Wire the provider health monitor into the manager, with gauges

--provider-check-interval (1m; 0 = off, else at least 10s) and
--provider-check-timeout (15s) are validated before the providers config
loads. Checkers are collected from the unwrapped providers in config
order, so checks never count as provider requests. Three scrape-time
gauges per {provider,target} read the monitor's snapshot: up, check
duration and check timestamp. The e2e suite waits for the kubernetes
ProviderHealth to turn Ready and asserts its gauge in the scrape.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
This commit is contained in:
2026-09-11 20:23:55 +00:00
co-authored by Claude Opus 5
parent 88a15ecdb5
commit 9a3ee963a0
6 changed files with 215 additions and 3 deletions
@@ -6,7 +6,7 @@ Plan: [docs/plans/2026-09-11-1526-provider-health-checks.md](../plans/2026-09-11
- [x] Step 2 — Contract plus the kubernetes and GCP checks
- [x] Step 3 — KVM check
- [x] Step 4 — Monitor
- [ ] Step 5 — Metrics and wiring
- [x] Step 5 — Metrics and wiring
- [ ] Step 6 — Docs
- [ ] Step 7 — Validate, record, push, open the PR
@@ -188,3 +188,50 @@ Worth noting:
- **Test renamed:** the planned `TestMonitor_timeoutBoundsBlockingChecker` is
`TestMonitor_checkOnce_timeoutBoundsEachRound`. It also asserts that a fast provider
still publishes next to a hung one.
## Step 5 — Metrics and wiring
`metrics.RegisterProviderTargets` registers a scrape-time collector for three gauges,
each labelled `{provider,target}`: `up`, check duration and check timestamp.
`Register`'s signature is unchanged.
`cmd/main.go`:
- gains `--provider-check-interval` (1m) and `--provider-check-timeout` (15s);
- validates them before even loading the providers config;
- collects the `Checker`s from the unwrapped providers in config order, just before the
metrics and tracing wrap loop;
- adds the Monitor with the manager's API reader;
- registers the collector through a small `providerTargetSamples` closure next to
`proxyPhaseCounts`.
Build, vet and metrics tests:
```bash
go build ./... && go vet ./cmd/... ./internal/metrics/ && go test -race -count=1 ./internal/metrics/
```
To confirm bad flags fail fast, before anything touches the cluster, I ran the binary
with a 5s interval. zap's development mode prints a stack trace after the error, hence
the grep:
```bash
go run ./cmd --providers-config=/dev/null --provider-check-interval=5s 2>&1 | grep -m1 "Invalid provider check flags"
# ERROR setup Invalid provider check flags {"error": "--provider-check-interval must be 0 (off) or at least 10s, got 5s"}
go vet -tags=e2e ./test/e2e/
```
The e2e suite gets a new ordered spec, "should publish provider health". It waits for
`providerhealth kubernetes` to report `Ready=True`. It runs before the existing metrics
spec, which now also asserts
`proxy_operator_provider_target_up{provider="kubernetes",target="apiserver"} 1`. By then
the first round is known to be done, so the series is certain to exist.
Worth noting:
- **Main-only wiring:** `providerTargetSamples` is an 8-line loop in `package main`,
which has no test file. The e2e metrics assertion is its only coverage.
- **The KVM entry isn't asserted in e2e:** the default providers config also ships
`kvm-home` (beelink), which would report `SSHMaterialUnavailable` in kind. That entry
is the user's own host, and asserting it would break the suite as soon as the default
config changes.
- **Manifest:** `config/manager/manager.yaml` only gained a comment about the two flags.
The defaults need no args.