Wire the provider health monitor into the manager, with gauges
--provider-check-interval (1m; 0 = off, else at least 10s) and
--provider-check-timeout (15s) are validated before the providers config
loads. Checkers are collected from the unwrapped providers in config
order, so checks never count as provider requests. Three scrape-time
gauges per {provider,target} read the monitor's snapshot: up, check
duration and check timestamp. The e2e suite waits for the kubernetes
ProviderHealth to turn Ready and asserts its gauge in the scrape.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
This commit is contained in:
@@ -6,7 +6,7 @@ Plan: [docs/plans/2026-09-11-1526-provider-health-checks.md](../plans/2026-09-11
|
||||
- [x] Step 2 — Contract plus the kubernetes and GCP checks
|
||||
- [x] Step 3 — KVM check
|
||||
- [x] Step 4 — Monitor
|
||||
- [ ] Step 5 — Metrics and wiring
|
||||
- [x] Step 5 — Metrics and wiring
|
||||
- [ ] Step 6 — Docs
|
||||
- [ ] Step 7 — Validate, record, push, open the PR
|
||||
|
||||
@@ -188,3 +188,50 @@ Worth noting:
|
||||
- **Test renamed:** the planned `TestMonitor_timeoutBoundsBlockingChecker` is
|
||||
`TestMonitor_checkOnce_timeoutBoundsEachRound`. It also asserts that a fast provider
|
||||
still publishes next to a hung one.
|
||||
|
||||
## Step 5 — Metrics and wiring
|
||||
|
||||
`metrics.RegisterProviderTargets` registers a scrape-time collector for three gauges,
|
||||
each labelled `{provider,target}`: `up`, check duration and check timestamp.
|
||||
`Register`'s signature is unchanged.
|
||||
|
||||
`cmd/main.go`:
|
||||
- gains `--provider-check-interval` (1m) and `--provider-check-timeout` (15s);
|
||||
- validates them before even loading the providers config;
|
||||
- collects the `Checker`s from the unwrapped providers in config order, just before the
|
||||
metrics and tracing wrap loop;
|
||||
- adds the Monitor with the manager's API reader;
|
||||
- registers the collector through a small `providerTargetSamples` closure next to
|
||||
`proxyPhaseCounts`.
|
||||
|
||||
Build, vet and metrics tests:
|
||||
|
||||
```bash
|
||||
go build ./... && go vet ./cmd/... ./internal/metrics/ && go test -race -count=1 ./internal/metrics/
|
||||
```
|
||||
|
||||
To confirm bad flags fail fast, before anything touches the cluster, I ran the binary
|
||||
with a 5s interval. zap's development mode prints a stack trace after the error, hence
|
||||
the grep:
|
||||
|
||||
```bash
|
||||
go run ./cmd --providers-config=/dev/null --provider-check-interval=5s 2>&1 | grep -m1 "Invalid provider check flags"
|
||||
# ERROR setup Invalid provider check flags {"error": "--provider-check-interval must be 0 (off) or at least 10s, got 5s"}
|
||||
go vet -tags=e2e ./test/e2e/
|
||||
```
|
||||
|
||||
The e2e suite gets a new ordered spec, "should publish provider health". It waits for
|
||||
`providerhealth kubernetes` to report `Ready=True`. It runs before the existing metrics
|
||||
spec, which now also asserts
|
||||
`proxy_operator_provider_target_up{provider="kubernetes",target="apiserver"} 1`. By then
|
||||
the first round is known to be done, so the series is certain to exist.
|
||||
|
||||
Worth noting:
|
||||
- **Main-only wiring:** `providerTargetSamples` is an 8-line loop in `package main`,
|
||||
which has no test file. The e2e metrics assertion is its only coverage.
|
||||
- **The KVM entry isn't asserted in e2e:** the default providers config also ships
|
||||
`kvm-home` (beelink), which would report `SSHMaterialUnavailable` in kind. That entry
|
||||
is the user's own host, and asserting it would break the suite as soon as the default
|
||||
config changes.
|
||||
- **Manifest:** `config/manager/manager.yaml` only gained a comment about the two flags.
|
||||
The defaults need no args.
|
||||
|
||||
Reference in New Issue
Block a user