Add the provider health Monitor

A leader-only runnable checks every configured provider concurrently
under one timeout per round (first round at start, then every interval),
keeps the latest results for the metrics collector, logs targets going
down, changing reason or coming back, and publishes one ProviderHealth
object per provider: created with a managed-by label when missing, then
a status merge patch. Per-target lastTransitionTime survives rounds in
which ready does not flip. Objects of providers that are no longer
configured are deleted once at start; with checks turned off, all are.
Reads go through the API reader, so no cluster-wide informer is needed.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
This commit is contained in:
2026-09-11 20:20:52 +00:00
co-authored by Claude Opus 5
parent 04dd2c0284
commit 88a15ecdb5
7 changed files with 1044 additions and 1 deletions
@@ -5,7 +5,7 @@ Plan: [docs/plans/2026-09-11-1526-provider-health-checks.md](../plans/2026-09-11
- [x] Step 1 — ProviderHealth API and CRD
- [x] Step 2 — Contract plus the kubernetes and GCP checks
- [x] Step 3 — KVM check
- [ ] Step 4 — Monitor
- [x] Step 4 — Monitor
- [ ] Step 5 — Metrics and wiring
- [ ] Step 6 — Docs
- [ ] Step 7 — Validate, record, push, open the PR
@@ -138,3 +138,53 @@ Worth noting:
- `TestDialError_keepsClassification`.
- **Not run yet:** `TestKVMCheck_realHost` in the integration file only compiles here.
It needs beelink, so it goes on the Step 7 checklist for the user to run.
## Step 4 — Monitor
`internal/providerhealth` has the `Monitor` runnable plus the pure `buildStatus`.
- **Scheduling:** the Monitor is leader-only. It cleans up once at start, checks
immediately, then checks on a ticker.
- **Each round:** providers are checked concurrently under one timeout, then logged and
published one after another. A failed write is logged, and the next round writes
again.
- **Reads:** objects are read through the reader it is given. `main.go` passes the
manager's API reader (Step 5).
The first `make manifests` left `config/rbac/role.yaml` unchanged, with no error.
I had put the `+kubebuilder:rbac` markers inside `Monitor`'s doc comment, and
controller-gen only reads RBAC markers from package-level comment groups. So they were
silently attached to the type and ignored. An empty `git diff config/rbac/role.yaml` was
the only sign. Moving them into their own comment block fixed it:
```bash
make manifests && git diff config/rbac/role.yaml
# + providerhealths: create, delete, get, list
# + providerhealths/status: get, patch
```
Vet, race tests and the envtest round-trip (using the `bin/k8s/1.36.2-linux-amd64`
binaries `make setup-envtest` had already fetched):
```bash
go vet ./internal/providerhealth/ && go test -race -count=1 -v ./internal/providerhealth/
# --- PASS: TestMonitor_envtestRoundTrip (6.45s) ... ok
```
Worth noting:
- **Fake client scope:** the fake client needs a RESTMapper that registers
ProviderHealth as `RESTScopeRoot`. Its default mapper treats every kind as namespaced.
- **Why the envtest calls `publish` directly:** `checkOnce` only logs write errors, so a
schema rejection would look like a missing object. Calling `publish` makes the API
server's answer the test failure.
- Its first round writes a 2000-character message; the stored message is truncated to
1024.
- Its second round proves that only the target that flipped gets a new
`lastTransitionTime`.
- **No guard against a hung checker:** the Monitor trusts `Checker`'s contract to return
once the context ends, and doesn't abandon a checker that ignores it. All three
checkers stop when the context ends: KVM through `awaitOrCancel`, and kubernetes and
GCP through their clients.
- **Test renamed:** the planned `TestMonitor_timeoutBoundsBlockingChecker` is
`TestMonitor_checkOnce_timeoutBoundsEachRound`. It also asserts that a fast provider
still publishes next to a hung one.