Add the provider health Monitor
A leader-only runnable checks every configured provider concurrently under one timeout per round (first round at start, then every interval), keeps the latest results for the metrics collector, logs targets going down, changing reason or coming back, and publishes one ProviderHealth object per provider: created with a managed-by label when missing, then a status merge patch. Per-target lastTransitionTime survives rounds in which ready does not flip. Objects of providers that are no longer configured are deleted once at start; with checks turned off, all are. Reads go through the API reader, so no cluster-wide informer is needed. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
This commit is contained in:
@@ -5,7 +5,7 @@ Plan: [docs/plans/2026-09-11-1526-provider-health-checks.md](../plans/2026-09-11
|
||||
- [x] Step 1 — ProviderHealth API and CRD
|
||||
- [x] Step 2 — Contract plus the kubernetes and GCP checks
|
||||
- [x] Step 3 — KVM check
|
||||
- [ ] Step 4 — Monitor
|
||||
- [x] Step 4 — Monitor
|
||||
- [ ] Step 5 — Metrics and wiring
|
||||
- [ ] Step 6 — Docs
|
||||
- [ ] Step 7 — Validate, record, push, open the PR
|
||||
@@ -138,3 +138,53 @@ Worth noting:
|
||||
- `TestDialError_keepsClassification`.
|
||||
- **Not run yet:** `TestKVMCheck_realHost` in the integration file only compiles here.
|
||||
It needs beelink, so it goes on the Step 7 checklist for the user to run.
|
||||
|
||||
## Step 4 — Monitor
|
||||
|
||||
`internal/providerhealth` has the `Monitor` runnable plus the pure `buildStatus`.
|
||||
|
||||
- **Scheduling:** the Monitor is leader-only. It cleans up once at start, checks
|
||||
immediately, then checks on a ticker.
|
||||
- **Each round:** providers are checked concurrently under one timeout, then logged and
|
||||
published one after another. A failed write is logged, and the next round writes
|
||||
again.
|
||||
- **Reads:** objects are read through the reader it is given. `main.go` passes the
|
||||
manager's API reader (Step 5).
|
||||
|
||||
The first `make manifests` left `config/rbac/role.yaml` unchanged, with no error.
|
||||
I had put the `+kubebuilder:rbac` markers inside `Monitor`'s doc comment, and
|
||||
controller-gen only reads RBAC markers from package-level comment groups. So they were
|
||||
silently attached to the type and ignored. An empty `git diff config/rbac/role.yaml` was
|
||||
the only sign. Moving them into their own comment block fixed it:
|
||||
|
||||
```bash
|
||||
make manifests && git diff config/rbac/role.yaml
|
||||
# + providerhealths: create, delete, get, list
|
||||
# + providerhealths/status: get, patch
|
||||
```
|
||||
|
||||
Vet, race tests and the envtest round-trip (using the `bin/k8s/1.36.2-linux-amd64`
|
||||
binaries `make setup-envtest` had already fetched):
|
||||
|
||||
```bash
|
||||
go vet ./internal/providerhealth/ && go test -race -count=1 -v ./internal/providerhealth/
|
||||
# --- PASS: TestMonitor_envtestRoundTrip (6.45s) ... ok
|
||||
```
|
||||
|
||||
Worth noting:
|
||||
- **Fake client scope:** the fake client needs a RESTMapper that registers
|
||||
ProviderHealth as `RESTScopeRoot`. Its default mapper treats every kind as namespaced.
|
||||
- **Why the envtest calls `publish` directly:** `checkOnce` only logs write errors, so a
|
||||
schema rejection would look like a missing object. Calling `publish` makes the API
|
||||
server's answer the test failure.
|
||||
- Its first round writes a 2000-character message; the stored message is truncated to
|
||||
1024.
|
||||
- Its second round proves that only the target that flipped gets a new
|
||||
`lastTransitionTime`.
|
||||
- **No guard against a hung checker:** the Monitor trusts `Checker`'s contract to return
|
||||
once the context ends, and doesn't abandon a checker that ignores it. All three
|
||||
checkers stop when the context ends: KVM through `awaitOrCancel`, and kubernetes and
|
||||
GCP through their clients.
|
||||
- **Test renamed:** the planned `TestMonitor_timeoutBoundsBlockingChecker` is
|
||||
`TestMonitor_checkOnce_timeoutBoundsEachRound`. It also asserts that a fast provider
|
||||
still publishes next to a hung one.
|
||||
|
||||
Reference in New Issue
Block a user