Document the provider health checks

Architecture gets a provider-health section, component rows and four
decisions; the KVM provider doc gets a health-check section with the
reason table and the rotated-Secret gap; the testing guide lists the new
suites. CLAUDE.md's Architecture TODO and stale "only go.mod exists"
gotcha now point at the real docs, plus the RBAC-marker gotcha found
while wiring the monitor.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

docs/architecture.md

@@ -4,9 +4,9 @@
 
 | Component | Package | Runs as | Leader-elected | Role |
 |---|---|---|---|---|
-| Proxy CRD + helpers | `api/v1alpha1` | types | — | `Proxy` spec/status, CEL validation, defaulting, pure helpers |
+| Proxy + ProviderHealth CRDs | `api/v1alpha1` | types | — | `Proxy` spec/status, CEL validation, defaulting, pure helpers; the status-only, cluster-scoped `ProviderHealth` |
 | Reconciler | `internal/controller` | controller | yes (with the manager) | the state machine: provision, replace, delete, represent health |
-| Provider contract | `internal/provider` | library | — | `Provider` interface, error taxonomy, deterministic naming, config, metrics decorator |
+| Provider contract | `internal/provider` | library | — | `Provider` interface, optional `Checker`, error taxonomy, deterministic naming, config, metrics decorator |
 | kubernetes provider | `internal/provider/kubernetes` | library | — | real Squid pods in this cluster (local dev/CI) |
 | gcp provider | `internal/provider/gcp` | library | — | Compute Engine VMs, four API calls, fire-and-forget ops |
 | kvm provider | `internal/provider/kvm` | library | — | VMs on libvirt hosts over SSH-forwarded sockets; image cache, overlay disks, NoCloud seeds |
@@ -14,6 +14,7 @@
 | Lease store | `internal/lease` | Runnable (expiry sweep) | no | in-memory leases + cooldowns, single mutex |
 | Discovery API | `internal/discovery` | Runnable | no | HTTP list/lease/release/report on `:8090` |
 | Orphan GC | `internal/gc` | Runnable | yes | deletes tagged instances whose CR is gone |
+| Provider health monitor | `internal/providerhealth` | Runnable | yes | periodic provider checks → one `ProviderHealth` per provider, target gauges, transition logs |
 | Metrics | `internal/metrics` | library | — | explicit registration, scrape-time collectors |
 | Composition root | `cmd/main.go` | binary | — | flags, provider registry, wires everything onto one manager |
 
@@ -306,6 +307,8 @@ registries). Two kinds:
 - **Scrape-time collectors** — `proxy_operator_proxies{phase}` and
   `proxy_operator_leases_active` read the cache / lease store at every
   scrape; reconcile-incremented gauges inevitably drift and leak series.
+  The provider-check gauges (§11) read the health monitor's snapshot the
+  same way.
 - **Fed vectors** — `healthcheck_duration_seconds{proxy}` and
   `healthcheck_failures_total{proxy}` observe EVERY probe (status writes
   are transition-only; metrics carry the high-frequency signal), and the
@@ -348,7 +351,8 @@ incoming trace context):
 - Discovery API: one server span per request (named from the route
   pattern), incoming W3C `traceparent` honored so clients' traces continue
   into the operator; `/healthz` excluded.
-- `gc.sweep` per GC pass; `health.probe` per probe **only** with
+- `gc.sweep` per GC pass; `providerhealth.check` per round of provider
+  checks; `health.probe` per probe **only** with
   `--trace-health-probes` (default off — probes run ~1/s per proxy).
 
 Log correlation: logr sinks never see a context, so trace IDs ride on the
@@ -359,6 +363,62 @@ Re-deriving from the base rather than layering keeps zap from emitting
 duplicate keys on nested spans. GCP V(5) wire logs get the same keys from
 the slog handler's ctx.
 
+### 11. Provider health checks (`internal/providerhealth/`)
+
+Answers "can this cluster reach each provider's backend right now?"
+without waiting for a Proxy to fail. A provider opts in by implementing
+`provider.Checker`: `Check(ctx) []TargetStatus` returns one verdict per
+target and never fails as a whole. All three providers do:
+
+| Provider | Target | What one check does |
+|---|---|---|
+| kvm | each `kvm.hosts` entry | over the pooled SSH+libvirt connection: libvirt version and hostname, then the configured pool must exist and be running; reports pool capacity and free space ([details](kvm-provider.md#health-check)) |
+| kubernetes | `apiserver` | lists at most one managed Pod cluster-wide: the verb and scope `ListByTag` and GC rely on |
+| gcp | the project | one page of an aggregated instance list with `maxResults=1`: credentials, Compute API enabled, `compute.instances.list` |
+
+```text
+Monitor.Start(ctx)   ── leader-only; --provider-check-interval (1m; 0 = off,
+  │                     otherwise ≥ 10s), --provider-check-timeout (15s)
+  ├─ once: delete managed-by-labelled ProviderHealth objects whose provider
+  │   is not configured (all of them when checks are off) — the providers
+  │   config is read once per process
+  └─ at start, then every Interval: checkOnce
+       │ every provider's Check concurrently, under one context.WithTimeout
+       │ snapshot → the scrape-time gauges
+       │ log transitions: down or reason changed → "WARNING: provider target
+       │   not ready"; back up → "Provider target ready"; steady → silent
+       └ per provider: APIReader.Get → Create if missing → status merge patch
+```
+
+- **One status write per provider per round**, since `lastCheckTime`
+  moves every round. At the 10s minimum interval that is still far below
+  leader-election renewals. Proxy status avoids writes like this only
+  because its probes run once a second per proxy.
+- **Reads go through the API reader**, not the cache. The objects are
+  cluster-scoped; a cached Get would start a cluster-wide informer (and
+  need `watch`) even with `--proxy-namespace`.
+- **Checks bypass the provider decorators.** `cmd/main.go` collects the
+  Checkers before wrapping the providers in `WithTracing(WithMetrics(...))`,
+  so a check never counts as a provider request. KVM RPCs still appear in
+  the V(1) `libvirt call` trace, and each round is one
+  `providerhealth.check` span.
+- **Not wired to `/readyz`.** A dead hypervisor must not make the manager
+  unready while the other providers keep working.
+- **Gauges** `proxy_operator_provider_target_up`,
+  `proxy_operator_provider_check_duration_seconds` and
+  `proxy_operator_provider_check_timestamp_seconds`, each labelled
+  `{provider,target}`. Only the leader exports them.
+
+Illustrative output:
+
+```sh
+$ kubectl get providerhealth          # short name: ph
+NAME         TYPE         READY   TARGETS   LAST CHECK   AGE
+kubernetes   kubernetes   True    1/1       12s          3d
+kvm-home     kvm          False   0/1       12s          3d
+$ kubectl get ph kvm-home -o yaml     # per target: reason, message, details
+```
+
 ## Decisions
 
 Judgment calls the spec left open, and deliberate deviations — recorded so
@@ -528,3 +588,22 @@ they read as choices, not accidents. Chronological by build step.
   external sites. The kube-apiserver ignores incoming `traceparent` by
   design (public endpoint), so k8s client spans are leaves — in-process
   traces, not cross-process ones.
+- **Provider health is a status-only, cluster-scoped CRD** (user
+  decision). `kubectl get providerhealth` works without Prometheus, which
+  the deployment does not ship (its ServiceMonitor is commented out). There
+  is no spec: the providers config file stays the one place providers are
+  configured, and the object only reports on it. The gauges carry the same
+  verdicts for alerting.
+- **KVM checks reuse the pooled connection instead of dialing fresh.** A
+  fresh dial would not pick up a rotated Secret anyway, since the SSH
+  material is cached. It would also open an sshd session per host per
+  minute. And a working cached connection is exactly what reconciles use.
+  The cost is that a check against a hung host drops the shared
+  connection, the same rule every cancelled reconcile already follows.
+- **Provider transitions are logs, not Events**, in line with the
+  EventRecorder decision above. They log at Info with a `WARNING:` prefix,
+  and each target's `lastTransitionTime` in the object records when it
+  flipped.
+- **A provider name that is not a valid object name** gets metrics but no
+  `ProviderHealth` object, with a warning at start. Tightening config
+  validation instead would crash-loop existing deployments on upgrade.