Adds scheduled health checks for every configured provider, reported in a new cluster-scoped ProviderHealth resource and as Prometheus gauges.
What's in it
ProviderHealth CRD (kubectl get ph): one object per provider, with a Ready condition, a reason, and per-target verdicts.
A Checker contract with three checks:
kubernetes: lists one object from the API server.
GCP: fetches exactly one instance page.
KVM: checks SSH/libvirt reachability and the storage pool on each host, in parallel.
Monitor runnable: runs the checks on --provider-check-interval, bounds each with --provider-check-timeout (defaults to min(15s, interval)), and logs only state transitions.
The KVM check doesn't disrupt reconciles: it uses hostPool.probe, which gives up on a slow RPC without dropping the shared connection, and waiting for a host slot can be interrupted.
Docs:docs/architecture.md, docs/kvm-provider.md, docs/testing.md (integration tests now use a dedicated KVM_TEST_KNOWN_HOSTS).
Also on this branch: the plan for the provider blueprint doc (docs/plans/2026-09-16-0841-provider-blueprint.md). That work starts on a new branch after this merges.
Validation
make test (envtest, -race): all packages pass. Coverage: internal/providerhealth 93.7 %, internal/provider/kvm 92.2 %.
make test-e2e on kind: 3 passed, 2 skipped (Tempo-gated). The kubernetes ProviderHealth went Ready=True, and its gauge showed up in the metrics scrape.
Real-hypervisor KVM checks against beelink: run by the author, passed.
make lint couldn't run. With the local Go 1.27 toolchain, the bundled staticcheck panics on the standard library before reaching any repo package. This isn't caused by this branch; go vet is clean with the default, integration and e2e tags.
Adds scheduled health checks for every configured provider, reported in a new cluster-scoped `ProviderHealth` resource and as Prometheus gauges.
## What's in it
- **`ProviderHealth` CRD** (`kubectl get ph`): one object per provider, with a `Ready` condition, a reason, and per-target verdicts.
- **A `Checker` contract** with three checks:
- **kubernetes:** lists one object from the API server.
- **GCP:** fetches exactly one instance page.
- **KVM:** checks SSH/libvirt reachability and the storage pool on each host, in parallel.
- **Monitor runnable:** runs the checks on `--provider-check-interval`, bounds each with `--provider-check-timeout` (defaults to `min(15s, interval)`), and logs only state transitions.
- **Metrics:** `proxy_operator_provider_target_up`, `proxy_operator_provider_check_duration_seconds`, `proxy_operator_provider_check_timestamp_seconds`.
- **The KVM check doesn't disrupt reconciles:** it uses `hostPool.probe`, which gives up on a slow RPC without dropping the shared connection, and waiting for a host slot can be interrupted.
- **Docs:** `docs/architecture.md`, `docs/kvm-provider.md`, `docs/testing.md` (integration tests now use a dedicated `KVM_TEST_KNOWN_HOSTS`).
- **Also on this branch:** the plan for the provider blueprint doc (`docs/plans/2026-09-16-0841-provider-blueprint.md`). That work starts on a new branch after this merges.
## Validation
- `make test` (envtest, `-race`): all packages pass. Coverage: `internal/providerhealth` 93.7 %, `internal/provider/kvm` 92.2 %.
- `make test-e2e` on kind: 3 passed, 2 skipped (Tempo-gated). The `kubernetes` ProviderHealth went `Ready=True`, and its gauge showed up in the metrics scrape.
- Real-hypervisor KVM checks against beelink: run by the author, passed.
- **`make lint` couldn't run.** With the local Go 1.27 toolchain, the bundled staticcheck panics on the standard library before reaching any repo package. This isn't caused by this branch; `go vet` is clean with the default, `integration` and `e2e` tags.
Plan: `docs/plans/2026-09-11-1526-provider-health-checks.md`. Execution log: `docs/plans-executions/2026-09-11-1526-provider-health-checks.md`.
🤖 Generated with [Claude Code](https://claude.com/claude-code)
Periodic, leader-only checks of every configured provider (KVM per host
over the pooled SSH+libvirt connection including the storage pool,
kubernetes apiserver/RBAC, GCP credentials/API), surfaced through a new
cluster-scoped ProviderHealth CRD, Prometheus gauges and transition logs.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
One operator-owned object per configured provider will carry the
periodic check result: a Ready condition, per-target verdicts with their
own transition times, and a ready/total count for kubectl get. There is
no spec; only a viewer role is shipped since the operator is the sole
writer. Written by hand (kubebuilder is not installed), registered in
the existing AddKnownTypes call rather than a second init().
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Checker is an optional interface next to Provider: Check returns one
verdict per target and never fails as a whole. The kubernetes provider
lists at most one managed Pod cluster-wide (the verb and scope GC relies
on); the GCP provider fetches a single page of at most one managed
instance through a new Probe seam method, since the SDK treats
MaxResults as a page size and the existing AggregatedList drains every
page.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Each configured host is checked concurrently over the pooled SSH+libvirt
connection: libvirt version and hostname prove the daemon answers, then
the configured pool must exist and be running, with its free space
reported. Dial failures are now tagged with their stage (SSH material,
TCP, SSH handshake, libvirt connect) by a dialError that keeps the
message and unwraps to the cause, so classification is unchanged and the
check can say why a host is unreachable. A pool verdict is decided after
the call returns so it never drops a working connection.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
A leader-only runnable checks every configured provider concurrently
under one timeout per round (first round at start, then every interval),
keeps the latest results for the metrics collector, logs targets going
down, changing reason or coming back, and publishes one ProviderHealth
object per provider: created with a managed-by label when missing, then
a status merge patch. Per-target lastTransitionTime survives rounds in
which ready does not flip. Objects of providers that are no longer
configured are deleted once at start; with checks turned off, all are.
Reads go through the API reader, so no cluster-wide informer is needed.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
--provider-check-interval (1m; 0 = off, else at least 10s) and
--provider-check-timeout (15s) are validated before the providers config
loads. Checkers are collected from the unwrapped providers in config
order, so checks never count as provider requests. Three scrape-time
gauges per {provider,target} read the monitor's snapshot: up, check
duration and check timestamp. The e2e suite waits for the kubernetes
ProviderHealth to turn Ready and asserts its gauge in the scrape.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Architecture gets a provider-health section, component rows and four
decisions; the KVM provider doc gets a health-check section with the
reason table and the rotated-Secret gap; the testing guide lists the new
suites. CLAUDE.md's Architecture TODO and stale "only go.mod exists"
gotcha now point at the real docs, plus the RBAC-marker gotcha found
while wiring the monitor.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
make test (envtest, -race) and the kind e2e suite pass; the e2e
ProviderHealth for the kubernetes provider turned Ready within a second
of the manager starting and its gauge appeared in the scrape. make lint
is blocked by the local go1.27 toolchain crashing staticcheck on the
standard library, unrelated to this branch.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The health check shares each host's libvirt connection with the reconciles,
so it must not be able to take that connection away from them.
- A timed-out check no longer drops the link. The check goes through the new
hostPool.probe, which abandons its RPCs when the context ends and leaves
the connection alone, so an image upload or a Create running on the same
link is not cut off. A link that really died is closed by the keepalive.
One abandoned probe per host may stay waiting; until it returns, the next
check reports Timeout at once (errProbeBusy).
- Waiting for a host's connection slot now ends with the caller's context.
hostEntry used a mutex, so a check could wait out a reconcile redialing a
dead host — minutes — past its own timeout. The slot is a one-slot channel
and acquire bounds only the wait, never the uncontended take.
- --provider-check-timeout defaults to 0 and derives min(15s, interval).
The flat 15s default contradicted the validation that the timeout may not
exceed the interval, so --provider-check-interval=12s alone failed to
start.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Leaving KVM_TEST_KNOWN_HOSTS unset silently disables host-key
verification for the run, so document it as part of the normal setup
rather than as an optional extra, and suggest a kvm_known_hosts holding
only the hypervisors.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The user ran the real-hypervisor checks against beelink and confirmed they
passed, which completes Step 7 of the provider health plan.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
kacerr
merged commit cacd9583a3 into main2026-09-16 10:46:47 +02:00
Blocking a user prevents them from interacting with repositories, such as opening or commenting on pull requests or issues. Learn more about blocking a user.
Adds scheduled health checks for every configured provider, reported in a new cluster-scoped
ProviderHealthresource and as Prometheus gauges.What's in it
ProviderHealthCRD (kubectl get ph): one object per provider, with aReadycondition, a reason, and per-target verdicts.Checkercontract with three checks:--provider-check-interval, bounds each with--provider-check-timeout(defaults tomin(15s, interval)), and logs only state transitions.proxy_operator_provider_target_up,proxy_operator_provider_check_duration_seconds,proxy_operator_provider_check_timestamp_seconds.hostPool.probe, which gives up on a slow RPC without dropping the shared connection, and waiting for a host slot can be interrupted.docs/architecture.md,docs/kvm-provider.md,docs/testing.md(integration tests now use a dedicatedKVM_TEST_KNOWN_HOSTS).docs/plans/2026-09-16-0841-provider-blueprint.md). That work starts on a new branch after this merges.Validation
make test(envtest,-race): all packages pass. Coverage:internal/providerhealth93.7 %,internal/provider/kvm92.2 %.make test-e2eon kind: 3 passed, 2 skipped (Tempo-gated). ThekubernetesProviderHealth wentReady=True, and its gauge showed up in the metrics scrape.make lintcouldn't run. With the local Go 1.27 toolchain, the bundled staticcheck panics on the standard library before reaching any repo package. This isn't caused by this branch;go vetis clean with the default,integrationande2etags.Plan:
docs/plans/2026-09-11-1526-provider-health-checks.md. Execution log:docs/plans-executions/2026-09-11-1526-provider-health-checks.md.🤖 Generated with Claude Code
--provider-check-interval (1m; 0 = off, else at least 10s) and --provider-check-timeout (15s) are validated before the providers config loads. Checkers are collected from the unwrapped providers in config order, so checks never count as provider requests. Three scrape-time gauges per {provider,target} read the monitor's snapshot: up, check duration and check timestamp. The e2e suite waits for the kubernetes ProviderHealth to turn Ready and asserts its gauge in the scrape. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>