The user ran the real-hypervisor checks against beelink and confirmed they
passed, which completes Step 7 of the provider health plan.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Leaving KVM_TEST_KNOWN_HOSTS unset silently disables host-key
verification for the run, so document it as part of the normal setup
rather than as an optional extra, and suggest a kvm_known_hosts holding
only the hypervisors.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The health check shares each host's libvirt connection with the reconciles,
so it must not be able to take that connection away from them.
- A timed-out check no longer drops the link. The check goes through the new
hostPool.probe, which abandons its RPCs when the context ends and leaves
the connection alone, so an image upload or a Create running on the same
link is not cut off. A link that really died is closed by the keepalive.
One abandoned probe per host may stay waiting; until it returns, the next
check reports Timeout at once (errProbeBusy).
- Waiting for a host's connection slot now ends with the caller's context.
hostEntry used a mutex, so a check could wait out a reconcile redialing a
dead host — minutes — past its own timeout. The slot is a one-slot channel
and acquire bounds only the wait, never the uncontended take.
- --provider-check-timeout defaults to 0 and derives min(15s, interval).
The flat 15s default contradicted the validation that the timeout may not
exceed the interval, so --provider-check-interval=12s alone failed to
start.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
make test (envtest, -race) and the kind e2e suite pass; the e2e
ProviderHealth for the kubernetes provider turned Ready within a second
of the manager starting and its gauge appeared in the scrape. make lint
is blocked by the local go1.27 toolchain crashing staticcheck on the
standard library, unrelated to this branch.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Architecture gets a provider-health section, component rows and four
decisions; the KVM provider doc gets a health-check section with the
reason table and the rotated-Secret gap; the testing guide lists the new
suites. CLAUDE.md's Architecture TODO and stale "only go.mod exists"
gotcha now point at the real docs, plus the RBAC-marker gotcha found
while wiring the monitor.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
--provider-check-interval (1m; 0 = off, else at least 10s) and
--provider-check-timeout (15s) are validated before the providers config
loads. Checkers are collected from the unwrapped providers in config
order, so checks never count as provider requests. Three scrape-time
gauges per {provider,target} read the monitor's snapshot: up, check
duration and check timestamp. The e2e suite waits for the kubernetes
ProviderHealth to turn Ready and asserts its gauge in the scrape.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
A leader-only runnable checks every configured provider concurrently
under one timeout per round (first round at start, then every interval),
keeps the latest results for the metrics collector, logs targets going
down, changing reason or coming back, and publishes one ProviderHealth
object per provider: created with a managed-by label when missing, then
a status merge patch. Per-target lastTransitionTime survives rounds in
which ready does not flip. Objects of providers that are no longer
configured are deleted once at start; with checks turned off, all are.
Reads go through the API reader, so no cluster-wide informer is needed.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Each configured host is checked concurrently over the pooled SSH+libvirt
connection: libvirt version and hostname prove the daemon answers, then
the configured pool must exist and be running, with its free space
reported. Dial failures are now tagged with their stage (SSH material,
TCP, SSH handshake, libvirt connect) by a dialError that keeps the
message and unwraps to the cause, so classification is unchanged and the
check can say why a host is unreachable. A pool verdict is decided after
the call returns so it never drops a working connection.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Checker is an optional interface next to Provider: Check returns one
verdict per target and never fails as a whole. The kubernetes provider
lists at most one managed Pod cluster-wide (the verb and scope GC relies
on); the GCP provider fetches a single page of at most one managed
instance through a new Probe seam method, since the SDK treats
MaxResults as a page size and the existing AggregatedList drains every
page.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
One operator-owned object per configured provider will carry the
periodic check result: a Ready condition, per-target verdicts with their
own transition times, and a ready/total count for kubectl get. There is
no spec; only a viewer role is shipped since the operator is the sole
writer. Written by hand (kubebuilder is not installed), registered in
the existing AddKnownTypes call rather than a second init().
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Periodic, leader-only checks of every configured provider (KVM per host
over the pooled SSH+libvirt connection including the storage pool,
kubernetes apiserver/RBAC, GCP credentials/API), surfaced through a new
cluster-scoped ProviderHealth CRD, Prometheus gauges and transition logs.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
kacerrandClaude Opus 5