Probe hosts for health instead of risking the shared connection

The health check shares each host's libvirt connection with the reconciles,
so it must not be able to take that connection away from them.

- A timed-out check no longer drops the link. The check goes through the new
  hostPool.probe, which abandons its RPCs when the context ends and leaves
  the connection alone, so an image upload or a Create running on the same
  link is not cut off. A link that really died is closed by the keepalive.
  One abandoned probe per host may stay waiting; until it returns, the next
  check reports Timeout at once (errProbeBusy).
- Waiting for a host's connection slot now ends with the caller's context.
  hostEntry used a mutex, so a check could wait out a reconcile redialing a
  dead host — minutes — past its own timeout. The slot is a one-slot channel
  and acquire bounds only the wait, never the uncontended take.
- --provider-check-timeout defaults to 0 and derives min(15s, interval).
  The flat 15s default contradicted the validation that the timeout may not
  exceed the interval, so --provider-check-interval=12s alone failed to
  start.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
This commit is contained in:
2026-09-12 05:04:45 +00:00
co-authored by Claude Opus 5
parent 0e456affdd
commit 4e5b5d62df
12 changed files with 349 additions and 43 deletions
+3 -2
View File
@@ -146,8 +146,9 @@ func main() {
"Interval between provider health checks (can the cluster reach each hypervisor, API "+
"and account), published as ProviderHealth objects and metrics. 0 turns them off; "+
"otherwise at least 10s.")
flag.DurationVar(&providerCheckTimeout, "provider-check-timeout", 15*time.Second,
"Time limit for one round of provider health checks; at most the interval.")
flag.DurationVar(&providerCheckTimeout, "provider-check-timeout", 0,
"Time limit for one round of provider health checks: 15s, or the interval when that is "+
"shorter. If set, at most the interval.")
opts := zap.Options{
Development: true,