Probe hosts for health instead of risking the shared connection

The health check shares each host's libvirt connection with the reconciles,
so it must not be able to take that connection away from them.

- A timed-out check no longer drops the link. The check goes through the new
  hostPool.probe, which abandons its RPCs when the context ends and leaves
  the connection alone, so an image upload or a Create running on the same
  link is not cut off. A link that really died is closed by the keepalive.
  One abandoned probe per host may stay waiting; until it returns, the next
  check reports Timeout at once (errProbeBusy).
- Waiting for a host's connection slot now ends with the caller's context.
  hostEntry used a mutex, so a check could wait out a reconcile redialing a
  dead host — minutes — past its own timeout. The slot is a one-slot channel
  and acquire bounds only the wait, never the uncontended take.
- --provider-check-timeout defaults to 0 and derives min(15s, interval).
  The flat 15s default contradicted the validation that the timeout may not
  exceed the interval, so --provider-check-interval=12s alone failed to
  start.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
This commit is contained in:
2026-09-12 05:04:45 +00:00
co-authored by Claude Opus 5
parent 0e456affdd
commit 4e5b5d62df
12 changed files with 349 additions and 43 deletions
+9 -3
View File
@@ -378,7 +378,8 @@ target and never fails as a whole. All three providers do:
```text
Monitor.Start(ctx) ── leader-only; --provider-check-interval (1m; 0 = off,
│ otherwise ≥ 10s), --provider-check-timeout (15s)
│ otherwise ≥ 10s), --provider-check-timeout (15s, or
│ the interval if that is shorter)
├─ once: delete managed-by-labelled ProviderHealth objects whose provider
│ is not configured (all of them when checks are off) — the providers
│ config is read once per process
@@ -598,8 +599,13 @@ they read as choices, not accidents. Chronological by build step.
fresh dial would not pick up a rotated Secret anyway, since the SSH
material is cached. It would also open an sshd session per host per
minute. And a working cached connection is exactly what reconciles use.
The cost is that a check against a hung host drops the shared
connection, the same rule every cancelled reconcile already follows.
Sharing must not cost the reconciles anything, so the check calls
through `hostPool.probe`. A check that runs out of time abandons its
RPCs instead of closing the connection, which would kill an upload or a
Create running on it. At most one abandoned check per host is left
waiting, and until it returns, the next check reports `Timeout` at once.
Waiting for a host's connection slot also ends with the caller's
deadline, so a reconcile redialing a dead host cannot stall the round.
- **Provider transitions are logs, not Events**, in line with the
EventRecorder decision above. They log at Info with a `WARNING:` prefix,
and each target's `lastTransitionTime` in the object records when it