Probe hosts for health instead of risking the shared connection
The health check shares each host's libvirt connection with the reconciles, so it must not be able to take that connection away from them. - A timed-out check no longer drops the link. The check goes through the new hostPool.probe, which abandons its RPCs when the context ends and leaves the connection alone, so an image upload or a Create running on the same link is not cut off. A link that really died is closed by the keepalive. One abandoned probe per host may stay waiting; until it returns, the next check reports Timeout at once (errProbeBusy). - Waiting for a host's connection slot now ends with the caller's context. hostEntry used a mutex, so a check could wait out a reconcile redialing a dead host — minutes — past its own timeout. The slot is a one-slot channel and acquire bounds only the wait, never the uncontended take. - --provider-check-timeout defaults to 0 and derives min(15s, interval). The flat 15s default contradicted the validation that the timeout may not exceed the interval, so --provider-check-interval=12s alone failed to start. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
This commit is contained in:
@@ -378,7 +378,8 @@ target and never fails as a whole. All three providers do:
|
||||
|
||||
```text
|
||||
Monitor.Start(ctx) ── leader-only; --provider-check-interval (1m; 0 = off,
|
||||
│ otherwise ≥ 10s), --provider-check-timeout (15s)
|
||||
│ otherwise ≥ 10s), --provider-check-timeout (15s, or
|
||||
│ the interval if that is shorter)
|
||||
├─ once: delete managed-by-labelled ProviderHealth objects whose provider
|
||||
│ is not configured (all of them when checks are off) — the providers
|
||||
│ config is read once per process
|
||||
@@ -598,8 +599,13 @@ they read as choices, not accidents. Chronological by build step.
|
||||
fresh dial would not pick up a rotated Secret anyway, since the SSH
|
||||
material is cached. It would also open an sshd session per host per
|
||||
minute. And a working cached connection is exactly what reconciles use.
|
||||
The cost is that a check against a hung host drops the shared
|
||||
connection, the same rule every cancelled reconcile already follows.
|
||||
Sharing must not cost the reconciles anything, so the check calls
|
||||
through `hostPool.probe`. A check that runs out of time abandons its
|
||||
RPCs instead of closing the connection, which would kill an upload or a
|
||||
Create running on it. At most one abandoned check per host is left
|
||||
waiting, and until it returns, the next check reports `Timeout` at once.
|
||||
Waiting for a host's connection slot also ends with the caller's
|
||||
deadline, so a reconcile redialing a dead host cannot stall the round.
|
||||
- **Provider transitions are logs, not Events**, in line with the
|
||||
EventRecorder decision above. They log at Info with a `WARNING:` prefix,
|
||||
and each target's `lastTransitionTime` in the object records when it
|
||||
|
||||
Reference in New Issue
Block a user