Probe hosts for health instead of risking the shared connection
The health check shares each host's libvirt connection with the reconciles, so it must not be able to take that connection away from them. - A timed-out check no longer drops the link. The check goes through the new hostPool.probe, which abandons its RPCs when the context ends and leaves the connection alone, so an image upload or a Create running on the same link is not cut off. A link that really died is closed by the keepalive. One abandoned probe per host may stay waiting; until it returns, the next check reports Timeout at once (errProbeBusy). - Waiting for a host's connection slot now ends with the caller's context. hostEntry used a mutex, so a check could wait out a reconcile redialing a dead host — minutes — past its own timeout. The slot is a one-slot channel and acquire bounds only the wait, never the uncontended take. - --provider-check-timeout defaults to 0 and derives min(15s, interval). The flat 15s default contradicted the validation that the timeout may not exceed the interval, so --provider-check-interval=12s alone failed to start. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
This commit is contained in:
@@ -292,7 +292,7 @@ shows its SSH `user`. A failing target says where the check broke:
|
||||
| `Unreachable` | the TCP connect failed, or the link died mid-check | DNS, Tailscale or a firewall between the cluster and `address` |
|
||||
| `SSHHandshakeFailed` | the host key is missing from known_hosts or has changed, or `user` rejected the key | the known_hosts entry; `user`'s `authorized_keys` |
|
||||
| `LibvirtConnectFailed` | SSH worked, the libvirt socket did not | `user` in the `libvirt` group, `libvirtd` running, the `socket` path |
|
||||
| `Timeout` | no answer within `--provider-check-timeout` (default 15s) | a hung `libvirtd` or a saturated link |
|
||||
| `Timeout` | no answer within `--provider-check-timeout` (default 15s), or the previous check of the host is still waiting | a hung `libvirtd`, a saturated link, or a reconcile stuck redialing the host |
|
||||
| `PoolNotFound` | there is no pool named `pool` | `virsh pool-list --all` |
|
||||
| `PoolNotActive` | the pool exists but is not running | `virsh pool-start <pool>` and `virsh pool-autostart <pool>` |
|
||||
| `LibvirtError` | any other libvirt error | the message carries libvirt's own text |
|
||||
@@ -302,9 +302,14 @@ Sharing the connection has two consequences:
|
||||
- The leader now keeps one SSH connection per host open from startup, with a
|
||||
keepalive every 15s. Before, the first connection opened with the first KVM
|
||||
reconcile.
|
||||
- A check that hits its timeout drops the host's connection, just as a
|
||||
cancelled reconcile already does. Calls in flight on a hung link then fail
|
||||
fast, and the next call redials.
|
||||
- A check that runs out of time gives up on its RPCs but leaves the
|
||||
connection alone, so an image upload or a Create running on it is not cut
|
||||
off. At most one abandoned check per host keeps waiting; until it
|
||||
returns, the next check reports `Timeout` straight away. A link that is
|
||||
really dead is closed by the keepalive within about two keepalive
|
||||
intervals.
|
||||
- A check never waits longer than its timeout for the host's connection,
|
||||
even while a reconcile holds it to redial a host that is down.
|
||||
|
||||
## Known gaps
|
||||
|
||||
|
||||
Reference in New Issue
Block a user