Probe hosts for health instead of risking the shared connection

The health check shares each host's libvirt connection with the reconciles,
so it must not be able to take that connection away from them.

- A timed-out check no longer drops the link. The check goes through the new
  hostPool.probe, which abandons its RPCs when the context ends and leaves
  the connection alone, so an image upload or a Create running on the same
  link is not cut off. A link that really died is closed by the keepalive.
  One abandoned probe per host may stay waiting; until it returns, the next
  check reports Timeout at once (errProbeBusy).
- Waiting for a host's connection slot now ends with the caller's context.
  hostEntry used a mutex, so a check could wait out a reconcile redialing a
  dead host — minutes — past its own timeout. The slot is a one-slot channel
  and acquire bounds only the wait, never the uncontended take.
- --provider-check-timeout defaults to 0 and derives min(15s, interval).
  The flat 15s default contradicted the validation that the timeout may not
  exceed the interval, so --provider-check-interval=12s alone failed to
  start.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
This commit is contained in:
2026-09-12 05:04:45 +00:00
co-authored by Claude Opus 5
parent 0e456affdd
commit 4e5b5d62df
12 changed files with 349 additions and 43 deletions
@@ -312,3 +312,54 @@ contained `proxy_operator_provider_target_up{provider="kubernetes",target="apise
That exercises the generated RBAC, the API-reader path and the `main.go` gauge wiring
end to end. The two skipped specs are the Tempo-gated tracing specs; `TEMPO_URL` was not
set.
### Follow-up — the check must not cost the reconciles their connection
Reviewing the shared-connection decision after the automated validation turned up two
ways the health check could hurt the work it shares the link with, both of which the
plan's "a check against a hung host drops the shared connection" line accepted too
easily:
- A check that timed out went through `hostPool.call`, whose cancel path drops the
connection. A `libvirtd` slow to answer one `ConnectGetLibVersion` would therefore
cut off an image upload or a `Create` running on the same link.
- `hostEntry` was guarded by a `sync.Mutex`, so waiting for a host's slot could not be
interrupted. A reconcile redialing a host that is down holds the slot for as long as
the SSH dial takes — minutes on a blackholed address — and the whole check round
would wait behind it, past its own timeout.
So the check now calls `hostPool.probe` instead of `call`: on `ctx` ending it abandons
the RPC and leaves the connection alone (a link that is really dead is closed by the
keepalive anyway, which ends the abandoned `fn` too). One abandoned probe per host is
allowed to stay waiting — `hostEntry.probing` is an `atomic.Bool`, not a counter, so a
daemon that never answers cannot pile up goroutines; further probes get the new
`errProbeBusy`, which `checkReason` maps to `Timeout`. `hostEntry.mu` became a one-slot
channel with an `acquire(ctx)` that only bounds *waiting*: an uncontended caller takes a
free slot even with an expired context, so `call`'s behaviour is unchanged where nothing
is contended.
The second change came out of the same pass: `--provider-check-timeout` defaulted to a
flat 15s while the flag validation rejected a timeout longer than the interval, so
`--provider-check-interval=12s` alone failed to start. The flag now defaults to 0 and
`Monitor.setDefaults` derives `min(15s, Interval)`; `ValidateSchedule` accepts 0 and
rejects only negatives and timeouts past the interval.
Re-verified with the same commands as above:
```bash
go vet ./...
go test -race -count=1 ./internal/provider/kvm/ ./internal/providerhealth/
make test
# every package ok; internal/provider/kvm 92.2 %, internal/providerhealth 93.7 % coverage
```
Worth noting:
- `make lint` still cannot run, for the toolchain reason recorded above; unchanged by
this follow-up.
- The two new `kvm` tests are timing-shaped by necessity — they assert that a caller
with a 50ms deadline does not wait out a dial that is blocked on a channel. The
assertions are one-sided (`> 1s` fails) so a slow machine does not make them flaky.
- `docs/architecture.md`, `docs/kvm-provider.md` and `docs/testing.md` were corrected in
the same commit: the old text promised the drop-on-timeout behaviour that is now gone.
- The real-hypervisor commands from the plan's Verification section are still the user's
to run; nothing here replaces them.