Probe hosts for health instead of risking the shared connection
The health check shares each host's libvirt connection with the reconciles, so it must not be able to take that connection away from them. - A timed-out check no longer drops the link. The check goes through the new hostPool.probe, which abandons its RPCs when the context ends and leaves the connection alone, so an image upload or a Create running on the same link is not cut off. A link that really died is closed by the keepalive. One abandoned probe per host may stay waiting; until it returns, the next check reports Timeout at once (errProbeBusy). - Waiting for a host's connection slot now ends with the caller's context. hostEntry used a mutex, so a check could wait out a reconcile redialing a dead host — minutes — past its own timeout. The slot is a one-slot channel and acquire bounds only the wait, never the uncontended take. - --provider-check-timeout defaults to 0 and derives min(15s, interval). The flat 15s default contradicted the validation that the timeout may not exceed the interval, so --provider-check-interval=12s alone failed to start. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
This commit is contained in:
@@ -312,3 +312,54 @@ contained `proxy_operator_provider_target_up{provider="kubernetes",target="apise
|
||||
That exercises the generated RBAC, the API-reader path and the `main.go` gauge wiring
|
||||
end to end. The two skipped specs are the Tempo-gated tracing specs; `TEMPO_URL` was not
|
||||
set.
|
||||
|
||||
### Follow-up — the check must not cost the reconciles their connection
|
||||
|
||||
Reviewing the shared-connection decision after the automated validation turned up two
|
||||
ways the health check could hurt the work it shares the link with, both of which the
|
||||
plan's "a check against a hung host drops the shared connection" line accepted too
|
||||
easily:
|
||||
|
||||
- A check that timed out went through `hostPool.call`, whose cancel path drops the
|
||||
connection. A `libvirtd` slow to answer one `ConnectGetLibVersion` would therefore
|
||||
cut off an image upload or a `Create` running on the same link.
|
||||
- `hostEntry` was guarded by a `sync.Mutex`, so waiting for a host's slot could not be
|
||||
interrupted. A reconcile redialing a host that is down holds the slot for as long as
|
||||
the SSH dial takes — minutes on a blackholed address — and the whole check round
|
||||
would wait behind it, past its own timeout.
|
||||
|
||||
So the check now calls `hostPool.probe` instead of `call`: on `ctx` ending it abandons
|
||||
the RPC and leaves the connection alone (a link that is really dead is closed by the
|
||||
keepalive anyway, which ends the abandoned `fn` too). One abandoned probe per host is
|
||||
allowed to stay waiting — `hostEntry.probing` is an `atomic.Bool`, not a counter, so a
|
||||
daemon that never answers cannot pile up goroutines; further probes get the new
|
||||
`errProbeBusy`, which `checkReason` maps to `Timeout`. `hostEntry.mu` became a one-slot
|
||||
channel with an `acquire(ctx)` that only bounds *waiting*: an uncontended caller takes a
|
||||
free slot even with an expired context, so `call`'s behaviour is unchanged where nothing
|
||||
is contended.
|
||||
|
||||
The second change came out of the same pass: `--provider-check-timeout` defaulted to a
|
||||
flat 15s while the flag validation rejected a timeout longer than the interval, so
|
||||
`--provider-check-interval=12s` alone failed to start. The flag now defaults to 0 and
|
||||
`Monitor.setDefaults` derives `min(15s, Interval)`; `ValidateSchedule` accepts 0 and
|
||||
rejects only negatives and timeouts past the interval.
|
||||
|
||||
Re-verified with the same commands as above:
|
||||
|
||||
```bash
|
||||
go vet ./...
|
||||
go test -race -count=1 ./internal/provider/kvm/ ./internal/providerhealth/
|
||||
make test
|
||||
# every package ok; internal/provider/kvm 92.2 %, internal/providerhealth 93.7 % coverage
|
||||
```
|
||||
|
||||
Worth noting:
|
||||
- `make lint` still cannot run, for the toolchain reason recorded above; unchanged by
|
||||
this follow-up.
|
||||
- The two new `kvm` tests are timing-shaped by necessity — they assert that a caller
|
||||
with a 50ms deadline does not wait out a dial that is blocked on a channel. The
|
||||
assertions are one-sided (`> 1s` fails) so a slow machine does not make them flaky.
|
||||
- `docs/architecture.md`, `docs/kvm-provider.md` and `docs/testing.md` were corrected in
|
||||
the same commit: the old text promised the drop-on-timeout behaviour that is now gone.
|
||||
- The real-hypervisor commands from the plan's Verification section are still the user's
|
||||
to run; nothing here replaces them.
|
||||
|
||||
Reference in New Issue
Block a user