kacerr/egress-proxies-operator
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

Probe hosts for health instead of risking the shared connection

Browse Source 
The health check shares each host's libvirt connection with the reconciles,
so it must not be able to take that connection away from them.

- A timed-out check no longer drops the link. The check goes through the new
  hostPool.probe, which abandons its RPCs when the context ends and leaves
  the connection alone, so an image upload or a Create running on the same
  link is not cut off. A link that really died is closed by the keepalive.
  One abandoned probe per host may stay waiting; until it returns, the next
  check reports Timeout at once (errProbeBusy).
- Waiting for a host's connection slot now ends with the caller's context.
  hostEntry used a mutex, so a check could wait out a reconcile redialing a
  dead host — minutes — past its own timeout. The slot is a one-slot channel
  and acquire bounds only the wait, never the uncontended take.
- --provider-check-timeout defaults to 0 and derives min(15s, interval).
  The flat 15s default contradicted the validation that the timeout may not
  exceed the interval, so --provider-check-interval=12s alone failed to
  start.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
This commit is contained in:
kacerr
2026-09-12 05:04:45 +00:00
co-authored by Claude Opus 5
parent 0e456affdd
commit 4e5b5d62df
12 changed files with 349 additions and 43 deletions
Download Patch File Download Diff File Expand all files Collapse all files
internal/provider/kvm/check.go
+8 -6
View File
@@ -48,10 +48,12 @@ func (p *Provider) Check(ctx context.Context) []provider.TargetStatus {
return out
}
// checkHost returns the RPC errors from fn untouched: call drops the
// shared connection on anything that is not a libvirt error, and a pool
// that exists but is stopped says nothing about the link. facts is read
// only after call returned nil, since an abandoned fn may still write it.
// checkHost goes through probe, so a check that runs out of time abandons
// its RPCs rather than closing the connection the reconciles share. fn
// returns the RPC errors untouched, since a transport error still drops the
// link; a pool that exists but is stopped says nothing about the link, so
// that verdict is reached afterwards. facts is read only after probe
// returned nil, since an abandoned fn may still write it.
func (p *Provider) checkHost(ctx context.Context, host provider.KVMHost) provider.TargetStatus {
ts := provider.TargetStatus{
Name: host.Name,
@@ -60,7 +62,7 @@ func (p *Provider) checkHost(ctx context.Context, host provider.KVMHost) provide
}
start := time.Now()
var facts hostFacts
err := p.hosts.call(ctx, p.logger(ctx), host, func(api libvirtAPI) error {
err := p.hosts.probe(ctx, p.logger(ctx), host, func(api libvirtAPI) error {
version, err := api.ConnectGetLibVersion()
if err != nil {
return err
@@ -125,7 +127,7 @@ func checkReason(err error) string {
}
}
switch {
case errors.Is(err, context.DeadlineExceeded), errors.Is(err, context.Canceled):
case errors.Is(err, context.DeadlineExceeded), errors.Is(err, context.Canceled), errors.Is(err, errProbeBusy):
return provider.ReasonTimeout
case hasCode(err, libvirt.ErrNoStoragePool):
return reasonPoolNotFound