Probe hosts for health instead of risking the shared connection

The health check shares each host's libvirt connection with the reconciles,
so it must not be able to take that connection away from them.

- A timed-out check no longer drops the link. The check goes through the new
  hostPool.probe, which abandons its RPCs when the context ends and leaves
  the connection alone, so an image upload or a Create running on the same
  link is not cut off. A link that really died is closed by the keepalive.
  One abandoned probe per host may stay waiting; until it returns, the next
  check reports Timeout at once (errProbeBusy).
- Waiting for a host's connection slot now ends with the caller's context.
  hostEntry used a mutex, so a check could wait out a reconcile redialing a
  dead host — minutes — past its own timeout. The slot is a one-slot channel
  and acquire bounds only the wait, never the uncontended take.
- --provider-check-timeout defaults to 0 and derives min(15s, interval).
  The flat 15s default contradicted the validation that the timeout may not
  exceed the interval, so --provider-check-interval=12s alone failed to
  start.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
This commit is contained in:
2026-09-12 05:04:45 +00:00
co-authored by Claude Opus 5
parent 0e456affdd
commit 4e5b5d62df
12 changed files with 349 additions and 43 deletions
+17 -11
View File
@@ -73,7 +73,8 @@ type Monitor struct {
// round runs at start: nothing races it, and early visibility is the
// point.
Interval time.Duration
// Timeout bounds each round (default 15s).
// Timeout bounds each round; 0 means 15s, or the interval when that is
// shorter.
Timeout time.Duration
now func() time.Time
@@ -92,31 +93,36 @@ func (m *Monitor) NeedLeaderElection() bool { return true }
// ValidateSchedule checks --provider-check-interval and
// --provider-check-timeout. The interval is 0 (off) or at least 10s, which
// is what keeps one status write per provider per round cheap; the timeout
// is positive and no longer than the interval.
// is what keeps one status write per provider per round cheap. The timeout
// is 0 (derived from the interval, see Monitor.Timeout) or positive and no
// longer than the interval.
func ValidateSchedule(interval, timeout time.Duration) error {
switch {
case interval == 0:
return nil
case interval < minInterval:
return fmt.Errorf("--provider-check-interval must be 0 (off) or at least %s, got %s", minInterval, interval)
case timeout <= 0 || timeout > interval:
return fmt.Errorf("--provider-check-timeout must be positive and at most the interval (%s), got %s", interval, timeout)
case timeout < 0 || timeout > interval:
return fmt.Errorf("--provider-check-timeout must be at most the interval (%s), got %s", interval, timeout)
}
return nil
}
func (m *Monitor) setDefaults() {
if m.Timeout == 0 {
m.Timeout = min(defaultTimeout, m.Interval)
}
if m.now == nil {
m.now = time.Now
}
}
// Start deletes the objects of providers that are no longer configured,
// then checks every Interval until ctx ends. The providers config is read
// once per process, so cleaning up once is enough. With checking off,
// every object goes, so none is left claiming a stale Ready.
func (m *Monitor) Start(ctx context.Context) error {
if m.Timeout == 0 {
m.Timeout = defaultTimeout
}
if m.now == nil {
m.now = time.Now
}
m.setDefaults()
log := logf.FromContext(ctx).WithName("provider-health")
ctx = logf.IntoContext(ctx, log)