kvm2: an unreadable address is not a missing one (H3)
address() tried the guest agent then the host's ARP table and skipped any
error that was not a context error, returning ("", nil) after both. An
active domain with no IP maps to StateProvisioning; the controller clears
status.IP on that, and the health engine prunes the proxy's counters and
calls ForgetProxy in one tick once EffectiveHost() goes empty.
The damaging trigger was not the guest. The libvirt connection is pooled
per host, so when another reconcile's cancel or budget overrun killed it,
call returned "connection is closed" -- carrying no context sentinel, so
the check did not recognise it -- and Get returned a *successful*
instance in Provisioning with an empty IP. An unrelated proxy's timeout
dropped a healthy proxy out of rotation.
The discriminator was already in the spec without the conclusion being
drawn: during a normal boot on beelink exactly one source fails. The
agent errors because it is not up yet, while ARP answers without error
and with nothing, since the host never sees the guest's traffic. So one
source failing stays "no address yet"; every source failing, or a
connection already dead, is ErrTransient.
The explicit provider.ErrTransient wrap is load-bearing, not decoration.
Without it classify picks the first libvirt code out of the joined error,
and ErrInvalidArg, ErrAccessDenied and ErrNoSupport are Permanent -- an
address read failing would latch a running proxy as Failed, a worse
version of the harm being fixed.
Half of the finding is deliberately still open, and it is the half that
fires more often: a guest agent restarting inside a running VM still
demotes it, because that is indistinguishable from a VM still booting.
The discriminator is the previous IP, which the provider does not hold by
design and the controller already has. Follow-up finding to file against
proxy_controller.go:208-211 -- on StateProvisioning, keep an existing IP
and Provisioned condition for a few polls instead of clearing on the
first. The kvm2-local alternative (use the mark's createdAt as a
provisioning window) was rejected: createdAt is creation, not boot, and
it invents a deadline that belongs to the controller.
TestGet_addressErrorIsProvisioning failed against the new rule, correctly
-- it fails both sources with ErrAgentUnresponsive, and code 86 is
agent-specific, so the ARP source cannot produce it. Re-pointed as
TestGet_agentFailsAndARPEmptyIsProvisioning, the beelink boot it was
reaching for, which nothing covered before.
TestGet_everyAddressSourceFailingIsTransient covers the other side, both
sources erroring and the pooled link dying under the read; against the
unfixed code both return a Provisioning instance instead of an error.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
This commit is contained in:
@@ -98,22 +98,36 @@ var addressSources = []struct {
|
||||
}
|
||||
|
||||
// address returns the first usable IPv4 from the first source that yields
|
||||
// one, or "" for "no address yet". A source that fails is logged and
|
||||
// skipped; only ctx ending is an error, so a cancelled Get does not read
|
||||
// as Provisioning.
|
||||
// one, or "" for "no address yet". One source failing is normal and is
|
||||
// skipped: a booting VM has no agent yet, while the host's ARP table
|
||||
// simply reports nothing, since it never sees traffic addressed
|
||||
// elsewhere. A connection already gone, and every source failing, are
|
||||
// errors instead — otherwise a link some other reconcile killed reads as
|
||||
// a running VM that lost its address, and the controller clears its IP
|
||||
// and the health engine drops its accumulated state.
|
||||
func address(ctx context.Context, log logr.Logger, be backend, dom libvirt.Domain) (string, error) {
|
||||
errs := make([]error, 0, len(addressSources))
|
||||
for _, s := range addressSources {
|
||||
ifaces, err := be.DomainInterfaceAddresses(ctx, dom, uint32(s.src), 0)
|
||||
if err != nil {
|
||||
if errors.Is(err, context.Canceled) || errors.Is(err, context.DeadlineExceeded) {
|
||||
return "", fmt.Errorf("reading addresses of domain %s: %w", dom.Name, err)
|
||||
if isDead(be) || errors.Is(err, context.Canceled) || errors.Is(err, context.DeadlineExceeded) {
|
||||
return "", fmt.Errorf("reading addresses of domain %s from the %s source: %w", dom.Name, s.name, err)
|
||||
}
|
||||
log.V(2).Info("no addresses from source", "source", s.name, "error", err.Error())
|
||||
errs = append(errs, fmt.Errorf("%s source: %w", s.name, err))
|
||||
continue
|
||||
}
|
||||
if ip := firstIPv4(ifaces); ip != "" {
|
||||
return ip, nil
|
||||
}
|
||||
}
|
||||
// Transient explicitly: classify would otherwise pick a libvirt code
|
||||
// out of the joined error, and ErrInvalidArg, ErrAccessDenied and
|
||||
// ErrNoSupport are Permanent, which would latch a running proxy as
|
||||
// Failed over an address read.
|
||||
if len(errs) == len(addressSources) {
|
||||
return "", fmt.Errorf("%w: reading addresses of domain %s: every source failed: %w",
|
||||
provider.ErrTransient, dom.Name, errors.Join(errs...))
|
||||
}
|
||||
return "", nil
|
||||
}
|
||||
|
||||
Reference in New Issue
Block a user