kvm2: an unreadable address is not a missing one (H3)
address() tried the guest agent then the host's ARP table and skipped any
error that was not a context error, returning ("", nil) after both. An
active domain with no IP maps to StateProvisioning; the controller clears
status.IP on that, and the health engine prunes the proxy's counters and
calls ForgetProxy in one tick once EffectiveHost() goes empty.
The damaging trigger was not the guest. The libvirt connection is pooled
per host, so when another reconcile's cancel or budget overrun killed it,
call returned "connection is closed" -- carrying no context sentinel, so
the check did not recognise it -- and Get returned a *successful*
instance in Provisioning with an empty IP. An unrelated proxy's timeout
dropped a healthy proxy out of rotation.
The discriminator was already in the spec without the conclusion being
drawn: during a normal boot on beelink exactly one source fails. The
agent errors because it is not up yet, while ARP answers without error
and with nothing, since the host never sees the guest's traffic. So one
source failing stays "no address yet"; every source failing, or a
connection already dead, is ErrTransient.
The explicit provider.ErrTransient wrap is load-bearing, not decoration.
Without it classify picks the first libvirt code out of the joined error,
and ErrInvalidArg, ErrAccessDenied and ErrNoSupport are Permanent -- an
address read failing would latch a running proxy as Failed, a worse
version of the harm being fixed.
Half of the finding is deliberately still open, and it is the half that
fires more often: a guest agent restarting inside a running VM still
demotes it, because that is indistinguishable from a VM still booting.
The discriminator is the previous IP, which the provider does not hold by
design and the controller already has. Follow-up finding to file against
proxy_controller.go:208-211 -- on StateProvisioning, keep an existing IP
and Provisioned condition for a few polls instead of clearing on the
first. The kvm2-local alternative (use the mark's createdAt as a
provisioning window) was rejected: createdAt is creation, not boot, and
it invents a deadline that belongs to the controller.
TestGet_addressErrorIsProvisioning failed against the new rule, correctly
-- it fails both sources with ErrAgentUnresponsive, and code 86 is
agent-specific, so the ARP source cannot produce it. Re-pointed as
TestGet_agentFailsAndARPEmptyIsProvisioning, the beelink boot it was
reaching for, which nothing covered before.
TestGet_everyAddressSourceFailingIsTransient covers the other side, both
sources erroring and the pooled link dying under the read; against the
unfixed code both return a Provisioning instance instead of an error.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
This commit is contained in:
@@ -546,7 +546,7 @@ waits 5 minutes on `ErrQuotaExceeded`, recreates on `ErrNotFound` from
|
||||
| `ErrPermanent` | placement values not in the catalogue; a malformed providerID; a host no longer in the config (from `Get`; `Delete` treats it as done); libvirt `ErrXMLError`, `ErrNoSupport`, `ErrInvalidArg`, `ErrAuthFailed`, `ErrAccessDenied` and `ErrNoStoragePool`; a pool that is not running; SSH authentication refused (`unable to authenticate`); a host key missing from known_hosts or changed (`knownhosts.KeyError`); a key or known_hosts file that exists but does not parse; a domain of the same name owned by another UID or carrying no kvm2 mark; a domain of the same name that is paused or pmsuspended; corrupt metadata on a domain; an image URL answering 4xx, a response without `Content-Length` or with length 0, a sha256 that does not match the configured one |
|
||||
| `ErrQuotaExceeded` | the pool-space precheck |
|
||||
| `ErrNotFound` | libvirt `ErrNoDomain` or `ErrNoStorageVol` from `Get`, `Delete` or `ListByTag`; inside `Create` the same codes are `ErrTransient`, because the contract forbids NotFound from Create |
|
||||
| `ErrTransient` | everything else: a missing key or known_hosts file (the Secret has not landed yet), TCP and network errors, context cancellation, an abandoned or budget-exceeded call, a connection killed by another caller, libvirt `ErrOperationInvalid` and every unrecognised libvirt code, an image URL answering 429 or 5xx or ending the body early, the wait for running expiring or the domain going shut off or crashed right after its start |
|
||||
| `ErrTransient` | everything else: a missing key or known_hosts file (the Secret has not landed yet), TCP and network errors, context cancellation, an abandoned or budget-exceeded call, a connection killed by another caller, libvirt `ErrOperationInvalid` and every unrecognised libvirt code, an image URL answering 429 or 5xx or ending the body early, every address source failing on an active domain, the wait for running expiring or the domain going shut off or crashed right after its start |
|
||||
|
||||
Error strings name paths, hosts, volumes and libvirt's own text. They
|
||||
never contain key bytes, the known_hosts content or cloud-init.
|
||||
@@ -627,7 +627,12 @@ never contain key bytes, the known_hosts content or cloud-init.
|
||||
vendor script's output in `/var/log/cloud-init-output.log` (a VM without
|
||||
outbound internet cannot install it). A user-data that disables
|
||||
vendor-data (`vendor_data: {enabled: false}`) also removes the agent, by
|
||||
design. `virsh domifaddr <name> --source agent` and `--source arp` show
|
||||
design. A host where the ARP source *errors* rather than answering with
|
||||
nothing is the one case this reads differently: with no source left to
|
||||
answer, a booting VM retries with `ErrTransient` instead of resting in
|
||||
`Provisioning`. It still converges; the giveaway is two "no addresses
|
||||
from source" lines per poll at `-v 2`.
|
||||
`virsh domifaddr <name> --source agent` and `--source arp` show
|
||||
what the operator sees, in that order.
|
||||
- **Debian 13 cloud images hang in GRUB under BIOS on a headless VM**
|
||||
([Debian bug #1111240](https://bugs.debian.org/1111240)), which is why
|
||||
|
||||
Reference in New Issue
Block a user