kvm2: an unreadable address is not a missing one (H3)

address() tried the guest agent then the host's ARP table and skipped any
error that was not a context error, returning ("", nil) after both. An
active domain with no IP maps to StateProvisioning; the controller clears
status.IP on that, and the health engine prunes the proxy's counters and
calls ForgetProxy in one tick once EffectiveHost() goes empty.

The damaging trigger was not the guest. The libvirt connection is pooled
per host, so when another reconcile's cancel or budget overrun killed it,
call returned "connection is closed" -- carrying no context sentinel, so
the check did not recognise it -- and Get returned a *successful*
instance in Provisioning with an empty IP. An unrelated proxy's timeout
dropped a healthy proxy out of rotation.

The discriminator was already in the spec without the conclusion being
drawn: during a normal boot on beelink exactly one source fails. The
agent errors because it is not up yet, while ARP answers without error
and with nothing, since the host never sees the guest's traffic. So one
source failing stays "no address yet"; every source failing, or a
connection already dead, is ErrTransient.

The explicit provider.ErrTransient wrap is load-bearing, not decoration.
Without it classify picks the first libvirt code out of the joined error,
and ErrInvalidArg, ErrAccessDenied and ErrNoSupport are Permanent -- an
address read failing would latch a running proxy as Failed, a worse
version of the harm being fixed.

Half of the finding is deliberately still open, and it is the half that
fires more often: a guest agent restarting inside a running VM still
demotes it, because that is indistinguishable from a VM still booting.
The discriminator is the previous IP, which the provider does not hold by
design and the controller already has. Follow-up finding to file against
proxy_controller.go:208-211 -- on StateProvisioning, keep an existing IP
and Provisioned condition for a few polls instead of clearing on the
first. The kvm2-local alternative (use the mark's createdAt as a
provisioning window) was rejected: createdAt is creation, not boot, and
it invents a deadline that belongs to the controller.

TestGet_addressErrorIsProvisioning failed against the new rule, correctly
-- it fails both sources with ErrAgentUnresponsive, and code 86 is
agent-specific, so the ARP source cannot produce it. Re-pointed as
TestGet_agentFailsAndARPEmptyIsProvisioning, the beelink boot it was
reaching for, which nothing covered before.
TestGet_everyAddressSourceFailingIsTransient covers the other side, both
sources erroring and the pooled link dying under the read; against the
unfixed code both return a Provisioning instance instead of an error.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

docs/kvm2-provider-spec.md

@@ -146,8 +146,12 @@ cluster can route to. The address comes from `DomainInterfaceAddresses`, guest
 agent source first, then the ARP table (on beelink the agent answered about
 20 s after start where ARP never did, because the host itself does not talk to
 the guest). The first IPv4 that is not loopback, link-local or unspecified is
-reported; errors from either source mean "no address yet" and `Get` returns
+reported. One source failing means "no address yet" and `Get` returns
-`StateProvisioning`. The guest needs the agent: it is installed by the
+`StateProvisioning`, which is the ordinary boot on beelink: the agent is not up
+while ARP answers with nothing. A connection already gone, or every source
+failing, is `ErrTransient` instead — a running VM whose address cannot be read
+is not a VM without an address, and reporting it as one would make the
+reconciler clear its IP and the health engine drop its counters. The guest needs the agent: it is installed by the
 vendor-data script (entry 5) unless `installGuestAgent: false`, which needs an
 image that ships it or a network where ARP sees guest traffic. `Port` is not
 used by the provider; nothing on the host filters it. Reachability is a

docs/kvm2-provider.md

@@ -546,7 +546,7 @@ waits 5 minutes on `ErrQuotaExceeded`, recreates on `ErrNotFound` from
 | `ErrPermanent` | placement values not in the catalogue; a malformed providerID; a host no longer in the config (from `Get`; `Delete` treats it as done); libvirt `ErrXMLError`, `ErrNoSupport`, `ErrInvalidArg`, `ErrAuthFailed`, `ErrAccessDenied` and `ErrNoStoragePool`; a pool that is not running; SSH authentication refused (`unable to authenticate`); a host key missing from known_hosts or changed (`knownhosts.KeyError`); a key or known_hosts file that exists but does not parse; a domain of the same name owned by another UID or carrying no kvm2 mark; a domain of the same name that is paused or pmsuspended; corrupt metadata on a domain; an image URL answering 4xx, a response without `Content-Length` or with length 0, a sha256 that does not match the configured one |
 | `ErrQuotaExceeded` | the pool-space precheck |
 | `ErrNotFound` | libvirt `ErrNoDomain` or `ErrNoStorageVol` from `Get`, `Delete` or `ListByTag`; inside `Create` the same codes are `ErrTransient`, because the contract forbids NotFound from Create |
-| `ErrTransient` | everything else: a missing key or known_hosts file (the Secret has not landed yet), TCP and network errors, context cancellation, an abandoned or budget-exceeded call, a connection killed by another caller, libvirt `ErrOperationInvalid` and every unrecognised libvirt code, an image URL answering 429 or 5xx or ending the body early, the wait for running expiring or the domain going shut off or crashed right after its start |
+| `ErrTransient` | everything else: a missing key or known_hosts file (the Secret has not landed yet), TCP and network errors, context cancellation, an abandoned or budget-exceeded call, a connection killed by another caller, libvirt `ErrOperationInvalid` and every unrecognised libvirt code, an image URL answering 429 or 5xx or ending the body early, every address source failing on an active domain, the wait for running expiring or the domain going shut off or crashed right after its start |
 
 Error strings name paths, hosts, volumes and libvirt's own text. They
 never contain key bytes, the known_hosts content or cloud-init.
@@ -627,7 +627,12 @@ never contain key bytes, the known_hosts content or cloud-init.
   vendor script's output in `/var/log/cloud-init-output.log` (a VM without
   outbound internet cannot install it). A user-data that disables
   vendor-data (`vendor_data: {enabled: false}`) also removes the agent, by
-  design. `virsh domifaddr <name> --source agent` and `--source arp` show
+  design. A host where the ARP source *errors* rather than answering with
+  nothing is the one case this reads differently: with no source left to
+  answer, a booting VM retries with `ErrTransient` instead of resting in
+  `Provisioning`. It still converges; the giveaway is two "no addresses
+  from source" lines per poll at `-v 2`.
+  `virsh domifaddr <name> --source agent` and `--source arp` show
   what the operator sees, in that order.
 - **Debian 13 cloud images hang in GRUB under BIOS on a headless VM**
   ([Debian bug #1111240](https://bugs.debian.org/1111240)), which is why

docs/plans-executions/2026-09-17-1515-kvm2-review-fixes.md

@@ -4,7 +4,7 @@ Plan: [docs/plans/2026-09-17-1515-kvm2-review-fixes.md](../plans/2026-09-17-1515
 - [x] Step 0 — Save plan into the repo
 - [x] Step 1 — H1: a stalled write must not delay the teardown
 - [x] Step 2 — H2: a source error must not reach go-libvirt's stream sender
-- [ ] Step 3 — H3: an unreadable address must not look like a missing one
+- [x] Step 3 — H3: an unreadable address must not look like a missing one
 - [ ] Step 4 — H4: the base image cache must not be shared across provider entries
 
 Remaining after those: the 48 mediums and lows in `review/REPORT.md`. Each gets its own
@@ -210,3 +210,76 @@ atomic-publish work then lands exactly where this step put the discard.
 would mean a 150-250 line libvirt RPC server stub whose only assertion is about *upstream's*
 goroutines. Same call as H1's: the test would cost more than it protects. The two tests that
 did land pin the one property the seam actually needs.
+
+## Step 3 — H3: an unreadable address must not look like a missing one
+
+`address()` tried the guest agent then the host's ARP table and skipped any error that was
+not a context error, returning `("", nil)` after both. An active domain with no IP maps to
+`StateProvisioning`, the controller clears `status.IP` on that, and the health engine prunes
+the proxy's counters and calls `ForgetProxy` in one tick once `EffectiveHost()` is empty.
+
+The damaging trigger was not the guest. The libvirt connection is pooled per host, so when
+another reconcile's cancel or budget overrun killed it, `call` returned
+`"connection is closed"` — carrying no context sentinel, so the check did not recognise it —
+and `Get` returned a *successful* instance in Provisioning with an empty IP. An unrelated
+proxy's timeout dropped a healthy proxy out of rotation.
+
+The fix rests on a discriminator the spec had already recorded without drawing the
+conclusion: during a normal boot on beelink exactly one source fails. The agent errors
+because it is not up yet, while ARP answers *without* error and with nothing, since the host
+never sees the guest's traffic. So one source failing is "no address yet"; every source
+failing, or a connection already dead, is `ErrTransient`.
+
+The explicit `provider.ErrTransient` wrap on the joined error is load-bearing rather than
+decoration. Without it `classify` picks the first libvirt code out of the join tree, and
+`ErrInvalidArg`, `ErrAccessDenied` and `ErrNoSupport` are Permanent — an address read failing
+would have latched a *running* proxy as Failed, a worse version of the harm being fixed.
+
+`isDead(be)` is checked first in the error branch. It is not strictly needed, since a dead
+link fails both sources and the count rule catches it anyway, but it fails on the first
+source instead of the second, attributes the error better, and is the same predicate M15 will
+want for `ListByTag`.
+
+### Tests
+
+`TestGet_addressErrorIsProvisioning` failed against the new rule, which is correct: it fails
+*both* sources with `ErrAgentUnresponsive`, and code 86 is agent-specific — the ARP source
+cannot produce it. Re-pointed as `TestGet_agentFailsAndARPEmptyIsProvisioning`, the beelink
+boot it was really reaching for: agent errors, ARP returns no addresses and no error, result
+is Provisioning. Nothing covered that before, and it is the case the new rule must not break.
+
+`TestGet_everyAddressSourceFailingIsTransient` covers the other side, two subtests: both
+sources erroring, and the pooled connection dying under the read. Against the unfixed code
+both return a Provisioning instance instead of an error, which is exactly the bug:
+
+```bash
+# with isDead() and the every-source-failed arm removed from address():
+go test -count=1 -run TestGet_everyAddressSourceFailingIsTransient ./internal/provider/kvm2/
+#   get_test.go:215: Get = &{... IP: State:Provisioning ...}, want no instance
+```
+
+`TestGet_arpFallbackWhenAgentFails` passes untouched — one source of two.
+
+### Worth noting
+
+**Half of H3 is deliberately still open, and it is the half that fires more often.** A guest
+agent restarting inside a running VM still demotes it to Provisioning and clears its IP: the
+agent errors, ARP answers empty, and that is indistinguishable from a VM still booting. The
+discriminator is the *previous* IP, which the provider does not hold by design
+(`TestGet_readsUIDAndCreatedAtFromBackend` pins "read back, not remembered") and the
+controller already has. Follow-up finding, to be filed against
+`internal/controller/proxy_controller.go:208-211`: on `StateProvisioning`, keep an existing
+IP and `Provisioned` condition for a few polls rather than clearing on the first, or give the
+health engine a grace period before pruning.
+
+The one kvm2-local alternative was considered and rejected: the mark carries `createdAt`, so
+`get` could treat "active, no address from any source, domain older than a provisioning
+window" as an error. It uses state the provider genuinely has, but `createdAt` is creation
+rather than boot, it fires on every genuinely stuck VM as a retry-forever error, and it
+invents a provisioning deadline that belongs to the controller. A design change wearing a bug
+fix's clothes.
+
+**L6 not folded in.** A domain that vanishes between `DomainGetState` and the address read
+now returns Transient rather than a fake Provisioning — better, and self-correcting on the
+next poll. Returning NotFound one poll earlier is a two-line addition at the same spot, left
+for when L6 comes up so this commit stays one idea.

internal/provider/kvm2/get.go

@@ -98,22 +98,36 @@ var addressSources = []struct {
 }
 
 // address returns the first usable IPv4 from the first source that yields
-// one, or "" for "no address yet". A source that fails is logged and
+// one, or "" for "no address yet". One source failing is normal and is
-// skipped; only ctx ending is an error, so a cancelled Get does not read
+// skipped: a booting VM has no agent yet, while the host's ARP table
-// as Provisioning.
+// simply reports nothing, since it never sees traffic addressed
+// elsewhere. A connection already gone, and every source failing, are
+// errors instead — otherwise a link some other reconcile killed reads as
+// a running VM that lost its address, and the controller clears its IP
+// and the health engine drops its accumulated state.
 func address(ctx context.Context, log logr.Logger, be backend, dom libvirt.Domain) (string, error) {
+	errs := make([]error, 0, len(addressSources))
 	for _, s := range addressSources {
 		ifaces, err := be.DomainInterfaceAddresses(ctx, dom, uint32(s.src), 0)
 		if err != nil {
-			if errors.Is(err, context.Canceled) || errors.Is(err, context.DeadlineExceeded) {
+			if isDead(be) || errors.Is(err, context.Canceled) || errors.Is(err, context.DeadlineExceeded) {
-				return "", fmt.Errorf("reading addresses of domain %s: %w", dom.Name, err)
+				return "", fmt.Errorf("reading addresses of domain %s from the %s source: %w", dom.Name, s.name, err)
 			}
 			log.V(2).Info("no addresses from source", "source", s.name, "error", err.Error())
+			errs = append(errs, fmt.Errorf("%s source: %w", s.name, err))
 			continue
 		}
 		if ip := firstIPv4(ifaces); ip != "" {
 			return ip, nil
 		}
 	}
+	// Transient explicitly: classify would otherwise pick a libvirt code
+	// out of the joined error, and ErrInvalidArg, ErrAccessDenied and
+	// ErrNoSupport are Permanent, which would latch a running proxy as
+	// Failed over an address read.
+	if len(errs) == len(addressSources) {
+		return "", fmt.Errorf("%w: reading addresses of domain %s: every source failed: %w",
+			provider.ErrTransient, dom.Name, errors.Join(errs...))
+	}
 	return "", nil
 }

internal/provider/kvm2/get_test.go

@@ -156,17 +156,69 @@ func TestGet_arpFallbackWhenAgentFails(t *testing.T) {
 	}
 }
 
-func TestGet_addressErrorIsProvisioning(t *testing.T) {
+// TestGet_agentFailsAndARPEmptyIsProvisioning is a VM part way through
+// booting on a plain bridge: the agent is not up yet, and the host's ARP
+// table answers without error because it has nothing to report. One
+// source failing must stay "no address yet", not an error.
+func TestGet_agentFailsAndARPEmptyIsProvisioning(t *testing.T) {
 	t.Parallel()
-	fx := newFixture(t, 1)
+	fx := newFixture(t, 2)
 	id := fx.create(testName)
-	fx.fake.failWith("DomainInterfaceAddresses", fakeErr(libvirt.ErrAgentUnresponsive, "Guest agent is not responding"))
+	var calls int
+	fx.fake.mu.Lock()
+	fx.fake.hooks["DomainInterfaceAddresses"] = func() error {
+		calls++
+		if calls%2 == 1 {
+			return fakeErr(libvirt.ErrAgentUnresponsive, "Guest agent is not responding")
+		}
+		return nil
+	}
+	fx.fake.mu.Unlock()
 	inst := fx.get(id)
 	if inst.State != provider.StateProvisioning || inst.IP != "" {
 		t.Errorf("Get = %+v, want Provisioning with no IP", inst)
 	}
 }
 
+// TestGet_everyAddressSourceFailingIsTransient covers the other side of
+// that line. A running VM whose address cannot be read is not a VM
+// without an address: reporting Provisioning would make the controller
+// clear its IP and the health engine drop its counters.
+func TestGet_everyAddressSourceFailingIsTransient(t *testing.T) {
+	t.Parallel()
+	tests := []struct {
+		name  string
+		setup func(fx *fixture)
+	}{
+		{
+			name: "both sources error",
+			setup: func(fx *fixture) {
+				fx.fake.failWith("DomainInterfaceAddresses", fakeErr(libvirt.ErrOperationInvalid, "domain is not running"))
+			},
+		},
+		{
+			name: "the pooled connection died under the read",
+			setup: func(fx *fixture) {
+				fx.fake.failWith("DomainInterfaceAddresses", transientf("libvirt call DomainInterfaceAddresses on beelink: connection is closed: keepalive failed"))
+				fx.fake.markDead()
+			},
+		},
+	}
+	for _, tc := range tests {
+		t.Run(tc.name, func(t *testing.T) {
+			t.Parallel()
+			fx := newFixture(t, 1)
+			id := fx.create(testName)
+			tc.setup(fx)
+			inst, err := fx.p.Get(fx.ctx, id)
+			if inst != nil {
+				t.Errorf("Get = %+v, want no instance", inst)
+			}
+			assertProviderError(t, err, "get", provider.ErrTransient)
+		})
+	}
+}
+
 func TestGet_readsUIDAndCreatedAtFromBackend(t *testing.T) {
 	t.Parallel()
 	fx := newFixture(t, 1)