egress-proxies-operator/docs/reviews/2026-09-17-kvm2-provider-review/escalations.md

kvm2 provider review: escalations for phase 2

18 raw Escalate items from the seven sweepers, merged to 15 root hypotheses, top 10 kept. Ranked by blast radius x uncertainty. deep-reviewer is static (Read, Grep, Glob, Bash; it can read the Go module cache and run local tests, but not touch the hypervisor), so each item says whether it can be settled statically or is likely to end PLAUSIBLE without a probe on beelink. Finding IDs (S…) refer to sweep.md.

#	Hypothesis in one line	Blast	Uncertainty	Static?
E1	One wedged markDead starves every reconcile worker, not just one host	operator-wide stall	medium	yes
E2	A reformatted GC mark makes GC delete a live Proxy's VM	live VM destroyed	medium	mostly
E3	kvm and kvm2 on one hypervisor: cross-provider destruction beyond the Get/Delete hole	other provider's VMs/disks	medium	yes
E4	No cross-process fencing on the base volume: leader handover deletes a base mid-upload	corrupt base under new overlays	medium	yes
E5	An abandoned StorageVolCreateXML leaves a corrupt overlay that the next Create adopts	silent guest disk corruption	high	partly
E6	go-libvirt leaks or races when the socket is killed mid-connect; per-round check dials accumulate	slow leak in long-running leader	medium	yes
E7	In-place base replacement corrupts existing overlays on their next boot	guest disk corruption, many VMs	low-medium	partly
E8	Pool-space precheck reads a stale available; quota backoff never engages	host pool fills	medium	partly
E9	libvirt error-code assumptions: idempotency codes and define-time rejection codes	retry loops that never converge	medium	partly
E10	Overlay capacity below the base's virtual size is accepted and truncates the disk	silent guest corruption	high	no

---

E1 — A wedged markDead blocks name locks and starves the reconcile workers

Sources: chunk 1 Escalate 2; builds on finding S1.1. Entry points: internal/provider/kvm2/kvm2.go:186-243 (lockName, keyedLocks), create.go, delete.go, backend.go:138-188, hosts.go:66-95, internal/gc/gc.go, internal/controller/proxy_controller.go (+ SetupWithManager, MaxConcurrentReconciles), cmd/main.go manager wiring. Hypothesis: a goroutine blocked inside markDead (S1.1) holds the per-name lock taken by Create/Delete, and reconcile and GC contexts carry no deadline, so keyedLocks.acquire's ctx escape never fires. Delete/GC for that name block indefinitely instead of requeueing, and with a small MaxConcurrentReconciles a single silent host can pin every worker, stalling all Proxies on all providers. Checked: lock acquisition is ctx-aware and refcounted correctly; only internal/providerhealth/monitor.go:180 sets a deadline; the health checker and image upload use dialFresh and are insulated; markDead at hosts.go:153 runs under e.sem. Unknown: whether the controller-runtime work queue tolerates a permanently blocked Reconcile for one key (worker starvation across all Proxies); the actual MaxConcurrentReconciles and any reconcile timeout in the manager wiring; whether GC's single goroutine is blocked the same way; how many distinct call sites can enter markDead while holding e.sem or a name lock.

E2 — GC mark parse fragility: a reformatted mark makes GC delete a live VM

Sources: chunk 3 Escalate 1; finding S5.3. Entry points: internal/provider/kvm2/metadata.go:44-78 (createdAt, parseInstanceMetadata), list.go:49, get.go:79-88, internal/gc/gc.go:113-121. Hypothesis: if libvirt hands <uid> or <createdAt> back with surrounding whitespace or re-indented content (another libvirt version, or an admin's virsh edit/virsh define through a reformatting tool), UID stops matching live[inst.UID] (exact string compare, gc.go:118) and/or createdAt() returns the zero time, which makes the instance immediately past MinAge. GC then deletes a live Proxy's VM. No trim or tolerant parse exists as a backstop. Checked: newInstanceMetadata writes a single-line element; kvm2_integration_test.go:422-449 asserts the UID round-trips and CreatedAt is non-zero (sweeper's claim that this ran against real libvirt 11.6 was wrong: phase 2 found the kvm2 integration test has not yet run on beelink, see deep.md); libxml2 does not re-indent an element whose children have text content. Unknown: behaviour on other libvirt versions and after virsh edit; whether encoding/xml unmarshal plus the current struct tags already tolerate whitespace; whether a UID mismatch alone (timestamp intact) is enough for deletion, and what GC does with UID == ""; whether GC has any second guard (name-to-Proxy lookup) before Delete.

E3 — kvm and kvm2 on the same hypervisor

Sources: chunk 3 Escalate 2, chunk 7 Escalate 4; builds on findings S2.3, S7.1, S7.4. Entry points: internal/provider/kvm2/list.go:20-56, get.go:52-88, delete.go:55-98, volume.go:14-18, internal/provider/kvm/kvm.go:379, internal/provider/kvm/id.go:11-13, internal/provider/kvm/volume.go:12, internal/provider/kvm/image.go:46, internal/gc/gc.go:86-131, internal/provider/config.go:108-130, internal/controller/proxy_controller.go:197,237,287, docs/kvm2-provider.md:346-370. Hypothesis: the only thing keeping kvm2's ListByTag from handing kvm-made domains to GC is libvirt's namespace-scoped DomainGetMetadata; both providers also share <name>.qcow2, <name>-seed.iso, base-<key>.<format> names and the providerID format. Beyond the confirmed Get/Delete hole (S2.3), GC's per-provider scoping may let provider A's list be reconciled against provider B's Proxies, and a Proxy moved between a kvm and a kvm2 entry (same UID, same domain name) may have one side delete volumes the other side's domain uses. Checked: domain names are UID-derived so one Proxy cannot exist under both at once; the two metadata URIs differ; ListByTag is mark-filtered; spec.provider is immutable but an entry's name is independent of its type; the fake cannot prove the namespace rule (S7.4). Unknown: whether GC builds its live-UID set per provider or globally, and what it deletes when the same UID is visible through two entries; whether anything (config reload guard, admission check on ProviderID, deploy tooling) prevents a re-typed entry from inheriting live status.ProviderIDs; the reverse direction (kvm v1 Get/Delete/ListByTag against kvm2-made domains).

E4 — No cross-process fencing on the base volume

Source: chunk 4 Escalate A. Entry points: internal/provider/kvm2/image.go:85-146 (ensureImage, lockImage, cachedImage), image.go:200-300 (fetchImage, uploadImage), kvm2.go:199-243, cmd/main.go:107 (leader election flags), config/manager/manager.yaml:24. Hypothesis: process A is inside fetchImage (base created, marker not yet written); process B's cachedImage sees base-without-marker and deletes it (image.go:118), then its StorageVolCreateXML either fails ErrStorageVolExist or succeeds against a half-uploaded file. A's stream writes into an unlinked inode and A writes a marker for a base that no longer exists. An overlay created in between backs onto a deleted or wrong-content path. Checked: keyedLocks is process-local; replicas: 1; --leader-elect defaults to false; Delete never touches base volumes; nothing else in the repo names base or marker volumes. Unknown: whether a leader-election handover or a rolling update (old pod still inside a 20 minute fetchImage while the new pod reconciles the same Proxy) reaches this with the shipped manifests; whether the manager cancels in-flight reconciles on lost lease and how long terminationGracePeriodSeconds: 10 leaves the old process uploading; whether B's marker written over A's bytes can ever validate.

E5 — Overlay atomicity under an abandoned RPC

Source: chunk 2 Escalate 1. Entry points: internal/provider/kvm2/create.go:151-181 (ensureOverlay), backend.go:157-188 (call, budget, markDead), backend.go:319-321 (StorageVolCreateXML under budgets.rpc = 30 s). Hypothesis: if StorageVolCreateXML for the overlay exceeds the 30 s budget or the caller's ctx ends, call kills the SSH transport mid-RPC. If libvirtd still registers the volume, possibly partial, the next Create hits ErrStorageVolExist, logs "overlay volume already exists" and defines a domain over a corrupt overlay. The base image has a marker and the seed is rebuilt unconditionally for exactly this reason; the overlay has neither. Checked: the reuse branch distinguishes only ErrStorageVolExist; no size, format or backing-path verification anywhere; Delete is the only thing that removes an overlay. Unknown: libvirtd's storage-driver behaviour when the client drops mid virStorageVolCreateXML (unlink vs leave registered); whether overlay creation can exceed 30 s in normal operation (NFS, LVM, preallocating filesystems) rather than only on cancellation; whether a cheap verification (StorageVolGetInfo capacity, or delete-and-recreate when no domain references it) is available through the existing backend seam.

E6 — go-libvirt internals when the socket is killed under it

Sources: chunk 1 Escalate 1, chunk 6 Escalate 1. Entry points: internal/provider/kvm2/backend.go:225-238 (libvirtBackend.connect, the Disconnected() watcher), backend.go:351-367 (Close), hosts.go:91-162 (dialFresh, connect), check.go:85-119; upstream go-libvirt@v0.0.0-20260814190004-1a83157e1858: libvirt.go:286-305 (ConnectToURI), :821/:834 (waitAndDisconnect), socket/. Hypothesis: (a) when call(…, "Connect", …) expires and markDead closes the socket just as ConnectToURI reaches l.disconnected = make(chan struct{}), there is a data race on that field against waitAndDisconnect, and the loser leaves a Libvirt whose Disconnected() never fires; (b) Check opens a fresh connection per host per round (~1440/day/host), so any goroutine or fd go-libvirt leaves behind on the killed-socket path (abandoned Connect, or Close hitting its 5 s budget mid ConnectClose) accumulates linearly in a long-running leader. Checked: kvm2's own goroutines are clean (watcher selects on link.closed, keepalive exits on stop, sshSocket.Close is once-guarded); Disconnected() is read only after a successful Connect and the backend is discarded on failure; go test -race -count=2 of the unit tests is clean. Unknown: whether waitAndDisconnect/listenAndRespond always terminate when the conn is closed mid-handshake; whether the socket layer leaves an fd; whether a looped dial-and-kill test (N=1000, sampling runtime.NumGoroutine()) against the in-process SSH test server shows growth. Note finding S4.1 already confirms one permanent go-libvirt goroutine leak on the stream path; this asks whether the connect/close path has another.

E7 — In-place base-image replacement under overlays that pin its path

Source: chunk 2 Escalate 2; the code behaviour is finding S4.6 (three sweepers). Entry points: internal/provider/kvm2/create.go:103,154-165, image.go:113-146 (cachedImage), image.go:235 (StoragePoolRefresh), docs/provider-contract.md ("Catalogue changes do not replace instances"). Hypothesis: when images.<key>.sha256 changes (or the marker is missing or unreadable), the base is deleted and different content is downloaded to the same deterministic path while every earlier overlay records that path as its qcow2 backing file. Any such overlay that is later restarted (adoption path, host reboot, ensureRunning on a shutoff domain) reads mismatched backing data: guest filesystem corruption across every VM on that key. Checked: Delete never removes the base; the path is stable per key+format; BackingPath is resolved once at define time and never re-verified; the contract explicitly allows catalogue edits not to replace instances. Unknown: which restart paths actually exist for an existing overlay in this operator (does anything restart a shutoff domain rather than delete and recreate it, given Stopped → Delete in the controller?); whether the missing/unreadable-marker trigger can fire without any admin edit (e.g. after E4 or a failed writeMarker), which would make this reachable in normal operation; whether libvirt refuses StorageVolDelete on a volume in use as a backing store.

E8 — Pool-space precheck freshness

Source: chunk 4 Escalate B. Entry points: internal/provider/kvm2/image.go:419-467 (imageSizeIfUncached, checkPoolSpace, neededBytes), create.go:96-106, check.go:85-119 (also reads pool info). Hypothesis: for a dir pool, StoragePoolGetInfo.available is what libvirtd computed at the last pool refresh, adjusted only for creates and deletes it performed; guest writes growing sparse overlays do not update it. create.go never issues StoragePoolRefresh before the check (the only refresh is after an upload, image.go:235), so on a host its VMs have filled, the precheck reports tens of GB free indefinitely, ErrQuotaExceeded/QuotaRetry never engages, and Create fails later with an ENOSPC-flavoured transient, which is also the case that wedges the upload stream (S4.1) and pauses guests (S2.2). Checked: no refresh anywhere on the pre-create path; available is used raw; unit tests set the fake's value directly; the spec describes the precheck but says nothing about freshness. Unknown: libvirt 11.x dir-pool semantics for available (statvfs at call time vs cached); whether Check's poolAvailableGiB has the same staleness; the cost of a per-Create refresh on the shared pooled connection.

E9 — libvirt error-code assumptions

Sources: chunk 5 Escalate 2, chunk 7 Escalate 2; findings S3.5, S5.2. Entry points: internal/provider/kvm2/errors.go:33-97 (classify, classifyLibvirt, libvirtCode, hasCode), create.go:139-142,166-177, get.go:79-88, delete.go:55-98, image.go:174-186, fake_test.go:491-493, kvm2_integration_test.go. Hypothesis: two halves of one assumption. (a) Idempotency depends on exact codes: ensureOverlay needs ErrStorageVolExist for a duplicate volume on a dir pool, readMetadata needs ErrNoDomainMetadata, the delete paths need ErrNoDomain/ErrNoStorageVol. The fake returns exactly the codes the code expects, so unit tests cannot refute a mismatch; if real libvirt returns ErrOperationFailed or ErrInternalError for any of them, a retried Create after a partial failure never converges. (b) Every define-time rejection of the generated XML (arch, machine, firmware feature, host-passthrough) surfaces as ErrConfigUnsupported (67) or ErrOperationUnsupported (84), which default to Transient: an unbounded retry with no ReasonPermanentError. Checked: only ErrNoStoragePool, ErrXMLError, ErrNoSupport, ErrInvalidArg, ErrAuthFailed, ErrAccessDenied map to Permanent; 67 and 84 exist in go-libvirt's const set and are unlisted; bare libvirt.ErrUnsupported bypasses libvirtCode; transient errors requeue forever. Unknown: which idempotency codes the integration test actually exercises against libvirt 11.6 (a second Create on an existing overlay? Delete twice? Get on a foreign domain?), which settles (a) for those paths; for (b), which code libvirt really returns for each rejection, and whether keeping unknown codes Transient is a deliberate spec decision. Mapping too much to Permanent latches Proxies that would recover.

E10 — Overlay capacity smaller than the backing image's virtual size

Source: chunk 5 Escalate 1; finding S5.6. Entry points: internal/provider/kvm2/volume.go:68-89 (buildVolumeXML), create.go:107,154-181, image.go:449-467, internal/provider/config_kvm.go:133 (diskSizeGb validation). Hypothesis: with kvm2.diskSizeGb below the base image's virtual size, libvirt runs qemu-img create -b <base> … <capacity> and succeeds, producing a truncated virtual disk whose partition table overruns the device. The VM fails to boot or boots with a corrupt root filesystem, and kvm2 reports a healthy running domain that never answers. Checked: nothing in the package compares overlay capacity to the base's virtual size; the validator rejects only diskSizeGb < 0; the 10 GiB default exceeds current Debian/Ubuntu/Fedora cloud images. Unknown: whether libvirt's storage backend or qemu-img errors on a short capacity (then this is only a poor error message). Needs a real qemu-img; the deep reviewer can at most establish whether kvm2 has any way to learn the base's virtual size through the existing seam (StorageVolGetInfo on the base) and what a guard would cost. Expect PLAUSIBLE.

---

Dropped (not in the top 10)

• D1 Deleting a volume while a libvirt upload stream is open on it (chunk 4 C): a consequence of confirmed finding S4.1, only changes whether its disk-space effect is real; needs a truncated-source probe on beelink.
• D2 cloud-init dispatch of a text/plain part inside vendor-data (chunk 5 E3): invisible-failure risk but only a boot test on a real image settles it; nothing static to add beyond S5.8.
• D3 ProviderHealth status churn every round waking watchers (chunk 6 E2): low blast radius, arguably intended per the code comment.
• D4 Nested volumeMount, kvm-ssh Secret inside the read-only providers-config mount (chunk 7 E1): pre-existing and shared with kvm; refuted if kvm has ever run in-cluster with the Secret present, otherwise one kubectl apply settles it.
• D5 Whether provider error strings (with the full image URL, S7.10) reach Proxy status conditions (chunk 7 E3): low blast radius; cheap and fully static if you want it added.