egress-proxies-operator/review/escalations.md

# kvm2 provider review: escalations for phase 2

18 raw Escalate items from the seven sweepers, merged to 15 root hypotheses, top 10 kept.
Ranked by blast radius x uncertainty. `deep-reviewer` is static (Read, Grep, Glob, Bash; it
can read the Go module cache and run local tests, but not touch the hypervisor), so each item
says whether it can be **settled statically** or is likely to end PLAUSIBLE without a probe on
beelink. Finding IDs (`S…`) refer to `sweep.md`.

| # | Hypothesis in one line | Blast | Uncertainty | Static? |
|---|---|---|---|---|
| E1 | One wedged `markDead` starves every reconcile worker, not just one host | operator-wide stall | medium | yes |
| E2 | A reformatted GC mark makes GC delete a live Proxy's VM | live VM destroyed | medium | mostly |
| E3 | kvm and kvm2 on one hypervisor: cross-provider destruction beyond the Get/Delete hole | other provider's VMs/disks | medium | yes |
| E4 | No cross-process fencing on the base volume: leader handover deletes a base mid-upload | corrupt base under new overlays | medium | yes |
| E5 | An abandoned `StorageVolCreateXML` leaves a corrupt overlay that the next Create adopts | silent guest disk corruption | high | partly |
| E6 | go-libvirt leaks or races when the socket is killed mid-connect; per-round check dials accumulate | slow leak in long-running leader | medium | yes |
| E7 | In-place base replacement corrupts existing overlays on their next boot | guest disk corruption, many VMs | low-medium | partly |
| E8 | Pool-space precheck reads a stale `available`; quota backoff never engages | host pool fills | medium | partly |
| E9 | libvirt error-code assumptions: idempotency codes and define-time rejection codes | retry loops that never converge | medium | partly |
| E10 | Overlay capacity below the base's virtual size is accepted and truncates the disk | silent guest corruption | high | no |

---

## E1 — A wedged `markDead` blocks name locks and starves the reconcile workers
Sources: chunk 1 Escalate 2; builds on finding S1.1.
**Entry points:** `internal/provider/kvm2/kvm2.go:186-243` (`lockName`, `keyedLocks`), `create.go`, `delete.go`, `backend.go:138-188`, `hosts.go:66-95`, `internal/gc/gc.go`, `internal/controller/proxy_controller.go` (+ `SetupWithManager`, `MaxConcurrentReconciles`), `cmd/main.go` manager wiring.
**Hypothesis:** a goroutine blocked inside `markDead` (S1.1) holds the per-name lock taken by Create/Delete, and reconcile and GC contexts carry no deadline, so `keyedLocks.acquire`'s ctx escape never fires. Delete/GC for that name block indefinitely instead of requeueing, and with a small `MaxConcurrentReconciles` a single silent host can pin every worker, stalling all Proxies on all providers.
**Checked:** lock acquisition is ctx-aware and refcounted correctly; only `internal/providerhealth/monitor.go:180` sets a deadline; the health checker and image upload use `dialFresh` and are insulated; `markDead` at `hosts.go:153` runs under `e.sem`.
**Unknown:** whether the controller-runtime work queue tolerates a permanently blocked Reconcile for one key (worker starvation across all Proxies); the actual `MaxConcurrentReconciles` and any reconcile timeout in the manager wiring; whether GC's single goroutine is blocked the same way; how many distinct call sites can enter `markDead` while holding `e.sem` or a name lock.

## E2 — GC mark parse fragility: a reformatted mark makes GC delete a live VM
Sources: chunk 3 Escalate 1; finding S5.3.
**Entry points:** `internal/provider/kvm2/metadata.go:44-78` (`createdAt`, `parseInstanceMetadata`), `list.go:49`, `get.go:79-88`, `internal/gc/gc.go:113-121`.
**Hypothesis:** if libvirt hands `<uid>` or `<createdAt>` back with surrounding whitespace or re-indented content (another libvirt version, or an admin's `virsh edit`/`virsh define` through a reformatting tool), `UID` stops matching `live[inst.UID]` (exact string compare, `gc.go:118`) and/or `createdAt()` returns the zero time, which makes the instance immediately past `MinAge`. GC then deletes a live Proxy's VM. No trim or tolerant parse exists as a backstop.
**Checked:** `newInstanceMetadata` writes a single-line element; `kvm2_integration_test.go:422-449` asserts the UID round-trips and `CreatedAt` is non-zero (sweeper's claim that this ran against real libvirt 11.6 was wrong: phase 2 found the kvm2 integration test has not yet run on beelink, see `deep.md`); libxml2 does not re-indent an element whose children have text content.
**Unknown:** behaviour on other libvirt versions and after `virsh edit`; whether `encoding/xml` unmarshal plus the current struct tags already tolerate whitespace; whether a UID mismatch alone (timestamp intact) is enough for deletion, and what GC does with `UID == ""`; whether GC has any second guard (name-to-Proxy lookup) before `Delete`.

## E3 — kvm and kvm2 on the same hypervisor
Sources: chunk 3 Escalate 2, chunk 7 Escalate 4; builds on findings S2.3, S7.1, S7.4.
**Entry points:** `internal/provider/kvm2/list.go:20-56`, `get.go:52-88`, `delete.go:55-98`, `volume.go:14-18`, `internal/provider/kvm/kvm.go:379`, `internal/provider/kvm/id.go:11-13`, `internal/provider/kvm/volume.go:12`, `internal/provider/kvm/image.go:46`, `internal/gc/gc.go:86-131`, `internal/provider/config.go:108-130`, `internal/controller/proxy_controller.go:197,237,287`, `docs/kvm2-provider.md:346-370`.
**Hypothesis:** the only thing keeping kvm2's `ListByTag` from handing kvm-made domains to GC is libvirt's namespace-scoped `DomainGetMetadata`; both providers also share `<name>.qcow2`, `<name>-seed.iso`, `base-<key>.<format>` names and the providerID format. Beyond the confirmed Get/Delete hole (S2.3), GC's per-provider scoping may let provider A's list be reconciled against provider B's Proxies, and a Proxy moved between a kvm and a kvm2 entry (same UID, same domain name) may have one side delete volumes the other side's domain uses.
**Checked:** domain names are UID-derived so one Proxy cannot exist under both at once; the two metadata URIs differ; `ListByTag` is mark-filtered; `spec.provider` is immutable but an entry's `name` is independent of its `type`; the fake cannot prove the namespace rule (S7.4).
**Unknown:** whether GC builds its live-UID set per provider or globally, and what it deletes when the same UID is visible through two entries; whether anything (config reload guard, admission check on ProviderID, deploy tooling) prevents a re-typed entry from inheriting live `status.ProviderID`s; the reverse direction (kvm v1 `Get`/`Delete`/`ListByTag` against kvm2-made domains).

## E4 — No cross-process fencing on the base volume
Source: chunk 4 Escalate A.
**Entry points:** `internal/provider/kvm2/image.go:85-146` (`ensureImage`, `lockImage`, `cachedImage`), `image.go:200-300` (`fetchImage`, `uploadImage`), `kvm2.go:199-243`, `cmd/main.go:107` (leader election flags), `config/manager/manager.yaml:24`.
**Hypothesis:** process A is inside `fetchImage` (base created, marker not yet written); process B's `cachedImage` sees base-without-marker and deletes it (`image.go:118`), then its `StorageVolCreateXML` either fails `ErrStorageVolExist` or succeeds against a half-uploaded file. A's stream writes into an unlinked inode and A writes a marker for a base that no longer exists. An overlay created in between backs onto a deleted or wrong-content path.
**Checked:** `keyedLocks` is process-local; `replicas: 1`; `--leader-elect` defaults to false; `Delete` never touches base volumes; nothing else in the repo names base or marker volumes.
**Unknown:** whether a leader-election handover or a rolling update (old pod still inside a 20 minute `fetchImage` while the new pod reconciles the same Proxy) reaches this with the shipped manifests; whether the manager cancels in-flight reconciles on lost lease and how long `terminationGracePeriodSeconds: 10` leaves the old process uploading; whether B's marker written over A's bytes can ever validate.

## E5 — Overlay atomicity under an abandoned RPC
Source: chunk 2 Escalate 1.
**Entry points:** `internal/provider/kvm2/create.go:151-181` (`ensureOverlay`), `backend.go:157-188` (`call`, budget, `markDead`), `backend.go:319-321` (`StorageVolCreateXML` under `budgets.rpc` = 30 s).
**Hypothesis:** if `StorageVolCreateXML` for the overlay exceeds the 30 s budget or the caller's ctx ends, `call` kills the SSH transport mid-RPC. If libvirtd still registers the volume, possibly partial, the next Create hits `ErrStorageVolExist`, logs "overlay volume already exists" and defines a domain over a corrupt overlay. The base image has a marker and the seed is rebuilt unconditionally for exactly this reason; the overlay has neither.
**Checked:** the reuse branch distinguishes only `ErrStorageVolExist`; no size, format or backing-path verification anywhere; `Delete` is the only thing that removes an overlay.
**Unknown:** libvirtd's storage-driver behaviour when the client drops mid `virStorageVolCreateXML` (unlink vs leave registered); whether overlay creation can exceed 30 s in normal operation (NFS, LVM, preallocating filesystems) rather than only on cancellation; whether a cheap verification (`StorageVolGetInfo` capacity, or delete-and-recreate when no domain references it) is available through the existing `backend` seam.

## E6 — go-libvirt internals when the socket is killed under it
Sources: chunk 1 Escalate 1, chunk 6 Escalate 1.
**Entry points:** `internal/provider/kvm2/backend.go:225-238` (`libvirtBackend.connect`, the `Disconnected()` watcher), `backend.go:351-367` (`Close`), `hosts.go:91-162` (`dialFresh`, `connect`), `check.go:85-119`; upstream `go-libvirt@v0.0.0-20260814190004-1a83157e1858`: `libvirt.go:286-305` (`ConnectToURI`), `:821/:834` (`waitAndDisconnect`), `socket/`.
**Hypothesis:** (a) when `call(…, "Connect", …)` expires and `markDead` closes the socket just as `ConnectToURI` reaches `l.disconnected = make(chan struct{})`, there is a data race on that field against `waitAndDisconnect`, and the loser leaves a `Libvirt` whose `Disconnected()` never fires; (b) `Check` opens a fresh connection per host per round (~1440/day/host), so any goroutine or fd go-libvirt leaves behind on the killed-socket path (abandoned Connect, or `Close` hitting its 5 s budget mid `ConnectClose`) accumulates linearly in a long-running leader.
**Checked:** kvm2's own goroutines are clean (watcher selects on `link.closed`, keepalive exits on `stop`, `sshSocket.Close` is once-guarded); `Disconnected()` is read only after a successful Connect and the backend is discarded on failure; `go test -race -count=2` of the unit tests is clean.
**Unknown:** whether `waitAndDisconnect`/`listenAndRespond` always terminate when the conn is closed mid-handshake; whether the socket layer leaves an fd; whether a looped dial-and-kill test (N=1000, sampling `runtime.NumGoroutine()`) against the in-process SSH test server shows growth. Note finding S4.1 already confirms one permanent go-libvirt goroutine leak on the stream path; this asks whether the connect/close path has another.

## E7 — In-place base-image replacement under overlays that pin its path
Source: chunk 2 Escalate 2; the code behaviour is finding S4.6 (three sweepers).
**Entry points:** `internal/provider/kvm2/create.go:103,154-165`, `image.go:113-146` (`cachedImage`), `image.go:235` (`StoragePoolRefresh`), `docs/provider-contract.md` ("Catalogue changes do not replace instances").
**Hypothesis:** when `images.<key>.sha256` changes (or the marker is missing or unreadable), the base is deleted and different content is downloaded to the same deterministic path while every earlier overlay records that path as its qcow2 backing file. Any such overlay that is later restarted (adoption path, host reboot, `ensureRunning` on a shutoff domain) reads mismatched backing data: guest filesystem corruption across every VM on that key.
**Checked:** `Delete` never removes the base; the path is stable per key+format; `BackingPath` is resolved once at define time and never re-verified; the contract explicitly allows catalogue edits not to replace instances.
**Unknown:** which restart paths actually exist for an existing overlay in this operator (does anything restart a shutoff domain rather than delete and recreate it, given Stopped → Delete in the controller?); whether the missing/unreadable-marker trigger can fire without any admin edit (e.g. after E4 or a failed `writeMarker`), which would make this reachable in normal operation; whether libvirt refuses `StorageVolDelete` on a volume in use as a backing store.

## E8 — Pool-space precheck freshness
Source: chunk 4 Escalate B.
**Entry points:** `internal/provider/kvm2/image.go:419-467` (`imageSizeIfUncached`, `checkPoolSpace`, `neededBytes`), `create.go:96-106`, `check.go:85-119` (also reads pool info).
**Hypothesis:** for a `dir` pool, `StoragePoolGetInfo.available` is what libvirtd computed at the last pool refresh, adjusted only for creates and deletes it performed; guest writes growing sparse overlays do not update it. `create.go` never issues `StoragePoolRefresh` before the check (the only refresh is after an upload, `image.go:235`), so on a host its VMs have filled, the precheck reports tens of GB free indefinitely, `ErrQuotaExceeded`/`QuotaRetry` never engages, and Create fails later with an ENOSPC-flavoured transient, which is also the case that wedges the upload stream (S4.1) and pauses guests (S2.2).
**Checked:** no refresh anywhere on the pre-create path; `available` is used raw; unit tests set the fake's value directly; the spec describes the precheck but says nothing about freshness.
**Unknown:** libvirt 11.x dir-pool semantics for `available` (statvfs at call time vs cached); whether `Check`'s `poolAvailableGiB` has the same staleness; the cost of a per-Create refresh on the shared pooled connection.

## E9 — libvirt error-code assumptions
Sources: chunk 5 Escalate 2, chunk 7 Escalate 2; findings S3.5, S5.2.
**Entry points:** `internal/provider/kvm2/errors.go:33-97` (`classify`, `classifyLibvirt`, `libvirtCode`, `hasCode`), `create.go:139-142,166-177`, `get.go:79-88`, `delete.go:55-98`, `image.go:174-186`, `fake_test.go:491-493`, `kvm2_integration_test.go`.
**Hypothesis:** two halves of one assumption. (a) Idempotency depends on exact codes: `ensureOverlay` needs `ErrStorageVolExist` for a duplicate volume on a dir pool, `readMetadata` needs `ErrNoDomainMetadata`, the delete paths need `ErrNoDomain`/`ErrNoStorageVol`. The fake returns exactly the codes the code expects, so unit tests cannot refute a mismatch; if real libvirt returns `ErrOperationFailed` or `ErrInternalError` for any of them, a retried Create after a partial failure never converges. (b) Every define-time rejection of the generated XML (arch, machine, firmware feature, host-passthrough) surfaces as `ErrConfigUnsupported` (67) or `ErrOperationUnsupported` (84), which default to Transient: an unbounded retry with no `ReasonPermanentError`.
**Checked:** only `ErrNoStoragePool`, `ErrXMLError`, `ErrNoSupport`, `ErrInvalidArg`, `ErrAuthFailed`, `ErrAccessDenied` map to Permanent; 67 and 84 exist in go-libvirt's const set and are unlisted; bare `libvirt.ErrUnsupported` bypasses `libvirtCode`; transient errors requeue forever.
**Unknown:** which idempotency codes the integration test actually exercises against libvirt 11.6 (a second Create on an existing overlay? Delete twice? Get on a foreign domain?), which settles (a) for those paths; for (b), which code libvirt really returns for each rejection, and whether keeping unknown codes Transient is a deliberate spec decision. Mapping too much to Permanent latches Proxies that would recover.

## E10 — Overlay capacity smaller than the backing image's virtual size
Source: chunk 5 Escalate 1; finding S5.6.
**Entry points:** `internal/provider/kvm2/volume.go:68-89` (`buildVolumeXML`), `create.go:107,154-181`, `image.go:449-467`, `internal/provider/config_kvm.go:133` (`diskSizeGb` validation).
**Hypothesis:** with `kvm2.diskSizeGb` below the base image's virtual size, libvirt runs `qemu-img create -b <base> … <capacity>` and succeeds, producing a truncated virtual disk whose partition table overruns the device. The VM fails to boot or boots with a corrupt root filesystem, and kvm2 reports a healthy running domain that never answers.
**Checked:** nothing in the package compares overlay capacity to the base's virtual size; the validator rejects only `diskSizeGb < 0`; the 10 GiB default exceeds current Debian/Ubuntu/Fedora cloud images.
**Unknown:** whether libvirt's storage backend or qemu-img errors on a short capacity (then this is only a poor error message). Needs a real qemu-img; the deep reviewer can at most establish whether kvm2 has any way to learn the base's virtual size through the existing seam (`StorageVolGetInfo` on the base) and what a guard would cost. **Expect PLAUSIBLE.**

---

## Dropped (not in the top 10)

- **D1** Deleting a volume while a libvirt upload stream is open on it (chunk 4 C): a consequence of confirmed finding S4.1, only changes whether its disk-space effect is real; needs a truncated-source probe on beelink.
- **D2** cloud-init dispatch of a `text/plain` part inside vendor-data (chunk 5 E3): invisible-failure risk but only a boot test on a real image settles it; nothing static to add beyond S5.8.
- **D3** ProviderHealth status churn every round waking watchers (chunk 6 E2): low blast radius, arguably intended per the code comment.
- **D4** Nested volumeMount, `kvm-ssh` Secret inside the read-only `providers-config` mount (chunk 7 E1): pre-existing and shared with kvm; refuted if kvm has ever run in-cluster with the Secret present, otherwise one `kubectl apply` settles it.
- **D5** Whether provider error strings (with the full image URL, S7.10) reach Proxy status conditions (chunk 7 E3): low blast radius; cheap and fully static if you want it added.