docs/plans/2026-09-17-1515-kvm2-review-fixes.md

@@ -20,7 +20,8 @@ Precedent for this shape: `docs/plans/2026-09-10-2222-kvm-review-fixes.md`.
 - [ ] **Step 2** — H2: an error during `StorageVolUpload` wedges go-libvirt's stream sender.
 - [ ] **Step 3** — H3: `address()` swallows every error and demotes healthy proxies.
 - [ ] **Step 4** — H4: the base image cache is shared across provider entries.
-- [ ] **Step 5+** — the mediums, from `review/REPORT.md`. Detailed when reached.
+- [ ] **Step 5** — Validate H1-H4 in-cluster against the real hypervisor (detailed below).
+- [ ] **Step 6+** — the mediums, from `review/REPORT.md`. Detailed when reached.
 
 ---
 
@@ -335,6 +336,148 @@ at `internal/provider/config_kvm.go:139-141` ("it becomes the cached volume name
 
 ---
 
+## Step 5 — Validate in-cluster against beelink
+
+### Why, and what this run can and cannot prove
+
+The kvm2 provider has **never run against real libvirt**: Step 11 is unchecked in
+`docs/plans-executions/2026-09-16-1333-kvm2-provider.md`, every `base-*` volume on beelink
+predates it, and both live `proxy-*` domains carry the kvm v1 metadata namespace. Four fixes
+have landed on code no hypervisor has executed. This run is first contact, and it is also the
+gate the campaign's `CHANGELOG` entry waits on.
+
+It proves the happy paths: the dial and health probe, a real 415 MB image download through the
+reader H2 now wraps, H4's new volume name appearing beside kvm's untouched one, a second Proxy
+hitting the cache, H3's address path reaching Running with an IP, and Squid answering a real
+CONNECT probe.
+
+It does **not** prove H2's error paths. The unverified ~85% claim — that libvirtd answers a
+short stream finish with OK — only arises when a body *fails*, and a healthy download never
+finishes short. The 19-minute HTTP deadline will not fire either. A green run means "the
+wrapper did not break normal uploads", not "the deadlock is settled". That still needs the
+scratch-pool probe recorded in the Step 2 execution entry.
+
+### The cluster this runs in, and the one hazard
+
+`kind-proxy-operator-demo` is **not a throwaway**. It runs the operator that currently owns
+four Ready proxies, two of them real VMs on beelink:
+
+```
+proxy-beelink-1           Managed  kvm-home     Ready  192.168.0.166  True
+proxy-beelink-2           Managed  kvm-home     Ready  192.168.0.94   True
+proxy-kubernetes-demo-1/2 Managed  kubernetes   Ready  10.244.0.7/.8  True
+```
+
+Validating kvm2 means rolling the operator that owns them. That is low risk — the controller
+and the whole `kvm` provider are untouched by H1-H4 — and rollback is one `kubectl set image`.
+
+**Do not run a second operator against beelink with `kvm-home` in its config.** Orphan GC
+(`internal/gc/gc.go:86-131`) lists every Proxy in *its own* cluster, and deletes any instance
+whose UID is not in that set and is older than `--gc-min-age` (10 m, swept every 10 m). A
+second cluster's live set would not contain `proxy-beelink-1/2`'s UIDs, so its GC would destroy
+them within ten minutes. Using the existing cluster removes the hazard entirely: its live set
+is the one that protects them.
+
+### Steps
+
+**0. Baseline** — record what is there before touching anything, so "did we break it" is
+answerable: `kubectl get px -A`, `kubectl get ph`, and a read-only `virsh vol-list default`
+plus `virsh list --all` on beelink.
+
+**1. Build and load, under a new tag.**
+
+```bash
+make docker-build IMG=egress-proxies-operator:kvm2-dev
+kind load docker-image egress-proxies-operator:kvm2-dev --name proxy-operator-demo
+```
+
+The tag **must** differ from the running `kvm-dev`. `make deploy` sets the image through
+kustomize; an identical tag leaves the Deployment spec unchanged, so no rollout happens, so the
+new process never starts, so the new providers config is never read. The ConfigMap carries no
+generator hash, so nothing else forces a restart either.
+
+The working tree is dirty (`review/`, `.claude/`), so `GIT_COMMIT` bakes in as `c10f98a-dirty`
+and `--version` will say so. Cosmetic; commit or stash `review/` first to avoid it.
+
+**2. Enable kvm2 in the deployed config.** Uncomment the `kvm2-home` block in
+`config/manager/providers_config.yaml` (it is the shipped default, commented out). Keep its
+`address: beelink.tail8c69c.ts.net:22` — the Tailscale FQDN, which is what
+`~/.ssh/kvm_known_hosts` pins. `config/samples/providers-config.yaml` uses the short
+`beelink:22`, which **fails host-key verification**; I hit exactly that when validating SSH
+earlier in this session.
+
+**3. Deploy.**
+
+```bash
+make deploy IMG=egress-proxies-operator:kvm2-dev
+kubectl -n egress-proxies-operator-system rollout status deploy/egress-proxies-operator-controller-manager
+```
+
+`make deploy` rewrites `config/manager/kustomization.yaml`'s image block as a side effect; it
+already carries `kvm-dev` from a previous run.
+
+**4. STOP GATE — provider health, before anything is created.**
+
+```bash
+kubectl get ph
+kubectl get ph kvm2-home -o yaml
+```
+
+`kvm2-home` must be `Ready 1/1` with reason `Reachable` and details carrying `hostname`,
+`libvirtVersion`, `poolState: running` and `poolAvailableGiB`. This proves SSH, the host key,
+the libvirt handshake and the pool **without creating a thing**, and it exercises the
+dedicated-connection dial and close that H1 touched. A failure here is free to diagnose:
+`CredentialsUnavailable`, `Unauthorized`, `Forbidden`, `PoolNotActive` each name their own
+cause. Report and wait for a go before step 5.
+
+**5. Two Proxies.** Two manifests in the scratchpad (not committed), modelled on
+`config/samples/proxy_kvm2.yaml` — `proxy-kvm2-1` and `proxy-kvm2-2` in `default`, matching the
+existing `proxy-beelink-*` convention, `provider: kvm2-home`, `zone: beelink`,
+`machineType: small`, `image: debian-13`, port 3128 and the sample's Squid cloud-init. Apply
+the first, let it finish, then the second — sequencing them is what makes the cache hit
+observable.
+
+```bash
+kubectl get px -w
+kubectl -n egress-proxies-operator-system logs -f deploy/egress-proxies-operator-controller-manager
+```
+
+**6. What to check, and which finding it exercises.**
+
+| Check | Finding |
+|---|---|
+| `fetching base image into host pool` then `base image cached` with size, duration, digest | H2's wrapped reader on a real 415 MB upload |
+| `virsh vol-list default` shows `base-kvm2-home-debian-13.qcow2` **and** `base-debian-13.qcow2` still there | H4: per-entry naming, kvm's cache untouched |
+| Second Proxy logs no `fetching`, and the server is not hit again | H4 + the image lock: cache hit under the new name |
+| `px` goes Provisioning → Running with an IP, without flapping back to Provisioning | H3: the address path, and no spurious demotion |
+| `Healthy=True` | Squid answers a real CONNECT probe end to end |
+| `virsh list --all` shows two new `proxy-*` domains | Create |
+| After deleting both: domains and their `.qcow2`/`-seed.iso` gone, base image still present | Delete, and the cache surviving |
+
+Optionally `curl -x http://<ip>:3128 https://example.com -I` from this box, if it routes to
+192.168.0.0/24 — the existing kvm proxies live on that LAN.
+
+**7. Cleanup.** `kubectl delete px proxy-kvm2-1 proxy-kvm2-2` and let the finalizer do the
+work, then verify on beelink read-only. **Any leftover domain or volume: I print the exact
+`virsh undefine --nvram` / `vol-delete` lines and you run them** — no write reaches the shared
+host from me. Then decide whether the `kvm2-home` ConfigMap entry stays (keep it if you want
+kvm2 running for real; `git checkout config/manager/providers_config.yaml` otherwise).
+
+**Rollback at any point:**
+
+```bash
+kubectl -n egress-proxies-operator-system set image \
+  deploy/egress-proxies-operator-controller-manager manager=egress-proxies-operator:kvm-dev
+```
+
+### Who runs what
+
+I drive steps 0-6 and stop at gate 4 for a go. Every beelink write is yours. Nothing here
+touches `proxy-beelink-1/2`, `base-debian-13.qcow2`, your unrelated VMs, or the five
+pre-existing orphan volumes the review found.
+
+---
+
 ## Ordering
 
 **H2 → H3 → H4**, three commits.