From 966807bf5776e0d5213d1a5327439e2a04cc1732 Mon Sep 17 00:00:00 2001 From: kacerr Date: Thu, 17 Sep 2026 19:23:44 +0000 Subject: [PATCH] Plan the in-cluster kvm2 validation (Step 5) First contact between the kvm2 provider and a real hypervisor: it has never run against libvirt, so H1-H4 landed on code no host has executed. The run happens in the existing kind cluster on purpose. That cluster already owns two live kvm-home VMs on beelink, and orphan GC deletes any instance whose UID is missing from its own cluster live set after gc-min-age. A second operator carrying the kvm-home entry would therefore destroy both within ten minutes; using the existing cluster makes its live set the thing that protects them. The plan also records what the run cannot prove: H2 error paths need a body that fails part way through, and a healthy download never finishes short, so the unverified libvirtd short-stream behaviour stays open. Co-Authored-By: Claude Opus 5 (1M context) --- .../2026-09-17-1515-kvm2-review-fixes.md | 145 +++++++++++++++++- 1 file changed, 144 insertions(+), 1 deletion(-) diff --git a/docs/plans/2026-09-17-1515-kvm2-review-fixes.md b/docs/plans/2026-09-17-1515-kvm2-review-fixes.md index 70b0ba9..70c94f6 100644 --- a/docs/plans/2026-09-17-1515-kvm2-review-fixes.md +++ b/docs/plans/2026-09-17-1515-kvm2-review-fixes.md @@ -20,7 +20,8 @@ Precedent for this shape: `docs/plans/2026-09-10-2222-kvm-review-fixes.md`. - [ ] **Step 2** — H2: an error during `StorageVolUpload` wedges go-libvirt's stream sender. - [ ] **Step 3** — H3: `address()` swallows every error and demotes healthy proxies. - [ ] **Step 4** — H4: the base image cache is shared across provider entries. -- [ ] **Step 5+** — the mediums, from `review/REPORT.md`. Detailed when reached. +- [ ] **Step 5** — Validate H1-H4 in-cluster against the real hypervisor (detailed below). +- [ ] **Step 6+** — the mediums, from `review/REPORT.md`. Detailed when reached. --- @@ -335,6 +336,148 @@ at `internal/provider/config_kvm.go:139-141` ("it becomes the cached volume name --- +## Step 5 — Validate in-cluster against beelink + +### Why, and what this run can and cannot prove + +The kvm2 provider has **never run against real libvirt**: Step 11 is unchecked in +`docs/plans-executions/2026-09-16-1333-kvm2-provider.md`, every `base-*` volume on beelink +predates it, and both live `proxy-*` domains carry the kvm v1 metadata namespace. Four fixes +have landed on code no hypervisor has executed. This run is first contact, and it is also the +gate the campaign's `CHANGELOG` entry waits on. + +It proves the happy paths: the dial and health probe, a real 415 MB image download through the +reader H2 now wraps, H4's new volume name appearing beside kvm's untouched one, a second Proxy +hitting the cache, H3's address path reaching Running with an IP, and Squid answering a real +CONNECT probe. + +It does **not** prove H2's error paths. The unverified ~85% claim — that libvirtd answers a +short stream finish with OK — only arises when a body *fails*, and a healthy download never +finishes short. The 19-minute HTTP deadline will not fire either. A green run means "the +wrapper did not break normal uploads", not "the deadlock is settled". That still needs the +scratch-pool probe recorded in the Step 2 execution entry. + +### The cluster this runs in, and the one hazard + +`kind-proxy-operator-demo` is **not a throwaway**. It runs the operator that currently owns +four Ready proxies, two of them real VMs on beelink: + +``` +proxy-beelink-1 Managed kvm-home Ready 192.168.0.166 True +proxy-beelink-2 Managed kvm-home Ready 192.168.0.94 True +proxy-kubernetes-demo-1/2 Managed kubernetes Ready 10.244.0.7/.8 True +``` + +Validating kvm2 means rolling the operator that owns them. That is low risk — the controller +and the whole `kvm` provider are untouched by H1-H4 — and rollback is one `kubectl set image`. + +**Do not run a second operator against beelink with `kvm-home` in its config.** Orphan GC +(`internal/gc/gc.go:86-131`) lists every Proxy in *its own* cluster, and deletes any instance +whose UID is not in that set and is older than `--gc-min-age` (10 m, swept every 10 m). A +second cluster's live set would not contain `proxy-beelink-1/2`'s UIDs, so its GC would destroy +them within ten minutes. Using the existing cluster removes the hazard entirely: its live set +is the one that protects them. + +### Steps + +**0. Baseline** — record what is there before touching anything, so "did we break it" is +answerable: `kubectl get px -A`, `kubectl get ph`, and a read-only `virsh vol-list default` +plus `virsh list --all` on beelink. + +**1. Build and load, under a new tag.** + +```bash +make docker-build IMG=egress-proxies-operator:kvm2-dev +kind load docker-image egress-proxies-operator:kvm2-dev --name proxy-operator-demo +``` + +The tag **must** differ from the running `kvm-dev`. `make deploy` sets the image through +kustomize; an identical tag leaves the Deployment spec unchanged, so no rollout happens, so the +new process never starts, so the new providers config is never read. The ConfigMap carries no +generator hash, so nothing else forces a restart either. + +The working tree is dirty (`review/`, `.claude/`), so `GIT_COMMIT` bakes in as `c10f98a-dirty` +and `--version` will say so. Cosmetic; commit or stash `review/` first to avoid it. + +**2. Enable kvm2 in the deployed config.** Uncomment the `kvm2-home` block in +`config/manager/providers_config.yaml` (it is the shipped default, commented out). Keep its +`address: beelink.tail8c69c.ts.net:22` — the Tailscale FQDN, which is what +`~/.ssh/kvm_known_hosts` pins. `config/samples/providers-config.yaml` uses the short +`beelink:22`, which **fails host-key verification**; I hit exactly that when validating SSH +earlier in this session. + +**3. Deploy.** + +```bash +make deploy IMG=egress-proxies-operator:kvm2-dev +kubectl -n egress-proxies-operator-system rollout status deploy/egress-proxies-operator-controller-manager +``` + +`make deploy` rewrites `config/manager/kustomization.yaml`'s image block as a side effect; it +already carries `kvm-dev` from a previous run. + +**4. STOP GATE — provider health, before anything is created.** + +```bash +kubectl get ph +kubectl get ph kvm2-home -o yaml +``` + +`kvm2-home` must be `Ready 1/1` with reason `Reachable` and details carrying `hostname`, +`libvirtVersion`, `poolState: running` and `poolAvailableGiB`. This proves SSH, the host key, +the libvirt handshake and the pool **without creating a thing**, and it exercises the +dedicated-connection dial and close that H1 touched. A failure here is free to diagnose: +`CredentialsUnavailable`, `Unauthorized`, `Forbidden`, `PoolNotActive` each name their own +cause. Report and wait for a go before step 5. + +**5. Two Proxies.** Two manifests in the scratchpad (not committed), modelled on +`config/samples/proxy_kvm2.yaml` — `proxy-kvm2-1` and `proxy-kvm2-2` in `default`, matching the +existing `proxy-beelink-*` convention, `provider: kvm2-home`, `zone: beelink`, +`machineType: small`, `image: debian-13`, port 3128 and the sample's Squid cloud-init. Apply +the first, let it finish, then the second — sequencing them is what makes the cache hit +observable. + +```bash +kubectl get px -w +kubectl -n egress-proxies-operator-system logs -f deploy/egress-proxies-operator-controller-manager +``` + +**6. What to check, and which finding it exercises.** + +| Check | Finding | +|---|---| +| `fetching base image into host pool` then `base image cached` with size, duration, digest | H2's wrapped reader on a real 415 MB upload | +| `virsh vol-list default` shows `base-kvm2-home-debian-13.qcow2` **and** `base-debian-13.qcow2` still there | H4: per-entry naming, kvm's cache untouched | +| Second Proxy logs no `fetching`, and the server is not hit again | H4 + the image lock: cache hit under the new name | +| `px` goes Provisioning → Running with an IP, without flapping back to Provisioning | H3: the address path, and no spurious demotion | +| `Healthy=True` | Squid answers a real CONNECT probe end to end | +| `virsh list --all` shows two new `proxy-*` domains | Create | +| After deleting both: domains and their `.qcow2`/`-seed.iso` gone, base image still present | Delete, and the cache surviving | + +Optionally `curl -x http://:3128 https://example.com -I` from this box, if it routes to +192.168.0.0/24 — the existing kvm proxies live on that LAN. + +**7. Cleanup.** `kubectl delete px proxy-kvm2-1 proxy-kvm2-2` and let the finalizer do the +work, then verify on beelink read-only. **Any leftover domain or volume: I print the exact +`virsh undefine --nvram` / `vol-delete` lines and you run them** — no write reaches the shared +host from me. Then decide whether the `kvm2-home` ConfigMap entry stays (keep it if you want +kvm2 running for real; `git checkout config/manager/providers_config.yaml` otherwise). + +**Rollback at any point:** + +```bash +kubectl -n egress-proxies-operator-system set image \ + deploy/egress-proxies-operator-controller-manager manager=egress-proxies-operator:kvm-dev +``` + +### Who runs what + +I drive steps 0-6 and stop at gate 4 for a go. Every beelink write is yours. Nothing here +touches `proxy-beelink-1/2`, `base-debian-13.qcow2`, your unrelated VMs, or the five +pre-existing orphan volumes the review found. + +--- + ## Ordering **H2 → H3 → H4**, three commits.