kvm2 provider: clean-room libvirt/KVM provider, code review, and the four high findings #8
@@ -20,7 +20,8 @@ Precedent for this shape: `docs/plans/2026-09-10-2222-kvm-review-fixes.md`.
|
||||
- [ ] **Step 2** — H2: an error during `StorageVolUpload` wedges go-libvirt's stream sender.
|
||||
- [ ] **Step 3** — H3: `address()` swallows every error and demotes healthy proxies.
|
||||
- [ ] **Step 4** — H4: the base image cache is shared across provider entries.
|
||||
- [ ] **Step 5+** — the mediums, from `review/REPORT.md`. Detailed when reached.
|
||||
- [ ] **Step 5** — Validate H1-H4 in-cluster against the real hypervisor (detailed below).
|
||||
- [ ] **Step 6+** — the mediums, from `review/REPORT.md`. Detailed when reached.
|
||||
|
||||
---
|
||||
|
||||
@@ -335,6 +336,148 @@ at `internal/provider/config_kvm.go:139-141` ("it becomes the cached volume name
|
||||
|
||||
---
|
||||
|
||||
## Step 5 — Validate in-cluster against beelink
|
||||
|
||||
### Why, and what this run can and cannot prove
|
||||
|
||||
The kvm2 provider has **never run against real libvirt**: Step 11 is unchecked in
|
||||
`docs/plans-executions/2026-09-16-1333-kvm2-provider.md`, every `base-*` volume on beelink
|
||||
predates it, and both live `proxy-*` domains carry the kvm v1 metadata namespace. Four fixes
|
||||
have landed on code no hypervisor has executed. This run is first contact, and it is also the
|
||||
gate the campaign's `CHANGELOG` entry waits on.
|
||||
|
||||
It proves the happy paths: the dial and health probe, a real 415 MB image download through the
|
||||
reader H2 now wraps, H4's new volume name appearing beside kvm's untouched one, a second Proxy
|
||||
hitting the cache, H3's address path reaching Running with an IP, and Squid answering a real
|
||||
CONNECT probe.
|
||||
|
||||
It does **not** prove H2's error paths. The unverified ~85% claim — that libvirtd answers a
|
||||
short stream finish with OK — only arises when a body *fails*, and a healthy download never
|
||||
finishes short. The 19-minute HTTP deadline will not fire either. A green run means "the
|
||||
wrapper did not break normal uploads", not "the deadlock is settled". That still needs the
|
||||
scratch-pool probe recorded in the Step 2 execution entry.
|
||||
|
||||
### The cluster this runs in, and the one hazard
|
||||
|
||||
`kind-proxy-operator-demo` is **not a throwaway**. It runs the operator that currently owns
|
||||
four Ready proxies, two of them real VMs on beelink:
|
||||
|
||||
```
|
||||
proxy-beelink-1 Managed kvm-home Ready 192.168.0.166 True
|
||||
proxy-beelink-2 Managed kvm-home Ready 192.168.0.94 True
|
||||
proxy-kubernetes-demo-1/2 Managed kubernetes Ready 10.244.0.7/.8 True
|
||||
```
|
||||
|
||||
Validating kvm2 means rolling the operator that owns them. That is low risk — the controller
|
||||
and the whole `kvm` provider are untouched by H1-H4 — and rollback is one `kubectl set image`.
|
||||
|
||||
**Do not run a second operator against beelink with `kvm-home` in its config.** Orphan GC
|
||||
(`internal/gc/gc.go:86-131`) lists every Proxy in *its own* cluster, and deletes any instance
|
||||
whose UID is not in that set and is older than `--gc-min-age` (10 m, swept every 10 m). A
|
||||
second cluster's live set would not contain `proxy-beelink-1/2`'s UIDs, so its GC would destroy
|
||||
them within ten minutes. Using the existing cluster removes the hazard entirely: its live set
|
||||
is the one that protects them.
|
||||
|
||||
### Steps
|
||||
|
||||
**0. Baseline** — record what is there before touching anything, so "did we break it" is
|
||||
answerable: `kubectl get px -A`, `kubectl get ph`, and a read-only `virsh vol-list default`
|
||||
plus `virsh list --all` on beelink.
|
||||
|
||||
**1. Build and load, under a new tag.**
|
||||
|
||||
```bash
|
||||
make docker-build IMG=egress-proxies-operator:kvm2-dev
|
||||
kind load docker-image egress-proxies-operator:kvm2-dev --name proxy-operator-demo
|
||||
```
|
||||
|
||||
The tag **must** differ from the running `kvm-dev`. `make deploy` sets the image through
|
||||
kustomize; an identical tag leaves the Deployment spec unchanged, so no rollout happens, so the
|
||||
new process never starts, so the new providers config is never read. The ConfigMap carries no
|
||||
generator hash, so nothing else forces a restart either.
|
||||
|
||||
The working tree is dirty (`review/`, `.claude/`), so `GIT_COMMIT` bakes in as `c10f98a-dirty`
|
||||
and `--version` will say so. Cosmetic; commit or stash `review/` first to avoid it.
|
||||
|
||||
**2. Enable kvm2 in the deployed config.** Uncomment the `kvm2-home` block in
|
||||
`config/manager/providers_config.yaml` (it is the shipped default, commented out). Keep its
|
||||
`address: beelink.tail8c69c.ts.net:22` — the Tailscale FQDN, which is what
|
||||
`~/.ssh/kvm_known_hosts` pins. `config/samples/providers-config.yaml` uses the short
|
||||
`beelink:22`, which **fails host-key verification**; I hit exactly that when validating SSH
|
||||
earlier in this session.
|
||||
|
||||
**3. Deploy.**
|
||||
|
||||
```bash
|
||||
make deploy IMG=egress-proxies-operator:kvm2-dev
|
||||
kubectl -n egress-proxies-operator-system rollout status deploy/egress-proxies-operator-controller-manager
|
||||
```
|
||||
|
||||
`make deploy` rewrites `config/manager/kustomization.yaml`'s image block as a side effect; it
|
||||
already carries `kvm-dev` from a previous run.
|
||||
|
||||
**4. STOP GATE — provider health, before anything is created.**
|
||||
|
||||
```bash
|
||||
kubectl get ph
|
||||
kubectl get ph kvm2-home -o yaml
|
||||
```
|
||||
|
||||
`kvm2-home` must be `Ready 1/1` with reason `Reachable` and details carrying `hostname`,
|
||||
`libvirtVersion`, `poolState: running` and `poolAvailableGiB`. This proves SSH, the host key,
|
||||
the libvirt handshake and the pool **without creating a thing**, and it exercises the
|
||||
dedicated-connection dial and close that H1 touched. A failure here is free to diagnose:
|
||||
`CredentialsUnavailable`, `Unauthorized`, `Forbidden`, `PoolNotActive` each name their own
|
||||
cause. Report and wait for a go before step 5.
|
||||
|
||||
**5. Two Proxies.** Two manifests in the scratchpad (not committed), modelled on
|
||||
`config/samples/proxy_kvm2.yaml` — `proxy-kvm2-1` and `proxy-kvm2-2` in `default`, matching the
|
||||
existing `proxy-beelink-*` convention, `provider: kvm2-home`, `zone: beelink`,
|
||||
`machineType: small`, `image: debian-13`, port 3128 and the sample's Squid cloud-init. Apply
|
||||
the first, let it finish, then the second — sequencing them is what makes the cache hit
|
||||
observable.
|
||||
|
||||
```bash
|
||||
kubectl get px -w
|
||||
kubectl -n egress-proxies-operator-system logs -f deploy/egress-proxies-operator-controller-manager
|
||||
```
|
||||
|
||||
**6. What to check, and which finding it exercises.**
|
||||
|
||||
| Check | Finding |
|
||||
|---|---|
|
||||
| `fetching base image into host pool` then `base image cached` with size, duration, digest | H2's wrapped reader on a real 415 MB upload |
|
||||
| `virsh vol-list default` shows `base-kvm2-home-debian-13.qcow2` **and** `base-debian-13.qcow2` still there | H4: per-entry naming, kvm's cache untouched |
|
||||
| Second Proxy logs no `fetching`, and the server is not hit again | H4 + the image lock: cache hit under the new name |
|
||||
| `px` goes Provisioning → Running with an IP, without flapping back to Provisioning | H3: the address path, and no spurious demotion |
|
||||
| `Healthy=True` | Squid answers a real CONNECT probe end to end |
|
||||
| `virsh list --all` shows two new `proxy-*` domains | Create |
|
||||
| After deleting both: domains and their `.qcow2`/`-seed.iso` gone, base image still present | Delete, and the cache surviving |
|
||||
|
||||
Optionally `curl -x http://<ip>:3128 https://example.com -I` from this box, if it routes to
|
||||
192.168.0.0/24 — the existing kvm proxies live on that LAN.
|
||||
|
||||
**7. Cleanup.** `kubectl delete px proxy-kvm2-1 proxy-kvm2-2` and let the finalizer do the
|
||||
work, then verify on beelink read-only. **Any leftover domain or volume: I print the exact
|
||||
`virsh undefine --nvram` / `vol-delete` lines and you run them** — no write reaches the shared
|
||||
host from me. Then decide whether the `kvm2-home` ConfigMap entry stays (keep it if you want
|
||||
kvm2 running for real; `git checkout config/manager/providers_config.yaml` otherwise).
|
||||
|
||||
**Rollback at any point:**
|
||||
|
||||
```bash
|
||||
kubectl -n egress-proxies-operator-system set image \
|
||||
deploy/egress-proxies-operator-controller-manager manager=egress-proxies-operator:kvm-dev
|
||||
```
|
||||
|
||||
### Who runs what
|
||||
|
||||
I drive steps 0-6 and stop at gate 4 for a go. Every beelink write is yours. Nothing here
|
||||
touches `proxy-beelink-1/2`, `base-debian-13.qcow2`, your unrelated VMs, or the five
|
||||
pre-existing orphan volumes the review found.
|
||||
|
||||
---
|
||||
|
||||
## Ordering
|
||||
|
||||
**H2 → H3 → H4**, three commits.
|
||||
|
||||
Reference in New Issue
Block a user