# Plan: fix the kvm2 code-review findings **Created:** 2026-09-17 15:15 ## Context The review of `feat/kvm2-provider` (`review/REPORT.md`) confirmed 52 findings: 4 high, 20 medium, 28 low. We are fixing them one at a time, highest severity first, each as its own step and its own commit. This plan is the campaign; the execution file's checklist is the authoritative progress record. Only **Step 1 (H1)** is detailed now — later steps get detailed as we reach them, so the plan is not a fiction about work not yet designed. Precedent for this shape: `docs/plans/2026-09-10-2222-kvm-review-fixes.md`, the same campaign for the kvm provider. ### The bug being fixed in Step 1 (H1) `sshSocket.Close` is what tears a kvm2 connection down. It is the `kill` of `link` (`backend.go:216`), and killing the transport is the only way to abandon a go-libvirt call, which takes no context and blocks. ```go // ssh.go:111 s.closeErr = errors.Join(s.Conn.Close(), s.client.Close()) ``` Go evaluates those arguments left to right, so **the blocking close runs before the unblocking one**: - `s.Conn` is the forwarded libvirt socket, a `*chanConn` over x/crypto's `*channel`. `channel.Close` → `sendMessage` → `channel.writePacket` (`channel.go:223`), which takes `ch.writeMu` and holds it across a raw TCP write. - `s.client.Close()` is `connection.Close()` → a bare `net.Conn.Close()` on the TCP socket (`connection.go:111`). It takes no lock, and Go's netpoll unblocks any pending write on it. While go-libvirt streams a base image through that same channel (`StorageVolUpload`), the writer sits inside `writePacket` holding `ch.writeMu`. Verified in x/crypto v0.54.0: the flow-control park (`remoteWin.reserve`, `channel.go:268`) happens **outside** `writeMu`, so only a writer stuck in the TCP write blocks `Close`. When the host is power-cut or partitioned (no RST, no FIN), that write blocks until the kernel stops retransmitting — `tcp_retries2`, about 15 minutes. `Close` queues behind it and never reaches the client close that would have freed everything. `markDead` then amplifies it (`backend.go:138-148`): ```go l.once.Do(func() { ... if l.kill != nil { l.kill() } // blocks here for ~15 min close(l.closed) // never reached }) ``` Three consequences, each independently bad: 1. `l.closed` is never closed, so callers parked in `call`'s `case <-l.closed` are never released. 2. `isDead()` stays false, so `hostTable.get` keeps handing out the dying pooled backend and each new call wedges the same way. 3. `sync.Once.Do` blocks *every* concurrent `markDead` caller — including `call`'s own ctx-cancel branch (`backend.go:180`). **Cancelling the reconcile does not help.** Blast radius: `MaxConcurrentReconciles: 3` (`proxy_controller.go:442`) and the per-image lock (`ensureImage`/`lockImage`), whose `keyedLocks.acquire` is ctx-aware but reconcile contexts carry no deadline. Three Proxies on one host with an uncached image park all three workers for ~15 minutes, stalling every Proxy on every provider. There is a **second route to the same pin**, fixed by the same edits: go-libvirt's `Disconnect` closes the `net.Conn` it was handed (`libvirt.go:331` → `socket.go:164`), and that conn *is* our `*sshSocket`. So `libvirtBackend.Close`'s `Disconnect` goroutine enters `sshSocket.closeOnce` and hangs; the 5 s `closeBudget` releases `Close`, which calls `markDead("closed")` → `kill()` → `sshSocket.Close()` → blocks in the other goroutine's `sync.Once`. The budget does not protect it. The older kvm provider does not have this bug: it closes `client` only (`kvm/conn.go:429,456`) and never closes the forwarded channel. kvm2 introduced it. ## Steps - **Step 0** — Save this plan into the repo, commit alone, before any code. - **Step 1** — H1: `sshSocket.Close` and `link.markDead` (detailed below). - **Step 2+** — the remaining findings from `review/REPORT.md` in severity order (H2 upload stream wedge, H3 `address()` swallowing errors, H4 shared base image cache, then the mediums). Detailed when reached; add a checklist line per step as it is planned. --- ## Step 1 — H1: a stalled write must never delay tearing the connection down ### 1a. `internal/provider/kvm2/ssh.go` — close the SSH client only Replace the body of `Close` (109-114) and correct the `sshSocket` doc comment (98-100), which currently says Close "also closes the SSH client, which is what unblocks anything reading the channel": ```go // sshSocket is the libvirt socket forwarded through one SSH connection. The // embedded Conn is the channel; Close closes the SSH client, whose TCP // connection carries it. Closing the channel too would be worse than useless: // channel.Close writes a close message under the channel's write lock, so it // would queue behind a transfer stuck in a TCP write — the case Close exists // to survive. type sshSocket struct { ... } // fields unchanged func (s *sshSocket) Close() error { s.closeOnce.Do(func() { s.closeErr = s.client.Close() }) return s.closeErr } ``` Dropping `s.Conn.Close()` rather than reordering it, because after the TCP conn is closed the channel close returns one of several errors (`net.ErrClosed`, a `*net.OpError`, the transport's sticky write error, or `io.EOF` once `mux.loop` has run `ch.close()`), and `closeErr` is memoised — a filter that misses a case would put a spurious error in every later `Close` and break `ssh_test.go:66,69`, which assert both closes return nil. Nothing needs the polite channel-close message: `dialSSHSocket` builds the client solely for this one channel, the `sshSocket` owns it exclusively, every Close is a full teardown, and `dialSSHSocket`'s own error paths (181, 188) already tear down by closing the client. Closing the client is sufficient: `mux.loop` exits and `chanList.dropAll()` → `ch.close()` unblocks the channel's readers and writers. `errors` stays imported (`errors.As` at 52 and 90). No production caller inspects the error: `backend.go:216`, `hosts.go:78`, `check.go:90`, `image.go:265` all discard it. ### 1b. `internal/provider/kvm2/backend.go` — signal dead before killing, and drop the `sync.Once` Reordering alone is not enough: `sync.Once.Do` blocks concurrent callers until the first body returns, which is consequence 3 above. Delete the `once sync.Once` field from `link` (109) and guard with the mutex the struct already has: ```go // markDead records why the connection is unusable, signals dead and then // closes the transport. Signalling first means a kill that takes its time // holds up neither the callers waiting on dead nor the next markDead. func (l *link) markDead(reason string) { l.mu.Lock() if l.isDead() { l.mu.Unlock() return } l.reason = reason close(l.closed) l.mu.Unlock() if l.kill != nil { l.kill() } } ``` `isDead` is a non-blocking select on `l.closed` and takes no lock, so it is safe under `mu` and keeps one source of truth. `mu` holds the invariant that exactly one goroutine closes the channel and runs `kill`. `reason` is set before `close`, so a caller released by `<-l.closed` always reads a populated `deadReason()`. `kill` runs outside the lock. Update the comment at 136-137 ("closes the transport and signals dead" — now reversed). The `link` doc (100-104) and `newLibvirtBackend`'s comment (210-212) stay accurate. Consequences checked, all safe: `call`'s five branches; `hostTable.get` may redial a few microseconds before the old socket is gone (two TCP connections overlap briefly); `libvirtBackend.Close`'s early return on `isDead()` skips no kill, since `isDead()` now means "some goroutine has closed `closed` and is running `kill`", still exactly once; the disconnect watcher and keepalive goroutine exit sooner instead of parking in `once.Do`; `hosts.go:153`, which calls `markDead` while holding `e.sem`, is bounded by one `net.Conn.Close`. When `Close` is itself the first marker, `kill` runs synchronously, so Close still returns only after the socket is closed — `TestLibvirtBackend_closeIsIdempotent` is unaffected. **Not** running `kill` in a goroutine: `Close` would return before the socket was really closed, which `mockAdapter`'s cleanup and `image.go:265`'s `defer fresh.Close()` rely on, and it would leave an fd-leak window on the failed-dial path. ### 1c. Test — one, not four Revised down from an earlier draft of four tests. What the other three would have asserted is either already covered or no longer has a code path to guard: - "Close tears the connection down properly" is covered by `TestDial_forwardsUnixSocket` (ssh_test.go:66-71), which closes twice and asserts both return nil. After 1a, `Close` is a single `s.client.Close()` — there is no ordering left to get wrong. - A stalled-write end-to-end test (TCP proxy, shrunken `SO_SNDBUF`/`SO_RCVBUF`, window priming) was dropped: ~150 lines of Linux-only, buffer-autotuning-dependent scaffolding whose own false-green guard makes it *skip* when the kernel does not cooperate. Its value was demonstrating the bug, which reading x/crypto already settles, and 1a removes the call it would exercise. The repro recipe goes in the commit message instead. **`TestMarkDead_slowKillDoesNotBlockCallers`** — `backend_test.go`. Kept because 1b *adds* a guarantee that does not exist today ("a slow kill blocks neither the callers waiting on dead nor the next `markDead`") and nothing else in the suite touches it — `TestCall_killedByAnotherCallerFailsTransient` passes `newLink("beelink", nil)`, a nil kill, so it cannot see the property. It is also the plausible regression: restoring the `sync.Once` looks like a correctness improvement and would silently reinstate the stall. A `kill` that signals it has started and then blocks on a channel closed by `t.Cleanup`. Reuses the existing `blockingFn` helper (backend_test.go:19). With a call parked in `l.run`, assert within one second each: `<-l.dead()` closes; the parked call returns `ErrTransient`; a second `markDead` from another goroutine returns; `killed.Load() == 1`; `deadReason()` is the first reason. Fails by timeout against the current code. `t.Parallel()`, generous bounds, `errors.Is` against `provider.ErrTransient`, per repo style. **Accepted risk:** nothing then stops someone re-adding `s.Conn.Close()` to `sshSocket.Close` as tidiness. The guard is the doc comment in 1a saying outright that closing the channel would queue behind a stalled transfer — in idiom for this package (`hosts.go:25-34`, `create.go:151-153`), but a comment, not a test. ### 1d. Docs `docs/kvm2-provider.md:451-453` says the adapter "kills the connection (socket and SSH client)". It now closes the SSH client, which takes the forwarded socket with it — reword, and optionally add the one-clause reason (a lock-free close is the only kind that survives a transfer stuck in a TCP write) near the 20 m upload budget row. `docs/kvm2-provider-spec.md` entries 18 and 19 say the adapter "closes the underlying connection" and describe the pool; neither is falsified, so no spec change. ### 1e. Explicitly out of scope - `ReconciliationTimeout` in `SetupWithManager`: does not exist in the repo today, and a per-reconcile deadline changes the contract for every provider (`hosts.go` and `image.go` both cite "no caller deadline"). Its own change, with its own spec entry. - `sshSocket.keepalive` having no test (sweep finding S1.2) — separate finding. - The kvm provider: verified not to have this bug. ### Verification ```bash go build ./... go test -race ./internal/provider/kvm2/ make test ``` Then the step that actually matters: **confirm the new test fails against the unfixed code.** Stash the `markDead` edit while keeping the test, run it, watch it fail by timeout, restore. A regression test that passes both ways is worthless. Record in the commit message: the reproduction recipe (shrunken `SO_SNDBUF` so the socket buffer fills before the 2 MiB SSH window, and why `remoteWin.reserve` sitting *outside* `writeMu` means only a writer inside `writePacket` can block the close), and the second route through go-libvirt's `Disconnect` closing the `net.Conn` it was handed. ### Commit One commit on `feat/kvm2-provider`: the two source edits, the one test, the doc reword, the execution-file entry with its checklist line ticked, and the `CHANGELOG.md` entry (timestamp from `date "+%Y-%m-%d %H:%M %Z"`) once the user confirms it works. Trailer: `Co-Authored-By: Claude Opus 5 (1M context) `.