kvm2: do not let a stalled write delay the connection teardown (H1)

sshSocket.Close was errors.Join(s.Conn.Close(), s.client.Close()). Go
evaluates those left to right, so the blocking close ran before the
unblocking one. s.Conn is the forwarded libvirt socket, and closing an SSH
channel writes a close message under ch.writeMu — which go-libvirt already
holds while streaming a base image through that same channel. Against a
host that vanished without RST or FIN, that write blocks until the kernel
stops retransmitting (tcp_retries2, ~15 min), and s.client.Close(), a bare
net.Conn.Close() that takes no lock and unblocks the stalled writer through
netpoll, was never reached. Close the client only; nothing wants the polite
channel-close message, and reordering instead would leave an error to
filter in a memoised closeErr.

link.markDead compounded it by running kill() before close(l.closed), both
inside a sync.Once. Reordering alone was not enough: sync.Once.Do blocks
concurrent callers until the first body returns, so every other markDead —
including the one in call is own ctx-cancel branch — queued behind the
stalled kill, and cancelling the reconcile did not help. The once field is
gone; the mutex the struct already had guards the close-exactly-once
invariant, reason is set before the signal, and kill runs outside the lock.

Blast radius was the whole operator, not one host: MaxConcurrentReconciles
is 3 and ensureImage holds a per-image lock whose acquire is ctx-aware but
reconcile contexts carry no deadline, so three Proxies on one host with an
uncached image parked every worker.

The same edits close a second route to the same pin: go-libvirt is
Disconnect closes the net.Conn it was handed (libvirt.go:331 ->
socket.go:164), and that conn is our sshSocket, so libvirtBackend.Close is
Disconnect goroutine entered closeOnce and hung; the 5 s closeBudget
released Close, which called markDead -> kill -> sshSocket.Close and
blocked in the other goroutine is sync.Once.

Verified against the pinned x/crypto v0.54.0 rather than assumed: the
flow-control park (remoteWin.reserve, channel.go:268) is outside writeMu
and only writePacket holds it across the TCP write, so a writer parked in
SSH flow control never blocks the close and a writer inside the TCP write
always does. To reproduce by hand, shrink SO_SNDBUF well under the 2 MiB
channel window (channelWindowSize = 64 * channelMaxPacket) so the socket
buffer fills before the window does.

The kvm provider does not have this bug: it closes client only and never
touches the forwarded channel.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

docs/kvm2-provider.md

@@ -450,9 +450,15 @@ applied by `call`):
 
 go-libvirt's calls take no context and block, so the only way to abandon
 one is to close the connection under it. That is what a cancelled or
-overlong call does: the adapter kills the connection (socket and SSH
+overlong call does: the adapter marks the connection dead so the next call
-client), returns `ErrTransient` wrapping the context or deadline error,
+redials, kills it by closing the SSH client's TCP connection, which takes
-and marks the connection dead so the next call redials. The consequence:
+the forwarded socket with it, and returns `ErrTransient` wrapping the
+context or deadline error. It marks dead before it kills, and closes the
+TCP connection rather than the forwarded channel, because both of those are
+lock-free: closing the channel writes a close message under the channel's
+write lock, so it would queue behind a transfer stuck in a TCP write to a
+host that has gone away — for as long as the kernel retransmits, which is
+the one case the kill exists to survive. The consequence:
 **other RPCs in flight on that host's pooled connection fail once with
 `ErrTransient`** and their callers retry on a fresh connection. In the
 reconciler that is one requeue with backoff; in GC it is one skipped

docs/plans-executions/2026-09-17-1515-kvm2-review-fixes.md

@@ -0,0 +1,124 @@
+# Execution: fix the kvm2 code-review findings
+Plan: [docs/plans/2026-09-17-1515-kvm2-review-fixes.md](../plans/2026-09-17-1515-kvm2-review-fixes.md)
+
+- [x] Step 0 — Save plan into the repo
+- [x] Step 1 — H1: a stalled write must not delay the teardown
+
+Remaining: the other 51 findings in `review/REPORT.md`, in severity order — H2 (the
+`StorageVolUpload` stream wedge), H3 (`address()` swallowing errors), H4 (the shared base
+image cache), then the mediums. Each gets its own checklist line here when it is planned,
+so this list stays a record of decided work rather than a wish.
+
+## Step 0 — plan committed
+
+Copied out of `~/.claude/plans/` and committed alone before touching code, as the repo's
+plan rules require.
+
+## Step 1 — H1: close the SSH client, and signal dead before killing
+
+Two edits, both small, in the order the bug unfolds.
+
+`sshSocket.Close` was `errors.Join(s.Conn.Close(), s.client.Close())`. Go evaluates those
+arguments left to right, so the blocking close ran before the unblocking one. `s.Conn` is
+the forwarded libvirt socket, and closing an SSH channel writes a close message under
+`ch.writeMu`, which go-libvirt already holds while streaming a base image through the same
+channel. Against a host that vanished without RST or FIN, that write blocks until the
+kernel stops retransmitting — `tcp_retries2`, about 15 minutes. `s.client.Close()`, a bare
+`net.Conn.Close()` on the TCP socket that takes no lock and unblocks the stalled writer
+through netpoll, was never reached. It now closes the client only.
+
+**Dropping the channel close rather than reordering it** was the judgment call. Reordering
+works, but afterwards the channel close returns an error — `net.ErrClosed`, a
+`*net.OpError`, the transport's sticky write error, or `io.EOF` once `mux.loop` has run
+`ch.close()` — and `closeErr` is memoised, so a filter that missed a case would poison
+every later `Close` and break `ssh_test.go:66,69`, which assert both closes return nil.
+Nothing wants the polite channel-close message: `dialSSHSocket` builds the client solely
+for this one channel, the `sshSocket` owns it exclusively, and `dialSSHSocket`'s own error
+paths already tear down by closing the client.
+
+`link.markDead` then compounded it by running `kill()` before `close(l.closed)`, and doing
+both inside a `sync.Once`. Reordering alone would not have been enough: `sync.Once.Do`
+blocks concurrent callers until the first body returns, so every other `markDead` — including
+the one in `call`'s own ctx-cancel branch — queued behind the stalled kill. **Cancelling the
+reconcile did not help.** The `once` field is gone; the mutex the struct already had now
+guards the "close it exactly once" invariant, `reason` is set before the signal so a
+released caller always reads a populated `deadReason()`, and `kill` runs outside the lock.
+
+**A second route to the same pin, found while planning and fixed by the same edits.**
+go-libvirt's `Disconnect` closes the `net.Conn` it was handed (`libvirt.go:331` →
+`socket.go:164`) — and that conn *is* our `*sshSocket`. So `libvirtBackend.Close`'s
+`Disconnect` goroutine entered `sshSocket.closeOnce` and hung there; the 5 s `closeBudget`
+released `Close`, which called `markDead("closed")` → `kill()` → `sshSocket.Close()` →
+blocked in the other goroutine's `sync.Once`. The budget did not protect it.
+
+The precondition was verified in the pinned x/crypto v0.54.0 rather than assumed: the
+flow-control park (`remoteWin.reserve`, `channel.go:268`) happens *outside* `writeMu`, and
+only `writePacket` holds it across the TCP write. So a writer parked in SSH flow control
+never blocks the close; a writer inside the TCP write always does. Since the channel window
+is 2 MiB (`channelWindowSize = 64 * channelMaxPacket`), reproducing it by hand needs a
+kernel send buffer that fills first — shrink `SO_SNDBUF` well under 2 MiB.
+
+The older kvm provider does not have this bug: it closes `client` only
+(`internal/provider/kvm/conn.go:429,456`) and never touches the forwarded channel. kvm2's
+rewrite introduced it.
+
+### Tests: planned four, shipped one
+
+The first draft had three hermetic tests plus a `//go:build integration && linux`
+stalled-write test with a TCP proxy and shrunken socket buffers. The user pushed back —
+once the bug is fixed the connection closes properly, so what is left to test? — and they
+were right about most of it. `TestDial_forwardsUnixSocket` already covers "Close tears the
+connection down", and after the fix `Close` is a single call with no ordering to get wrong.
+The stalled-write test was the worst of the four: ~150 lines of Linux-only,
+buffer-autotuning-dependent scaffolding whose own false-green guard made it *skip* whenever
+the kernel did not cooperate, to demonstrate a bug that reading x/crypto settles and that
+the fix deletes the code path for.
+
+What survived is `TestMarkDead_slowKillDoesNotBlockCallers`, because the `markDead` change
+*adds* a guarantee that did not exist before and nothing else touches it —
+`TestCall_killedByAnotherCallerFailsTransient` passes a nil kill, so it cannot see the
+property. It is also the plausible regression: restoring the `sync.Once` looks like a
+correctness improvement. The call's own `fn` signals when it is in flight, so the parked
+caller is genuinely parked rather than refused by the dead check on the way in.
+
+The step that actually validates a regression test is running it against the unfixed code.
+`markDead` was reverted in place, the test run, and the fixed version restored from a copy:
+
+```bash
+cp internal/provider/kvm2/backend.go "$SCRATCH/backend.go.fixed"
+# ... revert markDead to the sync.Once form and re-add the once field ...
+go test -race -count=1 -run 'TestMarkDead_slowKillDoesNotBlockCallers' ./internal/provider/kvm2/
+#   --- FAIL: TestMarkDead_slowKillDoesNotBlockCallers (1.00s)
+#       backend_test.go:186: link not signalled dead while its kill is still running
+cp "$SCRATCH/backend.go.fixed" internal/provider/kvm2/backend.go
+```
+
+It fails in a second with the right diagnostic rather than hanging the suite, because the
+blocked kill is released by `t.Cleanup`.
+
+```bash
+go build ./... && go vet ./... && go test -race -count=1 ./internal/provider/kvm2/
+```
+
+### Worth noting
+
+**An accepted risk, deliberately not covered by a test.** Nothing now stops someone
+re-adding `s.Conn.Close()` to `sshSocket.Close` as tidiness; it would look harmless and
+every test would still pass, because in a healthy test the channel close returns instantly.
+The guard is the doc comment on `sshSocket`, which says outright that closing the channel
+would queue behind a stalled transfer. That is in idiom for this package — `hosts.go:25-34`
+and `create.go:151-153` carry the same kind of hard-won decision — but it is a comment, not
+a test, and the failure it prevents is a 15-minute operator-wide stall.
+
+**Left for later, on purpose.** `ReconciliationTimeout` in `SetupWithManager` does not exist
+in this repo at all; adding a per-reconcile deadline would change the contract for every
+provider, since `hosts.go` and `image.go` both cite "no caller deadline", so it needs its own
+change and its own spec entry. `sshSocket.keepalive` still has no test (sweep finding S1.2).
+
+**Docs.** `docs/kvm2-provider.md` said the adapter "kills the connection (socket and SSH
+client)", which is no longer true and was the description that made the bug look correct.
+It now states the order (mark dead, then kill) and why both steps are lock-free.
+`docs/kvm2-provider-spec.md` entries 18 and 19 were checked and are not falsified.
+
+**No CHANGELOG entry yet** — the repo's rule is to add one once the user confirms the change
+works, so it lands with a later "validate and record" step, as in the kvm campaign.

internal/provider/kvm2/backend.go

@@ -106,7 +106,6 @@ type link struct {
 	label string
 	kill  func()
 
-	once   sync.Once
 	closed chan struct{}
 	mu     sync.Mutex
 	reason string
@@ -133,18 +132,23 @@ func (l *link) deadReason() string {
 	return l.reason
 }
 
-// markDead records why the connection is unusable, closes the transport and
+// markDead records why the connection is unusable, signals dead and then
-// signals dead. Later calls are no-ops.
+// closes the transport. Later calls are no-ops. Signalling before killing,
+// and killing outside the lock, is what keeps a kill that takes its time
+// from holding up the callers waiting on dead or the next markDead: a
+// transport close can block for as long as the kernel retransmits.
 func (l *link) markDead(reason string) {
-	l.once.Do(func() {
 	l.mu.Lock()
+	if l.isDead() {
+		l.mu.Unlock()
+		return
+	}
 	l.reason = reason
+	close(l.closed)
 	l.mu.Unlock()
 	if l.kill != nil {
 		l.kill()
 	}
-		close(l.closed)
-	})
 }
 
 // call executes a blocking go-libvirt call with a budget and returns its

internal/provider/kvm2/backend_test.go

@@ -148,6 +148,71 @@ func TestCall_killedByAnotherCallerFailsTransient(t *testing.T) {
 	}
 }
 
+// TestMarkDead_slowKillDoesNotBlockCallers pins the guarantee the pooled
+// connection rests on: a kill that takes its time — closing a transport can
+// block for as long as the kernel retransmits to a host that went away —
+// holds up neither the callers waiting on dead nor the next markDead. The
+// call's own fn signals when it is in flight, so the parked caller is
+// genuinely parked rather than refused by the dead check on the way in.
+func TestMarkDead_slowKillDoesNotBlockCallers(t *testing.T) {
+	t.Parallel()
+	release := make(chan struct{})
+	t.Cleanup(func() { close(release) })
+	killing := make(chan struct{})
+	var killed atomic.Int32
+	l := newLink("beelink", func() {
+		killed.Add(1)
+		close(killing)
+		<-release
+	})
+
+	inflight := make(chan struct{})
+	waiting := make(chan error, 1)
+	go func() {
+		waiting <- l.run(context.Background(), "StorageVolUpload", time.Minute, func() error {
+			close(inflight)
+			<-release
+			return nil
+		})
+	}()
+	<-inflight
+
+	go l.markDead("keepalive failed")
+	<-killing
+
+	select {
+	case <-l.dead():
+	case <-time.After(time.Second):
+		t.Fatal("link not signalled dead while its kill is still running")
+	}
+	select {
+	case err := <-waiting:
+		if !errors.Is(err, provider.ErrTransient) || !strings.Contains(err.Error(), "keepalive failed") {
+			t.Errorf("parked call error = %v, want ErrTransient with the death reason", err)
+		}
+	case <-time.After(time.Second):
+		t.Fatal("parked call not released while the kill is still running")
+	}
+
+	second := make(chan struct{})
+	go func() {
+		l.markDead("closed")
+		close(second)
+	}()
+	select {
+	case <-second:
+	case <-time.After(time.Second):
+		t.Fatal("second markDead blocked behind the running kill")
+	}
+
+	if killed.Load() != 1 {
+		t.Errorf("kill hook called %d times, want 1", killed.Load())
+	}
+	if !strings.Contains(l.deadReason(), "keepalive failed") {
+		t.Errorf("deadReason() = %q, want the first reason", l.deadReason())
+	}
+}
+
 func TestBudgets_upload(t *testing.T) {
 	t.Parallel()
 	tests := []struct {

internal/provider/kvm2/ssh.go

@@ -96,8 +96,12 @@ func loadCredentials(cfg provider.KVMConfig) (credentials, error) {
 }
 
 // sshSocket is the libvirt socket forwarded through one SSH connection. The
-// embedded Conn is the channel; Close also closes the SSH client, which is
+// embedded Conn is the channel; Close closes the SSH client, whose TCP
-// what unblocks anything reading the channel.
+// connection carries it, and that alone unblocks the channel's readers and
+// writers. Closing the channel as well would be worse than useless: its
+// Close writes a close message under the channel's write lock, so it would
+// queue behind a transfer stuck in a TCP write, which is the case Close
+// exists to survive.
 type sshSocket struct {
 	net.Conn
 	client *ssh.Client
@@ -107,9 +111,7 @@ type sshSocket struct {
 }
 
 func (s *sshSocket) Close() error {
-	s.closeOnce.Do(func() {
+	s.closeOnce.Do(func() { s.closeErr = s.client.Close() })
-		s.closeErr = errors.Join(s.Conn.Close(), s.client.Close())
-	})
 	return s.closeErr
 }