kvm2: do not let a stalled write delay the connection teardown (H1)
sshSocket.Close was errors.Join(s.Conn.Close(), s.client.Close()). Go evaluates those left to right, so the blocking close ran before the unblocking one. s.Conn is the forwarded libvirt socket, and closing an SSH channel writes a close message under ch.writeMu — which go-libvirt already holds while streaming a base image through that same channel. Against a host that vanished without RST or FIN, that write blocks until the kernel stops retransmitting (tcp_retries2, ~15 min), and s.client.Close(), a bare net.Conn.Close() that takes no lock and unblocks the stalled writer through netpoll, was never reached. Close the client only; nothing wants the polite channel-close message, and reordering instead would leave an error to filter in a memoised closeErr. link.markDead compounded it by running kill() before close(l.closed), both inside a sync.Once. Reordering alone was not enough: sync.Once.Do blocks concurrent callers until the first body returns, so every other markDead — including the one in call is own ctx-cancel branch — queued behind the stalled kill, and cancelling the reconcile did not help. The once field is gone; the mutex the struct already had guards the close-exactly-once invariant, reason is set before the signal, and kill runs outside the lock. Blast radius was the whole operator, not one host: MaxConcurrentReconciles is 3 and ensureImage holds a per-image lock whose acquire is ctx-aware but reconcile contexts carry no deadline, so three Proxies on one host with an uncached image parked every worker. The same edits close a second route to the same pin: go-libvirt is Disconnect closes the net.Conn it was handed (libvirt.go:331 -> socket.go:164), and that conn is our sshSocket, so libvirtBackend.Close is Disconnect goroutine entered closeOnce and hung; the 5 s closeBudget released Close, which called markDead -> kill -> sshSocket.Close and blocked in the other goroutine is sync.Once. Verified against the pinned x/crypto v0.54.0 rather than assumed: the flow-control park (remoteWin.reserve, channel.go:268) is outside writeMu and only writePacket holds it across the TCP write, so a writer parked in SSH flow control never blocks the close and a writer inside the TCP write always does. To reproduce by hand, shrink SO_SNDBUF well under the 2 MiB channel window (channelWindowSize = 64 * channelMaxPacket) so the socket buffer fills before the window does. The kvm provider does not have this bug: it closes client only and never touches the forwarded channel. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
This commit is contained in:
@@ -450,9 +450,15 @@ applied by `call`):
|
||||
|
||||
go-libvirt's calls take no context and block, so the only way to abandon
|
||||
one is to close the connection under it. That is what a cancelled or
|
||||
overlong call does: the adapter kills the connection (socket and SSH
|
||||
client), returns `ErrTransient` wrapping the context or deadline error,
|
||||
and marks the connection dead so the next call redials. The consequence:
|
||||
overlong call does: the adapter marks the connection dead so the next call
|
||||
redials, kills it by closing the SSH client's TCP connection, which takes
|
||||
the forwarded socket with it, and returns `ErrTransient` wrapping the
|
||||
context or deadline error. It marks dead before it kills, and closes the
|
||||
TCP connection rather than the forwarded channel, because both of those are
|
||||
lock-free: closing the channel writes a close message under the channel's
|
||||
write lock, so it would queue behind a transfer stuck in a TCP write to a
|
||||
host that has gone away — for as long as the kernel retransmits, which is
|
||||
the one case the kill exists to survive. The consequence:
|
||||
**other RPCs in flight on that host's pooled connection fail once with
|
||||
`ErrTransient`** and their callers retry on a fresh connection. In the
|
||||
reconciler that is one requeue with backoff; in GC it is one skipped
|
||||
|
||||
Reference in New Issue
Block a user