kvm2: stop handing go-libvirt a reader that can error (H2)

uploadImage streamed the HTTP body straight into StorageVolUpload. When
that body failed part way through, go-libvirt deadlocked: requestStream
starts its stream sender on two unbuffered channels and, on the error
path, returns without ever reading outErr, so SendStream emits a
StatusError packet and parks on `outErr <-` forever while the main
goroutine parks on `abort <- true` forever. StorageVolUpload never
returned. The 20-minute transfer budget rescued the caller only, and the
socket kill did nothing, because both ends are blocked on channel sends
rather than on I/O -- which is what the comment at backend.go:152 got
wrong. Each failed upload leaked kvm2's call goroutine, go-libvirt's
sender goroutine and a 4 MiB buffer, permanently, against a 128Mi pod
limit and a reconciler that retries on backoff.

The defect is upstream and stays there: no fork, no replace. Instead
eofReader, at the go-libvirt seam so it covers the seed and marker
uploads too, converts any source error to io.EOF -- the one outcome that
ends a stream the normal way -- and keeps the real error for the caller.
countingReader sits underneath untouched, so the existing length and
digest checks report exactly what they did before.

The download also gets its own deadline a minute inside the transfer
budget. fetchImage already bounded the whole fetch at 20 minutes, so the
body's death and the budget's socket kill were racing and the socket
usually won; with the body a minute ahead, a stalled mirror ends through
its reader and unwinds cleanly instead of leaking.

discardVolume moves from uploadImage into fetchImage's upload-error
branch, so the dedicated connection is closed and libvirtd has let go of
the stream's fd before the file is unlinked. Moving it rather than adding
a second Close keeps image_test's closeCount() == 1 true. That discard is
still unfenced against a second provider entry sharing the volume name;
H4 closes that by making the name per-entry.

What remains: a failure originating in libvirtd mid-upload still leaks,
since SendStream has already returned by then. Confirmed unfixable from
here. Known gaps quotes the two-character upstream fix
(make(chan bool, 1) / make(chan error, 1)) and the replace escape hatch.

Unverified at ~85%: that libvirtd answers a short-but-clean stream finish
with OK rather than an error. virFDStreamWrite enforces length only on
writes past it and nothing compares offset to length on the close path,
which is also why virsh vol-upload of a small file into a large volume
works. If it errors we are back to a deadlock by a different door, which
is exactly today's behaviour, so this cannot be worse. The execution file
carries a probe recipe to settle it.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
This commit is contained in:
2026-09-17 17:46:15 +00:00
co-authored by Claude Opus 5
parent d826114fd3
commit 62561c21fe
5 changed files with 244 additions and 13 deletions
+36
View File
@@ -285,6 +285,18 @@ operations share. A cancelled or overlong call kills the connection it
ran on (see *Timeouts and connections*), and a poll for some other Proxy
being cancelled must not cut off a 20-minute upload.
A body that fails part way through is reported to the caller **out of
band**: the reader handed to `StorageVolUpload` turns any source error
into a clean end of stream and keeps the real error for afterwards, so
the short upload is caught by the length and digest checks rather than
by the stream. This is not tidiness. go-libvirt answers a reader error
by parking its stream sender forever on a channel it no longer reads, so
handing it one wedges the call and leaks the sender with its 4 MiB
buffer; see *Known gaps*. For the same reason the download carries its
own deadline, a minute inside the transfer budget, so a stalled mirror
ends through its body rather than through the killed socket the budget
would otherwise leave behind.
The download needs a `Content-Length`: the volume's capacity is declared
before the upload starts. The request sends `Accept-Encoding: identity`
so a transparently gzipped body cannot hide the length. A response
@@ -541,6 +553,30 @@ never contain key bytes, the known_hosts content or cloud-init.
## Known gaps
- **An upload that libvirtd itself fails leaks a goroutine, upstream.**
go-libvirt's `requestStream` hands its stream sender an unbuffered
channel and then, on the error path, returns without ever reading it:
```go
abort := make(chan bool) // rpc.go
outErr := make(chan error)
go func() { outErr <- l.socket.SendStream(...) }()
resp, err = l.processIncomingStream(c, in)
if err != nil {
abort <- true
return resp, err // outErr is never drained
}
```
Whenever the daemon errors mid-upload — a pool that fills, a volume
removed underneath — the sender parks on `outErr` for good, taking its
4 MiB buffer with it. Closing the socket does not help: both ends are
blocked on channel sends, not on I/O. kvm2 avoids the two cases it
controls (a failing body, and its own transfer deadline; see *The image
cache*), so what is left needs a server-side failure. The fix is two
characters upstream, `make(chan bool, 1)` and `make(chan error, 1)`; if
it ever bites hard enough, a `replace` carrying that patch is the
escape hatch.
- **Create blocks**, including the first image download on each host,
holding one of the three reconcile workers for its duration (LAN
speed, a few hundred MB, once per image per host). Other Proxies keep