kvm2: stop handing go-libvirt a reader that can error (H2)

uploadImage streamed the HTTP body straight into StorageVolUpload. When
that body failed part way through, go-libvirt deadlocked: requestStream
starts its stream sender on two unbuffered channels and, on the error
path, returns without ever reading outErr, so SendStream emits a
StatusError packet and parks on `outErr <-` forever while the main
goroutine parks on `abort <- true` forever. StorageVolUpload never
returned. The 20-minute transfer budget rescued the caller only, and the
socket kill did nothing, because both ends are blocked on channel sends
rather than on I/O -- which is what the comment at backend.go:152 got
wrong. Each failed upload leaked kvm2's call goroutine, go-libvirt's
sender goroutine and a 4 MiB buffer, permanently, against a 128Mi pod
limit and a reconciler that retries on backoff.

The defect is upstream and stays there: no fork, no replace. Instead
eofReader, at the go-libvirt seam so it covers the seed and marker
uploads too, converts any source error to io.EOF -- the one outcome that
ends a stream the normal way -- and keeps the real error for the caller.
countingReader sits underneath untouched, so the existing length and
digest checks report exactly what they did before.

The download also gets its own deadline a minute inside the transfer
budget. fetchImage already bounded the whole fetch at 20 minutes, so the
body's death and the budget's socket kill were racing and the socket
usually won; with the body a minute ahead, a stalled mirror ends through
its reader and unwinds cleanly instead of leaking.

discardVolume moves from uploadImage into fetchImage's upload-error
branch, so the dedicated connection is closed and libvirtd has let go of
the stream's fd before the file is unlinked. Moving it rather than adding
a second Close keeps image_test's closeCount() == 1 true. That discard is
still unfenced against a second provider entry sharing the volume name;
H4 closes that by making the name per-entry.

What remains: a failure originating in libvirtd mid-upload still leaks,
since SendStream has already returned by then. Confirmed unfixable from
here. Known gaps quotes the two-character upstream fix
(make(chan bool, 1) / make(chan error, 1)) and the replace escape hatch.

Unverified at ~85%: that libvirtd answers a short-but-clean stream finish
with OK rather than an error. virFDStreamWrite enforces length only on
writes past it and nothing compares offset to length on the close path,
which is also why virsh vol-upload of a small file into a large volume
works. If it errors we are back to a deadlock by a different door, which
is exactly today's behaviour, so this cannot be worse. The execution file
carries a probe recipe to settle it.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

docs/kvm2-provider.md

@@ -285,6 +285,18 @@ operations share. A cancelled or overlong call kills the connection it
 ran on (see *Timeouts and connections*), and a poll for some other Proxy
 being cancelled must not cut off a 20-minute upload.
 
+A body that fails part way through is reported to the caller **out of
+band**: the reader handed to `StorageVolUpload` turns any source error
+into a clean end of stream and keeps the real error for afterwards, so
+the short upload is caught by the length and digest checks rather than
+by the stream. This is not tidiness. go-libvirt answers a reader error
+by parking its stream sender forever on a channel it no longer reads, so
+handing it one wedges the call and leaks the sender with its 4 MiB
+buffer; see *Known gaps*. For the same reason the download carries its
+own deadline, a minute inside the transfer budget, so a stalled mirror
+ends through its body rather than through the killed socket the budget
+would otherwise leave behind.
+
 The download needs a `Content-Length`: the volume's capacity is declared
 before the upload starts. The request sends `Accept-Encoding: identity`
 so a transparently gzipped body cannot hide the length. A response
@@ -541,6 +553,30 @@ never contain key bytes, the known_hosts content or cloud-init.
 
 ## Known gaps
 
+- **An upload that libvirtd itself fails leaks a goroutine, upstream.**
+  go-libvirt's `requestStream` hands its stream sender an unbuffered
+  channel and then, on the error path, returns without ever reading it:
+
+  ```go
+  abort  := make(chan bool)    // rpc.go
+  outErr := make(chan error)
+  go func() { outErr <- l.socket.SendStream(...) }()
+  resp, err = l.processIncomingStream(c, in)
+  if err != nil {
+      abort <- true
+      return resp, err          // outErr is never drained
+  }
+  ```
+
+  Whenever the daemon errors mid-upload — a pool that fills, a volume
+  removed underneath — the sender parks on `outErr` for good, taking its
+  4 MiB buffer with it. Closing the socket does not help: both ends are
+  blocked on channel sends, not on I/O. kvm2 avoids the two cases it
+  controls (a failing body, and its own transfer deadline; see *The image
+  cache*), so what is left needs a server-side failure. The fix is two
+  characters upstream, `make(chan bool, 1)` and `make(chan error, 1)`; if
+  it ever bites hard enough, a `replace` carrying that patch is the
+  escape hatch.
 - **Create blocks**, including the first image download on each host,
   holding one of the three reconcile workers for its duration (LAN
   speed, a few hundred MB, once per image per host). Other Proxies keep

docs/plans-executions/2026-09-17-1515-kvm2-review-fixes.md

@@ -3,11 +3,13 @@ Plan: [docs/plans/2026-09-17-1515-kvm2-review-fixes.md](../plans/2026-09-17-1515
 
 - [x] Step 0 — Save plan into the repo
 - [x] Step 1 — H1: a stalled write must not delay the teardown
+- [x] Step 2 — H2: a source error must not reach go-libvirt's stream sender
+- [ ] Step 3 — H3: an unreadable address must not look like a missing one
+- [ ] Step 4 — H4: the base image cache must not be shared across provider entries
 
-Remaining: the other 51 findings in `review/REPORT.md`, in severity order — H2 (the
+Remaining after those: the 48 mediums and lows in `review/REPORT.md`. Each gets its own
-`StorageVolUpload` stream wedge), H3 (`address()` swallowing errors), H4 (the shared base
+checklist line here when it is planned, so this list stays a record of decided work rather
-image cache), then the mediums. Each gets its own checklist line here when it is planned,
+than a wish.
-so this list stays a record of decided work rather than a wish.
 
 ## Step 0 — plan committed
 
@@ -122,3 +124,89 @@ It now states the order (mark dead, then kill) and why both steps are lock-free.
 
 **No CHANGELOG entry yet** — the repo's rule is to add one once the user confirms the change
 works, so it lands with a later "validate and record" step, as in the kvm campaign.
+
+## Step 2 — H2: a source error must not reach go-libvirt's stream sender
+
+This one is a workaround, not a repair. The defect is upstream, and the entry says so in
+three places (code comment, `docs/kvm2-provider.md` *Known gaps*, commit message) so nobody
+later reads the wrapper as ours to delete.
+
+`requestStream` in the pinned go-libvirt starts its stream sender on two **unbuffered**
+channels and then, on the error path, returns without ever reading `outErr`. A non-EOF error
+from the reader makes `SendStream` emit a StatusError packet and return, whereupon it parks on
+`outErr <-` forever; libvirtd answers the StatusError with an error, so the main goroutine
+parks on `abort <- true` forever. `StorageVolUpload` never returns. The 20-minute budget
+rescues only the caller, and the socket kill does nothing at all here, because both ends are
+blocked on channel sends rather than on I/O — which is what the comment at `backend.go:152`
+got wrong. Every failed upload leaked two goroutines and 4 MiB against a 128Mi pod limit,
+with the reconciler retrying on backoff.
+
+Three changes.
+
+`eofReader` sits at the go-libvirt seam in `backend.go`, not in `image.go`, so it also covers
+the seed and marker uploads, and so it stays out of the file Step 4 rewrites. It converts any
+source error to `io.EOF` — the one outcome that ends a stream the normal way — and keeps the
+real error for `StorageVolUpload` to return afterwards. `countingReader` is untouched
+underneath it and still sees the true error, so `uploadImage`'s existing length and digest
+checks produce exactly the messages they did before.
+
+The download now carries its own deadline, one minute inside the transfer budget.
+`fetchImage` already bounded the whole fetch at 20 minutes, so the body's death and the
+budget's socket kill were racing, and the socket usually won. With the body a minute ahead, a
+stalled mirror ends through the reader and unwinds cleanly. This is the difference between a
+slow mirror costing a Transient error and costing a permanent leak, and slow mirrors are the
+realistic repeat offender.
+
+`discardVolume` moved from `uploadImage` up into `fetchImage`'s upload-error branch, so the
+dedicated connection is closed — and libvirtd has let go of the stream's fd — before the file
+is unlinked. Moving it rather than adding a second `Close` is what keeps
+`image_test.go`'s `closeCount() == 1` assertion true. `uploadImage` lost its `be` parameter
+and its private logger in the process.
+
+**What is left, deliberately.** A failure that originates in libvirtd mid-upload (ENOSPC, or
+the volume deleted underneath) still leaks, because by then `SendStream` has already returned
+and nothing we pass, cancel or close can reach it. Confirmed unfixable from this side. The
+upstream fix is two characters and is quoted in *Known gaps* along with the `replace` escape
+hatch, should it ever bite hard enough to be worth the maintenance.
+
+**Unverified, ~85%.** That libvirtd answers a short-but-clean stream finish with OK rather
+than an error. `virFDStreamWrite` enforces `length` only on writes past it and nothing
+compares `offset` to `length` on the close path, which is also why `virsh vol-upload` of a
+small file into a large volume works. If it answers with an error we land back in a deadlock
+by a different door — but that is precisely today's behaviour, so the change cannot make
+things worse. To settle it, on any box with a libvirtd (not beelink, which is read-only
+here), against the pinned go-libvirt:
+
+```sh
+virsh -c qemu:///session pool-define-as probe dir --target ~/vmprobe
+virsh -c qemu:///session pool-start probe
+virsh -c qemu:///session vol-create-as probe short.raw 10M --format raw
+# then a ~40-line main.go: StorageVolUpload(vol, r, 0, 10<<20, 0) where r
+# yields 1 MiB then io.EOF; print the error, then runtime.NumGoroutine()
+# and a goroutine profile after a two-second settle.
+```
+
+Expected: `err == nil` and the goroutine count back at baseline. The same program with a
+reader that returns a non-EOF error should wedge against the *unpatched* seam, which also
+confirms the diagnosis end to end.
+
+The negative check that matters was run on the unit test instead — `eofReader.Read` was
+changed back to returning the real error, and both new tests failed with
+`io.Copy through eofReader = connection reset by peer, want nil`, then passed again once
+restored.
+
+```bash
+go build ./... && go vet ./... && go test -race -count=1 ./internal/provider/kvm2/
+```
+
+### Worth noting
+
+The `fetchImage` discard is still unfenced against a second provider entry that shares the
+volume name — Step 4 (H4) is what closes that, by making the name per-entry. The deferred
+atomic-publish work then lands exactly where this step put the discard.
+
+**Deliberately not tested:** the abort protocol end to end. `libvirttest` reads a fixed
+28-byte header per packet and implements no stream procedures, so pinning this hermetically
+would mean a 150-250 line libvirt RPC server stub whose only assertion is about *upstream's*
+goroutines. Same call as H1's: the test would cost more than it protects. The two tests that
+did land pin the one property the seam actually needs.

internal/provider/kvm2/backend.go

@@ -2,6 +2,7 @@ package kvm2
 
 import (
 	"context"
+	"errors"
 	"fmt"
 	"io"
 	"sync"
@@ -90,6 +91,12 @@ const smallUploadBytes = 16 << 20
 // closeBudget bounds the courtesy ConnectClose RPC in Close.
 const closeBudget = 5 * time.Second
 
+// transferGrace keeps the image download's own deadline inside the transfer
+// budget. A stalled mirror then ends by its body failing, which the upload
+// seam turns into a clean end of stream, instead of by the budget killing
+// the socket, which go-libvirt's stream sender does not survive.
+const transferGrace = time.Minute
+
 func (b budgets) upload(length uint64) time.Duration {
 	if length <= smallUploadBytes {
 		return b.seedUpload
@@ -325,9 +332,50 @@ func (b *libvirtBackend) StorageVolCreateXML(ctx context.Context, pool libvirt.S
 }
 
 func (b *libvirtBackend) StorageVolUpload(ctx context.Context, vol libvirt.StorageVol, r io.Reader, offset, length uint64, flags libvirt.StorageVolUploadFlags) error {
-	return b.link.run(ctx, "StorageVolUpload", b.budgets.upload(length), func() error {
+	src := &eofReader{r: r}
-		return b.lv.StorageVolUpload(vol, r, offset, length, flags)
+	err := b.link.run(ctx, "StorageVolUpload", b.budgets.upload(length), func() error {
+		return b.lv.StorageVolUpload(vol, src, offset, length, flags)
 	})
+	if err == nil {
+		if srcErr := src.sourceErr(); srcErr != nil {
+			return fmt.Errorf("reading the upload source for %s: %w", vol.Name, srcErr)
+		}
+	}
+	return err
+}
+
+// eofReader hides a source error from go-libvirt's stream sender and keeps
+// it for the caller. SendStream answers a non-EOF error with a StatusError
+// packet and then blocks forever handing its result to an unbuffered
+// channel that requestStream no longer reads (go-libvirt rpc.go), which
+// wedges the call and leaks the goroutine along with its 4 MiB buffer;
+// closing the socket does not help, because both ends are parked on
+// channel sends rather than on I/O. io.EOF is the one outcome that ends
+// the stream the normal way. The field is guarded because an abandoned
+// call returns while go-libvirt is still reading.
+type eofReader struct {
+	r   io.Reader
+	mu  sync.Mutex
+	err error
+}
+
+func (e *eofReader) Read(b []byte) (int, error) {
+	n, err := e.r.Read(b)
+	if err == nil || errors.Is(err, io.EOF) {
+		return n, err
+	}
+	e.mu.Lock()
+	if e.err == nil {
+		e.err = err
+	}
+	e.mu.Unlock()
+	return n, io.EOF
+}
+
+func (e *eofReader) sourceErr() error {
+	e.mu.Lock()
+	defer e.mu.Unlock()
+	return e.err
 }
 
 func (b *libvirtBackend) StorageVolDownload(ctx context.Context, vol libvirt.StorageVol, w io.Writer, offset, length uint64, flags libvirt.StorageVolDownloadFlags) error {

internal/provider/kvm2/backend_test.go

@@ -3,6 +3,7 @@ package kvm2
 import (
 	"context"
 	"errors"
+	"io"
 	"strings"
 	"sync/atomic"
 	"testing"
@@ -213,6 +214,57 @@ func TestMarkDead_slowKillDoesNotBlockCallers(t *testing.T) {
 	}
 }
 
+// TestEOFReader_sourceErrorNeverReachesTheStream pins the one property the
+// upload seam needs: go-libvirt's stream sender must never see an error
+// other than io.EOF, because it answers one by parking forever on a channel
+// nothing reads. The real error has to survive for the caller.
+func TestEOFReader_sourceErrorNeverReachesTheStream(t *testing.T) {
+	t.Parallel()
+	boom := errors.New("connection reset by peer")
+	tests := []struct {
+		name    string
+		reader  io.Reader
+		wantN   int
+		wantErr error
+	}{
+		{name: "error after data", reader: io.MultiReader(strings.NewReader("abc"), errReader{boom}), wantN: 3, wantErr: boom},
+		{name: "error with no data", reader: errReader{boom}, wantErr: boom},
+		{name: "clean eof passes through", reader: strings.NewReader("abc"), wantN: 3},
+	}
+	for _, tc := range tests {
+		t.Run(tc.name, func(t *testing.T) {
+			t.Parallel()
+			e := &eofReader{r: tc.reader}
+			n, err := io.Copy(io.Discard, e)
+			if err != nil {
+				t.Fatalf("io.Copy through eofReader = %v, want nil: the stream sender must only ever see io.EOF", err)
+			}
+			if n != int64(tc.wantN) {
+				t.Errorf("copied %d bytes, want %d", n, tc.wantN)
+			}
+			if !errors.Is(e.sourceErr(), tc.wantErr) {
+				t.Errorf("sourceErr() = %v, want %v", e.sourceErr(), tc.wantErr)
+			}
+		})
+	}
+}
+
+func TestEOFReader_keepsTheFirstError(t *testing.T) {
+	t.Parallel()
+	first, second := errors.New("first"), errors.New("second")
+	e := &eofReader{r: io.MultiReader(errReader{first}, errReader{second})}
+	if _, err := io.Copy(io.Discard, e); err != nil {
+		t.Fatalf("io.Copy through eofReader = %v, want nil", err)
+	}
+	if !errors.Is(e.sourceErr(), first) {
+		t.Errorf("sourceErr() = %v, want the first error", e.sourceErr())
+	}
+}
+
+type errReader struct{ err error }
+
+func (r errReader) Read([]byte) (int, error) { return 0, r.err }
+
 func TestBudgets_upload(t *testing.T) {
 	t.Parallel()
 	tests := []struct {

internal/provider/kvm2/image.go

@@ -214,7 +214,12 @@ func (p *Provider) fetchImage(ctx context.Context, host provider.KVMHost, pool l
 	log := p.logger(ctx).WithValues("host", host.Name, "volume", name)
 	start := time.Now()
 
-	resp, err := p.do(ctx, http.MethodGet, src.URL)
+	// The body runs out of time before the surrounding budget does, so a
+	// mirror that stalls mid-image ends the upload through its reader
+	// rather than through the killed socket the budget would leave.
+	httpCtx, cancelHTTP := context.WithTimeout(ctx, defaultBudgets.transfer-transferGrace)
+	defer cancelHTTP()
+	resp, err := p.do(httpCtx, http.MethodGet, src.URL)
 	if err == nil {
 		err = checkResponse(src.URL, resp)
 	}
@@ -227,8 +232,11 @@ func (p *Provider) fetchImage(ctx context.Context, host provider.KVMHost, pool l
 	span.SetAttributes(attribute.Int64("kvm2.bytes", size))
 	log.V(1).Info("fetching base image into host pool", "url", src.URL, "bytes", size)
 
-	vol, digest, err := p.uploadImage(ctx, host, pool, be, src, resp.Body, size)
+	vol, digest, err := p.uploadImage(ctx, host, pool, src, resp.Body, size)
 	if err != nil {
+		// uploadImage has returned, so its dedicated connection is closed
+		// and libvirtd has let go of the stream before the file goes.
+		p.discardVolume(ctx, log, be, pool, name)
 		span.SetStatus(codes.Error, err.Error())
 		return libvirt.StorageVol{}, err
 	}
@@ -249,10 +257,10 @@ func (p *Provider) fetchImage(ctx context.Context, host provider.KVMHost, pool l
 
 // uploadImage creates the base volume and streams body into it over a
 // dedicated connection with a running sha256, then checks length and
-// digest. On any failure the volume is discarded and the error says why:
+// digest. The error says why: a body that ended early is Transient, a
-// a body that ended early is Transient, a digest that differs from the
+// digest that differs from the configured one is Permanent. Discarding
-// configured one is Permanent.
+// the volume is the caller's, once this connection is closed.
-func (p *Provider) uploadImage(ctx context.Context, host provider.KVMHost, pool libvirt.StoragePool, be backend, src imageSource, body io.Reader, size int64) (libvirt.StorageVol, string, error) {
+func (p *Provider) uploadImage(ctx context.Context, host provider.KVMHost, pool libvirt.StoragePool, src imageSource, body io.Reader, size int64) (libvirt.StorageVol, string, error) {
 	name := src.volumeName()
 	xmlDoc, err := buildVolumeXML(volumeSpec{Name: name, CapacityBytes: size, Format: src.Format})
 	if err != nil {
@@ -289,7 +297,6 @@ func (p *Provider) uploadImage(ctx context.Context, host provider.KVMHost, pool
 		err = permanentf("image %s: sha256 %s does not match the configured %s", src.URL, digest, strings.ToLower(src.SHA256))
 	}
 	if err != nil {
-		p.discardVolume(ctx, p.logger(ctx).WithValues("host", host.Name, "volume", name), be, pool, name)
 		return libvirt.StorageVol{}, "", err
 	}
 	return vol, digest, nil