kacerr/egress-proxies-operator
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

Plan the kvm2 H2, H3 and H4 fixes

Browse Source 
Extends the campaign plan with the three remaining high findings, each its
own step and commit, in the order H2 -> H3 -> H4 (H2 and H4 both touch
image.go, and H2 restructures the failure path H4 would otherwise be
written against).

H2 is a workaround, not a repair: the deadlock is upstream in go-libvirt,
whose requestStream never drains an unbuffered outErr on its error path.
Per the decision taken while planning, no fork and no replace directive;
we stop handing it a reader that can error, move the transfer deadline
onto the HTTP source so a stalled mirror unwinds cleanly, and document the
residual with the two-character upstream patch.

H4 names the base image per provider entry, which is what was chosen over
a content fingerprint. That fixes cross-entry sharing but not the silent
no-op on a url edit, so the plan says so rather than implying the finding
is closed.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
This commit is contained in:
kacerr
2026-09-17 17:42:34 +00:00
co-authored by Claude Opus 5
parent 0239fb7e69
commit d826114fd3
1 changed files with 329 additions and 193 deletions
Download Patch File Download Diff File Expand all files Collapse all files
docs/plans/2026-09-17-1515-kvm2-review-fixes.md
+329 -193
View File
@@ -4,239 +4,375 @@
## Context ## Context
The review of `feat/kvm2-provider` (`review/REPORT.md`) confirmed 52 findings: 4 high, The review of `feat/kvm2-provider` (`review/REPORT.md`) confirmed 52 findings: 4 high,
20 medium, 28 low. We are fixing them one at a time, highest severity first, each as its 20 medium, 28 low. We fix them one at a time, highest severity first, each as its own step and
own step and its own commit. This plan is the campaign; the execution file's checklist is its own commit. This plan is the campaign; the execution file's checklist is the authoritative
the authoritative progress record. Only **Step 1 (H1)** is detailed now — later steps get progress record. Steps are detailed as we reach them, so the plan is never a fiction about work
detailed as we reach them, so the plan is not a fiction about work not yet designed. not yet designed.
Precedent for this shape: `docs/plans/2026-09-10-2222-kvm-review-fixes.md`, the same Precedent for this shape: `docs/plans/2026-09-10-2222-kvm-review-fixes.md`.
campaign for the kvm provider.
### The bug being fixed in Step 1 (H1)
`sshSocket.Close` is what tears a kvm2 connection down. It is the `kill` of `link`
(`backend.go:216`), and killing the transport is the only way to abandon a go-libvirt call,
which takes no context and blocks.
```go
// ssh.go:111
s.closeErr = errors.Join(s.Conn.Close(), s.client.Close())
```
Go evaluates those arguments left to right, so **the blocking close runs before the
unblocking one**:
- `s.Conn` is the forwarded libvirt socket, a `*chanConn` over x/crypto's `*channel`.
`channel.Close` → `sendMessage` → `channel.writePacket` (`channel.go:223`), which takes
`ch.writeMu` and holds it across a raw TCP write.
- `s.client.Close()` is `connection.Close()` → a bare `net.Conn.Close()` on the TCP socket
(`connection.go:111`). It takes no lock, and Go's netpoll unblocks any pending write on it.
While go-libvirt streams a base image through that same channel (`StorageVolUpload`), the
writer sits inside `writePacket` holding `ch.writeMu`. Verified in x/crypto v0.54.0: the
flow-control park (`remoteWin.reserve`, `channel.go:268`) happens **outside** `writeMu`, so
only a writer stuck in the TCP write blocks `Close`. When the host is power-cut or
partitioned (no RST, no FIN), that write blocks until the kernel stops retransmitting —
`tcp_retries2`, about 15 minutes. `Close` queues behind it and never reaches the client
close that would have freed everything.
`markDead` then amplifies it (`backend.go:138-148`):
```go
l.once.Do(func() {
...
if l.kill != nil { l.kill() } // blocks here for ~15 min
close(l.closed) // never reached
})
```
Three consequences, each independently bad:
1. `l.closed` is never closed, so callers parked in `call`'s `case <-l.closed` are never released.
2. `isDead()` stays false, so `hostTable.get` keeps handing out the dying pooled backend and
each new call wedges the same way.
3. `sync.Once.Do` blocks *every* concurrent `markDead` caller — including `call`'s own
ctx-cancel branch (`backend.go:180`). **Cancelling the reconcile does not help.**
Blast radius: `MaxConcurrentReconciles: 3` (`proxy_controller.go:442`) and the per-image lock
(`ensureImage`/`lockImage`), whose `keyedLocks.acquire` is ctx-aware but reconcile contexts
carry no deadline. Three Proxies on one host with an uncached image park all three workers
for ~15 minutes, stalling every Proxy on every provider.
There is a **second route to the same pin**, fixed by the same edits: go-libvirt's
`Disconnect` closes the `net.Conn` it was handed (`libvirt.go:331` → `socket.go:164`), and
that conn *is* our `*sshSocket`. So `libvirtBackend.Close`'s `Disconnect` goroutine enters
`sshSocket.closeOnce` and hangs; the 5 s `closeBudget` releases `Close`, which calls
`markDead("closed")` → `kill()` → `sshSocket.Close()` → blocks in the other goroutine's
`sync.Once`. The budget does not protect it.
The older kvm provider does not have this bug: it closes `client` only (`kvm/conn.go:429,456`)
and never closes the forwarded channel. kvm2 introduced it.
## Steps ## Steps
- **Step 0** — Save this plan into the repo, commit alone, before any code. - [x] **Step 0** — Save this plan into the repo, commit alone, before any code.
- **Step 1** — H1: `sshSocket.Close` and `link.markDead` (detailed below). - [x] **Step 1** — H1: `sshSocket.Close` ran the blocking close before the unblocking one, and
- **Step 2+** — the remaining findings from `review/REPORT.md` in severity order (H2 upload `markDead` killed inside a `sync.Once` before signalling dead. Shipped in `0239fb7`; the
stream wedge, H3 `address()` swallowing errors, H4 shared base image cache, then the reasoning is in the execution file.
mediums). Detailed when reached; add a checklist line per step as it is planned. - [ ] **Step 2** — H2: an error during `StorageVolUpload` wedges go-libvirt's stream sender.
- [ ] **Step 3** — H3: `address()` swallows every error and demotes healthy proxies.
- [ ] **Step 4** — H4: the base image cache is shared across provider entries.
- [ ] **Step 5+** — the mediums, from `review/REPORT.md`. Detailed when reached.
--- ---
## Step 1 — H1: a stalled write must never delay tearing the connection down ## Step 2 — H2: a source error must not reach go-libvirt's stream sender
### 1a. `internal/provider/kvm2/ssh.go` — close the SSH client only ### The bug
Replace the body of `Close` (109-114) and correct the `sshSocket` doc comment (98-100), `uploadImage` (`image.go:255-296`) hands a reader over `resp.Body` to `StorageVolUpload`. I
which currently says Close "also closes the SSH client, which is what unblocks anything verified the mechanism in the pinned `go-libvirt@v0.0.0-20260814190004`:
reading the channel":
```go ```go
// sshSocket is the libvirt socket forwarded through one SSH connection. The // rpc.go requestStream
// embedded Conn is the channel; Close closes the SSH client, whose TCP abort := make(chan bool) // UNBUFFERED
// connection carries it. Closing the channel too would be worse than useless: outErr := make(chan error) // UNBUFFERED
// channel.Close writes a close message under the channel's write lock, so it go func() { outErr <- l.socket.SendStream(serial, proc, program, out, abort) }()
// would queue behind a transfer stuck in a TCP write — the case Close exists resp, err = l.processIncomingStream(c, in)
// to survive. if err != nil {
type sshSocket struct { ... } // fields unchanged abort <- true
return resp, err // outErr is NEVER drained on this path
}
err = <-outErr
```
func (s *sshSocket) Close() error { `SendStream` (`socket/socket.go`) holds a 4 MiB buffer, polls `abort` once per loop, and on a
s.closeOnce.Do(func() { s.closeErr = s.client.Close() }) **non-EOF reader error** sends a `StatusError` packet and returns. It then blocks forever on
return s.closeErr `outErr <-`, because the main goroutine is still inside `processIncomingStream`. libvirtd
answers the `StatusError` with an error, so `processIncomingStream` returns an error and the
main goroutine blocks forever on `abort <- true`. **Deadlock: `StorageVolUpload` never returns.**
kvm2's `call` rescues only the caller, after `budgets.transfer` = 20 minutes, and the socket
kill does **not** unblock either goroutine — both are parked on channel sends, not on socket
I/O, which is what the comment at `backend.go:152-156` gets wrong. So every failed upload leaks
kvm2's call goroutine, go-libvirt's sender goroutine and 4 MiB, permanently. The pod limit is
128Mi and the reconciler retries on backoff.
Three shapes, only the first of which is ours to fix:
- **(A) reader error** — a truncated or failing HTTP body. The common case.
- **(B) server-side error mid-upload** — ENOSPC, or the volume deleted underneath. After
`SendStream` has returned there is nothing any reader, ctx or socket close can do: confirmed
unfixable from our side.
- **(C) our own abandonment** — ctx cancel or the 20 m budget kills the socket, which makes
`processIncomingStream` error. Same tail as (B).
### The fix (workaround; the defect stays upstream)
No fork, no `replace`. Two changes that between them close (A) and the realistic half of (C).
**2a. Never hand go-libvirt a reader that errors** — `internal/provider/kvm2/backend.go`, at the
seam rather than in `image.go`, so it covers the seed and marker uploads too and stays out of
the file Step 4 rewrites:
```go
type eofReader struct {
r io.Reader
mu sync.Mutex
err error
}
func (e *eofReader) Read(b []byte) (int, error) {
n, err := e.r.Read(b)
if err == nil || err == io.EOF {
return n, err
}
e.mu.Lock()
if e.err == nil {
e.err = err
}
e.mu.Unlock()
return n, io.EOF
}
func (e *eofReader) sourceErr() error { e.mu.Lock(); defer e.mu.Unlock(); return e.err }
```
`libvirtBackend.StorageVolUpload` wraps `r` in one, and after a nil-error return converts a
recorded `sourceErr()` into the returned error. `io.EOF` is the one outcome that ends the stream
the normal way: libvirtd gets `StatusOK`, replies OK, `processIncomingStream` returns nil, the
main goroutine reaches `err = <-outErr` and both goroutines unwind.
`countingReader` is untouched — it sits under the wrapper, still sees the real error, so
`uploadImage`'s `readErr != nil` branch and `TestEnsureImage_shortReadRemovesVolumeAndIsTransient`
keep working verbatim.
**Unverified, ~85%:** that libvirtd answers a short-but-clean stream finish with OK rather than
an error. `virFDStreamWrite` enforces `length` only on writes *past* it and nothing compares
`offset` to `length` on the close path, which is also why `virsh vol-upload` of a small file
into a large volume works. If it answers with an error instead, we land back in a deadlock by a
different door — but that is exactly today's behaviour, so the change is non-worsening either
way. Probe recipe goes in the commit message; it needs a real libvirtd and a write, so it is
for the user to run, not an agent.
**2b. Make the transfer deadline land on the HTTP source, not on the libvirt socket.**
`p.do` already builds the request with `http.NewRequestWithContext` (`image.go:360`), so the
body is ctx-bound, and `http.DefaultClient` has no timeout of its own by deliberate choice
(`kvm2.go:100`). Give the image GET in `fetchImage` a deadline of `defaultBudgets.transfer`
minus a grace of one minute (precedent for reaching for `defaultBudgets` directly:
`discardVolume`, `image.go:331`).
A stalled mirror then ends by the *body* failing, which 2a turns into `io.EOF` and a clean
unwind, roughly a minute before the libvirt budget would have killed the socket and leaked.
That converts the realistic repeated (C) — a slow or stalling mirror on a 415 MB image — from a
permanent leak into an ordinary Transient error. The socket kill stays as the backstop for
libvirtd itself hanging, which is the residual we cannot close.
**2c. Close the dedicated connection before unlinking the file.** Move the `discardVolume` call
out of `uploadImage` (`image.go:291-294`) up into `fetchImage`'s upload-error branch, where the
deferred `fresh.Close()` has already run, so libvirtd has let go of the stream's fd before the
volume is deleted. `uploadImage` then drops its `be` parameter and its private logger — a net
deletion. Doing it this way rather than adding an explicit `Close` keeps
`image_test.go:573`'s `closeCount() == 1` assertion true.
### Tests, docs, scope
One test: `TestEOFReader_sourceErrorNeverReachesTheStream` in `backend_test.go` — a table with
"error after data", "error with no data", "plain io.EOF passes through", "first error wins".
~35 lines.
**Deliberately not tested:** the abort protocol end to end. `libvirttest` reads a fixed 28-byte
header per packet and implements no stream procedures, so pinning this hermetically would mean
writing a libvirt RPC server stub — 150-250 lines whose only assertion is about *upstream's*
goroutines. That is the bloat we cut for H1.
Docs: a paragraph in `docs/kvm2-provider.md`'s *The image cache* section saying a truncated
response is reported out of band and why, and a *Known gaps* bullet naming shapes (B) and (C)
as an upstream defect with the two-character patch quoted, so a future reader knows it is not
ours and what the escape hatch is:
```go
abort := make(chan bool, 1)
outErr := make(chan error, 1)
```
File that as an upstream issue/PR against `digitalocean/go-libvirt` and link it from the bullet.
---
## Step 3 — H3: an unreadable address must not look like a missing one
### The bug
`address()` (`get.go:104-119`) tries the guest agent then the host's ARP table. Any error that
is not `context.Canceled`/`DeadlineExceeded` is logged at V(2) and skipped; after both it
returns `("", nil)`. `mapState(active, "")` is `StateProvisioning` (`state.go:20-21`), the
controller clears `p.Status.IP` on Provisioning (`proxy_controller.go:208-211`), and the health
engine prunes the proxy's accumulated state and calls `Metrics.ForgetProxy` in one tick once
`EffectiveHost()` is empty.
The trigger that matters: another reconcile's cancel or budget overrun kills the **shared**
pooled connection, and `call` then returns `transientf("... connection is closed: ...")` with no
ctx sentinel, which `get.go:108` does not recognise. `Get` returns a *successful* Instance in
Provisioning with `IP: ""`, and a healthy proxy is pulled out of rotation by an unrelated
reconcile's timeout.
### The fix
A booting VM and a dead link are distinguishable: on beelink the agent source errors while the
ARP source returns success-with-no-addresses (`docs/kvm2-provider-spec.md:146` — the host never
sees the guest's traffic). So *one* source failing is normal; *every* source failing, or a
connection already gone, is not.
```go
func address(ctx context.Context, log logr.Logger, be backend, dom libvirt.Domain) (string, error) {
errs := make([]error, 0, len(addressSources))
for _, s := range addressSources {
ifaces, err := be.DomainInterfaceAddresses(ctx, dom, uint32(s.src), 0)
if err != nil {
if isDead(be) || errors.Is(err, context.Canceled) || errors.Is(err, context.DeadlineExceeded) {
return "", fmt.Errorf("reading addresses of domain %s from the %s source: %w", dom.Name, s.name, err)
}
log.V(2).Info("no addresses from source", "source", s.name, "error", err.Error())
errs = append(errs, fmt.Errorf("%s source: %w", s.name, err))
continue
}
if ip := firstIPv4(ifaces); ip != "" {
return ip, nil
}
}
if len(errs) == len(addressSources) {
return "", fmt.Errorf("%w: reading addresses of domain %s: every source failed: %w",
provider.ErrTransient, dom.Name, errors.Join(errs...))
}
return "", nil
} }
``` ```
Dropping `s.Conn.Close()` rather than reordering it, because after the TCP conn is closed the The explicit `provider.ErrTransient` wrap is load-bearing, not decoration: without it `classify`
channel close returns one of several errors (`net.ErrClosed`, a `*net.OpError`, the transport's picks the first libvirt code out of the join tree, and `ErrInvalidArg`/`ErrAccessDenied`/
sticky write error, or `io.EOF` once `mux.loop` has run `ch.close()`), and `closeErr` is `ErrNoSupport` map to **Permanent** — an address read failing would latch a *running* proxy as
memoised — a filter that misses a case would put a spurious error in every later `Close` and Failed, a worse version of the harm being fixed. `isDead(be)` (`backend.go:50-61`, already
break `ssh_test.go:66,69`, which assert both closes return nil. Nothing needs the polite works through the `observed` decorator and the test fake) is not strictly needed — a dead link
channel-close message: `dialSSHSocket` builds the client solely for this one channel, the fails both sources anyway — but it fails faster, attributes better, and is the same predicate
`sshSocket` owns it exclusively, every Close is a full teardown, and `dialSSHSocket`'s own M15 will need for `ListByTag`.
error paths (181, 188) already tear down by closing the client. Closing the client is
sufficient: `mux.loop` exits and `chanList.dropAll()` → `ch.close()` unblocks the channel's
readers and writers.
`errors` stays imported (`errors.As` at 52 and 90). No production caller inspects the error: Also fix the now-wrong comment at `get.go:100-103` ("only ctx ending is an error").
`backend.go:216`, `hosts.go:78`, `check.go:90`, `image.go:265` all discard it.
### 1b. `internal/provider/kvm2/backend.go` — signal dead before killing, and drop the `sync.Once` **What this does not fix, on purpose.** A guest agent restarting inside a running VM still
demotes it: the agent errors, ARP returns empty, and the provider cannot tell that from a VM
that is still booting. The discriminator is the *previous* IP, which the provider deliberately
does not hold (`TestGet_readsUIDAndCreatedAtFromBackend` pins "read back, not remembered") and
the controller already has. File as a follow-up finding against
`proxy_controller.go:208-211`: on Provisioning, keep a Proxy's existing IP and `Provisioned`
condition for N polls before clearing, or give the health engine a grace period before pruning.
The one kvm2-local alternative — treating "active, no address, domain older than a provisioning
window" as an error using the mark's `createdAt` — is a design change wearing a bug fix's
clothes, and it invents a provisioning deadline that belongs to the controller. Rejected.
Reordering alone is not enough: `sync.Once.Do` blocks concurrent callers until the first body ### Tests
returns, which is consequence 3 above. Delete the `once sync.Once` field from `link` (109) and
guard with the mutex the struct already has:
```go - `TestGet_arpFallbackWhenAgentFails` (`get_test.go:135`) passes unchanged — one source of two.
// markDead records why the connection is unusable, signals dead and then - `TestGet_addressErrorIsProvisioning` (`get_test.go:159`) must be **re-pointed**. It fails both
// closes the transport. Signalling first means a kill that takes its time sources with `ErrAgentUnresponsive`, which is not realistic: code 86 is agent-specific and the
// holds up neither the callers waiting on dead nor the next markDead. ARP source never produces it. Rewrite as `TestGet_agentFailsAndARPEmptyIsProvisioning` — agent
func (l *link) markDead(reason string) { errors, ARP returns no addresses and no error — which is the beelink boot and the behaviour
l.mu.Lock() the new rule must not break. Nothing covers it today.
if l.isDead() { - One new test, `TestGet_everyAddressSourceFailingIsTransient`, two subtests: both sources error;
l.mu.Unlock() and the pooled link is dead before the address read (the fake implements `liveness`). Assert
return Transient, not NotFound, not Permanent.
}
l.reason = reason
close(l.closed)
l.mu.Unlock()
if l.kill != nil {
l.kill()
}
}
```
`isDead` is a non-blocking select on `l.closed` and takes no lock, so it is safe under `mu` Docs: `docs/kvm2-provider-spec.md:146-151` says "errors from either source mean 'no address yet'",
and keeps one source of truth. `mu` holds the invariant that exactly one goroutine closes the now false. The `ErrTransient` row of the error-class table in `docs/kvm2-provider.md`.
channel and runs `kill`. `reason` is set before `close`, so a caller released by `<-l.closed`
always reads a populated `deadReason()`. `kill` runs outside the lock.
Update the comment at 136-137 ("closes the transport and signals dead" — now reversed). The ---
`link` doc (100-104) and `newLibvirtBackend`'s comment (210-212) stay accurate.
Consequences checked, all safe: `call`'s five branches; `hostTable.get` may redial a few ## Step 4 — H4: the base image cache must not be shared across provider entries
microseconds before the old socket is gone (two TCP connections overlap briefly);
`libvirtBackend.Close`'s early return on `isDead()` skips no kill, since `isDead()` now means
"some goroutine has closed `closed` and is running `kill`", still exactly once; the disconnect
watcher and keepalive goroutine exit sooner instead of parking in `once.Do`; `hosts.go:153`,
which calls `markDead` while holding `e.sem`, is bounded by one `net.Conn.Close`. When `Close`
is itself the first marker, `kill` runs synchronously, so Close still returns only after the
socket is closed — `TestLibvirtBackend_closeIsIdempotent` is unaffected.
**Not** running `kill` in a goroutine: `Close` would return before the socket was really ### The bug
closed, which `mockAdapter`'s cleanup and `image.go:265`'s `defer fresh.Close()` rely on, and
it would leave an fd-leak window on the failed-dial path.
### 1c. Test — one, not four `baseVolumeName` (`volume.go:17`) is `base-<key>.<format>`, with a `.sha256` marker beside it.
`internal/provider/kvm/image.go:44-56` builds **byte-identical** names in the same pool — for
URL images too (kvm takes `hex(sum[:8])`, kvm2 takes `hex(sum[:])[:16]`, the same 8 bytes) —
with a completely separate lock table. The shipped config has a live `kvm-home` entry and a
commented-out `kvm2-home` entry on the same host, the same key `debian-13` and the same default
pool, and the comment tells the operator to enable it. Both integration tests use key
`test-image`. On beelink both live overlays back onto `base-debian-13.qcow2`, which kvm created.
Revised down from an earlier draft of four tests. What the other three would have asserted is Consequences, given `cachedImage` accepts *any* 64-hex marker when no `sha256` is configured and
either already covered or no longer has a code path to guard: never re-hashes the base:
- "Close tears the connection down properly" is covered by `TestDial_forwardsUnixSocket` - Two entries, one key, different URLs: whichever fetched first owns the file and the other
(ssh_test.go:66-71), which closes twice and asserts both return nil. After 1a, `Close` is a silently boots the wrong image.
single `s.client.Close()` — there is no ordering left to get wrong. - A concurrent cold fetch: B finds A's base-without-marker, deletes it, recreates and uploads
- A stalled-write end-to-end test (TCP proxy, shrunken `SO_SNDBUF`/`SO_RCVBUF`, window its own; A finishes into the unlinked inode, writes the marker, and `StorageVolGetPath`
priming) was dropped: ~150 lines of Linux-only, buffer-autotuning-dependent scaffolding resolves A's handle to B's partial file — a guest boots from a half-written base.
whose own false-green guard makes it *skip* when the kernel does not cooperate. Its value - A `sha256` edit deletes the base in place under live overlays (`cachedImage` branch E), and
was demonstrating the bug, which reading x/crypto already settles, and 1a removes the call libvirt does not refuse to delete a volume that running domains use as a backing store.
it would exercise. The repro recipe goes in the commit message instead.
**`TestMarkDead_slowKillDoesNotBlockCallers`** — `backend_test.go`. Kept because 1b *adds* a ### The fix — the provider entry's name in the volume name
guarantee that does not exist today ("a slow kill blocks neither the callers waiting on dead
nor the next `markDead`") and nothing else in the suite touches it —
`TestCall_killedByAnotherCallerFailsTransient` passes `newLink("beelink", nil)`, a nil kill, so
it cannot see the property. It is also the plausible regression: restoring the `sync.Once`
looks like a correctness improvement and would silently reinstate the stall.
A `kill` that signals it has started and then blocks on a channel closed by `t.Cleanup`. `base-<entry>-<key>.<format>`, e.g. `base-kvm2-home-debian-13.qcow2`, with the marker following
Reuses the existing `blockingFn` helper (backend_test.go:19). With a call parked in `l.run`, automatically. Every production call site already goes through `imageSource.volumeName()`
assert within one second each: `<-l.dead()` closes; the parked call returns `ErrTransient`; a (`image.go:49`), so this is a small change: give `resolveImage` the provider name and add an
second `markDead` from another goroutine returns; `killed.Load() == 1`; `deadReason()` is the `Entry` field to `imageSource`.
first reason. Fails by timeout against the current code. `t.Parallel()`, generous bounds,
`errors.Is` against `provider.ErrTransient`, per repo style.
**Accepted risk:** nothing then stops someone re-adding `s.Conn.Close()` to `sshSocket.Close` `Provider.name` is already carried (`kvm2.go:104`). But provider names are validated only as
as tidiness. The guard is the doc comment in 1a saying outright that closing the channel would non-empty and unique (`config.go:96-105`) — no charset rule — and a name with a `/` would
queue behind a stalled transfer — in idiom for this package (`hosts.go:25-34`, produce an unusable volume name, so **`New` must reject an entry name that is not a safe volume
`create.go:151-153`), but a comment, not a test. name component**, using the existing `kvmIdentifier` shape. Fail fast at startup, in kvm2 only:
tightening the rule for every provider type is a broader change than this fix.
### 1d. Docs **What this buys, and what it does not.** It fixes the cross-entry sharing outright — kvm↔kvm2
and kvm2↔kvm2 — which is the wrong-image bug and the concurrent-fetch corruption, and it makes
the existing per-`Provider` `imageLocks` correct rather than a latent bug, since no two entries
can now contend for one name. It does **not** make the name mean the content, so two things
stay open and must be said plainly rather than quietly skipped:
`docs/kvm2-provider.md:451-453` says the adapter "kills the connection (socket and SSH - Editing `images.<key>.url` without changing the key is still a silent no-op. Already
client)". It now closes the SSH client, which takes the forwarded socket with it — reword, and documented behaviour ("add a new key instead"); left as is.
optionally add the one-clause reason (a lock-free close is the only kind that survives a - Cross-*process* fencing still depends on the atomic-publish work (upload to a temp name,
transfer stuck in a TCP write) near the 20 m upload budget row. verify, then publish). E4 refuted the rollout and leader-handover triggers for the shipped
manifests, so this is a narrow residual. Its own later commit.
`docs/kvm2-provider-spec.md` entries 18 and 19 say the adapter "closes the underlying **One companion fix, in this commit**, because the naming alone does not stop it and it is the
connection" and describe the pool; neither is falsified, so no spec change. remaining path that unlinks a good file under a running domain: `cachedImage` branch E
(`image.go:129-130`, digest differs from the configured `sha256`) currently deletes the marker
and the base. Make it return Permanent instead — "this key is cached with digest X but Y is
configured; add the image under a new key, or delete the Proxies using it and evict the base."
That enforces the rule the contract already states instead of contradicting it. Keep the delete
for the no-marker and non-hex-marker branches, where the base is genuine debris.
### 1e. Explicitly out of scope ### Migration
- `ReconciliationTimeout` in `SetupWithManager`: does not exist in the repo today, and a Nothing to do. The existing `base-debian-13.qcow2` on beelink is kvm's, kvm keeps using it
per-reconcile deadline changes the contract for every provider (`hosts.go` and `image.go` untouched, and kvm2 downloads its own copy once per host (~415 MB) when `kvm2-home` is enabled.
both cite "no caller deadline"). Its own change, with its own spec entry. No window in which anything is replaced under a running guest. The pool needs room for both.
- `sshSocket.keepalive` having no test (sweep finding S1.2) — separate finding.
- The kvm provider: verified not to have this bug.
### Verification ### Tests and docs
Tests are mostly literals becoming helpers: `volume_test.go:122-125`, `image_test.go:184,190,196`
(`wantVolume`), `create_test.go:64-70`, `delete_test.go:67`, `provider_test.go:409`; a small
`fx.baseName()` helper covers the last three. `image_test.go:281,298` should assert the log
field and the `kvm2.volume` span attribute **by prefix**, which keeps pinning that the key stays
readable in logs. `kvm2_integration_test.go:481-484` switches from `baseVolumeName(...)` to
`resolveImage(...).volumeName()`; its assertion that the base survives `Delete` is unchanged.
Two new tests:
- `TestBaseVolumeName_isPerProviderEntry` — two entries, same key, different names; the produced
name never equals the kvm provider's `base-<key>.<format>`.
- `TestNew_rejectsUnsafeEntryName` — an entry name with a `/` fails at construction.
**Not tested:** that libvirt permits deleting an in-use backing file (established from libvirt
source by the review; re-proving it needs a write on a real host), and the cross-process race
(no hermetic way, and its fix is deferred).
Docs: the *What lands on the host* table (`docs/kvm2-provider.md:198-212`), the eviction recipe
(`:305-318`, which currently names `base-debian-13.qcow2` and would delete kvm's file), the
migration section (`:346-370` — kvm2 never reuses a kvm base, one extra download, nothing to
delete), the prerequisites note that the pool needs one base per distro **per entry**, the
naming contract in `docs/kvm2-provider-spec.md` entries 4 and 10, the shared validator's message
at `internal/provider/config_kvm.go:139-141` ("it becomes the cached volume name
`base-<key>.<format>`", now wrong for kvm2), and the kvm2 block's comment in both
`providers-config.yaml` files. Leave `docs/kvm-provider.md` alone; kvm is unchanged.
---
## Ordering
**H2 → H3 → H4**, three commits.
H3 is orthogonal: it touches only `get.go` and `get_test.go`.
H2 and H4 both touch `image.go` but at different heights, and the order matters. H2's `image.go`
edit is a deletion — `discardVolume` moves out of `uploadImage` and up into `fetchImage` — while
H4 edits `imageSource.volumeName()`, `resolveImage`, and `cachedImage`'s digest branch ten lines
below. Landing H2 first means H4 rebases cleanly; the reverse means H4 is written against a
failure path H2 is about to restructure. Keeping H2's guard in `backend.go` rather than in
`countingReader` is part of the same separation.
Semantically, H2 makes `fetchImage` delete the base after the transfer connection is closed, and
that delete is only unambiguously safe once names are per-entry — so H2's commit message should
say the discard is still unfenced against a second entry sharing the name and that H4 closes it,
and H4's should point back. The deferred atomic-publish work then lands exactly where H2 put the
discard, on a volume this call created under a name no other entry can produce.
## Verification
```bash ```bash
go build ./... go build ./... && go vet ./...
go test -race ./internal/provider/kvm2/ go test -race -count=1 ./internal/provider/kvm2/
make test make test
``` ```
Then the step that actually matters: **confirm the new test fails against the unfixed code.** Per step, the check that actually matters is that the new test fails against the unfixed code —
Stash the `markDead` edit while keeping the test, run it, watch it fail by timeout, restore. A stash the source edit, keep the test, watch it fail, restore. For H2 that means asserting
regression test that passes both ways is worthless. `eofReader` converts before the fix exists; for H3, that the dead-link subtest returns a
successful Provisioning instance today; for H4, that two entries produce one name today.
Record in the commit message: the reproduction recipe (shrunken `SO_SNDBUF` so the socket After H4, run the kvm2 integration test on beelink. **It has never run** — Step 11 is unchecked
buffer fills before the 2 MiB SSH window, and why `remoteWin.reserve` sitting *outside* in `docs/plans-executions/2026-09-16-1333-kvm2-provider.md`, every `base-*` on the host predates
`writeMu` means only a writer inside `writePacket` can block the close), and the second route kvm2, and both live domains carry the kvm v1 metadata namespace. It is the first end-to-end
through go-libvirt's `Disconnect` closing the `net.Conn` it was handed. exercise of the new name and does a real cold download beside kvm's untouched base.
### Commit Each step: its own commit on `feat/kvm2-provider` with the execution-file entry and its checklist
line ticked. `CHANGELOG.md` lands with a "validate and record" step once the user confirms the
One commit on `feat/kvm2-provider`: the two source edits, the one test, the doc reword, the work, as in the kvm campaign. Trailer:
execution-file entry with its checklist line ticked, and the `CHANGELOG.md` entry (timestamp
from `date "+%Y-%m-%d %H:%M %Z"`) once the user confirms it works. Trailer:
`Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>`. `Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>`.