feat/proxmox-provider
7
Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
  kacerrandClaude Opus 5.5
|
0a6c8624f3 |
provider: extract nocloud seed builder and keyedlock from kvm2
kvm2's NoCloud seed builder moves to internal/provider/nocloud (exported, with an ErrEmptyName sentinel instead of permanentf) and its keyed locks to internal/provider/keyedlock, tests moved under their existing names. kvm2 behaviour is unchanged. proxmox gains the per-name lock and the pure seed helpers (seedISOName, seedVolID, buildSeed); unlike kvm2 (review finding S5.1), a seed staging failure there is Transient, not Permanent. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> |
||
|
kacerrandClaude Opus 5
|
c10f98a352 |
kvm2: name the cached base image after the provider entry (H4)
baseVolumeName was base-<key>.<format>, byte-identical to what the kvm provider builds in the same pool -- for URL images too, since kvm takes hex(sum[:8]) and kvm2 takes hex(sum[:])[:16], the same eight bytes -- with a completely separate lock table. The shipped config has a live kvm-home entry and a commented-out kvm2-home entry on the same host, the same key debian-13 and the same default pool, and the comment invites the operator to enable it. Both integration tests use key test-image. On beelink both live overlays back onto kvm's base-debian-13.qcow2. Since cachedImage accepts any 64-hex marker when no sha256 is configured and never re-hashes the base, that meant two entries pointing one key at different URLs silently shared a file; a concurrent cold fetch let one entry delete the other's half-uploaded base and boot a guest from the remains; and a sha256 edit deleted the base in place under live overlays, which libvirt permits without complaint. The name now carries the provider entry: base-<entry>-<key>.<format>. imageSource gained an Entry field and resolveImage takes the provider name, so every production call site follows from imageSource.volumeName(). That closes the cross-entry sharing outright, kvm<->kvm2 and kvm2<->kvm2 alike, and turns the per-Provider imageLocks from a latent bug into a correct one: with names unique per entry, two entries cannot contend for a file, so no process-wide lock table is needed. The entry name now reaches a file name, so New checks it against the existing kvmIdentifier rule, exported as provider.IsKVMIdentifier rather than duplicated. Provider names are otherwise validated for all types at once and only for being non-empty and unique, so a name with a slash would have produced a volume nothing could address. One companion fix, because the naming alone does not stop it: cachedImage's digest-mismatch branch deleted the marker and the base and refetched. It now returns ErrPermanent naming both digests and telling the operator what to do. That branch was the remaining path that unlinks a good file under running domains, and the contract already said "add a new key instead" -- the code was contradicting it. The no-marker and non-hex-marker branches keep deleting, since there the base is debris. Not fixed, and said so in the docs rather than left to look closed: editing images.<key>.url is still a silent no-op, since nothing records the URL; and cross-process fencing still needs the deferred atomic publish (upload to a temp name, verify, publish), which lands where H2 put the discard. Migration is a non-event. The base-debian-13.qcow2 on beelink is kvm's, kvm keeps using it untouched, and kvm2 downloads its own copy once per host. Nothing is replaced under a running guest; the pool needs room for both, which the prerequisites now say. The eviction recipe in the guide named base-debian-13.qcow2, which after this change is kvm's file -- following it would have deleted the wrong provider's base. It now greps for the prefix and deletes the marker first. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> |
||
|
kacerrandClaude Opus 5
|
9682737c69 |
kvm2: an unreadable address is not a missing one (H3)
address() tried the guest agent then the host's ARP table and skipped any
error that was not a context error, returning ("", nil) after both. An
active domain with no IP maps to StateProvisioning; the controller clears
status.IP on that, and the health engine prunes the proxy's counters and
calls ForgetProxy in one tick once EffectiveHost() goes empty.
The damaging trigger was not the guest. The libvirt connection is pooled
per host, so when another reconcile's cancel or budget overrun killed it,
call returned "connection is closed" -- carrying no context sentinel, so
the check did not recognise it -- and Get returned a *successful*
instance in Provisioning with an empty IP. An unrelated proxy's timeout
dropped a healthy proxy out of rotation.
The discriminator was already in the spec without the conclusion being
drawn: during a normal boot on beelink exactly one source fails. The
agent errors because it is not up yet, while ARP answers without error
and with nothing, since the host never sees the guest's traffic. So one
source failing stays "no address yet"; every source failing, or a
connection already dead, is ErrTransient.
The explicit provider.ErrTransient wrap is load-bearing, not decoration.
Without it classify picks the first libvirt code out of the joined error,
and ErrInvalidArg, ErrAccessDenied and ErrNoSupport are Permanent -- an
address read failing would latch a running proxy as Failed, a worse
version of the harm being fixed.
Half of the finding is deliberately still open, and it is the half that
fires more often: a guest agent restarting inside a running VM still
demotes it, because that is indistinguishable from a VM still booting.
The discriminator is the previous IP, which the provider does not hold by
design and the controller already has. Follow-up finding to file against
proxy_controller.go:208-211 -- on StateProvisioning, keep an existing IP
and Provisioned condition for a few polls instead of clearing on the
first. The kvm2-local alternative (use the mark's createdAt as a
provisioning window) was rejected: createdAt is creation, not boot, and
it invents a deadline that belongs to the controller.
TestGet_addressErrorIsProvisioning failed against the new rule, correctly
-- it fails both sources with ErrAgentUnresponsive, and code 86 is
agent-specific, so the ARP source cannot produce it. Re-pointed as
TestGet_agentFailsAndARPEmptyIsProvisioning, the beelink boot it was
reaching for, which nothing covered before.
TestGet_everyAddressSourceFailingIsTransient covers the other side, both
sources erroring and the pooled link dying under the read; against the
unfixed code both return a Provisioning instance instead of an error.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
|
||
|
kacerrandClaude Opus 5
|
62561c21fe |
kvm2: stop handing go-libvirt a reader that can error (H2)
uploadImage streamed the HTTP body straight into StorageVolUpload. When that body failed part way through, go-libvirt deadlocked: requestStream starts its stream sender on two unbuffered channels and, on the error path, returns without ever reading outErr, so SendStream emits a StatusError packet and parks on `outErr <-` forever while the main goroutine parks on `abort <- true` forever. StorageVolUpload never returned. The 20-minute transfer budget rescued the caller only, and the socket kill did nothing, because both ends are blocked on channel sends rather than on I/O -- which is what the comment at backend.go:152 got wrong. Each failed upload leaked kvm2's call goroutine, go-libvirt's sender goroutine and a 4 MiB buffer, permanently, against a 128Mi pod limit and a reconciler that retries on backoff. The defect is upstream and stays there: no fork, no replace. Instead eofReader, at the go-libvirt seam so it covers the seed and marker uploads too, converts any source error to io.EOF -- the one outcome that ends a stream the normal way -- and keeps the real error for the caller. countingReader sits underneath untouched, so the existing length and digest checks report exactly what they did before. The download also gets its own deadline a minute inside the transfer budget. fetchImage already bounded the whole fetch at 20 minutes, so the body's death and the budget's socket kill were racing and the socket usually won; with the body a minute ahead, a stalled mirror ends through its reader and unwinds cleanly instead of leaking. discardVolume moves from uploadImage into fetchImage's upload-error branch, so the dedicated connection is closed and libvirtd has let go of the stream's fd before the file is unlinked. Moving it rather than adding a second Close keeps image_test's closeCount() == 1 true. That discard is still unfenced against a second provider entry sharing the volume name; H4 closes that by making the name per-entry. What remains: a failure originating in libvirtd mid-upload still leaks, since SendStream has already returned by then. Confirmed unfixable from here. Known gaps quotes the two-character upstream fix (make(chan bool, 1) / make(chan error, 1)) and the replace escape hatch. Unverified at ~85%: that libvirtd answers a short-but-clean stream finish with OK rather than an error. virFDStreamWrite enforces length only on writes past it and nothing compares offset to length on the close path, which is also why virsh vol-upload of a small file into a large volume works. If it errors we are back to a deadlock by a different door, which is exactly today's behaviour, so this cannot be worse. The execution file carries a probe recipe to settle it. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> |
||
|
kacerrandClaude Opus 5
|
0239fb7e69 |
kvm2: do not let a stalled write delay the connection teardown (H1)
sshSocket.Close was errors.Join(s.Conn.Close(), s.client.Close()). Go evaluates those left to right, so the blocking close ran before the unblocking one. s.Conn is the forwarded libvirt socket, and closing an SSH channel writes a close message under ch.writeMu — which go-libvirt already holds while streaming a base image through that same channel. Against a host that vanished without RST or FIN, that write blocks until the kernel stops retransmitting (tcp_retries2, ~15 min), and s.client.Close(), a bare net.Conn.Close() that takes no lock and unblocks the stalled writer through netpoll, was never reached. Close the client only; nothing wants the polite channel-close message, and reordering instead would leave an error to filter in a memoised closeErr. link.markDead compounded it by running kill() before close(l.closed), both inside a sync.Once. Reordering alone was not enough: sync.Once.Do blocks concurrent callers until the first body returns, so every other markDead — including the one in call is own ctx-cancel branch — queued behind the stalled kill, and cancelling the reconcile did not help. The once field is gone; the mutex the struct already had guards the close-exactly-once invariant, reason is set before the signal, and kill runs outside the lock. Blast radius was the whole operator, not one host: MaxConcurrentReconciles is 3 and ensureImage holds a per-image lock whose acquire is ctx-aware but reconcile contexts carry no deadline, so three Proxies on one host with an uncached image parked every worker. The same edits close a second route to the same pin: go-libvirt is Disconnect closes the net.Conn it was handed (libvirt.go:331 -> socket.go:164), and that conn is our sshSocket, so libvirtBackend.Close is Disconnect goroutine entered closeOnce and hung; the 5 s closeBudget released Close, which called markDead -> kill -> sshSocket.Close and blocked in the other goroutine is sync.Once. Verified against the pinned x/crypto v0.54.0 rather than assumed: the flow-control park (remoteWin.reserve, channel.go:268) is outside writeMu and only writePacket holds it across the TCP write, so a writer parked in SSH flow control never blocks the close and a writer inside the TCP write always does. To reproduce by hand, shrink SO_SNDBUF well under the 2 MiB channel window (channelWindowSize = 64 * channelMaxPacket) so the socket buffer fills before the window does. The kvm provider does not have this bug: it closes client only and never touches the forwarded channel. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> |
||
|
kacerrandClaude Fable 5.1
|
2510842396 |
kvm2: apply the milestone review
Shorter start wait with immediate verdicts for dead or paused domains, ListByTag without state or address RPCs, Timeout for an expired check round, no adoption of unmarked domains, reference-counted keyed locks, dead options removed, HEAD fallback on any non-2xx. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> |
||
|
kacerrandClaude Fable 5.1
|
fcbf8ddc91 |
kvm2: user guide, appendix column and section, architecture and README entries
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> |