Phase 10. Two review passes (go-operator-reviewer, /code-review high)
over the branch, recorded in docs/reviews/2026-09-23-proxmox-provider-review/:
9 findings fixed, 3 accepted as documented limits, 7 rejected with a
reason.
- Delete removes the seed between the stop and the destroy, so a failed
seed delete leaves the VM for the next Delete instead of leaking the
ISO; a Permanent refusal on the every-node path fails the call.
- Create skips a probe VMID whose listed name is someone else's without
waiting on its lock.
- The three answers Phase 0 never captured are fixtures now: a missing
storage is a 400 (the Checker reports StorageNotFound), a hibernated
VM is destroyable (Delete destroys ours), a paused VM stops.
- TestProxmox_lifecycle and TestProxmoxCheck_realHost passed on pv1-1
(PVE 9.2.20); the docs record the run and drop "not yet run".
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
The optional proxmox-token Secret (token, ca.pem) mounted at
/etc/proxy-operator/proxmox, make proxmox-token-secret with content
checks that never print the token, a commented proxmox-home entry in
both providers configs, and config/samples/proxy_proxmox.yaml.
TestLoadConfig_proxmoxSampleEntry loads the commented entry from both
files, and a second test ties the sample Proxy's placement to it.
docs/proxmox-provider.md is the user guide: prerequisites and privileges,
cluster setup, config reference, placement, what lands on Proxmox,
logging, tracing, timeouts, health check, error classes, known gaps and
testing. providers-existing.md gains the proxmox column and section;
architecture.md, the README, providers.md and CLAUDE.md gain their rows.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
The contract sweep in provider_test.go: every method and Check returns
within a second of a cancel, every wrapErr path carries the configured
name, op and id, the V0/V1/V2 logging tiers and the no-op lines, and a
Get over the real client whose HTTP spans are children of provider.get.
proxmox_integration_test.go (-tags=integration) holds TestProxmox_lifecycle
and TestProxmoxCheck_realHost; both compile and skip without
PROXMOX_TEST_*, and have not run against a host yet. make
test-integration-proxmox runs them. docs/testing.md gains the proxmox,
nocloud and keyedlock rows, the Live Proxmox gaps and the integration
section; spec entry 21's section 11 list is ticked.
Waits that could hang the package under a mutation are bounded: the
concurrent-Create loop and eight direct blocking calls in older tests.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Delete stops and destroys a VM of ours under the per-name lock, then
deletes its seed on the VM's node, or on every node when no VM of ours
was left. A hibernated VM of ours is Permanent, as in Create. The
Checker verifies each node concurrently (version, node status, ISO
storage, templates) and adds CertificateInvalid, CredentialsUnavailable,
StorageNotFound and TemplateNotFound to the shared reasons. The
ProviderHealth type comment gains proxmox.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Create validates placement, then under the per-name lock adopts the VM a
previous attempt left (by name, or at a taken VMID after the ownership
check of spec entry 2) or full-clones the template onto the UID's
deterministic VMID with the mark in the clone call. It then uploads the
seed over any existing file, writes cores, memory, agent, tag, ide2 and
net0 when configured, grows the boot disk when diskGiB is larger, starts,
and waits for running/running.
waitUnlocked no longer waits on a suspended lock: a hibernated VM is
judged at once, so ours is Permanent and a foreign one is probed past.
The template is checked only on the allocation path, and its shape
(net0, boot disk size) is refused before any clone exists. Spec entries
2, 4 and 10 updated to match.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Get follows spec entry 9's check order (lock and unknown status before
the name, suspended is Stopped after the name and mark checks), reads
status/current so paused is Stopped, and maps the three agent messages
of entry 7. ListByTag reads one config per candidate and fails whole
only on the resources call or an ended ctx. waitTask polls the UPID's
node every budgets.poll, logs failed tasks at V(1) and counts
"WARNINGS: <n>" as success; waitUnlocked serves Create's ownership check.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
httpAPI implements the sixteen seam methods over net/http: the token is
read per request, non-2xx answers become apiError from the body message or
the reason phrase, echoed secrets are redacted, the CA loads lazily, HTTP/2
is off, and uploads stream with an exact Content-Length. The observed
decorator logs one V(1) line per call and starts no span; New wires both.
fakeAPI simulates clone locks, tasks and ISOs with the captured error texts.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
kvm2's NoCloud seed builder moves to internal/provider/nocloud (exported,
with an ErrEmptyName sentinel instead of permanentf) and its keyed locks
to internal/provider/keyedlock, tests moved under their existing names.
kvm2 behaviour is unchanged. proxmox gains the per-name lock and the pure
seed helpers (seedISOName, seedVolID, buildSeed); unlike kvm2 (review
finding S5.1), a seed staging failure there is Transient, not Permanent.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Phase 2 of the proxmox provider plan. Adds the proxmox: config block with
its validation, registers the type in the config allow-list and the
constructor map, and starts internal/provider/proxmox: the api seam and
its data types, providerIDs (vms/<vmid>/<name>) with the UID-derived VMID,
the description mark, state mapping, and error classification.
Tests read the responses captured from pv1-1 in Phase 0 rather than
repeating Proxmox's wording, and every matcher is run against every
captured error so none fires on the wrong message. The contract methods
are Transient stubs until Phases 5 to 7.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Phase 1 of the proxmox provider plan: docs/proxmox-provider-spec.md
answers all 24 entries of the provider spec template against the
contract, citing the Phase 0 facts from pv1-1 as evidence.
It folds in the ten corrections recon found. The two that matter most:
Create inspects a taken VMID, waiting out a clone lock, before probing
past it, so a crash-retry adopts its own half-cloned VM instead of cloning
a duplicate; and Get checks the lock before the name, because a VM being
cloned has no name yet.
The plan now opens with a note that the spec wins where they disagree.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Phase 0 of the proxmox provider plan. 55 fixtures under
internal/provider/proxmox/testdata, each the status, reason phrase and
raw body of a real response from the user's host, and the execution log
answering the plan's facts F1 to F23.
Recon ran a clone of the Debian 13 template through the whole path the
provider will take, twice: untagged and on VLAN 4. On both, the NoCloud
seed on ide2 installed the guest agent and an address came back about 50
seconds after start. It also found two bugs in the plan: a VM being
cloned has no name in /cluster/resources, so name-first lookups would read
it as gone, and a crash-retry would clone a duplicate. Those and eight
smaller corrections are listed for the spec.
Decisions made with the user: seeds go to a dedicated proxy-seeds
storage where the token can delete them; proxies land on a VLAN set in
config (vlanTag); the API URL is https://192.168.0.8:8006, the address in
the node certificate.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
The host is 9.2.20, not 8.x. Three corrections that would have failed
at runtime:
- VM.Monitor was removed in PVE 9.0, so it is dropped from the role and
the agent read rests on VM.GuestAgent.Audit alone.
- The GC-assist tag becomes proxy_operator: current tag validation takes
only lowercase letters, digits and underscore, so proxy-operator would
have been rejected on every Create. The description mark keeps its
hyphenated keys; it is free text, not a tag.
- HTTP/2 is documented as absent from pveproxy on every release, so the
client disabling it is a fact rather than a guess.
Two new Phase 0 facts: F22 for the 9.2 unified authentication-failure
message, which invalidates any 401 text remembered from 8.x, and F23 for
the node /tmp tmpfs move. F12 records that the upload endpoint takes
iso, vztmpl or import on 9.x and still refuses snippets, which is what
keeps the seed ISO route the only API-only path for user-data.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Eleven self-contained phases (host recon and fixtures, spec, config and
skeleton, nocloud/keyedlock extraction from kvm2, HTTP client and fake,
Get/List, Create, Delete/Checker, test sweep and integration test,
deployment and docs, review and MR), with the Proxmox host setup
checklist and the backend facts Phase 0 must verify on 192.168.2.8.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Four unrelated working-tree changes, none of them touching operator
behaviour:
- review/*.md -> docs/reviews/2026-09-17-kvm2-provider-review/, byte for
byte, so the kvm2 review sits with the rest of the docs under a name
that says which review it was.
- AGENTS.md: replaced the generic kubebuilder scaffold guide with what is
actually true of this repo — the real component layout, the lint/test
commands that work here, and the behavioural gotchas (replace-on-change
hashing, the cloud-init Secret label, in-memory leases).
- .claude/settings.json: the allow list still carried /Users/jan.novak
paths from the macOS checkout; re-pointed at /home/novakj and collapsed
the one-off entries into prefixes.
- config/samples/proxy_kvm2_crawl_only_one.yaml: a one-VM cut of the crawl
pool, the rest of the members left commented out.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
A place for helpers against a running operator, as opposed to the design
docs. First entry is the jq filter for timestamp + rpc: both libvirt
providers log the same "libvirt call" msg, so one select covers kvm and
kvm2, and a fromjson? variant survives a stream with non-JSON lines in it.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Closes out the review-fix campaign. The kvm2 provider was deployed
in-cluster and validated against beelink -- the first time any of this
code had run against real libvirt rather than the in-memory fake.
config/manager/providers_config.yaml ships the kvm2-home entry enabled,
since the operator is now running it for real.
review/ carries the code review the four H-commits reference, so H1..H4
resolve to something rather than dangling: scope, the seven-chunk sweep,
the escalations, the ten deep dives and the final report. .claude/agents
carries the two reviewer definitions that produced them.
What the validation did not prove is recorded next to what it did: H2's
error paths need a body that fails part way through, and a healthy 433 MB
download never finishes short, so the unverified libvirtd short-stream
behaviour is still open. The kvm2 integration test has also still never
been run.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Some lines ignored the zap encoder entirely, so `kubectl logs | jq` could
never work:
I0917 19:50:02.580097 1 leaderelection.go:258] "Attempting to acquire
leader lease..." lock="egress-proxies-operator-system/b47711d1..."
That is klog's own stderr writer, not ours. client-go's leader elector
calls klog.FromContext(ctx), and controller-runtime hands the elector a
context with no logr in it -- mgr.Start receives ctrl.SetupSignalHandler(),
which carries no logger. FromContext falls back to Background(), and with
no logger registered that is klog's text sink. Nothing here imported or
configured klog; it was an indirect dependency that around 60 linked
packages use, including client-go/tools/{leaderelection,cache,record} and
client-go/transport. controller-runtime does not route it either -- its
only klog.SetLogger call is in its own test suite.
Plain klog.SetLogger, deliberately not SetLoggerWithOptions with
ContextualLogger(true). The contextual form is what klog's docs recommend
because it skips a klogr round trip, but it also disables klog's verbosity
gate and hands the decision to zap, whose level under Development is Debug
-- so client-go's V(1) chatter, suppressed today by klog's own -v=0, would
have started appearing. Plain SetLogger keeps klog's check in front and
still redirects what it does emit. Same fix to the format, no new volume,
which matters because volume is what prompted this.
The discovery API's http.Server set no ErrorLog, so net/http's own faults
(a failed Accept, a panic escaping a handler) went to the stdlib logger on
stderr. It now drains into logr through logr.ToSlogHandler, the pattern
internal/provider/gcp/wirelog.go already uses for the same purpose.
The deployment gets --zap-encoder=json and --zap-time-encoding=rfc3339.
The flag beats the in-code Development default -- addDefaults fills
NewEncoder only when it is nil -- and the time encoding is not optional,
since the JSON encoder defaults to epoch seconds. Verbosity stays at debug
on purpose; --zap-log-level=info drops the per-RPC lines.
zap.Options{Development: true} stays as the in-code default: flipping it
would change the format for every deployment of this operator, and the
flags get the same result here without that reach.
k8s.io/klog/v2 moves from indirect to direct in go.mod. It was already in
the module graph, so nothing new is downloaded.
No tests: three changes, all wiring, in a file with no test coverage, and
pinning klog.SetLogger means mutating process-global state that leaks into
every other test in the package. Verified in-cluster instead --
`kubectl logs | jq -e -s 'length > 0'` now passes, which fails if any
single line is not JSON.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Found while watching the Step 5 validation, not a review finding: the
leader election lines ignore the zap encoder because they are not ours.
client-go calls klog.FromContext, controller-runtime hands the elector a
context with no logr in it, and klog falls back to its own stderr writer.
Nothing in the repo configures klog; it is an indirect dependency that
~60 linked packages use.
Plain klog.SetLogger rather than the contextual form the klog docs
recommend: the contextual option disables klog verbosity checking and
hands the decision to zap, which at Development level would let
client-go V(1) chatter through that klog suppresses today. Same fix to
the format, no new volume.
Also the one other in-process stderr bypass (the discovery http.Server
has no ErrorLog) and json/rfc3339 encoder flags on the deployment.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
First contact between the kvm2 provider and a real hypervisor: it has
never run against libvirt, so H1-H4 landed on code no host has executed.
The run happens in the existing kind cluster on purpose. That cluster
already owns two live kvm-home VMs on beelink, and orphan GC deletes any
instance whose UID is missing from its own cluster live set after
gc-min-age. A second operator carrying the kvm-home entry would therefore
destroy both within ten minutes; using the existing cluster makes its
live set the thing that protects them.
The plan also records what the run cannot prove: H2 error paths need a
body that fails part way through, and a healthy download never finishes
short, so the unverified libvirtd short-stream behaviour stays open.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
baseVolumeName was base-<key>.<format>, byte-identical to what the kvm
provider builds in the same pool -- for URL images too, since kvm takes
hex(sum[:8]) and kvm2 takes hex(sum[:])[:16], the same eight bytes -- with
a completely separate lock table. The shipped config has a live kvm-home
entry and a commented-out kvm2-home entry on the same host, the same key
debian-13 and the same default pool, and the comment invites the operator
to enable it. Both integration tests use key test-image. On beelink both
live overlays back onto kvm's base-debian-13.qcow2.
Since cachedImage accepts any 64-hex marker when no sha256 is configured
and never re-hashes the base, that meant two entries pointing one key at
different URLs silently shared a file; a concurrent cold fetch let one
entry delete the other's half-uploaded base and boot a guest from the
remains; and a sha256 edit deleted the base in place under live overlays,
which libvirt permits without complaint.
The name now carries the provider entry: base-<entry>-<key>.<format>.
imageSource gained an Entry field and resolveImage takes the provider
name, so every production call site follows from imageSource.volumeName().
That closes the cross-entry sharing outright, kvm<->kvm2 and kvm2<->kvm2
alike, and turns the per-Provider imageLocks from a latent bug into a
correct one: with names unique per entry, two entries cannot contend for a
file, so no process-wide lock table is needed.
The entry name now reaches a file name, so New checks it against the
existing kvmIdentifier rule, exported as provider.IsKVMIdentifier rather
than duplicated. Provider names are otherwise validated for all types at
once and only for being non-empty and unique, so a name with a slash would
have produced a volume nothing could address.
One companion fix, because the naming alone does not stop it:
cachedImage's digest-mismatch branch deleted the marker and the base and
refetched. It now returns ErrPermanent naming both digests and telling the
operator what to do. That branch was the remaining path that unlinks a
good file under running domains, and the contract already said "add a new
key instead" -- the code was contradicting it. The no-marker and
non-hex-marker branches keep deleting, since there the base is debris.
Not fixed, and said so in the docs rather than left to look closed:
editing images.<key>.url is still a silent no-op, since nothing records
the URL; and cross-process fencing still needs the deferred atomic publish
(upload to a temp name, verify, publish), which lands where H2 put the
discard.
Migration is a non-event. The base-debian-13.qcow2 on beelink is kvm's,
kvm keeps using it untouched, and kvm2 downloads its own copy once per
host. Nothing is replaced under a running guest; the pool needs room for
both, which the prerequisites now say.
The eviction recipe in the guide named base-debian-13.qcow2, which after
this change is kvm's file -- following it would have deleted the wrong
provider's base. It now greps for the prefix and deletes the marker first.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
address() tried the guest agent then the host's ARP table and skipped any
error that was not a context error, returning ("", nil) after both. An
active domain with no IP maps to StateProvisioning; the controller clears
status.IP on that, and the health engine prunes the proxy's counters and
calls ForgetProxy in one tick once EffectiveHost() goes empty.
The damaging trigger was not the guest. The libvirt connection is pooled
per host, so when another reconcile's cancel or budget overrun killed it,
call returned "connection is closed" -- carrying no context sentinel, so
the check did not recognise it -- and Get returned a *successful*
instance in Provisioning with an empty IP. An unrelated proxy's timeout
dropped a healthy proxy out of rotation.
The discriminator was already in the spec without the conclusion being
drawn: during a normal boot on beelink exactly one source fails. The
agent errors because it is not up yet, while ARP answers without error
and with nothing, since the host never sees the guest's traffic. So one
source failing stays "no address yet"; every source failing, or a
connection already dead, is ErrTransient.
The explicit provider.ErrTransient wrap is load-bearing, not decoration.
Without it classify picks the first libvirt code out of the joined error,
and ErrInvalidArg, ErrAccessDenied and ErrNoSupport are Permanent -- an
address read failing would latch a running proxy as Failed, a worse
version of the harm being fixed.
Half of the finding is deliberately still open, and it is the half that
fires more often: a guest agent restarting inside a running VM still
demotes it, because that is indistinguishable from a VM still booting.
The discriminator is the previous IP, which the provider does not hold by
design and the controller already has. Follow-up finding to file against
proxy_controller.go:208-211 -- on StateProvisioning, keep an existing IP
and Provisioned condition for a few polls instead of clearing on the
first. The kvm2-local alternative (use the mark's createdAt as a
provisioning window) was rejected: createdAt is creation, not boot, and
it invents a deadline that belongs to the controller.
TestGet_addressErrorIsProvisioning failed against the new rule, correctly
-- it fails both sources with ErrAgentUnresponsive, and code 86 is
agent-specific, so the ARP source cannot produce it. Re-pointed as
TestGet_agentFailsAndARPEmptyIsProvisioning, the beelink boot it was
reaching for, which nothing covered before.
TestGet_everyAddressSourceFailingIsTransient covers the other side, both
sources erroring and the pooled link dying under the read; against the
unfixed code both return a Provisioning instance instead of an error.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
uploadImage streamed the HTTP body straight into StorageVolUpload. When
that body failed part way through, go-libvirt deadlocked: requestStream
starts its stream sender on two unbuffered channels and, on the error
path, returns without ever reading outErr, so SendStream emits a
StatusError packet and parks on `outErr <-` forever while the main
goroutine parks on `abort <- true` forever. StorageVolUpload never
returned. The 20-minute transfer budget rescued the caller only, and the
socket kill did nothing, because both ends are blocked on channel sends
rather than on I/O -- which is what the comment at backend.go:152 got
wrong. Each failed upload leaked kvm2's call goroutine, go-libvirt's
sender goroutine and a 4 MiB buffer, permanently, against a 128Mi pod
limit and a reconciler that retries on backoff.
The defect is upstream and stays there: no fork, no replace. Instead
eofReader, at the go-libvirt seam so it covers the seed and marker
uploads too, converts any source error to io.EOF -- the one outcome that
ends a stream the normal way -- and keeps the real error for the caller.
countingReader sits underneath untouched, so the existing length and
digest checks report exactly what they did before.
The download also gets its own deadline a minute inside the transfer
budget. fetchImage already bounded the whole fetch at 20 minutes, so the
body's death and the budget's socket kill were racing and the socket
usually won; with the body a minute ahead, a stalled mirror ends through
its reader and unwinds cleanly instead of leaking.
discardVolume moves from uploadImage into fetchImage's upload-error
branch, so the dedicated connection is closed and libvirtd has let go of
the stream's fd before the file is unlinked. Moving it rather than adding
a second Close keeps image_test's closeCount() == 1 true. That discard is
still unfenced against a second provider entry sharing the volume name;
H4 closes that by making the name per-entry.
What remains: a failure originating in libvirtd mid-upload still leaks,
since SendStream has already returned by then. Confirmed unfixable from
here. Known gaps quotes the two-character upstream fix
(make(chan bool, 1) / make(chan error, 1)) and the replace escape hatch.
Unverified at ~85%: that libvirtd answers a short-but-clean stream finish
with OK rather than an error. virFDStreamWrite enforces length only on
writes past it and nothing compares offset to length on the close path,
which is also why virsh vol-upload of a small file into a large volume
works. If it errors we are back to a deadlock by a different door, which
is exactly today's behaviour, so this cannot be worse. The execution file
carries a probe recipe to settle it.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Extends the campaign plan with the three remaining high findings, each its
own step and commit, in the order H2 -> H3 -> H4 (H2 and H4 both touch
image.go, and H2 restructures the failure path H4 would otherwise be
written against).
H2 is a workaround, not a repair: the deadlock is upstream in go-libvirt,
whose requestStream never drains an unbuffered outErr on its error path.
Per the decision taken while planning, no fork and no replace directive;
we stop handing it a reader that can error, move the transfer deadline
onto the HTTP source so a stalled mirror unwinds cleanly, and document the
residual with the two-character upstream patch.
H4 names the base image per provider entry, which is what was chosen over
a content fingerprint. That fixes cross-entry sharing but not the silent
no-op on a url edit, so the plan says so rather than implying the finding
is closed.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
sshSocket.Close was errors.Join(s.Conn.Close(), s.client.Close()). Go
evaluates those left to right, so the blocking close ran before the
unblocking one. s.Conn is the forwarded libvirt socket, and closing an SSH
channel writes a close message under ch.writeMu — which go-libvirt already
holds while streaming a base image through that same channel. Against a
host that vanished without RST or FIN, that write blocks until the kernel
stops retransmitting (tcp_retries2, ~15 min), and s.client.Close(), a bare
net.Conn.Close() that takes no lock and unblocks the stalled writer through
netpoll, was never reached. Close the client only; nothing wants the polite
channel-close message, and reordering instead would leave an error to
filter in a memoised closeErr.
link.markDead compounded it by running kill() before close(l.closed), both
inside a sync.Once. Reordering alone was not enough: sync.Once.Do blocks
concurrent callers until the first body returns, so every other markDead —
including the one in call is own ctx-cancel branch — queued behind the
stalled kill, and cancelling the reconcile did not help. The once field is
gone; the mutex the struct already had guards the close-exactly-once
invariant, reason is set before the signal, and kill runs outside the lock.
Blast radius was the whole operator, not one host: MaxConcurrentReconciles
is 3 and ensureImage holds a per-image lock whose acquire is ctx-aware but
reconcile contexts carry no deadline, so three Proxies on one host with an
uncached image parked every worker.
The same edits close a second route to the same pin: go-libvirt is
Disconnect closes the net.Conn it was handed (libvirt.go:331 ->
socket.go:164), and that conn is our sshSocket, so libvirtBackend.Close is
Disconnect goroutine entered closeOnce and hung; the 5 s closeBudget
released Close, which called markDead -> kill -> sshSocket.Close and
blocked in the other goroutine is sync.Once.
Verified against the pinned x/crypto v0.54.0 rather than assumed: the
flow-control park (remoteWin.reserve, channel.go:268) is outside writeMu
and only writePacket holds it across the TCP write, so a writer parked in
SSH flow control never blocks the close and a writer inside the TCP write
always does. To reproduce by hand, shrink SO_SNDBUF well under the 2 MiB
channel window (channelWindowSize = 64 * channelMaxPacket) so the socket
buffer fills before the window does.
The kvm provider does not have this bug: it closes client only and never
touches the forwarded channel.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The review of feat/kvm2-provider confirmed 52 findings (4 high, 20 medium,
28 low); review/REPORT.md has them. This is the campaign plan: one step and
one commit per finding, highest severity first, with only the step in hand
detailed so the plan is not a fiction about work not yet designed. Same
shape as the kvm campaign in 2026-09-10-2222-kvm-review-fixes.md.
Step 1 (H1) is detailed: sshSocket.Close runs the blocking close before the
unblocking one, so a transfer stuck in a TCP write pins the teardown for as
long as the kernel retransmits, and markDead compounds it by killing inside
a sync.Once before signalling dead.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Shorter start wait with immediate verdicts for dead or paused domains,
ListByTag without state or address RPCs, Timeout for an expired check
round, no adoption of unmarked domains, reference-counted keyed locks,
dead options removed, HEAD fallback on any non-2xx.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Every go-libvirt call runs under its own budget and a cancelled or overlong
call kills the connection; credentials are read on every dial; one span and
one log line per RPC come from a decorator applied where connections are
made.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
The kvm2 block reuses KVMConfig; validateKVM gains a block-name variant so
messages say kvm2. The package skeleton has a non-dialing New, strict
providerID parsing, the state table and a 34-row classify test.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Profile table keyed on the contract's axis codes, per-provider reference
sections for kubernetes, gcp and kvm, shared gaps and proposed features.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Twenty-four fill-in entries, one per decision the contract leaves to the
provider, numbered to match the contract's profile table.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Fully abstract: MUST/SHOULD/MAY rules with the caller behind each one,
19 behaviour-profile axes with stable codes, credential hygiene, testing
obligations, the checklist and the contract's known limits.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
A pass over the API types, spec hash, health and discovery call sites,
manager manifests, samples, README, provider docs, plans and the PR review
found rules the blueprint did not state: the IP must be reachable on the
proxy port by the health engine and clients, providers must be safe for
concurrent use, the manager's read-only/128Mi runtime envelope, catalogue
edits bypass the spec hash, Delete is re-called while in progress, and
provider names must be DNS-1123. Adds matrix rows, a deployment checklist
item, and the still-open review findings and broken samples as gaps.
README's "Adding a provider" restated an outdated contract; it now points
to docs/providers.md, and the providers table gains the kvm row.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The blueprint reduced tracing to "don't open per-operation spans", which
reads as "nothing to do". Add an observability section: the operation
spans the decorators give for free, the child spans per backend call a
provider should add (parent-gated transport or a span decorator over the
SDK seam), the ctx/logger rules that keep traceID on log lines, and the
status per provider: kubernetes and gcp have child spans, kvm has none.
Matrix rows, the checklist item, gaps and a kvm proposal follow suit.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
A re-check against the code found four mismatches: the kvm connection is
opened at startup by the leader's first health check, so that isn't doc
drift; cmd/main.go, not the monitor, type-asserts Checker; QuotaExceeded
and PermanentError are reasons on Provisioned, not conditions; and the
latch line range. The checklist also now matches how the existing
providers do interface assertions and RBAC markers.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
docs/providers.md collects the provider contract that was spread across
comments, callers and per-provider docs: MUST/SHOULD/MAY rules per method,
each tied to the reconciler or GC behaviour it protects, the error taxonomy,
a feature matrix for kubernetes/gcp/kvm with file:line notes, a checklist for
adding a provider, and the known gaps. Linked from architecture.md and
CLAUDE.md.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The user ran the real-hypervisor checks against beelink and confirmed they
passed, which completes Step 7 of the provider health plan.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Leaving KVM_TEST_KNOWN_HOSTS unset silently disables host-key
verification for the run, so document it as part of the normal setup
rather than as an optional extra, and suggest a kvm_known_hosts holding
only the hypervisors.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The health check shares each host's libvirt connection with the reconciles,
so it must not be able to take that connection away from them.
- A timed-out check no longer drops the link. The check goes through the new
hostPool.probe, which abandons its RPCs when the context ends and leaves
the connection alone, so an image upload or a Create running on the same
link is not cut off. A link that really died is closed by the keepalive.
One abandoned probe per host may stay waiting; until it returns, the next
check reports Timeout at once (errProbeBusy).
- Waiting for a host's connection slot now ends with the caller's context.
hostEntry used a mutex, so a check could wait out a reconcile redialing a
dead host — minutes — past its own timeout. The slot is a one-slot channel
and acquire bounds only the wait, never the uncontended take.
- --provider-check-timeout defaults to 0 and derives min(15s, interval).
The flat 15s default contradicted the validation that the timeout may not
exceed the interval, so --provider-check-interval=12s alone failed to
start.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
kacerrandClaude Opus 5.5