address() tried the guest agent then the host's ARP table and skipped any
error that was not a context error, returning ("", nil) after both. An
active domain with no IP maps to StateProvisioning; the controller clears
status.IP on that, and the health engine prunes the proxy's counters and
calls ForgetProxy in one tick once EffectiveHost() goes empty.
The damaging trigger was not the guest. The libvirt connection is pooled
per host, so when another reconcile's cancel or budget overrun killed it,
call returned "connection is closed" -- carrying no context sentinel, so
the check did not recognise it -- and Get returned a *successful*
instance in Provisioning with an empty IP. An unrelated proxy's timeout
dropped a healthy proxy out of rotation.
The discriminator was already in the spec without the conclusion being
drawn: during a normal boot on beelink exactly one source fails. The
agent errors because it is not up yet, while ARP answers without error
and with nothing, since the host never sees the guest's traffic. So one
source failing stays "no address yet"; every source failing, or a
connection already dead, is ErrTransient.
The explicit provider.ErrTransient wrap is load-bearing, not decoration.
Without it classify picks the first libvirt code out of the joined error,
and ErrInvalidArg, ErrAccessDenied and ErrNoSupport are Permanent -- an
address read failing would latch a running proxy as Failed, a worse
version of the harm being fixed.
Half of the finding is deliberately still open, and it is the half that
fires more often: a guest agent restarting inside a running VM still
demotes it, because that is indistinguishable from a VM still booting.
The discriminator is the previous IP, which the provider does not hold by
design and the controller already has. Follow-up finding to file against
proxy_controller.go:208-211 -- on StateProvisioning, keep an existing IP
and Provisioned condition for a few polls instead of clearing on the
first. The kvm2-local alternative (use the mark's createdAt as a
provisioning window) was rejected: createdAt is creation, not boot, and
it invents a deadline that belongs to the controller.
TestGet_addressErrorIsProvisioning failed against the new rule, correctly
-- it fails both sources with ErrAgentUnresponsive, and code 86 is
agent-specific, so the ARP source cannot produce it. Re-pointed as
TestGet_agentFailsAndARPEmptyIsProvisioning, the beelink boot it was
reaching for, which nothing covered before.
TestGet_everyAddressSourceFailingIsTransient covers the other side, both
sources erroring and the pooled link dying under the read; against the
unfixed code both return a Provisioning instance instead of an error.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
uploadImage streamed the HTTP body straight into StorageVolUpload. When
that body failed part way through, go-libvirt deadlocked: requestStream
starts its stream sender on two unbuffered channels and, on the error
path, returns without ever reading outErr, so SendStream emits a
StatusError packet and parks on `outErr <-` forever while the main
goroutine parks on `abort <- true` forever. StorageVolUpload never
returned. The 20-minute transfer budget rescued the caller only, and the
socket kill did nothing, because both ends are blocked on channel sends
rather than on I/O -- which is what the comment at backend.go:152 got
wrong. Each failed upload leaked kvm2's call goroutine, go-libvirt's
sender goroutine and a 4 MiB buffer, permanently, against a 128Mi pod
limit and a reconciler that retries on backoff.
The defect is upstream and stays there: no fork, no replace. Instead
eofReader, at the go-libvirt seam so it covers the seed and marker
uploads too, converts any source error to io.EOF -- the one outcome that
ends a stream the normal way -- and keeps the real error for the caller.
countingReader sits underneath untouched, so the existing length and
digest checks report exactly what they did before.
The download also gets its own deadline a minute inside the transfer
budget. fetchImage already bounded the whole fetch at 20 minutes, so the
body's death and the budget's socket kill were racing and the socket
usually won; with the body a minute ahead, a stalled mirror ends through
its reader and unwinds cleanly instead of leaking.
discardVolume moves from uploadImage into fetchImage's upload-error
branch, so the dedicated connection is closed and libvirtd has let go of
the stream's fd before the file is unlinked. Moving it rather than adding
a second Close keeps image_test's closeCount() == 1 true. That discard is
still unfenced against a second provider entry sharing the volume name;
H4 closes that by making the name per-entry.
What remains: a failure originating in libvirtd mid-upload still leaks,
since SendStream has already returned by then. Confirmed unfixable from
here. Known gaps quotes the two-character upstream fix
(make(chan bool, 1) / make(chan error, 1)) and the replace escape hatch.
Unverified at ~85%: that libvirtd answers a short-but-clean stream finish
with OK rather than an error. virFDStreamWrite enforces length only on
writes past it and nothing compares offset to length on the close path,
which is also why virsh vol-upload of a small file into a large volume
works. If it errors we are back to a deadlock by a different door, which
is exactly today's behaviour, so this cannot be worse. The execution file
carries a probe recipe to settle it.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Extends the campaign plan with the three remaining high findings, each its
own step and commit, in the order H2 -> H3 -> H4 (H2 and H4 both touch
image.go, and H2 restructures the failure path H4 would otherwise be
written against).
H2 is a workaround, not a repair: the deadlock is upstream in go-libvirt,
whose requestStream never drains an unbuffered outErr on its error path.
Per the decision taken while planning, no fork and no replace directive;
we stop handing it a reader that can error, move the transfer deadline
onto the HTTP source so a stalled mirror unwinds cleanly, and document the
residual with the two-character upstream patch.
H4 names the base image per provider entry, which is what was chosen over
a content fingerprint. That fixes cross-entry sharing but not the silent
no-op on a url edit, so the plan says so rather than implying the finding
is closed.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
sshSocket.Close was errors.Join(s.Conn.Close(), s.client.Close()). Go
evaluates those left to right, so the blocking close ran before the
unblocking one. s.Conn is the forwarded libvirt socket, and closing an SSH
channel writes a close message under ch.writeMu — which go-libvirt already
holds while streaming a base image through that same channel. Against a
host that vanished without RST or FIN, that write blocks until the kernel
stops retransmitting (tcp_retries2, ~15 min), and s.client.Close(), a bare
net.Conn.Close() that takes no lock and unblocks the stalled writer through
netpoll, was never reached. Close the client only; nothing wants the polite
channel-close message, and reordering instead would leave an error to
filter in a memoised closeErr.
link.markDead compounded it by running kill() before close(l.closed), both
inside a sync.Once. Reordering alone was not enough: sync.Once.Do blocks
concurrent callers until the first body returns, so every other markDead —
including the one in call is own ctx-cancel branch — queued behind the
stalled kill, and cancelling the reconcile did not help. The once field is
gone; the mutex the struct already had guards the close-exactly-once
invariant, reason is set before the signal, and kill runs outside the lock.
Blast radius was the whole operator, not one host: MaxConcurrentReconciles
is 3 and ensureImage holds a per-image lock whose acquire is ctx-aware but
reconcile contexts carry no deadline, so three Proxies on one host with an
uncached image parked every worker.
The same edits close a second route to the same pin: go-libvirt is
Disconnect closes the net.Conn it was handed (libvirt.go:331 ->
socket.go:164), and that conn is our sshSocket, so libvirtBackend.Close is
Disconnect goroutine entered closeOnce and hung; the 5 s closeBudget
released Close, which called markDead -> kill -> sshSocket.Close and
blocked in the other goroutine is sync.Once.
Verified against the pinned x/crypto v0.54.0 rather than assumed: the
flow-control park (remoteWin.reserve, channel.go:268) is outside writeMu
and only writePacket holds it across the TCP write, so a writer parked in
SSH flow control never blocks the close and a writer inside the TCP write
always does. To reproduce by hand, shrink SO_SNDBUF well under the 2 MiB
channel window (channelWindowSize = 64 * channelMaxPacket) so the socket
buffer fills before the window does.
The kvm provider does not have this bug: it closes client only and never
touches the forwarded channel.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The review of feat/kvm2-provider confirmed 52 findings (4 high, 20 medium,
28 low); review/REPORT.md has them. This is the campaign plan: one step and
one commit per finding, highest severity first, with only the step in hand
detailed so the plan is not a fiction about work not yet designed. Same
shape as the kvm campaign in 2026-09-10-2222-kvm-review-fixes.md.
Step 1 (H1) is detailed: sshSocket.Close runs the blocking close before the
unblocking one, so a transfer stuck in a TCP write pins the teardown for as
long as the kernel retransmits, and markDead compounds it by killing inside
a sync.Once before signalling dead.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Shorter start wait with immediate verdicts for dead or paused domains,
ListByTag without state or address RPCs, Timeout for an expired check
round, no adoption of unmarked domains, reference-counted keyed locks,
dead options removed, HEAD fallback on any non-2xx.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Every go-libvirt call runs under its own budget and a cancelled or overlong
call kills the connection; credentials are read on every dial; one span and
one log line per RPC come from a decorator applied where connections are
made.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
The kvm2 block reuses KVMConfig; validateKVM gains a block-name variant so
messages say kvm2. The package skeleton has a non-dialing New, strict
providerID parsing, the state table and a 34-row classify test.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Profile table keyed on the contract's axis codes, per-provider reference
sections for kubernetes, gcp and kvm, shared gaps and proposed features.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Twenty-four fill-in entries, one per decision the contract leaves to the
provider, numbered to match the contract's profile table.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Fully abstract: MUST/SHOULD/MAY rules with the caller behind each one,
19 behaviour-profile axes with stable codes, credential hygiene, testing
obligations, the checklist and the contract's known limits.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
A pass over the API types, spec hash, health and discovery call sites,
manager manifests, samples, README, provider docs, plans and the PR review
found rules the blueprint did not state: the IP must be reachable on the
proxy port by the health engine and clients, providers must be safe for
concurrent use, the manager's read-only/128Mi runtime envelope, catalogue
edits bypass the spec hash, Delete is re-called while in progress, and
provider names must be DNS-1123. Adds matrix rows, a deployment checklist
item, and the still-open review findings and broken samples as gaps.
README's "Adding a provider" restated an outdated contract; it now points
to docs/providers.md, and the providers table gains the kvm row.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The blueprint reduced tracing to "don't open per-operation spans", which
reads as "nothing to do". Add an observability section: the operation
spans the decorators give for free, the child spans per backend call a
provider should add (parent-gated transport or a span decorator over the
SDK seam), the ctx/logger rules that keep traceID on log lines, and the
status per provider: kubernetes and gcp have child spans, kvm has none.
Matrix rows, the checklist item, gaps and a kvm proposal follow suit.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
A re-check against the code found four mismatches: the kvm connection is
opened at startup by the leader's first health check, so that isn't doc
drift; cmd/main.go, not the monitor, type-asserts Checker; QuotaExceeded
and PermanentError are reasons on Provisioned, not conditions; and the
latch line range. The checklist also now matches how the existing
providers do interface assertions and RBAC markers.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
docs/providers.md collects the provider contract that was spread across
comments, callers and per-provider docs: MUST/SHOULD/MAY rules per method,
each tied to the reconciler or GC behaviour it protects, the error taxonomy,
a feature matrix for kubernetes/gcp/kvm with file:line notes, a checklist for
adding a provider, and the known gaps. Linked from architecture.md and
CLAUDE.md.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The user ran the real-hypervisor checks against beelink and confirmed they
passed, which completes Step 7 of the provider health plan.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Leaving KVM_TEST_KNOWN_HOSTS unset silently disables host-key
verification for the run, so document it as part of the normal setup
rather than as an optional extra, and suggest a kvm_known_hosts holding
only the hypervisors.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The health check shares each host's libvirt connection with the reconciles,
so it must not be able to take that connection away from them.
- A timed-out check no longer drops the link. The check goes through the new
hostPool.probe, which abandons its RPCs when the context ends and leaves
the connection alone, so an image upload or a Create running on the same
link is not cut off. A link that really died is closed by the keepalive.
One abandoned probe per host may stay waiting; until it returns, the next
check reports Timeout at once (errProbeBusy).
- Waiting for a host's connection slot now ends with the caller's context.
hostEntry used a mutex, so a check could wait out a reconcile redialing a
dead host — minutes — past its own timeout. The slot is a one-slot channel
and acquire bounds only the wait, never the uncontended take.
- --provider-check-timeout defaults to 0 and derives min(15s, interval).
The flat 15s default contradicted the validation that the timeout may not
exceed the interval, so --provider-check-interval=12s alone failed to
start.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
make test (envtest, -race) and the kind e2e suite pass; the e2e
ProviderHealth for the kubernetes provider turned Ready within a second
of the manager starting and its gauge appeared in the scrape. make lint
is blocked by the local go1.27 toolchain crashing staticcheck on the
standard library, unrelated to this branch.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Architecture gets a provider-health section, component rows and four
decisions; the KVM provider doc gets a health-check section with the
reason table and the rotated-Secret gap; the testing guide lists the new
suites. CLAUDE.md's Architecture TODO and stale "only go.mod exists"
gotcha now point at the real docs, plus the RBAC-marker gotcha found
while wiring the monitor.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
--provider-check-interval (1m; 0 = off, else at least 10s) and
--provider-check-timeout (15s) are validated before the providers config
loads. Checkers are collected from the unwrapped providers in config
order, so checks never count as provider requests. Three scrape-time
gauges per {provider,target} read the monitor's snapshot: up, check
duration and check timestamp. The e2e suite waits for the kubernetes
ProviderHealth to turn Ready and asserts its gauge in the scrape.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
A leader-only runnable checks every configured provider concurrently
under one timeout per round (first round at start, then every interval),
keeps the latest results for the metrics collector, logs targets going
down, changing reason or coming back, and publishes one ProviderHealth
object per provider: created with a managed-by label when missing, then
a status merge patch. Per-target lastTransitionTime survives rounds in
which ready does not flip. Objects of providers that are no longer
configured are deleted once at start; with checks turned off, all are.
Reads go through the API reader, so no cluster-wide informer is needed.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Each configured host is checked concurrently over the pooled SSH+libvirt
connection: libvirt version and hostname prove the daemon answers, then
the configured pool must exist and be running, with its free space
reported. Dial failures are now tagged with their stage (SSH material,
TCP, SSH handshake, libvirt connect) by a dialError that keeps the
message and unwraps to the cause, so classification is unchanged and the
check can say why a host is unreachable. A pool verdict is decided after
the call returns so it never drops a working connection.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Checker is an optional interface next to Provider: Check returns one
verdict per target and never fails as a whole. The kubernetes provider
lists at most one managed Pod cluster-wide (the verb and scope GC relies
on); the GCP provider fetches a single page of at most one managed
instance through a new Probe seam method, since the SDK treats
MaxResults as a page size and the existing AggregatedList drains every
page.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
One operator-owned object per configured provider will carry the
periodic check result: a Ready condition, per-target verdicts with their
own transition times, and a ready/total count for kubectl get. There is
no spec; only a viewer role is shipped since the operator is the sole
writer. Written by hand (kubebuilder is not installed), registered in
the existing AddKnownTypes call rather than a second init().
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Periodic, leader-only checks of every configured provider (KVM per host
over the pooled SSH+libvirt connection including the storage pool,
kubernetes apiserver/RBAC, GCP credentials/API), surfaced through a new
cluster-scoped ProviderHealth CRD, Prometheus gauges and transition logs.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The default manager config gains a kvm-home provider reaching beelink
over Tailscale, the provider sample names the same host, and the
manager kustomization uses the locally built kvm-dev image.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Collapse duplicated logic the review found in PR #5 without changing
behaviour: one releaseFinalizer helper for the four finalizer exits in
reconcileDelete, classify reusing isClassified and provider.Class, the
image URL check and format/firmware values shared from the provider
package, imageSource and the metadata parser embedding their config
structs instead of copying fields, and the SSH loader as a small struct
that keeps only the SSH fields and is seeded by New instead of loading
the key twice. Tracing builds its V(2)-only lists only at V(2).
Tests: drop a duplicate New test, fold near-identical tests into tables
(connection drops, metadata parsing, cached images), add mustSSHAuth,
and use wantClass where classes were checked by hand.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Merging the bootstrap into each Proxy's cloud-config was the wrong
mechanism twice over. Detection only matched a literal #cloud-config
first line, so a jinja template or a MIME archive still fell back to
vendor-data, where cloud-init's default mergers drop it — the exact
failure the merge existed to prevent. And the YAML round-trip re-typed
unquoted scalars: version: 1.10 reached the guest as 1.1, comments were
dropped and keys reordered.
Neither is worth patching, because vendor-data delivered as a shell
script bypasses the merger entirely: cloud-init hands vendor-data to its
own handler chain, writes scripts to a separate directory and runs them
from cc_scripts_vendor, after the user's packages and before the user's
runcmd. The merge could not have been rescued anyway — merge_how in
user-data governs only the first source, and merge_how in vendor-data is
discarded before the document is stored.
The Proxy's cloud-init now reaches the seed byte for byte in any format.
installGuestAgent (default true) turns the script off; a configured
vendorData travels beside it as a MIME multipart, declared text/plain so
cloud-init re-sniffs it and this code never classifies user content.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The kvm-ssh Secret is mounted optional, but kvm.New read the key and
known_hosts eagerly and registry.Build aborts the whole provider set on a
constructor error, so a missing or misnamed Secret exited the manager
before any controller started and took the kubernetes and gcp providers
down with it.
lazyAuth loads the material on the first connection and caches it once it
works, so a Secret created after the manager starts heals the affected
proxies on the next reconcile with no restart. A missing file classifies
transient for that reason; a key that does not parse is permanent.
New still checks both files and logs a WARNING: line when they are
unreadable, keeping the startup diagnostic the eager load was there for.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
A host name is pasted into the providerID hosts/<name>/domains/<vm>, which
parseProviderID splits on "/", so a name containing a slash produced an ID
that never parsed again: Get, Delete and GC all failed permanently and the
VM leaked with the Proxy's finalizer stuck.
The image map key has the same defect and lands in a file name rather than
an ID: base-<key>.<format> in the host's storage pool. Both now have to
match a plain identifier pattern. Machine-type keys are left alone; they
are only ever compared, never embedded.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
reconcileDelete asked Get first and dropped the finalizer only on
not-found, so a permanent Get retried forever: a KVM host removed from
the config, or a providerID the provider cannot resolve, left the Proxy
in Terminating with no way out. It now tries Delete anyway, which is
idempotent and can still finish where Get cannot converge, and releases
the object when that succeeds.
crawl.example.com/skip-cleanup abandons the instance for the cases that
stay stuck, such as revoked credentials. It is checked before the
provider lookup, so it also frees a Proxy whose provider is gone from
the config.
In the KVM provider, Delete over an unconfigured host is now a no-op
success, mirroring removeVolumes over a missing pool. Get keeps failing
permanently there on purpose: the live path reads not-found as "rebuild
it", so a typo in hosts must not look like a missing VM.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
kacerrandClaude Opus 5