Phase 10. Two review passes (go-operator-reviewer, /code-review high)
over the branch, recorded in docs/reviews/2026-09-23-proxmox-provider-review/:
9 findings fixed, 3 accepted as documented limits, 7 rejected with a
reason.
- Delete removes the seed between the stop and the destroy, so a failed
seed delete leaves the VM for the next Delete instead of leaking the
ISO; a Permanent refusal on the every-node path fails the call.
- Create skips a probe VMID whose listed name is someone else's without
waiting on its lock.
- The three answers Phase 0 never captured are fixtures now: a missing
storage is a 400 (the Checker reports StorageNotFound), a hibernated
VM is destroyable (Delete destroys ours), a paused VM stops.
- TestProxmox_lifecycle and TestProxmoxCheck_realHost passed on pv1-1
(PVE 9.2.20); the docs record the run and drop "not yet run".
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
The optional proxmox-token Secret (token, ca.pem) mounted at
/etc/proxy-operator/proxmox, make proxmox-token-secret with content
checks that never print the token, a commented proxmox-home entry in
both providers configs, and config/samples/proxy_proxmox.yaml.
TestLoadConfig_proxmoxSampleEntry loads the commented entry from both
files, and a second test ties the sample Proxy's placement to it.
docs/proxmox-provider.md is the user guide: prerequisites and privileges,
cluster setup, config reference, placement, what lands on Proxmox,
logging, tracing, timeouts, health check, error classes, known gaps and
testing. providers-existing.md gains the proxmox column and section;
architecture.md, the README, providers.md and CLAUDE.md gain their rows.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
The contract sweep in provider_test.go: every method and Check returns
within a second of a cancel, every wrapErr path carries the configured
name, op and id, the V0/V1/V2 logging tiers and the no-op lines, and a
Get over the real client whose HTTP spans are children of provider.get.
proxmox_integration_test.go (-tags=integration) holds TestProxmox_lifecycle
and TestProxmoxCheck_realHost; both compile and skip without
PROXMOX_TEST_*, and have not run against a host yet. make
test-integration-proxmox runs them. docs/testing.md gains the proxmox,
nocloud and keyedlock rows, the Live Proxmox gaps and the integration
section; spec entry 21's section 11 list is ticked.
Waits that could hang the package under a mutation are bounded: the
concurrent-Create loop and eight direct blocking calls in older tests.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Delete stops and destroys a VM of ours under the per-name lock, then
deletes its seed on the VM's node, or on every node when no VM of ours
was left. A hibernated VM of ours is Permanent, as in Create. The
Checker verifies each node concurrently (version, node status, ISO
storage, templates) and adds CertificateInvalid, CredentialsUnavailable,
StorageNotFound and TemplateNotFound to the shared reasons. The
ProviderHealth type comment gains proxmox.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Create validates placement, then under the per-name lock adopts the VM a
previous attempt left (by name, or at a taken VMID after the ownership
check of spec entry 2) or full-clones the template onto the UID's
deterministic VMID with the mark in the clone call. It then uploads the
seed over any existing file, writes cores, memory, agent, tag, ide2 and
net0 when configured, grows the boot disk when diskGiB is larger, starts,
and waits for running/running.
waitUnlocked no longer waits on a suspended lock: a hibernated VM is
judged at once, so ours is Permanent and a foreign one is probed past.
The template is checked only on the allocation path, and its shape
(net0, boot disk size) is refused before any clone exists. Spec entries
2, 4 and 10 updated to match.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Get follows spec entry 9's check order (lock and unknown status before
the name, suspended is Stopped after the name and mark checks), reads
status/current so paused is Stopped, and maps the three agent messages
of entry 7. ListByTag reads one config per candidate and fails whole
only on the resources call or an ended ctx. waitTask polls the UPID's
node every budgets.poll, logs failed tasks at V(1) and counts
"WARNINGS: <n>" as success; waitUnlocked serves Create's ownership check.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
httpAPI implements the sixteen seam methods over net/http: the token is
read per request, non-2xx answers become apiError from the body message or
the reason phrase, echoed secrets are redacted, the CA loads lazily, HTTP/2
is off, and uploads stream with an exact Content-Length. The observed
decorator logs one V(1) line per call and starts no span; New wires both.
fakeAPI simulates clone locks, tasks and ISOs with the captured error texts.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
kvm2's NoCloud seed builder moves to internal/provider/nocloud (exported,
with an ErrEmptyName sentinel instead of permanentf) and its keyed locks
to internal/provider/keyedlock, tests moved under their existing names.
kvm2 behaviour is unchanged. proxmox gains the per-name lock and the pure
seed helpers (seedISOName, seedVolID, buildSeed); unlike kvm2 (review
finding S5.1), a seed staging failure there is Transient, not Permanent.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Phase 2 of the proxmox provider plan. Adds the proxmox: config block with
its validation, registers the type in the config allow-list and the
constructor map, and starts internal/provider/proxmox: the api seam and
its data types, providerIDs (vms/<vmid>/<name>) with the UID-derived VMID,
the description mark, state mapping, and error classification.
Tests read the responses captured from pv1-1 in Phase 0 rather than
repeating Proxmox's wording, and every matcher is run against every
captured error so none fires on the wrong message. The contract methods
are Transient stubs until Phases 5 to 7.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Phase 0 of the proxmox provider plan. 55 fixtures under
internal/provider/proxmox/testdata, each the status, reason phrase and
raw body of a real response from the user's host, and the execution log
answering the plan's facts F1 to F23.
Recon ran a clone of the Debian 13 template through the whole path the
provider will take, twice: untagged and on VLAN 4. On both, the NoCloud
seed on ide2 installed the guest agent and an address came back about 50
seconds after start. It also found two bugs in the plan: a VM being
cloned has no name in /cluster/resources, so name-first lookups would read
it as gone, and a crash-retry would clone a duplicate. Those and eight
smaller corrections are listed for the spec.
Decisions made with the user: seeds go to a dedicated proxy-seeds
storage where the token can delete them; proxies land on a VLAN set in
config (vlanTag); the API URL is https://192.168.0.8:8006, the address in
the node certificate.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Making baseVolumeName multi-line in the H4 commit collapsed the
alignment gofmt gives a run of single-line funcs, and the commit went out
with the old spacing. Caught by make test, whose fmt target rewrites the
file and leaves the tree dirty.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Some lines ignored the zap encoder entirely, so `kubectl logs | jq` could
never work:
I0917 19:50:02.580097 1 leaderelection.go:258] "Attempting to acquire
leader lease..." lock="egress-proxies-operator-system/b47711d1..."
That is klog's own stderr writer, not ours. client-go's leader elector
calls klog.FromContext(ctx), and controller-runtime hands the elector a
context with no logr in it -- mgr.Start receives ctrl.SetupSignalHandler(),
which carries no logger. FromContext falls back to Background(), and with
no logger registered that is klog's text sink. Nothing here imported or
configured klog; it was an indirect dependency that around 60 linked
packages use, including client-go/tools/{leaderelection,cache,record} and
client-go/transport. controller-runtime does not route it either -- its
only klog.SetLogger call is in its own test suite.
Plain klog.SetLogger, deliberately not SetLoggerWithOptions with
ContextualLogger(true). The contextual form is what klog's docs recommend
because it skips a klogr round trip, but it also disables klog's verbosity
gate and hands the decision to zap, whose level under Development is Debug
-- so client-go's V(1) chatter, suppressed today by klog's own -v=0, would
have started appearing. Plain SetLogger keeps klog's check in front and
still redirects what it does emit. Same fix to the format, no new volume,
which matters because volume is what prompted this.
The discovery API's http.Server set no ErrorLog, so net/http's own faults
(a failed Accept, a panic escaping a handler) went to the stdlib logger on
stderr. It now drains into logr through logr.ToSlogHandler, the pattern
internal/provider/gcp/wirelog.go already uses for the same purpose.
The deployment gets --zap-encoder=json and --zap-time-encoding=rfc3339.
The flag beats the in-code Development default -- addDefaults fills
NewEncoder only when it is nil -- and the time encoding is not optional,
since the JSON encoder defaults to epoch seconds. Verbosity stays at debug
on purpose; --zap-log-level=info drops the per-RPC lines.
zap.Options{Development: true} stays as the in-code default: flipping it
would change the format for every deployment of this operator, and the
flags get the same result here without that reach.
k8s.io/klog/v2 moves from indirect to direct in go.mod. It was already in
the module graph, so nothing new is downloaded.
No tests: three changes, all wiring, in a file with no test coverage, and
pinning klog.SetLogger means mutating process-global state that leaks into
every other test in the package. Verified in-cluster instead --
`kubectl logs | jq -e -s 'length > 0'` now passes, which fails if any
single line is not JSON.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
baseVolumeName was base-<key>.<format>, byte-identical to what the kvm
provider builds in the same pool -- for URL images too, since kvm takes
hex(sum[:8]) and kvm2 takes hex(sum[:])[:16], the same eight bytes -- with
a completely separate lock table. The shipped config has a live kvm-home
entry and a commented-out kvm2-home entry on the same host, the same key
debian-13 and the same default pool, and the comment invites the operator
to enable it. Both integration tests use key test-image. On beelink both
live overlays back onto kvm's base-debian-13.qcow2.
Since cachedImage accepts any 64-hex marker when no sha256 is configured
and never re-hashes the base, that meant two entries pointing one key at
different URLs silently shared a file; a concurrent cold fetch let one
entry delete the other's half-uploaded base and boot a guest from the
remains; and a sha256 edit deleted the base in place under live overlays,
which libvirt permits without complaint.
The name now carries the provider entry: base-<entry>-<key>.<format>.
imageSource gained an Entry field and resolveImage takes the provider
name, so every production call site follows from imageSource.volumeName().
That closes the cross-entry sharing outright, kvm<->kvm2 and kvm2<->kvm2
alike, and turns the per-Provider imageLocks from a latent bug into a
correct one: with names unique per entry, two entries cannot contend for a
file, so no process-wide lock table is needed.
The entry name now reaches a file name, so New checks it against the
existing kvmIdentifier rule, exported as provider.IsKVMIdentifier rather
than duplicated. Provider names are otherwise validated for all types at
once and only for being non-empty and unique, so a name with a slash would
have produced a volume nothing could address.
One companion fix, because the naming alone does not stop it:
cachedImage's digest-mismatch branch deleted the marker and the base and
refetched. It now returns ErrPermanent naming both digests and telling the
operator what to do. That branch was the remaining path that unlinks a
good file under running domains, and the contract already said "add a new
key instead" -- the code was contradicting it. The no-marker and
non-hex-marker branches keep deleting, since there the base is debris.
Not fixed, and said so in the docs rather than left to look closed:
editing images.<key>.url is still a silent no-op, since nothing records
the URL; and cross-process fencing still needs the deferred atomic publish
(upload to a temp name, verify, publish), which lands where H2 put the
discard.
Migration is a non-event. The base-debian-13.qcow2 on beelink is kvm's,
kvm keeps using it untouched, and kvm2 downloads its own copy once per
host. Nothing is replaced under a running guest; the pool needs room for
both, which the prerequisites now say.
The eviction recipe in the guide named base-debian-13.qcow2, which after
this change is kvm's file -- following it would have deleted the wrong
provider's base. It now greps for the prefix and deletes the marker first.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
address() tried the guest agent then the host's ARP table and skipped any
error that was not a context error, returning ("", nil) after both. An
active domain with no IP maps to StateProvisioning; the controller clears
status.IP on that, and the health engine prunes the proxy's counters and
calls ForgetProxy in one tick once EffectiveHost() goes empty.
The damaging trigger was not the guest. The libvirt connection is pooled
per host, so when another reconcile's cancel or budget overrun killed it,
call returned "connection is closed" -- carrying no context sentinel, so
the check did not recognise it -- and Get returned a *successful*
instance in Provisioning with an empty IP. An unrelated proxy's timeout
dropped a healthy proxy out of rotation.
The discriminator was already in the spec without the conclusion being
drawn: during a normal boot on beelink exactly one source fails. The
agent errors because it is not up yet, while ARP answers without error
and with nothing, since the host never sees the guest's traffic. So one
source failing stays "no address yet"; every source failing, or a
connection already dead, is ErrTransient.
The explicit provider.ErrTransient wrap is load-bearing, not decoration.
Without it classify picks the first libvirt code out of the joined error,
and ErrInvalidArg, ErrAccessDenied and ErrNoSupport are Permanent -- an
address read failing would latch a running proxy as Failed, a worse
version of the harm being fixed.
Half of the finding is deliberately still open, and it is the half that
fires more often: a guest agent restarting inside a running VM still
demotes it, because that is indistinguishable from a VM still booting.
The discriminator is the previous IP, which the provider does not hold by
design and the controller already has. Follow-up finding to file against
proxy_controller.go:208-211 -- on StateProvisioning, keep an existing IP
and Provisioned condition for a few polls instead of clearing on the
first. The kvm2-local alternative (use the mark's createdAt as a
provisioning window) was rejected: createdAt is creation, not boot, and
it invents a deadline that belongs to the controller.
TestGet_addressErrorIsProvisioning failed against the new rule, correctly
-- it fails both sources with ErrAgentUnresponsive, and code 86 is
agent-specific, so the ARP source cannot produce it. Re-pointed as
TestGet_agentFailsAndARPEmptyIsProvisioning, the beelink boot it was
reaching for, which nothing covered before.
TestGet_everyAddressSourceFailingIsTransient covers the other side, both
sources erroring and the pooled link dying under the read; against the
unfixed code both return a Provisioning instance instead of an error.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
uploadImage streamed the HTTP body straight into StorageVolUpload. When
that body failed part way through, go-libvirt deadlocked: requestStream
starts its stream sender on two unbuffered channels and, on the error
path, returns without ever reading outErr, so SendStream emits a
StatusError packet and parks on `outErr <-` forever while the main
goroutine parks on `abort <- true` forever. StorageVolUpload never
returned. The 20-minute transfer budget rescued the caller only, and the
socket kill did nothing, because both ends are blocked on channel sends
rather than on I/O -- which is what the comment at backend.go:152 got
wrong. Each failed upload leaked kvm2's call goroutine, go-libvirt's
sender goroutine and a 4 MiB buffer, permanently, against a 128Mi pod
limit and a reconciler that retries on backoff.
The defect is upstream and stays there: no fork, no replace. Instead
eofReader, at the go-libvirt seam so it covers the seed and marker
uploads too, converts any source error to io.EOF -- the one outcome that
ends a stream the normal way -- and keeps the real error for the caller.
countingReader sits underneath untouched, so the existing length and
digest checks report exactly what they did before.
The download also gets its own deadline a minute inside the transfer
budget. fetchImage already bounded the whole fetch at 20 minutes, so the
body's death and the budget's socket kill were racing and the socket
usually won; with the body a minute ahead, a stalled mirror ends through
its reader and unwinds cleanly instead of leaking.
discardVolume moves from uploadImage into fetchImage's upload-error
branch, so the dedicated connection is closed and libvirtd has let go of
the stream's fd before the file is unlinked. Moving it rather than adding
a second Close keeps image_test's closeCount() == 1 true. That discard is
still unfenced against a second provider entry sharing the volume name;
H4 closes that by making the name per-entry.
What remains: a failure originating in libvirtd mid-upload still leaks,
since SendStream has already returned by then. Confirmed unfixable from
here. Known gaps quotes the two-character upstream fix
(make(chan bool, 1) / make(chan error, 1)) and the replace escape hatch.
Unverified at ~85%: that libvirtd answers a short-but-clean stream finish
with OK rather than an error. virFDStreamWrite enforces length only on
writes past it and nothing compares offset to length on the close path,
which is also why virsh vol-upload of a small file into a large volume
works. If it errors we are back to a deadlock by a different door, which
is exactly today's behaviour, so this cannot be worse. The execution file
carries a probe recipe to settle it.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
sshSocket.Close was errors.Join(s.Conn.Close(), s.client.Close()). Go
evaluates those left to right, so the blocking close ran before the
unblocking one. s.Conn is the forwarded libvirt socket, and closing an SSH
channel writes a close message under ch.writeMu — which go-libvirt already
holds while streaming a base image through that same channel. Against a
host that vanished without RST or FIN, that write blocks until the kernel
stops retransmitting (tcp_retries2, ~15 min), and s.client.Close(), a bare
net.Conn.Close() that takes no lock and unblocks the stalled writer through
netpoll, was never reached. Close the client only; nothing wants the polite
channel-close message, and reordering instead would leave an error to
filter in a memoised closeErr.
link.markDead compounded it by running kill() before close(l.closed), both
inside a sync.Once. Reordering alone was not enough: sync.Once.Do blocks
concurrent callers until the first body returns, so every other markDead —
including the one in call is own ctx-cancel branch — queued behind the
stalled kill, and cancelling the reconcile did not help. The once field is
gone; the mutex the struct already had guards the close-exactly-once
invariant, reason is set before the signal, and kill runs outside the lock.
Blast radius was the whole operator, not one host: MaxConcurrentReconciles
is 3 and ensureImage holds a per-image lock whose acquire is ctx-aware but
reconcile contexts carry no deadline, so three Proxies on one host with an
uncached image parked every worker.
The same edits close a second route to the same pin: go-libvirt is
Disconnect closes the net.Conn it was handed (libvirt.go:331 ->
socket.go:164), and that conn is our sshSocket, so libvirtBackend.Close is
Disconnect goroutine entered closeOnce and hung; the 5 s closeBudget
released Close, which called markDead -> kill -> sshSocket.Close and
blocked in the other goroutine is sync.Once.
Verified against the pinned x/crypto v0.54.0 rather than assumed: the
flow-control park (remoteWin.reserve, channel.go:268) is outside writeMu
and only writePacket holds it across the TCP write, so a writer parked in
SSH flow control never blocks the close and a writer inside the TCP write
always does. To reproduce by hand, shrink SO_SNDBUF well under the 2 MiB
channel window (channelWindowSize = 64 * channelMaxPacket) so the socket
buffer fills before the window does.
The kvm provider does not have this bug: it closes client only and never
touches the forwarded channel.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Shorter start wait with immediate verdicts for dead or paused domains,
ListByTag without state or address RPCs, Timeout for an expired check
round, no adoption of unmarked domains, reference-counted keyed locks,
dead options removed, HEAD fallback on any non-2xx.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Every go-libvirt call runs under its own budget and a cancelled or overlong
call kills the connection; credentials are read on every dial; one span and
one log line per RPC come from a decorator applied where connections are
made.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
The kvm2 block reuses KVMConfig; validateKVM gains a block-name variant so
messages say kvm2. The package skeleton has a non-dialing New, strict
providerID parsing, the state table and a 34-row classify test.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
The health check shares each host's libvirt connection with the reconciles,
so it must not be able to take that connection away from them.
- A timed-out check no longer drops the link. The check goes through the new
hostPool.probe, which abandons its RPCs when the context ends and leaves
the connection alone, so an image upload or a Create running on the same
link is not cut off. A link that really died is closed by the keepalive.
One abandoned probe per host may stay waiting; until it returns, the next
check reports Timeout at once (errProbeBusy).
- Waiting for a host's connection slot now ends with the caller's context.
hostEntry used a mutex, so a check could wait out a reconcile redialing a
dead host — minutes — past its own timeout. The slot is a one-slot channel
and acquire bounds only the wait, never the uncontended take.
- --provider-check-timeout defaults to 0 and derives min(15s, interval).
The flat 15s default contradicted the validation that the timeout may not
exceed the interval, so --provider-check-interval=12s alone failed to
start.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
--provider-check-interval (1m; 0 = off, else at least 10s) and
--provider-check-timeout (15s) are validated before the providers config
loads. Checkers are collected from the unwrapped providers in config
order, so checks never count as provider requests. Three scrape-time
gauges per {provider,target} read the monitor's snapshot: up, check
duration and check timestamp. The e2e suite waits for the kubernetes
ProviderHealth to turn Ready and asserts its gauge in the scrape.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
A leader-only runnable checks every configured provider concurrently
under one timeout per round (first round at start, then every interval),
keeps the latest results for the metrics collector, logs targets going
down, changing reason or coming back, and publishes one ProviderHealth
object per provider: created with a managed-by label when missing, then
a status merge patch. Per-target lastTransitionTime survives rounds in
which ready does not flip. Objects of providers that are no longer
configured are deleted once at start; with checks turned off, all are.
Reads go through the API reader, so no cluster-wide informer is needed.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Each configured host is checked concurrently over the pooled SSH+libvirt
connection: libvirt version and hostname prove the daemon answers, then
the configured pool must exist and be running, with its free space
reported. Dial failures are now tagged with their stage (SSH material,
TCP, SSH handshake, libvirt connect) by a dialError that keeps the
message and unwraps to the cause, so classification is unchanged and the
check can say why a host is unreachable. A pool verdict is decided after
the call returns so it never drops a working connection.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Checker is an optional interface next to Provider: Check returns one
verdict per target and never fails as a whole. The kubernetes provider
lists at most one managed Pod cluster-wide (the verb and scope GC relies
on); the GCP provider fetches a single page of at most one managed
instance through a new Probe seam method, since the SDK treats
MaxResults as a page size and the existing AggregatedList drains every
page.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Collapse duplicated logic the review found in PR #5 without changing
behaviour: one releaseFinalizer helper for the four finalizer exits in
reconcileDelete, classify reusing isClassified and provider.Class, the
image URL check and format/firmware values shared from the provider
package, imageSource and the metadata parser embedding their config
structs instead of copying fields, and the SSH loader as a small struct
that keeps only the SSH fields and is seeded by New instead of loading
the key twice. Tracing builds its V(2)-only lists only at V(2).
Tests: drop a duplicate New test, fold near-identical tests into tables
(connection drops, metadata parsing, cached images), add mustSSHAuth,
and use wantClass where classes were checked by hand.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Merging the bootstrap into each Proxy's cloud-config was the wrong
mechanism twice over. Detection only matched a literal #cloud-config
first line, so a jinja template or a MIME archive still fell back to
vendor-data, where cloud-init's default mergers drop it — the exact
failure the merge existed to prevent. And the YAML round-trip re-typed
unquoted scalars: version: 1.10 reached the guest as 1.1, comments were
dropped and keys reordered.
Neither is worth patching, because vendor-data delivered as a shell
script bypasses the merger entirely: cloud-init hands vendor-data to its
own handler chain, writes scripts to a separate directory and runs them
from cc_scripts_vendor, after the user's packages and before the user's
runcmd. The merge could not have been rescued anyway — merge_how in
user-data governs only the first source, and merge_how in vendor-data is
discarded before the document is stored.
The Proxy's cloud-init now reaches the seed byte for byte in any format.
installGuestAgent (default true) turns the script off; a configured
vendorData travels beside it as a MIME multipart, declared text/plain so
cloud-init re-sniffs it and this code never classifies user content.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The kvm-ssh Secret is mounted optional, but kvm.New read the key and
known_hosts eagerly and registry.Build aborts the whole provider set on a
constructor error, so a missing or misnamed Secret exited the manager
before any controller started and took the kubernetes and gcp providers
down with it.
lazyAuth loads the material on the first connection and caches it once it
works, so a Secret created after the manager starts heals the affected
proxies on the next reconcile with no restart. A missing file classifies
transient for that reason; a key that does not parse is permanent.
New still checks both files and logs a WARNING: line when they are
unreadable, keeping the startup diagnostic the eager load was there for.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
A host name is pasted into the providerID hosts/<name>/domains/<vm>, which
parseProviderID splits on "/", so a name containing a slash produced an ID
that never parsed again: Get, Delete and GC all failed permanently and the
VM leaked with the Proxy's finalizer stuck.
The image map key has the same defect and lands in a file name rather than
an ID: base-<key>.<format> in the host's storage pool. Both now have to
match a plain identifier pattern. Machine-type keys are left alone; they
are only ever compared, never embedded.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
reconcileDelete asked Get first and dropped the finalizer only on
not-found, so a permanent Get retried forever: a KVM host removed from
the config, or a providerID the provider cannot resolve, left the Proxy
in Terminating with no way out. It now tries Delete anyway, which is
idempotent and can still finish where Get cannot converge, and releases
the object when that succeeds.
crawl.example.com/skip-cleanup abandons the instance for the cases that
stay stuck, such as revoked credentials. It is checked before the
provider lookup, so it also frees a Proxy whose provider is gone from
the config.
In the KVM provider, Delete over an unconfigured host is now a no-op
success, mirroring removeVolumes over a missing pool. Get keeps failing
permanently there on purpose: the live path reads not-found as "rebuild
it", so a typo in hosts must not look like a missing VM.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
In the kind cluster two KVM Proxies with real cloud-configs booted fine
but never left Provisioning: the guest agent was not installed, so the
provider could not learn the IP. cloud-init merges vendor-data under
user-data with its default mergers, which keep the user-data's value
for any key both define and never append lists, so a cloud-config with
its own `packages:` silently dropped the vendor-data's packages and
runcmd. The integration test passed only because its user-data was
empty.
withGuestAgent now appends qemu-guest-agent to `packages` and the
enable command to `runcmd` of the Proxy's own cloud-config (skipping
entries already present) and re-renders it. Vendor-data is no longer
defaulted; `vendorData` is plain extra data. User-data that is not a
cloud-config document (script, #include, MIME) still gets the bootstrap
as vendor-data. A cloud-config that is not a YAML mapping, or whose
packages/runcmd is not a list, fails the Proxy as ErrPermanent.
Validated in-cluster: both Proxies Ready and healthy in two minutes, 204
through squid from the LAN. Docs, sample comment, CHANGELOG and the
plan execution log (Run 7) updated.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
The manager runs with a read-only root filesystem and the seed ISO
writer stages files in a temp directory, so the first in-cluster Create
failed with "creating iso writer: read-only file system". Mount a 64 MiB
emptyDir on /tmp.
That error is classified ErrPermanent by writeSeed, yet hostPool.call
treated every non-libvirt error as a transport failure and dropped the
cached SSH connection, redialling on each retry. Errors that already
carry a provider sentinel are local verdicts, not evidence about the
link; leave the connection alone for those.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
The known-gaps entry, decision log, sample comment and builder comment
blamed the host; the cause is trixie's GRUB config ("terminal_output
gfxterm serial" with a serial console and no video device), reported
and reproduced upstream and still open. UEFI remains the default for the
reason recorded; bookworm images are noted as unaffected.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Run 5 on the real host bisected the guest hang: the Debian 13 image
hangs after GRUB under SeaBIOS with or without our seed and with either
CPU model, and boots under OVMF. The image's legacy boot chain is the
problem, not the provider.
images.<key>.firmware is "uefi" (default) or "bios". The domain XML
emits <os firmware="efi"> with secure boot off, so libvirt selects the
OVMF binary and manages the NVRAM file, which the existing nvram undefine
flag removes. Direct-URL images default to UEFI too. OVMF joins the
hypervisor prerequisites; sample, config table, known gaps, decision log
and the execution log record the finding.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
The first live run showed the trace working but noisy: testr printed
ts/level/msg and the per-run constants on every line, the error text
appeared twice, host appeared twice on image-cache lines, and the Go
method name was less useful than the libvirt function it maps to.
The tracer now logs rpc=virDomainLookupByName (go-libvirt's names are the
public API names without the vir prefix), error plus libvirtError (code
name) and libvirtCode, and image.go stops re-adding host. The integration
test replaces testr with a compact logr sink: time, logger name, the rpc
as headline, then fields, with provider/host/name omitted since the first
line of the run states them. Docs and execution log updated.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
A tracingAPI decorator wraps the 17-method libvirt seam and logs one
"libvirt call" line per RPC after it returns: method, the identifiers it
was given, curated results, duration, and on failure the error with its
libvirt code, name and message. At V(2) the same line carries the domain
and volume XML, metadata and addresses; upload bodies are never logged.
The pool applies it in call() when V(1) is enabled and logs connect,
dial-failure and drop events, taking the caller's logger explicitly like
every other helper in the package.
The integration test installs a testr logger at KVM_TEST_LOG_LEVEL
(default 1) so the trace interleaves with t.Log output, which also ends
the "log.SetLogger(...) was never called" warning. Docs gained a Logging
section and the test env var; the KVM provider execution log records run
4 (guest hang at GRUB) that prompted this.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Run 3 on the real host got past Create and failed in Get: libvirt 11.6
selects the <instance> element by our URI in DomainGetMetadata but hands
it back with no namespace declaration, and the parser required the URI
on the element. The lookup already guarantees the namespace, so match on
the local name only. The fake now strips the declaration the way libvirt
does, and a new test pins the bare form. Execution log updated.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
The first run against a real hypervisor (beelink, Ubuntu, libvirt 11.6)
failed at DomainCreate with "Could not open base image: Permission denied"
even with correct DAC ownership. libvirt's virt-aa-helper only grants QEMU
access to disk sources that carry a path; the <disk type="volume"> form
chosen in Step 2 gives it pool and volume names instead, so the overlay and
its backing image get no AppArmor rule on any stock Debian/Ubuntu host.
buildDomainXML now takes DiskPath/SeedPath and emits type="file" disks.
ensureOverlay and writeSeed return the volume handles and Create resolves
both paths with StorageVolGetPath before defining the domain. The domain
XML test rejects any volume/pool reference so this cannot drift back; the
Create test asserts the sources equal the paths libvirt reported.
Execution log: Step 2 note marked reverted, Step 7 entry opened with the
two failed runs and the diagnosis.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Behind -tags=integration and self-skipping unless KVM_TEST_* names a
host: one sequential lifecycle (image fetch, Create, idempotent
re-Create, Running with IP, ListByTag, Delete, pool state afterwards),
driven through the provider directly so it needs no Step 5 wiring.
Adds `make test-integration-kvm` and documents the run and the
deviation from the testcontainers convention in docs/testing.md.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Create looks the domain up first and only starts it when it exists;
otherwise it caches the image, reuses or creates the overlay, rebuilds
the seed ISO unconditionally (a cancelled upload can leave a truncated
seed that no later attempt would repair), defines and starts the domain.
Get maps libvirt state plus the first routable IPv4 from ARP or the
guest agent onto the provider states. Delete destroys, undefines and
removes the per-VM volumes by name, tolerating every already-gone case.
ListByTag keeps managed domains across all hosts, skipping domains it
cannot read and failing the sweep only for an unreachable host.
Also corrects the Step 2 seed determinism test: the iso9660 writer
stamps the build time into the volume descriptor, so builds differ by a
few bytes; the test now compares label, files and size.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
resolveImage maps placement.image to a configured key or URL; imageCache
streams the download through StorageVolUpload with a sha256 tee, writes
a marker volume when the upload is verified, and re-fetches when the
marker is missing or the configured checksum changed. Errors carry
taxonomy sentinels that classify now honours. Adds the in-memory fake
libvirt API used from here on.
Step 3b of docs/plans/2026-09-04-1404-kvm-provider.md.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Per-host connection pool over SSH with the libvirt unix socket forwarded
through the session, lazy dial and redial after disconnect, context
cancellation for go-libvirt's context-less RPCs, handshake deadline and
SSH keepalives. Error classification onto the provider taxonomy. Tested
with a fake dialer and an in-process SSH server that forwards unix sockets.
Step 3 of docs/plans/2026-09-04-1404-kvm-provider.md.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Introduces internal/provider/kvm with the side-effect-free pieces of the
provider: providerID format/parse, the namespaced <metadata> element that
carries the GC contract, q35 domain XML over pool volumes, volume XML for
overlays and raw seeds, the NoCloud seed ISO built with kdomanski/iso9660,
and config defaults. All unit-tested.
Step 2 of docs/plans/2026-09-04-1404-kvm-provider.md.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Adds the kvm block to the providers config (SSH access, images map,
machine-type profiles, hosts list) with validation and table tests.
validate() now rejects mismatched type-specific blocks generically via
setBlocks() instead of pairwise checks.
Step 1 of docs/plans/2026-09-04-1404-kvm-provider.md.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
kacerrandClaude Opus 5.5