The contract sweep in provider_test.go: every method and Check returns
within a second of a cancel, every wrapErr path carries the configured
name, op and id, the V0/V1/V2 logging tiers and the no-op lines, and a
Get over the real client whose HTTP spans are children of provider.get.
proxmox_integration_test.go (-tags=integration) holds TestProxmox_lifecycle
and TestProxmoxCheck_realHost; both compile and skip without
PROXMOX_TEST_*, and have not run against a host yet. make
test-integration-proxmox runs them. docs/testing.md gains the proxmox,
nocloud and keyedlock rows, the Live Proxmox gaps and the integration
section; spec entry 21's section 11 list is ticked.
Waits that could hang the package under a mutation are bounded: the
concurrent-Create loop and eight direct blocking calls in older tests.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Delete stops and destroys a VM of ours under the per-name lock, then
deletes its seed on the VM's node, or on every node when no VM of ours
was left. A hibernated VM of ours is Permanent, as in Create. The
Checker verifies each node concurrently (version, node status, ISO
storage, templates) and adds CertificateInvalid, CredentialsUnavailable,
StorageNotFound and TemplateNotFound to the shared reasons. The
ProviderHealth type comment gains proxmox.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Create validates placement, then under the per-name lock adopts the VM a
previous attempt left (by name, or at a taken VMID after the ownership
check of spec entry 2) or full-clones the template onto the UID's
deterministic VMID with the mark in the clone call. It then uploads the
seed over any existing file, writes cores, memory, agent, tag, ide2 and
net0 when configured, grows the boot disk when diskGiB is larger, starts,
and waits for running/running.
waitUnlocked no longer waits on a suspended lock: a hibernated VM is
judged at once, so ours is Permanent and a foreign one is probed past.
The template is checked only on the allocation path, and its shape
(net0, boot disk size) is refused before any clone exists. Spec entries
2, 4 and 10 updated to match.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Get follows spec entry 9's check order (lock and unknown status before
the name, suspended is Stopped after the name and mark checks), reads
status/current so paused is Stopped, and maps the three agent messages
of entry 7. ListByTag reads one config per candidate and fails whole
only on the resources call or an ended ctx. waitTask polls the UPID's
node every budgets.poll, logs failed tasks at V(1) and counts
"WARNINGS: <n>" as success; waitUnlocked serves Create's ownership check.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
httpAPI implements the sixteen seam methods over net/http: the token is
read per request, non-2xx answers become apiError from the body message or
the reason phrase, echoed secrets are redacted, the CA loads lazily, HTTP/2
is off, and uploads stream with an exact Content-Length. The observed
decorator logs one V(1) line per call and starts no span; New wires both.
fakeAPI simulates clone locks, tasks and ISOs with the captured error texts.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
kvm2's NoCloud seed builder moves to internal/provider/nocloud (exported,
with an ErrEmptyName sentinel instead of permanentf) and its keyed locks
to internal/provider/keyedlock, tests moved under their existing names.
kvm2 behaviour is unchanged. proxmox gains the per-name lock and the pure
seed helpers (seedISOName, seedVolID, buildSeed); unlike kvm2 (review
finding S5.1), a seed staging failure there is Transient, not Permanent.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Phase 2 of the proxmox provider plan. Adds the proxmox: config block with
its validation, registers the type in the config allow-list and the
constructor map, and starts internal/provider/proxmox: the api seam and
its data types, providerIDs (vms/<vmid>/<name>) with the UID-derived VMID,
the description mark, state mapping, and error classification.
Tests read the responses captured from pv1-1 in Phase 0 rather than
repeating Proxmox's wording, and every matcher is run against every
captured error so none fires on the wrong message. The contract methods
are Transient stubs until Phases 5 to 7.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Phase 1 of the proxmox provider plan: docs/proxmox-provider-spec.md
answers all 24 entries of the provider spec template against the
contract, citing the Phase 0 facts from pv1-1 as evidence.
It folds in the ten corrections recon found. The two that matter most:
Create inspects a taken VMID, waiting out a clone lock, before probing
past it, so a crash-retry adopts its own half-cloned VM instead of cloning
a duplicate; and Get checks the lock before the name, because a VM being
cloned has no name yet.
The plan now opens with a note that the spec wins where they disagree.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Phase 0 of the proxmox provider plan. 55 fixtures under
internal/provider/proxmox/testdata, each the status, reason phrase and
raw body of a real response from the user's host, and the execution log
answering the plan's facts F1 to F23.
Recon ran a clone of the Debian 13 template through the whole path the
provider will take, twice: untagged and on VLAN 4. On both, the NoCloud
seed on ide2 installed the guest agent and an address came back about 50
seconds after start. It also found two bugs in the plan: a VM being
cloned has no name in /cluster/resources, so name-first lookups would read
it as gone, and a crash-retry would clone a duplicate. Those and eight
smaller corrections are listed for the spec.
Decisions made with the user: seeds go to a dedicated proxy-seeds
storage where the token can delete them; proxies land on a VLAN set in
config (vlanTag); the API URL is https://192.168.0.8:8006, the address in
the node certificate.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Closes out the review-fix campaign. The kvm2 provider was deployed
in-cluster and validated against beelink -- the first time any of this
code had run against real libvirt rather than the in-memory fake.
config/manager/providers_config.yaml ships the kvm2-home entry enabled,
since the operator is now running it for real.
review/ carries the code review the four H-commits reference, so H1..H4
resolve to something rather than dangling: scope, the seven-chunk sweep,
the escalations, the ten deep dives and the final report. .claude/agents
carries the two reviewer definitions that produced them.
What the validation did not prove is recorded next to what it did: H2's
error paths need a body that fails part way through, and a healthy 433 MB
download never finishes short, so the unverified libvirtd short-stream
behaviour is still open. The kvm2 integration test has also still never
been run.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Some lines ignored the zap encoder entirely, so `kubectl logs | jq` could
never work:
I0917 19:50:02.580097 1 leaderelection.go:258] "Attempting to acquire
leader lease..." lock="egress-proxies-operator-system/b47711d1..."
That is klog's own stderr writer, not ours. client-go's leader elector
calls klog.FromContext(ctx), and controller-runtime hands the elector a
context with no logr in it -- mgr.Start receives ctrl.SetupSignalHandler(),
which carries no logger. FromContext falls back to Background(), and with
no logger registered that is klog's text sink. Nothing here imported or
configured klog; it was an indirect dependency that around 60 linked
packages use, including client-go/tools/{leaderelection,cache,record} and
client-go/transport. controller-runtime does not route it either -- its
only klog.SetLogger call is in its own test suite.
Plain klog.SetLogger, deliberately not SetLoggerWithOptions with
ContextualLogger(true). The contextual form is what klog's docs recommend
because it skips a klogr round trip, but it also disables klog's verbosity
gate and hands the decision to zap, whose level under Development is Debug
-- so client-go's V(1) chatter, suppressed today by klog's own -v=0, would
have started appearing. Plain SetLogger keeps klog's check in front and
still redirects what it does emit. Same fix to the format, no new volume,
which matters because volume is what prompted this.
The discovery API's http.Server set no ErrorLog, so net/http's own faults
(a failed Accept, a panic escaping a handler) went to the stdlib logger on
stderr. It now drains into logr through logr.ToSlogHandler, the pattern
internal/provider/gcp/wirelog.go already uses for the same purpose.
The deployment gets --zap-encoder=json and --zap-time-encoding=rfc3339.
The flag beats the in-code Development default -- addDefaults fills
NewEncoder only when it is nil -- and the time encoding is not optional,
since the JSON encoder defaults to epoch seconds. Verbosity stays at debug
on purpose; --zap-log-level=info drops the per-RPC lines.
zap.Options{Development: true} stays as the in-code default: flipping it
would change the format for every deployment of this operator, and the
flags get the same result here without that reach.
k8s.io/klog/v2 moves from indirect to direct in go.mod. It was already in
the module graph, so nothing new is downloaded.
No tests: three changes, all wiring, in a file with no test coverage, and
pinning klog.SetLogger means mutating process-global state that leaks into
every other test in the package. Verified in-cluster instead --
`kubectl logs | jq -e -s 'length > 0'` now passes, which fails if any
single line is not JSON.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
baseVolumeName was base-<key>.<format>, byte-identical to what the kvm
provider builds in the same pool -- for URL images too, since kvm takes
hex(sum[:8]) and kvm2 takes hex(sum[:])[:16], the same eight bytes -- with
a completely separate lock table. The shipped config has a live kvm-home
entry and a commented-out kvm2-home entry on the same host, the same key
debian-13 and the same default pool, and the comment invites the operator
to enable it. Both integration tests use key test-image. On beelink both
live overlays back onto kvm's base-debian-13.qcow2.
Since cachedImage accepts any 64-hex marker when no sha256 is configured
and never re-hashes the base, that meant two entries pointing one key at
different URLs silently shared a file; a concurrent cold fetch let one
entry delete the other's half-uploaded base and boot a guest from the
remains; and a sha256 edit deleted the base in place under live overlays,
which libvirt permits without complaint.
The name now carries the provider entry: base-<entry>-<key>.<format>.
imageSource gained an Entry field and resolveImage takes the provider
name, so every production call site follows from imageSource.volumeName().
That closes the cross-entry sharing outright, kvm<->kvm2 and kvm2<->kvm2
alike, and turns the per-Provider imageLocks from a latent bug into a
correct one: with names unique per entry, two entries cannot contend for a
file, so no process-wide lock table is needed.
The entry name now reaches a file name, so New checks it against the
existing kvmIdentifier rule, exported as provider.IsKVMIdentifier rather
than duplicated. Provider names are otherwise validated for all types at
once and only for being non-empty and unique, so a name with a slash would
have produced a volume nothing could address.
One companion fix, because the naming alone does not stop it:
cachedImage's digest-mismatch branch deleted the marker and the base and
refetched. It now returns ErrPermanent naming both digests and telling the
operator what to do. That branch was the remaining path that unlinks a
good file under running domains, and the contract already said "add a new
key instead" -- the code was contradicting it. The no-marker and
non-hex-marker branches keep deleting, since there the base is debris.
Not fixed, and said so in the docs rather than left to look closed:
editing images.<key>.url is still a silent no-op, since nothing records
the URL; and cross-process fencing still needs the deferred atomic publish
(upload to a temp name, verify, publish), which lands where H2 put the
discard.
Migration is a non-event. The base-debian-13.qcow2 on beelink is kvm's,
kvm keeps using it untouched, and kvm2 downloads its own copy once per
host. Nothing is replaced under a running guest; the pool needs room for
both, which the prerequisites now say.
The eviction recipe in the guide named base-debian-13.qcow2, which after
this change is kvm's file -- following it would have deleted the wrong
provider's base. It now greps for the prefix and deletes the marker first.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
address() tried the guest agent then the host's ARP table and skipped any
error that was not a context error, returning ("", nil) after both. An
active domain with no IP maps to StateProvisioning; the controller clears
status.IP on that, and the health engine prunes the proxy's counters and
calls ForgetProxy in one tick once EffectiveHost() goes empty.
The damaging trigger was not the guest. The libvirt connection is pooled
per host, so when another reconcile's cancel or budget overrun killed it,
call returned "connection is closed" -- carrying no context sentinel, so
the check did not recognise it -- and Get returned a *successful*
instance in Provisioning with an empty IP. An unrelated proxy's timeout
dropped a healthy proxy out of rotation.
The discriminator was already in the spec without the conclusion being
drawn: during a normal boot on beelink exactly one source fails. The
agent errors because it is not up yet, while ARP answers without error
and with nothing, since the host never sees the guest's traffic. So one
source failing stays "no address yet"; every source failing, or a
connection already dead, is ErrTransient.
The explicit provider.ErrTransient wrap is load-bearing, not decoration.
Without it classify picks the first libvirt code out of the joined error,
and ErrInvalidArg, ErrAccessDenied and ErrNoSupport are Permanent -- an
address read failing would latch a running proxy as Failed, a worse
version of the harm being fixed.
Half of the finding is deliberately still open, and it is the half that
fires more often: a guest agent restarting inside a running VM still
demotes it, because that is indistinguishable from a VM still booting.
The discriminator is the previous IP, which the provider does not hold by
design and the controller already has. Follow-up finding to file against
proxy_controller.go:208-211 -- on StateProvisioning, keep an existing IP
and Provisioned condition for a few polls instead of clearing on the
first. The kvm2-local alternative (use the mark's createdAt as a
provisioning window) was rejected: createdAt is creation, not boot, and
it invents a deadline that belongs to the controller.
TestGet_addressErrorIsProvisioning failed against the new rule, correctly
-- it fails both sources with ErrAgentUnresponsive, and code 86 is
agent-specific, so the ARP source cannot produce it. Re-pointed as
TestGet_agentFailsAndARPEmptyIsProvisioning, the beelink boot it was
reaching for, which nothing covered before.
TestGet_everyAddressSourceFailingIsTransient covers the other side, both
sources erroring and the pooled link dying under the read; against the
unfixed code both return a Provisioning instance instead of an error.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
uploadImage streamed the HTTP body straight into StorageVolUpload. When
that body failed part way through, go-libvirt deadlocked: requestStream
starts its stream sender on two unbuffered channels and, on the error
path, returns without ever reading outErr, so SendStream emits a
StatusError packet and parks on `outErr <-` forever while the main
goroutine parks on `abort <- true` forever. StorageVolUpload never
returned. The 20-minute transfer budget rescued the caller only, and the
socket kill did nothing, because both ends are blocked on channel sends
rather than on I/O -- which is what the comment at backend.go:152 got
wrong. Each failed upload leaked kvm2's call goroutine, go-libvirt's
sender goroutine and a 4 MiB buffer, permanently, against a 128Mi pod
limit and a reconciler that retries on backoff.
The defect is upstream and stays there: no fork, no replace. Instead
eofReader, at the go-libvirt seam so it covers the seed and marker
uploads too, converts any source error to io.EOF -- the one outcome that
ends a stream the normal way -- and keeps the real error for the caller.
countingReader sits underneath untouched, so the existing length and
digest checks report exactly what they did before.
The download also gets its own deadline a minute inside the transfer
budget. fetchImage already bounded the whole fetch at 20 minutes, so the
body's death and the budget's socket kill were racing and the socket
usually won; with the body a minute ahead, a stalled mirror ends through
its reader and unwinds cleanly instead of leaking.
discardVolume moves from uploadImage into fetchImage's upload-error
branch, so the dedicated connection is closed and libvirtd has let go of
the stream's fd before the file is unlinked. Moving it rather than adding
a second Close keeps image_test's closeCount() == 1 true. That discard is
still unfenced against a second provider entry sharing the volume name;
H4 closes that by making the name per-entry.
What remains: a failure originating in libvirtd mid-upload still leaks,
since SendStream has already returned by then. Confirmed unfixable from
here. Known gaps quotes the two-character upstream fix
(make(chan bool, 1) / make(chan error, 1)) and the replace escape hatch.
Unverified at ~85%: that libvirtd answers a short-but-clean stream finish
with OK rather than an error. virFDStreamWrite enforces length only on
writes past it and nothing compares offset to length on the close path,
which is also why virsh vol-upload of a small file into a large volume
works. If it errors we are back to a deadlock by a different door, which
is exactly today's behaviour, so this cannot be worse. The execution file
carries a probe recipe to settle it.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
sshSocket.Close was errors.Join(s.Conn.Close(), s.client.Close()). Go
evaluates those left to right, so the blocking close ran before the
unblocking one. s.Conn is the forwarded libvirt socket, and closing an SSH
channel writes a close message under ch.writeMu — which go-libvirt already
holds while streaming a base image through that same channel. Against a
host that vanished without RST or FIN, that write blocks until the kernel
stops retransmitting (tcp_retries2, ~15 min), and s.client.Close(), a bare
net.Conn.Close() that takes no lock and unblocks the stalled writer through
netpoll, was never reached. Close the client only; nothing wants the polite
channel-close message, and reordering instead would leave an error to
filter in a memoised closeErr.
link.markDead compounded it by running kill() before close(l.closed), both
inside a sync.Once. Reordering alone was not enough: sync.Once.Do blocks
concurrent callers until the first body returns, so every other markDead —
including the one in call is own ctx-cancel branch — queued behind the
stalled kill, and cancelling the reconcile did not help. The once field is
gone; the mutex the struct already had guards the close-exactly-once
invariant, reason is set before the signal, and kill runs outside the lock.
Blast radius was the whole operator, not one host: MaxConcurrentReconciles
is 3 and ensureImage holds a per-image lock whose acquire is ctx-aware but
reconcile contexts carry no deadline, so three Proxies on one host with an
uncached image parked every worker.
The same edits close a second route to the same pin: go-libvirt is
Disconnect closes the net.Conn it was handed (libvirt.go:331 ->
socket.go:164), and that conn is our sshSocket, so libvirtBackend.Close is
Disconnect goroutine entered closeOnce and hung; the 5 s closeBudget
released Close, which called markDead -> kill -> sshSocket.Close and
blocked in the other goroutine is sync.Once.
Verified against the pinned x/crypto v0.54.0 rather than assumed: the
flow-control park (remoteWin.reserve, channel.go:268) is outside writeMu
and only writePacket holds it across the TCP write, so a writer parked in
SSH flow control never blocks the close and a writer inside the TCP write
always does. To reproduce by hand, shrink SO_SNDBUF well under the 2 MiB
channel window (channelWindowSize = 64 * channelMaxPacket) so the socket
buffer fills before the window does.
The kvm provider does not have this bug: it closes client only and never
touches the forwarded channel.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Shorter start wait with immediate verdicts for dead or paused domains,
ListByTag without state or address RPCs, Timeout for an expired check
round, no adoption of unmarked domains, reference-counted keyed locks,
dead options removed, HEAD fallback on any non-2xx.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Every go-libvirt call runs under its own budget and a cancelled or overlong
call kills the connection; credentials are read on every dial; one span and
one log line per RPC come from a decorator applied where connections are
made.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
The kvm2 block reuses KVMConfig; validateKVM gains a block-name variant so
messages say kvm2. The package skeleton has a non-dialing New, strict
providerID parsing, the state table and a 34-row classify test.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Profile table keyed on the contract's axis codes, per-provider reference
sections for kubernetes, gcp and kvm, shared gaps and proposed features.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Twenty-four fill-in entries, one per decision the contract leaves to the
provider, numbered to match the contract's profile table.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Fully abstract: MUST/SHOULD/MAY rules with the caller behind each one,
19 behaviour-profile axes with stable codes, credential hygiene, testing
obligations, the checklist and the contract's known limits.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
A pass over the API types, spec hash, health and discovery call sites,
manager manifests, samples, README, provider docs, plans and the PR review
found rules the blueprint did not state: the IP must be reachable on the
proxy port by the health engine and clients, providers must be safe for
concurrent use, the manager's read-only/128Mi runtime envelope, catalogue
edits bypass the spec hash, Delete is re-called while in progress, and
provider names must be DNS-1123. Adds matrix rows, a deployment checklist
item, and the still-open review findings and broken samples as gaps.
README's "Adding a provider" restated an outdated contract; it now points
to docs/providers.md, and the providers table gains the kvm row.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The blueprint reduced tracing to "don't open per-operation spans", which
reads as "nothing to do". Add an observability section: the operation
spans the decorators give for free, the child spans per backend call a
provider should add (parent-gated transport or a span decorator over the
SDK seam), the ctx/logger rules that keep traceID on log lines, and the
status per provider: kubernetes and gcp have child spans, kvm has none.
Matrix rows, the checklist item, gaps and a kvm proposal follow suit.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
A re-check against the code found four mismatches: the kvm connection is
opened at startup by the leader's first health check, so that isn't doc
drift; cmd/main.go, not the monitor, type-asserts Checker; QuotaExceeded
and PermanentError are reasons on Provisioned, not conditions; and the
latch line range. The checklist also now matches how the existing
providers do interface assertions and RBAC markers.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
docs/providers.md collects the provider contract that was spread across
comments, callers and per-provider docs: MUST/SHOULD/MAY rules per method,
each tied to the reconciler or GC behaviour it protects, the error taxonomy,
a feature matrix for kubernetes/gcp/kvm with file:line notes, a checklist for
adding a provider, and the known gaps. Linked from architecture.md and
CLAUDE.md.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The user ran the real-hypervisor checks against beelink and confirmed they
passed, which completes Step 7 of the provider health plan.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The health check shares each host's libvirt connection with the reconciles,
so it must not be able to take that connection away from them.
- A timed-out check no longer drops the link. The check goes through the new
hostPool.probe, which abandons its RPCs when the context ends and leaves
the connection alone, so an image upload or a Create running on the same
link is not cut off. A link that really died is closed by the keepalive.
One abandoned probe per host may stay waiting; until it returns, the next
check reports Timeout at once (errProbeBusy).
- Waiting for a host's connection slot now ends with the caller's context.
hostEntry used a mutex, so a check could wait out a reconcile redialing a
dead host — minutes — past its own timeout. The slot is a one-slot channel
and acquire bounds only the wait, never the uncontended take.
- --provider-check-timeout defaults to 0 and derives min(15s, interval).
The flat 15s default contradicted the validation that the timeout may not
exceed the interval, so --provider-check-interval=12s alone failed to
start.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
make test (envtest, -race) and the kind e2e suite pass; the e2e
ProviderHealth for the kubernetes provider turned Ready within a second
of the manager starting and its gauge appeared in the scrape. make lint
is blocked by the local go1.27 toolchain crashing staticcheck on the
standard library, unrelated to this branch.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Architecture gets a provider-health section, component rows and four
decisions; the KVM provider doc gets a health-check section with the
reason table and the rotated-Secret gap; the testing guide lists the new
suites. CLAUDE.md's Architecture TODO and stale "only go.mod exists"
gotcha now point at the real docs, plus the RBAC-marker gotcha found
while wiring the monitor.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
--provider-check-interval (1m; 0 = off, else at least 10s) and
--provider-check-timeout (15s) are validated before the providers config
loads. Checkers are collected from the unwrapped providers in config
order, so checks never count as provider requests. Three scrape-time
gauges per {provider,target} read the monitor's snapshot: up, check
duration and check timestamp. The e2e suite waits for the kubernetes
ProviderHealth to turn Ready and asserts its gauge in the scrape.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
A leader-only runnable checks every configured provider concurrently
under one timeout per round (first round at start, then every interval),
keeps the latest results for the metrics collector, logs targets going
down, changing reason or coming back, and publishes one ProviderHealth
object per provider: created with a managed-by label when missing, then
a status merge patch. Per-target lastTransitionTime survives rounds in
which ready does not flip. Objects of providers that are no longer
configured are deleted once at start; with checks turned off, all are.
Reads go through the API reader, so no cluster-wide informer is needed.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Each configured host is checked concurrently over the pooled SSH+libvirt
connection: libvirt version and hostname prove the daemon answers, then
the configured pool must exist and be running, with its free space
reported. Dial failures are now tagged with their stage (SSH material,
TCP, SSH handshake, libvirt connect) by a dialError that keeps the
message and unwraps to the cause, so classification is unchanged and the
check can say why a host is unreachable. A pool verdict is decided after
the call returns so it never drops a working connection.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Checker is an optional interface next to Provider: Check returns one
verdict per target and never fails as a whole. The kubernetes provider
lists at most one managed Pod cluster-wide (the verb and scope GC relies
on); the GCP provider fetches a single page of at most one managed
instance through a new Probe seam method, since the SDK treats
MaxResults as a page size and the existing AggregatedList drains every
page.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
One operator-owned object per configured provider will carry the
periodic check result: a Ready condition, per-target verdicts with their
own transition times, and a ready/total count for kubectl get. There is
no spec; only a viewer role is shipped since the operator is the sole
writer. Written by hand (kubebuilder is not installed), registered in
the existing AddKnownTypes call rather than a second init().
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Merging the bootstrap into each Proxy's cloud-config was the wrong
mechanism twice over. Detection only matched a literal #cloud-config
first line, so a jinja template or a MIME archive still fell back to
vendor-data, where cloud-init's default mergers drop it — the exact
failure the merge existed to prevent. And the YAML round-trip re-typed
unquoted scalars: version: 1.10 reached the guest as 1.1, comments were
dropped and keys reordered.
Neither is worth patching, because vendor-data delivered as a shell
script bypasses the merger entirely: cloud-init hands vendor-data to its
own handler chain, writes scripts to a separate directory and runs them
from cc_scripts_vendor, after the user's packages and before the user's
runcmd. The merge could not have been rescued anyway — merge_how in
user-data governs only the first source, and merge_how in vendor-data is
discarded before the document is stored.
The Proxy's cloud-init now reaches the seed byte for byte in any format.
installGuestAgent (default true) turns the script off; a configured
vendorData travels beside it as a MIME multipart, declared text/plain so
cloud-init re-sniffs it and this code never classifies user content.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The kvm-ssh Secret is mounted optional, but kvm.New read the key and
known_hosts eagerly and registry.Build aborts the whole provider set on a
constructor error, so a missing or misnamed Secret exited the manager
before any controller started and took the kubernetes and gcp providers
down with it.
lazyAuth loads the material on the first connection and caches it once it
works, so a Secret created after the manager starts heals the affected
proxies on the next reconcile with no restart. A missing file classifies
transient for that reason; a key that does not parse is permanent.
New still checks both files and logs a WARNING: line when they are
unreadable, keeping the startup diagnostic the eager load was there for.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
A host name is pasted into the providerID hosts/<name>/domains/<vm>, which
parseProviderID splits on "/", so a name containing a slash produced an ID
that never parsed again: Get, Delete and GC all failed permanently and the
VM leaked with the Proxy's finalizer stuck.
The image map key has the same defect and lands in a file name rather than
an ID: base-<key>.<format> in the host's storage pool. Both now have to
match a plain identifier pattern. Machine-type keys are left alone; they
are only ever compared, never embedded.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
reconcileDelete asked Get first and dropped the finalizer only on
not-found, so a permanent Get retried forever: a KVM host removed from
the config, or a providerID the provider cannot resolve, left the Proxy
in Terminating with no way out. It now tries Delete anyway, which is
idempotent and can still finish where Get cannot converge, and releases
the object when that succeeds.
crawl.example.com/skip-cleanup abandons the instance for the cases that
stay stuck, such as revoked credentials. It is checked before the
provider lookup, so it also frees a Proxy whose provider is gone from
the config.
In the KVM provider, Delete over an unconfigured host is now a no-op
success, mirroring removeVolumes over a missing pool. Get keeps failing
permanently there on purpose: the live path reads not-found as "rebuild
it", so a typo in hosts must not look like a missing VM.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
In the kind cluster two KVM Proxies with real cloud-configs booted fine
but never left Provisioning: the guest agent was not installed, so the
provider could not learn the IP. cloud-init merges vendor-data under
user-data with its default mergers, which keep the user-data's value
for any key both define and never append lists, so a cloud-config with
its own `packages:` silently dropped the vendor-data's packages and
runcmd. The integration test passed only because its user-data was
empty.
withGuestAgent now appends qemu-guest-agent to `packages` and the
enable command to `runcmd` of the Proxy's own cloud-config (skipping
entries already present) and re-renders it. Vendor-data is no longer
defaulted; `vendorData` is plain extra data. User-data that is not a
cloud-config document (script, #include, MIME) still gets the bootstrap
as vendor-data. A cloud-config that is not a YAML mapping, or whose
packages/runcmd is not a list, fails the Proxy as ErrPermanent.
Validated in-cluster: both Proxies Ready and healthy in two minutes, 204
through squid from the LAN. Docs, sample comment, CHANGELOG and the
plan execution log (Run 7) updated.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
kacerrandClaude Opus 5.5