Commit Graph
140 Commits
Author SHA1 Message Date
kacerr 5928c3fe75 Merge pull request 'kvm2 provider: clean-room libvirt/KVM provider, code review, and the four high findings' (#8) from feat/kvm2-provider into main
Reviewed-on: #8
2026-09-17 23:12:46 +02:00
kacerrandClaude Opus 5 54525bf67a kvm2: gofmt volume.go
Making baseVolumeName multi-line in the H4 commit collapsed the
alignment gofmt gives a run of single-line funcs, and the commit went out
with the old spacing. Caught by make test, whose fmt target rewrites the
file and leaves the tree dirty.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-09-17 21:07:10 +00:00
kacerrandClaude Opus 5 41fbd5e7e6 kvm2: record the review validation, enable the provider entry, changelog
Closes out the review-fix campaign. The kvm2 provider was deployed
in-cluster and validated against beelink -- the first time any of this
code had run against real libvirt rather than the in-memory fake.

config/manager/providers_config.yaml ships the kvm2-home entry enabled,
since the operator is now running it for real.

review/ carries the code review the four H-commits reference, so H1..H4
resolve to something rather than dangling: scope, the seven-chunk sweep,
the escalations, the ten deep dives and the final report. .claude/agents
carries the two reviewer definitions that produced them.

What the validation did not prove is recorded next to what it did: H2's
error paths need a body that fails part way through, and a healthy 433 MB
download never finishes short, so the unverified libvirtd short-stream
behaviour is still open. The kvm2 integration test has also still never
been run.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-09-17 20:50:45 +00:00
kacerrandClaude Opus 5 4114d43e87 Route klog and net/http's own errors into the structured logger
Some lines ignored the zap encoder entirely, so `kubectl logs | jq` could
never work:

  I0917 19:50:02.580097  1 leaderelection.go:258] "Attempting to acquire
  leader lease..." lock="egress-proxies-operator-system/b47711d1..."

That is klog's own stderr writer, not ours. client-go's leader elector
calls klog.FromContext(ctx), and controller-runtime hands the elector a
context with no logr in it -- mgr.Start receives ctrl.SetupSignalHandler(),
which carries no logger. FromContext falls back to Background(), and with
no logger registered that is klog's text sink. Nothing here imported or
configured klog; it was an indirect dependency that around 60 linked
packages use, including client-go/tools/{leaderelection,cache,record} and
client-go/transport. controller-runtime does not route it either -- its
only klog.SetLogger call is in its own test suite.

Plain klog.SetLogger, deliberately not SetLoggerWithOptions with
ContextualLogger(true). The contextual form is what klog's docs recommend
because it skips a klogr round trip, but it also disables klog's verbosity
gate and hands the decision to zap, whose level under Development is Debug
-- so client-go's V(1) chatter, suppressed today by klog's own -v=0, would
have started appearing. Plain SetLogger keeps klog's check in front and
still redirects what it does emit. Same fix to the format, no new volume,
which matters because volume is what prompted this.

The discovery API's http.Server set no ErrorLog, so net/http's own faults
(a failed Accept, a panic escaping a handler) went to the stdlib logger on
stderr. It now drains into logr through logr.ToSlogHandler, the pattern
internal/provider/gcp/wirelog.go already uses for the same purpose.

The deployment gets --zap-encoder=json and --zap-time-encoding=rfc3339.
The flag beats the in-code Development default -- addDefaults fills
NewEncoder only when it is nil -- and the time encoding is not optional,
since the JSON encoder defaults to epoch seconds. Verbosity stays at debug
on purpose; --zap-log-level=info drops the per-RPC lines.

zap.Options{Development: true} stays as the in-code default: flipping it
would change the format for every deployment of this operator, and the
flags get the same result here without that reach.

k8s.io/klog/v2 moves from indirect to direct in go.mod. It was already in
the module graph, so nothing new is downloaded.

No tests: three changes, all wiring, in a file with no test coverage, and
pinning klog.SetLogger means mutating process-global state that leaks into
every other test in the package. Verified in-cluster instead --
`kubectl logs | jq -e -s 'length > 0'` now passes, which fails if any
single line is not JSON.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-09-17 20:24:37 +00:00
kacerrandClaude Opus 5 82d01efb96 Plan routing klog into the structured logger (Step 6)
Found while watching the Step 5 validation, not a review finding: the
leader election lines ignore the zap encoder because they are not ours.
client-go calls klog.FromContext, controller-runtime hands the elector a
context with no logr in it, and klog falls back to its own stderr writer.
Nothing in the repo configures klog; it is an indirect dependency that
~60 linked packages use.

Plain klog.SetLogger rather than the contextual form the klog docs
recommend: the contextual option disables klog verbosity checking and
hands the decision to zap, which at Development level would let
client-go V(1) chatter through that klog suppresses today. Same fix to
the format, no new volume.

Also the one other in-process stderr bypass (the discovery http.Server
has no ErrorLog) and json/rfc3339 encoder flags on the deployment.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-09-17 20:05:02 +00:00
kacerrandClaude Opus 5 966807bf57 Plan the in-cluster kvm2 validation (Step 5)
First contact between the kvm2 provider and a real hypervisor: it has
never run against libvirt, so H1-H4 landed on code no host has executed.

The run happens in the existing kind cluster on purpose. That cluster
already owns two live kvm-home VMs on beelink, and orphan GC deletes any
instance whose UID is missing from its own cluster live set after
gc-min-age. A second operator carrying the kvm-home entry would therefore
destroy both within ten minutes; using the existing cluster makes its
live set the thing that protects them.

The plan also records what the run cannot prove: H2 error paths need a
body that fails part way through, and a healthy download never finishes
short, so the unverified libvirtd short-stream behaviour stays open.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-09-17 19:23:44 +00:00
kacerrandClaude Opus 5 c10f98a352 kvm2: name the cached base image after the provider entry (H4)
baseVolumeName was base-<key>.<format>, byte-identical to what the kvm
provider builds in the same pool -- for URL images too, since kvm takes
hex(sum[:8]) and kvm2 takes hex(sum[:])[:16], the same eight bytes -- with
a completely separate lock table. The shipped config has a live kvm-home
entry and a commented-out kvm2-home entry on the same host, the same key
debian-13 and the same default pool, and the comment invites the operator
to enable it. Both integration tests use key test-image. On beelink both
live overlays back onto kvm's base-debian-13.qcow2.

Since cachedImage accepts any 64-hex marker when no sha256 is configured
and never re-hashes the base, that meant two entries pointing one key at
different URLs silently shared a file; a concurrent cold fetch let one
entry delete the other's half-uploaded base and boot a guest from the
remains; and a sha256 edit deleted the base in place under live overlays,
which libvirt permits without complaint.

The name now carries the provider entry: base-<entry>-<key>.<format>.
imageSource gained an Entry field and resolveImage takes the provider
name, so every production call site follows from imageSource.volumeName().
That closes the cross-entry sharing outright, kvm<->kvm2 and kvm2<->kvm2
alike, and turns the per-Provider imageLocks from a latent bug into a
correct one: with names unique per entry, two entries cannot contend for a
file, so no process-wide lock table is needed.

The entry name now reaches a file name, so New checks it against the
existing kvmIdentifier rule, exported as provider.IsKVMIdentifier rather
than duplicated. Provider names are otherwise validated for all types at
once and only for being non-empty and unique, so a name with a slash would
have produced a volume nothing could address.

One companion fix, because the naming alone does not stop it:
cachedImage's digest-mismatch branch deleted the marker and the base and
refetched. It now returns ErrPermanent naming both digests and telling the
operator what to do. That branch was the remaining path that unlinks a
good file under running domains, and the contract already said "add a new
key instead" -- the code was contradicting it. The no-marker and
non-hex-marker branches keep deleting, since there the base is debris.

Not fixed, and said so in the docs rather than left to look closed:
editing images.<key>.url is still a silent no-op, since nothing records
the URL; and cross-process fencing still needs the deferred atomic publish
(upload to a temp name, verify, publish), which lands where H2 put the
discard.

Migration is a non-event. The base-debian-13.qcow2 on beelink is kvm's,
kvm keeps using it untouched, and kvm2 downloads its own copy once per
host. Nothing is replaced under a running guest; the pool needs room for
both, which the prerequisites now say.

The eviction recipe in the guide named base-debian-13.qcow2, which after
this change is kvm's file -- following it would have deleted the wrong
provider's base. It now greps for the prefix and deletes the marker first.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-09-17 17:57:02 +00:00
kacerrandClaude Opus 5 9682737c69 kvm2: an unreadable address is not a missing one (H3)
address() tried the guest agent then the host's ARP table and skipped any
error that was not a context error, returning ("", nil) after both. An
active domain with no IP maps to StateProvisioning; the controller clears
status.IP on that, and the health engine prunes the proxy's counters and
calls ForgetProxy in one tick once EffectiveHost() goes empty.

The damaging trigger was not the guest. The libvirt connection is pooled
per host, so when another reconcile's cancel or budget overrun killed it,
call returned "connection is closed" -- carrying no context sentinel, so
the check did not recognise it -- and Get returned a *successful*
instance in Provisioning with an empty IP. An unrelated proxy's timeout
dropped a healthy proxy out of rotation.

The discriminator was already in the spec without the conclusion being
drawn: during a normal boot on beelink exactly one source fails. The
agent errors because it is not up yet, while ARP answers without error
and with nothing, since the host never sees the guest's traffic. So one
source failing stays "no address yet"; every source failing, or a
connection already dead, is ErrTransient.

The explicit provider.ErrTransient wrap is load-bearing, not decoration.
Without it classify picks the first libvirt code out of the joined error,
and ErrInvalidArg, ErrAccessDenied and ErrNoSupport are Permanent -- an
address read failing would latch a running proxy as Failed, a worse
version of the harm being fixed.

Half of the finding is deliberately still open, and it is the half that
fires more often: a guest agent restarting inside a running VM still
demotes it, because that is indistinguishable from a VM still booting.
The discriminator is the previous IP, which the provider does not hold by
design and the controller already has. Follow-up finding to file against
proxy_controller.go:208-211 -- on StateProvisioning, keep an existing IP
and Provisioned condition for a few polls instead of clearing on the
first. The kvm2-local alternative (use the mark's createdAt as a
provisioning window) was rejected: createdAt is creation, not boot, and
it invents a deadline that belongs to the controller.

TestGet_addressErrorIsProvisioning failed against the new rule, correctly
-- it fails both sources with ErrAgentUnresponsive, and code 86 is
agent-specific, so the ARP source cannot produce it. Re-pointed as
TestGet_agentFailsAndARPEmptyIsProvisioning, the beelink boot it was
reaching for, which nothing covered before.
TestGet_everyAddressSourceFailingIsTransient covers the other side, both
sources erroring and the pooled link dying under the read; against the
unfixed code both return a Provisioning instance instead of an error.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-09-17 17:48:57 +00:00
kacerrandClaude Opus 5 62561c21fe kvm2: stop handing go-libvirt a reader that can error (H2)
uploadImage streamed the HTTP body straight into StorageVolUpload. When
that body failed part way through, go-libvirt deadlocked: requestStream
starts its stream sender on two unbuffered channels and, on the error
path, returns without ever reading outErr, so SendStream emits a
StatusError packet and parks on `outErr <-` forever while the main
goroutine parks on `abort <- true` forever. StorageVolUpload never
returned. The 20-minute transfer budget rescued the caller only, and the
socket kill did nothing, because both ends are blocked on channel sends
rather than on I/O -- which is what the comment at backend.go:152 got
wrong. Each failed upload leaked kvm2's call goroutine, go-libvirt's
sender goroutine and a 4 MiB buffer, permanently, against a 128Mi pod
limit and a reconciler that retries on backoff.

The defect is upstream and stays there: no fork, no replace. Instead
eofReader, at the go-libvirt seam so it covers the seed and marker
uploads too, converts any source error to io.EOF -- the one outcome that
ends a stream the normal way -- and keeps the real error for the caller.
countingReader sits underneath untouched, so the existing length and
digest checks report exactly what they did before.

The download also gets its own deadline a minute inside the transfer
budget. fetchImage already bounded the whole fetch at 20 minutes, so the
body's death and the budget's socket kill were racing and the socket
usually won; with the body a minute ahead, a stalled mirror ends through
its reader and unwinds cleanly instead of leaking.

discardVolume moves from uploadImage into fetchImage's upload-error
branch, so the dedicated connection is closed and libvirtd has let go of
the stream's fd before the file is unlinked. Moving it rather than adding
a second Close keeps image_test's closeCount() == 1 true. That discard is
still unfenced against a second provider entry sharing the volume name;
H4 closes that by making the name per-entry.

What remains: a failure originating in libvirtd mid-upload still leaks,
since SendStream has already returned by then. Confirmed unfixable from
here. Known gaps quotes the two-character upstream fix
(make(chan bool, 1) / make(chan error, 1)) and the replace escape hatch.

Unverified at ~85%: that libvirtd answers a short-but-clean stream finish
with OK rather than an error. virFDStreamWrite enforces length only on
writes past it and nothing compares offset to length on the close path,
which is also why virsh vol-upload of a small file into a large volume
works. If it errors we are back to a deadlock by a different door, which
is exactly today's behaviour, so this cannot be worse. The execution file
carries a probe recipe to settle it.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-09-17 17:46:15 +00:00
kacerrandClaude Opus 5 d826114fd3 Plan the kvm2 H2, H3 and H4 fixes
Extends the campaign plan with the three remaining high findings, each its
own step and commit, in the order H2 -> H3 -> H4 (H2 and H4 both touch
image.go, and H2 restructures the failure path H4 would otherwise be
written against).

H2 is a workaround, not a repair: the deadlock is upstream in go-libvirt,
whose requestStream never drains an unbuffered outErr on its error path.
Per the decision taken while planning, no fork and no replace directive;
we stop handing it a reader that can error, move the transfer deadline
onto the HTTP source so a stalled mirror unwinds cleanly, and document the
residual with the two-character upstream patch.

H4 names the base image per provider entry, which is what was chosen over
a content fingerprint. That fixes cross-entry sharing but not the silent
no-op on a url edit, so the plan says so rather than implying the finding
is closed.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-09-17 17:42:34 +00:00
kacerrandClaude Opus 5 0239fb7e69 kvm2: do not let a stalled write delay the connection teardown (H1)
sshSocket.Close was errors.Join(s.Conn.Close(), s.client.Close()). Go
evaluates those left to right, so the blocking close ran before the
unblocking one. s.Conn is the forwarded libvirt socket, and closing an SSH
channel writes a close message under ch.writeMu — which go-libvirt already
holds while streaming a base image through that same channel. Against a
host that vanished without RST or FIN, that write blocks until the kernel
stops retransmitting (tcp_retries2, ~15 min), and s.client.Close(), a bare
net.Conn.Close() that takes no lock and unblocks the stalled writer through
netpoll, was never reached. Close the client only; nothing wants the polite
channel-close message, and reordering instead would leave an error to
filter in a memoised closeErr.

link.markDead compounded it by running kill() before close(l.closed), both
inside a sync.Once. Reordering alone was not enough: sync.Once.Do blocks
concurrent callers until the first body returns, so every other markDead —
including the one in call is own ctx-cancel branch — queued behind the
stalled kill, and cancelling the reconcile did not help. The once field is
gone; the mutex the struct already had guards the close-exactly-once
invariant, reason is set before the signal, and kill runs outside the lock.

Blast radius was the whole operator, not one host: MaxConcurrentReconciles
is 3 and ensureImage holds a per-image lock whose acquire is ctx-aware but
reconcile contexts carry no deadline, so three Proxies on one host with an
uncached image parked every worker.

The same edits close a second route to the same pin: go-libvirt is
Disconnect closes the net.Conn it was handed (libvirt.go:331 ->
socket.go:164), and that conn is our sshSocket, so libvirtBackend.Close is
Disconnect goroutine entered closeOnce and hung; the 5 s closeBudget
released Close, which called markDead -> kill -> sshSocket.Close and
blocked in the other goroutine is sync.Once.

Verified against the pinned x/crypto v0.54.0 rather than assumed: the
flow-control park (remoteWin.reserve, channel.go:268) is outside writeMu
and only writePacket holds it across the TCP write, so a writer parked in
SSH flow control never blocks the close and a writer inside the TCP write
always does. To reproduce by hand, shrink SO_SNDBUF well under the 2 MiB
channel window (channelWindowSize = 64 * channelMaxPacket) so the socket
buffer fills before the window does.

The kvm provider does not have this bug: it closes client only and never
touches the forwarded channel.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-09-17 15:30:41 +00:00
kacerrandClaude Opus 5 85ea319cc2 Plan the kvm2 code-review fixes
The review of feat/kvm2-provider confirmed 52 findings (4 high, 20 medium,
28 low); review/REPORT.md has them. This is the campaign plan: one step and
one commit per finding, highest severity first, with only the step in hand
detailed so the plan is not a fiction about work not yet designed. Same
shape as the kvm campaign in 2026-09-10-2222-kvm-review-fixes.md.

Step 1 (H1) is detailed: sshSocket.Close runs the blocking close before the
unblocking one, so a transfer stuck in a TCP write pins the teardown for as
long as the kernel retransmits, and markDead compounds it by killing inside
a sync.Once before signalling dead.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-09-17 15:27:04 +00:00
kacerrandClaude Fable 5.1 2510842396 kvm2: apply the milestone review
Shorter start wait with immediate verdicts for dead or paused domains,
ListByTag without state or address RPCs, Timeout for an expired check
round, no adoption of unmarked domains, reference-counted keyed locks,
dead options removed, HEAD fallback on any non-2xx.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
2026-09-16 21:19:50 +00:00
kacerrandClaude Fable 5.1 fcbf8ddc91 kvm2: user guide, appendix column and section, architecture and README entries
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
2026-09-16 19:54:25 +00:00
kacerrandClaude Fable 5.1 fb6f59090b kvm2: integration test, sample Proxy, providers-config entries and Makefile target
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
2026-09-16 19:40:50 +00:00
kacerrandClaude Fable 5.1 ce62fe57f6 kvm2: health checker on dedicated connections; kvm2 in the ProviderHealth type comment
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
2026-09-16 19:26:51 +00:00
kacerrandClaude Fable 5.1 794a2aef3d kvm2: Create, Get, Delete and ListByTag with the contract tests
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
2026-09-16 19:23:49 +00:00
kacerrandClaude Fable 5.1 6b3c985b4a kvm2: base image cache with a streamed upload and the pool-space precheck
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
2026-09-16 18:19:05 +00:00
kacerrandClaude Fable 5.1 b45f83327a kvm2: SSH transport, ctx-first libvirt seam, host table and span decorator
Every go-libvirt call runs under its own budget and a cancelled or overlong
call kills the connection; credentials are read on every dial; one span and
one log line per RPC come from a decorator applied where connections are
made.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
2026-09-16 17:43:27 +00:00
kacerrandClaude Fable 5.1 1f10a51f82 kvm2: domain and volume XML, metadata element and NoCloud seed builders
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
2026-09-16 16:57:30 +00:00
kacerrandClaude Fable 5.1 95768281be kvm2: config block, constructor registration, IDs, state mapping and classify
The kvm2 block reuses KVMConfig; validateKVM gains a block-name variant so
messages say kvm2. The package skeleton has a non-dialing New, strict
providerID parsing, the state table and a 34-row classify test.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
2026-09-16 16:38:48 +00:00
kacerrandClaude Fable 5.1 4bd9aae03b Write the kvm2 provider spec from the template
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
2026-09-16 13:38:29 +00:00
kacerrandClaude Fable 5.1 854856cf0f Plan the clean-room kvm2 provider built from the provider contract
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
2026-09-16 13:33:56 +00:00
kacerr 0dc25a1637 Merge pull request 'Provider docs: generic contract, spec template and appendix on the existing providers' (#7) from feat/provider-blueprint into main
Reviewed-on: #7
2026-09-16 13:48:40 +02:00
kacerrandClaude Fable 5.1 625cc7cd64 Log the provider docs split in the changelog
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
2026-09-16 11:47:24 +00:00
kacerrandClaude Fable 5.1 c6dda03e51 Record the verification sweep for the provider docs split
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
2026-09-16 11:37:53 +00:00
kacerrandClaude Fable 5.1 86b8d21be3 Turn docs/providers.md into an index and repoint its inbound links
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
2026-09-16 11:36:49 +00:00
kacerrandClaude Fable 5.1 2d959af313 Add the appendix on the existing providers
Profile table keyed on the contract's axis codes, per-provider reference
sections for kubernetes, gcp and kvm, shared gaps and proposed features.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
2026-09-16 11:36:03 +00:00
kacerrandClaude Fable 5.1 d61868fe2d Add the provider spec template
Twenty-four fill-in entries, one per decision the contract leaves to the
provider, numbered to match the contract's profile table.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
2026-09-16 11:28:52 +00:00
kacerrandClaude Fable 5.1 8d764de336 Write the generic provider contract
Fully abstract: MUST/SHOULD/MAY rules with the caller behind each one,
19 behaviour-profile axes with stable codes, credential hygiene, testing
obligations, the checklist and the contract's known limits.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
2026-09-16 11:17:56 +00:00
kacerrandClaude Fable 5.1 cf89ac4c91 Plan the split of the provider blueprint into contract, appendix, template and index
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
2026-09-16 11:13:01 +00:00
kacerrandClaude Opus 5 0bb699bf09 Fill the provider blueprint's gaps after a full sweep
A pass over the API types, spec hash, health and discovery call sites,
manager manifests, samples, README, provider docs, plans and the PR review
found rules the blueprint did not state: the IP must be reachable on the
proxy port by the health engine and clients, providers must be safe for
concurrent use, the manager's read-only/128Mi runtime envelope, catalogue
edits bypass the spec hash, Delete is re-called while in progress, and
provider names must be DNS-1123. Adds matrix rows, a deployment checklist
item, and the still-open review findings and broken samples as gaps.

README's "Adding a provider" restated an outdated contract; it now points
to docs/providers.md, and the providers table gains the kvm row.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-09-16 09:18:40 +00:00
kacerrandClaude Opus 5 01c27b5f35 Cover tracing in the provider blueprint
The blueprint reduced tracing to "don't open per-operation spans", which
reads as "nothing to do". Add an observability section: the operation
spans the decorators give for free, the child spans per backend call a
provider should add (parent-gated transport or a span decorator over the
SDK seam), the ctx/logger rules that keep traceID on log lines, and the
status per provider: kubernetes and gcp have child spans, kvm has none.
Matrix rows, the checklist item, gaps and a kvm proposal follow suit.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-09-16 09:14:06 +00:00
kacerrandClaude Opus 5 e08f23bcb3 Correct the provider blueprint after verification
A re-check against the code found four mismatches: the kvm connection is
opened at startup by the leader's first health check, so that isn't doc
drift; cmd/main.go, not the monitor, type-asserts Checker; QuotaExceeded
and PermanentError are reasons on Provisioned, not conditions; and the
latch line range. The checklist also now matches how the existing
providers do interface assertions and RBAC markers.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-09-16 08:57:52 +00:00
kacerrandClaude Opus 5 d96e56f1a0 Write the provider blueprint
docs/providers.md collects the provider contract that was spread across
comments, callers and per-provider docs: MUST/SHOULD/MAY rules per method,
each tied to the reconciler or GC behaviour it protects, the error taxonomy,
a feature matrix for kubernetes/gcp/kvm with file:line notes, a checklist for
adding a provider, and the known gaps. Linked from architecture.md and
CLAUDE.md.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-09-16 08:54:40 +00:00
kacerr cacd9583a3 Merge pull request 'Provider health checks: ProviderHealth CRD, per-provider checks and gauges' (#6) from feat/provider-health into main
Reviewed-on: #6
2026-09-16 10:46:47 +02:00
kacerrandClaude Opus 5 e6ed46fd7c Record the provider health validation and add the changelog entry
The user ran the real-hypervisor checks against beelink and confirmed they
passed, which completes Step 7 of the provider health plan.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-09-16 08:45:58 +00:00
kacerrandClaude Opus 5 80b2e9c85e Plan the provider blueprint document
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-09-16 08:41:42 +00:00
kacerrandClaude Opus 5 4e23347760 Point the KVM integration tests at a dedicated known_hosts
Leaving KVM_TEST_KNOWN_HOSTS unset silently disables host-key
verification for the run, so document it as part of the normal setup
rather than as an optional extra, and suggest a kvm_known_hosts holding
only the hypervisors.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-09-12 05:59:22 +00:00
kacerrandClaude Opus 5 4e5b5d62df Probe hosts for health instead of risking the shared connection
The health check shares each host's libvirt connection with the reconciles,
so it must not be able to take that connection away from them.

- A timed-out check no longer drops the link. The check goes through the new
  hostPool.probe, which abandons its RPCs when the context ends and leaves
  the connection alone, so an image upload or a Create running on the same
  link is not cut off. A link that really died is closed by the keepalive.
  One abandoned probe per host may stay waiting; until it returns, the next
  check reports Timeout at once (errProbeBusy).
- Waiting for a host's connection slot now ends with the caller's context.
  hostEntry used a mutex, so a check could wait out a reconcile redialing a
  dead host — minutes — past its own timeout. The slot is a one-slot channel
  and acquire bounds only the wait, never the uncontended take.
- --provider-check-timeout defaults to 0 and derives min(15s, interval).
  The flat 15s default contradicted the validation that the timeout may not
  exceed the interval, so --provider-check-interval=12s alone failed to
  start.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-09-12 05:04:45 +00:00
kacerrandClaude Opus 5 0e456affdd Record the automated validation of the provider health checks
make test (envtest, -race) and the kind e2e suite pass; the e2e
ProviderHealth for the kubernetes provider turned Ready within a second
of the manager starting and its gauge appeared in the scrape. make lint
is blocked by the local go1.27 toolchain crashing staticcheck on the
standard library, unrelated to this branch.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-09-11 20:31:15 +00:00
kacerrandClaude Opus 5 19b4a82ae9 Document the provider health checks
Architecture gets a provider-health section, component rows and four
decisions; the KVM provider doc gets a health-check section with the
reason table and the rotated-Secret gap; the testing guide lists the new
suites. CLAUDE.md's Architecture TODO and stale "only go.mod exists"
gotcha now point at the real docs, plus the RBAC-marker gotcha found
while wiring the monitor.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-09-11 20:28:09 +00:00
kacerrandClaude Opus 5 9a3ee963a0 Wire the provider health monitor into the manager, with gauges
--provider-check-interval (1m; 0 = off, else at least 10s) and
--provider-check-timeout (15s) are validated before the providers config
loads. Checkers are collected from the unwrapped providers in config
order, so checks never count as provider requests. Three scrape-time
gauges per {provider,target} read the monitor's snapshot: up, check
duration and check timestamp. The e2e suite waits for the kubernetes
ProviderHealth to turn Ready and asserts its gauge in the scrape.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-09-11 20:23:55 +00:00
kacerrandClaude Opus 5 88a15ecdb5 Add the provider health Monitor
A leader-only runnable checks every configured provider concurrently
under one timeout per round (first round at start, then every interval),
keeps the latest results for the metrics collector, logs targets going
down, changing reason or coming back, and publishes one ProviderHealth
object per provider: created with a managed-by label when missing, then
a status merge patch. Per-target lastTransitionTime survives rounds in
which ready does not flip. Objects of providers that are no longer
configured are deleted once at start; with checks turned off, all are.
Reads go through the API reader, so no cluster-wide informer is needed.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-09-11 20:20:52 +00:00
kacerrandClaude Opus 5 04dd2c0284 Add the KVM health check: reachability and storage pool per host
Each configured host is checked concurrently over the pooled SSH+libvirt
connection: libvirt version and hostname prove the daemon answers, then
the configured pool must exist and be running, with its free space
reported. Dial failures are now tagged with their stage (SSH material,
TCP, SSH handshake, libvirt connect) by a dialError that keeps the
message and unwraps to the cause, so classification is unchanged and the
check can say why a host is unreachable. A pool verdict is decided after
the call returns so it never drops a working connection.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-09-11 20:17:35 +00:00
kacerrandClaude Opus 5 1962365f44 Add the provider Checker contract and the kubernetes and GCP checks
Checker is an optional interface next to Provider: Check returns one
verdict per target and never fails as a whole. The kubernetes provider
lists at most one managed Pod cluster-wide (the verb and scope GC relies
on); the GCP provider fetches a single page of at most one managed
instance through a new Probe seam method, since the SDK treats
MaxResults as a page size and the existing AggregatedList drains every
page.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-09-11 20:08:21 +00:00
kacerrandClaude Opus 5 1c773c24ad Add the cluster-scoped ProviderHealth CRD
One operator-owned object per configured provider will carry the
periodic check result: a Ready condition, per-target verdicts with their
own transition times, and a ready/total count for kubectl get. There is
no spec; only a viewer role is shipped since the operator is the sole
writer. Written by hand (kubebuilder is not installed), registered in
the existing AddKnownTypes call rather than a second init().

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-09-11 18:30:31 +00:00
kacerrandClaude Opus 5 18a3378507 Add plan: provider health checks
Periodic, leader-only checks of every configured provider (KVM per host
over the pooled SSH+libvirt connection including the storage pool,
kubernetes apiserver/RBAC, GCP credentials/API), surfaced through a new
cluster-scoped ProviderHealth CRD, Prometheus gauges and transition logs.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-09-11 18:08:59 +00:00
kacerr eb8f075f7a Merge pull request 'KVM/libvirt provider: proxies as VMs on your own hypervisors' (#5) from feat/kvm-provider into main
Reviewed-on: #5
2026-09-11 17:02:33 +02:00
kacerrandClaude Opus 5 43d429aef7 Point the manager config and sample at the beelink KVM host
The default manager config gains a kvm-home provider reaching beelink
over Tailscale, the provider sample names the same host, and the
manager kustomization uses the locally built kvm-dev image.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-09-11 06:26:44 +00:00