kacerr/egress-proxies-operator
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

Compare commits

merge into: kacerr/egress-proxies-operator:feat/kvm-provider
Branches Tags
kacerr/egress-proxies-operator:feat/proxy-operator kacerr/egress-proxies-operator:main kacerr/egress-proxies-operator:feat/proxmox-provider kacerr/egress-proxies-operator:feat/kvm2-provider kacerr/egress-proxies-operator:feat/provider-blueprint kacerr/egress-proxies-operator:feat/provider-health kacerr/egress-proxies-operator:feat/kvm-provider kacerr/egress-proxies-operator:feat/otel-tracing kacerr/egress-proxies-operator:feat/demo-scripts kacerr/egress-proxies-operator:feat/gitea-build-workflow
kacerr/egress-proxies-operator:0.01
...
pull from: kacerr/egress-proxies-operator:feat/kvm2-provider
Branches Tags
kacerr/egress-proxies-operator:main kacerr/egress-proxies-operator:feat/proxmox-provider kacerr/egress-proxies-operator:feat/kvm2-provider kacerr/egress-proxies-operator:feat/provider-blueprint kacerr/egress-proxies-operator:feat/provider-health kacerr/egress-proxies-operator:feat/kvm-provider kacerr/egress-proxies-operator:feat/proxy-operator kacerr/egress-proxies-operator:feat/otel-tracing kacerr/egress-proxies-operator:feat/demo-scripts kacerr/egress-proxies-operator:feat/gitea-build-workflow
kacerr/egress-proxies-operator:0.01
48 Commits
feat/kvm-provider ... feat/kvm2-provider
Author SHA1 Message Date
kacerrandClaude Opus 5 54525bf67a kvm2: gofmt volume.go 
Making baseVolumeName multi-line in the H4 commit collapsed the
alignment gofmt gives a run of single-line funcs, and the commit went out
with the old spacing. Caught by make test, whose fmt target rewrites the
file and leaves the tree dirty.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
 2026-09-17 21:07:10 +00:00
kacerrandClaude Opus 5 41fbd5e7e6 kvm2: record the review validation, enable the provider entry, changelog 
Closes out the review-fix campaign. The kvm2 provider was deployed
in-cluster and validated against beelink -- the first time any of this
code had run against real libvirt rather than the in-memory fake.

config/manager/providers_config.yaml ships the kvm2-home entry enabled,
since the operator is now running it for real.

review/ carries the code review the four H-commits reference, so H1..H4
resolve to something rather than dangling: scope, the seven-chunk sweep,
the escalations, the ten deep dives and the final report. .claude/agents
carries the two reviewer definitions that produced them.

What the validation did not prove is recorded next to what it did: H2's
error paths need a body that fails part way through, and a healthy 433 MB
download never finishes short, so the unverified libvirtd short-stream
behaviour is still open. The kvm2 integration test has also still never
been run.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
 2026-09-17 20:50:45 +00:00
kacerrandClaude Opus 5 4114d43e87 Route klog and net/http's own errors into the structured logger 
Some lines ignored the zap encoder entirely, so `kubectl logs | jq` could
never work:

  I0917 19:50:02.580097  1 leaderelection.go:258] "Attempting to acquire
  leader lease..." lock="egress-proxies-operator-system/b47711d1..."

That is klog's own stderr writer, not ours. client-go's leader elector
calls klog.FromContext(ctx), and controller-runtime hands the elector a
context with no logr in it -- mgr.Start receives ctrl.SetupSignalHandler(),
which carries no logger. FromContext falls back to Background(), and with
no logger registered that is klog's text sink. Nothing here imported or
configured klog; it was an indirect dependency that around 60 linked
packages use, including client-go/tools/{leaderelection,cache,record} and
client-go/transport. controller-runtime does not route it either -- its
only klog.SetLogger call is in its own test suite.

Plain klog.SetLogger, deliberately not SetLoggerWithOptions with
ContextualLogger(true). The contextual form is what klog's docs recommend
because it skips a klogr round trip, but it also disables klog's verbosity
gate and hands the decision to zap, whose level under Development is Debug
-- so client-go's V(1) chatter, suppressed today by klog's own -v=0, would
have started appearing. Plain SetLogger keeps klog's check in front and
still redirects what it does emit. Same fix to the format, no new volume,
which matters because volume is what prompted this.

The discovery API's http.Server set no ErrorLog, so net/http's own faults
(a failed Accept, a panic escaping a handler) went to the stdlib logger on
stderr. It now drains into logr through logr.ToSlogHandler, the pattern
internal/provider/gcp/wirelog.go already uses for the same purpose.

The deployment gets --zap-encoder=json and --zap-time-encoding=rfc3339.
The flag beats the in-code Development default -- addDefaults fills
NewEncoder only when it is nil -- and the time encoding is not optional,
since the JSON encoder defaults to epoch seconds. Verbosity stays at debug
on purpose; --zap-log-level=info drops the per-RPC lines.

zap.Options{Development: true} stays as the in-code default: flipping it
would change the format for every deployment of this operator, and the
flags get the same result here without that reach.

k8s.io/klog/v2 moves from indirect to direct in go.mod. It was already in
the module graph, so nothing new is downloaded.

No tests: three changes, all wiring, in a file with no test coverage, and
pinning klog.SetLogger means mutating process-global state that leaks into
every other test in the package. Verified in-cluster instead --
`kubectl logs | jq -e -s 'length > 0'` now passes, which fails if any
single line is not JSON.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
 2026-09-17 20:24:37 +00:00
kacerrandClaude Opus 5 82d01efb96 Plan routing klog into the structured logger (Step 6) 
Found while watching the Step 5 validation, not a review finding: the
leader election lines ignore the zap encoder because they are not ours.
client-go calls klog.FromContext, controller-runtime hands the elector a
context with no logr in it, and klog falls back to its own stderr writer.
Nothing in the repo configures klog; it is an indirect dependency that
~60 linked packages use.

Plain klog.SetLogger rather than the contextual form the klog docs
recommend: the contextual option disables klog verbosity checking and
hands the decision to zap, which at Development level would let
client-go V(1) chatter through that klog suppresses today. Same fix to
the format, no new volume.

Also the one other in-process stderr bypass (the discovery http.Server
has no ErrorLog) and json/rfc3339 encoder flags on the deployment.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
 2026-09-17 20:05:02 +00:00
kacerrandClaude Opus 5 966807bf57 Plan the in-cluster kvm2 validation (Step 5) 
First contact between the kvm2 provider and a real hypervisor: it has
never run against libvirt, so H1-H4 landed on code no host has executed.

The run happens in the existing kind cluster on purpose. That cluster
already owns two live kvm-home VMs on beelink, and orphan GC deletes any
instance whose UID is missing from its own cluster live set after
gc-min-age. A second operator carrying the kvm-home entry would therefore
destroy both within ten minutes; using the existing cluster makes its
live set the thing that protects them.

The plan also records what the run cannot prove: H2 error paths need a
body that fails part way through, and a healthy download never finishes
short, so the unverified libvirtd short-stream behaviour stays open.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
 2026-09-17 19:23:44 +00:00
kacerrandClaude Opus 5 c10f98a352 kvm2: name the cached base image after the provider entry (H4) 
baseVolumeName was base-<key>.<format>, byte-identical to what the kvm
provider builds in the same pool -- for URL images too, since kvm takes
hex(sum[:8]) and kvm2 takes hex(sum[:])[:16], the same eight bytes -- with
a completely separate lock table. The shipped config has a live kvm-home
entry and a commented-out kvm2-home entry on the same host, the same key
debian-13 and the same default pool, and the comment invites the operator
to enable it. Both integration tests use key test-image. On beelink both
live overlays back onto kvm's base-debian-13.qcow2.

Since cachedImage accepts any 64-hex marker when no sha256 is configured
and never re-hashes the base, that meant two entries pointing one key at
different URLs silently shared a file; a concurrent cold fetch let one
entry delete the other's half-uploaded base and boot a guest from the
remains; and a sha256 edit deleted the base in place under live overlays,
which libvirt permits without complaint.

The name now carries the provider entry: base-<entry>-<key>.<format>.
imageSource gained an Entry field and resolveImage takes the provider
name, so every production call site follows from imageSource.volumeName().
That closes the cross-entry sharing outright, kvm<->kvm2 and kvm2<->kvm2
alike, and turns the per-Provider imageLocks from a latent bug into a
correct one: with names unique per entry, two entries cannot contend for a
file, so no process-wide lock table is needed.

The entry name now reaches a file name, so New checks it against the
existing kvmIdentifier rule, exported as provider.IsKVMIdentifier rather
than duplicated. Provider names are otherwise validated for all types at
once and only for being non-empty and unique, so a name with a slash would
have produced a volume nothing could address.

One companion fix, because the naming alone does not stop it:
cachedImage's digest-mismatch branch deleted the marker and the base and
refetched. It now returns ErrPermanent naming both digests and telling the
operator what to do. That branch was the remaining path that unlinks a
good file under running domains, and the contract already said "add a new
key instead" -- the code was contradicting it. The no-marker and
non-hex-marker branches keep deleting, since there the base is debris.

Not fixed, and said so in the docs rather than left to look closed:
editing images.<key>.url is still a silent no-op, since nothing records
the URL; and cross-process fencing still needs the deferred atomic publish
(upload to a temp name, verify, publish), which lands where H2 put the
discard.

Migration is a non-event. The base-debian-13.qcow2 on beelink is kvm's,
kvm keeps using it untouched, and kvm2 downloads its own copy once per
host. Nothing is replaced under a running guest; the pool needs room for
both, which the prerequisites now say.

The eviction recipe in the guide named base-debian-13.qcow2, which after
this change is kvm's file -- following it would have deleted the wrong
provider's base. It now greps for the prefix and deletes the marker first.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
 2026-09-17 17:57:02 +00:00
kacerrandClaude Opus 5 9682737c69 kvm2: an unreadable address is not a missing one (H3) 
address() tried the guest agent then the host's ARP table and skipped any
error that was not a context error, returning ("", nil) after both. An
active domain with no IP maps to StateProvisioning; the controller clears
status.IP on that, and the health engine prunes the proxy's counters and
calls ForgetProxy in one tick once EffectiveHost() goes empty.

The damaging trigger was not the guest. The libvirt connection is pooled
per host, so when another reconcile's cancel or budget overrun killed it,
call returned "connection is closed" -- carrying no context sentinel, so
the check did not recognise it -- and Get returned a *successful*
instance in Provisioning with an empty IP. An unrelated proxy's timeout
dropped a healthy proxy out of rotation.

The discriminator was already in the spec without the conclusion being
drawn: during a normal boot on beelink exactly one source fails. The
agent errors because it is not up yet, while ARP answers without error
and with nothing, since the host never sees the guest's traffic. So one
source failing stays "no address yet"; every source failing, or a
connection already dead, is ErrTransient.

The explicit provider.ErrTransient wrap is load-bearing, not decoration.
Without it classify picks the first libvirt code out of the joined error,
and ErrInvalidArg, ErrAccessDenied and ErrNoSupport are Permanent -- an
address read failing would latch a running proxy as Failed, a worse
version of the harm being fixed.

Half of the finding is deliberately still open, and it is the half that
fires more often: a guest agent restarting inside a running VM still
demotes it, because that is indistinguishable from a VM still booting.
The discriminator is the previous IP, which the provider does not hold by
design and the controller already has. Follow-up finding to file against
proxy_controller.go:208-211 -- on StateProvisioning, keep an existing IP
and Provisioned condition for a few polls instead of clearing on the
first. The kvm2-local alternative (use the mark's createdAt as a
provisioning window) was rejected: createdAt is creation, not boot, and
it invents a deadline that belongs to the controller.

TestGet_addressErrorIsProvisioning failed against the new rule, correctly
-- it fails both sources with ErrAgentUnresponsive, and code 86 is
agent-specific, so the ARP source cannot produce it. Re-pointed as
TestGet_agentFailsAndARPEmptyIsProvisioning, the beelink boot it was
reaching for, which nothing covered before.
TestGet_everyAddressSourceFailingIsTransient covers the other side, both
sources erroring and the pooled link dying under the read; against the
unfixed code both return a Provisioning instance instead of an error.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
 2026-09-17 17:48:57 +00:00
kacerrandClaude Opus 5 62561c21fe kvm2: stop handing go-libvirt a reader that can error (H2) 
uploadImage streamed the HTTP body straight into StorageVolUpload. When
that body failed part way through, go-libvirt deadlocked: requestStream
starts its stream sender on two unbuffered channels and, on the error
path, returns without ever reading outErr, so SendStream emits a
StatusError packet and parks on `outErr <-` forever while the main
goroutine parks on `abort <- true` forever. StorageVolUpload never
returned. The 20-minute transfer budget rescued the caller only, and the
socket kill did nothing, because both ends are blocked on channel sends
rather than on I/O -- which is what the comment at backend.go:152 got
wrong. Each failed upload leaked kvm2's call goroutine, go-libvirt's
sender goroutine and a 4 MiB buffer, permanently, against a 128Mi pod
limit and a reconciler that retries on backoff.

The defect is upstream and stays there: no fork, no replace. Instead
eofReader, at the go-libvirt seam so it covers the seed and marker
uploads too, converts any source error to io.EOF -- the one outcome that
ends a stream the normal way -- and keeps the real error for the caller.
countingReader sits underneath untouched, so the existing length and
digest checks report exactly what they did before.

The download also gets its own deadline a minute inside the transfer
budget. fetchImage already bounded the whole fetch at 20 minutes, so the
body's death and the budget's socket kill were racing and the socket
usually won; with the body a minute ahead, a stalled mirror ends through
its reader and unwinds cleanly instead of leaking.

discardVolume moves from uploadImage into fetchImage's upload-error
branch, so the dedicated connection is closed and libvirtd has let go of
the stream's fd before the file is unlinked. Moving it rather than adding
a second Close keeps image_test's closeCount() == 1 true. That discard is
still unfenced against a second provider entry sharing the volume name;
H4 closes that by making the name per-entry.

What remains: a failure originating in libvirtd mid-upload still leaks,
since SendStream has already returned by then. Confirmed unfixable from
here. Known gaps quotes the two-character upstream fix
(make(chan bool, 1) / make(chan error, 1)) and the replace escape hatch.

Unverified at ~85%: that libvirtd answers a short-but-clean stream finish
with OK rather than an error. virFDStreamWrite enforces length only on
writes past it and nothing compares offset to length on the close path,
which is also why virsh vol-upload of a small file into a large volume
works. If it errors we are back to a deadlock by a different door, which
is exactly today's behaviour, so this cannot be worse. The execution file
carries a probe recipe to settle it.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
 2026-09-17 17:46:15 +00:00
kacerrandClaude Opus 5 d826114fd3 Plan the kvm2 H2, H3 and H4 fixes 
Extends the campaign plan with the three remaining high findings, each its
own step and commit, in the order H2 -> H3 -> H4 (H2 and H4 both touch
image.go, and H2 restructures the failure path H4 would otherwise be
written against).

H2 is a workaround, not a repair: the deadlock is upstream in go-libvirt,
whose requestStream never drains an unbuffered outErr on its error path.
Per the decision taken while planning, no fork and no replace directive;
we stop handing it a reader that can error, move the transfer deadline
onto the HTTP source so a stalled mirror unwinds cleanly, and document the
residual with the two-character upstream patch.

H4 names the base image per provider entry, which is what was chosen over
a content fingerprint. That fixes cross-entry sharing but not the silent
no-op on a url edit, so the plan says so rather than implying the finding
is closed.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
 2026-09-17 17:42:34 +00:00
kacerrandClaude Opus 5 0239fb7e69 kvm2: do not let a stalled write delay the connection teardown (H1) 
sshSocket.Close was errors.Join(s.Conn.Close(), s.client.Close()). Go
evaluates those left to right, so the blocking close ran before the
unblocking one. s.Conn is the forwarded libvirt socket, and closing an SSH
channel writes a close message under ch.writeMu — which go-libvirt already
holds while streaming a base image through that same channel. Against a
host that vanished without RST or FIN, that write blocks until the kernel
stops retransmitting (tcp_retries2, ~15 min), and s.client.Close(), a bare
net.Conn.Close() that takes no lock and unblocks the stalled writer through
netpoll, was never reached. Close the client only; nothing wants the polite
channel-close message, and reordering instead would leave an error to
filter in a memoised closeErr.

link.markDead compounded it by running kill() before close(l.closed), both
inside a sync.Once. Reordering alone was not enough: sync.Once.Do blocks
concurrent callers until the first body returns, so every other markDead —
including the one in call is own ctx-cancel branch — queued behind the
stalled kill, and cancelling the reconcile did not help. The once field is
gone; the mutex the struct already had guards the close-exactly-once
invariant, reason is set before the signal, and kill runs outside the lock.

Blast radius was the whole operator, not one host: MaxConcurrentReconciles
is 3 and ensureImage holds a per-image lock whose acquire is ctx-aware but
reconcile contexts carry no deadline, so three Proxies on one host with an
uncached image parked every worker.

The same edits close a second route to the same pin: go-libvirt is
Disconnect closes the net.Conn it was handed (libvirt.go:331 ->
socket.go:164), and that conn is our sshSocket, so libvirtBackend.Close is
Disconnect goroutine entered closeOnce and hung; the 5 s closeBudget
released Close, which called markDead -> kill -> sshSocket.Close and
blocked in the other goroutine is sync.Once.

Verified against the pinned x/crypto v0.54.0 rather than assumed: the
flow-control park (remoteWin.reserve, channel.go:268) is outside writeMu
and only writePacket holds it across the TCP write, so a writer parked in
SSH flow control never blocks the close and a writer inside the TCP write
always does. To reproduce by hand, shrink SO_SNDBUF well under the 2 MiB
channel window (channelWindowSize = 64 * channelMaxPacket) so the socket
buffer fills before the window does.

The kvm provider does not have this bug: it closes client only and never
touches the forwarded channel.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
 2026-09-17 15:30:41 +00:00
kacerrandClaude Opus 5 85ea319cc2 Plan the kvm2 code-review fixes 
The review of feat/kvm2-provider confirmed 52 findings (4 high, 20 medium,
28 low); review/REPORT.md has them. This is the campaign plan: one step and
one commit per finding, highest severity first, with only the step in hand
detailed so the plan is not a fiction about work not yet designed. Same
shape as the kvm campaign in 2026-09-10-2222-kvm-review-fixes.md.

Step 1 (H1) is detailed: sshSocket.Close runs the blocking close before the
unblocking one, so a transfer stuck in a TCP write pins the teardown for as
long as the kernel retransmits, and markDead compounds it by killing inside
a sync.Once before signalling dead.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
 2026-09-17 15:27:04 +00:00
kacerrandClaude Fable 5.1 2510842396 kvm2: apply the milestone review 
Shorter start wait with immediate verdicts for dead or paused domains,
ListByTag without state or address RPCs, Timeout for an expired check
round, no adoption of unmarked domains, reference-counted keyed locks,
dead options removed, HEAD fallback on any non-2xx.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
 2026-09-16 21:19:50 +00:00
kacerrandClaude Fable 5.1 fcbf8ddc91 kvm2: user guide, appendix column and section, architecture and README entries 
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
 2026-09-16 19:54:25 +00:00
kacerrandClaude Fable 5.1 fb6f59090b kvm2: integration test, sample Proxy, providers-config entries and Makefile target 
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
 2026-09-16 19:40:50 +00:00
kacerrandClaude Fable 5.1 ce62fe57f6 kvm2: health checker on dedicated connections; kvm2 in the ProviderHealth type comment 
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
 2026-09-16 19:26:51 +00:00
kacerrandClaude Fable 5.1 794a2aef3d kvm2: Create, Get, Delete and ListByTag with the contract tests 
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
 2026-09-16 19:23:49 +00:00
kacerrandClaude Fable 5.1 6b3c985b4a kvm2: base image cache with a streamed upload and the pool-space precheck 
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
 2026-09-16 18:19:05 +00:00
kacerrandClaude Fable 5.1 b45f83327a kvm2: SSH transport, ctx-first libvirt seam, host table and span decorator 
Every go-libvirt call runs under its own budget and a cancelled or overlong
call kills the connection; credentials are read on every dial; one span and
one log line per RPC come from a decorator applied where connections are
made.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
 2026-09-16 17:43:27 +00:00
kacerrandClaude Fable 5.1 1f10a51f82 kvm2: domain and volume XML, metadata element and NoCloud seed builders 
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
 2026-09-16 16:57:30 +00:00
kacerrandClaude Fable 5.1 95768281be kvm2: config block, constructor registration, IDs, state mapping and classify 
The kvm2 block reuses KVMConfig; validateKVM gains a block-name variant so
messages say kvm2. The package skeleton has a non-dialing New, strict
providerID parsing, the state table and a 34-row classify test.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
 2026-09-16 16:38:48 +00:00
kacerrandClaude Fable 5.1 4bd9aae03b Write the kvm2 provider spec from the template 
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
 2026-09-16 13:38:29 +00:00
kacerrandClaude Fable 5.1 854856cf0f Plan the clean-room kvm2 provider built from the provider contract 
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
 2026-09-16 13:33:56 +00:00
kacerr 0dc25a1637 Merge pull request 'Provider docs: generic contract, spec template and appendix on the existing providers' (#7) from feat/provider-blueprint into main 
Reviewed-on: #7
 2026-09-16 13:48:40 +02:00
kacerrandClaude Fable 5.1 625cc7cd64 Log the provider docs split in the changelog 
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
 2026-09-16 11:47:24 +00:00
kacerrandClaude Fable 5.1 c6dda03e51 Record the verification sweep for the provider docs split 
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
 2026-09-16 11:37:53 +00:00
kacerrandClaude Fable 5.1 86b8d21be3 Turn docs/providers.md into an index and repoint its inbound links 
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
 2026-09-16 11:36:49 +00:00
kacerrandClaude Fable 5.1 2d959af313 Add the appendix on the existing providers 
Profile table keyed on the contract's axis codes, per-provider reference
sections for kubernetes, gcp and kvm, shared gaps and proposed features.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
 2026-09-16 11:36:03 +00:00
kacerrandClaude Fable 5.1 d61868fe2d Add the provider spec template 
Twenty-four fill-in entries, one per decision the contract leaves to the
provider, numbered to match the contract's profile table.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
 2026-09-16 11:28:52 +00:00
kacerrandClaude Fable 5.1 8d764de336 Write the generic provider contract 
Fully abstract: MUST/SHOULD/MAY rules with the caller behind each one,
19 behaviour-profile axes with stable codes, credential hygiene, testing
obligations, the checklist and the contract's known limits.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
 2026-09-16 11:17:56 +00:00
kacerrandClaude Fable 5.1 cf89ac4c91 Plan the split of the provider blueprint into contract, appendix, template and index 
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
 2026-09-16 11:13:01 +00:00
kacerrandClaude Opus 5 0bb699bf09 Fill the provider blueprint's gaps after a full sweep 
A pass over the API types, spec hash, health and discovery call sites,
manager manifests, samples, README, provider docs, plans and the PR review
found rules the blueprint did not state: the IP must be reachable on the
proxy port by the health engine and clients, providers must be safe for
concurrent use, the manager's read-only/128Mi runtime envelope, catalogue
edits bypass the spec hash, Delete is re-called while in progress, and
provider names must be DNS-1123. Adds matrix rows, a deployment checklist
item, and the still-open review findings and broken samples as gaps.

README's "Adding a provider" restated an outdated contract; it now points
to docs/providers.md, and the providers table gains the kvm row.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
 2026-09-16 09:18:40 +00:00
kacerrandClaude Opus 5 01c27b5f35 Cover tracing in the provider blueprint 
The blueprint reduced tracing to "don't open per-operation spans", which
reads as "nothing to do". Add an observability section: the operation
spans the decorators give for free, the child spans per backend call a
provider should add (parent-gated transport or a span decorator over the
SDK seam), the ctx/logger rules that keep traceID on log lines, and the
status per provider: kubernetes and gcp have child spans, kvm has none.
Matrix rows, the checklist item, gaps and a kvm proposal follow suit.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
 2026-09-16 09:14:06 +00:00
kacerrandClaude Opus 5 e08f23bcb3 Correct the provider blueprint after verification 
A re-check against the code found four mismatches: the kvm connection is
opened at startup by the leader's first health check, so that isn't doc
drift; cmd/main.go, not the monitor, type-asserts Checker; QuotaExceeded
and PermanentError are reasons on Provisioned, not conditions; and the
latch line range. The checklist also now matches how the existing
providers do interface assertions and RBAC markers.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
 2026-09-16 08:57:52 +00:00
kacerrandClaude Opus 5 d96e56f1a0 Write the provider blueprint 
docs/providers.md collects the provider contract that was spread across
comments, callers and per-provider docs: MUST/SHOULD/MAY rules per method,
each tied to the reconciler or GC behaviour it protects, the error taxonomy,
a feature matrix for kubernetes/gcp/kvm with file:line notes, a checklist for
adding a provider, and the known gaps. Linked from architecture.md and
CLAUDE.md.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
 2026-09-16 08:54:40 +00:00
kacerr cacd9583a3 Merge pull request 'Provider health checks: ProviderHealth CRD, per-provider checks and gauges' (#6) from feat/provider-health into main 
Reviewed-on: #6
 2026-09-16 10:46:47 +02:00
kacerrandClaude Opus 5 e6ed46fd7c Record the provider health validation and add the changelog entry 
The user ran the real-hypervisor checks against beelink and confirmed they
passed, which completes Step 7 of the provider health plan.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
 2026-09-16 08:45:58 +00:00
kacerrandClaude Opus 5 80b2e9c85e Plan the provider blueprint document 
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
 2026-09-16 08:41:42 +00:00
kacerrandClaude Opus 5 4e23347760 Point the KVM integration tests at a dedicated known_hosts 
Leaving KVM_TEST_KNOWN_HOSTS unset silently disables host-key
verification for the run, so document it as part of the normal setup
rather than as an optional extra, and suggest a kvm_known_hosts holding
only the hypervisors.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
 2026-09-12 05:59:22 +00:00
kacerrandClaude Opus 5 4e5b5d62df Probe hosts for health instead of risking the shared connection 
The health check shares each host's libvirt connection with the reconciles,
so it must not be able to take that connection away from them.

- A timed-out check no longer drops the link. The check goes through the new
  hostPool.probe, which abandons its RPCs when the context ends and leaves
  the connection alone, so an image upload or a Create running on the same
  link is not cut off. A link that really died is closed by the keepalive.
  One abandoned probe per host may stay waiting; until it returns, the next
  check reports Timeout at once (errProbeBusy).
- Waiting for a host's connection slot now ends with the caller's context.
  hostEntry used a mutex, so a check could wait out a reconcile redialing a
  dead host — minutes — past its own timeout. The slot is a one-slot channel
  and acquire bounds only the wait, never the uncontended take.
- --provider-check-timeout defaults to 0 and derives min(15s, interval).
  The flat 15s default contradicted the validation that the timeout may not
  exceed the interval, so --provider-check-interval=12s alone failed to
  start.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
 2026-09-12 05:04:45 +00:00
kacerrandClaude Opus 5 0e456affdd Record the automated validation of the provider health checks 
make test (envtest, -race) and the kind e2e suite pass; the e2e
ProviderHealth for the kubernetes provider turned Ready within a second
of the manager starting and its gauge appeared in the scrape. make lint
is blocked by the local go1.27 toolchain crashing staticcheck on the
standard library, unrelated to this branch.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
 2026-09-11 20:31:15 +00:00
kacerrandClaude Opus 5 19b4a82ae9 Document the provider health checks 
Architecture gets a provider-health section, component rows and four
decisions; the KVM provider doc gets a health-check section with the
reason table and the rotated-Secret gap; the testing guide lists the new
suites. CLAUDE.md's Architecture TODO and stale "only go.mod exists"
gotcha now point at the real docs, plus the RBAC-marker gotcha found
while wiring the monitor.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
 2026-09-11 20:28:09 +00:00
kacerrandClaude Opus 5 9a3ee963a0 Wire the provider health monitor into the manager, with gauges 
--provider-check-interval (1m; 0 = off, else at least 10s) and
--provider-check-timeout (15s) are validated before the providers config
loads. Checkers are collected from the unwrapped providers in config
order, so checks never count as provider requests. Three scrape-time
gauges per {provider,target} read the monitor's snapshot: up, check
duration and check timestamp. The e2e suite waits for the kubernetes
ProviderHealth to turn Ready and asserts its gauge in the scrape.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
 2026-09-11 20:23:55 +00:00
kacerrandClaude Opus 5 88a15ecdb5 Add the provider health Monitor 
A leader-only runnable checks every configured provider concurrently
under one timeout per round (first round at start, then every interval),
keeps the latest results for the metrics collector, logs targets going
down, changing reason or coming back, and publishes one ProviderHealth
object per provider: created with a managed-by label when missing, then
a status merge patch. Per-target lastTransitionTime survives rounds in
which ready does not flip. Objects of providers that are no longer
configured are deleted once at start; with checks turned off, all are.
Reads go through the API reader, so no cluster-wide informer is needed.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
 2026-09-11 20:20:52 +00:00
kacerrandClaude Opus 5 04dd2c0284 Add the KVM health check: reachability and storage pool per host 
Each configured host is checked concurrently over the pooled SSH+libvirt
connection: libvirt version and hostname prove the daemon answers, then
the configured pool must exist and be running, with its free space
reported. Dial failures are now tagged with their stage (SSH material,
TCP, SSH handshake, libvirt connect) by a dialError that keeps the
message and unwraps to the cause, so classification is unchanged and the
check can say why a host is unreachable. A pool verdict is decided after
the call returns so it never drops a working connection.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
 2026-09-11 20:17:35 +00:00
kacerrandClaude Opus 5 1962365f44 Add the provider Checker contract and the kubernetes and GCP checks 
Checker is an optional interface next to Provider: Check returns one
verdict per target and never fails as a whole. The kubernetes provider
lists at most one managed Pod cluster-wide (the verb and scope GC relies
on); the GCP provider fetches a single page of at most one managed
instance through a new Probe seam method, since the SDK treats
MaxResults as a page size and the existing AggregatedList drains every
page.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
 2026-09-11 20:08:21 +00:00
kacerrandClaude Opus 5 1c773c24ad Add the cluster-scoped ProviderHealth CRD 
One operator-owned object per configured provider will carry the
periodic check result: a Ready condition, per-target verdicts with their
own transition times, and a ready/total count for kubectl get. There is
no spec; only a viewer role is shipped since the operator is the sole
writer. Written by hand (kubebuilder is not installed), registered in
the existing AddKnownTypes call rather than a second init().

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
 2026-09-11 18:30:31 +00:00
kacerrandClaude Opus 5 18a3378507 Add plan: provider health checks 
Periodic, leader-only checks of every configured provider (KVM per host
over the pooled SSH+libvirt connection including the storage pool,
kubernetes apiserver/RBAC, GCP credentials/API), surfaced through a new
cluster-scoped ProviderHealth CRD, Prometheus gauges and transition logs.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
 2026-09-11 18:08:59 +00:00
kacerr eb8f075f7a Merge pull request 'KVM/libvirt provider: proxies as VMs on your own hypervisors' (#5) from feat/kvm-provider into main 
Reviewed-on: #5
 2026-09-11 17:02:33 +02:00
115 changed files with 22021 additions and 106 deletions
Expand all files Collapse all files
.claude/agents/deep-reviewer.md
+11
View File
@@ -0,0 +1,11 @@
---
name: deep-reviewer
description: Deep-dive verification of a single escalated hypothesis from the sweep.
model: fable
effort: high
tools: Read, Grep, Glob, Bash
---
You get ONE hypothesis with entry points. Follow it across the whole repo: callers,
callees, tests, configs. Verdict: CONFIRMED / REFUTED / PLAUSIBLE, with the exact
path/interleaving that breaks, or the reason it can't. Propose a minimal fix.
Don't re-review the package generally. Stay on the hypothesis.
.claude/agents/sweep-reviewer.md
+23
View File
@@ -0,0 +1,23 @@
---
name: sweep-reviewer
description: First-pass broad code review of one package/module. Use for fan-out sweeps.
model: opus
effort: high
tools: Read, Grep, Glob, Bash
---
You review ONE package/module given in the prompt. Read-only, never edit.
Output two sections:
## Findings
Concrete defects you verified. Per item: file:line, severity, one-line defect,
failure scenario (inputs/state -> wrong behavior). No style nits unless they hide a bug.
## Escalate
Things you could NOT confirm or refute within this package, where being wrong is costly.
Escalate when: shared mutable state across goroutines, lock ordering, context/cancel
propagation, retry+timeout+idempotency interactions, reconcile races, authn/authz
boundaries, invariants that depend on code outside this package, state machines.
Per item: entry-point files, the specific hypothesis ("X may be read without mu
on path A->B->C"), what you already checked, what's still unknown.
Do NOT escalate things you can resolve yourself. Go resolve them.
CHANGELOG.md
+81
View File
@@ -1,5 +1,86 @@
# Changelog
## 2026-09-17 20:36 UTC — kvm2: the four high findings from the code review, and its first run on a real hypervisor
A scoped review of the kvm2 provider (`review/REPORT.md`: 52 findings, 4 high, 20 medium,
28 low) produced four fixes, one commit each, then the provider was deployed in-cluster and
validated against beelink — the first time any of this code had run against real libvirt.
- **H1 — a stalled write no longer delays a connection teardown.** `sshSocket.Close` ran the
blocking close before the unblocking one, so a transfer stuck in a TCP write pinned the
teardown for as long as the kernel retransmitted (~15 min). `link.markDead` compounded it by
killing inside a `sync.Once` before signalling dead, which made even cancelling the reconcile
useless. With `MaxConcurrentReconciles: 3`, one silent host could stall every Proxy on every
provider. `internal/provider/kvm2/{ssh,backend}.go`.
- **H2 — a source error no longer reaches go-libvirt's stream sender.** A failing HTTP body
deadlocked `StorageVolUpload` outright and leaked two goroutines plus 4 MiB per occurrence,
against a 128Mi pod limit. The defect is upstream (`requestStream` never drains an unbuffered
`outErr`); the workaround wraps the reader so it can only ever end the stream cleanly, and
moves the transfer deadline onto the HTTP source so a stalled mirror unwinds instead of
leaking. A daemon-side failure mid-upload still leaks — documented in *Known gaps* with the
two-character upstream patch. `internal/provider/kvm2/{backend,image}.go`.
- **H3 — an unreadable address is no longer reported as a missing one.** Any error from both
address sources read as "no address yet", so a pooled connection killed by an unrelated
reconcile demoted a healthy proxy to Provisioning, cleared its IP and made the health engine
drop its counters. One source failing stays "no address yet"; every source failing, or a dead
connection, is now Transient. `internal/provider/kvm2/get.go`.
- **H4 — the base image cache is no longer shared between provider entries.** kvm2 built
byte-identical volume names to the `kvm` provider in the same pool, so two entries could share
one file, a concurrent fetch could boot a guest from a half-written base, and a `sha256` edit
deleted the base under live overlays. Names now carry the entry
(`base-<entry>-<key>.<format>`), and a digest mismatch is refused rather than resolved by
deleting. `internal/provider/kvm2/{volume,image}.go`, `internal/provider/config_kvm.go`.
- **klog and `net/http` errors now join the structured log stream.** client-go's leader election
wrote `I0917 ...` lines straight to stderr because nothing routed klog into logr, and the
discovery server's `http.Server` had no `ErrorLog`. The deployment also gained
`--zap-encoder=json` and `--zap-time-encoding=rfc3339`, so `kubectl logs | jq` works.
`cmd/main.go`, `internal/discovery/server.go`, `config/manager/manager.yaml`.
- **The `kvm2-home` provider entry is enabled** in `config/manager/providers_config.yaml`.
Validated end to end: health check Ready, a 433 MB image cached in 8.3 s under the new name
beside kvm's untouched one, a second Proxy reusing the cache, Squid answering a real CONNECT
probe, and Delete leaving the base image while removing the domain and its volumes.
## 2026-09-16 11:46 UTC — Provider blueprint split into contract, spec template and appendix
- **`docs/provider-contract.md`** is the generic provider specification: MUST/SHOULD/MAY
rules with the caller behind each one, 19 behaviour-profile axes with stable codes
(where providers may differ, such as blocking vs submit-and-return Create, or a
backend with no provisioning state), error taxonomy, credential hygiene, testing
obligations, the adding-a-provider checklist and the contract's known limits. It
names no existing provider.
- **`docs/provider-spec-template.md`** is the fill-in questionnaire a new provider's
design answers before implementation; entry numbers match the contract's profile
table.
- **`docs/providers-existing.md`** maps kubernetes, gcp and kvm to every profile and
keeps their per-provider reference notes, shared gaps and proposed features.
- **`docs/providers.md`** is now an index; README, CLAUDE.md and architecture.md point
at the new files. This supersedes the single-file blueprint written earlier on the
same branch.
- Two rules the callers enforce that were not written down before: orphan GC sweeps
providers sequentially with no deadline, so a hung `ListByTag` stalls all collection;
and the reconciler deletes an instance a backend reports as Stopped before start.
## 2026-09-16 08:45 UTC — Provider health checks: ProviderHealth CRD, per-provider checks, gauges
- **Each configured provider is now checked on a schedule** and reported in a
cluster-scoped `ProviderHealth` object (`kubectl get ph`), one per provider, with a
`Ready` condition, a reason, and per-target verdicts.
- **Checks:** kubernetes lists one object from the API server; GCP fetches a single
instance page; KVM checks SSH/libvirt reachability and the storage pool on every
host, in parallel. Failures map to fixed reasons (`Unreachable`,
`SSHHandshakeFailed`, `PoolNotFound`, `Timeout`, …).
- **The KVM check can't hurt reconciles:** it uses `hostPool.probe`, which gives up on
a slow RPC without dropping the shared connection, and waiting for a host slot can
be interrupted.
- **Metrics:** `proxy_operator_provider_target_up`, `_check_duration_seconds` and
`_check_timestamp_seconds`. The flags are `--provider-check-interval` and
`--provider-check-timeout` (defaults to `min(15s, interval)`).
- Validated with `make test`, `make test-e2e` and the real-hypervisor KVM run against
beelink. `make lint` is blocked by a Go 1.27 toolchain/staticcheck panic that has
nothing to do with this change.
- Key files: `api/v1alpha1/providerhealth_types.go`, `internal/provider/check.go`,
`internal/provider/kvm/`, `internal/providerhealth/`, `docs/architecture.md`.
## 2026-09-10 22:58 UTC — Code-review fixes on the KVM branch: deletion, validation, lazy SSH, guest agent
Five findings from a review of `feat/kvm-provider`, all fixed on the branch.
CLAUDE.md
+16 -8
View File
@@ -25,10 +25,16 @@ entrypoint yet, so there's nothing to run locally.
## Architecture
Not yet built. TODO: once the controller-runtime manager and reconcile loop exist,
document the data-flow diagram and package/role table here, following the standard
reconcile-loop shape: fetch → resolve → list related → compute desired state → merge
→ patch only on diff.
Documented in [docs/architecture.md](docs/architecture.md). It holds the component
table, the reconcile flow, and one section per runnable: health engine, discovery and
lease API, orphan GC, metrics, tracing, and provider health checks. The generic
provider contract (MUST/SHOULD rules, behaviour profiles, checklist for adding a
provider, known limits) is in [docs/provider-contract.md](docs/provider-contract.md);
a new provider starts from [docs/provider-spec-template.md](docs/provider-spec-template.md);
how the existing providers meet the contract, with their gaps, is in
[docs/providers-existing.md](docs/providers-existing.md); [docs/providers.md](docs/providers.md)
indexes them. Provider-specific detail is in `docs/kvm-provider.md` and the GCP
docs; the test layout is in `docs/testing.md`.
## Conventions
@@ -214,7 +220,9 @@ A lightweight `check` job (`go vet` / `go build` / `go test -short`) gates the b
## Gotchas
- This repo currently contains only `go.mod` (module
`gitea.home.hrajfrisbee.cz/kacerr/egress-proxies-operator`) — no controller code,
`main.go`, Dockerfile, k8s manifests, Makefile, or CI config exist yet. Nothing in
this file describes working software; it's the convention scaffold to build against.
- The "What this is" and "Commands" sections above predate the code and are stale.
The Makefile targets (`make test`, `make manifests generate`, `make test-e2e`,
`make test-integration-kvm`) and `docs/testing.md` are authoritative.
- `+kubebuilder:rbac` markers are package-level. Keep them in their own comment block,
never inside a type's or function's doc comment: controller-gen silently ignores
them there. After `make manifests`, check `git diff config/rbac/role.yaml`.
Makefile
+4
View File
@@ -70,6 +70,10 @@ test: manifests generate fmt vet setup-envtest ## Run tests.
test-integration-kvm: ## Run the KVM provider test against a real hypervisor. Needs KVM_TEST_HOST/USER/SSH_KEY/BRIDGE/IMAGE_URL; see docs/testing.md.
go test -race -tags=integration -count=1 -v -timeout 40m -run TestKVM ./internal/provider/kvm/
.PHONY: test-integration-kvm2
test-integration-kvm2: ## Run the kvm2 provider test against a real hypervisor. Needs KVM2_TEST_HOST/USER/SSH_KEY/BRIDGE/IMAGE_URL; see docs/testing.md.
go test -race -tags=integration -count=1 -v -timeout 40m -run TestKVM2 ./internal/provider/kvm2/
# TODO(user): To use a different vendor for e2e tests, modify the setup under 'tests/e2e'.
# The default setup assumes Kind is pre-installed and builds/loads the Manager Docker image locally.
# kubectl kuberc is disabled by default for test isolation; enable with:
PROJECT
+7
View File
@@ -18,4 +18,11 @@ resources:
kind: Proxy
path: gitea.home.hrajfrisbee.cz/kacerr/egress-proxies-operator/api/v1alpha1
version: v1alpha1
- api:
crdVersion: v1
domain: example.com
group: crawl
kind: ProviderHealth
path: gitea.home.hrajfrisbee.cz/kacerr/egress-proxies-operator/api/v1alpha1
version: v1alpha1
version: "3"
README.md
+17 -38
View File
@@ -104,7 +104,8 @@ make deploy IMG=<registry>/egress-proxies-operator:dev
- Provider config comes from the `providers-config` ConfigMap
([config/manager/providers_config.yaml](config/manager/providers_config.yaml));
the default ships only the kubernetes provider.
the default ships the kubernetes provider plus a `kvm-home` entry that needs
the `kvm-ssh` Secret (without it the manager starts and warns).
- The discovery API is exposed by the
`controller-manager-discovery-service` Service on port 8090.
- Auth: create the token Secret, or the API serves **unauthenticated**
@@ -159,6 +160,10 @@ Managed proxies as VMs on libvirt/KVM hosts you own. Full guide:
size (`placement.machineType`) and an image (`placement.image`) — see
[config/samples/proxy_kvm.yaml](config/samples/proxy_kvm.yaml).
The `kvm2` provider uses the same Secret and the same block shape (`type: kvm2`,
block `kvm2:`), so the steps above apply to it unchanged; its guide is
[docs/kvm2-provider.md](docs/kvm2-provider.md).
Cloud-init from a Secret: the Secret **must** carry the label
`crawl.example.com/cloud-init: "true"` — the operator's cache only holds
labelled Secrets, so an unlabelled one is invisible (the Proxy reports
@@ -185,46 +190,20 @@ can be two differently-configured instances of the same `gcp` type (see
| --- | --- | --- | --- |
| `kubernetes` | A real Squid pod (`ubuntu/squid:6.6-24.04_edge` by default) in the same cluster the operator runs in | `image` (optional) | Needs no cloud account — local dev, CI, and the kind quickstart. Pods share the cluster's egress IP, so it exercises the full lifecycle but not distinct egress paths. |
| `gcp` | A Compute Engine VM with an ephemeral external IP | `project` (required), `network`, `networkTag`, `diskSizeGb` | The real egress fleet. Deliberately uses only four API calls (Insert / Get / Delete / AggregatedList), all fire-and-forget: `Create` returns as soon as the operation is submitted and the reconciler discovers progress by polling `Get`. Auth is Application Default Credentials — workload identity in-cluster, `gcloud` ADC locally; no key-file plumbing. |
| `kvm` | A libvirt VM on one of your own hypervisors, reached over SSH | `sshKeyFile`, `knownHostsFile` (unless `insecureIgnoreHostKey`), `machineTypes`, `hosts` (required); `images`, `diskSizeGb`, `vendorData`, `installGuestAgent` | Home-lab or on-prem egress. The base image is downloaded into each host's pool on first use; `Create` waits for the VM to start. See [docs/kvm-provider.md](docs/kvm-provider.md). |
| `kvm2` | The same: a libvirt VM on one of your own hypervisors, reached over SSH | the same block shape as `kvm`, under `kvm2:` | The second-generation libvirt provider, built from the provider contract. Adds a child span per libvirt RPC, a pool-space precheck reported as quota (the reconciler waits instead of failing), SSH credentials re-read on every dial, and image downloads and health checks on connections of their own. It does not adopt `kvm`'s VMs. See [docs/kvm2-provider.md](docs/kvm2-provider.md). |
### Adding a provider
A new backend (Hetzner, AWS, ...) is four pieces; the contract lives in
[internal/provider/provider.go](internal/provider/provider.go):
1. **Implement the 4-method `Provider` interface** in a new
`internal/provider/<type>/` package:
- `Create` submits and returns — it never blocks until the VM runs, and
must be idempotent keyed on `req.Name` (a deterministic name derived
from the Proxy's UID), so a repeat call after a crash finds the
existing instance instead of duplicating it.
- `Get` returns `provider.ErrNotFound` as a *normal* outcome — the
reconciler branches on it for replacement and adoption, so don't
treat it as exceptional.
- `Delete` is idempotent: deleting an already-gone instance is not an
error.
- `ListByTag` returns every instance the operator ever tagged, for
orphan GC.
2. **Tag every created resource** with `LabelManaged=true` and
`LabelUID=<Proxy UID>`, and report `CreatedAt` — orphan GC relies on
all three to find owned resources and skip in-flight creates.
3. **Classify every returned error** with `provider.Wrap` into the
four-sentinel taxonomy in
[internal/provider/errors.go](internal/provider/errors.go)
(`ErrNotFound` / `ErrQuotaExceeded` / `ErrTransient` / `ErrPermanent`)
— the reconciler decides retry, slow backoff, or latching `Failed`
purely from that classification, never from provider-specific types.
4. **Wire it up**: add a type-specific config block in
[internal/provider/config.go](internal/provider/config.go), and
register the constructor in the builtins map in
[cmd/main.go](cmd/main.go) (`"<type>": <pkg>.New`). The registry
([internal/provider/registry](internal/provider/registry/registry.go))
handles named instances, and the metrics wrapper is applied
automatically.
Test against a fake API seam rather than the real cloud — see the
`instancesAPI` seam in
[internal/provider/gcp/gcp.go](internal/provider/gcp/gcp.go) for the
pattern.
The generic contract is [docs/provider-contract.md](docs/provider-contract.md),
the questionnaire a new provider's design fills in is
[docs/provider-spec-template.md](docs/provider-spec-template.md), and how the
existing providers meet the contract is
[docs/providers-existing.md](docs/providers-existing.md);
[docs/providers.md](docs/providers.md) indexes them. Start there: several rules (a
self-contained `providerID`, `Get` returning `ErrNotFound` only when the
instance is really gone, `CreatedAt` read back for orphan GC) are enforced by
callers, not by the interface.
## Tracing
api/v1alpha1/providerhealth_types.go
+127
View File
@@ -0,0 +1,127 @@
/*
Copyright 2026.
Licensed under the Apache License, Version 2.0 (the "License");
you may not use this file except in compliance with the License.
You may obtain a copy of the License at
http://www.apache.org/licenses/LICENSE-2.0
Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
See the License for the specific language governing permissions and
limitations under the License.
*/
package v1alpha1
import (
metav1 "k8s.io/apimachinery/pkg/apis/meta/v1"
)
// ConditionReady summarizes a ProviderHealth's targets: True when every
// target passed its last check.
const ConditionReady = "Ready"
// ProviderTarget is the latest check verdict for one thing a provider
// depends on: a KVM hypervisor, the Kubernetes API server, a GCP project.
type ProviderTarget struct {
// name identifies the target within its provider: the KVM host name,
// "apiserver", or the GCP project.
// +kubebuilder:validation:MinLength=1
// +required
Name string `json:"name"`
// address is where the check connected to.
// +optional
Address string `json:"address,omitempty"`
// ready is true when the last check succeeded.
// +required
Ready bool `json:"ready"`
// reason is a CamelCase verdict such as Reachable, Unreachable,
// Forbidden, SSHHandshakeFailed or PoolNotActive.
// +optional
Reason string `json:"reason,omitempty"`
// message is human-readable detail, usually the error of a failed check.
// +kubebuilder:validation:MaxLength=1024
// +optional
Message string `json:"message,omitempty"`
// latencyMillis is how long the last check took.
// +optional
LatencyMillis int64 `json:"latencyMillis,omitempty"`
// lastTransitionTime is when ready last changed.
// +required
LastTransitionTime metav1.Time `json:"lastTransitionTime"`
// details are provider-specific facts from the last check, e.g. the
// libvirt version and free storage-pool space of a KVM host.
// +optional
Details map[string]string `json:"details,omitempty"`
}
// ProviderHealthStatus is the result of the latest check of one provider.
type ProviderHealthStatus struct {
// type is the provider's type from the providers config: kubernetes,
// gcp, kvm or kvm2.
// +optional
Type string `json:"type,omitempty"`
// conditions hold Ready, which summarizes the targets.
// +listType=map
// +listMapKey=type
// +optional
Conditions []metav1.Condition `json:"conditions,omitempty"`
// targets hold one verdict per checked target.
// +listType=map
// +listMapKey=name
// +optional
Targets []ProviderTarget `json:"targets,omitempty"`
// targetsReady counts the ready targets as "ready/total".
// +optional
TargetsReady string `json:"targetsReady,omitempty"`
// lastCheckTime is when the latest check finished.
// +optional
LastCheckTime *metav1.Time `json:"lastCheckTime,omitempty"`
}
// +kubebuilder:object:root=true
// +kubebuilder:subresource:status
// +kubebuilder:resource:scope=Cluster,shortName=ph
// +kubebuilder:printcolumn:name="Type",type=string,JSONPath=`.status.type`
// +kubebuilder:printcolumn:name="Ready",type=string,JSONPath=`.status.conditions[?(@.type=="Ready")].status`
// +kubebuilder:printcolumn:name="Targets",type=string,JSONPath=`.status.targetsReady`
// +kubebuilder:printcolumn:name="Last Check",type=date,JSONPath=`.status.lastCheckTime`
// +kubebuilder:printcolumn:name="Age",type=date,JSONPath=`.metadata.creationTimestamp`
// ProviderHealth reports the periodic health check of one configured
// provider. It is named after the provider, and the operator creates,
// updates and deletes it itself; there is no spec to edit.
type ProviderHealth struct {
metav1.TypeMeta `json:",inline"`
// metadata is a standard object metadata
// +optional
metav1.ObjectMeta `json:"metadata,omitzero"`
// status is the result of the latest check
// +optional
Status ProviderHealthStatus `json:"status,omitzero"`
}
// +kubebuilder:object:root=true
// ProviderHealthList contains a list of ProviderHealth
type ProviderHealthList struct {
metav1.TypeMeta `json:",inline"`
metav1.ListMeta `json:"metadata,omitzero"`
Items []ProviderHealth `json:"items"`
}
api/v1alpha1/proxy_types.go
+1 -1
View File
@@ -361,7 +361,7 @@ type ProxyList struct {
func init() {
SchemeBuilder.Register(func(s *runtime.Scheme) error {
s.AddKnownTypes(SchemeGroupVersion, &Proxy{}, &ProxyList{})
s.AddKnownTypes(SchemeGroupVersion, &Proxy{}, &ProxyList{}, &ProviderHealth{}, &ProviderHealthList{})
return nil
})
}
api/v1alpha1/zz_generated.deepcopy.go
+114
View File
@@ -95,6 +95,120 @@ func (in *PlacementSpec) DeepCopy() *PlacementSpec {
return out
}
// DeepCopyInto is an autogenerated deepcopy function, copying the receiver, writing into out. in must be non-nil.
func (in *ProviderHealth) DeepCopyInto(out *ProviderHealth) {
*out = *in
out.TypeMeta = in.TypeMeta
in.ObjectMeta.DeepCopyInto(&out.ObjectMeta)
in.Status.DeepCopyInto(&out.Status)
}
// DeepCopy is an autogenerated deepcopy function, copying the receiver, creating a new ProviderHealth.
func (in *ProviderHealth) DeepCopy() *ProviderHealth {
if in == nil {
return nil
}
out := new(ProviderHealth)
in.DeepCopyInto(out)
return out
}
// DeepCopyObject is an autogenerated deepcopy function, copying the receiver, creating a new runtime.Object.
func (in *ProviderHealth) DeepCopyObject() runtime.Object {
if c := in.DeepCopy(); c != nil {
return c
}
return nil
}
// DeepCopyInto is an autogenerated deepcopy function, copying the receiver, writing into out. in must be non-nil.
func (in *ProviderHealthList) DeepCopyInto(out *ProviderHealthList) {
*out = *in
out.TypeMeta = in.TypeMeta
in.ListMeta.DeepCopyInto(&out.ListMeta)
if in.Items != nil {
in, out := &in.Items, &out.Items
*out = make([]ProviderHealth, len(*in))
for i := range *in {
(*in)[i].DeepCopyInto(&(*out)[i])
}
}
}
// DeepCopy is an autogenerated deepcopy function, copying the receiver, creating a new ProviderHealthList.
func (in *ProviderHealthList) DeepCopy() *ProviderHealthList {
if in == nil {
return nil
}
out := new(ProviderHealthList)
in.DeepCopyInto(out)
return out
}
// DeepCopyObject is an autogenerated deepcopy function, copying the receiver, creating a new runtime.Object.
func (in *ProviderHealthList) DeepCopyObject() runtime.Object {
if c := in.DeepCopy(); c != nil {
return c
}
return nil
}
// DeepCopyInto is an autogenerated deepcopy function, copying the receiver, writing into out. in must be non-nil.
func (in *ProviderHealthStatus) DeepCopyInto(out *ProviderHealthStatus) {
*out = *in
if in.Conditions != nil {
in, out := &in.Conditions, &out.Conditions
*out = make([]v1.Condition, len(*in))
for i := range *in {
(*in)[i].DeepCopyInto(&(*out)[i])
}
}
if in.Targets != nil {
in, out := &in.Targets, &out.Targets
*out = make([]ProviderTarget, len(*in))
for i := range *in {
(*in)[i].DeepCopyInto(&(*out)[i])
}
}
if in.LastCheckTime != nil {
in, out := &in.LastCheckTime, &out.LastCheckTime
*out = (*in).DeepCopy()
}
}
// DeepCopy is an autogenerated deepcopy function, copying the receiver, creating a new ProviderHealthStatus.
func (in *ProviderHealthStatus) DeepCopy() *ProviderHealthStatus {
if in == nil {
return nil
}
out := new(ProviderHealthStatus)
in.DeepCopyInto(out)
return out
}
// DeepCopyInto is an autogenerated deepcopy function, copying the receiver, writing into out. in must be non-nil.
func (in *ProviderTarget) DeepCopyInto(out *ProviderTarget) {
*out = *in
in.LastTransitionTime.DeepCopyInto(&out.LastTransitionTime)
if in.Details != nil {
in, out := &in.Details, &out.Details
*out = make(map[string]string, len(*in))
for key, val := range *in {
(*out)[key] = val
}
}
}
// DeepCopy is an autogenerated deepcopy function, copying the receiver, creating a new ProviderTarget.
func (in *ProviderTarget) DeepCopy() *ProviderTarget {
if in == nil {
return nil
}
out := new(ProviderTarget)
in.DeepCopyInto(out)
return out
}
// DeepCopyInto is an autogenerated deepcopy function, copying the receiver, writing into out. in must be non-nil.
func (in *Proxy) DeepCopyInto(out *Proxy) {
*out = *in
cmd/main.go
+71 -3
View File
@@ -16,8 +16,8 @@ limitations under the License.
// Package main is the composition root: it loads the provider config (fail
// fast), assembles the provider registry, and wires the reconciler, health
// engine, lease store, discovery API, orphan GC, and metrics onto one
// controller-runtime manager.
// engine, lease store, discovery API, orphan GC, provider health checks and
// metrics onto one controller-runtime manager.
package main
import (
@@ -38,6 +38,7 @@ import (
"k8s.io/apimachinery/pkg/runtime"
utilruntime "k8s.io/apimachinery/pkg/util/runtime"
clientgoscheme "k8s.io/client-go/kubernetes/scheme"
"k8s.io/klog/v2"
ctrl "sigs.k8s.io/controller-runtime"
"sigs.k8s.io/controller-runtime/pkg/cache"
"sigs.k8s.io/controller-runtime/pkg/client"
@@ -59,7 +60,9 @@ import (
"gitea.home.hrajfrisbee.cz/kacerr/egress-proxies-operator/internal/provider/gcp"
"gitea.home.hrajfrisbee.cz/kacerr/egress-proxies-operator/internal/provider/kubernetes"
"gitea.home.hrajfrisbee.cz/kacerr/egress-proxies-operator/internal/provider/kvm"
"gitea.home.hrajfrisbee.cz/kacerr/egress-proxies-operator/internal/provider/kvm2"
"gitea.home.hrajfrisbee.cz/kacerr/egress-proxies-operator/internal/provider/registry"
"gitea.home.hrajfrisbee.cz/kacerr/egress-proxies-operator/internal/providerhealth"
"gitea.home.hrajfrisbee.cz/kacerr/egress-proxies-operator/internal/tracing"
"gitea.home.hrajfrisbee.cz/kacerr/egress-proxies-operator/internal/version"
// +kubebuilder:scaffold:imports
@@ -97,6 +100,7 @@ func main() {
var showVersion bool
var gcpWireFullPayloads bool
var traceHealthProbes bool
var providerCheckInterval, providerCheckTimeout time.Duration
flag.StringVar(&metricsAddr, "metrics-bind-address", "0", "The address the metrics endpoint binds to. "+
"Use :8443 for HTTPS or :8080 for HTTP, or leave as 0 to disable the metrics service.")
@@ -140,6 +144,13 @@ func main() {
flag.BoolVar(&traceHealthProbes, "trace-health-probes", false,
"Emit one trace span per health probe. Off by default: probes run about "+
"once per second per proxy and would dominate trace volume.")
flag.DurationVar(&providerCheckInterval, "provider-check-interval", time.Minute,
"Interval between provider health checks (can the cluster reach each hypervisor, API "+
"and account), published as ProviderHealth objects and metrics. 0 turns them off; "+
"otherwise at least 10s.")
flag.DurationVar(&providerCheckTimeout, "provider-check-timeout", 0,
"Time limit for one round of provider health checks: 15s, or the interval when that is "+
"shorter. If set, at most the interval.")
opts := zap.Options{
Development: true,
@@ -152,7 +163,15 @@ func main() {
os.Exit(0)
}
ctrl.SetLogger(zap.New(zap.UseFlagOptions(&opts)))
logger := zap.New(zap.UseFlagOptions(&opts))
ctrl.SetLogger(logger)
// client-go logs through klog, and controller-runtime hands the leader
// elector a context with no logr in it, so klog.FromContext falls back
// to klog's own stderr writer and those lines ignore the encoder
// entirely. Plain SetLogger rather than the contextual form: it keeps
// klog's verbosity check in front, so this changes the format of what
// klog already emits without letting client-go's V(1) chatter through.
klog.SetLogger(logger)
setupLog.Info("Starting egress-proxies-operator",
"commit", version.Resolve(), "goVersion", goruntime.Version())
@@ -174,6 +193,10 @@ func main() {
setupLog.Error(nil, "--providers-config is required")
os.Exit(1)
}
if err := providerhealth.ValidateSchedule(providerCheckInterval, providerCheckTimeout); err != nil {
setupLog.Error(err, "Invalid provider check flags")
os.Exit(1)
}
cfg, err := provider.LoadConfigFile(providersConfig)
if err != nil {
setupLog.Error(err, "Failed to load providers config", "path", providersConfig)
@@ -189,12 +212,21 @@ func main() {
return gcp.NewWithWireOptions(ctx, pc, gcp.WireLogOptions{FullPayloads: gcpWireFullPayloads})
},
"kvm": kvm.New,
"kvm2": kvm2.New,
})
if err != nil {
setupLog.Error(err, "Failed to build providers")
os.Exit(1)
}
m := metrics.New()
// Checkers come from the unwrapped providers, in config order: the
// metrics and tracing wrappers implement only Provider, and a check is
// not a provider request to be counted anyway.
checkTargets := make([]providerhealth.Target, 0, len(cfg.Providers))
for _, pc := range cfg.Providers {
checker, _ := providers[pc.Name].(provider.Checker)
checkTargets = append(checkTargets, providerhealth.Target{Name: pc.Name, Type: pc.Type, Checker: checker})
}
for name, p := range providers {
// Tracing outermost: the span covers the metrics recording too.
providers[name] = provider.WithTracing(name, provider.WithMetrics(name, p, m))
@@ -314,6 +346,18 @@ func main() {
os.Exit(1)
}
monitor := &providerhealth.Monitor{
Client: mgr.GetClient(),
Reader: mgr.GetAPIReader(),
Targets: checkTargets,
Interval: providerCheckInterval,
Timeout: providerCheckTimeout,
}
if err := mgr.Add(monitor); err != nil {
setupLog.Error(err, "Failed to add provider health monitor")
os.Exit(1)
}
if err := m.Register(ctrlmetrics.Registry,
proxyPhaseCounts(mgr.GetClient()),
func() int {
@@ -327,6 +371,10 @@ func main() {
setupLog.Error(err, "Failed to register metrics")
os.Exit(1)
}
if err := metrics.RegisterProviderTargets(ctrlmetrics.Registry, providerTargetSamples(monitor)); err != nil {
setupLog.Error(err, "Failed to register provider check metrics")
os.Exit(1)
}
if err := (&controller.ProxyReconciler{
Client: mgr.GetClient(),
@@ -389,3 +437,23 @@ func proxyPhaseCounts(c client.Reader) func() map[string]int {
return counts
}
}
// providerTargetSamples flattens the monitor's latest results for the
// scrape-time provider-check gauges.
func providerTargetSamples(mon *providerhealth.Monitor) func() []metrics.ProviderTarget {
return func() []metrics.ProviderTarget {
var out []metrics.ProviderTarget
for _, r := range mon.Snapshot() {
for _, ts := range r.Targets {
out = append(out, metrics.ProviderTarget{
Provider: r.Provider,
Target: ts.Name,
Up: ts.Ready,
Duration: ts.Latency,
CheckedAt: r.CheckedAt,
})
}
}
return out
}
}
config/crd/bases/crawl.example.com_providerhealths.yaml
+193
View File
@@ -0,0 +1,193 @@
---
apiVersion: apiextensions.k8s.io/v1
kind: CustomResourceDefinition
metadata:
annotations:
controller-gen.kubebuilder.io/version: v0.21.0
name: providerhealths.crawl.example.com
spec:
group: crawl.example.com
names:
kind: ProviderHealth
listKind: ProviderHealthList
plural: providerhealths
shortNames:
- ph
singular: providerhealth
scope: Cluster
versions:
- additionalPrinterColumns:
- jsonPath: .status.type
name: Type
type: string
- jsonPath: .status.conditions[?(@.type=="Ready")].status
name: Ready
type: string
- jsonPath: .status.targetsReady
name: Targets
type: string
- jsonPath: .status.lastCheckTime
name: Last Check
type: date
- jsonPath: .metadata.creationTimestamp
name: Age
type: date
name: v1alpha1
schema:
openAPIV3Schema:
description: |-
ProviderHealth reports the periodic health check of one configured
provider. It is named after the provider, and the operator creates,
updates and deletes it itself; there is no spec to edit.
properties:
apiVersion:
description: |-
APIVersion defines the versioned schema of this representation of an object.
Servers should convert recognized schemas to the latest internal value, and
may reject unrecognized values.
More info: https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#resources
type: string
kind:
description: |-
Kind is a string value representing the REST resource this object represents.
Servers may infer this from the endpoint the client submits requests to.
Cannot be updated.
In CamelCase.
More info: https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#types-kinds
type: string
metadata:
type: object
status:
description: status is the result of the latest check
properties:
conditions:
description: conditions hold Ready, which summarizes the targets.
items:
description: Condition contains details for one aspect of the current
state of this API Resource.
properties:
lastTransitionTime:
description: |-
lastTransitionTime is the last time the condition transitioned from one status to another.
This should be when the underlying condition changed. If that is not known, then using the time when the API field changed is acceptable.
format: date-time
type: string
message:
description: |-
message is a human readable message indicating details about the transition.
This may be an empty string.
maxLength: 32768
type: string
observedGeneration:
description: |-
observedGeneration represents the .metadata.generation that the condition was set based upon.
For instance, if .metadata.generation is currently 12, but the .status.conditions[x].observedGeneration is 9, the condition is out of date
with respect to the current state of the instance.
format: int64
minimum: 0
type: integer
reason:
description: |-
reason contains a programmatic identifier indicating the reason for the condition's last transition.
Producers of specific condition types may define expected values and meanings for this field,
and whether the values are considered a guaranteed API.
The value should be a CamelCase string.
This field may not be empty.
maxLength: 1024
minLength: 1
pattern: ^[A-Za-z]([A-Za-z0-9_,:]*[A-Za-z0-9_])?$
type: string
status:
description: status of the condition, one of True, False, Unknown.
enum:
- "True"
- "False"
- Unknown
type: string
type:
description: type of condition in CamelCase or in foo.example.com/CamelCase.
maxLength: 316
pattern: ^([a-z0-9]([-a-z0-9]*[a-z0-9])?(\.[a-z0-9]([-a-z0-9]*[a-z0-9])?)*/)?(([A-Za-z0-9][-A-Za-z0-9_.]*)?[A-Za-z0-9])$
type: string
required:
- lastTransitionTime
- message
- reason
- status
- type
type: object
type: array
x-kubernetes-list-map-keys:
- type
x-kubernetes-list-type: map
lastCheckTime:
description: lastCheckTime is when the latest check finished.
format: date-time
type: string
targets:
description: targets hold one verdict per checked target.
items:
description: |-
ProviderTarget is the latest check verdict for one thing a provider
depends on: a KVM hypervisor, the Kubernetes API server, a GCP project.
properties:
address:
description: address is where the check connected to.
type: string
details:
additionalProperties:
type: string
description: |-
details are provider-specific facts from the last check, e.g. the
libvirt version and free storage-pool space of a KVM host.
type: object
lastTransitionTime:
description: lastTransitionTime is when ready last changed.
format: date-time
type: string
latencyMillis:
description: latencyMillis is how long the last check took.
format: int64
type: integer
message:
description: message is human-readable detail, usually the error
of a failed check.
maxLength: 1024
type: string
name:
description: |-
name identifies the target within its provider: the KVM host name,
"apiserver", or the GCP project.
minLength: 1
type: string
ready:
description: ready is true when the last check succeeded.
type: boolean
reason:
description: |-
reason is a CamelCase verdict such as Reachable, Unreachable,
Forbidden, SSHHandshakeFailed or PoolNotActive.
type: string
required:
- lastTransitionTime
- name
- ready
type: object
type: array
x-kubernetes-list-map-keys:
- name
x-kubernetes-list-type: map
targetsReady:
description: targetsReady counts the ready targets as "ready/total".
type: string
type:
description: |-
type is the provider's type from the providers config: kubernetes,
gcp, kvm or kvm2.
type: string
type: object
type: object
served: true
storage: true
subresources:
status: {}
config/crd/kustomization.yaml
+1
View File
@@ -3,4 +3,5 @@
# It should be run by config/default
resources:
- bases/crawl.example.com_proxies.yaml
- bases/crawl.example.com_providerhealths.yaml
# +kubebuilder:scaffold:crdkustomizeresource
config/manager/manager.yaml
+16 -3
View File
@@ -64,8 +64,19 @@ spec:
- --leader-elect
- --health-probe-bind-address=:8081
- --providers-config=/etc/proxy-operator/providers.yaml
# Structured logs: every line is one JSON object, so `kubectl logs
# | jq` works. The time encoding is not optional — the JSON encoder
# defaults to epoch seconds. Verbosity stays at the binary's
# default (debug, one line per provider API call); add
# --zap-log-level=info for outcomes only.
- --zap-encoder=json
- --zap-time-encoding=rfc3339
# Add --trace-health-probes to emit one span per health probe
# (high volume: ~1/s per proxy).
# Provider health checks (kubectl get providerhealth) run every
# minute; tune with --provider-check-interval (0 turns them off,
# otherwise at least 10s) and --provider-check-timeout (15s, or
# the interval if that is shorter).
env:
# Bearer token for the discovery API. Optional: without the
# Secret the API serves unauthenticated (with a loud warning).
@@ -145,9 +156,10 @@ spec:
- name: providers-config
mountPath: /etc/proxy-operator
readOnly: true
# SSH material for kvm providers (sshKeyFile / knownHostsFile in the
# providers config point here). Optional: the mount is empty when the
# Secret does not exist, and only kvm entries need it. Create it with:
# SSH material for the kvm and kvm2 providers (sshKeyFile /
# knownHostsFile in the providers config point here). One Secret
# serves both types. Optional: the mount is empty when the Secret
# does not exist, and only kvm and kvm2 entries need it. Create it with:
# kubectl -n egress-proxies-operator-system create secret generic kvm-ssh \
# --from-file=id_ed25519=<private key> \
# --from-file=known_hosts=<known_hosts with every host's key>
@@ -163,6 +175,7 @@ spec:
- name: providers-config
configMap:
name: providers-config
# Shared by the kvm and kvm2 providers; see the mount above.
- name: kvm-ssh
secret:
secretName: kvm-ssh
config/manager/providers_config.yaml
+33
View File
@@ -51,4 +51,37 @@ data:
bridge: br0 # to a LAN with DHCP; the VM's IP comes from there
# socket: /var/run/libvirt/libvirt-sock
# pool: default
# kvm2 takes the same block shape as kvm: an entry ports by changing
# type to kvm2 and the block name to kvm2. kvm2 does not adopt domains
# made by kvm, so delete those Proxies first. It caches its base images
# under base-<entry>-<key>.<format>, so it downloads its own copy rather
# than sharing kvm's; budget the pool space. See docs/kvm2-provider.md.
- name: kvm2-home
type: kvm2
kvm2:
sshKeyFile: /etc/proxy-operator/kvm/id_ed25519
knownHostsFile: /etc/proxy-operator/kvm/known_hosts
# insecureIgnoreHostKey: true
# diskSizeGb: 10
# installGuestAgent: false
# vendorData: |
# #cloud-config
# ...
images:
debian-13:
url: https://cloud.debian.org/images/cloud/trixie/latest/debian-13-generic-amd64.qcow2
# sha256: "<64 hex>"
# firmware: uefi
ubuntu-24.04:
url: https://cloud-images.ubuntu.com/noble/current/noble-server-cloudimg-amd64.img
machineTypes:
small: {vcpus: 1, memoryMiB: 1024}
medium: {vcpus: 2, memoryMiB: 2048}
hosts:
- name: beelink
address: beelink.tail8c69c.ts.net:22
user: novakj
bridge: br0
# socket: /var/run/libvirt/libvirt-sock
# pool: default
config/rbac/kustomization.yaml
+3
View File
@@ -25,4 +25,7 @@ resources:
- proxy_admin_role.yaml
- proxy_editor_role.yaml
- proxy_viewer_role.yaml
# ProviderHealth objects are written only by the operator, so they get a
# viewer role and nothing else.
- providerhealth_viewer_role.yaml
config/rbac/providerhealth_viewer_role.yaml
+29
View File
@@ -0,0 +1,29 @@
# This rule is not used by the project egress-proxies-operator itself.
# It is provided to allow the cluster admin to help manage permissions for users.
#
# Grants read-only access to ProviderHealth objects, the results of the
# operator's periodic provider checks. The operator creates, updates and
# deletes them itself, so no admin or editor role is provided.
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
labels:
app.kubernetes.io/name: egress-proxies-operator
app.kubernetes.io/managed-by: kustomize
name: providerhealth-viewer-role
rules:
- apiGroups:
- crawl.example.com
resources:
- providerhealths
verbs:
- get
- list
- watch
- apiGroups:
- crawl.example.com
resources:
- providerhealths/status
verbs:
- get
config/rbac/role.yaml
+16
View File
@@ -22,6 +22,22 @@ rules:
- get
- list
- watch
- apiGroups:
- crawl.example.com
resources:
- providerhealths
verbs:
- create
- delete
- get
- list
- apiGroups:
- crawl.example.com
resources:
- providerhealths/status
verbs:
- get
- patch
- apiGroups:
- crawl.example.com
resources:
config/samples/kustomization.yaml
+1
View File
@@ -5,5 +5,6 @@ resources:
- proxy_kubernetes.yaml
- proxy_gcp.yaml
- proxy_kvm.yaml
- proxy_kvm2.yaml
- proxy_external.yaml
# +kubebuilder:scaffold:manifestskustomizesamples
config/samples/providers-config.yaml
+33
View File
@@ -55,3 +55,36 @@ providers:
bridge: br0 # to a LAN with DHCP; the VM's IP comes from there
# socket: /var/run/libvirt/libvirt-sock
# pool: default
# kvm2 takes the same block shape as kvm: an entry ports by changing type
# to kvm2 and the block name to kvm2. kvm2 does not adopt domains made by
# kvm, so delete those Proxies first. It caches its base images under
# base-<entry>-<key>.<format>, so it downloads its own copy rather than
# sharing kvm's; budget the pool space. See docs/kvm2-provider.md.
# - name: kvm2-home
# type: kvm2
# kvm2:
# sshKeyFile: /etc/proxy-operator/kvm/id_ed25519
# knownHostsFile: /etc/proxy-operator/kvm/known_hosts
# # insecureIgnoreHostKey: true
# # diskSizeGb: 10
# # installGuestAgent: false
# # vendorData: |
# # #cloud-config
# # ...
# images:
# debian-13:
# url: https://cloud.debian.org/images/cloud/trixie/latest/debian-13-generic-amd64.qcow2
# # sha256: "<64 hex>"
# # firmware: uefi
# ubuntu-24.04:
# url: https://cloud-images.ubuntu.com/noble/current/noble-server-cloudimg-amd64.img
# machineTypes:
# small: {vcpus: 1, memoryMiB: 1024}
# medium: {vcpus: 2, memoryMiB: 2048}
# hosts:
# - name: beelink
# address: beelink:22
# user: novakj
# bridge: br0
# # socket: /var/run/libvirt/libvirt-sock
# # pool: default
config/samples/proxy_kvm2.yaml
+44
View File
@@ -0,0 +1,44 @@
# A Managed proxy backed by a libvirt/KVM hypervisor through the kvm2
# provider: the operator boots a stock Debian cloud image as a VM on the
# named host, with a NIC on the host's bridge (IP from the LAN's DHCP), and
# installs Squid via cloud-init. Requires a providers-config entry named
# "kvm2-home" (see providers-config.yaml) and the kvm-ssh Secret, which the
# kvm and kvm2 providers share (docs/kvm2-provider.md).
#
# placement.zone is the host name from kvm2.hosts (optional when only one
# host is configured); machineType and image are keys in the provider's
# machineTypes and images maps. The first Proxy for an image on a host
# waits for the image download; later ones reuse the cached copy.
apiVersion: crawl.example.com/v1alpha1
kind: Proxy
metadata:
name: proxy-kvm2-sample
spec:
mode: Managed
provider: kvm2-home
placement:
zone: beelink
machineType: small
image: debian-13
port: 3128
cloudInit:
inline: |
#cloud-config
packages:
- squid
write_files:
- path: /etc/squid/conf.d/proxy-operator.conf
content: |
http_port 3128
http_access allow all
via off
forwarded_for off
runcmd:
- systemctl restart squid
# Optional: log in with `ssh debian@<ip>` (the Debian image's default
# user) to look around; `virsh console <name>` on the host works too.
# ssh_authorized_keys:
# - ssh-ed25519 AAAA... you@laptop
attributes:
geo: home
purpose: crawl
docs/architecture.md
+178 -3
View File
@@ -4,16 +4,18 @@
| Component | Package | Runs as | Leader-elected | Role |
|---|---|---|---|---|
| Proxy CRD + helpers | `api/v1alpha1` | types | — | `Proxy` spec/status, CEL validation, defaulting, pure helpers |
| Proxy + ProviderHealth CRDs | `api/v1alpha1` | types | — | `Proxy` spec/status, CEL validation, defaulting, pure helpers; the status-only, cluster-scoped `ProviderHealth` |
| Reconciler | `internal/controller` | controller | yes (with the manager) | the state machine: provision, replace, delete, represent health |
| Provider contract | `internal/provider` | library | — | `Provider` interface, error taxonomy, deterministic naming, config, metrics decorator |
| Provider contract | `internal/provider` | library | — | `Provider` interface, optional `Checker`, error taxonomy, deterministic naming, config, metrics decorator; rules in [provider-contract.md](provider-contract.md), per-provider profile table in [providers-existing.md](providers-existing.md) |
| kubernetes provider | `internal/provider/kubernetes` | library | — | real Squid pods in this cluster (local dev/CI) |
| gcp provider | `internal/provider/gcp` | library | — | Compute Engine VMs, four API calls, fire-and-forget ops |
| kvm provider | `internal/provider/kvm` | library | — | VMs on libvirt hosts over SSH-forwarded sockets; image cache, overlay disks, NoCloud seeds |
| kvm2 provider | `internal/provider/kvm2` | library | — | the same backend, built clean-room from the provider contract as a second type: per-RPC child spans, pool-space quota, per-call budgets, its own metadata namespace, credentials re-read on every dial |
| Health engine | `internal/health` | Runnable | yes | through-the-proxy probes, thresholds, transition events |
| Lease store | `internal/lease` | Runnable (expiry sweep) | no | in-memory leases + cooldowns, single mutex |
| Discovery API | `internal/discovery` | Runnable | no | HTTP list/lease/release/report on `:8090` |
| Orphan GC | `internal/gc` | Runnable | yes | deletes tagged instances whose CR is gone |
| Provider health monitor | `internal/providerhealth` | Runnable | yes | periodic provider checks → one `ProviderHealth` per provider, target gauges, transition logs |
| Metrics | `internal/metrics` | library | — | explicit registration, scrape-time collectors |
| Composition root | `cmd/main.go` | binary | — | flags, provider registry, wires everything onto one manager |
@@ -167,6 +169,40 @@ providerID = hosts/<host>/domains/<name>; GC identity lives in the domain's
<metadata> element (libvirt has no labels). All RPCs run under a context-
bounded call: go-libvirt never times out on its own, so a cancelled reconcile
closes the SSH connection to unblock it.
kvm2 provider (internal/provider/kvm2/) - the same transport, a second type built from the contract
prov.Create ──► placement checked against the catalogue (Permanent before any dial)
DomainLookupByName: exists → mark+uid guard (no mark or foreign ─┐ every step
uid = Permanent; paused = Permanent), │
start if needed, wait for running, done │ idempotent,
else: StoragePoolGetInfo precheck: image (if uncached) + diskSizeGb │ under a
+ 16 MiB must fit, else ErrQuotaExceeded (reconciler │ per-name lock
waits 5m) │ shared with
ensure base image: marker present = cached; otherwise GET │ Delete
streamed into StorageVolUpload on a DEDICATED connection, │
StoragePoolRefresh, marker written last │
overlay volume (exists = reuse) │
seed ISO (always rebuilt) │
DomainDefineXML (<metadata> in kvm2's own namespace: │
managed, uid, createdAt) + DomainCreate, poll until │
running (15s cap; a domain that dies at once = Transient) ─┘
prov.Get ──► DomainLookupByName + DomainGetMetadata + DomainGetState
+ DomainInterfaceAddresses (guest agent, then ARP; first routable IPv4)
running without IP = Provisioning; nostate = Provisioning;
only ErrNoDomain is NotFound
prov.Delete ──► destroy (not running = fine) → undefine (nvram, managed save, snapshots)
→ delete overlay + seed by name; unconfigured host = logged no-op
prov.ListByTag ─► ConnectListAllDomains on every host, keep domains whose kvm2
<metadata> says managed=true; one host down fails the sweep,
one unreadable domain is skipped
Check ──► per host, on a FRESH connection: version, hostname, pool lookup,
pool info, close; never the pooled one
providerID = hosts/<host>/domains/<name>. Every RPC has its own budget (dial 10s,
RPC 30s, DomainCreate 90s, seed upload 60s, image transfer 20m): a call that
outlives its ctx or budget kills the pooled connection (the only way to unblock
go-libvirt), so other calls in flight on that host fail once with Transient and
retry; the next call redials. Each RPC is a child span kvm2.<Method> under
provider.*, plus kvm2.dial and kvm2.download.
```
The reconciler never watches provider-side resources (Pods or GCP VMs).
@@ -306,6 +342,8 @@ registries). Two kinds:
- **Scrape-time collectors** — `proxy_operator_proxies{phase}` and
`proxy_operator_leases_active` read the cache / lease store at every
scrape; reconcile-incremented gauges inevitably drift and leak series.
The provider-check gauges (§11) read the health monitor's snapshot the
same way.
- **Fed vectors** — `healthcheck_duration_seconds{proxy}` and
`healthcheck_failures_total{proxy}` observe EVERY probe (status writes
are transition-only; metrics carry the high-frequency signal), and the
@@ -340,6 +378,15 @@ incoming trace context):
- `provider.create|get|delete|list` (client spans, `provider.WithTracing`
decorator wired outermost around `WithMetrics`); the GCP SDK's own
otelhttp transport contributes HTTP child spans automatically.
- kvm2 child spans: under `provider.*`, one `kvm2.<Method>` client span per
libvirt RPC (for example `kvm2.DomainDefineXML`, attributes `kvm2.host`,
`kvm2.domain`, `kvm2.pool`, `kvm2.volume`, `kvm2.libvirt_code` on
failure), `kvm2.dial` around the SSH handshake and libvirt connect, and
`kvm2.download` around an image fetch; under `providerhealth.check`,
`kvm2.dial` and the four `kvm2.<Method>` spans of a check. They come
from a span decorator over the seam, applied once in the dialer, so no
RPC is unspanned. The `libvirt call` log line of an RPC is written
inside its span. The `kvm` provider has no child spans.
- Kubernetes API calls become client child spans via a wrapped
`rest.Config` transport — gated on an existing parent span, so informer
list/watch long-polls and leader-election renewals never create root
@@ -348,7 +395,8 @@ incoming trace context):
- Discovery API: one server span per request (named from the route
pattern), incoming W3C `traceparent` honored so clients' traces continue
into the operator; `/healthz` excluded.
- `gc.sweep` per GC pass; `health.probe` per probe **only** with
- `gc.sweep` per GC pass; `providerhealth.check` per round of provider
checks; `health.probe` per probe **only** with
`--trace-health-probes` (default off — probes run ~1/s per proxy).
Log correlation: logr sinks never see a context, so trace IDs ride on the
@@ -359,6 +407,64 @@ Re-deriving from the base rather than layering keeps zap from emitting
duplicate keys on nested spans. GCP V(5) wire logs get the same keys from
the slog handler's ctx.
### 11. Provider health checks (`internal/providerhealth/`)
Answers "can this cluster reach each provider's backend right now?"
without waiting for a Proxy to fail. A provider opts in by implementing
`provider.Checker`: `Check(ctx) []TargetStatus` returns one verdict per
target and never fails as a whole. All four providers do:
| Provider | Target | What one check does |
|---|---|---|
| kvm | each `kvm.hosts` entry | over the pooled SSH+libvirt connection: libvirt version and hostname, then the configured pool must exist and be running; reports pool capacity and free space ([details](kvm-provider.md#health-check)) |
| kvm2 | each `kvm2.hosts` entry | on a fresh SSH+libvirt connection dialled for the check and closed after it: libvirt version and hostname, then the configured pool must exist and be running; `details` carry `user`, `pool`, `hostname`, `libvirtVersion`, `poolState`, `poolCapacityGiB` and `poolAvailableGiB` ([details](kvm2-provider.md#health-check)) |
| kubernetes | `apiserver` | lists at most one managed Pod cluster-wide: the verb and scope `ListByTag` and GC rely on |
| gcp | the project | one page of an aggregated instance list with `maxResults=1`: credentials, Compute API enabled, `compute.instances.list` |
```text
Monitor.Start(ctx) ── leader-only; --provider-check-interval (1m; 0 = off,
│ otherwise ≥ 10s), --provider-check-timeout (15s, or
│ the interval if that is shorter)
├─ once: delete managed-by-labelled ProviderHealth objects whose provider
│ is not configured (all of them when checks are off) — the providers
│ config is read once per process
└─ at start, then every Interval: checkOnce
│ every provider's Check concurrently, under one context.WithTimeout
│ snapshot → the scrape-time gauges
│ log transitions: down or reason changed → "WARNING: provider target
│ not ready"; back up → "Provider target ready"; steady → silent
└ per provider: APIReader.Get → Create if missing → status merge patch
```
- **One status write per provider per round**, since `lastCheckTime`
moves every round. At the 10s minimum interval that is still far below
leader-election renewals. Proxy status avoids writes like this only
because its probes run once a second per proxy.
- **Reads go through the API reader**, not the cache. The objects are
cluster-scoped; a cached Get would start a cluster-wide informer (and
need `watch`) even with `--proxy-namespace`.
- **Checks bypass the provider decorators.** `cmd/main.go` collects the
Checkers before wrapping the providers in `WithTracing(WithMetrics(...))`,
so a check never counts as a provider request. KVM RPCs still appear in
the V(1) `libvirt call` trace, and each round is one
`providerhealth.check` span.
- **Not wired to `/readyz`.** A dead hypervisor must not make the manager
unready while the other providers keep working.
- **Gauges** `proxy_operator_provider_target_up`,
`proxy_operator_provider_check_duration_seconds` and
`proxy_operator_provider_check_timestamp_seconds`, each labelled
`{provider,target}`. Only the leader exports them.
Illustrative output:
```sh
$ kubectl get providerhealth # short name: ph
NAME TYPE READY TARGETS LAST CHECK AGE
kubernetes kubernetes True 1/1 12s 3d
kvm-home kvm False 0/1 12s 3d
$ kubectl get ph kvm-home -o yaml # per target: reason, message, details
```
## Decisions
Judgment calls the spec left open, and deliberate deviations — recorded so
@@ -528,3 +634,72 @@ they read as choices, not accidents. Chronological by build step.
external sites. The kube-apiserver ignores incoming `traceparent` by
design (public endpoint), so k8s client spans are leaves — in-process
traces, not cross-process ones.
- **Provider health is a status-only, cluster-scoped CRD** (user
decision). `kubectl get providerhealth` works without Prometheus, which
the deployment does not ship (its ServiceMonitor is commented out). There
is no spec: the providers config file stays the one place providers are
configured, and the object only reports on it. The gauges carry the same
verdicts for alerting.
- **KVM checks reuse the pooled connection instead of dialing fresh.** A
fresh dial would not pick up a rotated Secret anyway, since the SSH
material is cached. It would also open an sshd session per host per
minute. And a working cached connection is exactly what reconciles use.
Sharing must not cost the reconciles anything, so the check calls
through `hostPool.probe`. A check that runs out of time abandons its
RPCs instead of closing the connection, which would kill an upload or a
Create running on it. At most one abandoned check per host is left
waiting, and until it returns, the next check reports `Timeout` at once.
Waiting for a host's connection slot also ends with the caller's
deadline, so a reconcile redialing a dead host cannot stall the round.
- **Provider transitions are logs, not Events**, in line with the
EventRecorder decision above. They log at Info with a `WARNING:` prefix,
and each target's `lastTransitionTime` in the object records when it
flipped.
- **A provider name that is not a valid object name** gets metrics but no
`ProviderHealth` object, with a warning at start. Tightening config
validation instead would crash-loop existing deployments on upgrade.
- **kvm2 is a second provider type built clean-room from the contract,**
not a rewrite of `kvm` in place. Its spec
(`docs/kvm2-provider-spec.md`) and code were written from
[provider-contract.md](provider-contract.md), the kvm user guide and
go-libvirt alone, never from `internal/provider/kvm/`, as a test of
whether the contract is enough to build a provider. Both types ship,
share the `KVMConfig` block shape and the `kvm-ssh` Secret, and may
share a host.
- **kvm2 image downloads run on a dedicated connection.** A cancelled or
overlong call kills the connection it ran on (the only way to unblock
go-libvirt), so a 435 MB upload on the pooled connection could die
because a poll for another Proxy was cancelled. The transfer dials its
own connection and closes it afterwards; the cheap checks and the marker
stay on the pooled one.
- **kvm2 health checks dial fresh connections**, the opposite of the kvm
decision above. With credentials re-read on every dial a fresh
connection does pick up a rotated Secret, and a check that runs out of
time then closes only its own connection, so it cannot disrupt a
reconcile. The cost is one sshd session per host per round.
- **A cancelled kvm2 call kills the pooled connection.** Every RPC has a
budget; when a call outlives its ctx or budget the adapter closes the
socket and the SSH client, returns Transient, and marks the connection
dead so the next call redials. Other calls in flight on that host fail
once with Transient and their callers retry. Documented in the guide
rather than avoided: the alternative, leaving a hung RPC blocked
forever, pins a worker.
- **kvm2 marks domains in its own metadata namespace**, so it never lists
or adopts domains the `kvm` provider made. Switching an entry from `kvm`
to `kvm2` means deleting its Proxies first; the two types can share a
host without one collecting the other's VMs.
- **kvm2 turns pool free space into the quota signal.** libvirt reports a
full pool as an ordinary error, which would retry on the fast backoff.
Before creating any volume, `Create` compares `StoragePoolGetInfo`'s
available bytes with the image (if uncached) plus `diskSizeGb` plus 16
MiB for the seed, and returns `ErrQuotaExceeded` below that, so the
reconciler waits 5 minutes instead.
- **kvm2 classifies SSH authentication and host-key failures as
Permanent.** A key missing from `authorized_keys` or a host key missing
from or different in known_hosts needs an admin; kvm retried these
forever. A missing key or known_hosts file stays Transient, because the
optional Secret may land later.
- **kvm2 re-reads the SSH key and known_hosts on every dial**, so rotating
the Secret needs no restart; nothing is cached. The constructor only
checks that the files exist, to warn early without failing the provider
set.
docs/kvm-provider.md
+57 -2
View File
@@ -265,12 +265,65 @@ The provider logs through controller-runtime's logger, named `kvm`, with
only `length` and `duration`. There is no raw wire tier like the GCP
provider's V(5); go-libvirt exposes no hook for it.
## Health check
The operator checks every configured host every `--provider-check-interval`
(default 1m). It publishes the result in two places: the `ProviderHealth`
object named after the provider (`kubectl get ph kvm-home -o yaml`), and the
`proxy_operator_provider_target_up{provider,target}` gauge.
All hosts are checked concurrently, each over the same pooled connection
that Create and Get use:
1. The SSH and libvirt handshakes. These only happen on the first check, or
after the link dropped.
2. `virConnectGetLibVersion` and `virConnectGetHostname`.
3. `virStoragePoolLookupByName` and `virStoragePoolGetInfo` on the host's
`pool`.
A passing target's `details` carry `hostname`, `libvirtVersion`, `pool`,
`poolState`, `poolCapacity` and `poolAvailable` (in GiB). The free space is
the closest thing this provider has to a capacity signal. Every target also
shows its SSH `user`. A failing target says where the check broke:
| Reason | Meaning | Look at |
|---|---|---|
| `SSHMaterialUnavailable` | the key or known_hosts file is missing or does not parse | the `kvm-ssh` Secret (see *Cluster-side setup*) |
| `Unreachable` | the TCP connect failed, or the link died mid-check | DNS, Tailscale or a firewall between the cluster and `address` |
| `SSHHandshakeFailed` | the host key is missing from known_hosts or has changed, or `user` rejected the key | the known_hosts entry; `user`'s `authorized_keys` |
| `LibvirtConnectFailed` | SSH worked, the libvirt socket did not | `user` in the `libvirt` group, `libvirtd` running, the `socket` path |
| `Timeout` | no answer within `--provider-check-timeout` (default 15s), or the previous check of the host is still waiting | a hung `libvirtd`, a saturated link, or a reconcile stuck redialing the host |
| `PoolNotFound` | there is no pool named `pool` | `virsh pool-list --all` |
| `PoolNotActive` | the pool exists but is not running | `virsh pool-start <pool>` and `virsh pool-autostart <pool>` |
| `LibvirtError` | any other libvirt error | the message carries libvirt's own text |
Sharing the connection has two consequences:
- The leader now keeps one SSH connection per host open from startup, with a
keepalive every 15s. Before, the first connection opened with the first KVM
reconcile.
- A check that runs out of time gives up on its RPCs but leaves the
connection alone, so an image upload or a Create running on it is not cut
off. At most one abandoned check per host keeps waiting; until it
returns, the next check reports `Timeout` straight away. A link that is
really dead is closed by the keepalive within about two keepalive
intervals.
- A check never waits longer than its timeout for the host's connection,
even while a reconcile holds it to redial a host that is down.
## Known gaps
- **No quota or capacity detection.** libvirt reports an out-of-space
pool or an over-committed host as an ordinary error; the reconciler
treats it as transient and retries with backoff rather than the slower
quota backoff.
quota backoff. The health check does report each pool's free space
(`poolAvailable`), so a pool that is filling up shows before Creates
start failing.
- **A rotated `kvm-ssh` Secret needs a manager restart.** The key and
known_hosts files are read on first use and cached for the life of the
process. Replacing them in the Secret therefore changes nothing for
later connections, health checks included. After a host key change, the
check keeps reporting `SSHHandshakeFailed` until the manager restarts.
- **Orphan volumes without a domain are invisible to GC.** A crash
between creating the volumes and defining the domain leaves an overlay
and seed that `ListByTag` cannot see (it lists domains). They are
@@ -315,4 +368,6 @@ Unit tests run over an in-memory fake of the libvirt API. The optional
integration test against a real host is described in
[testing.md](testing.md#the-kvm-integration-test-real-hypervisor); its
`-v` output includes the `libvirt call` trace above, at the verbosity
`KVM_TEST_LOG_LEVEL` selects (default 1).
`KVM_TEST_LOG_LEVEL` selects (default 1). `TestKVMCheck_realHost` runs the
health check against the same host and creates nothing:
`go test -tags=integration -count=1 -v -run TestKVMCheck ./internal/provider/kvm/`.
docs/kvm2-provider-spec.md
+665
View File
@@ -0,0 +1,665 @@
# Provider spec: kvm2
Filled from [provider-spec-template.md](provider-spec-template.md) against
[provider-contract.md](provider-contract.md). This is the design the `kvm2`
package is built from. Where an entry names an axis in backticks, the answer is
one of that axis's profiles in contract §4. Nothing in this spec was taken from
the code of the existing `kvm` provider; backend facts come from
[kvm-provider.md](kvm-provider.md), the execution log of the first build, the
go-libvirt module in the module cache, and the beelink host.
## Header
- **Type**: `kvm2`
- **Example configured name**: `kvm2-home`
- **Backend and API version this spec was written against**: libvirt 11.6.0 and
QEMU 10.1.0 on Ubuntu (host `beelink`), reached with go-libvirt
`github.com/digitalocean/go-libvirt v0.0.0-20260814190004-1a83157e1858` over an
SSH-forwarded unix socket.
- **Client**: `rpc-over-ssh`. go-libvirt's RPC client (`libvirt.New(conn)`
then `Connect()`) over a `direct-streamlocal` channel opened with
`golang.org/x/crypto/ssh` to the host's libvirt socket. Both modules are
already direct requires in `go.mod`; no new dependency.
- **Author and date**: Claude (orchestrator session), 2026-09-16.
## 1. Identity and providerID
**Answer:** `hosts/<host>/domains/<name>`, where `<host>` is `hosts[].name`
from the config block and `<name>` is `req.Name`. Parsing is strict: exactly
four segments, literal `hosts` and `domains`, non-empty host and name, no
further slashes; anything else is `ErrPermanent` from `Get` and `Delete`. Host
names are validated at config load by the shared `kvmIdentifier` regexp in
`internal/provider/config_kvm.go` (letters, digits, dots, dashes, underscores),
so a host name can never contain the separator. Instances do not migrate
between hosts (libvirt domains are per host), so the location belongs in the
ID.
**Evidence:** contract §6; `kvmIdentifier` in `internal/provider/config_kvm.go`.
## 2. Primary key (`primary-key`)
**Answer:** `name-is-key`. libvirt addresses domains by name
(`DomainLookupByName`); `req.Name` is used verbatim as the domain name and as
the base of the per-VM volume names. Create looks the name up first and adopts
an existing domain only if it carries the kvm2 mark (`managed` true, entry 8)
and its metadata `uid` is empty or equals `req.UID`. A domain without the
mark returns `ErrPermanent` saying it was not created by this provider: kvm2
never starts or takes over a VM someone else defined under that name. A
different non-empty `uid` returns `ErrPermanent` naming both UIDs. This
closes the contract's "adoption is not verified" limit for this provider.
**Evidence:** `DomainLookupByName(Name string) (Domain, error)` in the
go-libvirt module; contract §2.1 idempotency and §13.
## 3. Placement mapping
**Answer:** three fields are catalogue keys checked before any backend call, one is ignored:
| Field | Meaning | Required | Unknown value |
|---|---|---|---|
| `Zone` | a `hosts[].name` | only when more than one host is configured; with exactly one host an empty zone selects it | `ErrPermanent`: `placement.zone "x" is not in kvm2.hosts` |
| `MachineType` | a key of `machineTypes` (`vcpus`, `memoryMiB`) | yes | `ErrPermanent`: `placement.machineType "x" is not in kvm2.machineTypes` |
| `Image` | a key of `images`, or an absolute http(s) URL used directly | yes | `ErrPermanent`: `placement.image "x" is neither a key in kvm2.images nor an http(s) URL` |
| `Region` | ignored | | still part of the spec hash: changing it replaces the VM |
All three checks run before any backend call. The guide documents the ignored
field and that catalogue edits affect only new VMs.
**Evidence:** contract §2.1 (reject early, MAY ignore with documentation);
`IsHTTPURL` in `internal/provider/config_kvm.go`.
## 4. Image source
**Answer:** a catalogue in the config block (`images.<key>.url`, optional
`sha256`, `format` qcow2 or raw, `firmware` uefi or bios) or a URL given
directly in `placement.image`. The provider downloads the image into the
host's storage pool on first use and caches it there as a base volume named
`base-<entry>-<key>.<format>` (for a URL: `<key>` is the first 16 hex of
sha256(url)), where `<entry>` is the provider entry's name, with a sidecar
marker volume `base-<entry>-<key>.<format>.sha256` holding the digest of
the verified upload, written last. The entry is in the name because the `kvm`
provider builds `base-<key>.<format>` in the same pool from the same catalogue
shape, and two entries pointing one key at different URLs would otherwise share
a file; the cost is one copy per entry per host, and the entry's name is checked
against `kvmIdentifier` at construction. A base volume without a marker is
incomplete and is deleted and redone. When `sha256` is configured and differs
from the marker's content, the image is **not** replaced: that is `ErrPermanent`
naming both digests, because overlays back onto the base file and libvirt
unlinks it under running domains without complaint. The download streams
from the URL through the operator into `StorageVolUpload` with a running
sha256; nothing is buffered. `Content-Length` is required because the volume's
capacity must be declared before the upload; a response without it is
`ErrPermanent` naming the URL. Per `(host, volume)` a keyed mutex makes
concurrent Creates download once. The download runs on a dedicated
connection to the host, not the pooled one, so a cancelled poll on another
Proxy cannot cut it off. After a successful upload `StoragePoolRefresh` makes
the file usable as a backing image.
Catalogue edit warning for the guide: overlays back onto the base file, so
evicting a base volume while Proxies use it corrupts those VMs. A `sha256`
change is refused for that reason; a `url` change is a silent no-op, since the
cache is keyed by entry and key. Add a new key for either.
**Evidence:** kvm-provider.md "The image cache" and "Known gaps"; execution log
of 2026-09-04 (raw qcow2 upload plus pool refresh accepted by libvirt 11.6;
435 MB trixie image); contract §1.4 (stream, no buffering) and §2.1
(catalogue changes).
## 5. Cloud-init delivery
**Answer:** a NoCloud seed ISO built in-process with
`github.com/kdomanski/iso9660`, volume label `cidata`, files `user-data` (the
Proxy's `CloudInit`, byte for byte, never parsed), `meta-data`
(`instance-id: <name>` and `local-hostname: <name>`, so a reboot is never taken
for a new instance) and `vendor-data`. The ISO is written into a temp
directory created under the default temp dir (`/tmp`, a 64Mi emptyDir in the
manager), uploaded as a raw volume `<name>-seed.iso` and attached as a CD-ROM.
The seed is rebuilt on every Create attempt: a previous attempt may have been
cut off mid-upload, and a truncated seed would boot a VM without cloud-init.
`vendor-data` carries a shell script that installs and starts
`qemu-guest-agent` with whichever of apt-get, dnf, yum, zypper or apk the image
has. It is a script and not a cloud-config on purpose: cloud-init merges a
vendor-data cloud-config under the user-data with mergers that keep the
user-data's value for every top-level key both define and never append lists,
so a Proxy with its own `packages:` would silently lose the agent. Scripts
bypass the merge. When the config's `vendorData` is set, the two travel as
separate parts of a MIME multipart (`text/plain` for the configured document,
`text/x-shellscript` for the agent script), headed by `MIME-Version` within
the first bytes. With `installGuestAgent: false` and no `vendorData` the file
is empty. Size: cloud-init has no practical limit for NoCloud; the ISO is a
few hundred KB.
**Evidence:** kvm-provider.md "What lands on the host"; execution log run 7
(vendor-data as YAML silently dropped) and run 8 (script survives); contract
§2.1 (byte for byte, never logged) and §11 (sentinel test).
## 6. Machine type mapping
**Answer:** a catalogue: `machineTypes.<key>` is `{vcpus, memoryMiB}`,
validated at load (both greater than zero, at least one profile). Disk size is
provider-wide: `diskSizeGb` (default 10) is the capacity of every VM's overlay
volume, thin-provisioned; cloud-init's growpart expands the root filesystem
into it.
**Evidence:** `KVMMachineType` and `validateKVM` in
`internal/provider/config_kvm.go`.
## 7. Network and IP discovery (`readiness`, `ip-source`)
**Answer:** `readiness`: `provisioning-window` (boot plus cloud-init, plus the
image download on a cold pool). `ip-source`: `discovered`. The VM gets one
virtio NIC on `hosts[].bridge`, which must lead to a LAN with DHCP that the
cluster can route to. The address comes from `DomainInterfaceAddresses`, guest
agent source first, then the ARP table (on beelink the agent answered about
20 s after start where ARP never did, because the host itself does not talk to
the guest). The first IPv4 that is not loopback, link-local or unspecified is
reported. One source failing means "no address yet" and `Get` returns
`StateProvisioning`, which is the ordinary boot on beelink: the agent is not up
while ARP answers with nothing. A connection already gone, or every source
failing, is `ErrTransient` instead — a running VM whose address cannot be read
is not a VM without an address, and reporting it as one would make the
reconciler clear its IP and the health engine drop its counters. The guest needs the agent: it is installed by the
vendor-data script (entry 5) unless `installGuestAgent: false`, which needs an
image that ships it or a network where ARP sees guest traffic. `Port` is not
used by the provider; nothing on the host filters it. Reachability is a
prerequisite in the guide: the operator's pod network and the clients must
route to the bridge's LAN.
**Evidence:** kvm-provider.md "Hypervisor prerequisites" and "Known gaps"
(IP discovery); execution log run 6 (agent 20 s, ARP never); contract §1.3
and §2.2 (Running only with IP).
## 8. GC marks and CreatedAt (`gc-mark`, `created-at`)
**Answer:** `gc-mark`: `written-metadata`. `created-at`: `written-at-create`.
The domain XML passed to `DomainDefineXML` carries, under `<metadata>`, an
element `<instance xmlns="https://gitea.home.hrajfrisbee.cz/kacerr/egress-proxies-operator/kvm2">`
with children `<managed>true</managed>`, `<uid>req.UID</uid>` and
`<createdAt>RFC3339 UTC</createdAt>`. The mark lands in the same call that
defines the domain, so there is no mark window. Read back with
`DomainGetMetadata(dom, DomainMetadataElement, uri, DomainAffectCurrent)`;
libvirt 11.6 returns the element without its namespace declaration, so the
parser matches on the local name `instance` only (the URI was already used to
select it). A domain without the element is not managed and reports an empty
UID. A `createdAt` that does not parse gives the zero time, which GC treats as
old; that is accepted. The volumes carry no mark (documented GC blind spot,
entry 24). The namespace is kvm2's own, so kvm2 never lists or adopts domains
made by the `kvm` provider; the two types may share a host.
**Evidence:** execution log run 3 (namespace asymmetry);
`DomainGetMetadata(Dom Domain, Type int32, Uri OptString, Flags DomainModificationImpact)`
in the go-libvirt module; contract §2.1 (mark in the create call) and §4.5.
## 9. State mapping (`provisioning-state`)
**Answer:** `provisioning-state`: `create-blocks`. Create defines and starts the
domain in one call and returns only once `DomainGetState` reports running, so
`Get` never sees a domain kvm2 defined but did not start. A domain that is
shut off later stopped on its own or was stopped by someone, and Stopped is
the right answer.
| Backend state (`DomainGetState`) | Lock or task | Provider state |
|---|---|---|
| nostate (0) | none | Provisioning (libvirt could not determine it; next poll decides) |
| running (1), blocked (2) | no IP found | Provisioning |
| running (1), blocked (2) | IP found | Running |
| paused (3), shutdown (4), shutoff (5), crashed (6), pmsuspended (7) | none | Stopped |
| any other code | none | Stopped |
libvirt has no locks or tasks; every RPC is synchronous.
**Evidence:** go-libvirt `DomainState` constants; contract §2.2 and §4.11.
## 10. Create: blocking and task tracking (`create-return`, `async-ops`)
**Answer:** `create-return`: `blocking`. `async-ops`: `none`. Create returns
when the domain is running, holding one reconcile worker for the duration,
including the first image download on a host (minutes on a cold pool; the
guide says so). The wait after `DomainCreate` is capped at 15 seconds:
`DomainCreate` is synchronous in libvirt, so the first `DomainGetState`
normally answers running and the cap only covers state settling. Two rules
keep the wait from pinning a worker: a state after the start that is neither
active nor `nostate` (shutoff, crashed, paused, pmsuspended, shutdown) is
`ErrTransient` at once, since polling will not bring that guest up; and a
domain found paused or pmsuspended before the start is `ErrPermanent`
(`cannot be started by the operator; resume or destroy it on the host`), an
admin's doing that `DomainCreate` would refuse and the reconciler must not
retry. Every RPC has its own deadline (entry 18).
Steps, each idempotent, in order, all under the per-name lock (entry 19):
1. Validate placement (entry 3). No backend call yet.
2. `DomainLookupByName`. Found: adoption guard (entry 2), then start it if it
is not running (paused or pmsuspended is `ErrPermanent`; `DomainCreate`;
`ErrOperationInvalid` means already running), poll as in step 9, return
the ID.
3. Pool space precheck (entry 13, quota).
4. Ensure the base image (entry 4): marker present and matching means done.
5. Overlay volume `<name>.qcow2`, qcow2 backed by the base file with
`diskSizeGb` capacity; `ErrStorageVolExist` means reuse (creation is
atomic, so an existing overlay is complete).
6. Seed volume `<name>-seed.iso`: delete if it exists, create, upload
(entry 5). Always rebuilt.
7. `StorageVolGetPath` for the overlay and the seed.
8. `DomainDefineXML` with file-type disks (entry 15's AppArmor note), the
metadata element, firmware per image.
9. `DomainCreate`, then poll `DomainGetState` until running: `nostate` is
polled again, any other inactive state is `ErrTransient` at once, and the
15-second cap is `ErrTransient` too.
10. Return `hosts/<host>/domains/<name>`.
A retry after a crash finds whatever the previous attempt got to: an existing
overlay is reused, a seed is rebuilt, a defined domain is started, a running
domain is returned. `ErrNoDomain` and `ErrNoStorageVol` inside Create (a
volume vanishing between steps) are `ErrTransient`, never `ErrNotFound`. The
log line for a Create records `cloudInitBytes`, never the content.
**Evidence:** contract §2.1, §4.2, §5 ("do not return NotFound from Create");
execution log run 1 (AppArmor).
## 11. Delete and secondary resources (`delete-return`, `secondary-resources`)
**Answer:** `delete-return`: `waits`. `secondary-resources`: `explicit`. Under
the per-name lock:
1. Parse the ID (`ErrPermanent` if malformed). A host no longer in the config
is a logged no-op success: the provider cannot reach it and refusing would
pin the finalizer; the guide tells admins to delete Proxies before retiring
a host.
2. `DomainLookupByName`; `ErrNoDomain` means the domain is already gone, skip
to step 5.
3. `DomainDestroy`; `ErrOperationInvalid` (not running) and `ErrNoDomain` are
success.
4. `DomainUndefineFlags` with `DomainUndefineNvram`,
`DomainUndefineManagedSave` and `DomainUndefineSnapshotsMetadata`
(UEFI domains keep NVRAM; undefine refuses without the flag); `ErrNoDomain`
is success.
5. Delete `<name>.qcow2` and `<name>-seed.iso` by name from the host's pool;
`ErrNoStorageVol` is success, and `ErrNoStoragePool` means nothing is left
to clean.
6. Return nil. The base image and its marker stay (shared cache).
Everything is gone when Delete returns nil, so `Get` afterwards is NotFound.
A second Delete during the first waits on the lock and then finds nothing.
Volumes that exist without a domain (a crash between step 5 and 8 of Create)
are removed by the same Proxy's next Delete but are invisible to GC (entry 24).
**Evidence:** kvm-provider.md "Known gaps"; execution log run 4 (NVRAM);
contract §2.3.
## 12. ListByTag scope and partial failure (`list-partial`)
**Answer:** `fail-whole`, per host, with single unreadable domains skipped.
Scope: every host in `hosts`, in config order, `ConnectListAllDomains` with
active and inactive domains. For each domain, read the metadata element; keep
those with `managed` true and build the `Instance` from the domain name and
the element alone: `ID`, `UID` and `CreatedAt`, which is all GC reads
(contract §2.4). `State` and `IP` are left empty; no state or address RPC is
made. A domain whose metadata cannot be read (mid teardown, corrupt element)
is skipped with a V(1) line, unless ctx ended, which fails the call; a host
that cannot be dialled or whose listing fails fails the whole call, so GC
skips this provider for the sweep rather than act on a partial view.
Filtering is client-side; libvirt has no label index. Cost: exactly one
`DomainGetMetadata` per domain per sweep, about half a millisecond each over
the tunnel.
**Evidence:** execution log run 6 (per-domain metadata cost); contract §2.4
and §4.7.
## 13. Error classification (`not-found`, `quota`)
**Answer:** `not-found`: `error-code`. `quota`: `detected`, from a pool-space
precheck rather than a backend error: before creating any volume, Create reads
`StoragePoolGetInfo` and compares `available` with the bytes it is about to
commit: the image's `Content-Length` when the base is not cached, plus
`diskSizeGb` in bytes, plus 16 MiB for the seed. Below that it returns
`ErrQuotaExceeded` naming the pool, available and needed, and the reconciler
waits `QuotaRetry`.
One `classify(op string, err error)`:
| Backend response | Class | Why |
|---|---|---|
| `libvirt.Error` code `ErrNoDomain`, `ErrNoStorageVol` | NotFound (Transient when `op == "create"`) | the object is gone; inside Create it vanished mid-call |
| `ErrNoStoragePool` | Permanent | config names a pool that does not exist |
| `ErrXMLError`, `ErrNoSupport`, `ErrInvalidArg` | Permanent | the request itself is wrong |
| `ErrAuthFailed`, `ErrAccessDenied` | Permanent | the SSH user may not use the socket |
| `ErrOperationInvalid` | Transient (Delete and Create treat the specific cases as success before classify) | state race; the next poll decides |
| any other `libvirt.Error` | Transient | unknown means retry |
| SSH: `ssh: unable to authenticate` (from `x/crypto/ssh`) | Permanent | the key is not in `authorized_keys` |
| SSH: `knownhosts.KeyError` with a known key that differs, or with no known key | Permanent | host key changed or missing from known_hosts; the admin must act |
| SSH key or known_hosts file missing (`fs.ErrNotExist`) | Transient | the Secret has not landed yet |
| SSH key present but does not parse | Permanent | wrong file in the Secret |
| `context.Canceled`, `context.DeadlineExceeded`, `net.OpError`, `io.EOF`, `io.ErrUnexpectedEOF` | Transient | network or cancellation |
| HTTP 404, 403, 410 on the image URL; missing `Content-Length`; sha256 mismatch | Permanent | the catalogue entry is wrong |
| HTTP 5xx, 429, network errors on the image URL | Transient | retry |
| an error already wrapped with a class (`errors.Is` any sentinel) | that class | validation and builders classify at the source |
| anything else | Transient | contract default |
The unit test table uses a `libvirt.Error` per code, the real SSH error
strings produced by `x/crypto/ssh` against the in-process server, and a real
`knownhosts.KeyError`.
**Evidence:** go-libvirt `ErrorNumber` constants and `Error{Code uint32, Message}`
(value receiver, so `errors.As` uses a `libvirt.Error` value); contract §5;
kvm-provider.md "Known gaps" (SSH auth retried forever in v1: closed here).
## 14. Credentials and TLS (`credentials`)
**Answer:** `mounted-files`. The config block names `sshKeyFile` (private key,
OpenSSH or PEM, parsed with `ssh.ParsePrivateKey`) and `knownHostsFile`
(OpenSSH `known_hosts`, parsed with `knownhosts.New`), both expected under the
existing optional `kvm-ssh` Secret mount at `/etc/proxy-operator/kvm`
(`config/manager/manager.yaml`; the mount is shared with the `kvm` provider and
its comment says so). `insecureIgnoreHostKey: true` replaces known_hosts with
`ssh.InsecureIgnoreHostKey()` and is logged as a warning by `New`. Both files
are read on every dial, so rotating the Secret takes effect on the next
connection without a restart. `New` tries to read them once to log a warning
when they are unusable, but never fails on them and never dials.
Minimum privilege: the SSH user must be able to open the libvirt socket
(member of the `libvirt` group); libvirt itself is used as `qemu:///system`
through the socket, so no further host permissions.
TLS: none (SSH provides transport security); `system-roots` for the image
download's HTTPS, which uses the distroless image's CA bundle.
Hygiene: the private key bytes and the known_hosts content never appear in
logs, spans or error strings; errors from parsing say "parsing ssh key file
<path>" plus the parser's message. Tested with a sentinel placed in a bad key
file.
**Evidence:** kvm-provider.md "Cluster-side setup"; execution log run 8
(lazy load: a missing Secret must not abort the provider set); contract §7.
## 15. Config block
**Answer:** the block is `kvm2:` and reuses the Go type `provider.KVMConfig`
from `internal/provider/config_kvm.go` unchanged, so an existing `kvm-home`
entry ports by changing `type: kvm` to `type: kvm2` and `kvm:` to `kvm2:`.
Validation calls the shared validator with the block name `kvm2` so messages
read `kvm2.hosts` and not `kvm.hosts`.
| Field | Required | Default | Validation | Ends up in |
|---|---|---|---|---|
| `sshKeyFile` | yes | | non-empty | dial |
| `knownHostsFile` | unless `insecureIgnoreHostKey` | | non-empty | dial |
| `insecureIgnoreHostKey` | | false | | dial, warning at `New` |
| `diskSizeGb` | | 10 | not negative | overlay capacity, quota precheck |
| `vendorData` | | | | seed `vendor-data` |
| `installGuestAgent` | | true | | seed `vendor-data` |
| `images.<key>` | | | key matches `kvmIdentifier` | volume name `base-<entry>-<key>.<format>` |
| `images.<key>.url` | yes per entry | | http(s) URL | download |
| `images.<key>.sha256` | | | 64 hex | marker comparison |
| `images.<key>.format` | | qcow2 | qcow2 or raw | base volume XML, overlay backing format |
| `images.<key>.firmware` | | uefi | uefi or bios | domain XML `<os>` |
| `machineTypes.<key>` | at least one | | vcpus and memoryMiB greater than zero | domain XML |
| `hosts[].name` | yes | | unique, matches `kvmIdentifier` | providerID segment |
| `hosts[].address` | yes | | non-empty `host:port` | TCP dial |
| `hosts[].user` | yes | | non-empty | SSH |
| `hosts[].socket` | | `/var/run/libvirt/libvirt-sock` | | SSH channel target |
| `hosts[].pool` | | `default` | | every storage RPC |
| `hosts[].bridge` | yes | | non-empty | domain XML NIC |
Wiring: `ProviderConfig.KVM2 *KVMConfig` with tag `kvm2,omitempty`;
`setBlocks` reports `kvm2`; the allow-list in `Config.validate` and the
constructor map in `cmd/main.go` both gain `kvm2` in the same commit.
**Evidence:** `KVMConfig`, `validateKVM`, `kvmIdentifier` in
`internal/provider/config_kvm.go`; contract §10.
## 16. Checker
**Answer:** one target per `hosts[]` entry, `Name` the host name, `Address`
the SSH address, checked concurrently in config order on a fresh dedicated
connection each time (dial, `ConnectGetLibVersion`, `ConnectGetHostname`,
`StoragePoolLookupByName`, `StoragePoolGetInfo`, close). It never touches the
pooled connection, so a check that runs out of time closes only its own
connection and cannot disrupt a reconcile. Cost: one SSH handshake and four
cheap RPCs per host per round.
Reasons: `Reachable`; `Timeout` (the round's ctx ended before the check
finished, whichever stage was running, the dial included, or an RPC's own
budget ran out once the connection was up); `Unreachable` (TCP connect or the
SSH transport failed, including on the dial's own 10 s budget while the round
still had time, or the link died mid-check); `Unauthorized` (SSH authentication refused, host key unknown or
changed); `Forbidden` (`ErrAccessDenied` or `ErrAuthFailed` from libvirt: the
socket refused the user); `APIError` (any other libvirt error);
`CredentialsUnavailable` (the key or known_hosts file is missing or
unparsable, i.e. the Secret is absent or wrong: fixed in the cluster, not on
the host); `PoolNotFound`; `PoolNotActive`.
Details on success: `user`, `hostname`, `libvirtVersion` (decoded
major.minor.micro), `pool`, `poolState`, `poolCapacityGiB`, `poolAvailableGiB`
(the closest thing to a capacity signal, and the same number the quota
precheck reads). The `status.type` comment in
`api/v1alpha1/providerhealth_types.go` gains `kvm2` and the CRD is
regenerated.
**Evidence:** contract §8; `StoragePoolGetInfo` returning state, capacity,
allocation, available; `ConnectGetLibVersion` returning major*1e6 + minor*1e3
+ micro.
## 17. Client and child spans (`client-kind`, `child-spans`)
**Answer:** `client-kind`: `rpc-over-ssh`. `child-spans`: `span-decorator`.
The seam is an interface over exactly the go-libvirt methods the provider
uses, with `ctx` as the first parameter of every method (go-libvirt's own
methods take none and block): `DomainLookupByName`, `DomainDefineXML`,
`DomainCreate`, `DomainDestroy`, `DomainUndefineFlags`, `DomainGetState`,
`DomainGetMetadata`, `DomainInterfaceAddresses`, `ConnectListAllDomains`,
`StoragePoolLookupByName`, `StoragePoolRefresh`, `StoragePoolGetInfo`,
`StorageVolLookupByName`, `StorageVolGetPath`, `StorageVolCreateXML`,
`StorageVolUpload`, `StorageVolDownload`, `StorageVolDelete`,
`ConnectGetLibVersion`, `ConnectGetHostname`, plus `Close`. One production
adapter wraps `*libvirt.Libvirt` and applies the per-call deadline
(entry 18); one in-memory fake implements the seam for unit tests. A dialer
seam (`dial(ctx, host) (backend, error)`) separates the SSH transport so tests
inject fakes and an in-process SSH server test covers the real dialer.
The decorator is applied once, inside the real dialer, at the point where a
connection is created, so provider code never sees an undecorated backend. Per
call it starts a span `kvm2.<Method>` (for example `kvm2.DomainDefineXML`) with
attributes `kvm2.host` and the identifiers passed (`kvm2.domain`, `kvm2.pool`,
`kvm2.volume`), records the libvirt error code on failure, and inside the span
writes the V(1) log line, so the line carries that span's ID. Two more spans:
`kvm2.dial` around the SSH handshake and libvirt connect (attributes address
and user) and `kvm2.download` around an image fetch (attributes URL host,
bytes, whether a sha256 was configured). All spans are children of the
decorator's `provider.create`, `provider.get`, `provider.delete`,
`provider.list` or the monitor's `providerhealth.check`. No CloudInit, key
material or upload bodies in attributes. A reflection test walks the seam
interface and asserts every method produces exactly one span and one log
line.
**Evidence:** contract §4.15, §4.18, §9; `tracing.Start` in
`internal/tracing/logger.go`.
## 18. Timeouts (`timeouts`)
**Answer:** `per-call`. The adapter runs each blocking go-libvirt call in a
goroutine and waits on the result, `ctx.Done()` and a timer:
| Call | Budget |
|---|---|
| TCP connect plus SSH handshake plus libvirt connect | 10 s |
| ordinary RPC | 30 s |
| `DomainCreate` | 90 s |
| `StorageVolUpload` (seed) | 60 s |
| `StorageVolUpload` (image), HTTP download | ctx, capped at 20 m |
| wait for running after `DomainCreate` | 15 s (entry 10) |
| HTTP client for images | no client timeout; the per-request ctx carries the cap |
When a call is abandoned (ctx cancelled or budget exceeded), the adapter
closes the underlying connection, because that is the only way to unblock
go-libvirt, and marks it dead so the next call redials; the abandoned call
returns `ErrTransient` wrapping the ctx or deadline error. Consequence,
documented in the guide: other RPCs in flight on that host's pooled connection
fail once with Transient and their callers retry. The image download uses a
dedicated connection so the expensive case (a cancelled poll killing a
435 MB upload) cannot happen. The keepalive (`keepalive@openssh.com` request
every 15 s) marks the connection dead on failure.
**Evidence:** contract §1.5 (no caller deadline; a hung ListByTag stalls GC),
§4.16; go-libvirt methods take no ctx.
## 19. Concurrency
**Answer:** shared state is the host connection table (one pooled connection
per host, dialled lazily, under a mutex with a per-host dial mutex so
concurrent callers dial once and share the result), the per-`(host, volume)`
download locks, and a lock per instance name. Both lock sets are one instance
each of `keyedLocks`: a one-slot semaphore per key, so a waiter gives up when
its ctx ends, reference-counted so an entry is removed from the map once no
caller holds or waits for it and the map does not grow with every name ever
seen, while two waiters on one key always share one semaphore. Create and
Delete for the same name run under that name's lock, so two Deletes (GC and
the reconciler) never race on destroy and undefine, and a Delete cannot
interleave with a Create's steps. Get and ListByTag take no name lock. go-libvirt multiplexes
RPCs on one connection, so calls for different names proceed concurrently
on the shared connection. Three reconciles, a GC sweep and a check round at
once are safe; the check uses its own connections.
**Evidence:** contract §1.2 (concurrency), §4.15.
## 20. Logging (`logging`)
**Answer:** `tiered`. Logger from `logf.FromContext(ctx)` on every call,
named `kvm2`, with `provider` on every line and `host` and `name` on every
line about an instance.
- V(1): one line per operation outcome (`kvm2 domain ensured running`,
`kvm2 domain fetched` with state, ip and uid, `kvm2 domain deleted`,
`kvm2 domains listed` with counts), one `libvirt call` line per RPC from the
decorator (`rpc`, identifiers, `duration`, on failure `error`, `libvirtCode`
and `libvirtError` name), connection events (dialled, dropped with reason),
and a line per failed backend call with the raw error.
- V(2): the same lines also carry domain and volume XML, the metadata element,
interface addresses, listed domain names, and the resolved Create parameters
(`cloudInitBytes`, never the content).
- No V(5) wire tier; go-libvirt exposes no hook for it.
- Never logged at any level: `CloudInit`, key material, known_hosts content,
upload or download bodies.
**Evidence:** contract §4.17, §9.
## 21. Tests (`backend-test`)
**Answer:** `backend-test`: `integration-tag`. Unit tests run over the
in-memory fake behind the seam (and an in-process `x/crypto/ssh` server for
the dialer); the integration test is `kvm2_integration_test.go` behind
`//go:build integration`, skipped unless `KVM2_TEST_HOST`, `KVM2_TEST_USER`,
`KVM2_TEST_SSH_KEY`, `KVM2_TEST_BRIDGE` and `KVM2_TEST_IMAGE_URL` are set,
optional `KVM2_TEST_KNOWN_HOSTS` (without it host keys are not verified and the
log says so), `KVM2_TEST_IMAGE_SHA256`, `KVM2_TEST_POOL`, `KVM2_TEST_SOCKET`,
`KVM2_TEST_TIMEOUT` (default 20m), `KVM2_TEST_KEEP`, `KVM2_TEST_LOG_LEVEL`;
run with `make test-integration-kvm2`. It asserts, in order: `New` returns
without dialling; `Check` reports the host Ready with details; `Create`
returns `hosts/<host>/domains/<name>`; a second `Create` returns the same ID
and the host's domain count is unchanged; `Get` reaches Running with an IPv4
within the budget and its `UID` and `CreatedAt` equal what Create wrote;
`ListByTag` contains it; `Delete` returns nil; `Get` is `ErrNotFound`;
`Delete` again returns nil; the pool holds the base image and its marker and
no `<name>.qcow2` or `<name>-seed.iso`; with an in-memory span recorder,
`kvm2.DomainDefineXML` and `kvm2.dial` spans are children of
`provider.create`.
Contract §11 list, with the test that covers each item:
- [x] repeat `Create`: `TestCreate_repeatReturnsSameIDAndCreatesNothing`
- [x] `Get` after `Delete`: `TestGet_afterDeleteReturnsNotFound`
- [x] `Delete` twice: `TestDelete_twiceSucceeds`
- [x] malformed ID: `TestGet_malformedIDIsPermanent`, `TestDelete_malformedIDIsPermanent`, `TestParseProviderID_malformed`
- [x] unknown state maps to `StateStopped`: `TestMapState_table`
- [x] `StateRunning` requires an IP: `TestGet_runningWithoutIPIsProvisioning`, `TestGet_runningWithIPIsRunning`
- [x] `UID` and `CreatedAt` read back: `TestGet_readsUIDAndCreatedAtFromBackend`
- [x] `ListByTag` excludes unmarked instances: `TestListByTag_excludesUnmarkedDomains`, `TestListByTag_emptyUIDStillListed`
- [x] `Get` during an in-progress delete: `TestGet_duringDeleteStillReturnsInstance`
- [x] `Get` during an asynchronous operation or lock: `TestGet_backendBusyIsTransientNeverNotFound`
- [x] never started is never `StateStopped`: `TestCreate_returnsOnlyWhenRunning`
- [x] cancelled `ctx` returns promptly: `TestProvider_cancelledContextReturnsPromptly`, `TestCall_cancelUnblocksRPC`
- [x] `classify` table with captured responses: `TestClassify_table`
- [x] Checker reasons, expired `ctx`, stable order: `TestCheck_reasonsTable`, `TestCheck_expiredContextReturnsPromptly`, `TestCheck_oneEntryPerHostInConfigOrder`
- [x] sentinel tests for `CloudInit` and the credential: `TestCreate_cloudInitSentinelNeverLoggedOrInErrors`, `TestSSH_keyMaterialNeverInErrorsOrLogs`
- [x] child spans under the operation span: `TestObserve_childSpansUnderParent`, `TestObserve_everySeamMethodInstrumented`
- [x] real-backend lifecycle test: `TestKVM2_lifecycle` and `TestKVM2Check_realHost` in `kvm2_integration_test.go`
Further tests per file are listed in the plan
(`docs/plans/2026-09-16-1333-kvm2-provider.md`).
**Evidence:** contract §11; docs/testing.md conventions.
## 22. Deployment
**Answer:** the existing `kvm-ssh` mount is reused; the rest is samples and a Makefile target:
- `config/manager/manager.yaml`: no new mount; the existing optional `kvm-ssh`
Secret at `/etc/proxy-operator/kvm` serves both `kvm` and `kvm2`, and its
comment says so. `/tmp` is already an emptyDir.
- `config/manager/providers_config.yaml` and
`config/samples/providers-config.yaml`: a commented `kvm2-home` entry, the
same block as `kvm-home` under `kvm2:`.
- `config/samples/proxy_kvm2.yaml`, listed in `config/samples/kustomization.yaml`:
provider `kvm2-home`, zone `beelink` (matching the shipped config, unlike the
`kvm` sample's `nas1`), machine type `small`, image `debian-13`, a cloud-init
that installs Squid.
- Makefile: `test-integration-kvm2` (`go test -race -tags=integration -count=1
-v -timeout 40m -run TestKVM2 ./internal/provider/kvm2/`). The Secret helper
`make kvm-ssh-secret` already exists and is reused.
- RBAC: none; the provider does not talk to the cluster's API.
- `hack/providers-dev.yaml`: no entry (needs credentials).
**Evidence:** contract §12; `config/manager/manager.yaml`.
## 23. Docs
**Answer:** `docs/kvm2-provider.md` (hypervisor prerequisites including OVMF
and the `libvirt` group and a bridge with DHCP; cluster-side setup reusing the
`kvm-ssh` Secret; config reference; placement mapping; what lands on the host
including the metadata namespace; the image cache and the quota precheck; the
health check reason table; logging tiers and span names; migrating an entry
from `kvm`, which requires deleting its Proxies first because kvm2 does not
adopt v1 domains; known gaps). A fourth column in the three tables of
`docs/providers-existing.md` and a kvm2 section. `docs/architecture.md`:
components row, a `kvm2` block in §5, the span topology in §10, a §11 row, and
Decisions bullets (dedicated download connection, checks on fresh connections,
own metadata namespace, quota precheck, SSH auth failures Permanent). A README
row. `docs/testing.md` rows and a kvm2 integration section. A `CHANGELOG.md`
entry after the user confirms.
**Evidence:** contract §12.
## 24. Known limits accepted at design time
**Answer:** nine limits, each with its workaround:
- **Create blocks**, including the first image download per host, holding one
of three reconcile workers. Workaround: warm the cache with one Proxy per
image per host before creating many.
- **Volumes without a domain are invisible to GC** (crash between volume
creation and define). The same Proxy's next reconcile removes them; a Proxy
deleted for good leaves them for `virsh vol-list`.
- **A host removed from the config orphans its VMs**: Delete is a no-op
success for it and GC does not sweep it. Delete the Proxies first.
- **No adoption of `kvm` (v1) domains**: switching an entry's type requires
deleting its Proxies first, or the old domains stay behind.
- **Cancelling one call drops the host's pooled connection**, so other RPCs
in flight on that host fail once with Transient and retry. The download is
isolated on its own connection.
- **`Content-Length` is required** for image downloads; a chunked mirror is
rejected as Permanent.
- **A paused or pmsuspended domain is not started.** A repeat `Create` for
a domain in that state is `ErrPermanent` (entry 10), so the Proxy latches
as Failed until its spec changes, rather than retrying a `DomainCreate`
that libvirt refuses. Resume the domain on the host (`virsh resume`) and
touch the spec, or delete the Proxy.
- **No v1 log-format compatibility**: the `libvirt call` lines have the same
keys where the same facts exist, but dashboards keyed on `kvm` logger names
need `kvm2`.
- Contract §13 limits that apply unchanged: config read once per process
(adding a host needs a restart; credentials do not), no installation identity
in GC, an unconfigured provider latches the Proxy.
**Evidence:** contract §13; entries 4, 8, 11, 18 above.
docs/kvm2-provider.md
+679
View File
@@ -0,0 +1,679 @@
# The kvm2 provider
Runs Managed proxies as VMs on your own libvirt/KVM hypervisors. It is the
second-generation libvirt provider, built clean-room from
[provider-contract.md](provider-contract.md) and the design in
[kvm2-provider-spec.md](kvm2-provider-spec.md), and it ships side by side
with the first-generation `kvm` provider: both types can be configured at
once, both take the same config block shape, and both read the same
`kvm-ssh` Secret. What kvm2 adds over kvm: a child span per libvirt RPC,
a pool-space precheck that maps to the reconciler's quota wait, SSH
credentials re-read on every dial, image downloads and health checks on
connections of their own, an adoption guard on the owning UID, and SSH
authentication failures that stop retrying.
Like kvm, the operator reaches each host over SSH, forwards the libvirt
unix socket through that connection and speaks libvirt's own RPC over it,
so the hosts need nothing beyond `sshd` and `libvirtd` and the operator
image carries no libvirt libraries. VMs boot a stock distro cloud image
that the operator downloads into the host's storage pool on first use,
with the Proxy's cloud-init delivered on a NoCloud seed ISO built
in-process.
## Hypervisor prerequisites
Per host, once:
1. **libvirt and KVM** (`libvirtd` running, `/dev/kvm` present).
2. **An SSH user in the `libvirt` group**, so it may open
`/var/run/libvirt/libvirt-sock`. The operator authenticates with a
private key; nothing else on the host is touched. The socket is used
as `qemu:///system`, so no further host permissions are needed.
3. **A bridge to a network with DHCP** (for example `br0` bridged onto
the LAN). Each VM gets one virtio NIC on it and takes its IP from that
network's DHCP. The operator's pod network and the crawler clients
must route to those IPs: the health engine probes `http://<ip>:<port>`
through the proxy and clients get the same address from the discovery
API. Nothing on the host filters the port; the provider does not use
`spec.port` at all.
4. **A storage pool** (default `default`, usually `/var/lib/libvirt/images`)
with room for one base image per distro and provider entry (a few
hundred MB each) plus the
per-VM overlays (`diskSizeGb` each, thin-provisioned). The pool must be
running: a stopped pool fails the health check with `PoolNotActive` and
fails `Create` permanently.
5. **UEFI firmware for guests**: the `ovmf` package on Debian/Ubuntu,
`edk2-ovmf` on Fedora/RHEL. VMs boot UEFI by default with secure boot
off (libvirt picks the OVMF binary itself); a domain start fails with
"no firmware found" without it. Only `firmware: bios` images skip this.
6. **Outbound internet from the VMs**, in practice: a vendor-data script
installs `qemu-guest-agent` from the distro mirrors on first boot, and
the agent is how the operator learns the VM's address. Without it the
operator falls back to the host's ARP table, which only sees traffic
addressed to the host itself, so on a plain bridge a VM without the
agent stays `Provisioning`.
Check from the operator's point of view before wiring it in:
```sh
ssh novakj@beelink virsh -c qemu:///system list --all # user can drive libvirt
ssh novakj@beelink virsh -c qemu:///system pool-info default
ssh novakj@beelink ip link show br0
```
## Cluster-side setup
1. Create the SSH Secret from a key pair dedicated to the operator and a
`known_hosts` file holding every host's key. It is the same `kvm-ssh`
Secret the `kvm` provider uses; one Secret serves both types, and if
you already have it there is nothing to do here.
```sh
# once: a key pair for the operator; the .pub goes on every host
ssh-keygen -t ed25519 -f ~/.ssh/proxy-operator_ed25519 -N '' -C proxy-operator
ssh-copy-id -i ~/.ssh/proxy-operator_ed25519.pub novakj@beelink
# once per host: its host key, under the address the *operator* will
# dial (hosts[].address), which may differ from what your laptop uses
ssh-keyscan -H beelink.tail8c69c.ts.net >> ~/.ssh/kvm_known_hosts
# the Secret: the PRIVATE key (not the .pub) plus known_hosts
kubectl -n egress-proxies-operator-system create secret generic kvm-ssh \
--from-file=id_ed25519=$HOME/.ssh/proxy-operator_ed25519 \
--from-file=known_hosts=$HOME/.ssh/kvm_known_hosts
```
`make kvm-ssh-secret` runs that last command as an apply, so it also
updates an existing Secret; override the paths with
`KVM_SSH_KEY=... KVM_KNOWN_HOSTS=...`. Putting the `.pub` in the
Secret is the classic mistake: the target's `kvm-ssh-secret` check
refuses a file without `PRIVATE KEY` in it, and the provider itself
would fail every call permanently with a "parsing ssh key file" error.
Without `known_hosts`, set `insecureIgnoreHostKey: true` and accept
that a MITM could hand you a fake hypervisor.
`config/manager/manager.yaml` mounts the Secret (optional) at
`/etc/proxy-operator/kvm` as two files, `id_ed25519` and
`known_hosts`, which is what `sshKeyFile` and `knownHostsFile` in the
provider config point at. The deployment starts fine without the
Secret, even with a kvm2 provider configured: `New` only checks that
the files exist and logs a `WARNING:` line when one does not; the
provider's proxies fail transiently and recover once the Secret lands.
**Rotating the key or a host key needs no restart.** Both files are
read and parsed on every dial (`loadCredentials` in
`internal/provider/kvm2/ssh.go`); nothing is cached. Re-run the
command or the make target, and the next connection, health check
included, uses the new material. Connections that are already open
stay open until they drop.
2. Add a provider entry to the providers config
(`config/manager/providers_config.yaml` in-cluster; full sample in
`config/samples/providers-config.yaml`). The type is `kvm2` and the
block is `kvm2:`; its fields are those of the `kvm` block:
```yaml
- name: kvm2-home
type: kvm2
kvm2:
sshKeyFile: /etc/proxy-operator/kvm/id_ed25519
knownHostsFile: /etc/proxy-operator/kvm/known_hosts
images:
debian-13:
url: https://cloud.debian.org/images/cloud/trixie/latest/debian-13-generic-amd64.qcow2
machineTypes:
small: {vcpus: 1, memoryMiB: 1024}
hosts:
- name: beelink
address: beelink.tail8c69c.ts.net:22
user: novakj
bridge: br0
```
Validation messages name the block, so a mistake reads
`kvm2.hosts[0]: bridge is required`, not `kvm.hosts`. The providers
config is read once per process: adding a host or an image entry needs
a manager restart; credentials do not.
3. Apply a Proxy (`config/samples/proxy_kvm2.yaml`) and watch it:
```sh
kubectl get px -w
# NAME MODE PROVIDER PHASE IP HEALTHY
# proxy-kvm2-sample Managed kvm2-home Provisioning
# proxy-kvm2-sample Managed kvm2-home Running 192.168.1.73
# proxy-kvm2-sample Managed kvm2-home Ready 192.168.1.73 True
```
The first Proxy for a given image on a given host sits in
`Provisioning` for the length of the image download; the operator log
shows `fetching base image into host pool` with the URL and size and
`base image cached` with the size, duration and digest.
## Config reference
The block is `kvm2:` and its Go type is the shared `KVMConfig` in
`internal/provider/config_kvm.go`, so the fields, defaults and validation
are the same as for `kvm`. Defaults are applied by `withDefaults` in
`internal/provider/kvm2/kvm2.go`.
| Field | Required | Default | Meaning |
|---|---|---|---|
| `sshKeyFile` | yes | | private key used for every host; OpenSSH or PEM, parsed on every dial |
| `knownHostsFile` | unless `insecureIgnoreHostKey` | | OpenSSH `known_hosts` to verify host keys against, read on every dial |
| `insecureIgnoreHostKey` | | `false` | skip host-key verification entirely; `New` logs a `WARNING:` line when it is on |
| `diskSizeGb` | | `10` | capacity of each VM's overlay disk; cloud-init's growpart expands the root fs into it. Also what the pool-space precheck reserves per VM |
| `vendorData` | | | extra cloud-init vendor-data placed on every seed next to the Proxy's user-data. Beware that cloud-init drops any top-level key of a vendor-data **cloud-config** that the user-data also defines. Shipped as one part of a MIME multipart alongside the agent script when `installGuestAgent` is on |
| `installGuestAgent` | | `true` | add the vendor-data script that installs and starts `qemu-guest-agent`. Turn off for images that ship it already or cannot reach a package mirror; IP discovery then depends on the host's ARP table |
| `images.<key>` | | | the key names the image in `placement.image`. Letters, digits, dots, dashes and underscores only: it becomes the cached volume name `base-<key>.<format>` |
| `images.<key>.url` | yes per entry | | http(s) URL of a cloud image (qcow2 or raw). The server must send `Content-Length` |
| `images.<key>.sha256` | | | expected digest, 64 hex characters; verified after download, a change triggers a re-download |
| `images.<key>.format` | | `qcow2` | `qcow2` or `raw` |
| `images.<key>.firmware` | | `uefi` | `uefi` (OVMF, secure boot off) or `bios` (SeaBIOS) for images without an EFI system partition |
| `machineTypes.<key>` | at least one | | `{vcpus, memoryMiB}` size profile, both greater than zero |
| `hosts[].name` | yes | | the `placement.zone` value that selects this host; unique. Letters, digits, dots, dashes and underscores only: it becomes part of the providerID `hosts/<name>/domains/<vm>` |
| `hosts[].address` | yes | | SSH endpoint, `host:port` |
| `hosts[].user` | yes | | SSH user, member of the `libvirt` group |
| `hosts[].socket` | | `/var/run/libvirt/libvirt-sock` | libvirt socket path on the host |
| `hosts[].pool` | | `default` | storage pool for base images, overlays and seeds |
| `hosts[].bridge` | yes | | host bridge for the VM's NIC |
## Placement
All three fields are checked before any host is dialled. An unknown value
is `ErrPermanent`, which the reconciler latches as `Failed` with the
message below until the spec changes.
| Proxy field | Meaning for kvm2 | Error when unknown |
|---|---|---|
| `placement.zone` | host name from `hosts`. May be omitted when exactly one host is configured; otherwise required | `placement.zone "x" is not in kvm2.hosts`; with several hosts and no zone, `placement.zone is required when kvm2.hosts lists more than one host` |
| `placement.machineType` | key in `machineTypes` | `placement.machineType "x" is not in kvm2.machineTypes` |
| `placement.image` | key in `images`, or an `http(s)://` URL used directly with the defaults (`qcow2`, `uefi`) | `placement.image "x" is neither a key in kvm2.images nor an http(s) URL` |
| `placement.region` | ignored. It is still part of the spec hash, so changing it replaces the VM and does nothing else | |
The messages come from `Create` in `internal/provider/kvm2/create.go`,
`host` in `kvm2.go` and `resolveImage` in `image.go`. Fix the spec (which
replaces the VM) or the config. Catalogue edits affect only new VMs: the
spec hash covers the placement values, not what the config maps them to.
## What lands on the host
For a Proxy whose deterministic name is `proxy-abc…`, on the chosen
host's pool:
| Object | Name | Lifetime |
|---|---|---|
| base image | `base-<entry>-<key>.<format>`, where `<entry>` is the provider entry's name (for a URL image, `<key>` is the first 16 hex characters of the URL's sha256) | cached; survives Proxy deletion |
| image marker | `base-<entry>-<key>.<format>.sha256`, a 64-byte raw volume holding the digest | with the base image |
| overlay disk | `proxy-abc….qcow2`, qcow2 backed by the base file, `diskSizeGb` capacity | with the Proxy |
| seed ISO | `proxy-abc…-seed.iso`, raw | with the Proxy |
| domain | `proxy-abc…` | with the Proxy |
The names are built in `internal/provider/kvm2/volume.go`. The base image
carries the **provider entry's name** because the file is shared by every
Proxy on that host but must not be shared by two entries: the `kvm`
provider builds `base-<key>.<format>` in the same pool from the same
catalogue shape, and two kvm2 entries pointing one key at different URLs
would otherwise agree on the name and disagree on the contents. The cost
is one copy of each image per entry per host. For that reason the entry's
name must itself be a usable file name — letters, digits, dots, dashes and
underscores — which `New` checks at startup, since provider names are
otherwise only required to be non-empty and unique.
The domain (`buildDomainXML` in `xml.go`) is a `q35` machine with a
host-passthrough CPU, ACPI and APIC, memory in MiB and vcpus from the
machine type, the overlay on `vda` (virtio) and the seed as a read-only
SATA cdrom on `sda`, one virtio NIC on the bridge, a serial console on a
pty, the guest-agent channel `org.qemu.guest_agent.0`, a virtio RNG fed
from `/dev/urandom`, and no graphics. Both disks are `type="file"` with
the paths `StorageVolGetPath` returned: AppArmor derives QEMU's profile
from those paths, and volume-type disks fail with permission denied on
stock Debian and Ubuntu hosts. UEFI images get `<os firmware="efi">` with
the `secure-boot` feature disabled; BIOS images get neither.
Its `<metadata>` carries one element in kvm2's own namespace,
`https://gitea.home.hrajfrisbee.cz/kacerr/egress-proxies-operator/kvm2`
(`metadataURI` in `metadata.go`):
```xml
<instance xmlns="https://gitea.home.hrajfrisbee.cz/kacerr/egress-proxies-operator/kvm2">
<managed>true</managed>
<uid>0f3a…</uid>
<createdAt>2026-09-16T17:42:11Z</createdAt>
</instance>
```
That is how orphan GC recognises its own VMs, since libvirt has no
labels, and how a repeat `Create` knows whether an existing domain of the
same name belongs to this Proxy: a domain without the element is a
permanent error (`carries no kvm2 mark; it was not created by this
provider`), a different non-empty `uid` is a permanent error naming both
UIDs, and a marked domain with a matching or empty `uid` is adopted. The
namespace is kvm2's own, so the `kvm` provider's domains are invisible to
it (see *Migrating from kvm*).
The seed (`buildSeedISO` in `seed.go`) is an ISO 9660 image labelled
`cidata` with three files. `user-data` is the Proxy's `cloudInit`, **byte
for byte**: any format cloud-init accepts works, including a Jinja
template, a MIME archive, a shell script or a gzipped payload; the
operator never parses or re-renders it. `meta-data` holds
`instance-id: <name>` and `local-hostname: <name>`, so a reboot is never
taken for a new instance. `vendor-data` carries the guest-agent
bootstrap. The ISO is staged in a temp directory under `/tmp` (the
manager's 64Mi emptyDir), assembled in memory (tens of KiB) and uploaded
as a raw volume; the staging directory is removed before the call
returns. The seed is rebuilt on every `Create` attempt, because a
previous attempt may have been cut off mid-upload and a truncated seed
would boot a VM without cloud-init.
The guest agent rides along in `vendor-data` as a **shell script**
(`guestAgentScript` in `seed.go`), not a cloud-config, on purpose:
cloud-init merges a vendor-data cloud-config under the user-data with
mergers that keep the user-data's value for every top-level key both
define and never append lists, so a Proxy carrying its own `packages:`
would silently lose the agent. Scripts bypass the merge. The script is
idempotent, picks whichever of `apt-get`, `dnf`, `yum`, `zypper` or `apk`
the image has, enables the service with `systemctl` or OpenRC, and logs
into `/var/log/cloud-init-output.log`. With `vendorData` also configured,
the two travel as separate parts of a MIME multipart (`text/plain` for
your document, `text/x-shellscript` for the script), with the
`MIME-Version` header first so cloud-init detects it. With
`installGuestAgent: false` the file is your `vendorData` alone, or empty.
## The image cache
The base image is downloaded **by the operator, through libvirt**, into
the host's pool the first time a Proxy asks for it on that host. Nothing
is pre-placed on the hypervisor and the operator needs no filesystem
access there. The bytes stream from the URL through the operator into
`StorageVolUpload` with a running sha256; nothing is buffered
(`fetchImage` and `uploadImage` in `internal/provider/kvm2/image.go`).
The upload runs on a **dedicated connection** to the host, dialled for
the transfer and closed after it, not on the pooled connection the other
operations share. A cancelled or overlong call kills the connection it
ran on (see *Timeouts and connections*), and a poll for some other Proxy
being cancelled must not cut off a 20-minute upload.
A body that fails part way through is reported to the caller **out of
band**: the reader handed to `StorageVolUpload` turns any source error
into a clean end of stream and keeps the real error for afterwards, so
the short upload is caught by the length and digest checks rather than
by the stream. This is not tidiness. go-libvirt answers a reader error
by parking its stream sender forever on a channel it no longer reads, so
handing it one wedges the call and leaks the sender with its 4 MiB
buffer; see *Known gaps*. For the same reason the download carries its
own deadline, a minute inside the transfer budget, so a stalled mirror
ends through its body rather than through the killed socket the budget
would otherwise leave behind.
The download needs a `Content-Length`: the volume's capacity is declared
before the upload starts. The request sends `Accept-Encoding: identity`
so a transparently gzipped body cannot hide the length. A response
without it, or with length 0, is a permanent error naming the URL, and so
is any 4xx status; 429 and 5xx are transient and retried.
The marker volume is written **last**, on the pooled connection, after
`StoragePoolRefresh` has made libvirt read the file's real format. Its
presence is what marks a base complete: a crash mid-download leaves a
base volume without a marker, which the next attempt deletes and redoes
(`cachedImage`). A re-download also happens when the marker's content is
not a 64-character hex digest, or when `sha256` is configured and differs
from it. Otherwise the cached copy is used for as long as it exists, even
if the URL's content changed upstream: a `latest` URL is pinned at the
moment of first fetch, per host. Concurrent Creates for the same image
on the same host queue on a per-(host, volume) lock; the first fetches,
the rest find the marker.
To evict manually:
```sh
ssh novakj@beelink virsh -c qemu:///system vol-list --pool default | grep base-kvm2-home-
ssh novakj@beelink virsh -c qemu:///system vol-delete --pool default base-kvm2-home-debian-13.qcow2.sha256
ssh novakj@beelink virsh -c qemu:///system vol-delete --pool default base-kvm2-home-debian-13.qcow2
```
Marker first: a base without its marker is refetched, a marker without its
base is cleaned up, and neither leaves a Proxy pointing at nothing.
**Warning: do not evict a base volume while Proxies using that image exist
on the host.** The overlays back onto the base file, and libvirt will
unlink it under running domains without complaint; the guests keep running
on the open file descriptor and fail at their next start. Delete those
Proxies first.
Changing `sha256` on an entry whose base is already cached is **refused**,
permanently and by name, rather than resolved by replacing the file — that
replacement was the same corruption, just automated. Add the image under a
new key and move Proxies over by changing their `placement.image` (which
replaces the VM), or delete the Proxies and evict the base by hand.
Changing `url` alone does nothing at all: the cache is keyed by entry and
key, so the new URL is never fetched. Add a new key for that too.
## Quota: the pool-space precheck
Before creating any volume, `Create` reads `StoragePoolGetInfo` and
compares the pool's free space with what it is about to commit
(`checkPoolSpace` and `neededBytes` in `image.go`): the image's
`Content-Length` when the base is not cached yet (found with a `HEAD`
request, or a `GET` whose body is closed at once when `HEAD` answers
anything but 2xx, since mirrors refuse it with 405, 403 or 501 alike; the
`GET`'s status is then what counts), plus `diskSizeGb` in bytes for the
overlay, plus 16 MiB for the
seed. Below that it returns `ErrQuotaExceeded` with a message naming the
pool, the bytes available and the bytes needed. The reconciler answers
quota with a wait, not a failure: the Proxy shows `Provisioned=False`
with reason `QuotaExceeded` and is retried after `QuotaRetry` (5
minutes), off the error backoff curve and never `Failed`. Free up the
pool or lower `diskSizeGb` and the next attempt goes through.
The precheck is the only quota signal; libvirt itself reports a full pool
as an ordinary error, which would be transient. The health check exposes
the same number as `poolAvailableGiB`, so a pool that is filling up shows
there before Creates start waiting.
A pool that is not running fails the precheck permanently (`storage pool
<pool> is not running`): start it with `virsh pool-start` and
`virsh pool-autostart`.
## Migrating from `kvm`
The `kvm2` block is the `kvm` block under another name, so an entry ports
by changing `type: kvm` to `type: kvm2` and `kvm:` to `kvm2:`. The
`kvm-ssh` Secret is shared; nothing changes on the hosts.
What does not port is the VMs. kvm2 marks its domains in its own metadata
namespace and neither lists nor adopts domains the `kvm` provider made:
`ListByTag` skips them, so orphan GC never sees them, and `Create` refuses
a domain of the same name that lacks the kvm2 mark with a permanent error
rather than start it (a Proxy recreated under the new entry gets the same
deterministic name, so this is what happens when step 1 below is skipped).
A Proxy's `spec.provider` is immutable, so an existing Proxy cannot be
moved to the new entry anyway. Do it in this order:
1. Delete the Proxies on the `kvm` entry and wait for them to go; the
`kvm` provider removes their domains and volumes.
2. Change the entry (or add a second one under a new name and keep the
old one for a while; both types may share a host).
3. Restart the manager, then recreate the Proxies with the new
`spec.provider`.
A `kvm` domain left behind on a host is invisible to kvm2 forever; clean
it up on the host with `virsh undefine --nvram <name>` and
`virsh vol-delete` for its `<name>.qcow2` and `<name>-seed.iso`.
The base image cache is **not** shared between the two types, and that is
deliberate: kvm names its copy `base-<key>.<format>` while kvm2 names its
own `base-<entry>-<key>.<format>`, so kvm2 downloads the image once more
per host rather than adopting a file another provider maintains. Nothing
needs deleting; the `kvm` base stays where it is and keeps serving the
overlays that back onto it. Budget the extra few hundred MB in the pool.
## Logging
The provider logs through controller-runtime's logger, named `kvm2`, with
`provider` (the configured name) on every line, and `host` and `name`
(the domain) on every line about an instance. Every line inside a traced
operation also carries `traceID` and `spanID`.
- `--zap-log-level=debug` (logr V(1)):
- one line per operation outcome: `kvm2 domain ensured running` with the
`id`, `kvm2 domain fetched` with `state`, `ip` and `uid`,
`kvm2 domain not found`, `kvm2 domain deleted`, `kvm2 domains listed`
with `hosts` and `managed` counts, `kvm2 host checked` with `ready`,
`reason` and `latency`; plus the steps inside a Create
(`kvm2 domain already defined`, `overlay volume created`,
`seed volume uploaded` with `bytes`, `domain defined` with `uid` and
`firmware`, `domain started`, `fetching base image into host pool`,
`base image cached`);
- **one `libvirt call` line per libvirt RPC**, written by the decorator
in `internal/provider/kvm2/observe.go` inside that RPC's span: `rpc`
(the go-libvirt method name, for example `DomainLookupByName`), the
identifiers it was given (`domain`, `pool`, `volume`), `duration`, and
on failure `error` plus `libvirtCode` and `libvirtError` (the code's
name, for example `ErrNoDomain`). A Get is four or five RPCs; a
Create with a cached image is around twenty;
- connection events from `internal/provider/kvm2/hosts.go`:
`libvirt host connected` with `address`, `user` and `duration`, and
`libvirt host connect failed` with the same plus `error` and `stage`
(`credentials`, `tcp`, `ssh` or `socket`);
- `kvm2 backend call failed` with `op` and the raw `error` once per
failed operation, before the error is returned;
- a line per domain that `ListByTag` skips, and a line when the image
marker forces a refetch.
- `--zap-log-level=2`: the `libvirt call` lines also carry the domain and
volume `xml`, the `metadata` element, the interface `addresses`, the
listed `domains`, the pool `state` and sizes and the volume `path`; the
`domain defined` line carries the `xml`; `kvm2 create resolved` logs the
resolved parameters (pool, bridge, machine type, sizes, image volume,
firmware) and `cloudInitBytes`; `kvm2 listed instance` logs each listed
domain.
- Never logged at any level: the cloud-init content, the SSH key, the
known_hosts content, and upload or download bodies. The seed and image
uploads report `offset` and `length` only. There is no raw wire tier
like the GCP provider's V(5); go-libvirt exposes no hook for it.
## Tracing
With an OTLP exporter configured ([architecture.md
§10](architecture.md#10-tracing-internaltracing)), every backend call is
a child span of the operation span the decorators open
(`provider.create`, `provider.get`, `provider.delete`, `provider.list`)
or of the monitor's `providerhealth.check`:
| Span | Around | Attributes |
|---|---|---|
| `kvm2.<Method>`, for example `kvm2.DomainDefineXML` | one libvirt RPC (`observe.go`) | `kvm2.host`, and the identifiers passed: `kvm2.domain`, `kvm2.pool`, `kvm2.volume`; on failure the status is Error and `kvm2.libvirt_code` carries the libvirt error number |
| `kvm2.dial` | the SSH handshake and the libvirt connect (`hosts.go`) | `kvm2.host`, `kvm2.address`, `kvm2.user` |
| `kvm2.download` | one image fetch, from the HTTP request to the marker (`image.go`) | `kvm2.host`, `kvm2.volume`, `kvm2.url_host`, `kvm2.sha256_configured`, and `kvm2.bytes` once the response arrived |
All spans are client spans. No attribute carries cloud-init, key material
or a URL beyond its host. The `libvirt call` log line for an RPC is
written inside that RPC's span, so its `spanID` points at the span.
## Timeouts and connections
No caller puts a deadline on the context, so every backend call carries
its own budget (`defaultBudgets` in `internal/provider/kvm2/backend.go`,
applied by `call`):
| Call | Budget |
|---|---|
| TCP connect plus SSH handshake plus libvirt connect | 10 s |
| ordinary RPC | 30 s |
| `DomainCreate` | 90 s |
| `StorageVolUpload` of at most 16 MiB (the seed, the marker) | 60 s |
| `StorageVolUpload` of an image, `StorageVolDownload`, and the whole image fetch from HTTP request to marker | 20 m; the caller's ctx may end it earlier |
| wait for running after `DomainCreate` | 15 s, polled every 500 ms (`defaultRunningWait` and `defaultRunningPoll` in `create.go`); `DomainCreate` is synchronous, so the first poll normally answers running, and a domain that is shut off, crashed or paused right after the start fails at once rather than waiting out the cap |
| the `HEAD` request of the pool-space precheck | 30 s |
| the courtesy `ConnectClose` when a connection is closed | 5 s |
go-libvirt's calls take no context and block, so the only way to abandon
one is to close the connection under it. That is what a cancelled or
overlong call does: the adapter marks the connection dead so the next call
redials, kills it by closing the SSH client's TCP connection, which takes
the forwarded socket with it, and returns `ErrTransient` wrapping the
context or deadline error. It marks dead before it kills, and closes the
TCP connection rather than the forwarded channel, because both of those are
lock-free: closing the channel writes a close message under the channel's
write lock, so it would queue behind a transfer stuck in a TCP write to a
host that has gone away — for as long as the kernel retransmits, which is
the one case the kill exists to survive. The consequence:
**other RPCs in flight on that host's pooled connection fail once with
`ErrTransient`** and their callers retry on a fresh connection. In the
reconciler that is one requeue with backoff; in GC it is one skipped
sweep.
Two callers are kept off the pooled connection for that reason
(`dialFresh` in `hosts.go`): the image upload, because a 435 MB transfer
must not die because a poll for another Proxy was cancelled, and the
health check, because a check that runs out of the monitor's timeout must
close only its own connection.
Otherwise there is one pooled SSH plus libvirt connection per host,
dialled lazily on the first call that needs it and shared by every
operation for that host (go-libvirt multiplexes RPCs on one connection).
Concurrent callers wait for one dial and share the result; waiting ends
with the caller's context. A `keepalive@openssh.com` request every 15 s
marks the connection dead when it fails or gets no reply within another
15 s, as does go-libvirt noticing the socket is gone; the next call
redials. Create and Delete for the same domain name are serialised on a
per-name lock, so two Deletes (GC and the reconciler) never race on
destroy and undefine, and a Delete cannot interleave with a Create's
steps; Get and ListByTag take no lock.
## Health check
The operator checks every configured host every `--provider-check-interval`
(default 1m) and publishes the result in the `ProviderHealth` object named
after the provider (`kubectl get ph kvm2-home -o yaml`) and in the
`proxy_operator_provider_target_up{provider,target}` gauge. One target per
`hosts[]` entry, in config order, all checked concurrently.
Each check dials a **fresh connection** to the host and closes it when
done (`probeHost` in `internal/provider/kvm2/check.go`); the pooled
connection is never touched, so a check that runs out of time cannot fail
a reconcile's call in flight, and a fresh dial picks up a rotated Secret
at once. The sequence is the SSH and libvirt handshakes, then four RPCs:
`ConnectGetLibVersion`, `ConnectGetHostname`, `StoragePoolLookupByName`
and `StoragePoolGetInfo` on the host's `pool`. The cost per host per
round is one sshd session and those four calls.
Every target's `details` carry `user` and `pool`. As facts arrive they
gain `libvirtVersion` (decoded `major.minor.micro`), `hostname`,
`poolState` (virsh's word: `running`, `inactive`, ...),
`poolCapacityGiB` and `poolAvailableGiB`, so a target that fails late in
the sequence still says what the host answered before. The free space is
the number the quota precheck compares against.
A failing target says where the check broke (`checkReason` in `check.go`):
| Reason | Meaning | Look at |
|---|---|---|
| `Reachable` | ready | |
| `CredentialsUnavailable` | the key or known_hosts file is missing or does not parse | the `kvm-ssh` Secret (see *Cluster-side setup*); fixed in the cluster, not on the host |
| `Unreachable` | the TCP connect failed or ran out of the 10 s dial budget while the round still had time, the SSH transport failed for a reason other than authentication, the forwarded libvirt socket was refused, or the link died mid-check | DNS, Tailscale or a firewall between the cluster and `address`; `libvirtd` running and the `socket` path |
| `Unauthorized` | the SSH server rejected the key, or the host key is missing from known_hosts or has changed | `user`'s `authorized_keys`; the known_hosts entry (re-run `ssh-keyscan` and update the Secret) |
| `Forbidden` | SSH worked, but libvirt refused the user (`ErrAccessDenied` or `ErrAuthFailed`) | `user` in the `libvirt` group |
| `Timeout` | the monitor's `--provider-check-timeout` (default 15s) ran out before the check finished, whichever stage was running, the dial included; or the connection was up and an RPC's own budget ran out | a hung `libvirtd`, a saturated link, or a round timeout shorter than the dial takes |
| `PoolNotFound` | there is no pool named `pool` | `virsh pool-list --all` |
| `PoolNotActive` | the pool exists but is not running | `virsh pool-start <pool>` and `virsh pool-autostart <pool>` |
| `APIError` | any other libvirt error | the message carries libvirt's own text |
## Error classes
Every error is classified once, in `classify` in
`internal/provider/kvm2/errors.go`, with errors that carry a class from
their source (validation, the image fetch, the dial) keeping it. The
reconciler latches `ErrPermanent` as `Failed` until the spec changes,
waits 5 minutes on `ErrQuotaExceeded`, recreates on `ErrNotFound` from
`Get`, and retries everything else with backoff.
| Class | What lands there |
|---|---|
| `ErrPermanent` | placement values not in the catalogue; a malformed providerID; a host no longer in the config (from `Get`; `Delete` treats it as done); libvirt `ErrXMLError`, `ErrNoSupport`, `ErrInvalidArg`, `ErrAuthFailed`, `ErrAccessDenied` and `ErrNoStoragePool`; a pool that is not running; SSH authentication refused (`unable to authenticate`); a host key missing from known_hosts or changed (`knownhosts.KeyError`); a key or known_hosts file that exists but does not parse; a domain of the same name owned by another UID or carrying no kvm2 mark; a domain of the same name that is paused or pmsuspended; corrupt metadata on a domain; an image URL answering 4xx, a response without `Content-Length` or with length 0, a sha256 that does not match the configured one |
| `ErrQuotaExceeded` | the pool-space precheck |
| `ErrNotFound` | libvirt `ErrNoDomain` or `ErrNoStorageVol` from `Get`, `Delete` or `ListByTag`; inside `Create` the same codes are `ErrTransient`, because the contract forbids NotFound from Create |
| `ErrTransient` | everything else: a missing key or known_hosts file (the Secret has not landed yet), TCP and network errors, context cancellation, an abandoned or budget-exceeded call, a connection killed by another caller, libvirt `ErrOperationInvalid` and every unrecognised libvirt code, an image URL answering 429 or 5xx or ending the body early, every address source failing on an active domain, the wait for running expiring or the domain going shut off or crashed right after its start |
Error strings name paths, hosts, volumes and libvirt's own text. They
never contain key bytes, the known_hosts content or cloud-init.
## Known gaps
- **An upload that libvirtd itself fails leaks a goroutine, upstream.**
go-libvirt's `requestStream` hands its stream sender an unbuffered
channel and then, on the error path, returns without ever reading it:
```go
abort := make(chan bool) // rpc.go
outErr := make(chan error)
go func() { outErr <- l.socket.SendStream(...) }()
resp, err = l.processIncomingStream(c, in)
if err != nil {
abort <- true
return resp, err // outErr is never drained
}
```
Whenever the daemon errors mid-upload — a pool that fills, a volume
removed underneath — the sender parks on `outErr` for good, taking its
4 MiB buffer with it. Closing the socket does not help: both ends are
blocked on channel sends, not on I/O. kvm2 avoids the two cases it
controls (a failing body, and its own transfer deadline; see *The image
cache*), so what is left needs a server-side failure. The fix is two
characters upstream, `make(chan bool, 1)` and `make(chan error, 1)`; if
it ever bites hard enough, a `replace` carrying that patch is the
escape hatch.
- **Create blocks**, including the first image download on each host,
holding one of the three reconcile workers for its duration (LAN
speed, a few hundred MB, once per image per host). Other Proxies keep
reconciling on the remaining workers. Warm the cache with one Proxy per
image per host before creating many.
- **Volumes without a domain are invisible to GC.** A crash between
creating the overlay or seed and defining the domain leaves volumes
that `ListByTag` cannot see (it lists domains). The same Proxy's next
Create reuses the overlay and rebuilds the seed, and its Delete removes
both by name even with no domain, but a Proxy CR that is gone for good
leaves them for `virsh vol-list` to find.
- **A host removed from the config orphans its VMs.** `Delete` for a
providerID naming an unconfigured host is a logged no-op success, so
the Proxy finishes deleting instead of pinning its finalizer, but the
domain and its volumes stay on the hypervisor and orphan GC cannot see
them either (it only sweeps configured hosts). Delete the Proxies before
retiring a host, or clean up afterwards with `virsh undefine --nvram`
plus `virsh vol-delete`. `Get` fails permanently for such a providerID
on purpose: a typo in `hosts` must not look like a missing VM and
trigger a rebuild elsewhere.
- **No adoption of `kvm` domains.** See *Migrating from kvm*.
- **A cancelled call drops the host's pooled connection once**, so other
RPCs in flight on that host fail once with `ErrTransient` and retry.
The image upload and the health check are isolated on their own
connections.
- **`Content-Length` is required** for image downloads; a mirror that
only chunks is rejected permanently. Point the catalogue at one that
sends the length.
- **A paused or pmsuspended domain is not started.** A repeat `Create`
for a domain in that state fails permanently (`is DomainPaused and
cannot be started by the operator; resume or destroy it on the host`),
so the Proxy shows `Failed` until its spec changes; the operator never
resumes a VM an admin paused. Resume it on the host
(`virsh resume <name>`) and touch the spec, or delete the Proxy.
- **Parser text in credential errors.** A key or known_hosts file that
does not parse fails with the parser's own message, which can quote a
PEM header type or a malformed known_hosts token. Key bytes are never
quoted.
- **The sample cloud-init is an open proxy.**
`config/samples/proxy_kvm2.yaml` installs Squid with
`http_access allow all`; restrict it before exposing the bridge network
to anything but the cluster.
- **IP discovery needs the agent or guest traffic to the host.** A VM on
a bridge with no DHCP never gets an address and stays `Provisioning`
forever; check `virsh console <name>` on the host. The same symptom
with a booted, addressed VM means the guest agent never came up: check
`systemctl status qemu-guest-agent`, `cloud-init status --long` and the
vendor script's output in `/var/log/cloud-init-output.log` (a VM without
outbound internet cannot install it). A user-data that disables
vendor-data (`vendor_data: {enabled: false}`) also removes the agent, by
design. A host where the ARP source *errors* rather than answering with
nothing is the one case this reads differently: with no source left to
answer, a booting VM retries with `ErrTransient` instead of resting in
`Provisioning`. It still converges; the giveaway is two "no addresses
from source" lines per poll at `-v 2`.
`virsh domifaddr <name> --source agent` and `--source arp` show
what the operator sees, in that order.
- **Debian 13 cloud images hang in GRUB under BIOS on a headless VM**
([Debian bug #1111240](https://bugs.debian.org/1111240)), which is why
`firmware` defaults to `uefi`; the same image boots under UEFI with the
serial console intact. Images that need `firmware: bios` are the
exception.
## Testing
Unit tests run over an in-memory fake of the libvirt seam and an
in-process SSH server. The optional integration test against a real host
is described in
[testing.md](testing.md#the-kvm2-integration-test-real-hypervisor); run it
with `make test-integration-kvm2`. `TestKVM2Check_realHost` runs the
health check against the same host and creates nothing:
`go test -tags=integration -count=1 -v -run TestKVM2Check ./internal/provider/kvm2/`.
docs/plans-executions/2026-09-11-1526-provider-health-checks.md
+378
View File
@@ -0,0 +1,378 @@
# Execution: provider health checks
Plan: [docs/plans/2026-09-11-1526-provider-health-checks.md](../plans/2026-09-11-1526-provider-health-checks.md)
- [x] Step 0 — Save the plan
- [x] Step 1 — ProviderHealth API and CRD
- [x] Step 2 — Contract plus the kubernetes and GCP checks
- [x] Step 3 — KVM check
- [x] Step 4 — Monitor
- [x] Step 5 — Metrics and wiring
- [x] Step 6 — Docs
- [x] Step 7 — Validate, record, push, open the PR
## Step 0 — Save the plan
Branched off `main` (at the PR #5 merge, `eb8f075`). The plan was copied out of
`~/.claude/plans/` and committed on its own, before any code, as the repo's plan rules
require:
```bash
git checkout -b feat/provider-health main
cp ~/.claude/plans/now-i-would-like-flickering-wall.md docs/plans/2026-09-11-1526-provider-health-checks.md
```
The uncommitted `.claude/settings.json` change was left unstaged and stays out of this
branch.
## Step 1 — ProviderHealth API and CRD
`kubebuilder` isn't installed here (only `bin/controller-gen`, `bin/kustomize` and
`bin/setup-envtest` are), so `api/v1alpha1/providerhealth_types.go` and the `PROJECT`
entry were written by hand instead of with `kubebuilder create api`. For a cluster-scoped
resource `PROJECT` simply omits `namespaced`, which is what kubebuilder itself writes.
The new types are registered in the existing `AddKnownTypes` call in `proxy_types.go`,
not in a second `init()`.
The deepcopy code, the CRD and the RBAC come from the usual generators:
```bash
make generate manifests
go build ./... && go vet ./api/... && go test -short ./api/...
```
To confirm the new CRD and the viewer role actually reach the rendered install, I built
the kustomization and grepped for them:
```bash
bin/kustomize build config/default | grep -n providerhealth
# providerhealths.crawl.example.com (scope: Cluster, shortName ph)
# egress-proxies-operator-providerhealth-viewer-role
```
Worth noting:
- The only role added is the viewer role. The operator is the object's only writer, so
the admin and editor roles kubebuilder scaffolds for Proxy were not added.
- The type file has no unit test, since it has no logic. The schema, including the
required `ready`/`lastTransitionTime` and the list-map keys, is proven by the envtest
round-trip in Step 4.
- `config/rbac/role.yaml` is unchanged so far. The manager's own permissions arrive in
Step 4 with the Monitor's RBAC markers.
## Step 2 — Contract plus the kubernetes and GCP checks
`provider.Checker` is an optional interface in `internal/provider/check.go`, kept out of
`Provider`. Each implementation has a compile-time assertion, because the monitor's
type assertion would otherwise fail silently.
- **kubernetes:** lists at most one managed Pod cluster-wide, using the same client, verb
and scope as `ListByTag`. The target is `apiserver`, at the `restCfg.Host` now stored
in the new `apiHost` field.
- **GCP:** calls a new seam method, `Probe`. `realInstances.Probe` lives in `check.go`,
next to its only caller. `ListByTag`'s `Sprintf` filter became the shared
`managedFilter` constant, so the check and the GC sweep can't drift apart.
Before relying on "MaxResults is only a page size", I checked the SDK's REST
`AggregatedList` in the module cache:
```bash
grep -n "func (c \*instancesRESTClient) AggregatedList" -A80 \
"$(go list -m -f '{{.Dir}}' cloud.google.com/go/compute)"/apiv1/instances_client.go
# InternalFetch: req.MaxResults = pageSize ... return elems, resp.GetNextPageToken(), nil
```
```bash
go build ./... && go vet ./internal/provider/... && go test -race -count=1 ./internal/provider/...
```
Worth noting:
- The planned `TestRealInstances_Probe_fetchesOnePage` became a three-case
`TestRealInstances_Probe` table.
- It keeps the first page's `nextPageToken`, and asserts exactly one request with
`maxResults=1` for each case.
- "empty project" covers `iterator.Done`.
- "api error keeps its status" proves the real client returns a `*googleapi.Error`, so
a disabled Compute API is reported as `Forbidden` rather than `Unreachable`.
- Once a context has ended, both checks report `Timeout` whatever the error says. The
monitor's timeout is the usual cause, and the underlying error only repeats it.
## Step 3 — KVM check
`conn.go` now wraps each of the four dial stages in a `dialError`: ssh-material, tcp,
ssh-handshake and libvirt-connect. `Error()` returns the inner message unchanged and
`Unwrap()` returns the cause, so `classify`/`isClassified` and every existing
message-matching test behave as before. `TestDialError_keepsClassification` pins this
down.
The three new RPCs went into `libvirtAPI`, `tracingAPI` and `fakeAPI`. The method-count
guard in `TestTracingAPI_logsEveryMethod` fails until a traced row exists for each new
method, which is what forced the three new rows.
`Check` runs one goroutine per host, over the shared connection, through
`hostPool.call`. The pool verdict is decided only after `call` returns, because
returning it as an error from inside the callback would make `call` drop a perfectly
good connection.
Build, vet (also with the integration tag, since that file changed) and race tests:
```bash
go build ./internal/provider/... && go vet ./internal/provider/kvm/ \
&& go vet -tags=integration ./internal/provider/kvm/ \
&& go test -race -count=1 ./internal/provider/kvm/
```
Worth noting:
- **Pool state wording:** `poolState` uses virsh's words (`running`, `inactive`, …)
rather than go-libvirt's constant names, so it reads like `virsh pool-list` output.
- **Details:** each host also reports its SSH `user`. That is the first thing to check
when the reason is `SSHHandshakeFailed`.
- **Tests renamed or merged:**
- The planned `TestSSHAuth_dial_stages` is `TestLazySSH_dial_tagsTheFailingStage`. It
dials through `lazySSH`, so the ssh-material stage is covered as well as the three
stages the in-process SSH server can trigger.
- The pool-inactive and pool-missing cases are merged into `TestCheck_libvirtVerdicts`,
which also covers a daemon error. Each case runs two checks and asserts a single
dial.
- **Tests added beyond the plan:**
- `TestCheck_timeout`: a daemon that stops answering gives `Timeout`.
- `TestDialError_keepsClassification`.
- **Not run yet:** `TestKVMCheck_realHost` in the integration file only compiles here.
It needs beelink, so it goes on the Step 7 checklist for the user to run.
## Step 4 — Monitor
`internal/providerhealth` has the `Monitor` runnable plus the pure `buildStatus`.
- **Scheduling:** the Monitor is leader-only. It cleans up once at start, checks
immediately, then checks on a ticker.
- **Each round:** providers are checked concurrently under one timeout, then logged and
published one after another. A failed write is logged, and the next round writes
again.
- **Reads:** objects are read through the reader it is given. `main.go` passes the
manager's API reader (Step 5).
The first `make manifests` left `config/rbac/role.yaml` unchanged, with no error.
I had put the `+kubebuilder:rbac` markers inside `Monitor`'s doc comment, and
controller-gen only reads RBAC markers from package-level comment groups. So they were
silently attached to the type and ignored. An empty `git diff config/rbac/role.yaml` was
the only sign. Moving them into their own comment block fixed it:
```bash
make manifests && git diff config/rbac/role.yaml
# + providerhealths: create, delete, get, list
# + providerhealths/status: get, patch
```
Vet, race tests and the envtest round-trip (using the `bin/k8s/1.36.2-linux-amd64`
binaries `make setup-envtest` had already fetched):
```bash
go vet ./internal/providerhealth/ && go test -race -count=1 -v ./internal/providerhealth/
# --- PASS: TestMonitor_envtestRoundTrip (6.45s) ... ok
```
Worth noting:
- **Fake client scope:** the fake client needs a RESTMapper that registers
ProviderHealth as `RESTScopeRoot`. Its default mapper treats every kind as namespaced.
- **Why the envtest calls `publish` directly:** `checkOnce` only logs write errors, so a
schema rejection would look like a missing object. Calling `publish` makes the API
server's answer the test failure.
- Its first round writes a 2000-character message; the stored message is truncated to
1024.
- Its second round proves that only the target that flipped gets a new
`lastTransitionTime`.
- **No guard against a hung checker:** the Monitor trusts `Checker`'s contract to return
once the context ends, and doesn't abandon a checker that ignores it. All three
checkers stop when the context ends: KVM through `awaitOrCancel`, and kubernetes and
GCP through their clients.
- **Test renamed:** the planned `TestMonitor_timeoutBoundsBlockingChecker` is
`TestMonitor_checkOnce_timeoutBoundsEachRound`. It also asserts that a fast provider
still publishes next to a hung one.
## Step 5 — Metrics and wiring
`metrics.RegisterProviderTargets` registers a scrape-time collector for three gauges,
each labelled `{provider,target}`: `up`, check duration and check timestamp.
`Register`'s signature is unchanged.
`cmd/main.go`:
- gains `--provider-check-interval` (1m) and `--provider-check-timeout` (15s);
- validates them before even loading the providers config;
- collects the `Checker`s from the unwrapped providers in config order, just before the
metrics and tracing wrap loop;
- adds the Monitor with the manager's API reader;
- registers the collector through a small `providerTargetSamples` closure next to
`proxyPhaseCounts`.
Build, vet and metrics tests:
```bash
go build ./... && go vet ./cmd/... ./internal/metrics/ && go test -race -count=1 ./internal/metrics/
```
To confirm bad flags fail fast, before anything touches the cluster, I ran the binary
with a 5s interval. zap's development mode prints a stack trace after the error, hence
the grep:
```bash
go run ./cmd --providers-config=/dev/null --provider-check-interval=5s 2>&1 | grep -m1 "Invalid provider check flags"
# ERROR setup Invalid provider check flags {"error": "--provider-check-interval must be 0 (off) or at least 10s, got 5s"}
go vet -tags=e2e ./test/e2e/
```
The e2e suite gets a new ordered spec, "should publish provider health". It waits for
`providerhealth kubernetes` to report `Ready=True`. It runs before the existing metrics
spec, which now also asserts
`proxy_operator_provider_target_up{provider="kubernetes",target="apiserver"} 1`. By then
the first round is known to be done, so the series is certain to exist.
Worth noting:
- **Main-only wiring:** `providerTargetSamples` is an 8-line loop in `package main`,
which has no test file. The e2e metrics assertion is its only coverage.
- **The KVM entry isn't asserted in e2e:** the default providers config also ships
`kvm-home` (beelink), which would report `SSHMaterialUnavailable` in kind. That entry
is the user's own host, and asserting it would break the suite as soon as the default
config changes.
- **Manifest:** `config/manager/manager.yaml` only gained a comment about the two flags.
The defaults need no args.
## Step 6 — Docs
- **`docs/architecture.md`:**
- component rows for the CRD, the `Checker` contract and the monitor;
- a new §11 covering the provider table, the loop diagram, the write, read and
decorator choices, the gauges, and sample `kubectl get ph` output;
- cross-references from the metrics (§9) and tracing (§10) sections;
- four Decisions entries.
- **`docs/kvm-provider.md`:**
- a "Health check" section with the reason → meaning → where-to-look table and the
two consequences of sharing the connection;
- under Known gaps, the capacity bullet now points at `poolAvailable`, and a new
bullet says a rotated `kvm-ssh` Secret needs a restart;
- the Testing section names `TestKVMCheck_realHost`.
- **`docs/testing.md`:**
- suite rows for the three `check_test.go` files, `internal/providerhealth` (unit and
envtest) and the metrics gauges;
- "17 libvirt calls" is now 20;
- the integration and e2e sections describe the new tests.
- **`CLAUDE.md`:**
- the Architecture TODO now points at `docs/architecture.md`;
- the stale "only `go.mod` exists" gotcha is replaced by a note that the "What this
is" and "Commands" sections predate the code, plus Step 4's RBAC-marker gotcha.
Worth noting:
- `docs/api.md` is untouched. It documents the discovery HTTP API, which this change
doesn't alter.
- `CLAUDE.md`'s "What this is" and "Commands" sections were left stale on purpose.
Rewriting them is outside this plan, so the new gotcha flags them instead of leaving
them silently wrong.
## Step 7 — Validate, record, push, open the PR
The full suite runs codegen, `go fmt` and `go vet`, then `go test -race` on every
package, with both envtest suites included:
```bash
make test
# every package ok; internal/providerhealth 92.5 %, internal/provider/kvm 92.0 % coverage
```
`go fmt` rewrote nothing, and `git status` showed only the Step 6 docs.
`make lint` could not run, for a reason unrelated to this branch. The Makefile rebuilt
the custom golangci-lint, which reports "built with go1.27.0", and its bundled
staticcheck (honnef.co/go/tools v0.7.0) panics on the standard library before it
reaches any repo package:
```bash
make lint
# buildir: package "poll" (isInitialPkg: false, needAnalyzeSource: true): unexpected expr: *ast.KeyValueExpr
# make: *** [Makefile:107: lint] Error 3
```
The local toolchain is go1.27.0 while `go.mod` says 1.26.0; the otel-tracing plan's
log shows `make lint` running normally before the toolchain moved. Retrying with
`GOTOOLCHAIN=go1.26.0` would have meant downloading a toolchain, and the user declined
that. `go vet` is clean in all three configurations: plain inside `make test`, plus
`-tags=integration` and `-tags=e2e`.
The kind e2e suite creates and deletes its own `egress-proxies-operator-test-e2e`
cluster. I checked `KIND_CLUSTER` first, because the two clusters already on the
machine (`kind`, `proxy-operator-demo`) must not be torn down by `cleanup-test-e2e`:
```bash
make test-e2e > <scratch>/e2e.log 2>&1
# Ran 3 of 5 Specs in 86.210 seconds
# SUCCESS! -- 3 Passed | 0 Failed | 0 Pending | 2 Skipped
```
In the deployed manager, the `kubernetes` ProviderHealth reported `Ready=True` about a
second after the pod was up (20:30:05.6 → 20:30:06.7). The metrics scrape that followed
contained `proxy_operator_provider_target_up{provider="kubernetes",target="apiserver"} 1`.
That exercises the generated RBAC, the API-reader path and the `main.go` gauge wiring
end to end. The two skipped specs are the Tempo-gated tracing specs; `TEMPO_URL` was not
set.
### Follow-up — the check must not cost the reconciles their connection
Reviewing the shared-connection decision after the automated validation turned up two
ways the health check could hurt the work it shares the link with, both of which the
plan's "a check against a hung host drops the shared connection" line accepted too
easily:
- A check that timed out went through `hostPool.call`, whose cancel path drops the
connection. A `libvirtd` slow to answer one `ConnectGetLibVersion` would therefore
cut off an image upload or a `Create` running on the same link.
- `hostEntry` was guarded by a `sync.Mutex`, so waiting for a host's slot could not be
interrupted. A reconcile redialing a host that is down holds the slot for as long as
the SSH dial takes — minutes on a blackholed address — and the whole check round
would wait behind it, past its own timeout.
So the check now calls `hostPool.probe` instead of `call`: on `ctx` ending it abandons
the RPC and leaves the connection alone (a link that is really dead is closed by the
keepalive anyway, which ends the abandoned `fn` too). One abandoned probe per host is
allowed to stay waiting — `hostEntry.probing` is an `atomic.Bool`, not a counter, so a
daemon that never answers cannot pile up goroutines; further probes get the new
`errProbeBusy`, which `checkReason` maps to `Timeout`. `hostEntry.mu` became a one-slot
channel with an `acquire(ctx)` that only bounds *waiting*: an uncontended caller takes a
free slot even with an expired context, so `call`'s behaviour is unchanged where nothing
is contended.
The second change came out of the same pass: `--provider-check-timeout` defaulted to a
flat 15s while the flag validation rejected a timeout longer than the interval, so
`--provider-check-interval=12s` alone failed to start. The flag now defaults to 0 and
`Monitor.setDefaults` derives `min(15s, Interval)`; `ValidateSchedule` accepts 0 and
rejects only negatives and timeouts past the interval.
Re-verified with the same commands as above:
```bash
go vet ./...
go test -race -count=1 ./internal/provider/kvm/ ./internal/providerhealth/
make test
# every package ok; internal/provider/kvm 92.2 %, internal/providerhealth 93.7 % coverage
```
Worth noting:
- `make lint` still cannot run, for the toolchain reason recorded above; unchanged by
this follow-up.
- The two new `kvm` tests are timing-shaped by necessity — they assert that a caller
with a 50ms deadline does not wait out a dial that is blocked on a channel. The
assertions are one-sided (`> 1s` fails) so a slow machine does not make them flaky.
- `docs/architecture.md`, `docs/kvm-provider.md` and `docs/testing.md` were corrected in
the same commit: the old text promised the drop-on-timeout behaviour that is now gone.
- The real-hypervisor commands from the plan's Verification section are still the user's
to run; nothing here replaces them.
### Real-hypervisor validation and PR
The user ran the plan's real-hypervisor checks against beelink (the `TestKVMCheck`
integration test plus the manual `ProviderHealth` and metrics checks) and confirmed
they passed. After that the CHANGELOG entry went in, and the branch was pushed with
the token-file credential helper. The PR against `main` was opened through the Gitea
API because `tea` isn't installed.
Worth noting:
- The branch also carries `docs/plans/2026-09-16-0841-provider-blueprint.md`, on
purpose: that plan reaches `main` with this merge, and the blueprint work starts on a
new branch afterwards.
docs/plans-executions/2026-09-16-0841-provider-blueprint.md
+177
View File
@@ -0,0 +1,177 @@
# Execution: provider blueprint
Plan: [docs/plans/2026-09-16-0841-provider-blueprint.md](../plans/2026-09-16-0841-provider-blueprint.md)
- [x] Step 0 — Save the plan; wait for the merge
- [x] Step 1 — Collect evidence (read-only, in parallel)
- [x] Step 2 — Write `docs/providers.md`
- [x] Step 3 — Verify, then open the MR
## Step 0 — Save the plan; wait for the merge
The plan was committed on `feat/provider-health` (`80b2e9c`) and reached `main`
through PR #6. After the user confirmed the merge, the branch was cut from the
updated `main`:
```bash
git switch main && git pull --ff-only
git switch -c feat/provider-blueprint
```
Worth noting: `.claude/settings.json` has local, uncommitted permission edits
that carried over through the switch. They are deliberately left out of this
branch.
## Step 1 — Collect evidence (read-only, in parallel)
Three Explore agents answered the plan's 12-item questionnaire, one each for
kubernetes, gcp and kvm. Meanwhile I read the shared layer:
`internal/provider/*.go`, `registry.go`, the provider wiring in `cmd/main.go`,
`reconcileManaged`/`replaceInstance`/`reconcileDelete` in the controller,
`internal/gc/gc.go`, and how the monitor uses `Checker`.
Before relying on the agents' claims, I spot-checked them against the code. One
was wrong and did not go into the doc: the kvm agent said `Delete` skips volume
removal when `DomainDestroy` returns `ErrNoDomain`. In fact `removeDomain`
returns nil there, and `Delete` still calls `removeVolumes` afterwards
(`kvm.go:477-482`).
Rules that exist only in the callers, and not in any comment, were found by
reading the controller and GC. They went into the doc as MUST rules:
- A zero `CreatedAt` makes an instance immediately collectable (`gc.go:120`).
- `Get` returning `ErrPermanent` on the deletion path makes the reconciler call
`Delete` anyway.
- Replacement waits for `Get` to return NotFound before creating an instance
with the same name.
- `Stopped` and `Terminated` both lead to delete-and-recreate.
Worth noting, found outside this plan's scope and not changed here:
- `docs/gcp-vm-validation.md` filters on `labels.proxy-operator-managed=yes`,
but the value is `true`.
- According to the gcp agent, `docs/gcp-in-specific-project.md` holds a real
project ID, IAM output and an egress IP. The user was told.
## Step 2 — Write `docs/providers.md`
Wrote the eight sections the plan lists. Linked the new doc from the Provider
contract row in `docs/architecture.md` and from the CLAUDE.md "Architecture"
section.
Judgment calls:
- The file:line evidence sits in per-provider notes under the matrix, as
planned. The contract rules in §2 cite the reconciler and GC lines each rule
protects.
- The gap list includes documentation drift (kvm-provider.md's ARP order and
log keys, gcp-vm-validation.md's label value and its claim that async errors
reach status), as well as code gaps. Both would mislead the next
provider author in the same way.
- "Proposed features" stays a flat list with no design, as the user decided.
Worth noting: the line numbers in the doc are a snapshot. The doc says so and
asks readers to look for the named function when a line has moved.
## Step 3 — Verify, then open the MR
An Explore agent re-checked the doc against the code: about 150 claims,
including every matrix cell. It found four mismatches, and all four were fixed:
- **Startup dial:** I had listed kvm-provider.md's "connection kept open from
startup" as doc drift. It isn't: the pool dials lazily, but the provider
health monitor runs a check as soon as the leader starts
(`monitor.go:147`, default interval 1m), and that check opens the connection.
The bullet was removed and the kvm notes now say this.
- **Checker wording:** §5 said the monitor type-asserts `Checker`. It is
`cmd/main.go:216`; the monitor only sees nil or not.
- **Reasons, not conditions:** §3 called `QuotaExceeded` and `PermanentError`
conditions. They are reasons on `Provisioned`.
- **Latch lines:** the line range is `:133-139`, not `:130-135`.
It also noted that none of the providers assert `var _ provider.Provider`, and
that the kubernetes provider's RBAC markers live in the controller package. The
§7 checklist now says both.
Link check (every relative `](…)` target and anchor in the new and edited docs
resolves), then the sanity build:
```bash
go build ./... && go test -short ./...
# all packages ok
```
Worth noting: no CHANGELOG entry yet. Per CLAUDE.md it goes in after the user
confirms the doc.
### Follow-up — tracing was missing from the blueprint
The user pointed out, after the PR was opened, that the blueprint said almost
nothing about tracing. The plan's questionnaire item 10 had collected the
evidence, but the doc reduced it to a matrix row about logging and a checklist
line telling authors not to open spans. That line even suggests tracing needs
no work. A provider author needs the opposite message: the operation span comes
free, and child spans for backend calls are theirs to add.
Added a new §6, "Observability: tracing, logs and metrics", and renumbered the
matrix, checklist and gaps sections to §7–§9. It covers:
- **Three layers:** decorator operation spans; child spans per backend call,
with parent-gated HTTP transports or a span decorator over the SDK seam; and
logs.
- **The ctx and logger rules** that keep `traceID` on log lines.
- **A per-provider status table:**
- kubernetes: parent-gated `RestConfigWrapper` spans.
- gcp: the Google SDK's own `otelhttp`, never disabled by the provider.
- kvm: no spans. `trace.go` is a log decorator.
Also added two tracing rows to the §7 matrix, rewrote checklist item 8 as
"Tracing and logging" with an in-memory span-recorder test, and added gaps (kvm
has no child spans; only kubernetes is checked against Tempo, and only at
operation level). A proposal notes that the kvm `libvirtAPI` seam takes no
`ctx`, so per-RPC spans need that plumbed in.
The links were re-checked, including the new `architecture.md#10-tracing-internaltracing` anchor.
### Follow-up — a completeness sweep
The user asked for one more pass over everything provider-related. After the
tracing miss, this pass read beyond the provider packages:
- the Proxy API types and the spec hash;
- the health engine and discovery call sites that consume `Instance.IP`;
- the manager Deployment, the providers ConfigMap and the samples;
- README, testing.md, kvm-provider.md and the GCP docs;
- the initial prompt, the proxy-operator and kvm plans, and the 2026-08-10 PR
review;
- the CHANGELOG.
What the blueprint was missing, now added:
- **What the instance is for.** The IP and port must be reachable from the
operator's health probes and from discovery clients. Providers don't know
which proxy runs; the proxy comes from cloud-init, except for kubernetes.
- **Concurrency.** Three reconciles, GC and checks can call a provider at once,
so it MUST be safe for concurrent use.
- **The runtime envelope:** read-only root filesystem, a 64Mi `/tmp`, a 128Mi
memory limit, credentials as optional mounts.
- **Create:** `placement` can't be validated by the CRD, so error messages must
name the field. Catalogue edits in config aren't in the spec hash.
- **Delete** is re-called every `DeletionPoll` while a delete is in progress.
- **Provider names** must be DNS-1123 for `ProviderHealth`, and `spec.provider`
is immutable.
- **Metric names** in §6.
- **Three matrix rows:** proxy software, IP reachability, distinct egress.
- **A deployment checklist item:** credentials mount, config entries, sample
Proxy, Makefile helper.
- **Gaps:**
- still open from the PR review: GC has no installation identity; an
unconfigured provider latches Proxies; kubernetes Pod RBAC always installed
and no `OwnerReference` on Pods;
- the duplicated type list;
- samples that can't work: `proxy_gcp.yaml` uses a debian-cloud image without
cloud-init, and `proxy_kvm.yaml` places on `nas1` where configs define
`beelink`;
- the `"mock"` example in the `spec.provider` doc.
Outside the plan's "leave other docs alone" line, README.md changed:
- Its "Adding a provider" section restated the contract, with the old "Create
never blocks" rule that kvm contradicts. It now points to `docs/providers.md`
instead of keeping a second copy that disagrees.
- The "Implemented providers" table gained the missing kvm row.
- The deploy note no longer claims the default ConfigMap ships only the
kubernetes provider.
The links were re-checked for providers.md and README.md.
docs/plans-executions/2026-09-16-1108-provider-docs-split.md
+228
View File
@@ -0,0 +1,228 @@
# Execution: Split the provider blueprint into contract, appendix, template and index
Plan: [docs/plans/2026-09-16-1108-provider-docs-split.md](../plans/2026-09-16-1108-provider-docs-split.md)
- [x] Step 0 — Save and commit the plan
- [x] Step 1 — Write `docs/provider-contract.md`
- [x] Step 2 — Write `docs/provider-spec-template.md`
- [x] Step 3 — Write `docs/providers-existing.md`
- [x] Step 4 — Rewrite `docs/providers.md` as the index; link edits
- [x] Step 5 — Verification sweep
- [x] Step 6 — Final summary, CHANGELOG after confirmation
## Step 0 — Save and commit the plan
Committed as `cf89ac4` on the existing `feat/provider-blueprint` branch, on
its own, before any document work. The pre-existing local modification of
`.claude/settings.json` was left out of the commit.
## Step 1 — Write `docs/provider-contract.md`
2026-09-16 11:17. Written from scratch rather than edited down from the old
blueprint, so that nothing provider-specific could survive by accident. Every
MUST names its caller by function (`reconcileManaged`, `replaceInstance`,
`reconcileDelete`, `providerFailure`, `deletionFailure`, `Sweeper.sweep`,
`Monitor.checkOnce`, `Monitor.run`, `registry.Build`) and states what breaks.
**Axis codes frozen** (19, §4.1 to §4.19): `readiness`, `create-return`,
`delete-return`, `ip-source`, `gc-mark`, `created-at`, `list-partial`,
`quota`, `not-found`, `primary-key`, `provisioning-state`, `async-ops`,
`secondary-resources`, `credentials`, `client-kind`, `timeouts`, `logging`,
`child-spans`, `backend-test`. The appendix table and the template are keyed
on these strings; the §4.20 summary table's last column gives the template
entry number each axis maps to.
**Two rules that were not in the old blueprint**, both found by reading the
callers while verifying:
- `Sweeper.sweep` loops providers sequentially in one goroutine, and no
caller puts a deadline on the context (only `Monitor.checkOnce` uses
`WithTimeout`). So a hung `ListByTag` stalls orphan GC for every provider.
Recorded in §1.5 and the `timeouts` axis.
- The Stopped branch of `reconcileManaged` calls `Delete` and requeues
without clearing `ProviderID`, then deletes again on every poll until
NotFound. A backend that reports a freshly created, never started instance
as stopped would have it deleted right after creation. Recorded as a MUST
in §2.2 and as the `provisioning-state` axis with three allowed profiles.
Checks run on the draft (all clean):
```sh
grep -niE "kubernetes|gcp|kvm|libvirt|squid|google|compute engine" docs/provider-contract.md
grep -n "—" docs/provider-contract.md
grep -nE '\.go:[0-9]+|:[0-9]+-[0-9]+' docs/provider-contract.md
grep -ohE "(internal|cmd|config|docs|api|hack|test)/[A-Za-z0-9_./-]+" docs/provider-contract.md | sort -u | while read p; do test -e "$p" || echo "MISSING $p"; done
```
The path check reports three expected misses: `config/samples/proxy_<type>.yaml`
is a pattern, `docs/provider-spec-template.md` is written in Step 2, and
`internal/provider/providertest` is a proposal named in §13.
Worth noting: the contract refers to "the operator pod" and "the cluster's
own API" rather than naming the platform, so that the abstractness grep stays
meaningful. The `hack/providers-dev.yaml` rule is phrased as "only if the type
needs no credentials" because that file exists for `make run-dev` without any
Secret.
## Step 2 — Write `docs/provider-spec-template.md`
2026-09-16 11:28. Twenty-four entries in a fixed shape (Question, Allowed
answers, Obligations, Answer, Evidence). Thirteen of them carry an axis code
and their numbers are the ones the contract's §4.20 table points at; the
other eleven (identity, placement, image, cloud-init, machine type, config
block, Checker, concurrency, deployment, docs, accepted limits) are decisions
the contract leaves open without being a profile axis. Entries 9, 13 and 15
carry an empty table to fill in (state mapping, error classification, config
fields); entry 21 carries the contract §11 test list as checkboxes so a
filled spec doubles as the test plan.
Check that every axis code appears in the template:
```sh
for c in $(grep -oE '^### 4\.[0-9]+ `[a-z-]+`' docs/provider-contract.md | grep -oE '`[a-z-]+`' | tr -d '`'); do
grep -q "\`$c\`" docs/provider-spec-template.md || echo "template missing $c"
done
```
Clean. Worth noting: the preamble allows "verify against the target version"
only on an Evidence line, never as an Answer, because the whole point of the
file is that the implementer does not guess.
## Step 3 — Write `docs/providers-existing.md`
2026-09-16 11:35. The profile table has exactly one row per contract axis
code, checked by the same loop as Step 2 with a row count. Everything else
comes from the old blueprint's §6, §7 and §9 plus a fresh read of the three
packages, with every `file:line` replaced by a symbol name.
Corrections to earlier notes applied while writing: kvm's Checker `Details`
also carry `user` and `pool`; the kubernetes provider has no seam interface
(`newWithClient` injects a `client.Client`, the fake client stands in); kvm's
`classify` has an `isClassified` pass-through. Timeouts were checked per
provider for the `timeouts` row: kubernetes sets no `rest.Config.Timeout`,
gcp relies on SDK defaults, kvm bounds only the SSH handshake
(`defaultHandshakeTimeout` in `dialSSH`), so all three are weaker than the
contract's SHOULD and the table says so.
Symbol check on the appendix (every backticked identifier grepped for in
Go, Makefile and YAML sources):
```sh
grep -ohE '`[A-Za-z_][A-Za-z0-9_.]*`' docs/providers-existing.md | tr -d '`' | sort -u | while read s; do
grep -rqwF --include='*.go' --include='Makefile' --include='*.yaml' -- "$s" . || echo "$s"
done
```
The residue is file names, dotted `Type.method` forms, the `timeouts` axis
code, libvirt's `nostate`, and the API names `OwnerReference` and
`rest.Config.Timeout`; nothing unexpected.
Worth noting: matrix rows that the profile table now carries (Create
non-blocking, GC marks, IP source, quota detection, Checker presence,
credentials, logging, tracing, tests) were dropped from "Other differences"
rather than kept twice.
## Step 4 — Rewrite `docs/providers.md` as the index; link edits
2026-09-16 11:40. `docs/providers.md` was rewritten in place (same path, so
the inbound links from `README.md`, `CLAUDE.md`, `docs/architecture.md` and
the historical plan files still resolve, and the pre-split blueprint stays
reachable in the path's git history). The three inbound sentences were
updated to name the new files:
- `README.md` "### Adding a provider": the first sentence now names the
contract, the template and the appendix, with `docs/providers.md` as the
index; the "Start there" sentence is unchanged.
- `CLAUDE.md` "## Architecture": the providers.md sentence now names all
three files and the index.
- `docs/architecture.md` components row "Provider contract": "rules in
provider-contract.md, per-provider profile table in providers-existing.md".
The historical files under `docs/plans/` and `docs/plans-executions/` keep
their `providers.md` links on purpose.
## Step 5 — Verification sweep
2026-09-16 11:45. All ten checks from the plan, run against the working tree
after Step 4; nothing needed fixing.
Links (file targets and anchors) across the four new files plus `README.md`,
`CLAUDE.md` and `docs/architecture.md`, with anchors compared against
slugified headings of the target file:
```sh
python3 - <<'PY'
import re, pathlib
files = ["docs/provider-contract.md","docs/provider-spec-template.md","docs/providers-existing.md","docs/providers.md","README.md","CLAUDE.md","docs/architecture.md"]
def slug(h):
h = re.sub(r'`','',h).strip().lower(); h = re.sub(r'[^\w\s-]','',h)
return re.sub(r'\s+','-',h).strip('-')
def headings(p):
return {slug(m.group(2)) for m in (re.match(r'^(#{1,6})\s+(.*)$', l) for l in open(p)) if m}
bad = 0
for f in files:
for m in re.finditer(r'\]\(([^)\s]+)\)', open(f).read()):
t = m.group(1)
if t.startswith(('http://','https://','mailto:')): continue
path, _, anchor = t.partition('#')
target = pathlib.Path(f) if path == '' else pathlib.Path(f).parent/path
if not target.exists(): print(f, "missing", t); bad += 1; continue
if anchor and target.suffix == '.md' and anchor not in headings(target): print(f, "bad anchor", t); bad += 1
print("link problems:", bad)
PY
```
Output: `link problems: 0`.
Em-dashes, line references and the contract's abstractness:
```sh
grep -n "—" docs/provider-contract.md docs/provider-spec-template.md docs/providers-existing.md docs/providers.md
git diff 2d959af -- README.md CLAUDE.md docs/architecture.md | grep '^+' | grep "—"
grep -nE '\.go:[0-9]+|:[0-9]+-[0-9]+' docs/provider-contract.md docs/provider-spec-template.md docs/providers-existing.md docs/providers.md
grep -niE "kubernetes|gcp|kvm|libvirt|squid|google|compute engine" docs/provider-contract.md
```
The only em-dash hit is the pre-existing "Leader-elected" cell of the
`architecture.md` components row, which shows as an added line because the
row's last cell was edited; no new em-dash was written. The other three
greps are empty.
Nothing lost from the old blueprint's §9: the bold leads of every gap and
proposal bullet in `git show 0bb699b:docs/providers.md` were looked up in
the new files. The literal-prefix match reported 22 misses, all caused by
rewording ("isn't" to "is not", bold removed), so each topic was re-checked
by a distinctive phrase and all 22 are present in `providers-existing.md`
or contract §13.
Paths and symbols: every `path/...` token in the four files exists except
the `config/samples/proxy_<type>.yaml` pattern and the
`internal/provider/providertest` proposal; every backticked identifier in
the contract and template resolves in Go, Makefile or YAML sources except
file names, dotted `Type.method` forms, profile names and the `NewWith...`
placeholder.
Axis alignment: the 19 axis codes from contract §4 each appear once as a
row in the appendix table (six cells per row) and at least once in the
template.
`go build ./... && go test -short ./...`: all packages `ok`.
## Step 6 — Final summary
The split is complete on `feat/provider-blueprint`: five commits after the
plan commit, one per step. The CHANGELOG entry is deliberately not written
yet; per `CLAUDE.md` it is added once the user confirms the documents read
the way they want. Nothing has been pushed.
Worth noting for whoever picks this up:
- The contract's §4.20 table maps axis codes to template entry numbers; if
entries are renumbered, that column must follow.
- The `CHANGELOG.md` heading format in `CLAUDE.md` uses an em-dash; the
"no em-dashes" rule applied to the four new documents only.
- The stale `mock` comments in `internal/provider/provider.go`,
`internal/provider/registry/registry.go` and `api/v1alpha1/proxy_types.go`
are still listed as gaps, not fixed here.
- The next planned use of these files is a Proxmox provider: copy the
template to `docs/proxmox-provider-spec.md` and fill it in before any code.
docs/plans-executions/2026-09-16-1333-kvm2-provider.md
+382
View File
@@ -0,0 +1,382 @@
# Execution: kvm2, a clean-room libvirt/KVM provider built from the provider contract
Plan: [docs/plans/2026-09-16-1333-kvm2-provider.md](../plans/2026-09-16-1333-kvm2-provider.md)
- [x] Step 0 — Plan file and branch
- [x] Step 1 — The spec, `docs/kvm2-provider-spec.md`
- [x] Step 2 — Config wiring, IDs, state mapping, classify
- [x] Step 3 — Pure builders: domain and volume XML, metadata, seed
- [x] Step 4 — Transport, seam, decorator, fakes
- [x] Step 5 — Image cache and pool-space precheck
- [x] Step 6 — The four contract methods
- [x] Step 7 — Checker and the CRD comment
- [x] Step 8 — Deployment files, Makefile, integration test
- [x] Step 9 — Documentation
- [x] Step 10 — Milestone review
- [ ] Step 11 — Real-host validation, changelog, MR
## Step 0 — Plan file and branch
`main` was fast-forwarded to the merge of PR #7 (`0dc25a1`), which carries the
three provider documents, and `feat/kvm2-provider` was cut from it. The plan
was committed alone as `854856c`. The pre-existing local modification of
`.claude/settings.json` stays uncommitted throughout.
## Step 1 — The spec, `docs/kvm2-provider-spec.md`
2026-09-16 13:37. Written by the orchestrator, not a subagent, since the spec
is the "specifics" a user hands to the implementer. All 24 entries answered;
every Evidence line cites the v1 user guide, the 2026-09-04 execution log, a
go-libvirt symbol, or the contract. Nothing was taken from
`internal/provider/kvm/`.
Design choices that go beyond the user's profile answers, all recorded in the
spec: an adoption guard on `Create` (foreign UID is Permanent), a pool-space
precheck mapped to `ErrQuotaExceeded`, a dedicated connection for image
downloads so a cancelled poll cannot cut one off, the Checker on fresh
connections rather than the pool, SSH authentication and host-key failures
classified Permanent (v1 retried them forever), SSH material re-read on every
dial so rotation needs no restart, kvm2's own metadata namespace (no adoption
of v1 domains), and the shared `validateKVM` refactored to take the block name
so messages say `kvm2.`.
Checks run on the spec:
```sh
grep -c '^\*\*Answer:\*\*$' docs/kvm2-provider-spec.md # 0 blank answers
grep -n 'internal/provider/kvm/' docs/kvm2-provider-spec.md # empty
```
plus the axis-code check against contract §4 headings and the path-existence
loop from the docs-split log. All clean.
Worth noting: the spec's entry 21 lists the concrete test name for every
contract §11 item, so the later steps' briefs can quote it rather than
re-derive the mapping.
## Step 2 — Config wiring, IDs, state mapping, classify
2026-09-16 16:38. First subagent step. The brief named the allowed files,
the spec entries, the test names and the prohibition on
`internal/provider/kvm/`; the agent reported nothing opened there, with one
honest caveat: a `golangci-lint` run printed seven single source lines from
v1 files among its findings, which it did not act on. The review greps
(`grep -rn 'provider/kvm"' internal/provider/kvm2/`, the v1 identifier
smell list) are clean.
What landed: `ProviderConfig.KVM2` reusing `KVMConfig`, `setBlocks`, the
allow-list and dispatch in `Config.validate`, `validateKVM` refactored into
`validateKVMBlock(prefix, block, c)` with `validateKVM` kept as a wrapper so
v1 messages and tests are untouched, `"kvm2": kvm2.New` in `cmd/main.go`
(both type lists in one commit), and the package skeleton: `kvm2.go`
(`New` never dials, warns on unreadable files, `withDefaults`, `host`,
`wrapErr`, stubbed methods), `id.go`, `state.go`, `errors.go` with 34
`classify` rows.
Decisions the agent made that the spec left open, all accepted: `New`
checks file existence only (parsing belongs to the dialer); `withDefaults`
copies hosts and images; `firstIPv4` also drops an address whose type tag and
literal disagree; there is no explicit `fs.ErrNotExist` branch in `classify`
because it falls to the Transient default anyway.
Commands as run by the agent and re-run by the orchestrator:
```sh
go vet ./... && go build ./... && go test -race -count=1 ./internal/provider/... ./cmd/...
go test -short ./...
```
All packages `ok`.
Worth noting: `bin/golangci-lint` panics inside staticcheck's IR builder on
this machine (`unexpected expr: *ast.KeyValueExpr` on stdlib `internal/poll`)
because the local toolchain is Go 1.27.0 while `go.mod` says 1.26 and the
bundled staticcheck predates it; it reproduces on untouched packages, so lint
is deferred to Step 10 with a toolchain note. The `.claude/settings.json`
modification is still local and uncommitted.
## Step 3 — Pure builders: domain and volume XML, metadata, seed
2026-09-16 16:55. Second subagent step, new files only: `xml.go`,
`metadata.go`, `volume.go`, `seed.go` and their tests (26 tests). The
domain XML is built from `encoding/xml` structs, both disks are
`type="file"` with resolved paths (the AppArmor lesson), UEFI is the default
with the secure-boot feature disabled, the metadata element parses with or
without its namespace declaration, and the seed ISO is staged under
`os.TempDir()` (the library uses `os.MkdirTemp("", "")`, so `TMPDIR` is
honoured; the test points it at a `t.TempDir()` and asserts nothing is left
behind).
Decisions the agent made, all accepted: `buildDomainXML` defaults an empty
disk format to qcow2 and rejects anything but qcow2 or raw;
`buildVolumeXML` rejects a non-positive capacity; `buildSeedISO` rejects an
empty name; the guest-agent script still runs the enable-and-start step when
the agent is already installed; the multipart boundary is random, so
vendor-data is not byte-deterministic across calls (the seed is rebuilt every
Create anyway).
Commands:
```sh
gofmt -l ./internal/provider/kvm2 ; go vet ./internal/provider/kvm2/ && go test -race -count=1 ./internal/provider/kvm2/
go build ./... && go test -short ./...
```
All `ok`. Review greps for a v1 import or v1 identifiers are empty; the only
`type="volume"` in the package is the comment explaining why it is not used.
The agent reported "files read outside the allowed set: none".
## Step 4 — Transport, seam, decorator, fakes
2026-09-16 17:42. Third subagent step, the largest: `backend.go` (the
ctx-first seam `backend`, the production adapter `libvirtBackend`, a
generic `call[T]` helper with per-call budgets and the abandon policy),
`ssh.go` (credentials read on every dial, stage-tagged `dialError`, the
forwarded socket, keepalive), `hosts.go` (`hostTable` with a per-host
channel semaphore so waiting for a dial ends with the caller's ctx,
`dialFresh` for dedicated connections, the production `libvirtDialer`
starting the `kvm2.dial` span and applying the decorator once),
`observe.go` (one span `kvm2.<Method>` and one `libvirt call` line per
RPC), plus an in-memory `fakeBackend`, an in-process SSH server and 41
tests. The agent also drove the real adapter over go-libvirt's own
`libvirttest` mock through a `net.Pipe`, which the brief did not ask for.
Abandon policy as built: a call that outlives its ctx or budget kills the
link (closes the socket and the SSH client), returns Transient wrapping the
ctx or deadline error, and other in-flight calls on that connection fail once
with go-libvirt's `ErrInterrupted`; the next `get` redials. Results travel
through a channel so a late-finishing abandoned call never writes into its
caller's frame; the agent reported that the first `-race` run caught exactly
that bug in an earlier version.
Decisions the agent made, accepted: liveness is a separate unexported
interface so the reflection test over `backend` stays at the 20 RPCs plus
`Close`; `Close` sends a bounded `ConnectClose` before killing the link;
`observe` and the dialer take `...tracing.Option` so tests can inject a
recording tracer provider, as `provider.WithTracing` does.
Clean-room note: three type names coincide with names v1 uses
(`dialError`, `dialStage`, `hostEntry`). The brief described each concept
("an error type that records which stage failed", "a per-host entry") so the
names follow from the description; the greps for a v1 import and for the
listed v1 identifiers are empty and the agent attested to reading nothing
under `internal/provider/kvm/`.
Commands (agent and orchestrator):
```sh
gofmt -l ./internal/provider/kvm2 ; go vet ./internal/provider/kvm2/ && go test -race -count=2 ./internal/provider/kvm2/
go build ./... && go test -short ./...
grep -n 'context.Background()' internal/provider/kvm2/*.go | grep -v _test.go # empty
```
All `ok`; the agent also ran the timing-sensitive tests at `-count=8`.
## Step 5 — Image cache and pool-space precheck
2026-09-16 18:11. Fourth subagent step: `image.go` and `image_test.go`
(27 tests), plus additive changes to `fake_test.go` (a `twin()` sharing
state with its own call log, so a test can tell the pooled connection from
the dedicated one) and `kvm2.go` (`http` client, budgets and tracing
options behind functional options, so `New` is unchanged). The pipeline:
marker lookup and bounded read on the pooled connection, then a GET bound to
ctx plus the 20-minute cap under a `kvm2.download` span, `StorageVolCreateXML`
and a streamed `StorageVolUpload` through sha256 on a fresh connection,
`StoragePoolRefresh`, and the marker written last; every failure after the
volume exists discards it with the cancellation stripped from ctx so the
cleanup RPC still runs.
Streaming is asserted by counting: a hook on the dedicated fake's upload
snapshots how many bytes the 64 MiB test server had written when the upload
began, and asserts under 16 MiB. The agent chose that over a process-wide
`MemStats` delta because the tests run in parallel.
Decisions the agent made, accepted: the base volume is created on the
dedicated connection (nothing exists if the dial fails); `Accept-Encoding:
identity` is sent so transparent gunzip cannot hide the Content-Length; a
marker whose content is not 64 hex characters counts as debris;
`Content-Length: 0` is its own Permanent error; the image lock is a
one-slot channel so a waiter gives up with its ctx instead of sitting out a
transfer; the fetch logger is derived after the span starts so its lines
carry the span ID (the agent's first test run caught that).
Commands (agent and orchestrator):
```sh
gofmt -l ./internal/provider/kvm2 ; go vet ./internal/provider/kvm2/ && go test -race -count=2 ./internal/provider/kvm2/
go build ./... && go test -short ./...
grep -n 'io.ReadAll\|bytes.Buffer' internal/provider/kvm2/image.go # empty
grep -n 'context.Background()' internal/provider/kvm2/*.go | grep -v _test.go # empty
```
All `ok`. Attestation: "files read outside the allowed set: none".
## Step 6 — The four contract methods
2026-09-16 18:36. Fifth subagent step: `create.go`, `get.go`, `delete.go`,
`list.go` and five test files (the package now has 150 top-level tests).
Create follows spec entry 10 step for step under a per-name lock, with the
adoption guard, the quota precheck, the image cache, an overlay that is
reused and a seed that is always rebuilt, file-type disks from resolved
paths, the metadata element in the define call, and a wait for running
capped at two minutes. Get maps only `ErrNoDomain` to NotFound, prefers the
guest agent over ARP, and reads UID and CreatedAt back every time. Delete is
idempotent at every step and a no-op success for an unconfigured host.
ListByTag fails whole per host and skips single unreadable domains.
Decisions the agent made, accepted: the adoption path also waits for
running, so a repeat Create on a shut-off domain blocks like a first one;
`address` and `ListByTag` propagate ctx cancellation instead of reading it as
"no address" or "skip", so a cancelled Get is Transient rather than a false
Provisioning; NotFound from Get gets its own V(1) line because polling to
NotFound is expected; the domain XML rides on the `domain defined` line at
V(2). Flagged, not solved: adopting a paused domain fails Transient on every
retry (`DomainCreate` refuses and the wait expires), which the spec does not
cover; recorded as a known gap for the docs step.
The agent found a one-word conflict between the brief and the spec
(`DomainAffectCurrent` versus `DomainAffectConfig` for `DomainGetMetadata`);
the code uses `DomainAffectCurrent`, and spec entry 8 was corrected to
match in this commit.
Commands (agent and orchestrator):
```sh
gofmt -l ./internal/provider/kvm2 ; go vet ./internal/provider/kvm2/ && go test -race -count=2 ./internal/provider/kvm2/
go build ./... && go test -short ./...
grep -n 'CloudInit' internal/provider/kvm2/create.go # only len(req.CloudInit) and the seed builder call
grep -n 'not implemented' internal/provider/kvm2/*.go # empty
grep -n 'provider.Wrap(' internal/provider/kvm2/*.go | grep -v _test.go | grep -v 'p.name' # empty
```
All `ok`. Attestation: "files read outside the allowed set: none".
## Step 7 — Checker and the CRD comment
2026-09-16 18:52. Sixth subagent step: `check.go` and `check_test.go`
(ten tests, the reasons table has 19 rows). Every host is checked
concurrently on a fresh dedicated connection that is closed when the probe
ends; the pooled table is never consulted, which the test proves with a
counting dialer. Details fill in as facts arrive, so a `PoolNotFound`
verdict still says which libvirt version answered. The `status.type` comment
in `api/v1alpha1/providerhealth_types.go` gained `kvm2` and `make manifests`
changed only the CRD's description line:
```sh
make manifests && git diff --stat config/ api/
# api/v1alpha1/providerhealth_types.go | 2 +-
# config/crd/bases/crawl.example.com_providerhealths.yaml | 2 +-
```
Decisions the agent made, accepted: a dial that runs out of time is
`Unreachable` (the host did not answer), and `Timeout` is reserved for ctx or
budget expiry on an established connection; `Close` runs synchronously in
the probe so the close-count assertion is deterministic, at the cost of up
to five seconds past ctx if a disconnect hangs on a live link.
The agent also reported that the `captureContext` helper in `kvm2_test.go`
appends to an unlocked slice, which the race detector flagged once `Check`
logged from several goroutines. The helper was outside its allowed file set,
so it built its own locked capture; the orchestrator fixed `captureContext`
with a mutex in this commit and re-ran the package at `-race -count=3`.
Attestation: "files read outside the allowed set: none".
## Step 8 — Deployment files, Makefile, integration test
2026-09-16 19:39. Seventh subagent step. `kvm2_integration_test.go`
(`//go:build integration`, `KVM2_TEST_*` variables, the eleven ordered
assertions of spec entry 21 plus `TestKVM2Check_realHost`),
`config/samples/proxy_kvm2.yaml` (zone `beelink`, so the sample matches the
shipped config), a commented `kvm2-home` entry in both providers configs
with the porting note, comment-only edits on the shared `kvm-ssh` mount, the
`test-integration-kvm2` Makefile target and the kvm2 rows and section in
`docs/testing.md`. The integration test was not run; that is Step 11 with
the user's go-ahead.
The agent found that `New` builds the production dialer without tracing
options, so in production the dialer and decorator use the global tracer
provider (correct: `cmd/main.go` sets it up), while a test that wants every
span family in one recorder sets the dialer's option itself. The test does
so and asserts that `kvm2.DomainDefineXML` and `kvm2.dial` are children of
`provider.create`.
Decisions the agent made, accepted: the UID is generated from `crypto/rand`
rather than promoting `google/uuid` to a direct dependency; the sentinel
check looks for the full cloud-init string because the provider name also
appears in attributes; `-run TestKVM2` matches both tests, as the kvm
target's pattern does. Forward links to `docs/kvm2-provider.md` dangle until
Step 9.
Commands:
```sh
go vet ./... && go vet -tags=integration ./... && go vet -tags=e2e ./...
bin/kustomize build config/default > /dev/null && bin/kustomize build config/samples > /dev/null
make -n test-integration-kvm2
# go test -race -tags=integration -count=1 -v -timeout 40m -run TestKVM2 ./internal/provider/kvm2/
```
All clean. Attestation: "files read outside the allowed set: none".
## Step 9 — Documentation
2026-09-16 19:54. Eighth subagent step: `docs/kvm2-provider.md` (sixteen
sections modelled on the kvm guide), a `kvm2` column in the three tables of
`docs/providers-existing.md` plus a kvm2 section and renumbering, the
components row, §5 block, §10 span bullet, §11 row and eight Decisions
bullets in `docs/architecture.md`, a README row and paragraph, and the
guide in the `docs/providers.md` index. The agent verified each claim
against the package rather than the spec and dropped a draft sentence
claiming kvm and kvm2 share the base-image cache, because checking it would
have meant reading v1's volume names.
Checks re-run by the orchestrator: no em-dash in the guide, no line
references, each of the 19 axis codes appears once in the appendix table
with seven cells, every link and anchor across the seven touched documents
resolves. The one added em-dash is the `Leader-elected` cell of the new
components-table row, where the table's own convention wins.
Attestation: "files read outside the allowed set: none".
## Step 10 — Milestone review
2026-09-16 21:19. The `go-operator-reviewer` agent reviewed the package
against the contract and the spec (v1 code off limits to it as well) and
returned seven findings, all accepted and applied by a fresh fix agent in
one commit with regression tests:
1. `waitRunning`'s two-minute cap only ever elapsed in the bad cases and
pinned a reconcile worker; now 15 s, a domain that dies right after
start is Transient at once, and a paused or pmsuspended domain is
Permanent before `DomainCreate` (an admin's doing).
2. `ListByTag` read state and addresses that GC never uses, one agent RPC
per active domain per sweep; it now builds instances from the name and
the metadata only (`TestListByTag_doesNotReadStateOrAddresses`).
3. A health round that ran out of time was blamed on the host
(`Unreachable`); `checkHost` now reports `Timeout` whenever the round's
ctx ended, whichever dial stage was running, and the promptness test
uses the production error shape.
4. `Create` adopted an unmarked domain and never wrote the mark, so GC could
not see it; an unmarked domain under the name is now Permanent
(`TestCreate_adoptsOnlyMarkedDomains`).
5. `hostTable.closeAll` had no production caller and `withBudgets` could not
work (the dialer captured the defaults first); both removed.
6. The per-name and per-image lock maps grew without bound; replaced by a
reference-counted `keyedLocks` whose entries vanish when uncontended.
7. `imageSizeIfUncached` falls back to GET on any non-2xx HEAD, and two
Delete no-op lines moved from V(0) to V(1).
The spec (entries 2, 10, 12, 16, 18, 19, 24), the guide, the appendix and
the architecture flow were updated in the same commit. The reviewer also
listed what it checked and found correct: the error boundary, the state
rules, Delete's idempotency and the replacement window, ListByTag's
partial-failure policy, the abandon policy and double-close guards,
resource handling on every path, and secret hygiene in the format strings.
Lint: `bin/golangci-lint` still panics on this toolchain (Go 1.27 against
a staticcheck built for 1.26), reproducible on untouched packages, so
`make lint` could not gate this step; `go vet` with every build tag is
clean. Tests: `go test -race -count=3 ./internal/provider/kvm2/` (168
top-level tests) and `go test -short ./...` pass; `make test` with envtest
runs next.
docs/plans-executions/2026-09-17-1515-kvm2-review-fixes.md
+531
View File
@@ -0,0 +1,531 @@
# Execution: fix the kvm2 code-review findings
Plan: [docs/plans/2026-09-17-1515-kvm2-review-fixes.md](../plans/2026-09-17-1515-kvm2-review-fixes.md)
- [x] Step 0 — Save plan into the repo
- [x] Step 1 — H1: a stalled write must not delay the teardown
- [x] Step 2 — H2: a source error must not reach go-libvirt's stream sender
- [x] Step 3 — H3: an unreadable address must not look like a missing one
- [x] Step 4 — H4: the base image cache must not be shared across provider entries
- [x] Step 5 — Validate H1-H4 in-cluster against beelink
- [x] Step 6 — Route klog into the structured logger
Remaining after those: the 48 mediums and lows in `review/REPORT.md`. Each gets its own
checklist line here when it is planned, so this list stays a record of decided work rather
than a wish.
## Step 0 — plan committed
Copied out of `~/.claude/plans/` and committed alone before touching code, as the repo's
plan rules require.
## Step 1 — H1: close the SSH client, and signal dead before killing
Two edits, both small, in the order the bug unfolds.
`sshSocket.Close` was `errors.Join(s.Conn.Close(), s.client.Close())`. Go evaluates those
arguments left to right, so the blocking close ran before the unblocking one. `s.Conn` is
the forwarded libvirt socket, and closing an SSH channel writes a close message under
`ch.writeMu`, which go-libvirt already holds while streaming a base image through the same
channel. Against a host that vanished without RST or FIN, that write blocks until the
kernel stops retransmitting — `tcp_retries2`, about 15 minutes. `s.client.Close()`, a bare
`net.Conn.Close()` on the TCP socket that takes no lock and unblocks the stalled writer
through netpoll, was never reached. It now closes the client only.
**Dropping the channel close rather than reordering it** was the judgment call. Reordering
works, but afterwards the channel close returns an error — `net.ErrClosed`, a
`*net.OpError`, the transport's sticky write error, or `io.EOF` once `mux.loop` has run
`ch.close()` — and `closeErr` is memoised, so a filter that missed a case would poison
every later `Close` and break `ssh_test.go:66,69`, which assert both closes return nil.
Nothing wants the polite channel-close message: `dialSSHSocket` builds the client solely
for this one channel, the `sshSocket` owns it exclusively, and `dialSSHSocket`'s own error
paths already tear down by closing the client.
`link.markDead` then compounded it by running `kill()` before `close(l.closed)`, and doing
both inside a `sync.Once`. Reordering alone would not have been enough: `sync.Once.Do`
blocks concurrent callers until the first body returns, so every other `markDead` — including
the one in `call`'s own ctx-cancel branch — queued behind the stalled kill. **Cancelling the
reconcile did not help.** The `once` field is gone; the mutex the struct already had now
guards the "close it exactly once" invariant, `reason` is set before the signal so a
released caller always reads a populated `deadReason()`, and `kill` runs outside the lock.
**A second route to the same pin, found while planning and fixed by the same edits.**
go-libvirt's `Disconnect` closes the `net.Conn` it was handed (`libvirt.go:331` →
`socket.go:164`) — and that conn *is* our `*sshSocket`. So `libvirtBackend.Close`'s
`Disconnect` goroutine entered `sshSocket.closeOnce` and hung there; the 5 s `closeBudget`
released `Close`, which called `markDead("closed")` → `kill()` → `sshSocket.Close()` →
blocked in the other goroutine's `sync.Once`. The budget did not protect it.
The precondition was verified in the pinned x/crypto v0.54.0 rather than assumed: the
flow-control park (`remoteWin.reserve`, `channel.go:268`) happens *outside* `writeMu`, and
only `writePacket` holds it across the TCP write. So a writer parked in SSH flow control
never blocks the close; a writer inside the TCP write always does. Since the channel window
is 2 MiB (`channelWindowSize = 64 * channelMaxPacket`), reproducing it by hand needs a
kernel send buffer that fills first — shrink `SO_SNDBUF` well under 2 MiB.
The older kvm provider does not have this bug: it closes `client` only
(`internal/provider/kvm/conn.go:429,456`) and never touches the forwarded channel. kvm2's
rewrite introduced it.
### Tests: planned four, shipped one
The first draft had three hermetic tests plus a `//go:build integration && linux`
stalled-write test with a TCP proxy and shrunken socket buffers. The user pushed back —
once the bug is fixed the connection closes properly, so what is left to test? — and they
were right about most of it. `TestDial_forwardsUnixSocket` already covers "Close tears the
connection down", and after the fix `Close` is a single call with no ordering to get wrong.
The stalled-write test was the worst of the four: ~150 lines of Linux-only,
buffer-autotuning-dependent scaffolding whose own false-green guard made it *skip* whenever
the kernel did not cooperate, to demonstrate a bug that reading x/crypto settles and that
the fix deletes the code path for.
What survived is `TestMarkDead_slowKillDoesNotBlockCallers`, because the `markDead` change
*adds* a guarantee that did not exist before and nothing else touches it —
`TestCall_killedByAnotherCallerFailsTransient` passes a nil kill, so it cannot see the
property. It is also the plausible regression: restoring the `sync.Once` looks like a
correctness improvement. The call's own `fn` signals when it is in flight, so the parked
caller is genuinely parked rather than refused by the dead check on the way in.
The step that actually validates a regression test is running it against the unfixed code.
`markDead` was reverted in place, the test run, and the fixed version restored from a copy:
```bash
cp internal/provider/kvm2/backend.go "$SCRATCH/backend.go.fixed"
# ... revert markDead to the sync.Once form and re-add the once field ...
go test -race -count=1 -run 'TestMarkDead_slowKillDoesNotBlockCallers' ./internal/provider/kvm2/
# --- FAIL: TestMarkDead_slowKillDoesNotBlockCallers (1.00s)
# backend_test.go:186: link not signalled dead while its kill is still running
cp "$SCRATCH/backend.go.fixed" internal/provider/kvm2/backend.go
```
It fails in a second with the right diagnostic rather than hanging the suite, because the
blocked kill is released by `t.Cleanup`.
```bash
go build ./... && go vet ./... && go test -race -count=1 ./internal/provider/kvm2/
```
### Worth noting
**An accepted risk, deliberately not covered by a test.** Nothing now stops someone
re-adding `s.Conn.Close()` to `sshSocket.Close` as tidiness; it would look harmless and
every test would still pass, because in a healthy test the channel close returns instantly.
The guard is the doc comment on `sshSocket`, which says outright that closing the channel
would queue behind a stalled transfer. That is in idiom for this package — `hosts.go:25-34`
and `create.go:151-153` carry the same kind of hard-won decision — but it is a comment, not
a test, and the failure it prevents is a 15-minute operator-wide stall.
**Left for later, on purpose.** `ReconciliationTimeout` in `SetupWithManager` does not exist
in this repo at all; adding a per-reconcile deadline would change the contract for every
provider, since `hosts.go` and `image.go` both cite "no caller deadline", so it needs its own
change and its own spec entry. `sshSocket.keepalive` still has no test (sweep finding S1.2).
**Docs.** `docs/kvm2-provider.md` said the adapter "kills the connection (socket and SSH
client)", which is no longer true and was the description that made the bug look correct.
It now states the order (mark dead, then kill) and why both steps are lock-free.
`docs/kvm2-provider-spec.md` entries 18 and 19 were checked and are not falsified.
**No CHANGELOG entry yet** — the repo's rule is to add one once the user confirms the change
works, so it lands with a later "validate and record" step, as in the kvm campaign.
## Step 2 — H2: a source error must not reach go-libvirt's stream sender
This one is a workaround, not a repair. The defect is upstream, and the entry says so in
three places (code comment, `docs/kvm2-provider.md` *Known gaps*, commit message) so nobody
later reads the wrapper as ours to delete.
`requestStream` in the pinned go-libvirt starts its stream sender on two **unbuffered**
channels and then, on the error path, returns without ever reading `outErr`. A non-EOF error
from the reader makes `SendStream` emit a StatusError packet and return, whereupon it parks on
`outErr <-` forever; libvirtd answers the StatusError with an error, so the main goroutine
parks on `abort <- true` forever. `StorageVolUpload` never returns. The 20-minute budget
rescues only the caller, and the socket kill does nothing at all here, because both ends are
blocked on channel sends rather than on I/O — which is what the comment at `backend.go:152`
got wrong. Every failed upload leaked two goroutines and 4 MiB against a 128Mi pod limit,
with the reconciler retrying on backoff.
Three changes.
`eofReader` sits at the go-libvirt seam in `backend.go`, not in `image.go`, so it also covers
the seed and marker uploads, and so it stays out of the file Step 4 rewrites. It converts any
source error to `io.EOF` — the one outcome that ends a stream the normal way — and keeps the
real error for `StorageVolUpload` to return afterwards. `countingReader` is untouched
underneath it and still sees the true error, so `uploadImage`'s existing length and digest
checks produce exactly the messages they did before.
The download now carries its own deadline, one minute inside the transfer budget.
`fetchImage` already bounded the whole fetch at 20 minutes, so the body's death and the
budget's socket kill were racing, and the socket usually won. With the body a minute ahead, a
stalled mirror ends through the reader and unwinds cleanly. This is the difference between a
slow mirror costing a Transient error and costing a permanent leak, and slow mirrors are the
realistic repeat offender.
`discardVolume` moved from `uploadImage` up into `fetchImage`'s upload-error branch, so the
dedicated connection is closed — and libvirtd has let go of the stream's fd — before the file
is unlinked. Moving it rather than adding a second `Close` is what keeps
`image_test.go`'s `closeCount() == 1` assertion true. `uploadImage` lost its `be` parameter
and its private logger in the process.
**What is left, deliberately.** A failure that originates in libvirtd mid-upload (ENOSPC, or
the volume deleted underneath) still leaks, because by then `SendStream` has already returned
and nothing we pass, cancel or close can reach it. Confirmed unfixable from this side. The
upstream fix is two characters and is quoted in *Known gaps* along with the `replace` escape
hatch, should it ever bite hard enough to be worth the maintenance.
**Unverified, ~85%.** That libvirtd answers a short-but-clean stream finish with OK rather
than an error. `virFDStreamWrite` enforces `length` only on writes past it and nothing
compares `offset` to `length` on the close path, which is also why `virsh vol-upload` of a
small file into a large volume works. If it answers with an error we land back in a deadlock
by a different door — but that is precisely today's behaviour, so the change cannot make
things worse. To settle it, on any box with a libvirtd (not beelink, which is read-only
here), against the pinned go-libvirt:
```sh
virsh -c qemu:///session pool-define-as probe dir --target ~/vmprobe
virsh -c qemu:///session pool-start probe
virsh -c qemu:///session vol-create-as probe short.raw 10M --format raw
# then a ~40-line main.go: StorageVolUpload(vol, r, 0, 10<<20, 0) where r
# yields 1 MiB then io.EOF; print the error, then runtime.NumGoroutine()
# and a goroutine profile after a two-second settle.
```
Expected: `err == nil` and the goroutine count back at baseline. The same program with a
reader that returns a non-EOF error should wedge against the *unpatched* seam, which also
confirms the diagnosis end to end.
The negative check that matters was run on the unit test instead — `eofReader.Read` was
changed back to returning the real error, and both new tests failed with
`io.Copy through eofReader = connection reset by peer, want nil`, then passed again once
restored.
```bash
go build ./... && go vet ./... && go test -race -count=1 ./internal/provider/kvm2/
```
### Worth noting
The `fetchImage` discard is still unfenced against a second provider entry that shares the
volume name — Step 4 (H4) is what closes that, by making the name per-entry. The deferred
atomic-publish work then lands exactly where this step put the discard.
**Deliberately not tested:** the abort protocol end to end. `libvirttest` reads a fixed
28-byte header per packet and implements no stream procedures, so pinning this hermetically
would mean a 150-250 line libvirt RPC server stub whose only assertion is about *upstream's*
goroutines. Same call as H1's: the test would cost more than it protects. The two tests that
did land pin the one property the seam actually needs.
## Step 3 — H3: an unreadable address must not look like a missing one
`address()` tried the guest agent then the host's ARP table and skipped any error that was
not a context error, returning `("", nil)` after both. An active domain with no IP maps to
`StateProvisioning`, the controller clears `status.IP` on that, and the health engine prunes
the proxy's counters and calls `ForgetProxy` in one tick once `EffectiveHost()` is empty.
The damaging trigger was not the guest. The libvirt connection is pooled per host, so when
another reconcile's cancel or budget overrun killed it, `call` returned
`"connection is closed"` — carrying no context sentinel, so the check did not recognise it —
and `Get` returned a *successful* instance in Provisioning with an empty IP. An unrelated
proxy's timeout dropped a healthy proxy out of rotation.
The fix rests on a discriminator the spec had already recorded without drawing the
conclusion: during a normal boot on beelink exactly one source fails. The agent errors
because it is not up yet, while ARP answers *without* error and with nothing, since the host
never sees the guest's traffic. So one source failing is "no address yet"; every source
failing, or a connection already dead, is `ErrTransient`.
The explicit `provider.ErrTransient` wrap on the joined error is load-bearing rather than
decoration. Without it `classify` picks the first libvirt code out of the join tree, and
`ErrInvalidArg`, `ErrAccessDenied` and `ErrNoSupport` are Permanent — an address read failing
would have latched a *running* proxy as Failed, a worse version of the harm being fixed.
`isDead(be)` is checked first in the error branch. It is not strictly needed, since a dead
link fails both sources and the count rule catches it anyway, but it fails on the first
source instead of the second, attributes the error better, and is the same predicate M15 will
want for `ListByTag`.
### Tests
`TestGet_addressErrorIsProvisioning` failed against the new rule, which is correct: it fails
*both* sources with `ErrAgentUnresponsive`, and code 86 is agent-specific — the ARP source
cannot produce it. Re-pointed as `TestGet_agentFailsAndARPEmptyIsProvisioning`, the beelink
boot it was really reaching for: agent errors, ARP returns no addresses and no error, result
is Provisioning. Nothing covered that before, and it is the case the new rule must not break.
`TestGet_everyAddressSourceFailingIsTransient` covers the other side, two subtests: both
sources erroring, and the pooled connection dying under the read. Against the unfixed code
both return a Provisioning instance instead of an error, which is exactly the bug:
```bash
# with isDead() and the every-source-failed arm removed from address():
go test -count=1 -run TestGet_everyAddressSourceFailingIsTransient ./internal/provider/kvm2/
# get_test.go:215: Get = &{... IP: State:Provisioning ...}, want no instance
```
`TestGet_arpFallbackWhenAgentFails` passes untouched — one source of two.
### Worth noting
**Half of H3 is deliberately still open, and it is the half that fires more often.** A guest
agent restarting inside a running VM still demotes it to Provisioning and clears its IP: the
agent errors, ARP answers empty, and that is indistinguishable from a VM still booting. The
discriminator is the *previous* IP, which the provider does not hold by design
(`TestGet_readsUIDAndCreatedAtFromBackend` pins "read back, not remembered") and the
controller already has. Follow-up finding, to be filed against
`internal/controller/proxy_controller.go:208-211`: on `StateProvisioning`, keep an existing
IP and `Provisioned` condition for a few polls rather than clearing on the first, or give the
health engine a grace period before pruning.
The one kvm2-local alternative was considered and rejected: the mark carries `createdAt`, so
`get` could treat "active, no address from any source, domain older than a provisioning
window" as an error. It uses state the provider genuinely has, but `createdAt` is creation
rather than boot, it fires on every genuinely stuck VM as a retry-forever error, and it
invents a provisioning deadline that belongs to the controller. A design change wearing a bug
fix's clothes.
**L6 not folded in.** A domain that vanishes between `DomainGetState` and the address read
now returns Transient rather than a fake Provisioning — better, and self-correcting on the
next poll. Returning NotFound one poll earlier is a two-line addition at the same spot, left
for when L6 comes up so this commit stays one idea.
## Step 4 — H4: the base image cache must not be shared across provider entries
`baseVolumeName` was `base-<key>.<format>` — byte-identical to what
`internal/provider/kvm/image.go` builds in the same pool, for URL images too (kvm takes
`hex(sum[:8])`, kvm2 takes `hex(sum[:])[:16]`, the same eight bytes), with a completely
separate lock table. The shipped config has a live `kvm-home` entry and a commented-out
`kvm2-home` entry on the same host, the same key `debian-13` and the same default pool, and
the comment invites the operator to enable it. Both integration tests use key `test-image`.
On beelink both live overlays back onto kvm's `base-debian-13.qcow2`.
Since `cachedImage` accepts any 64-hex marker when no `sha256` is configured and never
re-hashes the base, that meant: two entries pointing one key at different URLs silently share
a file; a concurrent cold fetch lets one entry delete the other's half-uploaded base and boot
a guest from the remains; and a `sha256` edit deletes the base in place under live overlays,
which libvirt permits without complaint.
The name now carries the provider entry: `base-<entry>-<key>.<format>`. `imageSource` gained
an `Entry` field and `resolveImage` takes the provider name, so the four production call
sites all follow from `imageSource.volumeName()`. That closes the cross-entry sharing
outright, kvm↔kvm2 and kvm2↔kvm2 alike, and it turns the existing per-`Provider` `imageLocks`
from a latent bug into a correct one — with names unique per entry, no two entries can
contend for a file, so a process-wide lock table is not needed.
**The entry name reaches a file name, so it now has to be a usable one.** Provider names are
validated for all types at once and only for being non-empty and unique
(`internal/provider/config.go`), so `New` checks its own against the existing `kvmIdentifier`
rule, exported as `provider.IsKVMIdentifier` rather than duplicated. A name with a slash
fails at startup instead of producing a volume nothing can address.
**One companion fix, because the naming alone does not stop it.** `cachedImage`'s
digest-mismatch branch deleted the marker and the base and refetched. It now returns
`ErrPermanent` naming both digests and telling the operator what to do. That branch was the
remaining path that unlinks a good file under running domains, and the contract already said
"add a new key instead" — the code was contradicting it. The no-marker and non-hex-marker
branches keep deleting, since there the base is genuine debris.
### What this does NOT fix
Stated plainly rather than left to look closed:
- **Editing `images.<key>.url` is still a silent no-op.** The cache is keyed by entry and
key, and nothing records the URL, so the new one is never fetched. Pre-existing, already
documented as "add a new key"; the eviction section now says it outright.
- **Cross-process fencing.** Two managers, or a manager plus a hand-run `virsh`, can still
meet on one name. E4 refuted the rollout and leader-handover triggers for the shipped
manifests, so this is narrow, and its fix is the deferred atomic publish — upload to a temp
name, verify, publish — which lands where H2 put the discard.
### Migration
Nothing to do. The `base-debian-13.qcow2` on beelink is kvm's, kvm keeps using it untouched,
and kvm2 downloads its own copy once per host when `kvm2-home` is enabled. No window in which
anything is replaced under a running guest. The pool needs room for both, which the
prerequisites now say.
### Tests
Mostly literals becoming helpers, plus a new `fx.baseName()` in `provider_test.go` for the
three fixtures that asserted `"base-debian-13.qcow2"` by hand. The log and span assertions in
`TestEnsureImage_downloadsAndWritesMarkerLast` keep asserting a full name rather than loosening
to a prefix, so they still pin that the key stays readable in operator-facing output.
One test changed behaviour rather than spelling: `TestEnsureImage_sha256ChangeRedownloads`
became `TestEnsureImage_sha256ChangeRefusesInsteadOfReplacing`, asserting Permanent, zero
server hits, zero deletes, and both volumes untouched. Two added:
`TestNew_rejectsUnsafeEntryName`, and a `TestVolumeNames` row for a second entry with the same
key.
Both negative checks were run:
```bash
# baseVolumeName reverted to base-<key>.<format>:
go test -count=1 -run TestVolumeNames ./internal/provider/kvm2/
# volume_test.go:135: got "base-debian-13.qcow2", want "base-kvm2-lab-debian-13.qcow2"
# the digest branch reverted to delete-and-refetch:
go test -count=1 -run TestEnsureImage_sha256ChangeRefuses ./internal/provider/kvm2/
# image_test.go:412: ensureImage() error = <nil>, want ErrPermanent telling the operator what to do
```
`go vet -tags=integration` was run too, since `kvm2_integration_test.go:484` builds the base
name and only compiles under that tag.
### Worth noting
**The integration test has still never run on beelink.** Step 11 is unchecked in
`docs/plans-executions/2026-09-16-1333-kvm2-provider.md`, every `base-*` on the host predates
kvm2, and both live domains carry the kvm v1 metadata namespace. After this change it is also
the first end-to-end exercise of the new name, and it will do a real cold download under
`base-kvm2-integration-test-image.qcow2` beside kvm's untouched `base-test-image.qcow2`.
**Docs touched beyond the obvious:** the eviction recipe named `base-debian-13.qcow2`, which
after this change is *kvm's* file — following it would have deleted the wrong provider's base.
It now greps for the prefix and deletes the marker first.
## Step 6 — Route klog into the structured logger
Not a review finding. Found while watching the Step 5 validation: some lines ignored the zap
encoder no matter what, so `kubectl logs | jq` could never work.
```
I0917 19:50:02.580097 1 leaderelection.go:258] "Attempting to acquire leader lease..." lock="..."
```
That is klog's own stderr writer, not ours. client-go's leader elector calls
`klog.FromContext(ctx)` (`leaderelection.go:257`), and controller-runtime hands the elector a
context with no logr in it — `mgr.Start` receives `ctrl.SetupSignalHandler()`, which carries no
logger (`pkg/manager/internal.go:461-473`). `FromContext` falls back to `Background()`, and with
no logger registered that is klog's text sink. Nothing in this repo imported or configured klog;
it was an indirect dependency that around 60 linked packages use, including
`client-go/tools/{leaderelection,cache,record}` and `client-go/transport`. controller-runtime
does not route it either — its only `klog.SetLogger` call is in its own test suite.
**`klog.SetLogger(logger)`, deliberately not `SetLoggerWithOptions(logger, ContextualLogger(true))`.**
The contextual form is what klog's docs recommend, because it skips a klogr round trip. It also
disables klog's verbosity gate ("Such a logger cannot rely on verbosity checking in klog",
`contextual.go:83-86`), handing the decision to zap — whose level under `Development: true` is
Debug, which enables V(1). client-go's V(1) chatter, suppressed today by klog's own `-v=0`,
would have started appearing. Plain `SetLogger` keeps klog's check in front and still redirects
what it does emit ("all log lines will be suppressed from the regular output, and redirected to
the logr implementation", `contextual.go:47-48`). Same fix to the format, no new volume — which
matters, because the complaint that started this was volume.
`k8s.io/klog/v2` moves from indirect to direct in `go.mod`. It was already in the module graph;
nothing new is downloaded.
**The other bypass.** The discovery API's `http.Server` set no `ErrorLog`, so `net/http`'s own
faults — a failed `Accept`, a panic escaping a handler — went to the stdlib logger on stderr.
It now drains into logr through `logr.ToSlogHandler`, the pattern
`internal/provider/gcp/wirelog.go:31-36` already uses. This server is plain HTTP with no TLS, so
`ErrorLog` carries real faults rather than handshake noise, and Error level is right.
**Deployment.** `--zap-encoder=json` and `--zap-time-encoding=rfc3339` on the manager.
Verified the flag beats the in-code default rather than assuming it: `addDefaults` fills
`NewEncoder` only `if o.NewEncoder == nil` (`zap.go:176-191`) and `BindFlags` sets it from the
flag. The time encoding is not optional — the JSON encoder defaults to epoch seconds.
Verbosity was left at debug on purpose; `--zap-log-level=info` drops the per-RPC lines.
`opts := zap.Options{Development: true}` stays. Flipping it would change the format for every
deployment of this operator, and the flags get the same result here without that reach.
### Tests
None. Three changes, all wiring: two struct fields and a manifest key. `cmd/main.go` has no test
coverage, and pinning `klog.SetLogger` means mutating process-global state that leaks into every
other test in the package. Verified by observation instead, which for this change is stronger
than a unit test would be.
### Verification
```bash
kubectl -n egress-proxies-operator-system logs deploy/... | grep -i leader
# {"level":"info","ts":"2026-09-17T20:10:00Z","msg":"Attempting to acquire leader lease...",
# "lock":"egress-proxies-operator-system/b47711d1.example.com"}
kubectl -n egress-proxies-operator-system logs deploy/... | jq -e -s 'length > 0'
# YES - every line is valid JSON
```
The `jq -s` check is the one that counts: it fails if *any* line is not JSON, which is exactly
the property being fixed. All four existing proxies stayed Ready across the rollout and all
three ProviderHealth entries stayed Ready.
### Worth noting
**The JSON encoder renders durations as float seconds**, so `"duration":"566.12µs"` became
`"duration":0.000336664`. Machine-friendlier, worse to read. Either format in the query —
`(.duration*1000000|round)` for µs — or pass an encoder config option in code, which
controller-runtime exposes (`zap.Options.EncoderConfigOptions`, `zap.go:140-144`):
```go
opts := zap.Options{
Development: true,
EncoderConfigOptions: []zap.EncoderConfigOption{
func(c *zapcore.EncoderConfig) { c.EncodeDuration = zapcore.StringDurationEncoder },
},
}
```
Left out of this step as scope the plan did not cover.
## Step 5 — Validation against beelink
First contact between the kvm2 provider and real libvirt. Everything before this ran against
an in-memory fake of the twenty libvirt calls the provider uses.
Run in the existing `kind-proxy-operator-demo` cluster rather than a fresh one, deliberately.
That cluster already owned two live `kvm-home` VMs on beelink, and orphan GC deletes any
instance whose UID is missing from *its own* cluster's live set after `gc-min-age`. A second
operator carrying the `kvm-home` entry would have destroyed both within ten minutes. Using the
existing cluster made its live set the thing that protected them — the isolation argument
pointed the opposite way from instinct.
Baseline recorded first (4 Ready proxies, 3 ProviderHealth entries, beelink's domains, volumes
and pool space), so "did we break something" stayed answerable.
The image tag had to change from the running `kvm-dev`. `make deploy` sets the image through
kustomize, so an identical tag leaves the Deployment spec untouched, no rollout happens, the
new process never starts, and the providers config — which is read once per process — is never
re-read. The ConfigMap carries no generator hash either, so nothing else would have forced it.
Results, in the order they were checked:
| Check | Outcome |
|---|---|
| `kvm2-home` ProviderHealth | `Ready 1/1`, `Reachable`, 189 ms — hostname, libvirt 11.6.0, pool running, 247.8 GiB free |
| First Create, cold cache | `base image cached`, 433,651,712 bytes in 8.3 s, digest verified |
| Volume naming (H4) | `base-kvm2-home-debian-13.qcow2` created **beside** kvm's untouched `base-debian-13.qcow2` |
| Second Proxy | no `fetching` line at all — cache hit under the new name |
| State reporting (H3) | `Running` held steady across every poll, no flapping to Provisioning |
| Squid | `Healthy=True` on a real CONNECT probe |
| Delete | domain and both per-VM volumes gone; base image survived |
| Existing infrastructure | all four pre-existing proxies stayed Ready across two rollouts |
The health check alone is worth calling out as a gate: it dials SSH, verifies the host key,
completes the libvirt handshake, runs four RPCs and tears down a dedicated connection — the
exact path H1 changed — without creating anything. Putting the stop gate there meant the
riskiest unknown was settled before a single byte was written to the hypervisor.
### What this run did NOT prove
**H2's error paths.** The unverified claim — that libvirtd answers a short stream finish with
OK rather than an error — only arises when a body *fails* part way through, and a healthy
433 MB download never finishes short. The 19-minute HTTP deadline never fired either. What was
proved is narrower: the wrapper does not break normal uploads. Settling the rest still needs
the scratch-pool probe in the Step 2 entry, and it would be easy to mistake this green run for
more than it is.
**The `kvm2` integration test still has not run.** `make test-integration-kvm2` remains
unexecuted; this was a cluster-level validation, not that suite.
### Worth noting
`config/manager/providers_config.yaml` now ships with the `kvm2-home` entry **enabled**, since
the operator is running it for real. `config/manager/kustomization.yaml` was left at its
committed tag: `make deploy` rewrites it to whatever local image was last deployed, which is
churn rather than work.
Two host-side leftovers were *not* touched, both pre-existing and unrelated to kvm2: five
orphan `proxy-*` volumes from earlier kvm runs, and the `base-test-image.qcow2` pair from the
kvm integration test. Every write to beelink in this step came from the operator itself; no
`virsh` write was issued by hand.
docs/plans/2026-09-11-1526-provider-health-checks.md
+354
View File
@@ -0,0 +1,354 @@
# Plan: Provider health checks — see whether the cluster can reach each provider
**Created:** 2026-09-11 15:26 UTC
## Context
Today the operator can't say whether a provider's backend is reachable. The `Provider`
interface ([provider.go:89-109](internal/provider/provider.go#L89-L109)) only has
Create/Get/Delete/ListByTag, and `/healthz` and `/readyz` are bare `healthz.Ping`. A
broken KVM path only shows up later: as a Proxy stuck in Provisioning, or as an orphan-GC
sweep logging an error. Several failures stay invisible until a Proxy needs that host:
- a missing `kvm-ssh` Secret;
- a rotated host key;
- a Tailscale route that is down;
- a storage pool that is inactive or full.
The goal is a periodic sanity check of every configured provider from inside the cluster,
with results you can see through `kubectl` and Prometheus.
Decisions made by the user during planning:
1. **Surface:** a new operator-owned, read-only, cluster-scoped **`ProviderHealth`** CRD,
with one object per configured provider, a `Ready` condition and per-target detail.
Also Prometheus gauges, and a log line whenever a target goes up or down. No
EventRecorder, matching the earlier decision.
2. **Scope:** real checks for all three providers.
- **kvm:** checked per host.
- **kubernetes:** confirms the API server and RBAC work.
- **gcp:** confirms credentials and the API work.
3. **KVM depth:** reachability plus the storage pool.
- Over the existing pooled SSH+libvirt connection, call the libvirt version and
hostname RPCs.
- Confirm the configured pool is running and report its free space.
- No bridge check.
Not affected: the Proxy reconcile behaviour, and the manager's `/healthz` and `/readyz`.
A dead hypervisor must never make the manager unready.
## Design
### Contract: `internal/provider/check.go`
```go
type TargetStatus struct {
Name, Address string
Ready bool
Reason, Message string
Latency time.Duration
Details map[string]string
}
// Check never errors; each target carries its own verdict. Must return promptly once ctx ends.
type Checker interface{ Check(ctx context.Context) []TargetStatus }
```
- **Shared reasons:** `Reachable`, `Timeout`, `Unreachable`, `Unauthorized`, `Forbidden`,
`APIError`.
- **Compile-time checks:** each provider's `check.go` has `var _ provider.Checker = (*Provider)(nil)`.
- **Bypass the wrappers.** `cmd/main.go:197-201` wraps every provider in
`WithTracing(WithMetrics(...))`, so the Checkers are collected from the **raw** providers
map, before that loop. As a result, checks don't inflate `provider_requests_total`.
### The checks
**kvm** (`internal/provider/kvm/check.go`)
- **Concurrency:** one goroutine per `cfg.Hosts` entry (`wg.Go`), writing to `out[i]` so
results stay in config order.
- **Inside `p.hosts.call`:** `ConnectGetLibVersion`, `ConnectGetHostname`,
`StoragePoolLookupByName(host.Pool)`, then `StoragePoolGetInfo`.
- **Rules for `fn`:**
- It returns only raw RPC errors. `call` drops the shared connection on any other error
([conn.go:160](internal/provider/kvm/conn.go#L160)), so "pool not running" is decided
after `call` returns, never returned from `fn`.
- It fills a captured result struct only after all four calls succeed. That struct is
read only when `call` returned nil, which avoids a race with an abandoned goroutine.
- **Details:** `hostname`, `libvirtVersion` (decoded as major.minor.patch), `pool`,
`poolState`, `poolCapacity` and `poolAvailable` (in GiB).
- **New RPCs:** add the three to `libvirtAPI` ([conn.go:29-47](internal/provider/kvm/conn.go#L29-L47)),
`tracingAPI` (`trace.go`) and `fakeAPI` (`fake_test.go`). `fakeAPI` gets a hostname, a
fixed version, and a per-pool state map that defaults to Running.
- **Telling failure stages apart without matching strings:** add
`type dialError struct{ stage dialStage; err error }` in `conn.go`, with `Error()`
returning the inner message unchanged and `Unwrap()`. It wraps at four places:
- `lazySSH.dial`, when loading the key material fails (`ssh-material`)
- `dialSSH`, at the dial step (`tcp`)
- `dialSSH`, at the handshake (`ssh-handshake`)
- `connectLibvirt` (`libvirt-connect`)
Because `Unwrap` is kept, `classify` and `isClassified` still see `ErrPermanent` and
`fs.ErrNotExist`, so reconcile behaviour doesn't change.
- **`checkReason(err)`, tested in order:**
| Condition | Reason |
|---|---|
| stage ssh-material | `SSHMaterialUnavailable` |
| stage tcp | `Unreachable` |
| stage ssh-handshake | `SSHHandshakeFailed` |
| stage libvirt-connect | `LibvirtConnectFailed` |
| ctx deadline or cancel | `Timeout` |
| `ErrNoStoragePool` | `PoolNotFound` |
| any other libvirt error | `LibvirtError` |
| anything else | `Unreachable` |
| success, pool not running | `PoolNotActive` |
| success | `Reachable` |
- **Accepted behaviour:**
- A check that hits its timeout drops the shared connection. That already happens for
any cancelled reconcile context, and it only fires when the host isn't answering.
- The leader now keeps one SSH connection per host open from startup, with keepalives
every 15s. Before, the connection only opened with the first KVM reconcile.
- No forced periodic redial. `lazySSH` caches the key material forever, so a redial
would not pick up a rotated Secret anyway; that gap goes into the docs under
"Known gaps".
**kubernetes** (`internal/provider/kubernetes/check.go`)
- **Call:** `List` pods with the managed-label selector and `client.Limit(1)`,
cluster-wide. It uses the same uncached client, verb and RBAC as `ListByTag` and GC.
There is no namespace setting in the kubernetes config.
- **Target:** one target named `apiserver`, with the address from a new `apiHost` field
set in `NewWithTransportWrapper`.
- **Reasons:**
- `Timeout`: the context ended, or the error is `IsTimeout`/`IsServerTimeout`.
- `Unauthorized`, `Forbidden`.
- `APIError`: any other API status.
- `Unreachable`: anything else.
**gcp** (`internal/provider/gcp/check.go`, plus a seam change in `gcp.go`)
- **Why a new seam method:** the existing `AggregatedList` seam reads every page
([gcp.go:63-78](internal/provider/gcp/gcp.go#L63-L78)), and MaxResults is only the page
size. So add `Probe(ctx, *computepb.AggregatedListInstancesRequest) error`, which makes
one `it.Next()` call and treats `iterator.Done` as nil.
- **Request:** `Project`, `MaxResults: 1`, `ReturnPartialSuccess: true`, and the same
managed-label filter `ListByTag` uses.
- **Target:** one per project, `Name: cfg.Project`, `Address: compute.googleapis.com`.
GCP config has no zones, and reading them from Proxy placements would tie the check to
the Proxy cache.
- **Reasons:**
- `Timeout`: the context ended.
- For `*googleapi.Error`: 401 is `Unauthorized`, 403 is `Forbidden` (this also covers a
disabled API), anything else is `APIError`.
- `Unreachable`: anything else, including a failure to get a token.
### The `ProviderHealth` CRD: `api/v1alpha1/providerhealth_types.go`
- **Written by hand**, since kubebuilder isn't installed; `PROJECT` is edited by hand too.
- **Scope and markers:** cluster-scoped, `shortName=ph`, status subresource, no spec.
- **Scheme registration:** add the type to the existing `AddKnownTypes` call in
[proxy_types.go:362](api/v1alpha1/proxy_types.go#L362), not a second `init()`.
- **Status:**
- `type`
- `conditions` (`Ready`; `ConditionReady` constant)
- `targets[]`, as a list-map keyed by `name`, each with: `name`, `address`, `ready`,
`reason`, `message` (at most 1024 characters), `latencyMillis`,
`lastTransitionTime`, `details`
- `targetsReady` (for example `"1/2"`, used by a printer column)
- `lastCheckTime`
- **Printer columns:** Type, Ready, Targets, Last Check, Age.
- **Viewer role:** `config/rbac/providerhealth_viewer_role.yaml`. No editor role, since the
operator owns the object.
### Monitor: `internal/providerhealth`
- **Fields:** `Monitor{Client, Reader, Targets []Target{Name, Type, Checker}, Interval, Timeout}`.
- **Scheduling:**
- `NeedLeaderElection() true`.
- Checks once immediately, then on a ticker. Copy the ticker loop from
[gc.go:54-80](internal/gc/gc.go#L54-L80).
- **Each cycle:**
- Runs every provider concurrently under `context.WithTimeout(Timeout)`, inside one
`tracing.Start` span.
- Stores the snapshot under a mutex; `Snapshot()` returns a deep copy.
- Logs changes. Going down logs `WARNING: provider target not ready` with provider,
target, address, reason and message. Coming back logs `Provider target ready`. A
target that is ready on first sight logs at V(1) only.
- **Writes:**
- Get through `mgr.GetAPIReader()`, not the cache. That avoids a cluster-wide informer
and the watch permission it would need, whatever `--proxy-namespace` is set to.
- Create the object if it is missing, labelled
`app.kubernetes.io/managed-by=egress-proxies-operator`.
- Then `Status().Patch(MergeFrom(base))` when the status changed, following
[status.go:114-121](internal/controller/status.go#L114-L121).
- This means one write per provider per interval, because `lastCheckTime` moves each
time. That's acceptable given the 10s minimum interval.
- **`buildStatus(prev, result, now)` is a pure function:**
- A target keeps its `lastTransitionTime` unless `ready` flipped.
- The `Ready` condition is set through `apimeta.SetStatusCondition`:
- `True/AllTargetsReady` when every target is ready
- `False/TargetsNotReady` otherwise, with the failing targets in the message
- `Unknown/ChecksNotImplemented` when the provider has no Checker
- `Unknown/NoTargets` when there are zero targets
- **Cleanup, once at `Start`:** delete labelled objects whose name isn't in the config.
The config is loaded once per process, and unlabelled objects are never touched.
- **Interval 0 (disabled):** run the cleanup with nothing kept, then return nil, so no
object is left showing a stale `Ready=True`.
- **Invalid provider names:** provider names aren't validated as DNS-1123 names. For an
invalid name, warn once and skip the object writes; metrics are still emitted.
- **`ValidateSchedule(interval, timeout)`** is called in `main.go` before the manager
starts. Interval must be 0 or at least 10s; timeout must be in (0, interval].
- **RBAC markers on `Monitor`:**
- `providerhealths`: get, list, create, delete
- `providerhealths/status`: get, patch
- `make manifests` regenerates `config/rbac/role.yaml`.
### Metrics and wiring
- **New function:** `metrics.RegisterProviderTargets(reg, read func() []ProviderTarget) error`.
`Register`'s signature is left alone.
- **Gauges:** a scrape-time collector, in the style of `constCollector`, reads the Monitor
snapshot and emits three gauges labelled `{provider,target}`:
- `proxy_operator_provider_target_up`
- `proxy_operator_provider_check_duration_seconds`
- `proxy_operator_provider_check_timestamp_seconds`
- **Conversion closure:** the one that turns the Monitor snapshot into gauges lives in
`main.go`, next to `proxyPhaseCounts`. So `providerhealth` doesn't import `metrics`.
- **No pool-bytes gauge:** that value is in status, and node-exporter is the right source
for disk alerts.
- **`cmd/main.go`:**
- New flags `--provider-check-interval` (1m) and `--provider-check-timeout` (15s).
- Build the `Targets` from `cfg.Providers` in config order, with
`c, _ := raw[pc.Name].(provider.Checker)`.
- `mgr.Add(&providerhealth.Monitor{Client: mgr.GetClient(), Reader: mgr.GetAPIReader(), ...})`,
then register the collector.
## Steps (one commit each, execution-log entry per step)
**Step 0 — Save the plan.**
- Create branch `feat/provider-health` off `main`.
- Copy this plan to `docs/plans/2026-09-11-1526-provider-health-checks.md` and commit it
alone.
- The uncommitted `.claude/settings.json` change stays out of every commit.
**Step 1 — ProviderHealth API and CRD.**
- `api/v1alpha1/providerhealth_types.go`
- `zz_generated.deepcopy.go`, from `make generate`
- `config/crd/bases/crawl.example.com_providerhealths.yaml`, from `make manifests`
- the CRD and RBAC `kustomization.yaml` files, plus the viewer role
- `PROJECT`
**Step 2 — Contract plus the kubernetes and GCP checks.**
- `internal/provider/check.go`
- `internal/provider/kubernetes/{kubernetes.go,check.go,check_test.go}`
- `internal/provider/gcp/{gcp.go,check.go,check_test.go,gcp_test.go}`
**Step 3 — KVM check.**
- `internal/provider/kvm/{conn.go,trace.go,check.go,check_test.go,fake_test.go,conn_test.go,trace_test.go,kvm_integration_test.go}`
**Step 4 — Monitor.**
- `internal/providerhealth/{monitor.go,status.go,monitor_test.go,status_test.go,envtest_test.go}`
- `config/rbac/role.yaml`, from `make manifests`
**Step 5 — Metrics and wiring.**
- `internal/metrics/{metrics.go,metrics_test.go}`
- `cmd/main.go`
- `config/manager/manager.yaml`: a comment for the new flags
- `test/e2e/e2e_test.go`
**Step 6 — Docs.**
- `docs/architecture.md`: a component row plus a provider-health section
- `docs/kvm-provider.md`: the reason table, the shared-connection behaviour, and the
Secret-rotation gap under "Known gaps"
- `docs/testing.md`: the new tests and the commands
- `CLAUDE.md`: its Architecture and Gotchas sections still say "empty scaffold"; point
them at `docs/architecture.md`
**Step 7 — Validate, record, push, open the PR.**
- Run `go vet ./...`, `go test -short ./...`, `make test` and `make test-e2e`.
- Hand the user the KVM commands below, and record their results in the execution log.
- Add a CHANGELOG entry once the user confirms.
- Push, using the token-file credential helper from memory.
- Open the PR against `main` through the Gitea API (`tea` isn't installed).
- Don't merge.
## Tests
**Provider checks** (table-driven, `Test<Func>_<scenario>`):
- kubernetes:
- reachable
- list options: `Limit==1`, the label selector, no namespace
- failure reasons: forbidden, unauthorized, timeout, cancelled, internal, and a
network `*url.Error`
- gcp:
- reachable, asserting the request fields
- failure reasons: 401, 403, 500, deadline, network
- `TestRealInstances_Probe_fetchesOnePage` against `httptest`: exactly one request with
`maxResults=1`, even though the page has a `nextPageToken`
- kvm (using `hostFakes`/`newTestProvider` from `kvm_test.go`):
- reachable, with details filled in
- per-host verdicts, order preserved
- pool inactive, pool missing
- hosts checked concurrently (a sequential check would deadlock)
- a pool verdict keeps the connection (the dial count stays 1)
- a `checkReason` table
- `TestSSHAuth_dial_stages`, using the in-process SSH server in `conn_test.go`
- three new rows in the tracing table
**Monitor:**
- Setup: the fake client is built with a status subresource and a RESTMapper that
registers ProviderHealth as cluster-scoped.
- Tests:
- `buildStatus` table
- objects are created with the label, and existing ones are patched
- the timeout bounds a checker that blocks
- a nil checker gives Unknown
- only transitions are logged (`funcr` logger)
- cleanup deletes only labelled objects that aren't configured
- the disabled path cleans up and returns
- an invalid name skips the writes
- `ValidateSchedule` table
- An envtest round-trip, skipped under `-short`, which proves the generated schema is
accepted by a real apiserver.
**Metrics:** `TestRegisterProviderTargets_scrape`, using a fresh registry and
`GatherAndCompare`; an empty snapshot produces no series.
**e2e (kind):** `providerhealth kubernetes` reaches `Ready=True`, which proves the
generated RBAC works end to end.
## Verification
**Automated (Claude runs):** `go vet ./...`, `go test -short ./...`, `make test` (envtest
with `-race`) and `make test-e2e`.
**Against the real hypervisor (handed to the user, not run by Claude)**, with beelink as
the host:
```bash
KVM_TEST_HOST=beelink.tail8c69c.ts.net:22 KVM_TEST_USER=novakj KVM_TEST_SSH_KEY=~/.ssh/id_ed25519 \
KVM_TEST_KNOWN_HOSTS=~/.ssh/known_hosts KVM_TEST_BRIDGE=<bridge> KVM_TEST_IMAGE_URL=<image-url> \
go test -tags=integration -count=1 -v -run TestKVMCheck ./internal/provider/kvm/
make install && go run ./cmd/main.go --providers-config <scratch copy of kvm-home config, local key paths> \
--metrics-bind-address :8080 --metrics-secure=false --provider-check-interval=15s
kubectl get ph -w
kubectl get ph kvm-home -o yaml
curl -s localhost:8080/metrics | grep -E 'proxy_operator_provider_(target_up|check_)'
```
Each negative case is a one-line edit to the scratch config; nothing changes on the host.
| Edit | Expected reason |
|---|---|
| `pool: nope` | `PoolNotFound` |
| `address: …:2` | `Unreachable` |
| `knownHostsFile: /dev/null` or `user: nobody` | `SSHHandshakeFailed` |
| `sshKeyFile: /nonexistent` | `SSHMaterialUnavailable` |
| `socket: /nonexistent` | `LibvirtConnectFailed` |
## Out of scope
- Gating Proxy creates on a host that is known to be down.
- Tying `/readyz` to provider health.
- Bridge checks.
- SelfSubjectAccessReview checks of create/delete rights.
- Hot-reloading the providers config.
- Picking up a rotated `kvm-ssh` Secret without a restart. This is an existing gap; it
gets documented.
docs/plans/2026-09-16-0841-provider-blueprint.md
+119
View File
@@ -0,0 +1,119 @@
# Plan: Provider blueprint (docs/providers.md)
**Created:** 2026-09-16 08:41
## Context
The provider contract is spread across `docs/prompts/__initial-prompt.md` §5,
the comments in `internal/provider/{provider,errors,check}.go`, `docs/architecture.md`
§5 and §11, `docs/kvm-provider.md`, and plan Step 2/3/8. Many rules exist only in
code: providerID must be self-contained, UID/CreatedAt must be read back, Create
must survive a crash partway through, calls must stop when ctx is cancelled,
decorators are wired in `cmd/main.go`, and so on. With "~5 providers soon", the
next provider author needs one blueprint. It should say what a provider MUST,
SHOULD or MAY do, show which features each existing provider (kubernetes, gcp,
kvm) actually implements (with file:line evidence), give a checklist for adding
one, and list the gaps.
Decisions (user): store in `docs/providers.md`; scope = current state plus a
separate "Gaps and proposed features" section (nothing implemented); wait for
`feat/provider-health` to merge, then branch off `main`.
## Step 0 — Save the plan; wait for the merge
- Right after ExitPlanMode, copy this plan to `docs/plans/2026-09-16-0841-provider-blueprint.md`
and commit it on its own. The current branch is `feat/provider-health`, so the
plan reaches `main` with that merge. CLAUDE.md requires the plan commit to
come before any branching.
- Stop and wait for the user to confirm that `feat/provider-health` is merged.
Then `git switch main && git pull`, and `git switch -c feat/provider-blueprint`.
## Step 1 — Collect evidence (read-only, in parallel)
Run 3 Explore agents at the same time, one each for `internal/provider/kubernetes`,
`internal/provider/gcp` and `internal/provider/kvm`, including their tests. Each
answers the **same questionnaire**, citing `file:line` for every answer and writing
"not implemented" explicitly when that's the case:
1. **Identity**: providerID format and parsing; what it carries (zone, host, namespace).
2. **Create**: how it stays idempotent (name lookup, 409, DomainLookupByName);
whether it returns before the VM is up or waits; what happens when an earlier
attempt crashed partway; which Placement fields are used or ignored; how
cloud-init is delivered; how the port is used; how GC labels/UID are attached.
3. **Get**: native state → InstanceState mapping; IP discovery rules; UID and
CreatedAt readback; how NotFound is detected.
4. **Delete**: idempotency; what gets removed (disks, volumes, seeds); returns
before or after the resource is gone; unknown/malformed providerID behaviour.
5. **ListByTag**: scope (namespaces, zones, hosts); partial-failure behaviour; how
managed resources are recognised.
6. **Error classification**: table of backend error → NotFound/Quota/Transient/
Permanent; whether quota is detected at all.
7. **Health check** (`Checker`): targets, provider-specific Reasons, Details keys.
8. **Config**: YAML section, defaults, validation (`internal/provider/config*.go`),
credentials/Secrets, which settings are read once per process.
9. **Connections and cancellation**: pooling, reconnects, timeouts, ctx honouring.
10. **Observability**: log keys and verbosity levels, spans (`kvm/trace.go`, gcp
`wirelog.go`), anything beyond the shared metrics/tracing decorators.
11. **Testing**: fakes, envtest/kind, `//go:build integration` tests, what is
not covered.
12. **Known gaps**: TODOs, stale comments, documented limitations.
Meanwhile, read the shared layer myself: `internal/provider/{provider,errors,check,
name,config,metrics,tracing}.go`, `internal/provider/registry/registry.go`,
`cmd/main.go` (registry wiring, decorator order, Checker collection), and the
places that consume the contract, so every MUST rule traces to a caller that depends on it:
`internal/controller` (error-class branching, ErrNotFound → replace/adopt),
`internal/gc` (UID, CreatedAt age skip, ListByTag errors),
`internal/providerhealth`. Then cross-check against the existing docs listed in Context.
## Step 2 — Write `docs/providers.md`
Sections:
1. **What a provider is**: a few sentences plus links to architecture.md §5/§11.
2. **The contract**: per method (Create/Get/Delete/ListByTag), MUST/SHOULD/MAY
rules, each followed by *why*: the reconciler or GC behaviour it protects,
linked to code.
3. **Error taxonomy**: what each class makes the reconciler do; how to map backend
errors; "unclassified = transient" pitfall.
4. **Identity and GC labels**: providerID guidance, `LabelManaged`/`LabelUID`,
CreatedAt, deterministic `NameFromUID`.
5. **Optional interfaces**: `Checker` (shared reasons, never fails as a whole,
collected before decorators).
6. **Feature matrix**: one row per questionnaire item, columns kubernetes/gcp/kvm,
cells `yes`/`partial`/`no` plus a short note. Put the file:line evidence in a
compact per-provider notes subsection under the table, not in the cells.
7. **Adding a provider: checklist**: package layout, config section and validation,
registry constructor, `cmd/main.go` wiring, Checker, error mapping with tests,
`name`/label tests, integration test behind a build tag, provider doc page,
architecture.md rows, demo script (optional).
8. **Gaps and proposed features**: current-state gaps per provider (e.g. no kvm
quota detection, orphan volumes invisible to GC, Secret rotation needs a
restart, ListByTag partial-failure inconsistency, stale `mock` references in
`provider.go` comments), plus proposals that are clearly marked as not
implemented. Only list them; don't design them.
Also: link it from `docs/architecture.md` (Components table → provider contract row)
and from the CLAUDE.md "Architecture" section. Leave per-provider docs where they are.
Follow the plan-execution convention: create `docs/plans-executions/2026-09-16-0841-provider-blueprint.md`
with the status checklist and step entries, and commit it with each step.
## Step 3 — Verify, then open the MR
- A verification agent (Explore) re-checks each matrix cell and each MUST rule
against the cited file:line and reports any mismatch. Fix what it finds.
- Check that every relative link in the new and edited docs resolves (grep the
`](` targets and `ls` each one).
- `go build ./... && go test -short ./...` as a sanity check (docs-only change,
expected no-op).
- Push `feat/provider-blueprint` and open the MR against `main` via the Gitea API,
following the project's push/MR memory note. After the user confirms, add a CHANGELOG.md entry.
## Critical files
- New: `docs/providers.md`, `docs/plans/2026-09-16-0841-provider-blueprint.md`,
`docs/plans-executions/2026-09-16-0841-provider-blueprint.md`
- Edited: `docs/architecture.md` (link), `CLAUDE.md` (link)
- Read-only sources: `internal/provider/**`, `cmd/main.go`, `internal/controller`,
`internal/gc`, `internal/providerhealth`, `docs/prompts/__initial-prompt.md`,
`docs/kvm-provider.md`, `docs/gcp-vm-validation.md`
docs/plans/2026-09-16-1108-provider-docs-split.md
+99
View File
@@ -0,0 +1,99 @@
# Plan: Split the provider blueprint into contract, appendix, template and index
**Created:** 2026-09-16 11:08
## Context
`docs/providers.md` is one 731-line blueprint that mixes the generic provider contract with how kubernetes, gcp and kvm happen to meet it, plus gaps and proposals. The user wants to hand an agent a self-contained set of rules for building a new provider (Proxmox is next), so the generic part must stand alone, state explicitly where providers are allowed to differ (a pod is ready at once, a VM has a provisioning window, kvm blocks in Create, gcp does not), and leave implementation detail to a separate appendix. A third file, a fill-in spec template, lets the user write the new provider's specifics in the same vocabulary. `docs/providers.md` becomes a short index so every existing link keeps working.
Decisions already made by the user: two new files plus a template; `providers.md` becomes an index; the contract is fully abstract (never names an existing provider); work continues on the checked-out branch `feat/provider-blueprint`.
Conventions for all four files: code references are `path` plus function or symbol name, never line numbers; rule keywords MUST / SHOULD / MAY as today, and every MUST names the caller and what breaks; no em-dashes in prose. The contract cites only shared-layer files (`internal/provider/*.go`, `internal/provider/registry`, `internal/controller`, `internal/gc`, `internal/providerhealth`, `internal/tracing`, `cmd/main.go`, `config/manager`, `api/v1alpha1`) and never `internal/provider/{kubernetes,gcp,kvm}`.
## Facts verified against the code that the new contract must carry
- Callers put no deadline on the context. Only `Monitor.checkOnce` (`internal/providerhealth/monitor.go`) uses `WithTimeout`; the reconcile and GC contexts are cancel-only. `Sweeper.sweep` (`internal/gc/gc.go`) loops providers sequentially, so one hung `ListByTag` stalls orphan GC for every provider.
- The reconciler's Stopped/Terminated branch in `reconcileManaged` (`internal/controller/proxy_controller.go`) deletes and requeues without clearing `ProviderID`, so a backend that reports a freshly created, not-yet-started instance as Stopped gets it deleted right after creation. kvm avoids this only by blocking `Create` through start.
- `Sweeper.sweep` treats a zero `CreatedAt` as collectable now, skips `UID == ""`, and skips anything younger than `MinAge` (10m).
- `cmd/main.go` type-asserts `provider.Checker` on the unwrapped provider before `WithTracing(WithMetrics(...))`; `registry.Build` fails the whole manager on a constructor error, so constructors must not dial.
- Manager runtime: non-root, read-only rootfs, `/tmp` emptyDir 64Mi, 128Mi limit, distroless static image with CA roots (`config/manager/manager.yaml`, `Dockerfile`).
- Corrections to earlier notes, for the appendix: kvm Checker `Details` also carry `user` and `pool`; the kubernetes provider has no seam interface (it injects `client.Client` via `newWithClient` and tests use the fake client); kvm `classify` has an `isClassified` pass-through for errors already wrapped with a class.
## Files
### 1. `docs/provider-contract.md` (new, the generic contract)
Every behaviour axis gets a stable kebab-case code in backticks, used verbatim by the appendix table and the template so the three files can be cross-checked by grep.
Sections:
1. **Preamble**: audience, reading order (contract, template, appendix), rule keywords, "path plus symbol, never line numbers", where the rest lives.
2. **What a provider is and who calls it**: the types in `provider.go`, `name.go`; callers and entry points (`reconcileManaged`, `replaceInstance`, `reconcileDelete`, `providerFailure`, `deletionFailure`, `placementFrom`, poll intervals, `MaxConcurrentReconciles`; `Sweeper.sweep` and `MinAge`; `Monitor.checkOnce`; decorators; `registry.Build`); what the instance is for (health probe CONNECT in `internal/health`, discovery handing out ip and port); leader-only vs every-replica; concurrency (MUST be safe, including two calls for one name); runtime envelope; time and cancellation (no caller deadline, MUST honour cancel, SHOULD bound each backend call, what breaks: a pinned worker, a stalled GC sweep, a wedged check round).
3. **The contract, method by method**: current §2 rewritten abstractly, plus the new points. Create: idempotent by name, mark on create or within a documented window shorter than `MinAge`, self-contained ID, ctx, blocking vs submit, reject-early with `ErrPermanent` naming field and value, catalogue changes are outside the spec hash (`spechash.go`), MAY ignore Placement fields but MUST document, SHOULD deliver CloudInit, MUST NOT log it, and an eventually consistent backend MUST NOT let `Get` answer NotFound for an ID `Create` just returned. Get: NotFound only when gone, keep answering during delete, Running only with IP, Stopped/Terminated only when it will not recover, the "defined but not started" rule (block Create through start, or report Provisioning using written metadata or a task handle), behaviour during async operations and locks (last known state or Transient, never NotFound or Permanent), UID and CreatedAt read back, Permanent for an unresolvable ID. Delete: idempotent, re-entrant while deleting, removes secondary resources, MAY return before gone, SHOULD succeed where Get fails permanently. ListByTag: marked instances only with ID, UID, CreatedAt; error rather than an unvouched list; MAY omit unreadable parts; SHOULD cover the whole creatable scope; no installation identity.
4. **Behaviour profiles**: one subsection per axis with the question, allowed profiles, what callers tolerate, obligations per profile. Axes: `readiness`, `create-return`, `delete-return`, `ip-source`, `gc-mark`, `created-at`, `list-partial`, `quota`, `not-found`, `primary-key`, `provisioning-state`, `async-ops`, `secondary-resources`, `credentials`, `client-kind`, `timeouts`, `logging`, `child-spans`, `backend-test`. Closing table: code, profiles, tolerated by which caller, template entry.
5. **Error taxonomy**: `Wrap`, `Class`, `Error` fields, `Op` values, configured name not type; the class-to-reaction table; pitfalls; NotFound detection guidance (404, error code, list-and-miss, and why list-and-miss is forbidden with a partial list); the quota-vs-permission 403 case stated abstractly.
6. **Identity, names, marks, CreatedAt**: `NameFromUID` shape, providerID strict parsing, marks and written metadata, provider name as DNS-1123 (`Monitor` skip, `spec.provider` MaxLength), primary key not the name.
7. **Credentials, TLS and secret hygiene** (new): source and read time (constructor MUST NOT dial, SHOULD warn on a missing optional mount and fail at first use), never in logs, spans, error strings or `Error.Err`, TLS trust options and the naming of an insecure toggle, scratch only under `/tmp`.
8. **Optional interfaces: Checker**: current §5 rules; `status.type` comment in `api/v1alpha1/providerhealth_types.go` must list the new type and the CRD be regenerated.
9. **Observability**: the three layers, span and metric names, ctx and logger rules, child-span strategies by `client-kind`, no secrets or CloudInit in attributes, log tiers.
10. **Configuration and wiring**: `ProviderConfig`, `setBlocks`, the allow-list in `Config.validate`, `UnmarshalStrict`, `config_<type>.go`, identifiers pasted into IDs, `registry.Constructor`, the map in `cmd/main.go`, decorator order, the duplicated type list.
11. **Testing obligations**: fake behind a seam; the contract test list (current checklist item 5 plus Get during an async operation, the not-yet-started state, NotFound after Create is forbidden); classify table test; Checker tests; sentinel tests for CloudInit and credentials; span recorder test; integration test behind `//go:build integration` with `<TYPE>_TEST_*` variables, `make test-integration-<type>`, a section in `docs/testing.md`.
12. **Checklist for adding a provider**: current §8 generalised, plus: Makefile target, testing.md section, `hack/providers-dev.yaml` decision, `ProviderHealth` type comment and `make manifests`, RBAC diff, CHANGELOG, appendix row and section, the filled template committed as `docs/<type>-provider-spec.md`, README row, architecture.md rows.
13. **Known limits of the contract**: adoption unverified; Stopped and Terminated indistinguishable; config read once; no installation identity in GC; unconfigured provider latches; type list duplicated; Delete resubmitted every poll; no caller deadline; no conformance suite.
14. **Related documents**.
### 2. `docs/provider-spec-template.md` (new, the questionnaire)
Preamble: copy to `docs/<type>-provider-spec.md`, fill every Answer line, hand it to the implementing agent with the contract and appendix; unanswered entries block implementation. Header block: type, example configured name, backend and API version, SDK or thin client, author, date.
Fixed entry format: Question; Allowed answers (profile names linking to the contract §4 subsection); What each answer obliges; `Answer:` blank; `Evidence:` blank.
Entries: identity and providerID shape; primary key (`primary-key`); placement mapping (Region, Zone, MachineType, Image, ignored fields); image source; cloud-init delivery; machine type mapping; network and IP discovery (`ip-source`, `readiness`); GC marks and CreatedAt (`gc-mark`, `created-at`); state mapping table incl. not yet started (`provisioning-state`); Create blocking and task tracking (`create-return`, `async-ops`); Delete and secondary resources (`delete-return`, `secondary-resources`); ListByTag scope and partial failure (`list-partial`); error classification table (`not-found`, `quota`); credentials and TLS (`credentials`); config block fields, defaults, validation; Checker targets, reasons, Details; SDK vs thin client and child spans (`client-kind`, `child-spans`); timeouts (`timeouts`); concurrency; logging tiers (`logging`); tests (`backend-test`); deployment (mounts, samples, Makefile helper, RBAC, dev config); docs; known limits accepted at design time.
### 3. `docs/providers-existing.md` (new, the appendix)
Sources: current `providers.md` §6 table, §7 matrix and per-provider notes, §9 per-provider gaps and proposed features, `docs/kvm-provider.md`, the gcp and kubernetes package facts. Links to architecture.md §5, §11 and Decisions, and to the README table, rather than duplicating them.
Sections: preamble; **profile table** with exactly one row per contract axis code and one column per existing provider, each cell a profile name plus a symbol as evidence; **other differences** (matrix rows that are not axes: Placement used, Port used, CloudInit delivered, states produced, Error.Provider correctness, credential rotation, connection handling, Checker Details, fakes, proxy software, reachability, distinct egress, adopted-UID check); **observability status** (current §6 table); one section each for **kubernetes**, **gcp**, **kvm** with the same sub-headings: package layout and seam, config block, ID format, Create, Get and state mapping table, IP source, Delete, ListByTag, classify table, Check (targets, reasons, Details), logging, tracing layer, tests and fakes, deployment bits, known gaps; **gaps shared by the existing providers**; **proposed features** (current list, unchanged).
### 4. `docs/providers.md` (rewritten in place as the index)
One paragraph on the split; one line each for contract, template, appendix; the per-provider guides (`docs/kvm-provider.md`, `docs/gcp-vm-validation.md`, `docs/gcp-in-specific-project.md`); the architecture.md sections; reading order for a new provider; a note that the pre-split blueprint is in git history under this path.
### 5. One-line link edits
- `README.md` "### Adding a provider": point at the contract, appendix and template, with `docs/providers.md` named as the index. Keep the "Start there" sentence.
- `CLAUDE.md` "## Architecture": same replacement for the providers.md sentence.
- `docs/architecture.md` components row "Provider contract": rules in `provider-contract.md`, profile table in `providers-existing.md`.
- Historical files under `docs/plans*` keep their links; they resolve to the index.
## Steps (one commit per step, on `feat/provider-blueprint`)
- **Step 0**: copy this plan to `docs/plans/<ts>-provider-docs-split.md` with the real timestamp and commit it alone, before anything else.
- **Step 1**: write `docs/provider-contract.md`; freeze the axis code list. Create `docs/plans-executions/<ts>-provider-docs-split.md` with the status checklist.
- **Step 2**: write `docs/provider-spec-template.md` from contract §4 and §12.
- **Step 3**: write `docs/providers-existing.md`, replacing every `file:line` from the old blueprint with `file` plus symbol.
- **Step 4**: rewrite `docs/providers.md` as the index; apply the three link edits.
- **Step 5**: run the verification below; fix findings.
- **Step 6**: final execution summary entry. CHANGELOG entry at the top only after the user confirms. Push and MR only when the user asks (per memory, push uses the token helper and the MR goes through the Gitea API).
## Verification (read-only shell, from the repo root)
1. Every MUST names a caller: `grep -n MUST docs/provider-contract.md` and confirm each bullet cites a shared-layer path or symbol and states what breaks.
2. Every cited path exists: extract `(internal|cmd|config|docs|api|hack|test)/...` tokens from the four files and `test -e` each.
3. Every cited symbol exists: extract backticked identifiers, drop paths, keywords and YAML keys, and `grep -rqw` each across `*.go`, `Makefile`, `*.yaml`.
4. Contract is abstract: `grep -niE "kubernetes|gcp|kvm|libvirt|squid|google|compute engine|internal/provider/(kubernetes|gcp|kvm)" docs/provider-contract.md` is empty.
5. Profiles line up 1:1: for each axis code from contract §4 headings, it appears in the appendix table and in the template; the appendix table has one row per code.
6. Links resolve: extract `](target)` from the four files plus README, CLAUDE.md and architecture.md; check file targets exist and anchors match slugified headings.
7. No em-dashes in the four files or in the added lines of the three edited files.
8. No line references: `grep -nE '\.go:[0-9]+|:[0-9]+-[0-9]+' <four files>` is empty.
9. Nothing lost: compare the gap and proposal bullets of `git show HEAD:docs/providers.md` against the appendix and contract §13 by keyword.
10. `go build ./... && go test -short ./...` still pass (docs-only change).
## Execution summary must record
- Step 1: the frozen axis list; the two contract additions found in verification (sequential GC sweep; the Stopped branch re-deleting a not-yet-started instance); per-call timeout facts as verified for each existing provider.
- Step 3: the fact corrections applied (kvm Details keys, kubernetes seam, kvm `isClassified`); matrix rows dropped in favour of the profile table.
- Step 4: the exact link edits.
- Step 5: the verification commands as run with trimmed output.
- Worth noting: CHANGELOG headings keep the repo's own em-dash format; the stale `mock` comments in `provider.go`, `registry.go` and `proxy_types.go` stay listed as gaps and are not fixed here.
docs/plans/2026-09-16-1333-kvm2-provider.md
+301
View File
@@ -0,0 +1,301 @@
# Plan: kvm2, a clean-room libvirt/KVM provider built from the provider contract
**Created:** 2026-09-16 13:33
## Context
The provider documentation was just split into a generic contract
(`docs/provider-contract.md`), a spec template (`docs/provider-spec-template.md`)
and an appendix on the existing providers (`docs/providers-existing.md`), so that
a new provider can be specified and built from those files alone. Before using
them for Proxmox, the user wants a dry run on a known backend: rebuild the
libvirt/KVM provider from scratch as a second type, `kvm2`, following the
documents, without touching or reading the existing `internal/provider/kvm/`
code. The result is a validated workflow (spec first, then implementation by
fresh agents against the contract) and a v2 provider that can replace v1 once it
passes on the real hypervisor.
User decisions:
- **Placement:** new package `internal/provider/kvm2`, type `kvm2`, config block
`kvm2:`, side by side with `kvm`. The Go config struct reuses
`provider.KVMConfig` (`internal/provider/config_kvm.go` is shared code, not
the kvm package), so an entry ports by changing `type` and the block name.
- **Clean room:** implementers never open, grep or import
`internal/provider/kvm/`. Docs (`docs/kvm-provider.md`) and the operational
lessons from the first build's execution log are allowed.
- **Implementer:** the orchestrator (this session) writes the spec and the step
briefs; each implementation step runs in a fresh general-purpose subagent
given only the three docs, the filled spec, the backend facts and its brief.
The orchestrator reviews each result against the contract, runs the tests and
commits.
- **Profiles:** `create-return` blocking, `provisioning-state` create-blocks,
`child-spans` span-decorator, `quota` detected (pool free space precheck),
`credentials` mounted-files (the existing `kvm-ssh` Secret), `client-kind`
rpc-over-ssh, `gc-mark` written-metadata, `created-at` written-at-create,
`ip-source` discovered (agent, then ARP), `list-partial` fail-whole per host
with unreadable domains skipped, `not-found` error-code, `delete-return`
waits, `secondary-resources` explicit, `timeouts` per-call, `logging` tiered,
`backend-test` integration-tag.
## Backend facts the spec carries (allowed lessons)
- Real host `beelink`: libvirt 11.6, QEMU 10.1, pool `default` at
`/srv/vms/images`, bridge `br0`, `beelink.tail8c69c.ts.net:22`, user `novakj`,
key `~/.ssh/proxy-operator_ed25519`, known_hosts `~/.ssh/kvm_known_hosts`.
The Debian trixie qcow2 is about 435 MB.
- AppArmor confines QEMU from the domain XML: only `<disk type="file">` with a
`<source file>` path gets a rule, so volume paths are resolved with
`StorageVolGetPath` and emitted as file disks.
- Debian 13 hangs in GRUB under BIOS with a serial console and no video: UEFI
by default (`<os firmware="efi">`, secure boot feature disabled), `firmware:
bios` per image; OVMF is a host prerequisite; undefine needs the NVRAM flag.
- libvirt hands the metadata element back without its namespace declaration:
parse by local name.
- Vendor-data must be a shell script, not cloud-config, to survive cloud-init's
merge with user-data; it installs `qemu-guest-agent`. The agent answered after
about 20 s where ARP never did.
- Read-only rootfs: `iso9660.NewWriter` needs a temp dir under `/tmp` (64Mi).
- go-libvirt (`github.com/digitalocean/go-libvirt`, in go.mod): every method
blocks and takes no ctx; `libvirt.Error{Code uint32, Message}` has a value
receiver; needed `ErrorNumber` constants exist (`ErrNoDomain`,
`ErrNoStorageVol`, `ErrNoStoragePool`, `ErrOperationInvalid`,
`ErrNoDomainMetadata`, `ErrStorageVolExist`, `ErrXMLError`, `ErrNoSupport`,
`ErrInvalidArg`, `ErrAuthFailed`, `ErrAccessDenied`). `libvirt.New(net.Conn)`
then `Connect()`. No new dependencies: `x/crypto/ssh`, `kdomanski/iso9660`
are already direct requires; use `slices`, never `sort` (depguard).
## Design decisions recorded in the spec (step 1)
- **Package layout** (`internal/provider/kvm2/`): `kvm2.go` (type, `New`,
logger, per-name keyed lock), `id.go`, `state.go`, `errors.go`, `xml.go`,
`seed.go`, `backend.go` (ctx-first seam over the 20 go-libvirt calls, adapter
with a per-RPC deadline), `ssh.go`, `hosts.go` (one cached connection per
host, lazy dial, dialer seam), `observe.go` (the single decorator: one span
`kvm2.<Call>` per RPC plus the V(1)/V(2) log line), `image.go`, `create.go`,
`get.go`, `delete.go`, `list.go`, `check.go`, plus `fake_test.go`,
`sshserver_test.go`, `kvm2_integration_test.go`. Names are suggestions for
the briefs; the implementer may choose others by responsibility.
- **Seam:** every seam method takes `ctx`; the adapter runs the blocking call in
a goroutine and selects on result, `ctx.Done()` and a timer (dial 10s, RPC
30s, `DomainCreate` 90s, upload/download ctx plus a 20m cap). On abandon it
closes the connection and marks it dead; the next call redials. The image
download runs on a dedicated connection so a cancelled poll cannot cut it off.
- **Decorator wrapped once:** applied inside the real dialer where a connection
is born, so provider code never sees an undecorated backend. Spans `kvm2.dial`
(in the dialer) and `kvm2.download` (in `image.go`) are the two non-RPC spans.
A reflection test asserts every seam method is instrumented.
- **Metadata:** own namespace
`https://gitea.home.hrajfrisbee.cz/kacerr/egress-proxies-operator/kvm2`,
element `<instance>` with `managed`, `uid`, `createdAt`. kvm2 never lists or
adopts v1 domains; a `kvm` and a `kvm2` entry may share a host.
- **Adoption guard:** an existing domain whose metadata `uid` is non-empty and
differs from `req.UID` makes `Create` return `ErrPermanent`.
- **Quota precheck:** before creating volumes, `StoragePoolGetInfo`; needed =
image `Content-Length` (if not cached) + `diskSizeGb` + 16 MiB; below that
returns `ErrQuotaExceeded` naming pool, available and needed.
- **Download** requires `Content-Length` (Permanent otherwise), streams through
`sha256` into `StorageVolUpload`, refreshes the pool, writes the marker
volume last; a failed attempt deletes the partial volume.
- **Classify:** `ErrNoDomain`, `ErrNoStorageVol` NotFound, but Transient inside
`Create`; `ErrNoStoragePool`, `ErrXMLError`, `ErrNoSupport`, `ErrInvalidArg`,
`ErrAuthFailed`, `ErrAccessDenied` Permanent; SSH authentication failures and
host-key mismatches or unknown host keys Permanent; missing key or
known_hosts file Transient (Secret pending), unparsable key Permanent;
cancellation, deadlines, `net.OpError`, `io.EOF` and anything else Transient.
- **Same-name concurrency:** a keyed mutex per instance name serialises Create
and Delete for one name.
- **State:** nostate Provisioning; running or blocked without IP Provisioning,
with IP Running; everything else, including unknown codes, Stopped. IP from
the guest agent first, then ARP, first non-loopback non-link-local IPv4.
- **Checker:** per host concurrently on a fresh dedicated connection (dial,
`ConnectGetLibVersion`, `ConnectGetHostname`, pool lookup and info, close), so
it never touches the pooled connection. Reasons: shared ones plus
`PoolNotFound`, `PoolNotActive`, `CredentialsUnavailable`. Details `user`,
`hostname`, `libvirtVersion`, `pool`, `poolState`, `poolCapacityGiB`,
`poolAvailableGiB`.
- **Rotation:** SSH material is read on every dial, so Secret rotation needs no
restart.
- **Config validation messages:** `validateKVM` is refactored into
`validateKVMBlock(prefix, block string, c *KVMConfig)` so messages say
`kvm2.` for the new block; `validateKVM` stays as a wrapper for `kvm`.
## Steps (one commit each, on `feat/kvm2-provider` off `main`)
PR #7 is merged, so `main` carries the three docs. Step 0 begins with
`git fetch origin && git switch main && git pull --ff-only`.
Every subagent brief opens and closes with: "Do not open, grep, list or import
anything under `internal/provider/kvm/`. If a detail seems to need that code,
stop and report the question instead." Each brief names the files it may create
or edit and nothing else, lists the inputs (the three docs, the spec,
`docs/kvm-provider.md` for backend facts, CLAUDE.md conventions), the
acceptance tests by name and the commands to run, and asks the agent to end
with "files read outside the allowed set: none".
- **Step 0 (orchestrator):** copy this plan to `docs/plans/<ts>-kvm2-provider.md`
with the real timestamp, commit alone, create the branch. Execution summary
`docs/plans-executions/<ts>-kvm2-provider.md` with the checklist after step 1.
- **Step 1 (orchestrator): the spec.** Copy `docs/provider-spec-template.md` to
`docs/kvm2-provider-spec.md` and answer all 24 entries with the profiles and
decisions above; every Evidence line cites the user guide, a go-libvirt symbol,
or "verified on beelink in step 11". Entry 21's checklist lists the concrete
test names from steps 4 to 7. Verify: no blank Answer lines, every axis code
used matches contract §4.20, no reference to `internal/provider/kvm/`.
- **Step 2 (subagent): config wiring, IDs, state, classify.** Edits
`internal/provider/config.go` (`KVM2 *KVMConfig` field, `setBlocks`,
allow-list, dispatch), `config_kvm.go` (`validateKVMBlock`),
`config_test.go` and `config_kvm_test.go` rows. Creates `kvm2.go` (type,
assertions, `New` that never dials and warns on unreadable SSH files, stub
methods), `id.go`, `state.go`, `errors.go` with tests:
`TestFormatProviderID_roundTrip`, `TestParseProviderID_malformed`,
`TestMapState_table`, `TestFirstIPv4_filters`, `TestClassify_table`. Also
registers `"kvm2": kvm2.New` in `cmd/main.go` so the two type lists change in
one commit. Verify: three `"kvm2"` sites in `config.go`, no import of
`provider/kvm`, `go test -race ./internal/provider/... && go vet ./...`.
- **Step 3 (subagent): pure builders.** `xml.go`, `seed.go` and tests:
`TestDomainXML_fileDisksWithResolvedPaths`, `TestDomainXML_uefiDefaultSecureBootOff`,
`TestDomainXML_biosEscape`, `TestDomainXML_metadataElement`,
`TestDomainXML_guestAgentChannel`, `TestParseMetadata_withNamespace`,
`TestParseMetadata_withoutNamespaceDeclaration`,
`TestParseMetadata_absentGivesEmptyUID`, `TestVolumeXML_overlayBackingStore`,
`TestSeedISO_filesAndLabel`, `TestSeedISO_userDataByteForByte`,
`TestSeedISO_tempDirRemoved`, `TestVendorData_isShellScriptNotCloudConfig`,
`TestVendorData_multipartWhenExtraProvided`. Verify: no `type="volume"` in
`xml.go`; temp dir under `/tmp` with a deferred removal.
- **Step 4 (subagent): transport, seam, decorator, fakes.** `backend.go`,
`ssh.go`, `hosts.go`, `observe.go`, `fake_test.go`, `sshserver_test.go` and
tests: `TestDial_forwardsUnixSocket`, `TestDial_unknownHostKeyRejected`,
`TestDial_mismatchedHostKeyRejected`, `TestDial_insecureIgnoreHostKeyAccepts`,
`TestDial_missingKeyFileIsTransient`, `TestDial_unparsableKeyIsPermanent`,
`TestDial_handshakeTimeout`, `TestCall_deadlineClosesConnectionAndReturnsTransient`,
`TestCall_cancelUnblocksRPC`, `TestHosts_oneDialPerHostUnderConcurrency`,
`TestHosts_redialsAfterDrop`, `TestHosts_unconfiguredHostIsPermanent`,
`TestObserve_everySeamMethodInstrumented`, `TestObserve_childSpansUnderParent`,
`TestObserve_logsOneLinePerCallAtV1`, `TestObserve_detailOnlyAtV2`,
`TestObserve_uploadBodyNeverLogged`, `TestSSH_keyMaterialNeverInErrorsOrLogs`.
Verify: no `context.Background()` outside tests and the keepalive goroutine;
`tracing.Start` in the decorator and the dialer; `go test -race -count=2`.
- **Step 5 (subagent): image cache and pool-space precheck.** `image.go` and
tests: `TestEnsureImage_downloadsAndWritesMarkerLast`,
`TestEnsureImage_cachedWithMarkerSkipsDownload`,
`TestEnsureImage_missingMarkerRedownloads`,
`TestEnsureImage_sha256ChangeRedownloads`,
`TestEnsureImage_sha256MismatchIsPermanentAndRemovesVolume`,
`TestEnsureImage_noContentLengthIsPermanent`, `TestEnsureImage_http404IsPermanent`,
`TestEnsureImage_http503IsTransient`, `TestEnsureImage_concurrentCallersFetchOnce`,
`TestEnsureImage_cancelledMidStreamReturnsPromptly`,
`TestEnsureImage_streamsWithoutBuffering`, `TestCheckPoolSpace_belowNeededIsQuota`.
Verify: no `io.ReadAll` or `bytes.Buffer` on the download path; download span.
- **Step 6 (subagent): the four contract methods.** `create.go`, `get.go`,
`delete.go`, `list.go`, `kvm2_test.go`, finishing `kvm2.go`. Tests map 1:1 to
contract §11: `TestCreate_repeatReturnsSameIDAndCreatesNothing`,
`TestCreate_resumesAfterEachPartialStep`, `TestCreate_returnsOnlyWhenRunning`,
`TestCreate_rejectsUnknownPlacement`, `TestCreate_adoptionWithForeignUIDIsPermanent`,
`TestCreate_poolOutOfSpaceIsQuotaExceeded`,
`TestCreate_libvirtNotFoundIsTransientNotNotFound`,
`TestCreate_cloudInitSentinelNeverLoggedOrInErrors`,
`TestGet_afterDeleteReturnsNotFound`, `TestGet_duringDeleteStillReturnsInstance`,
`TestGet_backendBusyIsTransientNeverNotFound`, `TestGet_runningWithoutIPIsProvisioning`,
`TestGet_runningWithIPIsRunning`, `TestGet_agentPreferredOverARP`,
`TestGet_addressErrorIsProvisioning`, `TestGet_readsUIDAndCreatedAtFromBackend`,
`TestGet_malformedIDIsPermanent`, `TestGet_unconfiguredHostIsPermanent`,
`TestDelete_malformedIDIsPermanent`, `TestDelete_twiceSucceeds`,
`TestDelete_removesOverlayAndSeed`, `TestDelete_runningDomainDestroyedFirst`,
`TestDelete_toleratesMissingVolumesAndPool`, `TestDelete_unconfiguredHostIsNoOp`,
`TestDelete_concurrentSameNameSafe`, `TestListByTag_excludesUnmarkedDomains`,
`TestListByTag_emptyUIDStillListed`, `TestListByTag_acrossHostsInConfigOrder`,
`TestListByTag_unreachableHostFailsWhole`, `TestListByTag_unreadableDomainSkipped`,
`TestProvider_cancelledContextReturnsPromptly`,
`TestProvider_everyErrorIsWrappedWithConfiguredName`,
`TestLogging_v0SilentV1OutcomeV2Detail`. Verify: `CloudInit` appears in
`create.go` only as `len(req.CloudInit)`; stub strings gone; `Wrap` always
gets the configured name; `go test -race -count=2 ./internal/provider/kvm2/`.
- **Step 7 (subagent): Checker and the CRD comment.** `check.go`,
`check_test.go`: `TestCheck_oneEntryPerHostInConfigOrder`,
`TestCheck_hostsCheckedConcurrently`, `TestCheck_expiredContextReturnsPromptly`,
`TestCheck_reasonsTable`, `TestCheck_neverUsesPooledConnection`,
`TestCheck_detailsKeys`. Edit `api/v1alpha1/providerhealth_types.go`
(`status.type` comment) and run `make manifests`. Verify: only the CRD
description line changes under `config/`; `go test ./internal/providerhealth/`.
- **Step 8 (subagent): deployment, Makefile, integration test.** Edit
`config/manager/manager.yaml` (comment: the `kvm-ssh` mount serves kvm and
kvm2), `config/manager/providers_config.yaml` and
`config/samples/providers-config.yaml` (commented `kvm2` entry),
`config/samples/kustomization.yaml`, `Makefile` (`test-integration-kvm2`,
`-run TestKVM2`, `-timeout 40m`), `docs/testing.md` (suite rows and a kvm2
integration section). Create `config/samples/proxy_kvm2.yaml` (zone
`beelink`, provider `kvm2-home`) and `kvm2_integration_test.go`
(`KVM2_TEST_HOST`, `_USER`, `_SSH_KEY`, `_KNOWN_HOSTS`, `_BRIDGE`,
`_IMAGE_URL`; optional `_IMAGE_SHA256`, `_POOL`, `_SOCKET`, `_TIMEOUT`,
`_KEEP`, `_LOG_LEVEL`) asserting in order: `New` does not dial; `Check`
Ready with details; `Create` returns `hosts/<host>/domains/<name>`; second
`Create` same ID, domain count unchanged; `Get` reaches Running with an IPv4
and the written UID and CreatedAt; `ListByTag` contains it; `Delete` nil;
`Get` NotFound; `Delete` again nil; pool holds base and marker, no per-VM
volumes; with an in-memory span recorder, `kvm2.DomainDefineXML` and
`kvm2.dial` spans are children of `provider.create`. Verify:
`go vet -tags=integration ./... && go vet -tags=e2e ./...`,
`bin/kustomize build config/default` and `config/samples` parse,
`make -n test-integration-kvm2`.
- **Step 9 (subagent): documentation.** Create `docs/kvm2-provider.md`
(prerequisites incl. OVMF, cluster setup reusing `kvm-ssh`, config reference
pointing at the shared `KVMConfig` table, placement mapping and the
add-a-key-never-edit warning, what lands on the host incl. the metadata
namespace, image cache and quota precheck, health-check reasons, logging and
span names, migration from `kvm`, known gaps). Edit
`docs/providers-existing.md` (fourth column in the three tables and a kvm2
section), `docs/architecture.md` (components row, §5 block, §10 span
topology, §11 row, Decisions bullets), `README.md` row, `docs/providers.md`
guide link. Verify with the link and symbol sweep from the docs-split
execution log; no em-dashes in new prose.
- **Step 10 (orchestrator): milestone review.** `go-operator-reviewer` on the
branch with the spec and contract as inputs, then `/code-review high`,
`make lint`, `make test` (full, with envtest). Fix accepted findings with
regression tests in one commit; record rejected ones in the summary.
- **Step 11 (orchestrator with the user): real-host validation, changelog, MR.**
Hand the user a host checklist (leftover `proxy-*` domains and volumes from
earlier runs, OVMF installed, pool running with free space, `br0` up, the
known_hosts entry) and the `KVM2_TEST_*` export lines; do not run remote
probes unprompted. With the user's go-ahead run `make test-integration-kvm2`
(a permission prompt is expected; the allow-list entry
`make test-integration-kvm *` does not match) and, if wanted, the in-cluster
pass on kind: build and load the image, deploy, a scratch ConfigMap with the
`kvm2-home` entry, apply `proxy_kvm2.yaml`, wait for `Ready`,
`kubectl get ph kvm2-home -o yaml`, a `curl -x` through the proxy, delete
and confirm the domain and volumes are gone; scratch overrides are not
committed. Record cold download duration and time to IP. CHANGELOG entry
after the user confirms; push with the token helper; MR via the Gitea API
with base `main`; never merge.
## Verification
- Per step: the checks listed above, plus `go build ./... && go test -short ./...`
before every commit and `git diff --stat` reviewed for files outside the
step's allowed set.
- Clean-room check on every step's diff: `grep -rn 'provider/kvm"' internal/provider/kvm2/`
empty; the agent's closing line "files read outside the allowed set: none";
identifiers that coincide with v1's names from the appendix (`libvirtAPI`,
`hostPool`, `imageCache`, `tracingAPI`, `lazySSH`) are treated as a smell and
questioned.
- Contract mapping: after step 7, every rule of contract §2 and §8 has a named
test in the package; the §11 list maps 1:1 to test names (recorded in the
spec's entry 21).
- End to end: the integration run's ordered assertions (step 8) on beelink, and
the optional kind pass (step 11).
## Risks
- Subagent scope creep into `internal/provider/kvm/`: instruction-only guard,
mitigated by the closing attestation and the diff greps.
- `.claude/settings.json` is locally modified and uncommitted; it stays out of
feature commits, and the permission prompt for the new make target is
accepted rather than pre-approved.
- `Content-Length` is required for downloads; the Debian and Ubuntu mirrors send
it.
- The cancelled-call policy still drops other in-flight RPCs on that host once;
the dedicated download connection removes the expensive case and the spec
states the rest.
- `go-operator-reviewer` hunts over-abstraction; the seam, the decorator and
the dialer interface each have one production implementation and one fake,
which the contract requires (§4.15, §11).
docs/plans/2026-09-17-1515-kvm2-review-fixes.md
+640
View File
@@ -0,0 +1,640 @@
# Plan: fix the kvm2 code-review findings
**Created:** 2026-09-17 15:15
## Context
The review of `feat/kvm2-provider` (`review/REPORT.md`) confirmed 52 findings: 4 high,
20 medium, 28 low. We fix them one at a time, highest severity first, each as its own step and
its own commit. This plan is the campaign; the execution file's checklist is the authoritative
progress record. Steps are detailed as we reach them, so the plan is never a fiction about work
not yet designed.
Precedent for this shape: `docs/plans/2026-09-10-2222-kvm-review-fixes.md`.
## Steps
- [x] **Step 0** — Save this plan into the repo, commit alone, before any code.
- [x] **Step 1** — H1: `sshSocket.Close` ran the blocking close before the unblocking one, and
`markDead` killed inside a `sync.Once` before signalling dead. Shipped in `0239fb7`; the
reasoning is in the execution file.
- [ ] **Step 2** — H2: an error during `StorageVolUpload` wedges go-libvirt's stream sender.
- [ ] **Step 3** — H3: `address()` swallows every error and demotes healthy proxies.
- [ ] **Step 4** — H4: the base image cache is shared across provider entries.
- [ ] **Step 5** — Validate H1-H4 in-cluster against the real hypervisor (detailed below).
- [ ] **Step 6** — Route klog into the structured logger (found during Step 5; detailed below).
- [ ] **Step 7+** — the mediums, from `review/REPORT.md`. Detailed when reached.
---
## Step 2 — H2: a source error must not reach go-libvirt's stream sender
### The bug
`uploadImage` (`image.go:255-296`) hands a reader over `resp.Body` to `StorageVolUpload`. I
verified the mechanism in the pinned `go-libvirt@v0.0.0-20260814190004`:
```go
// rpc.go requestStream
abort := make(chan bool) // UNBUFFERED
outErr := make(chan error) // UNBUFFERED
go func() { outErr <- l.socket.SendStream(serial, proc, program, out, abort) }()
resp, err = l.processIncomingStream(c, in)
if err != nil {
abort <- true
return resp, err // outErr is NEVER drained on this path
}
err = <-outErr
```
`SendStream` (`socket/socket.go`) holds a 4 MiB buffer, polls `abort` once per loop, and on a
**non-EOF reader error** sends a `StatusError` packet and returns. It then blocks forever on
`outErr <-`, because the main goroutine is still inside `processIncomingStream`. libvirtd
answers the `StatusError` with an error, so `processIncomingStream` returns an error and the
main goroutine blocks forever on `abort <- true`. **Deadlock: `StorageVolUpload` never returns.**
kvm2's `call` rescues only the caller, after `budgets.transfer` = 20 minutes, and the socket
kill does **not** unblock either goroutine — both are parked on channel sends, not on socket
I/O, which is what the comment at `backend.go:152-156` gets wrong. So every failed upload leaks
kvm2's call goroutine, go-libvirt's sender goroutine and 4 MiB, permanently. The pod limit is
128Mi and the reconciler retries on backoff.
Three shapes, only the first of which is ours to fix:
- **(A) reader error** — a truncated or failing HTTP body. The common case.
- **(B) server-side error mid-upload** — ENOSPC, or the volume deleted underneath. After
`SendStream` has returned there is nothing any reader, ctx or socket close can do: confirmed
unfixable from our side.
- **(C) our own abandonment** — ctx cancel or the 20 m budget kills the socket, which makes
`processIncomingStream` error. Same tail as (B).
### The fix (workaround; the defect stays upstream)
No fork, no `replace`. Two changes that between them close (A) and the realistic half of (C).
**2a. Never hand go-libvirt a reader that errors** — `internal/provider/kvm2/backend.go`, at the
seam rather than in `image.go`, so it covers the seed and marker uploads too and stays out of
the file Step 4 rewrites:
```go
type eofReader struct {
r io.Reader
mu sync.Mutex
err error
}
func (e *eofReader) Read(b []byte) (int, error) {
n, err := e.r.Read(b)
if err == nil || err == io.EOF {
return n, err
}
e.mu.Lock()
if e.err == nil {
e.err = err
}
e.mu.Unlock()
return n, io.EOF
}
func (e *eofReader) sourceErr() error { e.mu.Lock(); defer e.mu.Unlock(); return e.err }
```
`libvirtBackend.StorageVolUpload` wraps `r` in one, and after a nil-error return converts a
recorded `sourceErr()` into the returned error. `io.EOF` is the one outcome that ends the stream
the normal way: libvirtd gets `StatusOK`, replies OK, `processIncomingStream` returns nil, the
main goroutine reaches `err = <-outErr` and both goroutines unwind.
`countingReader` is untouched — it sits under the wrapper, still sees the real error, so
`uploadImage`'s `readErr != nil` branch and `TestEnsureImage_shortReadRemovesVolumeAndIsTransient`
keep working verbatim.
**Unverified, ~85%:** that libvirtd answers a short-but-clean stream finish with OK rather than
an error. `virFDStreamWrite` enforces `length` only on writes *past* it and nothing compares
`offset` to `length` on the close path, which is also why `virsh vol-upload` of a small file
into a large volume works. If it answers with an error instead, we land back in a deadlock by a
different door — but that is exactly today's behaviour, so the change is non-worsening either
way. Probe recipe goes in the commit message; it needs a real libvirtd and a write, so it is
for the user to run, not an agent.
**2b. Make the transfer deadline land on the HTTP source, not on the libvirt socket.**
`p.do` already builds the request with `http.NewRequestWithContext` (`image.go:360`), so the
body is ctx-bound, and `http.DefaultClient` has no timeout of its own by deliberate choice
(`kvm2.go:100`). Give the image GET in `fetchImage` a deadline of `defaultBudgets.transfer`
minus a grace of one minute (precedent for reaching for `defaultBudgets` directly:
`discardVolume`, `image.go:331`).
A stalled mirror then ends by the *body* failing, which 2a turns into `io.EOF` and a clean
unwind, roughly a minute before the libvirt budget would have killed the socket and leaked.
That converts the realistic repeated (C) — a slow or stalling mirror on a 415 MB image — from a
permanent leak into an ordinary Transient error. The socket kill stays as the backstop for
libvirtd itself hanging, which is the residual we cannot close.
**2c. Close the dedicated connection before unlinking the file.** Move the `discardVolume` call
out of `uploadImage` (`image.go:291-294`) up into `fetchImage`'s upload-error branch, where the
deferred `fresh.Close()` has already run, so libvirtd has let go of the stream's fd before the
volume is deleted. `uploadImage` then drops its `be` parameter and its private logger — a net
deletion. Doing it this way rather than adding an explicit `Close` keeps
`image_test.go:573`'s `closeCount() == 1` assertion true.
### Tests, docs, scope
One test: `TestEOFReader_sourceErrorNeverReachesTheStream` in `backend_test.go` — a table with
"error after data", "error with no data", "plain io.EOF passes through", "first error wins".
~35 lines.
**Deliberately not tested:** the abort protocol end to end. `libvirttest` reads a fixed 28-byte
header per packet and implements no stream procedures, so pinning this hermetically would mean
writing a libvirt RPC server stub — 150-250 lines whose only assertion is about *upstream's*
goroutines. That is the bloat we cut for H1.
Docs: a paragraph in `docs/kvm2-provider.md`'s *The image cache* section saying a truncated
response is reported out of band and why, and a *Known gaps* bullet naming shapes (B) and (C)
as an upstream defect with the two-character patch quoted, so a future reader knows it is not
ours and what the escape hatch is:
```go
abort := make(chan bool, 1)
outErr := make(chan error, 1)
```
File that as an upstream issue/PR against `digitalocean/go-libvirt` and link it from the bullet.
---
## Step 3 — H3: an unreadable address must not look like a missing one
### The bug
`address()` (`get.go:104-119`) tries the guest agent then the host's ARP table. Any error that
is not `context.Canceled`/`DeadlineExceeded` is logged at V(2) and skipped; after both it
returns `("", nil)`. `mapState(active, "")` is `StateProvisioning` (`state.go:20-21`), the
controller clears `p.Status.IP` on Provisioning (`proxy_controller.go:208-211`), and the health
engine prunes the proxy's accumulated state and calls `Metrics.ForgetProxy` in one tick once
`EffectiveHost()` is empty.
The trigger that matters: another reconcile's cancel or budget overrun kills the **shared**
pooled connection, and `call` then returns `transientf("... connection is closed: ...")` with no
ctx sentinel, which `get.go:108` does not recognise. `Get` returns a *successful* Instance in
Provisioning with `IP: ""`, and a healthy proxy is pulled out of rotation by an unrelated
reconcile's timeout.
### The fix
A booting VM and a dead link are distinguishable: on beelink the agent source errors while the
ARP source returns success-with-no-addresses (`docs/kvm2-provider-spec.md:146` — the host never
sees the guest's traffic). So *one* source failing is normal; *every* source failing, or a
connection already gone, is not.
```go
func address(ctx context.Context, log logr.Logger, be backend, dom libvirt.Domain) (string, error) {
errs := make([]error, 0, len(addressSources))
for _, s := range addressSources {
ifaces, err := be.DomainInterfaceAddresses(ctx, dom, uint32(s.src), 0)
if err != nil {
if isDead(be) || errors.Is(err, context.Canceled) || errors.Is(err, context.DeadlineExceeded) {
return "", fmt.Errorf("reading addresses of domain %s from the %s source: %w", dom.Name, s.name, err)
}
log.V(2).Info("no addresses from source", "source", s.name, "error", err.Error())
errs = append(errs, fmt.Errorf("%s source: %w", s.name, err))
continue
}
if ip := firstIPv4(ifaces); ip != "" {
return ip, nil
}
}
if len(errs) == len(addressSources) {
return "", fmt.Errorf("%w: reading addresses of domain %s: every source failed: %w",
provider.ErrTransient, dom.Name, errors.Join(errs...))
}
return "", nil
}
```
The explicit `provider.ErrTransient` wrap is load-bearing, not decoration: without it `classify`
picks the first libvirt code out of the join tree, and `ErrInvalidArg`/`ErrAccessDenied`/
`ErrNoSupport` map to **Permanent** — an address read failing would latch a *running* proxy as
Failed, a worse version of the harm being fixed. `isDead(be)` (`backend.go:50-61`, already
works through the `observed` decorator and the test fake) is not strictly needed — a dead link
fails both sources anyway — but it fails faster, attributes better, and is the same predicate
M15 will need for `ListByTag`.
Also fix the now-wrong comment at `get.go:100-103` ("only ctx ending is an error").
**What this does not fix, on purpose.** A guest agent restarting inside a running VM still
demotes it: the agent errors, ARP returns empty, and the provider cannot tell that from a VM
that is still booting. The discriminator is the *previous* IP, which the provider deliberately
does not hold (`TestGet_readsUIDAndCreatedAtFromBackend` pins "read back, not remembered") and
the controller already has. File as a follow-up finding against
`proxy_controller.go:208-211`: on Provisioning, keep a Proxy's existing IP and `Provisioned`
condition for N polls before clearing, or give the health engine a grace period before pruning.
The one kvm2-local alternative — treating "active, no address, domain older than a provisioning
window" as an error using the mark's `createdAt` — is a design change wearing a bug fix's
clothes, and it invents a provisioning deadline that belongs to the controller. Rejected.
### Tests
- `TestGet_arpFallbackWhenAgentFails` (`get_test.go:135`) passes unchanged — one source of two.
- `TestGet_addressErrorIsProvisioning` (`get_test.go:159`) must be **re-pointed**. It fails both
sources with `ErrAgentUnresponsive`, which is not realistic: code 86 is agent-specific and the
ARP source never produces it. Rewrite as `TestGet_agentFailsAndARPEmptyIsProvisioning` — agent
errors, ARP returns no addresses and no error — which is the beelink boot and the behaviour
the new rule must not break. Nothing covers it today.
- One new test, `TestGet_everyAddressSourceFailingIsTransient`, two subtests: both sources error;
and the pooled link is dead before the address read (the fake implements `liveness`). Assert
Transient, not NotFound, not Permanent.
Docs: `docs/kvm2-provider-spec.md:146-151` says "errors from either source mean 'no address yet'",
now false. The `ErrTransient` row of the error-class table in `docs/kvm2-provider.md`.
---
## Step 4 — H4: the base image cache must not be shared across provider entries
### The bug
`baseVolumeName` (`volume.go:17`) is `base-<key>.<format>`, with a `.sha256` marker beside it.
`internal/provider/kvm/image.go:44-56` builds **byte-identical** names in the same pool — for
URL images too (kvm takes `hex(sum[:8])`, kvm2 takes `hex(sum[:])[:16]`, the same 8 bytes) —
with a completely separate lock table. The shipped config has a live `kvm-home` entry and a
commented-out `kvm2-home` entry on the same host, the same key `debian-13` and the same default
pool, and the comment tells the operator to enable it. Both integration tests use key
`test-image`. On beelink both live overlays back onto `base-debian-13.qcow2`, which kvm created.
Consequences, given `cachedImage` accepts *any* 64-hex marker when no `sha256` is configured and
never re-hashes the base:
- Two entries, one key, different URLs: whichever fetched first owns the file and the other
silently boots the wrong image.
- A concurrent cold fetch: B finds A's base-without-marker, deletes it, recreates and uploads
its own; A finishes into the unlinked inode, writes the marker, and `StorageVolGetPath`
resolves A's handle to B's partial file — a guest boots from a half-written base.
- A `sha256` edit deletes the base in place under live overlays (`cachedImage` branch E), and
libvirt does not refuse to delete a volume that running domains use as a backing store.
### The fix — the provider entry's name in the volume name
`base-<entry>-<key>.<format>`, e.g. `base-kvm2-home-debian-13.qcow2`, with the marker following
automatically. Every production call site already goes through `imageSource.volumeName()`
(`image.go:49`), so this is a small change: give `resolveImage` the provider name and add an
`Entry` field to `imageSource`.
`Provider.name` is already carried (`kvm2.go:104`). But provider names are validated only as
non-empty and unique (`config.go:96-105`) — no charset rule — and a name with a `/` would
produce an unusable volume name, so **`New` must reject an entry name that is not a safe volume
name component**, using the existing `kvmIdentifier` shape. Fail fast at startup, in kvm2 only:
tightening the rule for every provider type is a broader change than this fix.
**What this buys, and what it does not.** It fixes the cross-entry sharing outright — kvm↔kvm2
and kvm2↔kvm2 — which is the wrong-image bug and the concurrent-fetch corruption, and it makes
the existing per-`Provider` `imageLocks` correct rather than a latent bug, since no two entries
can now contend for one name. It does **not** make the name mean the content, so two things
stay open and must be said plainly rather than quietly skipped:
- Editing `images.<key>.url` without changing the key is still a silent no-op. Already
documented behaviour ("add a new key instead"); left as is.
- Cross-*process* fencing still depends on the atomic-publish work (upload to a temp name,
verify, then publish). E4 refuted the rollout and leader-handover triggers for the shipped
manifests, so this is a narrow residual. Its own later commit.
**One companion fix, in this commit**, because the naming alone does not stop it and it is the
remaining path that unlinks a good file under a running domain: `cachedImage` branch E
(`image.go:129-130`, digest differs from the configured `sha256`) currently deletes the marker
and the base. Make it return Permanent instead — "this key is cached with digest X but Y is
configured; add the image under a new key, or delete the Proxies using it and evict the base."
That enforces the rule the contract already states instead of contradicting it. Keep the delete
for the no-marker and non-hex-marker branches, where the base is genuine debris.
### Migration
Nothing to do. The existing `base-debian-13.qcow2` on beelink is kvm's, kvm keeps using it
untouched, and kvm2 downloads its own copy once per host (~415 MB) when `kvm2-home` is enabled.
No window in which anything is replaced under a running guest. The pool needs room for both.
### Tests and docs
Tests are mostly literals becoming helpers: `volume_test.go:122-125`, `image_test.go:184,190,196`
(`wantVolume`), `create_test.go:64-70`, `delete_test.go:67`, `provider_test.go:409`; a small
`fx.baseName()` helper covers the last three. `image_test.go:281,298` should assert the log
field and the `kvm2.volume` span attribute **by prefix**, which keeps pinning that the key stays
readable in logs. `kvm2_integration_test.go:481-484` switches from `baseVolumeName(...)` to
`resolveImage(...).volumeName()`; its assertion that the base survives `Delete` is unchanged.
Two new tests:
- `TestBaseVolumeName_isPerProviderEntry` — two entries, same key, different names; the produced
name never equals the kvm provider's `base-<key>.<format>`.
- `TestNew_rejectsUnsafeEntryName` — an entry name with a `/` fails at construction.
**Not tested:** that libvirt permits deleting an in-use backing file (established from libvirt
source by the review; re-proving it needs a write on a real host), and the cross-process race
(no hermetic way, and its fix is deferred).
Docs: the *What lands on the host* table (`docs/kvm2-provider.md:198-212`), the eviction recipe
(`:305-318`, which currently names `base-debian-13.qcow2` and would delete kvm's file), the
migration section (`:346-370` — kvm2 never reuses a kvm base, one extra download, nothing to
delete), the prerequisites note that the pool needs one base per distro **per entry**, the
naming contract in `docs/kvm2-provider-spec.md` entries 4 and 10, the shared validator's message
at `internal/provider/config_kvm.go:139-141` ("it becomes the cached volume name
`base-<key>.<format>`", now wrong for kvm2), and the kvm2 block's comment in both
`providers-config.yaml` files. Leave `docs/kvm-provider.md` alone; kvm is unchanged.
---
## Step 5 — Validate in-cluster against beelink
### Why, and what this run can and cannot prove
The kvm2 provider has **never run against real libvirt**: Step 11 is unchecked in
`docs/plans-executions/2026-09-16-1333-kvm2-provider.md`, every `base-*` volume on beelink
predates it, and both live `proxy-*` domains carry the kvm v1 metadata namespace. Four fixes
have landed on code no hypervisor has executed. This run is first contact, and it is also the
gate the campaign's `CHANGELOG` entry waits on.
It proves the happy paths: the dial and health probe, a real 415 MB image download through the
reader H2 now wraps, H4's new volume name appearing beside kvm's untouched one, a second Proxy
hitting the cache, H3's address path reaching Running with an IP, and Squid answering a real
CONNECT probe.
It does **not** prove H2's error paths. The unverified ~85% claim — that libvirtd answers a
short stream finish with OK — only arises when a body *fails*, and a healthy download never
finishes short. The 19-minute HTTP deadline will not fire either. A green run means "the
wrapper did not break normal uploads", not "the deadlock is settled". That still needs the
scratch-pool probe recorded in the Step 2 execution entry.
### The cluster this runs in, and the one hazard
`kind-proxy-operator-demo` is **not a throwaway**. It runs the operator that currently owns
four Ready proxies, two of them real VMs on beelink:
```
proxy-beelink-1 Managed kvm-home Ready 192.168.0.166 True
proxy-beelink-2 Managed kvm-home Ready 192.168.0.94 True
proxy-kubernetes-demo-1/2 Managed kubernetes Ready 10.244.0.7/.8 True
```
Validating kvm2 means rolling the operator that owns them. That is low risk — the controller
and the whole `kvm` provider are untouched by H1-H4 — and rollback is one `kubectl set image`.
**Do not run a second operator against beelink with `kvm-home` in its config.** Orphan GC
(`internal/gc/gc.go:86-131`) lists every Proxy in *its own* cluster, and deletes any instance
whose UID is not in that set and is older than `--gc-min-age` (10 m, swept every 10 m). A
second cluster's live set would not contain `proxy-beelink-1/2`'s UIDs, so its GC would destroy
them within ten minutes. Using the existing cluster removes the hazard entirely: its live set
is the one that protects them.
### Steps
**0. Baseline** — record what is there before touching anything, so "did we break it" is
answerable: `kubectl get px -A`, `kubectl get ph`, and a read-only `virsh vol-list default`
plus `virsh list --all` on beelink.
**1. Build and load, under a new tag.**
```bash
make docker-build IMG=egress-proxies-operator:kvm2-dev
kind load docker-image egress-proxies-operator:kvm2-dev --name proxy-operator-demo
```
The tag **must** differ from the running `kvm-dev`. `make deploy` sets the image through
kustomize; an identical tag leaves the Deployment spec unchanged, so no rollout happens, so the
new process never starts, so the new providers config is never read. The ConfigMap carries no
generator hash, so nothing else forces a restart either.
The working tree is dirty (`review/`, `.claude/`), so `GIT_COMMIT` bakes in as `c10f98a-dirty`
and `--version` will say so. Cosmetic; commit or stash `review/` first to avoid it.
**2. Enable kvm2 in the deployed config.** Uncomment the `kvm2-home` block in
`config/manager/providers_config.yaml` (it is the shipped default, commented out). Keep its
`address: beelink.tail8c69c.ts.net:22` — the Tailscale FQDN, which is what
`~/.ssh/kvm_known_hosts` pins. `config/samples/providers-config.yaml` uses the short
`beelink:22`, which **fails host-key verification**; I hit exactly that when validating SSH
earlier in this session.
**3. Deploy.**
```bash
make deploy IMG=egress-proxies-operator:kvm2-dev
kubectl -n egress-proxies-operator-system rollout status deploy/egress-proxies-operator-controller-manager
```
`make deploy` rewrites `config/manager/kustomization.yaml`'s image block as a side effect; it
already carries `kvm-dev` from a previous run.
**4. STOP GATE — provider health, before anything is created.**
```bash
kubectl get ph
kubectl get ph kvm2-home -o yaml
```
`kvm2-home` must be `Ready 1/1` with reason `Reachable` and details carrying `hostname`,
`libvirtVersion`, `poolState: running` and `poolAvailableGiB`. This proves SSH, the host key,
the libvirt handshake and the pool **without creating a thing**, and it exercises the
dedicated-connection dial and close that H1 touched. A failure here is free to diagnose:
`CredentialsUnavailable`, `Unauthorized`, `Forbidden`, `PoolNotActive` each name their own
cause. Report and wait for a go before step 5.
**5. Two Proxies.** Two manifests in the scratchpad (not committed), modelled on
`config/samples/proxy_kvm2.yaml` — `proxy-kvm2-1` and `proxy-kvm2-2` in `default`, matching the
existing `proxy-beelink-*` convention, `provider: kvm2-home`, `zone: beelink`,
`machineType: small`, `image: debian-13`, port 3128 and the sample's Squid cloud-init. Apply
the first, let it finish, then the second — sequencing them is what makes the cache hit
observable.
```bash
kubectl get px -w
kubectl -n egress-proxies-operator-system logs -f deploy/egress-proxies-operator-controller-manager
```
**6. What to check, and which finding it exercises.**
| Check | Finding |
|---|---|
| `fetching base image into host pool` then `base image cached` with size, duration, digest | H2's wrapped reader on a real 415 MB upload |
| `virsh vol-list default` shows `base-kvm2-home-debian-13.qcow2` **and** `base-debian-13.qcow2` still there | H4: per-entry naming, kvm's cache untouched |
| Second Proxy logs no `fetching`, and the server is not hit again | H4 + the image lock: cache hit under the new name |
| `px` goes Provisioning → Running with an IP, without flapping back to Provisioning | H3: the address path, and no spurious demotion |
| `Healthy=True` | Squid answers a real CONNECT probe end to end |
| `virsh list --all` shows two new `proxy-*` domains | Create |
| After deleting both: domains and their `.qcow2`/`-seed.iso` gone, base image still present | Delete, and the cache surviving |
Optionally `curl -x http://<ip>:3128 https://example.com -I` from this box, if it routes to
192.168.0.0/24 — the existing kvm proxies live on that LAN.
**7. Cleanup.** `kubectl delete px proxy-kvm2-1 proxy-kvm2-2` and let the finalizer do the
work, then verify on beelink read-only. **Any leftover domain or volume: I print the exact
`virsh undefine --nvram` / `vol-delete` lines and you run them** — no write reaches the shared
host from me. Then decide whether the `kvm2-home` ConfigMap entry stays (keep it if you want
kvm2 running for real; `git checkout config/manager/providers_config.yaml` otherwise).
**Rollback at any point:**
```bash
kubectl -n egress-proxies-operator-system set image \
deploy/egress-proxies-operator-controller-manager manager=egress-proxies-operator:kvm-dev
```
### Who runs what
I drive steps 0-6 and stop at gate 4 for a go. Every beelink write is yours. Nothing here
touches `proxy-beelink-1/2`, `base-debian-13.qcow2`, your unrelated VMs, or the five
pre-existing orphan volumes the review found.
---
## Step 6 — Route klog into the structured logger
**Not a review finding.** Found while watching the Step 5 validation: some lines ignore the
zap encoder entirely, so no amount of `--zap-encoder=json` makes the stream parseable.
```
I0917 19:50:02.580097 1 leaderelection.go:258] "Attempting to acquire leader lease..." lock="..."
```
That is klog's own text sink. client-go's leader elector does `logger := klog.FromContext(ctx)`
(`leaderelection.go:257`), and controller-runtime hands the elector a context with no logr in it
(`pkg/manager/internal.go:461-473` — `mgr.Start` gets `ctrl.SetupSignalHandler()`, which carries
no logger). `klog.FromContext` falls back to `Background()`, and with no logger registered that
is klog's stderr writer. Nothing in this repo imports or configures klog; it is an indirect
dependency (`k8s.io/klog/v2 v2.140.0`) that ~60 linked packages use, including
`client-go/tools/{leaderelection,cache,record}`, `client-go/transport` and `client-go/rest`.
controller-runtime does not route it either — its only `klog.SetLogger` call is in its own test
suite.
### 6a. `cmd/main.go` — one line and an import
```go
logger := zap.New(zap.UseFlagOptions(&opts))
ctrl.SetLogger(logger)
klog.SetLogger(logger)
```
**Plain `SetLogger`, deliberately not `SetLoggerWithOptions(logger, klog.ContextualLogger(true))`.**
The contextual form is what klog's own docs recommend, because it skips a klogr round trip — but
it also disables klog's verbosity gate ("Such a logger cannot rely on verbosity checking in
klog", `contextual.go:83-86`), handing the decision to zap's level. With `Development: true`
that level is Debug, which enables V(1), so client-go's V(1) chatter that klog currently
suppresses at its default `-v=0` would start appearing. Plain `SetLogger` keeps klog's own check
in front ("klog will do its own verbosity checks before calling logger.V().Info",
`contextual.go:43-44`) and still redirects everything it does emit ("all log lines will be
suppressed from the regular output, and redirected to the logr implementation", `:47-48`).
Same fix, no new volume. Performance is irrelevant at this scale.
`go.mod` promotes `k8s.io/klog/v2` from indirect to direct. It is already in the module graph,
so nothing new is downloaded.
### 6b. `internal/discovery/server.go` — the other stderr bypass
The discovery API's `http.Server` (`:113-119`) sets no `ErrorLog`, so `net/http`'s own errors —
`Accept` failures, handler panics that escape — go to `log.Default()` and out to stderr in
stdlib format. `s.log` is already in scope two lines above:
```go
srv := &http.Server{
Handler: s.handler(),
ErrorLog: slog.NewLogLogger(logr.ToSlogHandler(s.log), slog.LevelError),
ReadHeaderTimeout: readHeadTimeout,
...
}
```
`logr.ToSlogHandler` is the pattern the repo already uses for the same purpose in
`internal/provider/gcp/wirelog.go:31-36`. This server is plain HTTP with no TLS, so `ErrorLog`
carries genuine faults rather than handshake noise from scanners — Error level is right.
### 6c. `config/manager/manager.yaml` — make the deployed stream parseable
```yaml
args:
- --leader-elect
- --health-probe-bind-address=:8081
- --providers-config=/etc/proxy-operator/providers.yaml
- --zap-encoder=json
- --zap-time-encoding=rfc3339
```
Verified that the flag wins over the in-code default: `addDefaults` only fills `NewEncoder`
`if o.NewEncoder == nil` (`zap.go:176-191`), and `BindFlags` sets it from `--zap-encoder`.
`--zap-time-encoding=rfc3339` matters because the JSON encoder defaults to `epoch`, which would
give `1758138422.31` instead of a timestamp anyone can read. Verbosity is left at debug, so the
per-RPC lines stay — that was the choice.
### Deliberately not doing
`opts := zap.Options{Development: true}` (`cmd/main.go:154-155`) stays. Flipping it to a
production default would change the format for every existing deployment, not just this one, and
the flags in 6c get the same result here without that blast radius.
### What stays unformatted regardless, and why that is fine
Flag-parse errors and `--version` (`cmd/main.go:160-163`) print before any logger exists, and
`utilruntime.Must` panics in `init()` run before `main`. controller-runtime's own metrics and
health servers also set no `ErrorLog` (`pkg/internal/httpserver/server.go:11-19`) and expose no
option to set one. All three are outside reach; none can fire during steady-state operation.
### Tests
None. All three changes are wiring: two struct fields and one manifest key. `cmd/main.go` has no
test coverage at all, and pinning `klog.SetLogger` hermetically means mutating global state that
would leak into every other test in the package. Verified by observation instead — see below.
Same call as H1's: a test that costs more than it protects does not earn its place.
### Verification
Rebuild, redeploy, and watch the rollout produce the line that started this:
```bash
kubectl -n egress-proxies-operator-system logs deploy/egress-proxies-operator-controller-manager \
| grep -i 'leader' | head -3
```
It must be JSON with an rfc3339 `ts`, not `I0917 ...`. Then confirm the whole stream parses:
```bash
kubectl -n egress-proxies-operator-system logs deploy/egress-proxies-operator-controller-manager \
| jq -e -s 'length > 0' >/dev/null && echo "every line is JSON"
```
That second command is the real check — it fails if *any* line is not JSON, which is exactly the
property being fixed.
---
## Ordering
**H2 → H3 → H4**, three commits.
H3 is orthogonal: it touches only `get.go` and `get_test.go`.
H2 and H4 both touch `image.go` but at different heights, and the order matters. H2's `image.go`
edit is a deletion — `discardVolume` moves out of `uploadImage` and up into `fetchImage` — while
H4 edits `imageSource.volumeName()`, `resolveImage`, and `cachedImage`'s digest branch ten lines
below. Landing H2 first means H4 rebases cleanly; the reverse means H4 is written against a
failure path H2 is about to restructure. Keeping H2's guard in `backend.go` rather than in
`countingReader` is part of the same separation.
Semantically, H2 makes `fetchImage` delete the base after the transfer connection is closed, and
that delete is only unambiguously safe once names are per-entry — so H2's commit message should
say the discard is still unfenced against a second entry sharing the name and that H4 closes it,
and H4's should point back. The deferred atomic-publish work then lands exactly where H2 put the
discard, on a volume this call created under a name no other entry can produce.
## Verification
```bash
go build ./... && go vet ./...
go test -race -count=1 ./internal/provider/kvm2/
make test
```
Per step, the check that actually matters is that the new test fails against the unfixed code —
stash the source edit, keep the test, watch it fail, restore. For H2 that means asserting
`eofReader` converts before the fix exists; for H3, that the dead-link subtest returns a
successful Provisioning instance today; for H4, that two entries produce one name today.
After H4, run the kvm2 integration test on beelink. **It has never run** — Step 11 is unchecked
in `docs/plans-executions/2026-09-16-1333-kvm2-provider.md`, every `base-*` on the host predates
kvm2, and both live domains carry the kvm v1 metadata namespace. It is the first end-to-end
exercise of the new name and does a real cold download beside kvm's untouched base.
Each step: its own commit on `feat/kvm2-provider` with the execution-file entry and its checklist
line ticked. `CHANGELOG.md` lands with a "validate and record" step once the user confirms the
work, as in the kvm campaign. Trailer:
`Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>`.
docs/provider-contract.md
+860
View File
@@ -0,0 +1,860 @@
# The provider contract
This document is the specification a provider must meet. It is written for
whoever builds the next one, human or agent, and it deliberately says nothing
about how any existing provider is built. Those details are in
[providers-existing.md](providers-existing.md); the questionnaire a new
provider's design must answer is [provider-spec-template.md](provider-spec-template.md).
Read the three in that order: this contract, then the template, then the
appendix for worked examples.
Rule keywords: **MUST** means breaking the rule breaks a caller, and every MUST
below names the caller and what breaks. **SHOULD** means every current provider
does it and a new one needs a written reason not to. **MAY** means it is
allowed and the choice must be recorded in the provider's spec.
Code references are a file path plus a function or symbol name, never a line
number. Where a rule is enforced by a caller, that caller is one of:
- the **reconciler**, `internal/controller/proxy_controller.go`;
- **orphan GC**, `internal/gc/gc.go`;
- the **provider health monitor**, `internal/providerhealth/monitor.go`;
- the **decorators**, `internal/provider/metrics.go` and
`internal/provider/tracing.go`;
- the **composition root**, `cmd/main.go`.
## 1. What a provider is and who calls it
### 1.1 The types
A provider is a Go type implementing `provider.Provider`
(`internal/provider/provider.go`): four methods over one VM-like instance,
identified by an opaque `providerID` string.
- `Create(ctx, CreateRequest) (providerID, error)`
- `Get(ctx, providerID) (*Instance, error)`
- `Delete(ctx, providerID) error`
- `ListByTag(ctx) ([]Instance, error)`
`CreateRequest` carries `Name` (from `NameFromUID` in `internal/provider/name.go`,
22 characters, `proxy-` plus base32), `UID` (the owning Proxy's UID),
`Namespace`, `ProxyName`, `Placement` (`Region`, `Zone`, `MachineType`,
`Image`), `CloudInit` (already resolved from inline text or a Secret) and
`Port`.
`Instance` carries `ID`, `IP`, `State` (one of `StateProvisioning`,
`StateRunning`, `StateStopped`, `StateTerminated`), `UID` and `CreatedAt`.
The GC marks are the constants `LabelManaged` (value `LabelManagedYes`) and
`LabelUID`, also in `provider.go`.
Providers are named instances of a type: two entries of one type with
different config blocks are two providers. The configured name is what
`Proxy.spec.provider` refers to, and it is immutable, so a Proxy never moves
between providers.
### 1.2 The callers
Nothing above the provider watches backend resources. Everything the operator
knows about an instance comes from polling `Get`.
- **The reconciler** calls `Create`, `Get` and `Delete` from `reconcileManaged`,
`replaceInstance` and `reconcileDelete`, and branches only on the error class
(`providerFailure`, `deletionFailure`) and on `Instance.State`. It maps the
CRD's placement to `Placement` one to one in `placementFrom`. Its poll
intervals are set in `applyDefaults`: `ProvisioningPoll` and `DeletionPoll`
10s, `DriftPoll` 2m, `RequeueNow` 1s, `QuotaRetry` 5m. Up to 3 Proxies
reconcile at once (`MaxConcurrentReconciles` in `SetupWithManager`).
- **Orphan GC** calls `ListByTag` and `Delete` from `Sweeper.sweep`, every
`Interval` (10m), and never deletes an instance younger than `MinAge` (10m).
It loops over providers one after another in a single goroutine.
- **The provider health monitor** calls the optional `Checker` from
`Monitor.checkOnce`, every `--provider-check-interval`, all providers
concurrently under one timeout.
- **The composition root** builds every provider through `registry.Build`
(`internal/provider/registry/registry.go`) before the manager starts, on
every replica, then wraps each one in `WithTracing(WithMetrics(...))`.
**Where the calls come from.** The reconciler, GC, the health engine and the
monitor run on the leader only. The constructors run on every replica at
startup.
**Concurrency.** Up to 3 reconciles, a GC sweep and a health-check round can
call the same provider at the same time, and GC's `Delete` can overlap the
reconciler's `Delete` for the same instance. A provider **MUST be safe for
concurrent use, including two calls for the same instance name**. Nothing
above the provider serialises them.
### 1.3 What the instance is for
The instance is a forward proxy with its own egress path. Two parties connect
to `http://<Instance.IP>:<spec.port>`:
- the health engine, from the operator pod, which sends a real `CONNECT`
through it (`probe` in `internal/health/probe.go`);
- clients, which get `ip` and `port` from the discovery API (`viewOf` in
`internal/discovery/handlers.go`).
So the IP a provider reports **MUST be reachable on that port from the
operator's pod network and from the clients**. Whatever that takes on the
backend (firewall rules, security groups, routing) is either done by the
provider or documented as a prerequisite in the provider's guide.
The provider does not know which proxy software runs. The proxy is installed
and configured by the Proxy's `cloudInit`; building install logic into the
operator is a stated non-goal (`docs/prompts/__initial-prompt.md`, non-goals).
A backend that cannot take user-data (see `readiness` in §4) documents that
it ignores the field and how the proxy gets configured instead.
### 1.4 Where it runs
The manager container (`config/manager/manager.yaml`) runs as non-root with a
read-only root filesystem, a 64Mi `emptyDir` at `/tmp` and a 128Mi memory
limit, from a distroless static image (`Dockerfile`) that carries the public
CA roots and nothing else. So a provider:
- **MUST stream large payloads** rather than buffering them in memory;
- **MUST write scratch files only under `/tmp`**, and delete them;
- gets credentials either as mounted files (the providers ConfigMap at
`/etc/proxy-operator`, optional Secret mounts) or from the environment or
the pod's own identity. See §7.
### 1.5 Time and cancellation
**No caller puts a deadline on the context.** The reconcile and GC contexts
are cancelled on shutdown and on loss of leadership, and otherwise never.
The only bounded context is the health monitor's (`Monitor.checkOnce`, one
timeout per round).
- **MUST honour `ctx`**: return promptly once it is cancelled, on every
method.
- **SHOULD bound every backend call itself**: a dial timeout, a per-request
timeout, a cap on any wait loop. What breaks otherwise: a hung `Create`,
`Get` or `Delete` pins one of the 3 reconcile workers until the manager
stops; a hung `ListByTag` stalls orphan GC for every provider, because
`Sweeper.sweep` runs them sequentially; a `Check` that ignores its context
wedges a monitor round. Choose budgets against the polls: a `Get` that
routinely outlives `ProvisioningPoll` gains nothing.
## 2. The contract, method by method
### 2.1 Create(ctx, CreateRequest) (providerID, error)
- **MUST be idempotent by `req.Name`.** A repeat call for a name that already
exists returns that instance's providerID with a nil error, creates no
duplicate and returns no error.
*Caller:* the reconciler records the ID in status only after `Create`
returns (`reconcileManaged`). A crash between the two, or a lost status
write, runs `Create` again for the same Proxy; `replaceInstance` recovers
from a crash between its two writes the same way.
- **MUST mark the instance as managed**, with `LabelManaged=LabelManagedYes`
and `LabelUID=req.UID` or the backend's equivalent, in the same call that
creates it. When the backend's create call cannot carry the mark, the
window between creation and marking **MUST be far shorter than `MinAge`**
and documented in the provider's spec (`gc-mark` in §4).
*Caller:* orphan GC finds instances only through the mark and never deletes
one whose UID is empty (`Sweeper.sweep`). An instance that exists without
the mark leaks if the operator crashes before marking it.
- **MUST return a providerID that is self-contained.** `Get` and `Delete`
receive only the ID, and the config or spec may have changed since
`Create` (a zone edit during a replacement). Put whatever `Get` needs to
find the instance into the ID, and validate config values that end up in
the ID so they cannot contain the separator. See `primary-key` in §4 for
backends that allocate their own identifier.
- **MUST NOT let `Get` answer NotFound for an ID that `Create` just
returned.** A backend with eventual consistency between create and read
needs a read-after-write guarantee or a retry inside `Create`.
*Caller:* on NotFound the reconciler clears the ID and creates again
(`reconcileManaged`), so a lagging read makes a duplicate.
- **MUST honour `ctx`** (§1.5).
- **MAY return once the request is submitted, or block until the instance is
running** (`create-return` in §4). Submit-and-return is preferred: readiness
is discovered by polling `Get`, and a blocking `Create` holds one of the 3
workers for its whole duration. A provider that blocks MUST still honour
cancellation and SHOULD cap the wait.
- **SHOULD reject a request it can never satisfy** with `ErrPermanent`
before touching the backend: a missing or unknown zone, machine type or
image, or any placement value not in its catalogue. The CRD cannot validate
`placement`, because it is provider-agnostic, so this error is the user's
only feedback. The message SHOULD name the offending field and value.
- **Catalogue changes do not replace instances.** The spec hash
(`specHashInput` in `internal/controller/spechash.go`) covers `placement`,
the resolved `cloudInit` and `port`, not the provider config. When a
placement value names a config entry, changing what that entry means affects
only new instances. The provider's guide MUST say so and tell users to add a
new key rather than edit one in use.
- **MAY ignore `Placement` fields that do not apply.** If a field is ignored,
the provider's guide MUST say so, because the field is still part of the
spec hash: changing it replaces the instance and has no other effect.
- **SHOULD deliver `CloudInit` as user-data**, byte for byte, without parsing
or re-rendering it. A provider that cannot documents that it ignores the
field.
- **MUST NOT log `CloudInit` contents**, put them in span attributes, or
include them in error messages. It can hold secrets. Log its byte count.
Test this with a sentinel string (§11).
### 2.2 Get(ctx, providerID) (*Instance, error)
- **MUST return `ErrNotFound` when, and only when, the instance is gone.**
*Caller:* `ErrNotFound` triggers three actions. In normal reconciling the
reconciler clears the ID and creates again (`reconcileManaged`). During
replacement it is the signal to create the new instance
(`replaceInstance`). During deletion it releases the finalizer
(`reconcileDelete`). A false NotFound during an outage creates a duplicate
or loses the instance. "Unreachable", "wrong credentials" and "cannot tell"
are Transient or Permanent, never NotFound.
- **MUST keep returning the instance, not NotFound, while a delete is in
progress** and the name is still taken.
*Caller:* the replacement instance gets the same name. `replaceInstance`
waits for NotFound before creating it; a NotFound too early makes `Create`
adopt the dying instance.
- **MUST report `StateRunning` only with a non-empty `IP`.** Running without
an address yet is `StateProvisioning`.
*Caller:* the IP from a Running answer goes straight into status and to the
health engine (`reconcileManaged`).
- **MUST report `StateStopped` or `StateTerminated` only when the instance
will not recover by itself.** SHOULD map states it does not recognise to
`StateStopped`.
*Caller:* the reconciler reacts to both states identically: it calls
`Delete` and requeues after `DeletionPoll` without clearing the ID, then
polls `Get` and deletes again until NotFound, then creates
(`reconcileManaged`). Instances are disposable, so this is the safe answer
for anything unknown, but it makes the next rule critical.
- **MUST NOT report a freshly created instance that has not been started yet
as `StateStopped`.** A backend with no provisioning state of its own, where
"defined but not yet started" and "stopped" look the same, MUST resolve this
in one of two ways (`provisioning-state` in §4): block `Create` until the
instance is started, or record enough at creation (written metadata, a task
handle) for `Get` to answer `StateProvisioning`.
*Caller:* the same branch of `reconcileManaged` would delete the instance
right after it was created, forever.
- **During an asynchronous operation or a backend lock** (a clone or a delete
still running, an instance locked by the backend), `Get` MUST return either
the last known state or `ErrTransient`, never NotFound and never
Permanent (`async-ops` in §4).
- **MUST fill `UID` and `CreatedAt` by reading them back from the backend**,
never from memory.
*Caller:* GC compares `UID` with the live Proxies and deletes anything whose
`CreatedAt` is older than `MinAge`. **A zero `CreatedAt` counts as very
old** (`Sweeper.sweep`), so a provider that forgets it makes instances still
being created collectable. An unparsable timestamp giving the zero time is
accepted: it never shields an orphan forever.
- **SHOULD return `ErrPermanent` for an ID it can never resolve**: a malformed
ID, a location no longer in the config.
*Caller:* `reconcileDelete` then tries `Delete` anyway and releases the
finalizer if that works. If the provider returned Transient, deletion would
poll forever.
### 2.3 Delete(ctx, providerID) error
- **MUST be idempotent.** Deleting something already gone returns nil.
*Caller:* GC and the reconciler race, and a crash repeats the call.
- **MUST accept being called again while an earlier delete is still in
progress.** Treat "already deleting" as success, and SHOULD keep that cheap.
*Caller:* while `Get` still returns the instance, `reconcileDelete` and the
Stopped branch of `reconcileManaged` call `Delete` again every
`DeletionPoll` until `Get` returns NotFound.
- **MUST remove everything `Create` made**: disks, volumes, seeds, addresses,
uploaded files. Either delete them explicitly or rely on the backend
cascading (`secondary-resources` in §4).
*Caller:* GC sees only what `ListByTag` returns, so secondary resources
have no other cleanup path.
- **MAY return before the instance is gone** (`delete-return` in §4), as long
as `Get` keeps showing it until it is.
- **SHOULD succeed for an ID whose `Get` fails permanently** when there is
nothing left the provider could reach, and log that it did nothing.
*Caller:* on the deletion path a permanent error is not latched; it retries
forever (`deletionFailure`), and the Proxy is then only released through the
`AnnotationSkipCleanup` annotation (`api/v1alpha1/proxy_types.go`).
### 2.4 ListByTag(ctx) ([]Instance, error)
- **MUST return only instances that carry the managed mark**, each with an
`ID` that `Delete` accepts, plus `UID` and `CreatedAt`. `State` and `IP`
are not used by GC.
*Caller:* GC deletes every listed instance whose UID is not live and whose
age is past `MinAge` (`Sweeper.sweep`). Listing something without the mark
is how an operator deletes an instance it does not own.
- **MUST return an error rather than a list it cannot vouch for.** An error
makes GC skip that provider for this sweep (`Sweeper.sweep`).
- **MAY leave out instances it could not read.** Leaving something out is
always safe, because GC never deletes what is not listed. Pick a policy
(`list-partial` in §4) and document it.
- **SHOULD cover the whole scope the provider can create in**: every
namespace, zone, node or host. Anything outside that scope is never
collected.
- **Scope caveat:** GC protects instances by UID, not by installation. Two
operator installs sharing one backend delete each other's instances (§13).
Two provider entries of one install that share a scope are safe, because
they share the live-UID set.
## 3. What a container-style backend and a VM-style backend have in common
The contract above is the same for both, but the profiles in §4 differ.
A backend that runs a container has an instance that is ready seconds after
`Create`, gets its address from the platform, is deleted in one call and has
no secondary resources; a backend that boots a VM has a provisioning window
of minutes, may discover the address only after the guest reports it, may
need several deletes, and leaves disks and seeds behind unless told
otherwise. Neither is preferred. What matters is that the provider's spec
names its profile on every axis, because the appendix table and the test list
are keyed by those names.
## 4. Behaviour profiles
Each axis below has a stable code. The provider's spec (the template) answers
every axis; the appendix maps each existing provider to its profile on every
axis. A profile is not a permission to skip a MUST in §2; it says which of
the allowed ways of meeting it the provider chose.
### 4.1 `readiness`
*When is an instance usable after `Create`?*
- **immediate**: the platform starts it at once; `Get` reports Running
within a poll or two.
- **provisioning-window**: minutes of booting; `Get` reports Provisioning
for that time.
Callers tolerate both: the reconciler polls at `ProvisioningPoll` either
way. An immediate provider still MUST report Provisioning until the IP is
known.
### 4.2 `create-return`
*Does `Create` return once the request is submitted or once the instance is
running?*
- **submit**: preferred. Readiness comes from `Get`.
- **blocking**: allowed. Costs one of 3 workers for the duration, and any
first-use setup (an image download) runs inside it. MUST honour `ctx`,
SHOULD cap the wait, and MUST keep every step idempotent so a cut-off
attempt resumes.
### 4.3 `delete-return`
*Does `Delete` return before the instance is gone?*
- **returns-early**: `Get` keeps returning the instance until it is gone
(§2.2), and repeated `Delete` calls are cheap.
- **waits**: `Delete` returns nil only when nothing is left. Bound the wait
and honour `ctx`.
### 4.4 `ip-source`
*Where does `Instance.IP` come from?*
- **assigned-at-create**: the backend reports it with the instance.
- **discovered**: an agent in the guest, a DHCP lease, a neighbour table.
Discovery failures mean "no address yet", never an error; `Get` returns
Provisioning. The provider's guide MUST say what the guest needs (an agent
package, outbound access to install it) and what happens without it.
### 4.5 `gc-mark`
*Where do `LabelManaged` and `LabelUID` live?*
- **native-labels**: labels or tags the backend indexes; `ListByTag` can
filter server-side.
- **written-metadata**: a description, an annotation or a metadata element
written by the provider; `ListByTag` lists everything and filters
client-side.
- **mark-window**: the create call cannot carry the mark and a second call
writes it. Allowed only when the window is a few seconds, documented, and
the unmarked instance is cleaned by the same Proxy's next reconcile (§2.1).
### 4.6 `created-at`
*Where does `CreatedAt` come from?*
- **backend-timestamp**: the backend's own creation time.
- **written-at-create**: the provider writes the time into the mark
metadata in the create call. A backend with no timestamp MUST do this.
### 4.7 `list-partial`
*What does `ListByTag` do when part of its scope is unreachable?*
- **fail-whole**: one unreachable part fails the list; GC skips the provider
this sweep. Nothing is collected until every part answers.
- **omit-unreachable**: unreachable parts are left out and the rest is
returned. Orphans in the reachable parts are collected; the provider MUST
log what it left out.
Both are safe (§2.4). A single unreadable instance MAY always be skipped.
### 4.8 `quota`
*Can the provider tell "no capacity" from other failures?*
- **detected**: a recognisable quota, rate-limit or out-of-resources signal
maps to `ErrQuotaExceeded`.
- **absent**: capacity errors map to `ErrTransient`. The guide MUST say so,
and the `Checker` SHOULD expose a capacity figure in `Details` instead.
### 4.9 `not-found`
*How does the provider know an instance is gone?*
- **status-404**: a distinct not-found status or error code from the
backend.
- **error-code**: a code or message the backend uses for "does not exist",
matched exactly and covered by the classify table test with real captured
responses.
- **list-and-miss**: absence from a listing. Allowed only when the listing
can never be partial; otherwise a missing part reads as "gone" and §2.2 is
broken.
### 4.10 `primary-key`
*Is `req.Name` the backend's key?*
- **name-is-key**: the backend addresses instances by name. `Create` looks
the name up, the ID is `<location>/<name>`.
- **allocated-id**: the backend allocates an identifier and the name is a
label. `Create` MUST first find an existing instance by name (idempotency),
then allocate. Allocation MUST be safe against a concurrent `Create` for
another Proxy: a deterministic identifier derived from `req.UID`, or an
allocate-then-verify loop. The ID SHOULD embed the allocated key and the
name; `Get` and `Delete` use the allocated key. If instances can move
between locations (migration), the ID MUST NOT depend on the location.
### 4.11 `provisioning-state`
*Does the backend have a state for "being created"?*
- **native**: the backend reports a provisioning or staging state, mapped to
`StateProvisioning`.
- **create-blocks**: no such state; `Create` blocks until the instance is
started, so `Get` never sees "defined but not started".
- **marker**: no such state and `Create` returns early; the provider records
at creation (metadata or a task handle) what lets `Get` answer
`StateProvisioning` for an instance that was never started, and
`StateStopped` for one that stopped afterwards.
### 4.12 `async-ops`
*Does the backend run create and delete as asynchronous tasks?*
- **none**: calls complete synchronously.
- **task-handle**: the backend returns a task the provider can poll. The
provider SHOULD keep the handle where `Get` can find it (metadata or the
ID) so a failed create can be reported instead of read as NotFound.
- **backend-lock**: the backend locks the instance during a task. `Get`
returns the last known state or Transient while locked (§2.2); `Delete`
on a locked instance returns Transient, not Permanent.
### 4.13 `secondary-resources`
*What else does `Create` make, and who removes it?*
- **none**.
- **cascade**: the backend deletes them with the instance (auto-delete
flags). The provider MUST set those flags at create.
- **explicit**: the provider deletes them by name in `Delete`, tolerating
their absence. Resources that can exist without an instance (a crash
between creating them and creating the instance) are a documented GC blind
spot unless `ListByTag` covers them.
### 4.14 `credentials`
*Where do credentials and trust material come from?*
- **mounted-files**: paths in the config block, pointing into an optional
Secret mount (§7).
- **environment**: standard SDK environment discovery.
- **workload-identity**: the pod's own identity.
TLS trust for an HTTPS backend with a private CA is one of **system-roots**,
**ca-file** (a path in the config block) or **insecure** (a boolean whose
name starts with `insecure`, warned about at startup).
### 4.15 `client-kind`
*What talks to the backend?*
- **vendor-sdk**: put it behind a small interface (the seam) so a fake can
stand in.
- **thin-client**: a `net/http` client written in the provider package.
Preferred when no maintained SDK exists (`CLAUDE.md`: stdlib over
third-party). It MUST set transport timeouts, MUST carry the backend's
status and body in one error type that `classify` reads, and SHOULD wrap
its transport for child spans (§9).
- **rpc-over-ssh** or another non-HTTP transport: one pooled connection per
target, a keepalive, and a documented answer to "what does a cancelled call
do to the shared connection".
### 4.16 `timeouts`
*How is each backend call bounded?* (§1.5)
- **per-call**: every request has its own deadline, independent of `ctx`.
- **ctx-only**: the provider relies on `ctx`. Allowed only for backends
whose client already enforces a timeout; the spec MUST cite it.
### 4.17 `logging`
*Which log tiers does the provider produce?* The logger MUST come from
`logf.FromContext(ctx)` at call time (§9).
- **tiered**: V(1) one line per operation outcome and per failed backend
call; V(2) request and response detail; V(5) raw wire if there is one.
- **outcome-only**: V(1) only.
- **none**: allowed only for a backend whose client logs on its own.
### 4.18 `child-spans`
*How do backend calls appear under the operation span?* (§9)
- **sdk-instrumentation**: the SDK emits spans itself.
- **transport-wrapper**: a parent-gated `otelhttp` transport.
- **span-decorator**: a decorator over the seam that starts one span per
call.
- **none**: a documented gap, with log correlation through `traceID` only.
### 4.19 `backend-test`
*How is the provider proven against the real backend?*
- **cluster-e2e**: the kind end-to-end suite.
- **integration-tag**: a test behind `//go:build integration`, skipped
without its environment variables, with a `make test-integration-<type>`
target and a section in `docs/testing.md`.
- **manual-runbook**: a documented by-hand run. Allowed only when the
backend cannot be reached from CI or a developer machine.
### 4.20 Summary
| Axis | Profiles | Who tolerates the difference | Template entry |
|---|---|---|---|
| `readiness` | immediate, provisioning-window | reconciler polls either way | 7 |
| `create-return` | submit, blocking | reconciler; worker budget | 10 |
| `delete-return` | returns-early, waits | reconciler re-calls Delete | 11 |
| `ip-source` | assigned-at-create, discovered | reconciler gates Running on IP | 7 |
| `gc-mark` | native-labels, written-metadata, mark-window | GC | 8 |
| `created-at` | backend-timestamp, written-at-create | GC `MinAge` | 8 |
| `list-partial` | fail-whole, omit-unreachable | GC | 12 |
| `quota` | detected, absent | reconciler `QuotaRetry` | 13 |
| `not-found` | status-404, error-code, list-and-miss | reconciler | 13 |
| `primary-key` | name-is-key, allocated-id | reconciler idempotency | 2 |
| `provisioning-state` | native, create-blocks, marker | reconciler Stopped branch | 9 |
| `async-ops` | none, task-handle, backend-lock | reconciler | 10 |
| `secondary-resources` | none, cascade, explicit | GC blind spots | 11 |
| `credentials` | mounted-files, environment, workload-identity | deployment | 14 |
| `client-kind` | vendor-sdk, thin-client, rpc-over-ssh | tests, tracing | 17 |
| `timeouts` | per-call, ctx-only | worker and GC budget | 18 |
| `logging` | tiered, outcome-only, none | operators | 20 |
| `child-spans` | sdk-instrumentation, transport-wrapper, span-decorator, none | tracing | 17 |
| `backend-test` | cluster-e2e, integration-tag, manual-runbook | CI | 21 |
## 5. Error taxonomy
Every error a provider method returns **MUST be built with
`provider.Wrap(class, op, providerName, id, err)`** (`internal/provider/errors.go`).
`op` is one of `create`, `get`, `delete` or `list`, and `providerName` is the
**configured name** (`ProviderConfig.Name`), not the type.
*Caller:* the decorators label metrics and spans with the class
(`resultLabel`), and the reconciler branches on `provider.Class`.
`Wrap` keeps both `errors.Is(err, class)` and `errors.As` to the backend's
error type working (`Error.Unwrap` returns both).
| Class | Use it for | During normal reconciling | While deleting |
|---|---|---|---|
| `ErrNotFound` | the instance does not exist | Get: recreate or advance (§2.2); other ops: returned as the error, so exponential backoff | Get: release the finalizer |
| `ErrQuotaExceeded` | quota, rate limit, out of capacity | `Provisioned=False` with reason `QuotaExceeded`, requeue after `QuotaRetry`, not counted as an error (`providerFailure`) | requeue after `QuotaRetry` (`deletionFailure`) |
| `ErrTransient` | network, 5xx, timeouts, cancellation, anything unknown | error returned, so workqueue exponential backoff | same |
| `ErrPermanent` | bad config, bad argument, auth or permission denied, malformed ID | **latched**: `Provisioned=False` with reason `PermanentError`, no more provider calls until `metadata.generation` changes (`reconcileManaged`) | error returned and retried forever; a Permanent `Get` also triggers `Delete` (`reconcileDelete`) |
Pitfalls:
- **Unclassified means Transient.** `provider.Class` returns `ErrTransient`
for any error not built with `Wrap`. That is the safe default, but a
permanent failure that was never classified retries forever, and a quota
error backs off far too fast.
- **Permanent is sticky.** Use it only for errors that editing the Proxy spec
or fixing the config can resolve. A permission-denied status that the
backend also uses for quota is the hard case: read the backend's reason if
it gives one, and if it does not, pick Permanent and document it.
- **Do not return NotFound from `Create`.** The reconciler treats it as
Transient there, but it misleads the metrics and anyone reading logs.
- **NotFound detection** follows the `not-found` profile (§4.9). A backend
that reports "does not exist" as a generic server error needs the exact
message or code matched, with real captured responses in the test table.
- **Classify in one `classify` function** with a table-driven test covering
every class, including "not a backend error at all, so Transient".
## 6. Identity, names, marks, CreatedAt
- **Names:** `NameFromUID(uid)` gives `proxy-` plus 16 lowercase base32
characters. It is a valid DNS label. A backend with stricter rules SHOULD
map it deterministically rather than invent a new scheme, because
idempotency depends on the name being the same every time. A backend that
keys on an allocated identifier follows `primary-key` (§4.10).
- **providerID:** opaque to everyone except the provider. Keep it
`<location>/<name>`-shaped or `<location>/<allocated-key>`-shaped, parse
it strictly, and return `ErrPermanent` for a malformed one (§2.2).
- **Marks:** `LabelManaged` with value `LabelManagedYes`, and `LabelUID`.
Use native labels or tags where the backend has them; otherwise write them
into metadata in the create call (§4.5).
- **`CreatedAt`:** the backend's creation timestamp, or written into the mark
metadata at creation (§4.6).
- **Provider names:** a configured `name` MUST be a lowercase DNS-1123 label
of at most 63 characters (`spec.provider` in
`api/v1alpha1/proxy_types.go`). The `ProviderHealth` object is named after
the provider; `Monitor.Start` warns and `Monitor.publish` skips the object
for a name that is not a valid object name. Config loading does not check
this yet (§13).
## 7. Credentials, TLS and secret hygiene
- **The constructor MUST NOT dial the backend.** `registry.Build` aborts the
whole provider set on a constructor error, and the manager exits: one
unreachable backend must not take the other providers down. Fail in the
constructor only on config errors.
- **Credential files SHOULD be read lazily.** A mount marked `optional: true`
is empty when the Secret is missing; the constructor logs a warning and the
first call fails Transient until the Secret lands. A file that exists but
does not parse is Permanent.
- **Credentials MUST NOT appear in logs, span attributes, error strings or
`Error.Err`.** Backend errors that echo a request URL or header with a
token in it MUST be sanitised before wrapping. The same holds for
`CloudInit` (§2.1). Test both with a sentinel (§11).
- **TLS trust** follows `credentials` (§4.14). An insecure toggle MUST be
named so that it reads as insecure, default off, and MUST be logged as a
warning at startup when on.
- **Rotation.** State in the spec whether rotating the Secret takes effect
without a restart. Reading files on every dial makes rotation free; caching
them after first load needs a restart and must say so.
- **Scratch files** only under `/tmp` (§1.4), removed when the call ends.
## 8. Optional interfaces: `Checker`
`provider.Checker` is `Check(ctx) []TargetStatus` (`internal/provider/check.go`).
`cmd/main.go` type-asserts it on the **unwrapped** provider before adding the
decorators, and the monitor treats a nil Checker as unsupported, not failed
(`Monitor.run`).
- **MUST never fail as a whole.** Every failure is a `TargetStatus` with
`Ready=false`.
*Caller:* `Monitor.checkOnce` has no error path per provider.
- **MUST return promptly once `ctx` ends.** The monitor gives the whole round
one timeout.
- **MUST return one entry per target, in a stable order.** Targets are what
can fail independently: an apiserver, a project, each host or node.
*Caller:* the monitor logs transitions per target (`logTransitions`) and
the gauges are labelled `{provider,target}`.
- **MUST NOT disrupt reconciles.** A check that shares a connection with
provisioning must not tear that connection down when it times out.
- **SHOULD use the shared reasons** (`ReasonReachable`, `ReasonTimeout`,
`ReasonUnreachable`, `ReasonUnauthorized`, `ReasonForbidden`,
`ReasonAPIError`). Add provider-specific reasons only for failures an admin
would fix differently, and document each one in the guide.
- **SHOULD check with the same credentials, verb and scope that `ListByTag`
and `Create` use**, and be cheap: one page, one object.
- **MAY fill `Details`** with facts useful when triaging: versions, capacity,
the account in use.
- The `status.type` field of `ProviderHealth`
(`api/v1alpha1/providerhealth_types.go`) lists the known types in its doc
comment; add the new type there and run `make manifests`.
## 9. Observability
Tracing is OpenTelemetry, set up in `internal/tracing` from the standard
`OTEL_*` environment variables. With no exporter configured, spans are no-ops
and logs are unchanged ([architecture.md §10](architecture.md#10-tracing-internaltracing)).
**Layer 1, operation spans and metrics: nothing to write.** `cmd/main.go`
wraps every provider in `WithTracing(WithMetrics(...))`:
- `WithTracing` opens one client span per call, named `provider.create`,
`provider.get`, `provider.delete` or `provider.list`, with
`provider.name`, `provider.id` and `provider.result` (the error class) as
attributes. `ErrNotFound` keeps the span status Ok, because polling to
NotFound is expected.
- `WithMetrics` counts each call as
`proxy_operator_provider_requests_total{provider,op,result}`
(`internal/metrics/metrics.go`). A `Checker` feeds
`proxy_operator_provider_target_up`,
`proxy_operator_provider_check_duration_seconds` and
`proxy_operator_provider_check_timestamp_seconds`, labelled
`{provider,target}`.
Rules:
- **MUST NOT open its own per-operation span or count its own requests.**
They would duplicate the decorators.
- **MUST pass the caller's `ctx` to every backend call and take the logger
from `logf.FromContext(ctx)` at call time.** `tracing.StartSpan`
(`internal/tracing/logger.go`) puts a logger carrying `traceID` and
`spanID` into `ctx`. A logger cached at construction, or a backend call
made with `context.Background()`, loses the trace.
- **Health checks bypass the decorators.** A `Checker` runs under the
monitor's `providerhealth.check` span, and the same `ctx` rules apply.
**Layer 2, child spans per backend call: SHOULD.** One operation span says
that `Create` took 40 seconds; child spans say which backend call took them.
Follow `child-spans` (§4.18):
- HTTP backends: wrap the transport with `otelhttp`, gated on an existing
parent span as `tracing.RestConfigWrapper` does
(`internal/tracing/transport.go`), so background traffic (token refreshes,
keepalives) does not become root spans. Check first whether the SDK already
does this.
- Non-HTTP backends: start child spans yourself with
`tracing.Start(ctx, "<type>.<call>")` around each call and around long
steps such as a dial or a download, in a decorator over the seam, not at
call sites.
- **MUST NOT put secrets or CloudInit in span attributes.** Put identifiers,
sizes and error codes there.
**Layer 3, logs.** Follow `logging` (§4.17). Log lines join a trace through
the `traceID` key from layer 1.
## 10. Configuration and wiring
The providers file is parsed by `LoadConfig` (`internal/provider/config.go`)
with `UnmarshalStrict`, so unknown keys already fail.
- Add a `<Type>Config` struct and a `ProviderConfig` field for it.
- Add the block to `setBlocks`, so a block that does not match `Type` is
rejected.
- Add the type to the allow-list in `Config.validate`. The same list exists a
second time as the constructor map in `cmd/main.go`; both must change
(§13).
- Add required-field validation in its own `config_<type>.go` when it is
more than a few lines, with rows in `config_test.go`.
- **Validate every config value that is pasted into an identifier**: a
location name that becomes part of the providerID must not contain the
separator; a catalogue key that becomes a file or volume name must be a
safe filename.
- The constructor has the signature `registry.Constructor`:
`func(ctx, provider.ProviderConfig) (provider.Provider, error)`. Register
it in the map in `cmd/main.go`. Flags a provider needs are defined there
too and passed through a `NewWith...` variant, never read from the
environment inside the package.
- Decorators are applied by the composition root, in the order
`WithTracing(WithMetrics(p))`, after the Checker has been collected from
the unwrapped provider.
## 11. Testing obligations
`CLAUDE.md` applies: stdlib `testing`, table-driven, tests next to source,
`t.Parallel()` where safe.
- **A fake behind the seam.** Every unit test runs against an in-memory
fake of the backend, never the network.
- **Contract tests**, each against the fake:
- repeat `Create` returns the same ID and creates nothing;
- `Get` after `Delete` returns `ErrNotFound`;
- `Delete` twice succeeds;
- malformed ID is `ErrPermanent` from `Get` and `Delete`;
- an unknown backend state maps to `StateStopped`;
- `StateRunning` requires an IP, and a running instance without one is
`StateProvisioning`;
- `UID` and `CreatedAt` are read back from the fake, not remembered;
- `ListByTag` excludes unmarked instances and instances with an empty UID
are still listed with `UID == ""`;
- `Get` during an in-progress delete still returns the instance;
- `Get` during an asynchronous operation or lock returns the last state or
Transient (`async-ops`);
- a created but not yet started instance is never `StateStopped`
(`provisioning-state`);
- a cancelled `ctx` makes every method return promptly.
- **Classify**: one table test covering every class, every backend code the
provider matches, and "not a backend error at all, so Transient". Use
captured real responses for message matching.
- **Checker**: a table test for the reasons, a test that `Check` returns
promptly with an expired `ctx`, and a test that it returns one entry per
target in config order.
- **Sentinel tests**: a `CloudInit` containing a sentinel string and a
credential containing another never appear in any log line, span attribute
or error string.
- **Tracing**: with an in-memory span recorder
(`sdktrace.NewTracerProvider(sdktrace.WithSpanProcessor(tracetest.NewSpanRecorder()))`,
as in `internal/provider/tracing_test.go`), the child spans are children
of the operation span.
- **Real-backend test** per `backend-test` (§4.19). It runs the whole
lifecycle: create, create again, poll until Running, list, delete, delete
again, confirm it is gone and that no secondary resource remains. Keep it
compiling with `go vet -tags=integration ./...`.
## 12. Checklist for adding a provider
1. **Spec first.** Copy `docs/provider-spec-template.md` to
`docs/<type>-provider-spec.md` and answer every entry. Unanswered entries
block implementation.
2. **Package** `internal/provider/<type>/` with the provider type, a
compile-time assertion for `provider.Provider` and one for
`provider.Checker`, the seam interface, and a logger built from
`logf.FromContext(ctx).WithName("<type>").WithValues("provider", name)`.
3. **Config** per §10, including `config_test.go` rows.
4. **Constructor** `New(ctx, provider.ProviderConfig)` registered in
`cmd/main.go`; no dialing.
5. **Contract** per §2, tested per §11.
6. **Errors** per §5, `classify` plus its table test, `ProviderConfig.Name`
in every `Wrap`.
7. **Checker** per §8, with the `status.type` comment updated and
`make manifests` run.
8. **Tracing and logging** per §9 and §4.17, §4.18.
9. **Real-backend test** per §4.19, with a `make test-integration-<type>`
target and a section in `docs/testing.md`.
10. **RBAC**: if the provider talks to the cluster's own API, add
`+kubebuilder:rbac` markers in their own comment block, never inside a
doc comment, then `make manifests` and check
`git diff config/rbac/role.yaml`.
11. **Deployment**: an `optional: true` credentials mount in
`config/manager/manager.yaml`; a commented entry in
`config/manager/providers_config.yaml` and
`config/samples/providers-config.yaml`; a `config/samples/proxy_<type>.yaml`
whose cloud-init installs a proxy on the sample image; a Makefile helper
for the Secret if there is one; `hack/providers-dev.yaml` only if the
type needs no credentials.
12. **Docs**: `docs/<type>-provider.md` (prerequisites, config reference,
placement mapping, what lands on the backend, the health check, logging,
known gaps); a row and a section in `providers-existing.md`; a row in the
`architecture.md` components table and a block in its §5 and §11
(§10 too if the provider adds spans); a row in the README "Implemented
providers" table; a `CHANGELOG.md` entry once the user has confirmed it
works.
13. **Optional:** a demo script under `docs/demo/`.
## 13. Known limits of the contract
These are facts a new provider has to live with today. None is designed or
scheduled.
- **Adoption is not verified.** `Create` adopts any existing instance with
the right name without checking its `LabelUID`.
- **`StateStopped` and `StateTerminated` are indistinguishable** to the
reconciler; both mean delete and recreate.
- **Config is read once per process.** Adding a provider or a location, or
rotating credentials a provider caches, needs a manager restart.
- **GC has no installation identity.** Two installs sharing one backend
delete each other's instances, because GC only knows its own live UIDs.
- **An unconfigured provider latches the Proxy.** A Proxy naming a provider
missing from the config is latched `PermanentError`; adding the provider
and restarting does not change `metadata.generation`, and `spec.provider`
is immutable, so the Proxy stays latched until it is recreated.
- **Provider names are not validated** as DNS-1123 at config load.
- **The type list exists twice**: `Config.validate` and the constructor map
in `cmd/main.go`.
- **`Delete` is resubmitted on every deletion poll** until `Get` says
NotFound.
- **Callers set no deadline** (§1.5); each provider bounds itself.
- **There is no conformance suite.** The §11 contract tests are written per
provider; `internal/provider/providertest` is a proposal in the appendix.
- **Stale comments** in `internal/provider/provider.go`,
`internal/provider/registry/registry.go` and `api/v1alpha1/proxy_types.go`
still describe a `mock` provider that no longer exists.
## 14. Related documents
- [providers.md](providers.md): the index.
- [provider-spec-template.md](provider-spec-template.md): the questionnaire.
- [providers-existing.md](providers-existing.md): how the existing providers
meet this contract, and their gaps.
- [architecture.md §5](architecture.md#5-what-provider-calls-do-in-the-outside-world),
[§10](architecture.md#10-tracing-internaltracing) and
[§11](architecture.md#11-provider-health-checks-internalproviderhealth).
- [testing.md](testing.md).
docs/provider-spec-template.md
+455
View File
@@ -0,0 +1,455 @@
# Provider spec template
Copy this file to `docs/<type>-provider-spec.md`, fill in every **Answer**
line, and hand the result to whoever implements the provider together with
[provider-contract.md](provider-contract.md) and
[providers-existing.md](providers-existing.md). An unanswered entry blocks
implementation: it is a decision the contract leaves to the provider, and the
implementer must not guess it.
Where an entry names an axis in backticks, the allowed answers are the
profiles of that axis in contract §4, and the entry number matches the last
column of the contract's §4.20 table. **Evidence** is where the answer was
confirmed: a link into the backend's API documentation, or the experiment
that was run. "Verify against the target version" is a valid interim
answer only for the Evidence line, never for the Answer.
## Header
- **Type** (the `type:` value in the providers config, lowercase):
- **Example configured name**:
- **Backend and API version this spec was written against**:
- **Client** (`vendor-sdk` with the module path, or `thin-client`):
- **Author and date**:
## 1. Identity and providerID
**Question.** What is the providerID's shape, which location parts does it
embed, and what validates the config values pasted into it?
**Allowed answers.** `<location>/<name>`, `<location>/<allocated-key>`, or a
location-free `<allocated-key>` (contract §6; required when instances can
migrate, see entry 2).
**Obligations.** Strict parsing; `ErrPermanent` for a malformed ID from `Get`
and `Delete`; a regexp on every config value that becomes an ID segment, in
`config_<type>.go`.
**Answer:**
**Evidence:**
## 2. Primary key (`primary-key`)
**Question.** Is `req.Name` the backend's key, or does the backend allocate an
identifier?
**Allowed answers.** `name-is-key`; `allocated-id`.
**Obligations.** `allocated-id`: `Create` finds an existing instance by name
before allocating; allocation is safe against a concurrent `Create` for
another Proxy (deterministic from `req.UID`, or allocate-then-verify); `Get`
and `Delete` use the allocated key; state how a collision with an instance
the operator does not own is detected and reported.
**Answer:**
**Evidence:**
## 3. Placement mapping
**Question.** What do `Placement.Region`, `Zone`, `MachineType` and `Image`
mean for this backend, which are required, which are ignored, and what
happens when a value is unknown?
**Allowed answers.** Per field: a config catalogue key, a backend identifier
passed through, or ignored (contract §2.1, MAY ignore with documentation).
**Obligations.** Unknown values are rejected with `ErrPermanent` naming the
field and value before any backend call; ignored fields are listed in the
provider's guide because they are still in the spec hash; an empty value with
a single-choice catalogue may default, and the guide says so.
**Answer:**
**Evidence:**
## 4. Image source
**Question.** Where does the boot image come from, and what does a catalogue
edit do to running instances?
**Allowed answers.** A catalogue in the config block (key to URL or backend
image ID); a backend template or image referenced directly by
`Placement.Image`; a URL fetched by the provider into the backend on first
use.
**Obligations.** If the provider fetches or converts images: streaming, no
buffering (contract §1.4), a completeness marker so a cut-off fetch is redone,
concurrency control so two Creates fetch once, and a written warning that
replacing an image in use may corrupt instances that depend on it. If the
backend owns the image: the prerequisite is documented in the guide.
**Answer:**
**Evidence:**
## 5. Cloud-init delivery
**Question.** How does `CreateRequest.CloudInit` reach the guest, byte for
byte?
**Allowed answers.** User-data metadata on the instance; a NoCloud seed
built in-process and attached as media; a file placed on the backend and
referenced from the instance config; not delivered (documented, with how the
proxy gets configured instead).
**Obligations.** Size limits of the channel stated; whether vendor-data or a
second document can ride along; the sentinel test (contract §11); the channel
must be fully reachable through the API the provider uses (no step that needs
shell access on the backend unless the spec says the provider has it).
**Answer:**
**Evidence:**
## 6. Machine type mapping
**Question.** How does `Placement.MachineType` become CPU, memory and disk?
**Allowed answers.** A catalogue of named profiles in the config block; a
backend size name passed through.
**Obligations.** A catalogue needs validation of each profile; disk size
default and its config field named.
**Answer:**
**Evidence:**
## 7. Network and IP discovery (`readiness`, `ip-source`)
**Question.** Which network does the instance attach to, where does its
address come from, how long until it is usable, and what makes it reachable
from the operator pod and from clients on `spec.port`?
**Allowed answers.** `readiness`: `immediate`; `provisioning-window`.
`ip-source`: `assigned-at-create`; `discovered` (name the mechanism: guest
agent, DHCP lease, neighbour table, and the order if several).
**Obligations.** `discovered`: what the guest must run, how it gets
installed, what happens without it; discovery failure is Provisioning, not an
error; only routable IPv4 is reported (state the filter). Reachability:
firewall or routing steps the provider performs, or the prerequisite the
guide states. Whether `Port` is used by the provider at all.
**Answer:**
**Evidence:**
## 8. GC marks and CreatedAt (`gc-mark`, `created-at`)
**Question.** Where do `LabelManaged` and `LabelUID` live, where does
`CreatedAt` come from, and which secondary resources are marked?
**Allowed answers.** `gc-mark`: `native-labels`; `written-metadata`;
`mark-window`. `created-at`: `backend-timestamp`; `written-at-create`.
**Obligations.** `mark-window`: the window's length in seconds, why it is
far below `MinAge`, and how the same Proxy's next reconcile cleans an
unmarked instance. Any value-format constraint of the backend's labels
(allowed characters, length) and how a Proxy UID fits it.
**Answer:**
**Evidence:**
## 9. State mapping (`provisioning-state`)
**Question.** How does every backend state map to `StateProvisioning`,
`StateRunning`, `StateStopped`, `StateTerminated`, and how does `Get` tell
"defined but never started" from "stopped"?
**Allowed answers.** A table of backend state (plus lock or task flags) to
provider state, with the unknown-state row mapping to `StateStopped`.
`provisioning-state`: `native`; `create-blocks`; `marker`.
**Obligations.** Running requires an IP; `marker` names what is recorded at
creation and how `Get` reads it; `create-blocks` names the cap on the wait.
**Answer:**
| Backend state | Lock or task | Provider state |
|---|---|---|
| | | |
**Evidence:**
## 10. Create: blocking and task tracking (`create-return`, `async-ops`)
**Question.** What does `Create` wait for before returning, and how are
asynchronous backend tasks tracked?
**Allowed answers.** `create-return`: `submit`; `blocking`. `async-ops`:
`none`; `task-handle`; `backend-lock`.
**Obligations.** `task-handle`: where the handle is kept and how a failed
task surfaces (in `Get`, as which class, with which message). `backend-lock`:
what `Get` and `Delete` return while locked. Every step of `Create` is
idempotent so a cut-off attempt resumes; list the steps and what each finds
on a retry.
**Answer:**
**Evidence:**
## 11. Delete and secondary resources (`delete-return`, `secondary-resources`)
**Question.** In what order does `Delete` tear things down, what does it do
when called again mid-delete, and what does `Get` return meanwhile?
**Allowed answers.** `delete-return`: `returns-early`; `waits`.
`secondary-resources`: `none`; `cascade` (name the flags set at create);
`explicit` (name each resource and how it is found by name).
**Obligations.** Idempotent at every step; "already deleting" is success and
cheap; a location no longer in the config is a logged no-op success; any
resource that can outlive the instance without being listable is a documented
GC blind spot.
**Answer:**
**Evidence:**
## 12. ListByTag scope and partial failure (`list-partial`)
**Question.** What is the full scope `ListByTag` enumerates, and what happens
when part of it is unreachable?
**Allowed answers.** `fail-whole`; `omit-unreachable`.
**Obligations.** The scope covers everything the provider can create in;
server-side filtering on the mark where the backend allows it; a single
unreadable instance may be skipped with a log line; the cost of one list at
fleet size.
**Answer:**
**Evidence:**
## 13. Error classification (`not-found`, `quota`)
**Question.** Which backend responses map to which of the four classes, and
which signal means "does not exist"?
**Allowed answers.** A table of backend status or code (with message pattern
where needed) to class. `not-found`: `status-404`; `error-code`;
`list-and-miss`. `quota`: `detected`; `absent`.
**Obligations.** One `classify`, a table test with captured real responses
for every matched message; the permission-denied-versus-quota case decided;
authentication failures, host-key or certificate failures, and cancellation
each assigned a class with a reason.
**Answer:**
| Backend response | Class | Why |
|---|---|---|
| | | |
**Evidence:**
## 14. Credentials and TLS (`credentials`)
**Question.** What credential does the operator use, where does it come from,
when is it read, and how is the backend's TLS certificate trusted?
**Allowed answers.** `mounted-files` (Secret name, mount path, file names,
`optional: true`); `environment`; `workload-identity`. TLS: `system-roots`;
`ca-file`; `insecure` (the flag's exact name).
**Obligations.** The minimum privilege set the credential needs, listed;
rotation behaviour (free, or restart needed); the sentinel test for the
credential (contract §7); the constructor never dials.
**Answer:**
**Evidence:**
## 15. Config block
**Question.** What are the fields of `<Type>Config`, their defaults, and their
validation?
**Allowed answers.** A table: field, required, default, validation, and which
identifier or path the value ends up in.
**Obligations.** Contract §10: `setBlocks`, the type allow-list in
`Config.validate` and the constructor map in `cmd/main.go`, `config_<type>.go`
with its tests, regexps for values pasted into IDs or file names.
**Answer:**
| Field | Required | Default | Validation | Ends up in |
|---|---|---|---|---|
| | | | | |
**Evidence:**
## 16. Checker
**Question.** What are the check targets, what does one check do, which
reasons can it report, and what goes into `Details`?
**Allowed answers.** One target per thing that fails independently; the
shared reasons from `internal/provider/check.go` first, provider-specific
reasons only with a stated "what the admin does differently"; `Details` keys
listed.
**Obligations.** Same credential, verb and scope as `ListByTag` and `Create`;
cheap (one page, one object); never disrupts a reconcile that shares a
connection; returns promptly on `ctx`; the `status.type` comment in
`api/v1alpha1/providerhealth_types.go` updated.
**Answer:**
**Evidence:**
## 17. Client and child spans (`client-kind`, `child-spans`)
**Question.** What talks to the backend, and how do its calls appear under the
operation span?
**Allowed answers.** `client-kind`: `vendor-sdk` (module path, and whether it
is maintained); `thin-client`; `rpc-over-ssh`. `child-spans`:
`sdk-instrumentation`; `transport-wrapper`; `span-decorator`; `none`.
**Obligations.** The seam interface's name and the fake that implements it;
`thin-client`: one error type carrying status and body, transport timeouts,
parent-gated `otelhttp`; `none` is a documented gap; the span recorder test
(contract §11).
**Answer:**
**Evidence:**
## 18. Timeouts (`timeouts`)
**Question.** What bounds each backend call, the dial, and any long step such
as a download or a wait loop, and what does a cancelled `ctx` do to a shared
connection?
**Allowed answers.** `per-call` (the budgets, per operation); `ctx-only`
(cite the client's own timeout).
**Obligations.** Contract §1.5: no caller sets a deadline; a hung call pins a
worker or stalls GC. Cancellation behaviour on a shared connection stated
(abandon the call, or close the connection).
**Answer:**
**Evidence:**
## 19. Concurrency
**Question.** What is shared between concurrent calls (connections, caches,
locks), and what happens when two calls for the same instance name overlap?
**Allowed answers.** A description of the shared state and its locking.
**Obligations.** Safe for 3 reconciles, a GC sweep and a check round at once;
two `Delete` calls for one instance overlap safely; one fetch per image per
location when several Creates race.
**Answer:**
**Evidence:**
## 20. Logging (`logging`)
**Question.** Which tiers does the provider log, with which keys on every
line?
**Allowed answers.** `tiered`; `outcome-only`; `none` (cite the client's own
logging).
**Obligations.** Logger from `logf.FromContext(ctx)` on every call; never
`CloudInit` or credentials; `provider` and the instance name on every line
that concerns an instance.
**Answer:**
**Evidence:**
## 21. Tests (`backend-test`)
**Question.** What is the fake, which contract tests exist, and how is the
provider proven against the real backend?
**Allowed answers.** `backend-test`: `cluster-e2e`; `integration-tag` (the
`<TYPE>_TEST_*` variables, the `make test-integration-<type>` target); `manual-runbook`.
**Obligations.** Every item of the contract §11 list, ticked here:
- [ ] repeat `Create`
- [ ] `Get` after `Delete`
- [ ] `Delete` twice
- [ ] malformed ID
- [ ] unknown state maps to `StateStopped`
- [ ] `StateRunning` requires an IP
- [ ] `UID` and `CreatedAt` read back
- [ ] `ListByTag` excludes unmarked instances
- [ ] `Get` during an in-progress delete
- [ ] `Get` during an asynchronous operation or lock
- [ ] never started is never `StateStopped`
- [ ] cancelled `ctx` returns promptly
- [ ] `classify` table with captured responses
- [ ] Checker reasons, expired `ctx`, stable order
- [ ] sentinel tests for `CloudInit` and the credential
- [ ] child spans under the operation span
- [ ] real-backend lifecycle test
**Answer:**
**Evidence:**
## 22. Deployment
**Question.** What changes in `config/` and the Makefile?
**Allowed answers.** The list: the `optional: true` mount in
`config/manager/manager.yaml`; the commented entries in
`config/manager/providers_config.yaml` and
`config/samples/providers-config.yaml`; `config/samples/proxy_<type>.yaml`
with a cloud-init that installs a proxy on the sample image; a Makefile
helper for the Secret; RBAC markers if the provider talks to the cluster's
own API; whether `hack/providers-dev.yaml` gets an entry.
**Answer:**
**Evidence:**
## 23. Docs
**Question.** Which documents are written or updated?
**Allowed answers.** `docs/<type>-provider.md` with its section list; the row
and section in `providers-existing.md`; the `architecture.md` rows (components
table, §5, §11, and §10 if spans are added); the README row; the
`docs/testing.md` section; the `CHANGELOG.md` entry after confirmation.
**Answer:**
**Evidence:**
## 24. Known limits accepted at design time
**Question.** Which of the contract's §13 limits bite this provider in
practice, and which new gaps does this design accept?
**Allowed answers.** A list, each with the consequence for an operator and
the workaround.
**Answer:**
**Evidence:**
docs/providers-existing.md
+599
View File
@@ -0,0 +1,599 @@
# The existing providers
This is the appendix to [provider-contract.md](provider-contract.md): how the
four providers that exist today (kubernetes, gcp, kvm, kvm2) meet the
contract, which profile each one chose on every axis, and what is still
missing. It is reference material for whoever writes the next provider, not
a user guide; the user guides are [kvm-provider.md](kvm-provider.md),
[kvm2-provider.md](kvm2-provider.md),
[gcp-vm-validation.md](gcp-vm-validation.md) and
[gcp-in-specific-project.md](gcp-in-specific-project.md). The README's
"Implemented providers" table stays the short public summary. kvm2 is the
one provider built from the contract itself (clean-room, from
[kvm2-provider-spec.md](kvm2-provider-spec.md), without reading kvm's code),
so its column is also a record of what the contract produced.
The per-provider call flows are drawn in
[architecture.md §5](architecture.md#5-what-provider-calls-do-in-the-outside-world),
the health checks are tabulated in
[§11](architecture.md#11-provider-health-checks-internalproviderhealth), and the
judgment calls behind several rows below are recorded under its "Decisions"
list. They are linked, not repeated.
Code references are a file path plus a function or symbol name.
## 1. Profile table
One row per axis of contract §4, in the same order, with the profile name
and the symbol that shows it.
| Axis | kubernetes | gcp | kvm | kvm2 |
|---|---|---|---|---|
| `readiness` | immediate: a Pod is scheduled within seconds (`stateFromPod`) | provisioning-window: minutes of boot (`mapState`) | provisioning-window: boot plus, on first use, an image download (`imageCache.ensure`) | provisioning-window: boot plus, on first use, an image download (`ensureImage`) |
| `create-return` | submit: returns after `client.Create` | submit: `Insert` is never awaited, `Operation.Wait` is not called | **blocking**: waits through `DomainCreate` (`ensureRunning`), and the first image fetch runs inside (`imageCache.fetch`) | **blocking**: waits through `DomainCreate` and a 15-second poll for running (`ensureRunning`, `waitRunning`), and the first image fetch runs inside (`fetchImage`) |
| `delete-return` | returns-early: `client.Delete`, Pod terminates on its own | returns-early: `Delete` submits an operation each time | waits: destroy, undefine and volume removal complete before return (`removeDomain`, `removeVolumes`) | waits: destroy, undefine and volume removal complete before return (`removeDomain`, `removeVolumes`) |
| `ip-source` | assigned-at-create: `PodIP` (`instanceFromPod`) | assigned-at-create: the external NAT IP of the first NIC (`toInstance`) | discovered: ARP table first, then the guest agent, first routable IPv4 (`address`, `firstIPv4`) | discovered: the guest agent first, then the ARP table, first routable IPv4 (`addressSources`, `address`, `firstIPv4`) |
| `gc-mark` | native-labels: Pod labels (`buildPod`) | native-labels: instance labels (`buildInsertRequest`) | written-metadata: a namespaced `<metadata>` element written in `DomainDefineXML` (`metadata.go`) | written-metadata: an element in kvm2's own namespace written in `DomainDefineXML` (`metadataURI`, `newInstanceMetadata`) |
| `created-at` | backend-timestamp: `CreationTimestamp` | backend-timestamp: `CreationTimestamp`, parse error gives zero | written-at-create: `createdAt` in the metadata element (`newInstanceMetadata`) | written-at-create: `createdAt` in the metadata element, parse error gives zero (`newInstanceMetadata`, `createdAt`) |
| `list-partial` | n/a: one call, all namespaces | omit-unreachable: `ReturnPartialSuccess` on `AggregatedList` | fail-whole per host, one unreadable domain skipped (`ListByTag`, `managedInstance`) | fail-whole per host, one unreadable domain skipped unless ctx ended; `ID`, `UID` and `CreatedAt` from the name and the mark only, no state or address RPC (`ListByTag`) |
| `quota` | absent: a quota 403 is Permanent (`classify`) | detected: 429, and 403 with reason `quotaExceeded` or `rateLimitExceeded` (`classify`, `hasReason`) | absent (`classify`) | detected: a pool-space precheck before any volume is created (`checkPoolSpace`, `neededBytes`) |
| `not-found` | status-404: `apierrors.IsNotFound` | status-404: `isNotFound` | error-code: `ErrNoDomain`, `ErrNoStorageVol` (`hasCode`) | error-code: `ErrNoDomain`, `ErrNoStorageVol`, Transient inside Create (`classifyLibvirt`) |
| `primary-key` | name-is-key: `namespace/name` | name-is-key: `zones/<zone>/instances/<name>` | name-is-key: `hosts/<host>/domains/<name>` | name-is-key: `hosts/<host>/domains/<name>`, with a UID guard on adoption (`create`) |
| `provisioning-state` | native: `PodPending` | native: `PROVISIONING`, `STAGING`, `REPAIRING` | create-blocks: a domain is defined and started in one `Create`; `nostate` maps to Provisioning | create-blocks: defined, started and polled to running in one `Create`; `nostate` maps to Provisioning (`mapState`) |
| `async-ops` | none | task-handle, but the handle is discarded: an insert that fails after acceptance is never read | none | none |
| `secondary-resources` | none | cascade: `AutoDelete` on the boot disk (`buildInsertRequest`) | explicit: overlay and seed removed by name (`removeVolumes`); base image kept | explicit: overlay and seed removed by name (`removeVolumes`); base image kept |
| `credentials` | workload-identity: in-cluster config or kubeconfig (`NewWithTransportWrapper`) | environment: Application Default Credentials | mounted-files: SSH key and known_hosts from the optional `kvm-ssh` Secret, read lazily (`newLazySSH`) | mounted-files: the same `kvm-ssh` Secret, read on every dial and never cached (`loadCredentials`) |
| `client-kind` | vendor-sdk: controller-runtime `client.Client`, no seam interface | vendor-sdk: `cloud.google.com/go/compute` behind `instancesAPI` | rpc-over-ssh: `go-libvirt` over an SSH-forwarded unix socket, behind `libvirtAPI` (`conn.go`) | rpc-over-ssh: `go-libvirt` over an SSH-forwarded unix socket, behind the ctx-first `backend` seam (`backend.go`) |
| `timeouts` | ctx-only: no `rest.Config.Timeout` is set | ctx-only: SDK defaults, with SDK retries on Get and List | per-call for the dial only (`defaultHandshakeTimeout` in `dialSSH`); RPCs are ctx-only and a cancelled `call` closes the connection | per-call: a budget around the dial, every RPC, the uploads and the download (`defaultBudgets`, `call`); a cancelled or overlong call kills the pooled connection |
| `logging` | none | tiered: V(1) outcomes and `logAPIError`, V(2) detail, V(5) wire log (`wirelog.go`) | tiered: V(1) outcomes and one `libvirt call` line per RPC (`trace.go`), V(2) XML and addresses; no wire tier | tiered: V(1) outcomes and one `libvirt call` line per RPC (`observe.go`), V(2) XML and addresses; no wire tier |
| `child-spans` | transport-wrapper: `tracing.RestConfigWrapper()` on its own `rest.Config`, parent-gated | sdk-instrumentation: the SDK adds `otelhttp` itself; not parent-gated, but every call site has a parent | **none**: `trace.go` is a log decorator, not spans | span-decorator: `kvm2.<Method>` per RPC (`observe`), plus `kvm2.dial` (`libvirtDialer`) and `kvm2.download` (`fetchImage`) |
| `backend-test` | cluster-e2e: kind (`test/e2e`) | manual-runbook: [gcp-vm-validation.md](gcp-vm-validation.md) | integration-tag: `kvm_integration_test.go`, `make test-integration-kvm` | integration-tag: `kvm2_integration_test.go`, `make test-integration-kvm2` |
## 2. Other differences
Rows that are not profile axes but that a new provider author compares
against.
| Feature | kubernetes | gcp | kvm | kvm2 |
|---|---|---|---|---|
| Create verifies the adopted instance's UID | no | no | no | **yes**: a domain without the kvm2 mark or with a different non-empty UID is Permanent; a marked one with a matching or empty UID is adopted (`create`) |
| Placement used | none | Zone, MachineType, Image | Zone (host), MachineType, Image | Zone (host), MachineType, Image |
| CloudInit delivered | **no**: ignored | yes: `user-data` metadata | yes: NoCloud seed ISO built in-process (`buildSeedISO`) | yes: NoCloud seed ISO built in-process (`buildSeedISO`) |
| Port used | yes: Squid config and containerPort (`squidConf`) | no: firewall is manual | no | no |
| States produced | Provisioning, Running, Terminated | all four | Provisioning, Running, Stopped | Provisioning, Running, Stopped |
| Deleting instance still reports its state | yes: a Pod being deleted reports its phase | n/a | n/a | n/a: Delete waits, and a concurrent Get sees the domain until it is undefined |
| Error.Provider is the configured name | **no**: hardcoded `"kubernetes"` | yes | yes | yes (`wrapErr`) |
| Checker target | `apiserver` | the project | each host, concurrently | each host, concurrently, on a fresh connection |
| Checker Details | none | none | `user`, `pool`, `hostname`, `libvirtVersion`, `poolState`, `poolCapacity`, `poolAvailable` | `user`, `pool`, `hostname`, `libvirtVersion`, `poolState`, `poolCapacityGiB`, `poolAvailableGiB`, filled in as far as the check got |
| Credential rotation without restart | n/a | via ADC | no: cached after first successful load | yes: both files read on every dial (`loadCredentials`) |
| Connection handling | client-go | SDK client, never closed | one pooled SSH plus libvirt connection per host, 15s keepalive (`hostPool`) | one pooled SSH plus libvirt connection per host, 15s keepalive (`hostTable`); dedicated connections for the image upload and the check (`dialFresh`) |
| Unit-test double | controller-runtime fake client | `fakeAPI` behind `instancesAPI`, plus `httptest` for the real SDK client in `check_test.go` | `fakeAPI` behind `libvirtAPI`, plus an in-process `x/crypto/ssh` server in `conn_test.go` | `fakeBackend` behind `backend` (`fake_test.go`), plus an in-process `x/crypto/ssh` server (`sshserver_test.go`) and go-libvirt's own mock for the adapter |
| Proxy software | generated Squid config (`pod.go`) | from `cloudInit`; the image must run cloud-init | from `cloudInit`, plus a vendor-data script that installs the guest agent (`seed.go`) | from `cloudInit`, plus a vendor-data script that installs the guest agent (`seed.go`) |
| IP reachable from the operator | Pod network, so in-cluster only (not from `make run-dev` on a laptop) | external IP, once a firewall rule allows the cluster's egress IP | LAN address on the host's bridge; the operator needs a route to that LAN | LAN address on the host's bridge; the operator needs a route to that LAN |
| Distinct egress IP per proxy | no: the cluster's egress | yes: ephemeral external IP | whatever the host's LAN routes through; not per VM by itself | whatever the host's LAN routes through; not per VM by itself |
## 3. Observability status
| Layer | kubernetes | gcp | kvm | kvm2 |
|---|---|---|---|---|
| 1. Operation spans and metrics | yes: decorators | yes: decorators | yes: decorators | yes: decorators |
| 2. Child spans per backend call | yes: `tracing.RestConfigWrapper()`, parent-gated (`cmd/main.go`, `NewWithTransportWrapper`) | yes: the SDK's own `otelhttp`; not parent-gated | **no** | yes: `kvm2.<Method>` from the `observe` decorator, `kvm2.dial`, `kvm2.download`; always under the caller's span, since every call carries the caller's ctx |
| 3. Logs carry `traceID` and `spanID` | n/a: the package does not log | yes, including the V(5) wire log | yes: every `libvirt call` line at V(1) | yes: every `libvirt call` line at V(1), written inside its RPC's span |
| Tested against a real collector | yes: the kind e2e checks `provider.create` and `provider.delete` in Tempo (`test/e2e/tracing_test.go`), operation spans only | no | no | no; the integration test asserts the parent-child topology in an in-memory recorder (`assertLifecycleSpans`) |
## 4. kubernetes (`internal/provider/kubernetes`)
- **Package and seam.** `kubernetes.go`, `pod.go`, `check.go`. There is no
seam interface: `newWithClient` injects a controller-runtime
`client.Client`, and tests use the fake client. `New` calls
`NewWithTransportWrapper`, which builds its own uncached client from
`ctrl.GetConfig()` with an optional `transport.WrapperFunc` for tracing.
- **Config.** `KubernetesConfig.Image` only, defaulting to the Squid image
constant in `kubernetes.go`.
- **ID.** `providerID(namespace, name)` gives `namespace/name`;
`parseProviderID` uses `cache.SplitMetaNamespaceKey` and rejects an empty
namespace with `ErrPermanent`. An empty name part is not rejected.
- **Create.** `buildPod` (pure) then `client.Create`; `IsAlreadyExists` is
success. Only `Name`, `Namespace`, `UID` and `Port` are used. The Squid
config is generated by `squidConf` from the port and passed through the
`SQUID_CONF` environment variable, so there is exactly one object per
proxy.
- **Get and state.** `stateFromPod`:
| Pod phase | PodIP | Provider state |
|---|---|---|
| Pending | any | Provisioning |
| Running | empty | Provisioning |
| Running | set | Running |
| Succeeded, Failed, Unknown | any | Terminated |
`instanceFromPod` sets `IP` only when the state is Running. A Pod with a
deletion timestamp still reports its phase. `UID` from the `LabelUID`
label, `CreatedAt` from `CreationTimestamp`.
- **Delete.** `client.Delete`, `IsNotFound` ignored.
- **ListByTag.** One `client.List` across all namespaces with
`MatchingLabels{LabelManaged: LabelManagedYes}`; this is what the
cluster-wide Pod RBAC is for.
- **Errors.** `classify`: `IsNotFound` NotFound; `IsTooManyRequests`,
`IsServerTimeout`, `IsTimeout` Transient; `IsForbidden`, `IsInvalid`,
`IsBadRequest`, `IsUnauthorized` Permanent; default Transient. Nothing is
Quota, because a quota 403 and an RBAC 403 cannot be told apart.
- **Check.** Lists at most one managed Pod cluster-wide (`client.Limit(1)`),
proving read and list access, not create. Target `apiserver`, address the
rest config host. `checkReason`: context errors and timeouts
`ReasonTimeout`; `IsUnauthorized`; `IsForbidden`; any `APIStatus`
`ReasonAPIError`; else `ReasonUnreachable`.
- **Logging and tracing.** No log lines in the package. Child spans come
from the parent-gated transport wrapper.
- **Tests.** `kubernetes_test.go`, `pod_test.go`, `check_test.go` over the
fake client; the kind e2e (`test/e2e`) is the real-backend run.
- **Deployment.** RBAC markers for Pods live next to the reconciler's in
`internal/controller/proxy_controller.go`. `hack/providers-dev.yaml` holds
only this provider, for `make run-dev`.
- **Known gaps.**
- `Error.Provider` is hardcoded to `"kubernetes"`, so errors and logs show
the wrong name when the configured name differs.
- `CloudInit` and all of `Placement` are ignored.
- No quota class: a quota 403 is Permanent.
- A Pod being deleted still reports its phase, so it can read as Running.
- No securityContext, no resources, and `http_access allow all` in
`pod.go`.
- No logging in the package, and no Checker `Details`.
- No `OwnerReference` on Pods: a bypassed finalizer leaves the Pod behind.
- The Pod RBAC is always installed, even when no kubernetes provider is
configured.
## 5. gcp (`internal/provider/gcp`)
- **Package and seam.** `gcp.go`, `insert.go`, `errors.go`, `check.go`,
`wirelog.go`. The seam is `instancesAPI` (`Insert`, `Get`, `Delete`,
`AggregatedList`, `Probe`); `realInstances` adapts the SDK client and
`newWithAPI` is the test constructor. `New` calls `NewWithWireOptions`,
which takes the `--gcp-wire-log-full-payloads` flag from `cmd/main.go`.
- **Config.** `GCPConfig`: `project` required; `network`, `networkTag`,
`diskSizeGb` defaulted in `withDefaults`.
- **ID.** `formatProviderID(zone, name)` gives
`zones/<zone>/instances/<name>`; `parseProviderID` rejects anything else
with `ErrPermanent`. The project comes from the config.
- **Create.** Validates `Zone`, `MachineType` and `Image` up front and
returns `ErrPermanent` naming the missing field. `buildInsertRequest`
(pure) builds one NIC with an ephemeral external IP (`ONE_TO_ONE_NAT`) and
a boot disk with `AutoDelete`, labelled with both marks. `Insert` is
submitted and never awaited; 409 (`isAlreadyExists`) is success.
- **Get and state.** `mapState(status, ip)`:
| Instance status | NAT IP | Provider state |
|---|---|---|
| PROVISIONING, STAGING, REPAIRING | any | Provisioning |
| RUNNING | empty | Provisioning |
| RUNNING | set | Running |
| STOPPING, STOPPED, SUSPENDING, SUSPENDED | any | Stopped |
| TERMINATED | any | Terminated |
| anything else | any | Stopped |
`toInstance` reads the NAT IP of the first access config of the first
NIC, `UID` from the label, and `CreatedAt` from `CreationTimestamp` with
the parse error dropped (zero time on failure).
- **Delete.** Submits a delete operation and returns; 404 (`isNotFound`) is
success. Every deletion poll submits a new operation.
- **ListByTag.** `AggregatedList` with `managedFilter` on the managed label
only and `ReturnPartialSuccess`, so an unreachable zone is left out
silently; the zone is recovered from the URL by `lastPathSegment`.
- **Errors.** `classify`: non-`googleapi` errors Transient; 404 NotFound;
429 Quota; 403 with reason `quotaExceeded` or `rateLimitExceeded`
(`hasReason`) Quota; 400, 401 and other 403s Permanent; other codes
Transient. `logAPIError` records the HTTP status and reasons at V(1)
because `classify` collapses them.
- **Check.** `Probe` makes one `AggregatedList` page with `maxResults=1`
and the managed filter: credentials, the Compute API and
`compute.instances.list` in one call. Target is the project, address
`compute.googleapis.com`. `checkReason`: context errors `ReasonTimeout`;
non-`googleapi` errors `ReasonUnreachable`; 401; 403; else
`ReasonAPIError`.
- **Logging.** `logger(ctx)` from the context, named `gcp` with
`provider`. V(2) for the curated insert request (never the user-data),
V(1) per operation outcome, V(2) per listed instance. The wire log in
`wirelog.go` runs at V(5) with payloads elided unless the flag is set.
- **Tracing.** The SDK adds `otelhttp` itself unless telemetry is disabled;
not parent-gated.
- **Tests.** `insert_test.go` (pure builder), `errors_test.go` (code
table), `gcp_test.go` (IDs, 409, state mapping, list filter and partial
success) over `fakeAPI`; `check_test.go` also drives the real SDK client
against `httptest` to prove exactly one request with `maxResults=1`.
Real backend: the manual runbook in [gcp-vm-validation.md](gcp-vm-validation.md).
- **Deployment.** Credentials by ADC; the runbook mounts a service-account
key and points `GOOGLE_APPLICATION_CREDENTIALS` at it. A commented entry
in both providers configs, `config/samples/proxy_gcp.yaml`, and demo
scripts under `docs/demo/`.
- **Known gaps.**
- Asynchronous insert failures are never read: an insert the API accepts
but that fails later (quota, bad image) is seen only as NotFound on the
next `Get`, and the reconciler creates again, so the reason never
reaches status. [gcp-vm-validation.md](gcp-vm-validation.md#gotchas)
still says such errors do reach status.
- External IP only: a VM without an external NAT IP stays Provisioning
forever.
- No firewall management: the rule for the proxy port is a manual
prerequisite, and `req.Port` is unused.
- The SDK client is never closed, and `Placement.Region` is unused.
- Wrong label value in the docs: `gcp-vm-validation.md` filters on
`labels.proxy-operator-managed=yes`; the value is `true`.
- The sample cannot work: `config/samples/proxy_gcp.yaml` uses a
`debian-cloud` image, which has no cloud-init, so its Squid install never
runs ([gcp-in-specific-project.md](gcp-in-specific-project.md) uses
Ubuntu for that reason).
## 6. kvm (`internal/provider/kvm`)
The full user guide is [kvm-provider.md](kvm-provider.md); this section is
the author's view.
- **Package and seam.** `kvm.go` (the four methods), `conn.go` (SSH dial,
`hostPool`, `libvirtAPI`), `image.go` (`imageCache`), `seed.go` (NoCloud
ISO and vendor-data), `domainxml.go`, `volume.go`, `metadata.go`, `id.go`,
`errors.go`, `check.go`, `trace.go`. The seam is `libvirtAPI`, the twenty
go-libvirt calls the provider uses; `fake_test.go` implements it in memory
and `newWithDialer` injects the dialer and the HTTP client.
- **Config.** `KVMConfig` in `internal/provider/config_kvm.go`:
`sshKeyFile`, `knownHostsFile` or `insecureIgnoreHostKey`, `diskSizeGb`,
`vendorData`, `installGuestAgent`, `images` (`url`, `sha256`, `format`,
`firmware`), `machineTypes` (`vcpus`, `memoryMiB`), `hosts` (`name`,
`address`, `user`, `socket`, `pool`, `bridge`). `validateKVM` restricts
image keys and host names with `kvmIdentifier` because they become a volume
name and an ID segment.
- **ID.** `formatProviderID(host, name)` gives `hosts/<host>/domains/<name>`.
- **Create.** `host` resolves `Placement.Zone` (optional with one host),
the machine type must be a catalogue key, `resolveImage` accepts a
catalogue key or an http(s) URL. Then, inside one `hostPool.call`:
`DomainLookupByName`, and if found `ensureRunning`; otherwise
`imageCache.ensure` (download on first use, streamed through the operator
into the pool with a sha256 marker volume), `ensureOverlay` (reused if it
exists), `writeSeed` (always rebuilt: a cut-off upload must not be
trusted), `defineDomain` with the metadata element, `ensureRunning`.
`Create` returns only when the domain is running.
- **Get and state.** `mapState(state, ip)`:
| libvirt state | IP | Provider state |
|---|---|---|
| nostate | any | Provisioning |
| running, blocked | empty | Provisioning |
| running, blocked | set | Running |
| paused, shutdown, shutoff, crashed, pmsuspended | any | Stopped |
`address` asks the ARP table first, then the guest agent, and
`firstIPv4` drops loopback, link-local and unspecified addresses. Metadata
is read back by `metadata` and parsed by `parseInstanceMetadata`; a domain
without the element reports an empty UID.
- **Delete.** `removeDomain` (destroy, tolerating not running or gone; then
`DomainUndefineFlags` with NVRAM, managed save and snapshot metadata) and
`removeVolumes` (overlay and seed by name, a missing pool is a no-op). A
host no longer in the config is a logged no-op success.
- **ListByTag.** `ConnectListAllDomains` on every host; `managedInstance`
keeps domains whose metadata says managed and skips a domain whose
metadata or state cannot be read. One unreachable host fails the whole
list.
- **Errors.** `classify`: `isClassified` errors keep their class;
`ErrNoDomain`, `ErrNoStorageVol` NotFound; XML, unsupported,
invalid-argument, auth and access errors, and a missing pool, Permanent;
everything else Transient, including SSH handshake and auth failures.
Nothing is Quota. `logAPIError` adds `libvirtErrorFields`.
- **Check.** `checkHost` per host concurrently through `hostPool.probe`,
not `call`, so a check that runs out of time abandons its RPCs instead of
closing the connection a reconcile may be using. Two cheap RPCs
(`ConnectGetLibVersion`, `ConnectGetHostname`) then the pool must exist
and be running. Reasons: the shared ones plus `SSHMaterialUnavailable`,
`SSHHandshakeFailed`, `LibvirtConnectFailed`, `LibvirtError`,
`PoolNotFound`, `PoolNotActive`, chosen by `checkReason` from the
`dialError` stage. Details: `user`, `pool`, `hostname`, `libvirtVersion`,
`poolState`, `poolCapacity`, `poolAvailable`.
- **Connections.** `hostPool` keeps one SSH plus libvirt connection per
host, dialled lazily (`lazySSH.dial`, `sshAuth.dial`, `dialSSH`,
`connectLibvirt`), with a 15s `keepalive`. `hostEntry.acquire` is a
one-slot semaphore so waiting can end with the caller's context. A
cancelled `call` closes the shared connection to unblock the RPC;
`probe` abandons it and leaves the connection open. The SSH material is
read once by `lazySSH.load` and cached.
- **Logging.** Logger from the context, named `kvm`, with `provider`,
`host` and `name`. V(1): one line per operation outcome and one
`libvirt call` line per RPC from the `tracingAPI` decorator in `trace.go`
(rpc name, identifiers, duration, error fields). V(2): domain and volume
XML, metadata, addresses. Upload and download bodies are never logged.
- **Tracing.** No child spans; `trace.go` is a log decorator.
- **Tests.** Pure builders (`domainxml_test.go`, `volume_test.go`,
`seed_test.go`, `id_test.go`, `metadata_test.go`); `errors_test.go`,
`conn_test.go` (pool behaviour and a real in-process SSH server);
`image_test.go`, `kvm_test.go` (Create resumes after every partial step,
every Delete already-gone case, ListByTag policies, logging tiers, a
cloud-init sentinel never logged); `trace_test.go`; `check_test.go`. Real
backend: `kvm_integration_test.go` behind `//go:build integration`, driven
by `KVM_TEST_*` variables and `make test-integration-kvm`, described in
[testing.md](testing.md#the-kvm-integration-test-real-hypervisor).
- **Deployment.** The optional `kvm-ssh` Secret mounted at
`/etc/proxy-operator/kvm` (`config/manager/manager.yaml`),
`make kvm-ssh-secret`, a commented entry in both providers configs,
`config/samples/proxy_kvm.yaml`.
- **Known gaps** (see also [kvm-provider.md § Known gaps](kvm-provider.md#known-gaps)).
- No quota or capacity detection.
- `Create` blocks, including the first image download on each host.
- GC blind spots: volumes without a domain are invisible to GC, and so are
VMs on a host removed from the config, which `Delete` treats as done.
- SSH material is cached after the first successful load, so rotating
keys needs a restart.
- Cancelling one operation cuts off the others on that host: a cancelled
`call` closes the shared per-host connection.
- SSH auth failures retry forever: SSH authentication and host-key
failures are Transient, while libvirt auth errors are Permanent.
- `Create` can return `ErrNotFound` when a volume or domain disappears
during the call.
- No reused-overlay check: an existing overlay is reused without checking
its backing image or size.
- Sample mismatch: `config/samples/proxy_kvm.yaml` places on host `nas1`,
while both providers configs define only `beelink`.
- Doc drift in kvm-provider.md: it says ARP is the fallback, but the code
tries ARP first; it says `host` and `name` are on every log line, but
the check and list paths lack them.
## 7. kvm2 (`internal/provider/kvm2`)
The full user guide is [kvm2-provider.md](kvm2-provider.md) and the design
it was built from is [kvm2-provider-spec.md](kvm2-provider-spec.md); this
section is the author's view. Nothing in the package was taken from
`internal/provider/kvm`; where the two agree it is because both follow the
same libvirt.
- **Package and seam.** `kvm2.go` (`Provider`, `New`, `withDefaults`,
`host`, `wrapErr`, the per-name lock), `create.go`, `get.go`,
`delete.go`, `list.go` (the four methods), `backend.go` (the seam, the
production adapter and the budgets), `ssh.go` (credentials, the forwarded
socket, keepalive), `hosts.go` (the pooled table and the production
dialer), `observe.go` (the span and log decorator), `image.go` (cache,
download, quota precheck), `seed.go`, `xml.go`, `volume.go`,
`metadata.go`, `id.go`, `state.go`, `errors.go`, `check.go`. The seam is
`backend`: the twenty go-libvirt calls the provider uses, each with `ctx`
first (go-libvirt's own methods take none and block), plus `Close`.
`libvirtBackend` adapts `*libvirt.Libvirt` and runs every call through
`call`, which applies the budget and the abandon policy. The `dialer`
seam separates the SSH transport: `newProvider` injects one for tests,
`newLibvirtDialer` is production. The `observe` decorator is applied
once, in `libvirtDialer.dial`, so provider code never holds an
undecorated backend.
- **Config.** `ProviderConfig.KVM2`, the same `KVMConfig` type as kvm,
validated by `validateKVMBlock` with block name `kvm2` so messages read
`kvm2.hosts`. Defaults (`diskSizeGb` 10, socket, pool, image format and
firmware) are filled by `withDefaults`. `New` never dials; it checks that
the credential files exist and warns when one does not, or when
`insecureIgnoreHostKey` is on.
- **ID.** `formatProviderID(host, name)` gives `hosts/<host>/domains/<name>`;
`parseProviderID` accepts exactly four segments with the literal
`hosts` and `domains` and non-empty host and name, else `ErrPermanent`.
- **Create.** `Create` validates `Zone` (`host`; optional with one host),
`MachineType` (a catalogue key) and `Image` (`resolveImage`: a catalogue
key or an http(s) URL with qcow2 and uefi defaults) before any dial, and
logs the resolved parameters with `cloudInitBytes` at V(2). Then, under
the per-name lock (`lockName`) and on the pooled connection
(`hostTable.get`), `create`: `DomainLookupByName`; found means the
adoption guard (no kvm2 mark, or a non-empty metadata UID that differs
from `req.UID`, is Permanent) and `ensureRunning`. Otherwise
`StoragePoolLookupByName`, `imageSizeIfUncached` (marker lookup, then
`HEAD`, or a closed `GET` when `HEAD` is not 2xx) and
`checkPoolSpace` with `neededBytes` (image if uncached, plus
`diskSizeGb`, plus 16 MiB), then `ensureImage` (`cachedImage` on the
pooled connection, `fetchImage` streaming into `uploadImage` on a
`dialFresh` connection, `StoragePoolRefresh`, `writeMarker` last),
`ensureOverlay` (`ErrStorageVolExist` means reuse), `writeSeed` (delete,
build, upload; always rebuilt), two `StorageVolGetPath`, `buildDomainXML`
with file-type disks and the metadata element, `DomainDefineXML`,
`ensureRunning` (`DomainGetState`, where paused or pmsuspended is
Permanent; `DomainCreate` with `ErrOperationInvalid` as already running;
`waitRunning` polls every 500 ms for up to 15 seconds, Transient on
expiry and at once for any inactive state other than `nostate`).
`Create` returns only when the domain is running.
- **Get and state.** `mapState(state, ip)`:
| libvirt state | IP | Provider state |
|---|---|---|
| nostate | any | Provisioning |
| running, blocked | empty | Provisioning |
| running, blocked | set | Running |
| paused, shutdown, shutoff, crashed, pmsuspended, anything else | any | Stopped |
`get`: `DomainLookupByName` (only `ErrNoDomain` reaches NotFound),
`readMetadata` (`DomainGetMetadata` with `metadataURI`;
`ErrNoDomainMetadata` means unmanaged with an empty UID; an element that
does not parse is Permanent), then `DomainGetState` and, for an active
domain, `address`. `addressSources` asks the guest agent first,
then ARP; a source that fails is skipped at V(2), only ctx ending is an
error, so a cancelled Get is Transient and not a false Provisioning.
`firstIPv4` drops loopback, link-local and unspecified addresses and an
address whose type tag and literal disagree. `UID` and `CreatedAt` are
read back on every call; an unparsable `createdAt` gives the zero time.
- **Delete.** Under the per-name lock: `removeDomain` (`DomainDestroy`,
tolerating not running and gone; `DomainUndefineFlags` with NVRAM,
managed save and snapshot metadata) and `removeVolumes` (overlay and seed
by name through `deleteVolumeIfPresent`; a missing pool is a no-op). A
host no longer in the config is a logged no-op success. Nothing is left
when nil is returned.
- **ListByTag.** `ConnectListAllDomains` (active and inactive) on every
host in config order; `readMetadata` and `managed` keep kvm2's own
domains, and the `Instance` is `ID`, `UID` and `CreatedAt` from the name
and the element, with `State` and `IP` empty: GC reads nothing else
(contract §2.4), so no state or address RPC is made. A domain whose
metadata cannot be read is skipped at V(1), unless ctx ended, which
fails the call. A host that cannot be dialled or listed fails the whole
call.
- **Errors.** `classify(op, err)`: an error already carrying a class keeps
it; libvirt codes through `classifyLibvirt` (`ErrNoDomain` and
`ErrNoStorageVol` NotFound, Transient when `op == "create"`;
`ErrNoStoragePool`, `ErrXMLError`, `ErrNoSupport`, `ErrInvalidArg`,
`ErrAuthFailed`, `ErrAccessDenied` Permanent; the rest Transient);
`knownhosts.KeyError` and the `unable to authenticate` text from
`x/crypto/ssh` Permanent; everything else Transient, including a missing
credential file. `permanentf` and `transientf` classify at the source:
placement, XML builders, a parse failure of the key or known_hosts, an
image 4xx, a missing or zero `Content-Length`, a sha256 mismatch, a pool
that is not running (Permanent); an image 429 or 5xx, a short body, the
running wait expiring (Transient). `checkPoolSpace` is the only source of
`ErrQuotaExceeded`. `logAPIError` writes `kvm2 backend call failed` with
`libvirtErrorFields`.
- **Check.** `Check` runs `checkHost` per host concurrently, each through
`probeHost` on a `dialFresh` connection that is closed when the probe
ends; the pooled table is never consulted. `ConnectGetLibVersion`,
`ConnectGetHostname`, `StoragePoolLookupByName`, `StoragePoolGetInfo`;
a pool that is not running is `poolNotActiveError`. Details are filled
as facts arrive (`user` and `pool` always; `libvirtVersion`, `hostname`,
`poolState`, `poolCapacityGiB`, `poolAvailableGiB`). `checkHost` answers
`Timeout` whenever the round's ctx ended, whatever stage was running and
whatever shape the error has; otherwise `checkReason`: a `dialError` is
judged by its stage (`credentials` gives `CredentialsUnavailable`; `ssh`
with a `knownhosts.KeyError` or the auth text gives `Unauthorized`; any
other stage, including a dial that ran out of its own budget, gives
`Unreachable`); then `context.DeadlineExceeded` or `Canceled` `Timeout`,
`ErrAccessDenied` or `ErrAuthFailed` `Forbidden`, `ErrNoStoragePool`
`PoolNotFound`, any other libvirt code `APIError`, anything else
`Unreachable`; `PoolNotActive` from the pool state.
- **Connections.** `hostTable` keeps one pooled connection per host,
dialled lazily under a per-host one-slot semaphore so waiting ends with
the caller's ctx; a dead backend (`liveness`) is closed and redialled on
the next `get`. `dialSSHSocket` runs credentials, TCP, the SSH handshake
and the `direct-streamlocal` channel to the socket, each failure a
`dialError` with its stage; `libvirtDialer.connect` bounds the whole
sequence plus the libvirt `Connect` with the dial budget and starts the
`keepalive` (15 s, `keepalive@openssh.com`) and a watcher on go-libvirt's
`Disconnected`. `call` runs each blocking go-libvirt call in a goroutine
and waits on the result, ctx and a budget timer; abandoning a call
(`link.markDead`) closes the socket and the SSH client, returns Transient
wrapping the ctx or deadline error, and every other call waiting on that
link fails once with Transient. `loadCredentials` runs on every dial.
- **Logging.** Logger from the context, named `kvm2`, with `provider`,
and `host` and `name` on lines about an instance. V(1): one line per
operation outcome (`kvm2 domain ensured running`, `kvm2 domain fetched`,
`kvm2 domain deleted`, `kvm2 domains listed`, `kvm2 host checked`), one
`libvirt call` line per RPC from `observe` (rpc name, identifiers,
duration, error fields), the connection events `libvirt host connected`
and `libvirt host connect failed` with `stage`, and `kvm2 backend call
failed`. V(2): XML, metadata, addresses, listed names, pool sizes, the
resolved Create parameters. Upload and download bodies, cloud-init and
key material are never logged.
- **Tracing.** `observe.call` starts `kvm2.<Method>` per RPC with
`kvm2.host` and the identifiers (`kvm2.domain`, `kvm2.pool`,
`kvm2.volume`), sets `kvm2.libvirt_code` on failure and writes the
`libvirt call` line inside the span; `libvirtDialer.dial` starts
`kvm2.dial` (`kvm2.address`, `kvm2.user`); `fetchImage` starts
`kvm2.download` (`kvm2.url_host`, `kvm2.sha256_configured`,
`kvm2.bytes`). Both the decorator and the dialer take `tracing.Option`
so tests inject a recorder; `New` leaves them on the global provider.
- **Tests.** Pure builders (`xml_test.go`, `volume_test.go`,
`seed_test.go`, `id_test.go`, `metadata_test.go`, `state_test.go`);
`errors_test.go` (the classify table with real SSH error strings and a
real `knownhosts.KeyError`); `backend_test.go` (the adapter over
go-libvirt's own mock through a `net.Pipe`: deadline kills the
connection, cancel unblocks, killed by another caller); `hosts_test.go`;
`ssh_test.go` over `sshserver_test.go` (every dial stage, host-key
policies, key material never in errors or logs); `image_test.go`
(marker rules, streaming asserted by counting bytes, dedicated upload
connection); `create_test.go`, `get_test.go`, `delete_test.go`,
`list_test.go`, `kvm2_test.go`, `provider_test.go` (the contract §11
list, named test by test in the spec's entry 21); `observe_test.go`
(every seam method spanned and logged exactly once, by reflection);
`check_test.go` (the reason table, never touches the pool). The double is
`fakeBackend` in `fake_test.go`, whose `twin` shares state under its own
call log so a test can tell the pooled connection from a dedicated one.
Real backend: `kvm2_integration_test.go` behind `//go:build integration`,
driven by `KVM2_TEST_*` variables and `make test-integration-kvm2`,
described in
[testing.md](testing.md#the-kvm2-integration-test-real-hypervisor).
- **Deployment.** The `kvm-ssh` Secret and its mount are shared with kvm
(`config/manager/manager.yaml` says so), `make kvm-ssh-secret` is reused,
a commented `kvm2-home` entry in both providers configs,
`config/samples/proxy_kvm2.yaml` (zone `beelink`, matching the shipped
config), the `kvm2` type in the `status.type` comment of
`api/v1alpha1/providerhealth_types.go`. No RBAC.
- **Known gaps** (see also [kvm2-provider.md § Known gaps](kvm2-provider.md#known-gaps)).
- `Create` blocks, including the first image download on each host.
- GC blind spots: volumes without a domain are invisible to GC, and so are
VMs on a host removed from the config, which `Delete` treats as done.
- No adoption of kvm's domains: a switched entry needs its Proxies
deleted first.
- Cancelling one operation cuts off the others on that host once: a
cancelled `call` kills the pooled connection.
- `Content-Length` is required for image downloads.
- A paused or pmsuspended domain is not started by a repeat `Create`:
`ensureRunning` returns Permanent, so the Proxy latches until its spec
changes; the admin resumes the VM on the host.
- Credential parse errors carry the parser's own text, which can quote a
PEM header type or a malformed known_hosts token (never key bytes).
- No reused-overlay check: an existing overlay is reused without checking
its backing image or size.
- The sample cloud-init installs Squid with `http_access allow all`.
## 8. Gaps shared by the existing providers
- **Adoption is not verified in three of the four.** kubernetes, gcp and
kvm adopt any existing instance with the right name without checking
`LabelUID`; kvm2 returns `ErrPermanent` when the UID differs.
- **The Create comment does not match kvm and kvm2.** The interface comment
in `internal/provider/provider.go` says `Create` returns as soon as the
request is submitted; both libvirt providers block until the domain has
started.
- **`Stopped` and `Terminated` are produced inconsistently.** kubernetes
never produces Stopped, kvm and kvm2 never produce Terminated, and the
reconciler treats them the same.
- **Different `ListByTag` partial-failure policies.** gcp leaves failed
parts out, kvm and kvm2 fail the whole list. Both are safe.
- **Stale `mock` comments** in `internal/provider/provider.go`,
`internal/provider/registry/registry.go` and `api/v1alpha1/proxy_types.go`
(and so the generated CRD's `spec.provider` description).
- **GC does not know about installations.** Two clusters sharing one project
or hypervisor delete each other's instances. This was a merge-blocker in
the 2026-08-10 PR review
([docs/reviews](reviews/2026-08-10-1134-proxy-operator-pr-review.md)) and
is still open.
- **Tracing.** kvm has no child spans, so a kvm trace shows `provider.create`
and nothing under it; correlation works only through `traceID` on its V(1)
log lines. kvm2 has them, but only an in-memory recorder checks their
topology. Only kubernetes is tested against a real collector, and that
test checks only the operation spans; nothing checks that gcp's SDK spans,
the kubernetes API spans or kvm2's spans nest under `provider.*` in a
real collector.
## 9. Proposed features
Listed only. Nothing below is designed or implemented, except where a
bullet says kvm2 already does it.
- **Child spans in kvm:** extend the `tracingAPI` decorator in
`kvm/trace.go`, which already wraps every libvirt call, to also start a
span per RPC, and add spans for the SSH dial and the image download. The
seam's methods take no `ctx` today (`hostPool.call` hands `fn` only the
API), so the span context has to be plumbed in. kvm2's `observe.go` over
its ctx-first seam is the shape this would take.
- **Tempo e2e checks for child spans**, and for gcp, kvm and kvm2 through
their real backend runbooks.
- **A shared conformance suite:** for example `internal/provider/providertest`,
which runs the contract §11 tests against any provider plus a fake
backend, so a new provider gets the contract tests for free.
- **Verify the UID when `Create` adopts an instance.** Return `ErrPermanent`
on a mismatch instead of adopting an instance that belongs to someone else.
kvm2 does this; the other three do not.
- **Report asynchronous create failures:** gcp keeps the operation name, and
`Get` reports the failure instead of NotFound.
- **Capacity or quota signals** for kvm, such as pool free space and host
memory, mapped to `ErrQuotaExceeded`. kvm2 has the pool-space precheck;
host memory is still unchecked in both.
- **GC for secondary resources:** mark volumes and similar resources, and
let `ListByTag` or a second listing method cover them.
- **Reload credentials and config without a restart.** kvm2 re-reads its
SSH material on every dial; the providers config is still read once per
process for every type.
- **Checker `Details`** for kubernetes (server version) and gcp (quota
headroom).
- **Internal-IP mode** for gcp, and a firewall rule managed per provider.
- **An installation-identity mark**, set on create and filtered on in
`ListByTag`, so installations sharing a backend do not collect each
other's instances.
- **A permanent-failure latch that sees provider availability**, so adding
a missing provider releases the Proxies waiting on it.
- **Validate provider names at load**, and derive the config type
allow-list from the constructor map.
- **Per-provider RBAC** as an optional kustomize component, and an
`OwnerReference` on kubernetes Pods.
docs/providers.md
+41
View File
@@ -0,0 +1,41 @@
# Providers
The provider documentation is split into three files so that a new provider
can be specified and built from the generic rules alone, without reading how
the existing ones happen to work.
- **[provider-contract.md](provider-contract.md)**: what every provider MUST,
SHOULD and MAY do, which caller depends on each rule, the behaviour profiles
where providers are allowed to differ, the error taxonomy, credential
hygiene, testing obligations, the checklist for adding a provider, and the
contract's known limits. It never names an existing provider.
- **[provider-spec-template.md](provider-spec-template.md)**: the questionnaire
a new provider's design answers, one entry per decision the contract leaves
open. Copy it to `docs/<type>-provider-spec.md`, fill it in, and hand it to
the implementer with the other two files.
- **[providers-existing.md](providers-existing.md)**: how kubernetes, gcp,
kvm and kvm2 meet the contract, a table mapping each of them to every
profile, their per-provider reference notes, shared gaps and proposed
features.
Per-provider user guides: [kvm-provider.md](kvm-provider.md),
[kvm2-provider.md](kvm2-provider.md),
[gcp-vm-validation.md](gcp-vm-validation.md) and
[gcp-in-specific-project.md](gcp-in-specific-project.md).
Related sections of [architecture.md](architecture.md): the components table,
[§5 what provider calls do](architecture.md#5-what-provider-calls-do-in-the-outside-world),
[§10 tracing](architecture.md#10-tracing-internaltracing),
[§11 provider health checks](architecture.md#11-provider-health-checks-internalproviderhealth)
and the "Decisions" list. The test layout is in [testing.md](testing.md).
## Reading order for a new provider
1. The contract, end to end.
2. The template, filled in for the new backend; every unanswered entry is a
design decision still to make.
3. The appendix, for a worked example of each profile the spec chose.
4. The user guide of the existing provider closest to the new one.
The single-file blueprint that preceded this split is in the git history of
this path.
docs/testing.md
+127 -11
View File
@@ -53,18 +53,28 @@ manual runs too.
| | `config_test.go` | providers-config parsing + fail-fast validation | pure |
| | `metrics_test.go` | `WithMetrics` decorator: classified result labels, error passthrough | fake recorder + static provider |
| `internal/provider/kubernetes` | `kubernetes_test.go`, `pod_test.go` | Create/Get/Delete/ListByTag over real Pod objects; pure `buildPod`/`squidConf` incl. the `max_filedescriptors` OOM-regression assertion | `client/fake` (real `corev1.Pod`s, no kubelet) |
| | `check_test.go` | health check: a single cluster-wide list with `Limit=1` and the managed-label selector; the reason for every API error class | `client/fake` + interceptor |
| `internal/provider/gcp` | `insert_test.go`, `errors_test.go`, `gcp_test.go` | field-by-field `buildInsertRequest`; HTTP-code classification table; zone-qualified IDs, 409-is-success, 9-row state mapping, ListByTag filter + `ReturnPartialSuccess` | pure builder needs **no fake**; the rest uses the flattened `instancesAPI` seam |
| | `check_test.go` | health check: request shape and the reason per HTTP code or network error; the **real SDK client** makes exactly one request with `maxResults=1`, even when a next page exists | fake seam + `httptest` |
| `internal/provider/kvm` | `domainxml_test.go`, `volume_test.go`, `seed_test.go`, `id_test.go`, `metadata_test.go` | pure builders: domain/volume XML unmarshalled back and checked field by field, seed ISO reopened with the same library, providerID round-trip, metadata in both namespace spellings | pure |
| | `errors_test.go`, `conn_test.go` | libvirt-code classification table; connection pool (reuse, redial, drop on transport error, cancel releases a blocked RPC, one dial per host under concurrency); SSH dial, host-key policy, keepalive | fake dialer; a **real in-process `x/crypto/ssh` server** forwarding a unix socket |
| | `image_test.go`, `kvm_test.go` | image cache (fresh fetch, ready, marker missing, checksum change/mismatch, 4xx/5xx, concurrent callers); Create resume after every partial step, leftover truncated seed, state/IP mapping, every Delete already-gone case, ListByTag across hosts / one down / unreadable domain; logging tiers (V0 silent, V1 one line per RPC, V2 XML) with cloud-init never logged | `httptest` image server + in-memory `fakeAPI` behind the `libvirtAPI` seam |
| | `trace_test.go`, `conn_test.go`, `errors_test.go` | every `libvirtAPI` method traced exactly once (guarded by reflection on the interface), results and errors pass through, libvirt error fields, detail keys only at V2, upload content never logged; pool connect/drop lines, no wrapping below V1 | `fakeAPI` + `funcr` capture |
| | `kvm_integration_test.go` (`-tags=integration`) | one real VM: image fetch, Create, idempotent re-Create, Running with IP, ListByTag, Delete, pool state afterwards | a **real hypervisor** — see below |
| | `check_test.go`, `conn_test.go` | health check: details, per-host verdicts in config order, hosts checked concurrently, a timeout that keeps the connection and refuses a second check while the first still waits, no waiting behind a reconcile's dial, the reason table; `probe` abandons without closing, but still drops on a transport error; waiting for a host's slot ends with the caller's deadline; an inactive or missing pool and daemon errors keep the connection; every dial stage tagged through the real SSH server, with classification unchanged | `fakeAPI`, fake dialers, in-process SSH server |
| | `kvm_integration_test.go` (`-tags=integration`) | one real VM: image fetch, Create, idempotent re-Create, Running with IP, ListByTag, Delete, pool state afterwards; `TestKVMCheck_realHost` runs the health check against the host and creates nothing | a **real hypervisor** — see below |
| `internal/provider/kvm2` | `xml_test.go`, `volume_test.go`, `seed_test.go`, `id_test.go`, `metadata_test.go`, `state_test.go` | pure builders: domain XML (file disks with resolved paths, UEFI with secure boot off, BIOS escape, the metadata element, the guest-agent channel), volume XML with and without a backing store, seed ISO reopened (files, label, user-data byte for byte, temp dir removed) and the vendor-data script vs. multipart rules, providerID round-trip and malformed forms, metadata in both namespace spellings, state mapping and the IPv4 filter | pure |
| | `errors_test.go`, `backend_test.go`, `hosts_test.go`, `ssh_test.go` | classification table (libvirt codes, SSH auth text, `knownhosts.KeyError`, missing files); the per-call budget adapter (deadline kills the connection and is Transient, cancel unblocks, killed by another caller, close idempotent) against a mocked libvirt socket; the host table (one dial per host under concurrency, redial after drop, `dialFresh` never cached, waiting ends with ctx); the real dialer end to end: socket forwarding, unknown and mismatched host keys, insecure mode, missing vs. unparsable key and known_hosts, wrong key, handshake timeout, cancel, every stage tagged, key material never in errors or logs | fake dialer; a **real in-process `x/crypto/ssh` server** forwarding a unix socket |
| | `image_test.go`, `create_test.go`, `get_test.go`, `delete_test.go`, `list_test.go`, `kvm2_test.go`, `provider_test.go` | image cache (fetch with the marker written last, cached, marker missing or without a digest, sha256 change or mismatch, 4xx/5xx, no Content-Length, short read, concurrent callers fetch once, cancel mid-stream, streamed without buffering, upload on a dedicated connection) and the pool-space quota precheck; Create resume after every partial step, returns only when running, adoption guard on UID, seed always rebuilt, cloud-init sentinel never logged; Get state and address mapping incl. agent-before-ARP; every Delete already-gone case, NVRAM flag, per-name lock; ListByTag across hosts, one down fails the whole call, unreadable domain skipped; `New` never dials and warns on missing files; every error wrapped with the configured name; logging tiers V0/V1/V2 | `httptest` image server + in-memory `fakeBackend` behind the `backend` seam |
| | `observe_test.go`, `check_test.go` | every seam method traced and logged exactly once (guarded by reflection), child spans under the parent, detail only at V2, upload bodies never logged, error fields; health check: one entry per host in config order, hosts checked concurrently, expired ctx returns promptly, never touches the pooled connection, the reason table, detail keys | `fakeBackend` + `funcr` capture + span recorder |
| | `kvm2_integration_test.go` (`-tags=integration`) | one real VM: `New` without a dial, `Check` Ready with details, Create, idempotent re-Create with an unchanged domain count, Running with IP and the metadata round-trip, ListByTag, Delete, NotFound, pool state afterwards, `kvm2.*` spans under `provider.create`; `TestKVM2Check_realHost` runs the health check against the host and creates nothing | a **real hypervisor** — see below |
| `internal/health` | `probe_test.go` | probes through a **real CONNECT-capable proxy stub** to a real TLS server: tunnel success, refused CONNECT, wrong status, dead proxy, plain-http forward | `httptest` + hijacked bidirectional tunnel |
| | `engine_test.go` | thresholds, first-verdict, latency suppression matrix, dropped-event retry, stale-probe UID guard, tick scheduling/seeding/pruning, `Start` end-to-end | fake `client.Reader` + injected `probeFn` |
| `internal/lease` | `store_test.go` | capacity, `MaxLeases=0`, all three selection tie-breaks, target-scoped vs global cooldowns, expiry via fake clock, report-on-retained-lease, idempotent release, **40 concurrent acquires never exceeding capacity** | injectable clock; the concurrency case is why `-race` matters |
| `internal/discovery` | `server_test.go` | auth matrix (incl. `/healthz` exemption), list filtering, grant shape, full 409 arithmetic, invalid TTL/body/result, idempotent release, report→cooldown→409 round trip, `Start`/shutdown | `httptest` over the real handler chain, fake cache reader, **real** `lease.Store` |
| `internal/gc` | `gc_test.go` | the true-orphan matrix (live/deleting/young/unlabelled all kept), broken-provider isolation, list-failure skips sweep, namespace guard, sweep loop | fake reader + canned-list provider |
| `internal/metrics` | `metrics_test.go` | scrape-time collectors (`GatherAndCompare`), per-proxy series cleanup via `ForgetProxy`, label counts, fresh-registry-per-test property | `prometheus/client_golang/testutil` |
| `internal/providerhealth` | `status_test.go`, `monitor_test.go` | `buildStatus` (the Ready condition, per-target transition times, truncation); objects created with the label and patched, the round timeout, no-checker Unknown, invalid names, transition-only logging, cleanup of labelled unconfigured objects, the checks-off path, `ValidateSchedule` | fake client with a cluster-scoped RESTMapper + `funcr` capture |
| | `envtest_test.go` | the generated `ProviderHealth` schema against a real API server: cluster scope, status subresource, required fields, list-map keys, the 1024-character message limit | envtest (skipped under `-short`) |
| `internal/metrics` | `metrics_test.go` | scrape-time collectors (`GatherAndCompare`) incl. the provider-target gauges, per-proxy series cleanup via `ForgetProxy`, label counts, fresh-registry-per-test property | `prometheus/client_golang/testutil` |
Conventions (from `CLAUDE.md`): stdlib `testing`, table-driven by default,
`t.Parallel()` where safe, tests next to source, ginkgo only in the
@@ -88,12 +98,12 @@ envtest suite the scaffold generated.
bug (below).
- **Live GCP**: no test talks to Google. The seam boundary
(`buildInsertRequest` + classification) is tested exhaustively instead.
- **Live libvirt, by default**: the KVM unit tests run over an in-memory
fake of the 17 libvirt calls the provider uses. Whether libvirt accepts
the domain XML, whether a raw qcow2 upload plus pool refresh yields a
usable backing image, and whether the guest actually boots and reports
an address are provable only on a real host — that is the opt-in
integration test below.
- **Live libvirt, by default**: the KVM and kvm2 unit tests run over an
in-memory fake of the 20 libvirt calls each provider uses. Whether
libvirt accepts the domain XML, whether a raw qcow2 upload plus pool
refresh yields a usable backing image, and whether the guest actually
boots and reports an address are provable only on a real host — that is
the opt-in integration test below (one per provider).
## The KVM integration test (real hypervisor)
@@ -109,13 +119,19 @@ had.
export KVM_TEST_HOST=nas1.home.lan:22 # SSH endpoint
export KVM_TEST_USER=virt # must be in the libvirt group
export KVM_TEST_SSH_KEY=$HOME/.ssh/id_ed25519 # private key for that user
export KVM_TEST_KNOWN_HOSTS=$HOME/.ssh/kvm_known_hosts # the hypervisors' host keys
export KVM_TEST_BRIDGE=br0 # bridge to a network with DHCP
export KVM_TEST_IMAGE_URL=https://cloud.debian.org/images/cloud/trixie/latest/debian-13-generic-amd64.qcow2
make test-integration-kvm
```
Optional: `KVM_TEST_KNOWN_HOSTS` (without it host keys are **not**
verified — the log says so), `KVM_TEST_IMAGE_SHA256`, `KVM_TEST_POOL`
`KVM_TEST_KNOWN_HOSTS` is technically optional, but leaving it out turns
host-key verification **off** for the run (the log says so); keep a
`kvm_known_hosts` holding just the hypervisors rather than pointing the
test at your everyday `known_hosts`, so a re-keyed hypervisor is a
one-line fix in a file nothing else uses.
Also optional: `KVM_TEST_IMAGE_SHA256`, `KVM_TEST_POOL`
(default `default`), `KVM_TEST_SOCKET`, `KVM_TEST_TIMEOUT` (default
`20m`, must cover the first-time image download), `KVM_TEST_KEEP`
to leave the VM behind for `virsh console` when something goes wrong,
@@ -139,13 +155,113 @@ but no per-VM overlay or seed. It also tries a TCP connect to port 22 on
the guest, but only logs the outcome — the machine running the test may
not route to the bridge network.
`TestKVMCheck_realHost` runs the provider health check
([kvm-provider.md](kvm-provider.md#health-check)) against the same host.
It must find the host reachable and the pool running, and it creates
nothing. It needs the same variables; to run only this test:
```sh
go test -tags=integration -count=1 -v -run TestKVMCheck ./internal/provider/kvm/
```
The test is not part of `make test` and not run in CI. Keep it compiling
with `go vet -tags=integration ./...` when the provider's internals move.
## The kvm2 integration test (real hypervisor)
`internal/provider/kvm2/kvm2_integration_test.go` compiles only under
`-tags=integration` and skips itself unless the environment names a
host. Like the kvm one it deviates from the "testcontainers for real
infra" convention on purpose: libvirt needs `/dev/kvm` and a bridge to a
DHCP network, and a container has neither. It targets whichever
hypervisor you point it at and leaves that host's pool with one more
cached base image (`base-test-image.qcow2` and its `.sha256` marker) than
it had. The kvm and kvm2 tests can share a host: kvm2 marks its domains
in its own metadata namespace and never lists or adopts the other's.
```sh
export KVM2_TEST_HOST=beelink:22 # SSH endpoint
export KVM2_TEST_USER=novakj # must be in the libvirt group
export KVM2_TEST_SSH_KEY=$HOME/.ssh/id_ed25519 # private key for that user
export KVM2_TEST_KNOWN_HOSTS=$HOME/.ssh/kvm_known_hosts # the hypervisors' host keys
export KVM2_TEST_BRIDGE=br0 # bridge to a network with DHCP
export KVM2_TEST_IMAGE_URL=https://cloud.debian.org/images/cloud/trixie/latest/debian-13-generic-amd64.qcow2
make test-integration-kvm2
```
`KVM2_TEST_KNOWN_HOSTS` is technically optional, but leaving it out turns
host-key verification **off** for the run (the log says so); keep a
`kvm_known_hosts` holding just the hypervisors rather than pointing the
test at your everyday `known_hosts`, so a re-keyed hypervisor is a
one-line fix in a file nothing else uses.
Also optional: `KVM2_TEST_IMAGE_SHA256` (verified after download and
compared with the cached marker), `KVM2_TEST_POOL` (default `default`),
`KVM2_TEST_SOCKET` (default `/var/run/libvirt/libvirt-sock`),
`KVM2_TEST_TIMEOUT` (default `20m`, the budget for the whole lifecycle;
it must cover the first-time image download), `KVM2_TEST_KEEP` to leave
the VM behind for `virsh console` when something goes wrong (the log then
prints the `virsh undefine --nvram` and `virsh vol-delete` lines that
clean it up), and `KVM2_TEST_LOG_LEVEL` (default `1`) for how much of the
provider's logging the `-v` output shows: at `1` every libvirt RPC is one
compact `libvirt call` line (`HH:MM:SS.mmm kvm2 "level"=1 "msg"="libvirt
call" "rpc"="DomainLookupByName" "domain"=... "duration"=...`, plus the
per-operation outcome lines and connection events), interleaved with the
test's own `t.Log` lines; `2` adds the domain and volume XML, metadata
and addresses; `0` leaves only `t.Log`. The per-run constants (provider,
host, domain name) are stated once in the first line and left off every
provider line. See the Logging section of
[kvm2-provider.md](kvm2-provider.md).
What one run asserts, in order, each step logged with its duration:
1. `New` returns within 100 ms and without dialling (no `libvirt host
connected` line before the first call).
2. `Check` reports the one host `Ready` with reason `Reachable` and the
`hostname`, `libvirtVersion`, `poolState` (`running`) and
`poolAvailableGiB` details, which are logged.
3. Every later call goes through `provider.WithTracing` with an in-memory
span recorder that the package's own spans also land in.
4. `Create` returns `hosts/kvm2-test/domains/<name>` (downloading the
image on a cold pool — the log prints how long it took).
5. A second `Create` returns the same ID and the host's domain count,
read over a dedicated connection, is unchanged.
6. `Get`, polled every 5 s, reaches `Running` with an IPv4 within the
budget; its `UID` is the one Create wrote and `CreatedAt` is non-zero
and within a few minutes of now. The time from Create's return to the
first address is logged.
7. `ListByTag` contains the ID with that UID.
8. A TCP connect to port 22 on the guest, logged only — the machine
running the test may not route to the bridge network.
9. `Delete` returns nil, `Get` is `ErrNotFound`, a second `Delete` is a
no-op.
10. The pool, inspected over a dedicated connection, holds the base image
and its marker but no `<name>.qcow2` or `<name>-seed.iso`.
11. Among the recorded spans, a `provider.create` span has at least one
`kvm2.DomainDefineXML` and one `kvm2.dial` child, and no span
attribute carries the cloud-init text.
With `KVM2_TEST_KEEP` steps 9 and 10 are skipped; otherwise a `t.Cleanup`
attempts the Delete even when an earlier assertion failed.
`TestKVM2Check_realHost` runs the provider health check
([kvm2-provider.md](kvm2-provider.md)) against the same host. It must
find the host reachable and the pool running, and it creates nothing. It
needs the same variables; to run only this test:
```sh
go test -tags=integration -count=1 -v -run TestKVM2Check ./internal/provider/kvm2/
```
The test is not part of `make test` and not run in CI. Keep it compiling
with `go vet -tags=integration ./...` when the provider's internals move.
## The scaffolded `make test-e2e` suite
`test/e2e/` is the kubebuilder-generated smoke suite (build image → kind
cluster → deploy → assert the manager pod runs and serves metrics). It
cluster → deploy → assert the manager pod runs, publishes a Ready
`ProviderHealth` for the `kubernetes` provider, and serves metrics that
include that provider's `proxy_operator_provider_target_up` series). It
compiles only under `-tags=e2e`, manages its own kind cluster
(`make test-e2e` / `make cleanup-test-e2e`), and has been kept compiling
(`go vet -tags=e2e ./...` is part of the routine) but is **not part of
go.mod
+1 -1
View File
@@ -23,6 +23,7 @@ require (
k8s.io/api v0.36.0
k8s.io/apimachinery v0.36.0
k8s.io/client-go v0.36.0
k8s.io/klog/v2 v2.140.0
sigs.k8s.io/controller-runtime v0.24.1
sigs.k8s.io/yaml v1.6.0
)
@@ -104,7 +105,6 @@ require (
k8s.io/apiextensions-apiserver v0.36.0 // indirect
k8s.io/apiserver v0.36.0 // indirect
k8s.io/component-base v0.36.0 // indirect
k8s.io/klog/v2 v2.140.0 // indirect
k8s.io/kube-openapi v0.0.0-20260317180543-43fb72c5454a // indirect
k8s.io/streaming v0.36.0 // indirect
k8s.io/utils v0.0.0-20260210185600-b8788abfbbc2 // indirect
internal/discovery/server.go
+5
View File
@@ -9,6 +9,7 @@ import (
"context"
"crypto/subtle"
"encoding/json"
"log/slog"
"net"
"net/http"
"strings"
@@ -112,6 +113,10 @@ func (s *Server) Start(ctx context.Context) error {
srv := &http.Server{
Handler: s.handler(),
// Without this, net/http's own faults (a failed Accept, a panic that
// escapes a handler) go to the stdlib logger on stderr, outside the
// structured stream.
ErrorLog: slog.NewLogLogger(logr.ToSlogHandler(s.log), slog.LevelError),
ReadHeaderTimeout: readHeadTimeout,
ReadTimeout: 10 * time.Second,
WriteTimeout: 10 * time.Second,
internal/metrics/metrics.go
+48
View File
@@ -117,3 +117,51 @@ func (c *constCollector) Collect(ch chan<- prometheus.Metric) {
ch <- prometheus.MustNewConstMetric(c.desc, prometheus.GaugeValue, float64(value), label)
}
}
// ProviderTarget is one provider health-check target as the gauges see it.
type ProviderTarget struct {
Provider, Target string
Up bool
Duration time.Duration
CheckedAt time.Time
}
// RegisterProviderTargets registers the provider-check gauges. Like the
// other scrape-time collectors they are read from the source at every
// scrape, so a target that goes away stops being reported rather than
// leaving a stale series behind.
func RegisterProviderTargets(reg prometheus.Registerer, read func() []ProviderTarget) error {
labels := []string{"provider", "target"}
return reg.Register(&providerTargetCollector{
up: prometheus.NewDesc("proxy_operator_provider_target_up",
"Whether the target passed its latest provider health check (1) or not (0).", labels, nil),
duration: prometheus.NewDesc("proxy_operator_provider_check_duration_seconds",
"Duration of the target's latest provider health check.", labels, nil),
checked: prometheus.NewDesc("proxy_operator_provider_check_timestamp_seconds",
"Unix time of the target's latest provider health check.", labels, nil),
read: read,
})
}
type providerTargetCollector struct {
up, duration, checked *prometheus.Desc
read func() []ProviderTarget
}
func (c *providerTargetCollector) Describe(ch chan<- *prometheus.Desc) {
ch <- c.up
ch <- c.duration
ch <- c.checked
}
func (c *providerTargetCollector) Collect(ch chan<- prometheus.Metric) {
for _, t := range c.read() {
up := 0.0
if t.Up {
up = 1
}
ch <- prometheus.MustNewConstMetric(c.up, prometheus.GaugeValue, up, t.Provider, t.Target)
ch <- prometheus.MustNewConstMetric(c.duration, prometheus.GaugeValue, t.Duration.Seconds(), t.Provider, t.Target)
ch <- prometheus.MustNewConstMetric(c.checked, prometheus.GaugeValue, float64(t.CheckedAt.UnixNano())/1e9, t.Provider, t.Target)
}
}
internal/metrics/metrics_test.go
+42
View File
@@ -111,3 +111,45 @@ func TestRegister_freshRegistryPerTest(t *testing.T) {
register(t, m1, nil, 0)
register(t, m2, nil, 0)
}
func TestRegisterProviderTargets_scrape(t *testing.T) {
t.Parallel()
checked := time.Unix(1757602800, 0)
targets := []ProviderTarget{
{Provider: "kvm-home", Target: "nas1", Up: true, Duration: 84 * time.Millisecond, CheckedAt: checked},
{Provider: "kvm-home", Target: "nas2", Up: false, Duration: 15 * time.Second, CheckedAt: checked},
}
reg := prometheus.NewRegistry()
if err := RegisterProviderTargets(reg, func() []ProviderTarget { return targets }); err != nil {
t.Fatalf("RegisterProviderTargets: %v", err)
}
expected := `
# HELP proxy_operator_provider_check_duration_seconds Duration of the target's latest provider health check.
# TYPE proxy_operator_provider_check_duration_seconds gauge
proxy_operator_provider_check_duration_seconds{provider="kvm-home",target="nas1"} 0.084
proxy_operator_provider_check_duration_seconds{provider="kvm-home",target="nas2"} 15
# HELP proxy_operator_provider_check_timestamp_seconds Unix time of the target's latest provider health check.
# TYPE proxy_operator_provider_check_timestamp_seconds gauge
proxy_operator_provider_check_timestamp_seconds{provider="kvm-home",target="nas1"} 1.7576028e+09
proxy_operator_provider_check_timestamp_seconds{provider="kvm-home",target="nas2"} 1.7576028e+09
# HELP proxy_operator_provider_target_up Whether the target passed its latest provider health check (1) or not (0).
# TYPE proxy_operator_provider_target_up gauge
proxy_operator_provider_target_up{provider="kvm-home",target="nas1"} 1
proxy_operator_provider_target_up{provider="kvm-home",target="nas2"} 0
`
if err := testutil.GatherAndCompare(reg, strings.NewReader(expected)); err != nil {
t.Error(err)
}
}
func TestRegisterProviderTargets_emptySnapshotHasNoSeries(t *testing.T) {
t.Parallel()
reg := prometheus.NewRegistry()
if err := RegisterProviderTargets(reg, func() []ProviderTarget { return nil }); err != nil {
t.Fatalf("RegisterProviderTargets: %v", err)
}
if n, err := testutil.GatherAndCount(reg); err != nil || n != 0 {
t.Errorf("series before the first check = %d (%v), want 0", n, err)
}
}
internal/provider/check.go
+46
View File
@@ -0,0 +1,46 @@
package provider
import (
"context"
"time"
)
// Reasons shared by every Checker, so a listing reads the same across
// providers. Providers add their own for failures only they can tell
// apart (KVM's SSHHandshakeFailed or PoolNotActive, say).
const (
ReasonReachable = "Reachable"
ReasonTimeout = "Timeout"
ReasonUnreachable = "Unreachable"
ReasonUnauthorized = "Unauthorized"
ReasonForbidden = "Forbidden"
ReasonAPIError = "APIError"
)
// TargetStatus is one target's verdict from a Checker.
type TargetStatus struct {
// Name identifies the target within its provider: a KVM host name,
// "apiserver", a GCP project.
Name string
// Address is what the check connected to, for humans.
Address string
Ready bool
// Reason is ReasonReachable when Ready, otherwise why not.
Reason string
// Message is human-readable detail, usually the error.
Message string
Latency time.Duration
// Details are provider-specific facts gathered by the check.
Details map[string]string
}
// Checker is the optional interface of a provider that can verify its
// backend is reachable and usable without touching any instance. The
// provider health monitor type-asserts for it on the unwrapped providers,
// so it stays out of the Provider contract.
type Checker interface {
// Check returns one verdict per target, in a stable order. It never
// fails as a whole — a failure is a verdict — and must return promptly
// once ctx ends.
Check(ctx context.Context) []TargetStatus
}
internal/provider/config.go
+11 -1
View File
@@ -22,6 +22,9 @@ type ProviderConfig struct {
Kubernetes *KubernetesConfig `json:"kubernetes,omitempty"`
GCP *GCPConfig `json:"gcp,omitempty"`
KVM *KVMConfig `json:"kvm,omitempty"`
// KVM2 is the same block shape as KVM, for the kvm2 provider type: one
// or more libvirt/KVM hypervisors reached over SSH.
KVM2 *KVMConfig `json:"kvm2,omitempty"`
}
// setBlocks names the type-specific blocks that are non-nil, so validate can
@@ -37,6 +40,9 @@ func (p ProviderConfig) setBlocks() []string {
if p.KVM != nil {
out = append(out, "kvm")
}
if p.KVM2 != nil {
out = append(out, "kvm2")
}
return out
}
@@ -100,7 +106,7 @@ func (c *Config) validate() error {
prefix := fmt.Sprintf("providers[%d] %q", i, p.Name)
switch p.Type {
case "kubernetes", "gcp", "kvm":
case "kubernetes", "gcp", "kvm", "kvm2":
case "":
return fmt.Errorf("%s: type is required", prefix)
default:
@@ -120,6 +126,10 @@ func (c *Config) validate() error {
if err := validateKVM(prefix, p.KVM); err != nil {
return err
}
case "kvm2":
if err := validateKVMBlock(prefix, "kvm2", p.KVM2); err != nil {
return err
}
}
}
return nil
internal/provider/config_kvm.go
+26 -13
View File
@@ -108,29 +108,42 @@ var sha256Hex = regexp.MustCompile(`^[0-9a-fA-F]{64}$`)
// kvmIdentifier constrains the config names that are pasted into
// identifiers elsewhere: a host name goes into the providerID
// hosts/<name>/domains/<vm>, which is split on "/" to parse, and an image
// key becomes the file name of the cached base volume. A slash in either
// produces an ID or a path that never resolves again, so the VM leaks with
// nothing able to address it.
// key becomes part of the file name of the cached base volume. A slash in
// either produces an ID or a path that never resolves again, so the VM
// leaks with nothing able to address it.
var kvmIdentifier = regexp.MustCompile(`^[A-Za-z0-9][A-Za-z0-9_.-]*$`)
// IsKVMIdentifier reports whether s is safe to paste into a providerID or a
// volume name. The provider entry's own name reaches a volume name too, and
// that name is validated for every provider type at once (only non-empty
// and unique), so the kvm2 provider checks its own at construction.
func IsKVMIdentifier(s string) bool { return kvmIdentifier.MatchString(s) }
func validateKVM(prefix string, c *KVMConfig) error {
return validateKVMBlock(prefix, "kvm", c)
}
// validateKVMBlock checks a KVMConfig under the given block name, so the
// kvm and kvm2 provider types share one validator and each reports its own
// block in messages.
func validateKVMBlock(prefix, block string, c *KVMConfig) error {
if c == nil {
return fmt.Errorf("%s: kvm block is required", prefix)
return fmt.Errorf("%s: %s block is required", prefix, block)
}
if c.SSHKeyFile == "" {
return fmt.Errorf("%s: kvm.sshKeyFile is required", prefix)
return fmt.Errorf("%s: %s.sshKeyFile is required", prefix, block)
}
if c.KnownHostsFile == "" && !c.InsecureIgnoreHostKey {
return fmt.Errorf("%s: kvm.knownHostsFile is required unless kvm.insecureIgnoreHostKey is true", prefix)
return fmt.Errorf("%s: %s.knownHostsFile is required unless %s.insecureIgnoreHostKey is true", prefix, block, block)
}
if c.DiskSizeGB < 0 {
return fmt.Errorf("%s: kvm.diskSizeGb must not be negative", prefix)
return fmt.Errorf("%s: %s.diskSizeGb must not be negative", prefix, block)
}
for name, img := range c.Images {
p := fmt.Sprintf("%s: kvm.images[%q]", prefix, name)
p := fmt.Sprintf("%s: %s.images[%q]", prefix, block, name)
if !kvmIdentifier.MatchString(name) {
return fmt.Errorf("%s: key must be letters, digits, dots, dashes or underscores"+
" — it becomes the cached volume name base-<key>.<format>", p)
" — it becomes part of the cached base volume's name", p)
}
if !IsHTTPURL(img.URL) {
return fmt.Errorf("%s.url must be an http(s) URL (got %q)", p, img.URL)
@@ -146,19 +159,19 @@ func validateKVM(prefix string, c *KVMConfig) error {
}
}
if len(c.MachineTypes) == 0 {
return fmt.Errorf("%s: kvm.machineTypes must define at least one profile", prefix)
return fmt.Errorf("%s: %s.machineTypes must define at least one profile", prefix, block)
}
for name, mt := range c.MachineTypes {
if mt.VCPUs <= 0 || mt.MemoryMiB <= 0 {
return fmt.Errorf("%s: kvm.machineTypes[%q] needs vcpus > 0 and memoryMiB > 0", prefix, name)
return fmt.Errorf("%s: %s.machineTypes[%q] needs vcpus > 0 and memoryMiB > 0", prefix, block, name)
}
}
if len(c.Hosts) == 0 {
return fmt.Errorf("%s: kvm.hosts must list at least one host", prefix)
return fmt.Errorf("%s: %s.hosts must list at least one host", prefix, block)
}
seen := make(map[string]bool, len(c.Hosts))
for i, h := range c.Hosts {
p := fmt.Sprintf("%s: kvm.hosts[%d]", prefix, i)
p := fmt.Sprintf("%s: %s.hosts[%d]", prefix, block, i)
if h.Name == "" {
return fmt.Errorf("%s: name is required", p)
}
internal/provider/config_kvm_test.go
+94
View File
@@ -214,6 +214,100 @@ providers:
}
}
func TestLoadConfig_kvm2Valid(t *testing.T) {
t.Parallel()
yaml := strings.Replace(validKVMYAML, "type: kvm\n kvm:\n", "type: kvm2\n kvm2:\n", 1)
cfg, err := LoadConfig([]byte(yaml))
if err != nil {
t.Fatalf("LoadConfig() error = %v, want nil", err)
}
if cfg.Providers[0].KVM != nil {
t.Errorf("providers[0].kvm = %+v, want nil", cfg.Providers[0].KVM)
}
k := cfg.Providers[0].KVM2
if k == nil {
t.Fatal("providers[0].kvm2 = nil")
}
if k.SSHKeyFile != "/etc/proxy-operator/kvm/id_ed25519" || k.KnownHostsFile != "/etc/proxy-operator/kvm/known_hosts" || k.DiskSizeGB != 20 {
t.Errorf("kvm2 = %+v, want sshKeyFile, knownHostsFile and diskSizeGb parsed", k)
}
if k.Images["ubuntu"].Format != "raw" {
t.Errorf("images[ubuntu].format = %q, want raw", k.Images["ubuntu"].Format)
}
if mt := k.MachineTypes["small"]; mt.VCPUs != 1 || mt.MemoryMiB != 1024 {
t.Errorf("machineTypes[small] = %+v, want {1 1024}", mt)
}
if len(k.Hosts) != 2 || k.Hosts[0].Name != "nas1" || k.Hosts[1].Pool != "fast" {
t.Errorf("hosts = %+v, want nas1 and lab2 with lab2's pool parsed", k.Hosts)
}
}
func TestLoadConfig_kvm2Invalid(t *testing.T) {
t.Parallel()
minimal := func(body string) string {
return "providers:\n - name: kvm2-home\n type: kvm2\n kvm2:\n" + indent(body, 6)
}
hostsAndTypes := `
machineTypes:
small: {vcpus: 1, memoryMiB: 1024}
hosts:
- {name: nas1, address: nas1:22, user: virt, bridge: br0}
`
tests := []struct {
name string
yaml string
wantErrSub string
}{
{
name: "missing kvm2 block",
yaml: "providers:\n - name: kvm2-home\n type: kvm2\n",
wantErrSub: "kvm2 block is required",
},
{
name: "missing sshKeyFile",
yaml: minimal("knownHostsFile: /kh\n" + hostsAndTypes),
wantErrSub: "kvm2.sshKeyFile is required",
},
{
name: "missing knownHostsFile without insecure flag",
yaml: minimal("sshKeyFile: /key\n" + hostsAndTypes),
wantErrSub: "kvm2.knownHostsFile is required unless kvm2.insecureIgnoreHostKey is true",
},
{
name: "no machine types",
yaml: minimal("sshKeyFile: /key\nknownHostsFile: /kh\nhosts:\n - {name: nas1, address: nas1:22, user: virt, bridge: br0}\n"),
wantErrSub: "kvm2.machineTypes must define at least one",
},
{
name: "no hosts",
yaml: minimal("sshKeyFile: /key\nknownHostsFile: /kh\nmachineTypes:\n small: {vcpus: 1, memoryMiB: 1024}\n"),
wantErrSub: "kvm2.hosts must list at least one host",
},
{
name: "duplicate host name",
yaml: minimal("sshKeyFile: /key\nknownHostsFile: /kh\nmachineTypes:\n small: {vcpus: 1, memoryMiB: 1024}\nhosts:\n - {name: nas1, address: a:22, user: virt, bridge: br0}\n - {name: nas1, address: b:22, user: virt, bridge: br0}\n"),
wantErrSub: `kvm2.hosts[1]: duplicate host name "nas1"`,
},
{
name: "image key with a slash",
yaml: minimal("sshKeyFile: /key\nknownHostsFile: /kh\nimages:\n ../etc/d: {url: https://x/y.qcow2}\n" + hostsAndTypes),
wantErrSub: `kvm2.images["../etc/d"]: key must be letters, digits, dots, dashes or underscores`,
},
}
for _, tc := range tests {
t.Run(tc.name, func(t *testing.T) {
t.Parallel()
_, err := LoadConfig([]byte(tc.yaml))
if err == nil {
t.Fatalf("LoadConfig() error = nil, want error containing %q", tc.wantErrSub)
}
if !strings.Contains(err.Error(), tc.wantErrSub) {
t.Errorf("LoadConfig() error = %q, want substring %q", err.Error(), tc.wantErrSub)
}
})
}
}
func indent(s string, n int) string {
pad := strings.Repeat(" ", n)
lines := strings.Split(strings.TrimLeft(s, "\n"), "\n")
internal/provider/config_test.go
+20
View File
@@ -118,6 +118,26 @@ providers:
image: custom-image`,
wantErrSub: "type is gcp but a kubernetes block is set",
},
{
name: "kvm type with kvm2 block",
yaml: `
providers:
- name: p1
type: kvm
kvm2:
sshKeyFile: /key`,
wantErrSub: "type is kvm but a kvm2 block is set",
},
{
name: "kvm2 type with kvm block",
yaml: `
providers:
- name: p1
type: kvm2
kvm:
sshKeyFile: /key`,
wantErrSub: "type is kvm2 but a kvm block is set",
},
{
name: "strict mode rejects unknown top-level key",
yaml: `
internal/provider/gcp/check.go
+78
View File
@@ -0,0 +1,78 @@
package gcp
import (
"context"
"errors"
"net/http"
"time"
"cloud.google.com/go/compute/apiv1/computepb"
"google.golang.org/api/googleapi"
"google.golang.org/api/iterator"
"google.golang.org/protobuf/proto"
"gitea.home.hrajfrisbee.cz/kacerr/egress-proxies-operator/internal/provider"
)
var _ provider.Checker = (*Provider)(nil)
// computeAddress is the endpoint the SDK talks to, shown only to humans.
const computeAddress = "compute.googleapis.com"
// managedFilter selects the instances this operator created.
const managedFilter = "labels." + provider.LabelManaged + " = " + provider.LabelManagedYes
// Check asks the project for one page holding at most one managed
// instance. That exercises the credentials, the Compute API and the
// compute.instances.list permission orphan GC depends on, in a single
// request whatever the project's size. The target is the project: the
// provider config names no zones.
func (p *Provider) Check(ctx context.Context) []provider.TargetStatus {
start := time.Now()
err := p.api.Probe(ctx, &computepb.AggregatedListInstancesRequest{
Project: p.cfg.Project,
Filter: proto.String(managedFilter),
MaxResults: proto.Uint32(1),
ReturnPartialSuccess: proto.Bool(true),
})
ts := provider.TargetStatus{Name: p.cfg.Project, Address: computeAddress, Latency: time.Since(start)}
if err != nil {
logAPIError(p.logger(ctx), "check", err)
ts.Reason = checkReason(ctx, err)
ts.Message = err.Error()
return []provider.TargetStatus{ts}
}
ts.Ready = true
ts.Reason = provider.ReasonReachable
return []provider.TargetStatus{ts}
}
// Probe fetches the first page of an aggregated list and discards it. The
// SDK treats MaxResults as the page size, so AggregatedList, which drains
// the iterator, would still walk every page of the project.
func (r *realInstances) Probe(ctx context.Context, req *computepb.AggregatedListInstancesRequest) error {
_, err := r.client.AggregatedList(ctx, req).Next()
if err == iterator.Done {
return nil
}
return err
}
// checkReason maps a failed probe to a reason. A token that cannot be
// obtained never reaches the API, so it has no status code and lands on
// Unreachable, with the detail in the message.
func checkReason(ctx context.Context, err error) string {
var gerr *googleapi.Error
switch {
case ctx.Err() != nil, errors.Is(err, context.DeadlineExceeded):
return provider.ReasonTimeout
case !errors.As(err, &gerr):
return provider.ReasonUnreachable
case gerr.Code == http.StatusUnauthorized:
return provider.ReasonUnauthorized
case gerr.Code == http.StatusForbidden:
return provider.ReasonForbidden
default:
return provider.ReasonAPIError
}
}
internal/provider/gcp/check_test.go
+165
View File
@@ -0,0 +1,165 @@
package gcp
import (
"context"
"errors"
"net/http"
"net/http/httptest"
"net/url"
"sync"
"syscall"
"testing"
compute "cloud.google.com/go/compute/apiv1"
"cloud.google.com/go/compute/apiv1/computepb"
"google.golang.org/api/googleapi"
"google.golang.org/api/option"
"google.golang.org/protobuf/proto"
"gitea.home.hrajfrisbee.cz/kacerr/egress-proxies-operator/internal/provider"
)
func TestCheck_reachable(t *testing.T) {
t.Parallel()
api := &fakeAPI{}
p := newTestProvider(api)
got := p.Check(context.Background())
if len(got) != 1 {
t.Fatalf("Check() returned %d targets, want 1", len(got))
}
ts := got[0]
if !ts.Ready || ts.Reason != provider.ReasonReachable || ts.Message != "" {
t.Errorf("Check() = %+v, want ready and Reachable", ts)
}
if ts.Name != "my-project" || ts.Address != "compute.googleapis.com" {
t.Errorf("target = %q at %q, want my-project at compute.googleapis.com", ts.Name, ts.Address)
}
want := &computepb.AggregatedListInstancesRequest{
Project: "my-project",
Filter: proto.String("labels.proxy-operator-managed = true"),
MaxResults: proto.Uint32(1),
ReturnPartialSuccess: proto.Bool(true),
}
if !proto.Equal(api.probeReq, want) {
t.Errorf("probe request = %v, want %v", api.probeReq, want)
}
}
func TestCheck_failureReasons(t *testing.T) {
t.Parallel()
ended, cancel := context.WithCancel(context.Background())
cancel()
tests := []struct {
name string
ctx context.Context
err error
want string
}{
{"unauthorized", context.Background(), gerr(401), provider.ReasonUnauthorized},
{"forbidden", context.Background(), gerr(403, "forbidden"), provider.ReasonForbidden},
{"api disabled", context.Background(), gerr(403, "accessNotConfigured"), provider.ReasonForbidden},
{"server error", context.Background(), gerr(500), provider.ReasonAPIError},
{"deadline", context.Background(), &url.Error{
Op: "Get", URL: "https://compute.googleapis.com/", Err: context.DeadlineExceeded,
}, provider.ReasonTimeout},
{"context ended", ended, context.Canceled, provider.ReasonTimeout},
{"connection refused", context.Background(), &url.Error{
Op: "Get", URL: "https://compute.googleapis.com/", Err: syscall.ECONNREFUSED,
}, provider.ReasonUnreachable},
}
for _, tc := range tests {
t.Run(tc.name, func(t *testing.T) {
t.Parallel()
p := newTestProvider(&fakeAPI{probeErr: tc.err})
got := p.Check(tc.ctx)
if len(got) != 1 {
t.Fatalf("Check() returned %d targets, want 1", len(got))
}
if got[0].Ready || got[0].Reason != tc.want || got[0].Message != tc.err.Error() {
t.Errorf("Check() = %+v, want not ready, reason %s, message %q", got[0], tc.want, tc.err.Error())
}
})
}
}
// TestRealInstances_Probe runs the real SDK client against a local server,
// since the point of Probe is how many requests the SDK makes.
func TestRealInstances_Probe(t *testing.T) {
t.Parallel()
tests := []struct {
name string
status int
body string
wantCode int
}{
{
name: "fetches only the first page",
status: http.StatusOK,
body: `{"items":{"zones/europe-west1-b":{"instances":[{"name":"proxy-a"}]}},"nextPageToken":"page-2"}`,
},
{
name: "empty project",
status: http.StatusOK,
body: `{}`,
},
{
name: "api error keeps its status",
status: http.StatusForbidden,
body: `{"error":{"code":403,"message":"Compute Engine API has not been used","errors":[{"reason":"accessNotConfigured"}]}}`,
wantCode: http.StatusForbidden,
},
}
for _, tc := range tests {
t.Run(tc.name, func(t *testing.T) {
t.Parallel()
var (
mu sync.Mutex
reqs []*url.URL
)
srv := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
mu.Lock()
reqs = append(reqs, r.URL)
mu.Unlock()
w.Header().Set("Content-Type", "application/json")
w.WriteHeader(tc.status)
_, _ = w.Write([]byte(tc.body))
}))
t.Cleanup(srv.Close)
ctx := context.Background()
c, err := compute.NewInstancesRESTClient(ctx, option.WithEndpoint(srv.URL), option.WithoutAuthentication())
if err != nil {
t.Fatalf("NewInstancesRESTClient: %v", err)
}
t.Cleanup(func() { _ = c.Close() })
err = (&realInstances{client: c}).Probe(ctx, &computepb.AggregatedListInstancesRequest{
Project: "my-project",
MaxResults: proto.Uint32(1),
})
var gerr *googleapi.Error
switch {
case tc.wantCode == 0 && err != nil:
t.Fatalf("Probe() = %v, want nil", err)
case tc.wantCode != 0 && (!errors.As(err, &gerr) || gerr.Code != tc.wantCode):
t.Fatalf("Probe() = %v, want a googleapi error with code %d", err, tc.wantCode)
}
mu.Lock()
defer mu.Unlock()
if len(reqs) != 1 {
t.Fatalf("server saw %d requests, want 1", len(reqs))
}
if got := reqs[0].Query().Get("maxResults"); got != "1" {
t.Errorf("maxResults = %q, want 1", got)
}
if want := "/compute/v1/projects/my-project/aggregated/instances"; reqs[0].Path != want {
t.Errorf("path = %q, want %q", reqs[0].Path, want)
}
})
}
}
internal/provider/gcp/gcp.go
+4 -3
View File
@@ -33,6 +33,8 @@ type instancesAPI interface {
Get(ctx context.Context, req *computepb.GetInstanceRequest) (*computepb.Instance, error)
Delete(ctx context.Context, req *computepb.DeleteInstanceRequest) (opName string, err error)
AggregatedList(ctx context.Context, req *computepb.AggregatedListInstancesRequest) ([]*computepb.Instance, error)
// Probe fetches only the first page of an aggregated list; see Check.
Probe(ctx context.Context, req *computepb.AggregatedListInstancesRequest) error
}
// realInstances adapts *compute.InstancesClient to the seam.
@@ -221,17 +223,16 @@ func (p *Provider) Delete(ctx context.Context, providerID string) error {
// entire GC sweep.
func (p *Provider) ListByTag(ctx context.Context) ([]provider.Instance, error) {
log := p.logger(ctx)
filter := fmt.Sprintf("labels.%s = %s", provider.LabelManaged, provider.LabelManagedYes)
instances, err := p.api.AggregatedList(ctx, &computepb.AggregatedListInstancesRequest{
Project: p.cfg.Project,
Filter: proto.String(filter),
Filter: proto.String(managedFilter),
ReturnPartialSuccess: proto.Bool(true),
})
if err != nil {
logAPIError(log, "list", err)
return nil, p.wrapErr("list", "", err)
}
log.V(1).Info("GCP instances listed", "filter", filter, "count", len(instances))
log.V(1).Info("GCP instances listed", "filter", managedFilter, "count", len(instances))
out := make([]provider.Instance, 0, len(instances))
for _, inst := range instances {
conv := toInstance(inst, lastPathSegment(inst.GetZone()))
internal/provider/gcp/gcp_test.go
+8
View File
@@ -31,6 +31,14 @@ type fakeAPI struct {
listReq *computepb.AggregatedListInstancesRequest
listInsts []*computepb.Instance
listErr error
probeReq *computepb.AggregatedListInstancesRequest
probeErr error
}
func (f *fakeAPI) Probe(_ context.Context, req *computepb.AggregatedListInstancesRequest) error {
f.probeReq = req
return f.probeErr
}
func (f *fakeAPI) Insert(_ context.Context, req *computepb.InsertInstanceRequest) (string, error) {
internal/provider/kubernetes/check.go
+53
View File
@@ -0,0 +1,53 @@
package kubernetes
import (
"context"
"errors"
"time"
corev1 "k8s.io/api/core/v1"
apierrors "k8s.io/apimachinery/pkg/api/errors"
"sigs.k8s.io/controller-runtime/pkg/client"
"gitea.home.hrajfrisbee.cz/kacerr/egress-proxies-operator/internal/provider"
)
var _ provider.Checker = (*Provider)(nil)
// Check lists at most one managed Pod across the cluster: the same client,
// verb and scope ListByTag and orphan GC rely on, so a pass means the API
// server answers and the operator's RBAC still covers them. It does not
// prove create or delete rights.
func (p *Provider) Check(ctx context.Context) []provider.TargetStatus {
start := time.Now()
var pods corev1.PodList
err := p.client.List(ctx, &pods, client.MatchingLabels{
provider.LabelManaged: provider.LabelManagedYes,
}, client.Limit(1))
ts := provider.TargetStatus{Name: "apiserver", Address: p.apiHost, Latency: time.Since(start)}
if err != nil {
ts.Reason = checkReason(ctx, err)
ts.Message = err.Error()
return []provider.TargetStatus{ts}
}
ts.Ready = true
ts.Reason = provider.ReasonReachable
return []provider.TargetStatus{ts}
}
func checkReason(ctx context.Context, err error) string {
var status apierrors.APIStatus
switch {
case ctx.Err() != nil, errors.Is(err, context.DeadlineExceeded),
apierrors.IsTimeout(err), apierrors.IsServerTimeout(err):
return provider.ReasonTimeout
case apierrors.IsUnauthorized(err):
return provider.ReasonUnauthorized
case apierrors.IsForbidden(err):
return provider.ReasonForbidden
case errors.As(err, &status):
return provider.ReasonAPIError
default:
return provider.ReasonUnreachable
}
}
internal/provider/kubernetes/check_test.go
+114
View File
@@ -0,0 +1,114 @@
package kubernetes
import (
"context"
"errors"
"net/url"
"syscall"
"testing"
corev1 "k8s.io/api/core/v1"
apierrors "k8s.io/apimachinery/pkg/api/errors"
"k8s.io/apimachinery/pkg/runtime"
"k8s.io/apimachinery/pkg/runtime/schema"
"sigs.k8s.io/controller-runtime/pkg/client"
"sigs.k8s.io/controller-runtime/pkg/client/fake"
"sigs.k8s.io/controller-runtime/pkg/client/interceptor"
"gitea.home.hrajfrisbee.cz/kacerr/egress-proxies-operator/internal/provider"
)
type listFunc func(ctx context.Context, c client.WithWatch, list client.ObjectList, opts ...client.ListOption) error
func newCheckProvider(list listFunc) *Provider {
scheme := runtime.NewScheme()
if err := corev1.AddToScheme(scheme); err != nil {
panic(err)
}
c := fake.NewClientBuilder().WithScheme(scheme).WithInterceptorFuncs(interceptor.Funcs{List: list}).Build()
p := newWithClient(c, provider.ProviderConfig{Name: "kubernetes"})
p.apiHost = "https://10.96.0.1:443"
return p
}
func TestCheck_reachable(t *testing.T) {
t.Parallel()
p := newTestProvider(testPod("ns", "a", "uid-a", corev1.PodRunning, "10.0.0.1"))
p.apiHost = "https://10.96.0.1:443"
got := p.Check(context.Background())
if len(got) != 1 {
t.Fatalf("Check() returned %d targets, want 1", len(got))
}
ts := got[0]
if !ts.Ready || ts.Reason != provider.ReasonReachable || ts.Message != "" {
t.Errorf("Check() = %+v, want ready and Reachable", ts)
}
if ts.Name != "apiserver" || ts.Address != "https://10.96.0.1:443" {
t.Errorf("target = %q at %q, want apiserver at https://10.96.0.1:443", ts.Name, ts.Address)
}
}
func TestCheck_listOptions(t *testing.T) {
t.Parallel()
var got client.ListOptions
p := newCheckProvider(func(ctx context.Context, c client.WithWatch, list client.ObjectList, opts ...client.ListOption) error {
got.ApplyOptions(opts)
return c.List(ctx, list, opts...)
})
p.Check(context.Background())
if got.Limit != 1 {
t.Errorf("Limit = %d, want 1", got.Limit)
}
if got.Namespace != "" {
t.Errorf("Namespace = %q, want cluster-wide", got.Namespace)
}
want := provider.LabelManaged + "=" + provider.LabelManagedYes
if got.LabelSelector == nil || got.LabelSelector.String() != want {
t.Errorf("LabelSelector = %v, want %s", got.LabelSelector, want)
}
}
func TestCheck_failureReasons(t *testing.T) {
t.Parallel()
pods := schema.GroupResource{Resource: "pods"}
ended, cancel := context.WithCancel(context.Background())
cancel()
tests := []struct {
name string
ctx context.Context
err error
want string
}{
{"forbidden", context.Background(), apierrors.NewForbidden(pods, "", errors.New("rbac denied")), provider.ReasonForbidden},
{"unauthorized", context.Background(), apierrors.NewUnauthorized("token expired"), provider.ReasonUnauthorized},
{"api timeout", context.Background(), apierrors.NewTimeoutError("slow etcd", 1), provider.ReasonTimeout},
{"server timeout", context.Background(), apierrors.NewServerTimeout(pods, "list", 1), provider.ReasonTimeout},
{"context ended", ended, context.Canceled, provider.ReasonTimeout},
{"internal error", context.Background(), apierrors.NewInternalError(errors.New("etcd down")), provider.ReasonAPIError},
{"connection refused", context.Background(), &url.Error{
Op: "Get", URL: "https://10.96.0.1:443/api/v1/pods", Err: syscall.ECONNREFUSED,
}, provider.ReasonUnreachable},
}
for _, tc := range tests {
t.Run(tc.name, func(t *testing.T) {
t.Parallel()
p := newCheckProvider(func(context.Context, client.WithWatch, client.ObjectList, ...client.ListOption) error {
return tc.err
})
got := p.Check(tc.ctx)
if len(got) != 1 {
t.Fatalf("Check() returned %d targets, want 1", len(got))
}
if got[0].Ready || got[0].Reason != tc.want || got[0].Message != tc.err.Error() {
t.Errorf("Check() = %+v, want not ready, reason %s, message %q", got[0], tc.want, tc.err.Error())
}
})
}
}
internal/provider/kubernetes/kubernetes.go
+5 -1
View File
@@ -36,6 +36,8 @@ const defaultImage = "ubuntu/squid:6.6-24.04_edge"
type Provider struct {
client client.Client
image string
// apiHost is the API server URL, shown by Check; empty in tests.
apiHost string
}
// New builds a kubernetes Provider from its config block. Satisfies
@@ -64,7 +66,9 @@ func NewWithTransportWrapper(_ context.Context, cfg provider.ProviderConfig, wra
if err != nil {
return nil, fmt.Errorf("kubernetes provider %q: building client: %w", cfg.Name, err)
}
return newWithClient(c, cfg), nil
p := newWithClient(c, cfg)
p.apiHost = restCfg.Host
return p, nil
}
// newWithClient builds a Provider around an already-constructed client,
internal/provider/kvm/check.go
+166
View File
@@ -0,0 +1,166 @@
package kvm
import (
"context"
"errors"
"fmt"
"sync"
"time"
libvirt "github.com/digitalocean/go-libvirt"
"gitea.home.hrajfrisbee.cz/kacerr/egress-proxies-operator/internal/provider"
)
var _ provider.Checker = (*Provider)(nil)
// Reasons only the KVM check can tell apart; the rest are provider-wide.
const (
reasonSSHMaterialUnavailable = "SSHMaterialUnavailable"
reasonSSHHandshakeFailed = "SSHHandshakeFailed"
reasonLibvirtConnectFailed = "LibvirtConnectFailed"
reasonLibvirtError = "LibvirtError"
reasonPoolNotFound = "PoolNotFound"
reasonPoolNotActive = "PoolNotActive"
)
// hostFacts is what a successful check learns about a host.
type hostFacts struct {
version uint64
hostname string
poolState libvirt.StoragePoolState
capacity uint64
available uint64
}
// Check verifies every configured host concurrently, in kvm.hosts order.
// It goes through the same pooled connection Create and Get use, so the
// SSH and libvirt handshakes only happen on the first check or after the
// link dropped; then two cheap RPCs prove the daemon answers, and the
// storage pool VMs are created in must exist and be running.
func (p *Provider) Check(ctx context.Context) []provider.TargetStatus {
out := make([]provider.TargetStatus, len(p.cfg.Hosts))
var wg sync.WaitGroup
for i, h := range p.cfg.Hosts {
wg.Go(func() { out[i] = p.checkHost(ctx, h) })
}
wg.Wait()
return out
}
// checkHost goes through probe, so a check that runs out of time abandons
// its RPCs rather than closing the connection the reconciles share. fn
// returns the RPC errors untouched, since a transport error still drops the
// link; a pool that exists but is stopped says nothing about the link, so
// that verdict is reached afterwards. facts is read only after probe
// returned nil, since an abandoned fn may still write it.
func (p *Provider) checkHost(ctx context.Context, host provider.KVMHost) provider.TargetStatus {
ts := provider.TargetStatus{
Name: host.Name,
Address: host.Address,
Details: map[string]string{"user": host.User, "pool": host.Pool},
}
start := time.Now()
var facts hostFacts
err := p.hosts.probe(ctx, p.logger(ctx), host, func(api libvirtAPI) error {
version, err := api.ConnectGetLibVersion()
if err != nil {
return err
}
hostname, err := api.ConnectGetHostname()
if err != nil {
return err
}
pool, err := api.StoragePoolLookupByName(host.Pool)
if err != nil {
return err
}
state, capacity, _, available, err := api.StoragePoolGetInfo(pool)
if err != nil {
return err
}
facts = hostFacts{
version: version,
hostname: hostname,
poolState: libvirt.StoragePoolState(state),
capacity: capacity,
available: available,
}
return nil
})
ts.Latency = time.Since(start)
if err != nil {
ts.Reason = checkReason(err)
ts.Message = err.Error()
return ts
}
ts.Details["hostname"] = facts.hostname
ts.Details["libvirtVersion"] = formatLibVersion(facts.version)
ts.Details["poolState"] = poolStateName(facts.poolState)
ts.Details["poolCapacity"] = formatGiB(facts.capacity)
ts.Details["poolAvailable"] = formatGiB(facts.available)
if facts.poolState != libvirt.StoragePoolRunning {
ts.Reason = reasonPoolNotActive
ts.Message = fmt.Sprintf("storage pool %q is %s", host.Pool, poolStateName(facts.poolState))
return ts
}
ts.Ready = true
ts.Reason = provider.ReasonReachable
return ts
}
// checkReason names where a failed check broke. The dial stage wins over
// a timeout: "the SSH handshake did not finish" says more than "the check
// ran out of time", and the message still carries the deadline.
func checkReason(err error) string {
var de *dialError
if errors.As(err, &de) {
switch de.stage {
case stageSSHMaterial:
return reasonSSHMaterialUnavailable
case stageSSHHandshake:
return reasonSSHHandshakeFailed
case stageLibvirtConnect:
return reasonLibvirtConnectFailed
default:
return provider.ReasonUnreachable
}
}
switch {
case errors.Is(err, context.DeadlineExceeded), errors.Is(err, context.Canceled), errors.Is(err, errProbeBusy):
return provider.ReasonTimeout
case hasCode(err, libvirt.ErrNoStoragePool):
return reasonPoolNotFound
case isLibvirtError(err):
return reasonLibvirtError
default:
return provider.ReasonUnreachable
}
}
// formatLibVersion decodes libvirt's major*1e6 + minor*1e3 + micro.
func formatLibVersion(v uint64) string {
return fmt.Sprintf("%d.%d.%d", v/1_000_000, v/1_000%1_000, v%1_000)
}
// poolStateName uses virsh's words for a pool state.
func poolStateName(s libvirt.StoragePoolState) string {
switch s {
case libvirt.StoragePoolInactive:
return "inactive"
case libvirt.StoragePoolBuilding:
return "building"
case libvirt.StoragePoolRunning:
return "running"
case libvirt.StoragePoolDegraded:
return "degraded"
case libvirt.StoragePoolInaccessible:
return "inaccessible"
default:
return fmt.Sprintf("state %d", s)
}
}
func formatGiB(bytes uint64) string {
return fmt.Sprintf("%.1fGiB", float64(bytes)/(1<<30))
}
internal/provider/kvm/check_test.go
+319
View File
@@ -0,0 +1,319 @@
package kvm
import (
"context"
"errors"
"fmt"
"io"
"maps"
"strings"
"sync"
"sync/atomic"
"testing"
"time"
libvirt "github.com/digitalocean/go-libvirt"
"gitea.home.hrajfrisbee.cz/kacerr/egress-proxies-operator/internal/provider"
)
// withoutLatency zeroes the one field a test cannot predict.
func withoutLatency(in []provider.TargetStatus) []provider.TargetStatus {
out := make([]provider.TargetStatus, len(in))
for i, ts := range in {
ts.Latency = 0
out[i] = ts
}
return out
}
func TestCheck_reachable(t *testing.T) {
t.Parallel()
api := newFakeAPI("default")
api.poolCapacity = 10 << 30
api.pools["default"]["disk.qcow2"] = &fakeVol{capacity: 4 << 30}
p := newTestProvider(hostFakes{"nas1": api}, "nas1")
got := withoutLatency(p.Check(context.Background()))
want := provider.TargetStatus{
Name: "nas1",
Address: "nas1:22",
Ready: true,
Reason: provider.ReasonReachable,
Details: map[string]string{
"user": "virt",
"pool": "default",
"hostname": "fake-host",
"libvirtVersion": "10.2.0",
"poolState": "running",
"poolCapacity": "10.0GiB",
"poolAvailable": "6.0GiB",
},
}
if len(got) != 1 {
t.Fatalf("Check() returned %d targets, want 1", len(got))
}
if got[0].Name != want.Name || got[0].Address != want.Address || got[0].Ready != want.Ready ||
got[0].Reason != want.Reason || got[0].Message != "" || !maps.Equal(got[0].Details, want.Details) {
t.Errorf("Check() = %+v, want %+v", got[0], want)
}
}
func TestCheck_perHostVerdicts(t *testing.T) {
t.Parallel()
p := newTestProvider(hostFakes{"nas1": newFakeAPI("default"), "nas3": newFakeAPI("default")}, "nas1", "nas2", "nas3")
got := p.Check(context.Background())
if len(got) != 3 {
t.Fatalf("Check() returned %d targets, want 3", len(got))
}
for i, want := range []struct {
name string
ready bool
}{{"nas1", true}, {"nas2", false}, {"nas3", true}} {
if got[i].Name != want.name || got[i].Ready != want.ready {
t.Errorf("target %d = %s ready=%v, want %s ready=%v", i, got[i].Name, got[i].Ready, want.name, want.ready)
}
}
if got[1].Reason != provider.ReasonUnreachable || !strings.Contains(got[1].Message, errHostOff.Error()) {
t.Errorf("unreachable host = %s %q, want Unreachable carrying %q", got[1].Reason, got[1].Message, errHostOff)
}
if got[1].Details["pool"] != "default" || got[1].Details["hostname"] != "" {
t.Errorf("unreachable host details = %v, want the configured pool and nothing learned", got[1].Details)
}
}
// TestCheck_libvirtVerdicts also checks twice per case: none of these
// verdicts is evidence against the link, so the connection must survive.
func TestCheck_libvirtVerdicts(t *testing.T) {
t.Parallel()
tests := []struct {
name string
setup func(api *fakeAPI)
pools []string
wantReason string
wantMessage string
}{
{
name: "pool inactive",
pools: []string{"default"},
setup: func(api *fakeAPI) { api.poolStates["default"] = libvirt.StoragePoolInactive },
wantReason: reasonPoolNotActive,
wantMessage: `storage pool "default" is inactive`,
},
{
name: "pool missing",
pools: []string{"other"},
wantReason: reasonPoolNotFound,
wantMessage: "no storage pool with matching name 'default'",
},
{
name: "daemon error",
pools: []string{"default"},
setup: func(api *fakeAPI) { api.fail["ConnectGetHostname"] = lvError(libvirt.ErrInternalError, "boom") },
wantReason: reasonLibvirtError,
wantMessage: "boom",
},
}
for _, tc := range tests {
t.Run(tc.name, func(t *testing.T) {
t.Parallel()
api := newFakeAPI(tc.pools...)
if tc.setup != nil {
tc.setup(api)
}
var dials atomic.Int32
dial := func(ctx context.Context, h provider.KVMHost) (*hostClient, error) {
dials.Add(1)
return hostFakes{"nas1": api}.dial(ctx, h)
}
pc := testConfig("nas1")
p := newWithDialer(pc.Name, *pc.KVM, dial, nil)
for range 2 {
got := p.Check(context.Background())
if got[0].Ready || got[0].Reason != tc.wantReason || !strings.Contains(got[0].Message, tc.wantMessage) {
t.Errorf("Check() = %+v, want not ready, %s, message containing %q", got[0], tc.wantReason, tc.wantMessage)
}
}
if n := dials.Load(); n != 1 {
t.Errorf("dialed %d times over two checks, want 1", n)
}
})
}
}
func TestCheck_hostsCheckedConcurrently(t *testing.T) {
t.Parallel()
nas2Dialing := make(chan struct{})
dial := func(ctx context.Context, h provider.KVMHost) (*hostClient, error) {
switch h.Name {
case "nas1":
select {
case <-nas2Dialing:
case <-ctx.Done():
return nil, ctx.Err()
}
case "nas2":
close(nas2Dialing)
}
return &hostClient{api: newFakeAPI("default"), close: func() {}}, nil
}
pc := testConfig("nas1", "nas2")
p := newWithDialer(pc.Name, *pc.KVM, dial, nil)
ctx, cancel := context.WithTimeout(context.Background(), 5*time.Second)
defer cancel()
got := p.Check(ctx)
for _, ts := range got {
if !ts.Ready {
t.Errorf("%s = %s %q, want ready: a sequential check would time out nas1", ts.Name, ts.Reason, ts.Message)
}
}
}
// A daemon that stopped answering times the check out, but the connection
// the reconciles share stays up, and the next check does not stack a
// second hung RPC on top of the first.
func TestCheck_timeoutKeepsTheConnection(t *testing.T) {
t.Parallel()
api := newFakeAPI("default")
block := make(chan struct{})
t.Cleanup(func() { close(block) })
blocked := &blockingAPI{fakeAPI: api, block: block}
var closed atomic.Bool
dial := func(context.Context, provider.KVMHost) (*hostClient, error) {
return &hostClient{api: blocked, close: func() { closed.Store(true) }}, nil
}
pc := testConfig("nas1")
p := newWithDialer(pc.Name, *pc.KVM, dial, nil)
ctx, cancel := context.WithTimeout(context.Background(), 50*time.Millisecond)
defer cancel()
got := p.Check(ctx)
if got[0].Ready || got[0].Reason != provider.ReasonTimeout {
t.Errorf("Check() = %+v, want Timeout", got[0])
}
if closed.Load() {
t.Error("the timed-out check closed the connection the reconciles share")
}
again := p.Check(context.Background())
if again[0].Reason != provider.ReasonTimeout || !strings.Contains(again[0].Message, "still waiting") {
t.Errorf("second Check() = %+v, want an immediate Timeout while the first still waits", again[0])
}
}
// A reconcile redialing a dead host holds the host's slot for minutes; the
// check must give up at its own deadline rather than wait that out.
func TestCheck_doesNotWaitBehindAReconcileDial(t *testing.T) {
t.Parallel()
release := make(chan struct{})
dialing := make(chan struct{})
var once sync.Once
dial := func(context.Context, provider.KVMHost) (*hostClient, error) {
once.Do(func() { close(dialing) })
<-release
return nil, errHostOff
}
pc := testConfig("nas1")
p := newWithDialer(pc.Name, *pc.KVM, dial, nil)
reconcile := make(chan error, 1)
go func() {
_, err := p.Get(context.Background(), formatProviderID("nas1", testName))
reconcile <- err
}()
<-dialing
ctx, cancel := context.WithTimeout(context.Background(), 50*time.Millisecond)
defer cancel()
start := time.Now()
got := p.Check(ctx)
if waited := time.Since(start); waited > time.Second {
t.Errorf("Check() took %s behind the reconcile's dial, want about 50ms", waited)
}
if got[0].Reason != provider.ReasonTimeout {
t.Errorf("Check() = %+v, want Timeout", got[0])
}
close(release)
<-reconcile
}
// blockingAPI hangs in ConnectGetLibVersion until block closes, like a
// daemon that stopped answering.
type blockingAPI struct {
*fakeAPI
block chan struct{}
}
func (b *blockingAPI) ConnectGetLibVersion() (uint64, error) {
<-b.block
return 0, io.EOF
}
func TestCheckReason(t *testing.T) {
t.Parallel()
tests := []struct {
name string
err error
want string
}{
{"ssh material", &dialError{stage: stageSSHMaterial, err: fmt.Errorf("%w: parsing key", provider.ErrPermanent)}, reasonSSHMaterialUnavailable},
{"tcp", &dialError{stage: stageTCP, err: errors.New("ssh dial: connection refused")}, provider.ReasonUnreachable},
{"ssh handshake", &dialError{stage: stageSSHHandshake, err: errors.New("ssh handshake: unable to authenticate")}, reasonSSHHandshakeFailed},
{"libvirt connect", &dialError{stage: stageLibvirtConnect, err: errors.New("libvirt connect via /sock: EOF")}, reasonLibvirtConnectFailed},
{"stage survives the pool's wrapping", fmt.Errorf("connecting to host nas1 (nas1:22): %w",
&dialError{stage: stageSSHHandshake, err: context.DeadlineExceeded}), reasonSSHHandshakeFailed},
{"deadline", context.DeadlineExceeded, provider.ReasonTimeout},
{"cancelled", context.Canceled, provider.ReasonTimeout},
{"previous check still waiting", errProbeBusy, provider.ReasonTimeout},
{"no pool", lvError(libvirt.ErrNoStoragePool, "no pool"), reasonPoolNotFound},
{"other libvirt error", lvError(libvirt.ErrInternalError, "boom"), reasonLibvirtError},
{"link died mid-call", io.EOF, provider.ReasonUnreachable},
}
for _, tc := range tests {
t.Run(tc.name, func(t *testing.T) {
t.Parallel()
if got := checkReason(tc.err); got != tc.want {
t.Errorf("checkReason(%v) = %s, want %s", tc.err, got, tc.want)
}
})
}
}
func TestDialError_keepsClassification(t *testing.T) {
t.Parallel()
err := fmt.Errorf("connecting to host nas1: %w",
&dialError{stage: stageSSHMaterial, err: fmt.Errorf("%w: parsing kvm.sshKeyFile: bad", provider.ErrPermanent)})
if got := classify(err); !errors.Is(got, provider.ErrPermanent) {
t.Errorf("classify() = %v, want ErrPermanent through the dialError", got)
}
if !isClassified(err) {
t.Error("isClassified() = false, want true")
}
if want := "connecting to host nas1: permanent: parsing kvm.sshKeyFile: bad"; !strings.HasSuffix(err.Error(), "parsing kvm.sshKeyFile: bad") {
t.Errorf("message = %q, want it unchanged, like %q", err.Error(), want)
}
}
func TestFormatLibVersion(t *testing.T) {
t.Parallel()
tests := []struct {
in uint64
want string
}{
{10002000, "10.2.0"},
{11006001, "11.6.1"},
{9000000, "9.0.0"},
}
for _, tc := range tests {
if got := formatLibVersion(tc.in); got != tc.want {
t.Errorf("formatLibVersion(%d) = %s, want %s", tc.in, got, tc.want)
}
}
}
internal/provider/kvm/conn.go
+111 -16
View File
@@ -9,6 +9,7 @@ import (
"net"
"os"
"sync"
"sync/atomic"
"time"
libvirt "github.com/digitalocean/go-libvirt"
@@ -44,6 +45,9 @@ type libvirtAPI interface {
StorageVolUpload(vol libvirt.StorageVol, r io.Reader, offset, length uint64, flags libvirt.StorageVolUploadFlags) error
StorageVolDownload(vol libvirt.StorageVol, w io.Writer, offset, length uint64, flags libvirt.StorageVolDownloadFlags) error
StorageVolDelete(vol libvirt.StorageVol, flags libvirt.StorageVolDeleteFlags) error
ConnectGetLibVersion() (uint64, error)
ConnectGetHostname() (string, error)
StoragePoolGetInfo(pool libvirt.StoragePool) (state uint8, capacity, allocation, available uint64, err error)
}
// hostClient is one live connection to a hypervisor.
@@ -68,6 +72,27 @@ func (c *hostClient) alive() bool {
// real one; tests inject their own.
type hostDialer func(ctx context.Context, host provider.KVMHost) (*hostClient, error)
// dialStage names the step of a connection attempt that failed, so the
// health check can say why a host is unreachable without parsing messages.
type dialStage string
const (
stageSSHMaterial dialStage = "ssh-material"
stageTCP dialStage = "tcp"
stageSSHHandshake dialStage = "ssh-handshake"
stageLibvirtConnect dialStage = "libvirt-connect"
)
// dialError tags a dial failure with its stage. It keeps the message and
// unwraps to the cause, so classify sees exactly what it saw before.
type dialError struct {
stage dialStage
err error
}
func (e *dialError) Error() string { return e.err.Error() }
func (e *dialError) Unwrap() error { return e.err }
// hostPool caches one connection per host, dialing lazily and redialing
// after the connection drops. Dials for different hosts don't block each
// other; concurrent callers for the same host share a single dial.
@@ -78,9 +103,16 @@ type hostPool struct {
entries map[string]*hostEntry
}
// hostEntry is one host's slot in the pool. lock is a one-slot semaphore
// rather than a mutex so that waiting for it can end with the caller's
// context: whoever holds it may be dialing a dead host for as long as the
// dial takes.
type hostEntry struct {
mu sync.Mutex
lock chan struct{}
client *hostClient
// probing is set while a probe's fn runs, including one whose caller
// has already given up on it.
probing atomic.Bool
}
func newHostPool(dial hostDialer) *hostPool {
@@ -92,16 +124,37 @@ func (p *hostPool) entry(name string) *hostEntry {
defer p.mu.Unlock()
e, ok := p.entries[name]
if !ok {
e = &hostEntry{}
e = &hostEntry{lock: make(chan struct{}, 1)}
p.entries[name] = e
}
return e
}
// acquire takes e.lock. Only waiting is bounded by ctx: a free slot is
// taken even when ctx has already ended, so an uncontended caller behaves
// as it did with a mutex, and a contended one gives up with ctx's error.
func (e *hostEntry) acquire(ctx context.Context) error {
select {
case e.lock <- struct{}{}:
return nil
default:
}
select {
case e.lock <- struct{}{}:
return nil
case <-ctx.Done():
return ctx.Err()
}
}
func (e *hostEntry) release() { <-e.lock }
func (p *hostPool) get(ctx context.Context, log logr.Logger, host provider.KVMHost) (*hostClient, error) {
e := p.entry(host.Name)
e.mu.Lock()
defer e.mu.Unlock()
if err := e.acquire(ctx); err != nil {
return nil, fmt.Errorf("waiting for the connection to host %s (%s): %w", host.Name, host.Address, err)
}
defer e.release()
if e.client != nil {
if e.client.alive() {
return e.client, nil
@@ -125,8 +178,8 @@ func (p *hostPool) get(ctx context.Context, log logr.Logger, host provider.KVMHo
// connection that was already replaced by a redial is left alone.
func (p *hostPool) drop(host string, c *hostClient) {
e := p.entry(host)
e.mu.Lock()
defer e.mu.Unlock()
e.lock <- struct{}{}
defer e.release()
if e.client == c {
e.client = nil
}
@@ -149,19 +202,61 @@ func (p *hostPool) call(ctx context.Context, log logr.Logger, host provider.KVMH
if err != nil {
return err
}
api := c.api
if log.V(1).Enabled() {
api = &tracingAPI{inner: c.api, log: log}
}
err = awaitOrCancel(ctx, func() error { return fn(api) }, func() {
err = awaitOrCancel(ctx, func() error { return fn(traced(c, log)) }, func() {
log.V(1).Info("libvirt connection dropped", "reason", "context cancelled", "error", ctx.Err().Error())
p.drop(host.Name, c)
})
p.dropOnTransportError(ctx, log, host, c, err)
return err
}
// errProbeBusy refuses a probe while the previous one on the same host has
// not come back yet.
var errProbeBusy = errors.New("the previous check of this host is still waiting for libvirt to answer")
// probe is call for the health check, which must never cost the reconciles
// their connection: when ctx ends first, fn is abandoned and the connection
// left alone. A link that is really dead is closed by the keepalive, which
// ends fn too. At most one probe per host runs at a time, an abandoned one
// included, so a daemon that stopped answering cannot pile up goroutines;
// until it answers, further probes get errProbeBusy.
func (p *hostPool) probe(ctx context.Context, log logr.Logger, host provider.KVMHost, fn func(api libvirtAPI) error) error {
log = log.WithName("libvirt")
e := p.entry(host.Name)
if !e.probing.CompareAndSwap(false, true) {
return errProbeBusy
}
c, err := p.get(ctx, log, host)
if err != nil {
e.probing.Store(false)
return err
}
err = awaitOrCancel(ctx, func() error {
defer e.probing.Store(false)
return fn(traced(c, log))
}, func() {
log.V(1).Info("libvirt probe abandoned, connection kept", "error", ctx.Err().Error())
})
p.dropOnTransportError(ctx, log, host, c, err)
return err
}
// traced returns c's API, logging every RPC when log has V(1) enabled.
func traced(c *hostClient, log logr.Logger) libvirtAPI {
if log.V(1).Enabled() {
return &tracingAPI{inner: c.api, log: log}
}
return c.api
}
// dropOnTransportError drops c when err is evidence against the link:
// neither a libvirt error, nor a verdict already classified, nor the
// caller's own context ending.
func (p *hostPool) dropOnTransportError(ctx context.Context, log logr.Logger, host provider.KVMHost, c *hostClient, err error) {
if err != nil && !isLibvirtError(err) && !isClassified(err) && !errors.Is(err, ctx.Err()) {
log.V(1).Info("libvirt connection dropped", "reason", "transport error", "error", err.Error())
p.drop(host.Name, c)
}
return err
}
// awaitOrCancel runs fn in the background and returns its result, unless ctx
@@ -254,7 +349,7 @@ func (l *lazySSH) load() (*sshAuth, error) {
func (l *lazySSH) dial(ctx context.Context, host provider.KVMHost) (*hostClient, error) {
a, err := l.load()
if err != nil {
return nil, err
return nil, &dialError{stage: stageSSHMaterial, err: err}
}
return a.dial(ctx, host)
}
@@ -285,7 +380,7 @@ func (a *sshAuth) dialSSH(ctx context.Context, host provider.KVMHost) (*ssh.Clie
var d net.Dialer
nc, err := d.DialContext(ctx, "tcp", host.Address)
if err != nil {
return nil, fmt.Errorf("ssh dial: %w", err)
return nil, &dialError{stage: stageTCP, err: fmt.Errorf("ssh dial: %w", err)}
}
_ = nc.SetDeadline(deadline)
conn, chans, reqs, err := ssh.NewClientConn(nc, host.Address, &ssh.ClientConfig{
@@ -295,7 +390,7 @@ func (a *sshAuth) dialSSH(ctx context.Context, host provider.KVMHost) (*ssh.Clie
})
if err != nil {
nc.Close()
return nil, fmt.Errorf("ssh handshake: %w", err)
return nil, &dialError{stage: stageSSHHandshake, err: fmt.Errorf("ssh handshake: %w", err)}
}
_ = nc.SetDeadline(time.Time{})
return ssh.NewClient(conn, chans, reqs), nil
@@ -319,7 +414,7 @@ func (u unixForward) Dial() (net.Conn, error) {
func connectLibvirt(ctx context.Context, client *ssh.Client, socket string) (*hostClient, error) {
lv := libvirt.NewWithDialer(unixForward{client: client, path: socket})
if err := awaitOrCancel(ctx, lv.Connect, func() { client.Close() }); err != nil {
return nil, fmt.Errorf("libvirt connect via %s: %w", socket, err)
return nil, &dialError{stage: stageLibvirtConnect, err: fmt.Errorf("libvirt connect via %s: %w", socket, err)}
}
stop := make(chan struct{})
var once sync.Once
internal/provider/kvm/conn_test.go
+171
View File
@@ -232,6 +232,97 @@ func TestHostPool_call_hostsAreIndependent(t *testing.T) {
}
}
// A dial to a dead host holds the host's slot for as long as it takes; a
// caller with a deadline must not wait past it.
func TestHostPool_get_waitingRespectsContext(t *testing.T) {
t.Parallel()
release := make(chan struct{})
dialing := make(chan struct{})
var once sync.Once
dial := func(context.Context, provider.KVMHost) (*hostClient, error) {
once.Do(func() { close(dialing) })
<-release
return nil, errors.New("ssh dial: connection timed out")
}
p := newHostPool(dial)
slow := make(chan error, 1)
go func() { slow <- p.call(context.Background(), discard, testHost, noop) }()
<-dialing
ctx, cancel := context.WithTimeout(context.Background(), 50*time.Millisecond)
defer cancel()
start := time.Now()
err := p.call(ctx, discard, testHost, noop)
if !errors.Is(err, context.DeadlineExceeded) {
t.Errorf("call() error = %v, want the caller's deadline", err)
}
if waited := time.Since(start); waited > time.Second {
t.Errorf("call() waited %s behind another caller's dial, want about 50ms", waited)
}
close(release)
if err := <-slow; err == nil {
t.Error("the blocked dial succeeded, want its error")
}
}
func TestHostPool_probe_abandonsWithoutClosing(t *testing.T) {
t.Parallel()
d := &fakeDialer{}
p := newHostPool(d.dial)
unblock := make(chan struct{})
ctx, cancel := context.WithTimeout(context.Background(), 20*time.Millisecond)
defer cancel()
err := p.probe(ctx, discard, testHost, func(libvirtAPI) error {
<-unblock // a daemon that stopped answering
return nil
})
if !errors.Is(err, context.DeadlineExceeded) {
t.Fatalf("probe() error = %v, want DeadlineExceeded", err)
}
if d.clients[0].closed.Load() {
t.Error("probe closed the shared connection when it gave up")
}
if err := p.probe(context.Background(), discard, testHost, noop); !errors.Is(err, errProbeBusy) {
t.Errorf("probe() while the abandoned one still waits = %v, want errProbeBusy", err)
}
if err := p.call(context.Background(), discard, testHost, noop); err != nil {
t.Errorf("call() on the kept connection = %v", err)
}
close(unblock)
deadline := time.Now().Add(time.Second)
for {
err := p.probe(context.Background(), discard, testHost, noop)
if err == nil {
break
}
if !errors.Is(err, errProbeBusy) || time.Now().After(deadline) {
t.Fatalf("probe() after the abandoned one returned = %v, want nil", err)
}
time.Sleep(time.Millisecond)
}
if d.dialCount() != 1 {
t.Errorf("dials = %d, want 1: nothing should have redialed", d.dialCount())
}
}
func TestHostPool_probe_dropsConnectionOnTransportError(t *testing.T) {
t.Parallel()
d := &fakeDialer{}
p := newHostPool(d.dial)
err := p.probe(context.Background(), discard, testHost, func(libvirtAPI) error { return io.ErrUnexpectedEOF })
if !errors.Is(err, io.ErrUnexpectedEOF) {
t.Fatalf("probe() error = %v, want io.ErrUnexpectedEOF passed through", err)
}
if !d.clients[0].closed.Load() {
t.Error("a transport error during a probe left the connection cached")
}
}
// --- SSH dial against an in-process server ---
func newKeyPair(t *testing.T) (ssh.Signer, []byte) {
@@ -614,6 +705,86 @@ func TestNew_missingSSHMaterialStillBuilds(t *testing.T) {
}
}
// The health check reports where a connection broke, so every stage of a
// real dial must come back tagged.
func TestLazySSH_dial_tagsTheFailingStage(t *testing.T) {
t.Parallel()
tests := []struct {
name string
mutate func(t *testing.T, f *sshFixture)
want dialStage
wantReason string
}{
{
name: "key material missing",
mutate: func(t *testing.T, f *sshFixture) { f.cfg.SSHKeyFile = filepath.Join(t.TempDir(), "absent") },
want: stageSSHMaterial,
wantReason: reasonSSHMaterialUnavailable,
},
{
name: "nothing listening",
mutate: func(t *testing.T, f *sshFixture) {
ln, err := net.Listen("tcp", "127.0.0.1:0")
if err != nil {
t.Fatal(err)
}
f.host.Address = ln.Addr().String()
ln.Close()
},
want: stageTCP,
wantReason: provider.ReasonUnreachable,
},
{
name: "host key not in known_hosts",
mutate: func(t *testing.T, f *sshFixture) {
impostor, _ := newKeyPair(t)
line := knownhosts.Line([]string{f.server.addr}, impostor.PublicKey())
if err := os.WriteFile(f.cfg.KnownHostsFile, []byte(line+"\n"), 0o600); err != nil {
t.Fatal(err)
}
},
want: stageSSHHandshake,
wantReason: reasonSSHHandshakeFailed,
},
{
name: "client key rejected",
mutate: func(t *testing.T, f *sshFixture) {
_, otherPEM := newKeyPair(t)
if err := os.WriteFile(f.keyFile, otherPEM, 0o600); err != nil {
t.Fatal(err)
}
},
want: stageSSHHandshake,
wantReason: reasonSSHHandshakeFailed,
},
{
name: "libvirt socket missing",
mutate: func(_ *testing.T, f *sshFixture) { f.host.Socket = "/nonexistent/libvirt-sock" },
want: stageLibvirtConnect,
wantReason: reasonLibvirtConnectFailed,
},
}
for _, tc := range tests {
t.Run(tc.name, func(t *testing.T) {
t.Parallel()
f := newSSHFixture(t, true)
tc.mutate(t, f)
ctx, cancel := context.WithTimeout(context.Background(), 5*time.Second)
defer cancel()
_, err := newLazySSH(f.cfg).dial(ctx, f.host)
var de *dialError
if !errors.As(err, &de) || de.stage != tc.want {
t.Fatalf("dial() error = %v, want a %s dialError", err, tc.want)
}
if got := checkReason(err); got != tc.wantReason {
t.Errorf("checkReason() = %s, want %s", got, tc.wantReason)
}
})
}
}
func TestKeepalive_closesClientWhenServerStalls(t *testing.T) {
t.Parallel()
f := newSSHFixture(t, false)
internal/provider/kvm/fake_test.go
+54
View File
@@ -28,6 +28,12 @@ type fakeAPI struct {
// uploadHook, when set, sees every StorageVolUpload before the fake
// reads the stream; it can return an error to simulate a broken upload.
uploadHook func(vol string, r io.Reader) error
// hostname, libVersion, poolCapacity and poolStates answer the health
// check's RPCs. A pool missing from poolStates is running.
hostname string
libVersion uint64
poolCapacity uint64
poolStates map[string]libvirt.StoragePoolState
}
type fakeVol struct {
@@ -53,6 +59,11 @@ func newFakeAPI(pools ...string) *fakeAPI {
domains: map[string]*fakeDomain{},
fail: map[string]error{},
failOnce: map[string]error{},
hostname: "fake-host",
libVersion: 10002000,
poolCapacity: 100 << 30,
poolStates: map[string]libvirt.StoragePoolState{},
}
for _, p := range pools {
f.pools[p] = map[string]*fakeVol{}
@@ -409,3 +420,46 @@ func (f *fakeAPI) StorageVolDelete(vol libvirt.StorageVol, _ libvirt.StorageVolD
delete(f.pools[vol.Pool], vol.Name)
return nil
}
// --- host ---
func (f *fakeAPI) ConnectGetLibVersion() (uint64, error) {
f.mu.Lock()
defer f.mu.Unlock()
if err := f.enter("ConnectGetLibVersion"); err != nil {
return 0, err
}
return f.libVersion, nil
}
func (f *fakeAPI) ConnectGetHostname() (string, error) {
f.mu.Lock()
defer f.mu.Unlock()
if err := f.enter("ConnectGetHostname"); err != nil {
return "", err
}
return f.hostname, nil
}
// StoragePoolGetInfo reports the pool's state and poolCapacity minus the
// volumes' capacities as available space.
func (f *fakeAPI) StoragePoolGetInfo(pool libvirt.StoragePool) (uint8, uint64, uint64, uint64, error) {
f.mu.Lock()
defer f.mu.Unlock()
if err := f.enter("StoragePoolGetInfo"); err != nil {
return 0, 0, 0, 0, err
}
vols, ok := f.pools[pool.Name]
if !ok {
return 0, 0, 0, 0, lvError(libvirt.ErrNoStoragePool, "no pool %s", pool.Name)
}
var used uint64
for _, v := range vols {
used += uint64(v.capacity)
}
state := libvirt.StoragePoolRunning
if s, ok := f.poolStates[pool.Name]; ok {
state = s
}
return uint8(state), f.poolCapacity, used, f.poolCapacity - used, nil
}
internal/provider/kvm/kvm_integration_test.go
+30
View File
@@ -347,6 +347,36 @@ func TestKVM_lifecycle(t *testing.T) {
env.keep = false
}
// TestKVMCheck_realHost runs the provider health check against the host.
// It creates nothing: it must reach the host and find the pool running.
func TestKVMCheck_realHost(t *testing.T) {
env := loadIntegrationEnv(t)
log := integrationLogger(t, env.logLevel)
ctx, cancel := context.WithTimeout(logr.NewContext(context.Background(), log), time.Minute)
defer cancel()
prov, err := New(ctx, env.cfg)
if err != nil {
t.Fatalf("New: %v", err)
}
got := prov.(*Provider).Check(ctx)
if len(got) != 1 {
t.Fatalf("Check returned %d targets, want 1", len(got))
}
ts := got[0]
t.Logf("%s (%s): ready=%v reason=%s latency=%s details=%v",
ts.Name, ts.Address, ts.Ready, ts.Reason, ts.Latency.Round(time.Millisecond), ts.Details)
if !ts.Ready {
t.Fatalf("check failed: %s: %s", ts.Reason, ts.Message)
}
for _, key := range []string{"hostname", "libvirtVersion", "poolState", "poolAvailable"} {
if ts.Details[key] == "" {
t.Errorf("details[%s] is empty, want what the host reported", key)
}
}
}
// waitForRunning polls Get until the instance is Running with an IP or ctx
// ends. Transient errors are retried; anything else fails the test.
func waitForRunning(ctx context.Context, t *testing.T, p *Provider, id string) *provider.Instance {
internal/provider/kvm/trace.go
+21
View File
@@ -209,3 +209,24 @@ func (t *tracingAPI) StorageVolDelete(vol libvirt.StorageVol, flags libvirt.Stor
c.done(err)
return err
}
func (t *tracingAPI) ConnectGetLibVersion() (uint64, error) {
c := t.begin("ConnectGetLibVersion")
v, err := t.inner.ConnectGetLibVersion()
c.done(err, "version", formatLibVersion(v))
return v, err
}
func (t *tracingAPI) ConnectGetHostname() (string, error) {
c := t.begin("ConnectGetHostname")
h, err := t.inner.ConnectGetHostname()
c.done(err, "hostname", h)
return h, err
}
func (t *tracingAPI) StoragePoolGetInfo(pool libvirt.StoragePool) (uint8, uint64, uint64, uint64, error) {
c := t.begin("StoragePoolGetInfo", "pool", pool.Name)
state, capacity, allocation, available, err := t.inner.StoragePoolGetInfo(pool)
c.done(err, "state", poolStateName(libvirt.StoragePoolState(state)), "capacity", capacity, "available", available)
return state, capacity, allocation, available, err
}
internal/provider/kvm/trace_test.go
+3
View File
@@ -103,6 +103,9 @@ func TestTracingAPI_logsEveryMethod(t *testing.T) {
return api.StorageVolDownload(overlay, io.Discard, 0, 0, 0)
}},
{"StorageVolDelete", func(api libvirtAPI) error { return api.StorageVolDelete(overlay, 0) }},
{"ConnectGetLibVersion", func(api libvirtAPI) error { _, err := api.ConnectGetLibVersion(); return err }},
{"ConnectGetHostname", func(api libvirtAPI) error { _, err := api.ConnectGetHostname(); return err }},
{"StoragePoolGetInfo", func(api libvirtAPI) error { _, _, _, _, err := api.StoragePoolGetInfo(pool); return err }},
}
if n := reflect.TypeOf((*libvirtAPI)(nil)).Elem().NumMethod(); n != len(cases) {
t.Fatalf("libvirtAPI has %d methods, this test covers %d; trace the new one", n, len(cases))
internal/provider/kvm2/backend.go
+421
View File
@@ -0,0 +1,421 @@
package kvm2
import (
"context"
"errors"
"fmt"
"io"
"sync"
"time"
"github.com/digitalocean/go-libvirt"
"gitea.home.hrajfrisbee.cz/kacerr/egress-proxies-operator/internal/provider"
)
// backend is the seam over go-libvirt (spec entry 17): exactly the calls the
// provider uses, each with ctx first, since go-libvirt's own methods take
// none and block. The production adapter is libvirtBackend; tests use an
// in-memory fake. Nothing outside this package sees it.
type backend interface {
DomainLookupByName(ctx context.Context, name string) (libvirt.Domain, error)
DomainDefineXML(ctx context.Context, xml string) (libvirt.Domain, error)
DomainCreate(ctx context.Context, dom libvirt.Domain) error
DomainDestroy(ctx context.Context, dom libvirt.Domain) error
DomainUndefineFlags(ctx context.Context, dom libvirt.Domain, flags libvirt.DomainUndefineFlagsValues) error
DomainGetState(ctx context.Context, dom libvirt.Domain, flags uint32) (state int32, reason int32, err error)
DomainGetMetadata(ctx context.Context, dom libvirt.Domain, typ int32, uri libvirt.OptString, flags libvirt.DomainModificationImpact) (string, error)
DomainInterfaceAddresses(ctx context.Context, dom libvirt.Domain, source uint32, flags uint32) ([]libvirt.DomainInterface, error)
ConnectListAllDomains(ctx context.Context, needResults int32, flags libvirt.ConnectListAllDomainsFlags) ([]libvirt.Domain, uint32, error)
StoragePoolLookupByName(ctx context.Context, name string) (libvirt.StoragePool, error)
StoragePoolRefresh(ctx context.Context, pool libvirt.StoragePool, flags uint32) error
StoragePoolGetInfo(ctx context.Context, pool libvirt.StoragePool) (state uint8, capacity, allocation, available uint64, err error)
StorageVolLookupByName(ctx context.Context, pool libvirt.StoragePool, name string) (libvirt.StorageVol, error)
StorageVolGetPath(ctx context.Context, vol libvirt.StorageVol) (string, error)
StorageVolCreateXML(ctx context.Context, pool libvirt.StoragePool, xml string, flags libvirt.StorageVolCreateFlags) (libvirt.StorageVol, error)
StorageVolUpload(ctx context.Context, vol libvirt.StorageVol, r io.Reader, offset, length uint64, flags libvirt.StorageVolUploadFlags) error
StorageVolDownload(ctx context.Context, vol libvirt.StorageVol, w io.Writer, offset, length uint64, flags libvirt.StorageVolDownloadFlags) error
StorageVolDelete(ctx context.Context, vol libvirt.StorageVol, flags libvirt.StorageVolDeleteFlags) error
ConnectGetLibVersion(ctx context.Context) (uint64, error)
ConnectGetHostname(ctx context.Context) (string, error)
Close() error
}
// liveness is implemented by backends whose connection can die underneath
// them. The host table drops a backend whose dead channel is closed. It is
// deliberately not part of backend: the fake needs it, the seam does not.
type liveness interface {
dead() <-chan struct{}
}
func isDead(b backend) bool {
l, ok := b.(liveness)
if !ok {
return false
}
select {
case <-l.dead():
return true
default:
return false
}
}
// budgets are the per-call deadlines of spec entry 18, independent of the
// caller's ctx, which by contract carries none.
type budgets struct {
// dial bounds TCP connect, SSH handshake and libvirt connect together.
dial time.Duration
// rpc bounds every ordinary call.
rpc time.Duration
// create bounds DomainCreate, which waits for QEMU to start.
create time.Duration
// seedUpload bounds a StorageVolUpload of at most smallUploadBytes.
seedUpload time.Duration
// transfer caps a large upload or a download; ctx may end it earlier.
transfer time.Duration
}
var defaultBudgets = budgets{
dial: 10 * time.Second,
rpc: 30 * time.Second,
create: 90 * time.Second,
seedUpload: 60 * time.Second,
transfer: 20 * time.Minute,
}
// smallUploadBytes separates a seed ISO (tens of KiB) from a base image
// (hundreds of MiB) for the upload budget.
const smallUploadBytes = 16 << 20
// closeBudget bounds the courtesy ConnectClose RPC in Close.
const closeBudget = 5 * time.Second
// transferGrace keeps the image download's own deadline inside the transfer
// budget. A stalled mirror then ends by its body failing, which the upload
// seam turns into a clean end of stream, instead of by the budget killing
// the socket, which go-libvirt's stream sender does not survive.
const transferGrace = time.Minute
func (b budgets) upload(length uint64) time.Duration {
if length <= smallUploadBytes {
return b.seedUpload
}
return b.transfer
}
// link is the liveness of one connection. dead is closed exactly once, by
// the first of: an abandoned call, a keepalive failure, go-libvirt noticing
// the socket went away, or Close. kill closes the transport, which is the
// only way to unblock a go-libvirt call; go-libvirt then fails every other
// call in flight on the connection with ErrInterrupted.
type link struct {
label string
kill func()
closed chan struct{}
mu sync.Mutex
reason string
}
func newLink(label string, kill func()) *link {
return &link{label: label, kill: kill, closed: make(chan struct{})}
}
func (l *link) dead() <-chan struct{} { return l.closed }
func (l *link) isDead() bool {
select {
case <-l.closed:
return true
default:
return false
}
}
func (l *link) deadReason() string {
l.mu.Lock()
defer l.mu.Unlock()
return l.reason
}
// markDead records why the connection is unusable, signals dead and then
// closes the transport. Later calls are no-ops. Signalling before killing,
// and killing outside the lock, is what keeps a kill that takes its time
// from holding up the callers waiting on dead or the next markDead: a
// transport close can block for as long as the kernel retransmits.
func (l *link) markDead(reason string) {
l.mu.Lock()
if l.isDead() {
l.mu.Unlock()
return
}
l.reason = reason
close(l.closed)
l.mu.Unlock()
if l.kill != nil {
l.kill()
}
}
// call executes a blocking go-libvirt call with a budget and returns its
// result, or Transient when the link is already dead, ctx ends, the budget
// runs out, or another caller kills the link meanwhile. Abandoning a call
// kills the link (spec entry 18); the goroutine running fn then returns on
// its own once go-libvirt sees the closed socket, and its result is dropped.
// Results travel through the channel so an abandoned call never writes
// into its caller's frame.
func call[T any](ctx context.Context, l *link, name string, budget time.Duration, fn func() (T, error)) (T, error) {
var zero T
if l.isDead() {
return zero, transientf("libvirt call %s on %s: connection is closed: %s", name, l.label, l.deadReason())
}
if err := ctx.Err(); err != nil {
return zero, fmt.Errorf("%w: libvirt call %s on %s not started: %w", provider.ErrTransient, name, l.label, err)
}
type result struct {
v T
err error
}
done := make(chan result, 1)
go func() {
v, err := fn()
done <- result{v, err}
}()
timer := time.NewTimer(budget)
defer timer.Stop()
select {
case r := <-done:
return r.v, r.err
case <-ctx.Done():
l.markDead(fmt.Sprintf("call %s abandoned: %v", name, ctx.Err()))
return zero, fmt.Errorf("%w: libvirt call %s on %s abandoned: %w", provider.ErrTransient, name, l.label, ctx.Err())
case <-timer.C:
l.markDead(fmt.Sprintf("call %s exceeded its %s budget", name, budget))
return zero, fmt.Errorf("%w: libvirt call %s on %s exceeded its %s budget: %w", provider.ErrTransient, name, l.label, budget, context.DeadlineExceeded)
case <-l.closed:
return zero, transientf("libvirt call %s on %s: connection closed while waiting: %s", name, l.label, l.deadReason())
}
}
// run is call for go-libvirt methods that return only an error.
func (l *link) run(ctx context.Context, name string, budget time.Duration, fn func() error) error {
_, err := call(ctx, l, name, budget, func() (struct{}, error) { return struct{}{}, fn() })
return err
}
// libvirtBackend is the production adapter: a go-libvirt client over one
// forwarded socket, with the budgets applied around every call. It never
// logs; the observe decorator does.
type libvirtBackend struct {
lv *libvirt.Libvirt
link *link
budgets budgets
closeOnce sync.Once
closeErr error
}
var _ backend = (*libvirtBackend)(nil)
// newLibvirtBackend wraps lv, whose socket runs over transport. Closing the
// transport is how a hung call is abandoned, so it must close the socket
// conn given to libvirt.New as well as whatever carries it.
func newLibvirtBackend(lv *libvirt.Libvirt, label string, transport io.Closer, b budgets) *libvirtBackend {
return &libvirtBackend{
lv: lv,
link: newLink(label, func() { _ = transport.Close() }),
budgets: b,
}
}
func (b *libvirtBackend) dead() <-chan struct{} { return b.link.dead() }
// connect runs the libvirt handshake under the dial budget and, once it
// holds, drops the link as soon as go-libvirt reports the socket gone.
func (b *libvirtBackend) connect(ctx context.Context) error {
if err := b.link.run(ctx, "Connect", b.budgets.dial, b.lv.Connect); err != nil {
return err
}
disconnected := b.lv.Disconnected()
go func() {
select {
case <-disconnected:
b.link.markDead("connection lost")
case <-b.link.closed:
}
}()
return nil
}
func (b *libvirtBackend) DomainLookupByName(ctx context.Context, name string) (libvirt.Domain, error) {
return call(ctx, b.link, "DomainLookupByName", b.budgets.rpc, func() (libvirt.Domain, error) { return b.lv.DomainLookupByName(name) })
}
func (b *libvirtBackend) DomainDefineXML(ctx context.Context, xml string) (libvirt.Domain, error) {
return call(ctx, b.link, "DomainDefineXML", b.budgets.rpc, func() (libvirt.Domain, error) { return b.lv.DomainDefineXML(xml) })
}
func (b *libvirtBackend) DomainCreate(ctx context.Context, dom libvirt.Domain) error {
return b.link.run(ctx, "DomainCreate", b.budgets.create, func() error { return b.lv.DomainCreate(dom) })
}
func (b *libvirtBackend) DomainDestroy(ctx context.Context, dom libvirt.Domain) error {
return b.link.run(ctx, "DomainDestroy", b.budgets.rpc, func() error { return b.lv.DomainDestroy(dom) })
}
func (b *libvirtBackend) DomainUndefineFlags(ctx context.Context, dom libvirt.Domain, flags libvirt.DomainUndefineFlagsValues) error {
return b.link.run(ctx, "DomainUndefineFlags", b.budgets.rpc, func() error { return b.lv.DomainUndefineFlags(dom, flags) })
}
func (b *libvirtBackend) DomainGetState(ctx context.Context, dom libvirt.Domain, flags uint32) (int32, int32, error) {
type stateReason struct{ state, reason int32 }
r, err := call(ctx, b.link, "DomainGetState", b.budgets.rpc, func() (stateReason, error) {
state, reason, err := b.lv.DomainGetState(dom, flags)
return stateReason{state, reason}, err
})
return r.state, r.reason, err
}
func (b *libvirtBackend) DomainGetMetadata(ctx context.Context, dom libvirt.Domain, typ int32, uri libvirt.OptString, flags libvirt.DomainModificationImpact) (string, error) {
return call(ctx, b.link, "DomainGetMetadata", b.budgets.rpc, func() (string, error) { return b.lv.DomainGetMetadata(dom, typ, uri, flags) })
}
func (b *libvirtBackend) DomainInterfaceAddresses(ctx context.Context, dom libvirt.Domain, source uint32, flags uint32) ([]libvirt.DomainInterface, error) {
return call(ctx, b.link, "DomainInterfaceAddresses", b.budgets.rpc, func() ([]libvirt.DomainInterface, error) {
return b.lv.DomainInterfaceAddresses(dom, source, flags)
})
}
func (b *libvirtBackend) ConnectListAllDomains(ctx context.Context, needResults int32, flags libvirt.ConnectListAllDomainsFlags) ([]libvirt.Domain, uint32, error) {
type listed struct {
doms []libvirt.Domain
n uint32
}
r, err := call(ctx, b.link, "ConnectListAllDomains", b.budgets.rpc, func() (listed, error) {
doms, n, err := b.lv.ConnectListAllDomains(needResults, flags)
return listed{doms, n}, err
})
return r.doms, r.n, err
}
func (b *libvirtBackend) StoragePoolLookupByName(ctx context.Context, name string) (libvirt.StoragePool, error) {
return call(ctx, b.link, "StoragePoolLookupByName", b.budgets.rpc, func() (libvirt.StoragePool, error) { return b.lv.StoragePoolLookupByName(name) })
}
func (b *libvirtBackend) StoragePoolRefresh(ctx context.Context, pool libvirt.StoragePool, flags uint32) error {
return b.link.run(ctx, "StoragePoolRefresh", b.budgets.rpc, func() error { return b.lv.StoragePoolRefresh(pool, flags) })
}
func (b *libvirtBackend) StoragePoolGetInfo(ctx context.Context, pool libvirt.StoragePool) (uint8, uint64, uint64, uint64, error) {
type info struct {
state uint8
capacity, allocation, available uint64
}
r, err := call(ctx, b.link, "StoragePoolGetInfo", b.budgets.rpc, func() (info, error) {
state, capacity, allocation, available, err := b.lv.StoragePoolGetInfo(pool)
return info{state, capacity, allocation, available}, err
})
return r.state, r.capacity, r.allocation, r.available, err
}
func (b *libvirtBackend) StorageVolLookupByName(ctx context.Context, pool libvirt.StoragePool, name string) (libvirt.StorageVol, error) {
return call(ctx, b.link, "StorageVolLookupByName", b.budgets.rpc, func() (libvirt.StorageVol, error) { return b.lv.StorageVolLookupByName(pool, name) })
}
func (b *libvirtBackend) StorageVolGetPath(ctx context.Context, vol libvirt.StorageVol) (string, error) {
return call(ctx, b.link, "StorageVolGetPath", b.budgets.rpc, func() (string, error) { return b.lv.StorageVolGetPath(vol) })
}
func (b *libvirtBackend) StorageVolCreateXML(ctx context.Context, pool libvirt.StoragePool, xml string, flags libvirt.StorageVolCreateFlags) (libvirt.StorageVol, error) {
return call(ctx, b.link, "StorageVolCreateXML", b.budgets.rpc, func() (libvirt.StorageVol, error) { return b.lv.StorageVolCreateXML(pool, xml, flags) })
}
func (b *libvirtBackend) StorageVolUpload(ctx context.Context, vol libvirt.StorageVol, r io.Reader, offset, length uint64, flags libvirt.StorageVolUploadFlags) error {
src := &eofReader{r: r}
err := b.link.run(ctx, "StorageVolUpload", b.budgets.upload(length), func() error {
return b.lv.StorageVolUpload(vol, src, offset, length, flags)
})
if err == nil {
if srcErr := src.sourceErr(); srcErr != nil {
return fmt.Errorf("reading the upload source for %s: %w", vol.Name, srcErr)
}
}
return err
}
// eofReader hides a source error from go-libvirt's stream sender and keeps
// it for the caller. SendStream answers a non-EOF error with a StatusError
// packet and then blocks forever handing its result to an unbuffered
// channel that requestStream no longer reads (go-libvirt rpc.go), which
// wedges the call and leaks the goroutine along with its 4 MiB buffer;
// closing the socket does not help, because both ends are parked on
// channel sends rather than on I/O. io.EOF is the one outcome that ends
// the stream the normal way. The field is guarded because an abandoned
// call returns while go-libvirt is still reading.
type eofReader struct {
r io.Reader
mu sync.Mutex
err error
}
func (e *eofReader) Read(b []byte) (int, error) {
n, err := e.r.Read(b)
if err == nil || errors.Is(err, io.EOF) {
return n, err
}
e.mu.Lock()
if e.err == nil {
e.err = err
}
e.mu.Unlock()
return n, io.EOF
}
func (e *eofReader) sourceErr() error {
e.mu.Lock()
defer e.mu.Unlock()
return e.err
}
func (b *libvirtBackend) StorageVolDownload(ctx context.Context, vol libvirt.StorageVol, w io.Writer, offset, length uint64, flags libvirt.StorageVolDownloadFlags) error {
return b.link.run(ctx, "StorageVolDownload", b.budgets.transfer, func() error {
return b.lv.StorageVolDownload(vol, w, offset, length, flags)
})
}
func (b *libvirtBackend) StorageVolDelete(ctx context.Context, vol libvirt.StorageVol, flags libvirt.StorageVolDeleteFlags) error {
return b.link.run(ctx, "StorageVolDelete", b.budgets.rpc, func() error { return b.lv.StorageVolDelete(vol, flags) })
}
func (b *libvirtBackend) ConnectGetLibVersion(ctx context.Context) (uint64, error) {
return call(ctx, b.link, "ConnectGetLibVersion", b.budgets.rpc, b.lv.ConnectGetLibVersion)
}
func (b *libvirtBackend) ConnectGetHostname(ctx context.Context) (string, error) {
return call(ctx, b.link, "ConnectGetHostname", b.budgets.rpc, b.lv.ConnectGetHostname)
}
// Close sends libvirt's ConnectClose when the link is still live, bounded
// because a hung host would otherwise pin the caller, then kills the link,
// which closes the socket and the SSH client. Repeat calls return the first
// result.
func (b *libvirtBackend) Close() error {
b.closeOnce.Do(func() {
if b.link.isDead() {
return
}
done := make(chan error, 1)
go func() { done <- b.lv.Disconnect() }()
timer := time.NewTimer(closeBudget)
defer timer.Stop()
select {
case b.closeErr = <-done:
case <-timer.C:
b.closeErr = transientf("libvirt disconnect on %s exceeded its %s budget", b.link.label, closeBudget)
case <-b.link.closed:
}
b.link.markDead("closed")
})
return b.closeErr
}
internal/provider/kvm2/backend_test.go
+369
View File
@@ -0,0 +1,369 @@
package kvm2
import (
"context"
"errors"
"io"
"strings"
"sync/atomic"
"testing"
"time"
"github.com/digitalocean/go-libvirt"
"github.com/digitalocean/go-libvirt/libvirttest"
"gitea.home.hrajfrisbee.cz/kacerr/egress-proxies-operator/internal/provider"
)
// blockingFn returns a func that blocks until release is closed, plus a
// counter of how often it was started.
func blockingFn(t *testing.T) (fn func() error, started *atomic.Int32) {
t.Helper()
release := make(chan struct{})
t.Cleanup(func() { close(release) })
started = &atomic.Int32{}
return func() error {
started.Add(1)
<-release
return nil
}, started
}
func TestCall_deadlineClosesConnectionAndReturnsTransient(t *testing.T) {
t.Parallel()
var killed atomic.Int32
l := newLink("beelink", func() { killed.Add(1) })
fn, started := blockingFn(t)
start := time.Now()
err := l.run(context.Background(), "DomainCreate", 50*time.Millisecond, fn)
elapsed := time.Since(start)
if !errors.Is(err, provider.ErrTransient) {
t.Fatalf("run() error = %v, want ErrTransient", err)
}
if !errors.Is(err, context.DeadlineExceeded) {
t.Errorf("run() error = %v, want it to wrap context.DeadlineExceeded", err)
}
if !strings.Contains(err.Error(), "DomainCreate") || !strings.Contains(err.Error(), "beelink") {
t.Errorf("run() error = %q, want the call name and host in it", err)
}
if elapsed > time.Second {
t.Errorf("run() took %v, want about the 50ms budget", elapsed)
}
if killed.Load() != 1 {
t.Errorf("kill hook called %d times, want 1", killed.Load())
}
if !l.isDead() {
t.Error("link not dead after an abandoned call")
}
if !strings.Contains(l.deadReason(), "budget") {
t.Errorf("deadReason() = %q, want the budget mentioned", l.deadReason())
}
start = time.Now()
err = l.run(context.Background(), "DomainGetState", time.Minute, fn)
if !errors.Is(err, provider.ErrTransient) {
t.Fatalf("second run() error = %v, want ErrTransient", err)
}
if time.Since(start) > 100*time.Millisecond {
t.Errorf("second run() took %v, want immediate", time.Since(start))
}
if started.Load() != 1 {
t.Errorf("fn started %d times, want 1: a dead link must not touch the socket", started.Load())
}
if killed.Load() != 1 {
t.Errorf("kill hook called %d times, want still 1", killed.Load())
}
}
func TestCall_cancelUnblocksRPC(t *testing.T) {
t.Parallel()
var killed atomic.Int32
l := newLink("beelink", func() { killed.Add(1) })
fn, _ := blockingFn(t)
ctx, cancel := context.WithCancel(context.Background())
go func() {
time.Sleep(10 * time.Millisecond)
cancel()
}()
start := time.Now()
err := l.run(ctx, "ConnectListAllDomains", time.Minute, fn)
if time.Since(start) > 100*time.Millisecond {
t.Errorf("run() took %v after cancel, want under 100ms", time.Since(start))
}
if !errors.Is(err, provider.ErrTransient) || !errors.Is(err, context.Canceled) {
t.Errorf("run() error = %v, want ErrTransient wrapping context.Canceled", err)
}
if killed.Load() != 1 {
t.Errorf("kill hook called %d times, want 1", killed.Load())
}
}
func TestCall_alreadyCancelledDoesNotStart(t *testing.T) {
t.Parallel()
l := newLink("beelink", nil)
fn, started := blockingFn(t)
ctx, cancel := context.WithCancel(context.Background())
cancel()
err := l.run(ctx, "DomainGetState", time.Minute, fn)
if !errors.Is(err, provider.ErrTransient) || !errors.Is(err, context.Canceled) {
t.Errorf("run() error = %v, want ErrTransient wrapping context.Canceled", err)
}
if started.Load() != 0 {
t.Error("fn started although ctx was already cancelled")
}
if l.isDead() {
t.Error("link marked dead although nothing was abandoned")
}
}
func TestCall_returnsFnResult(t *testing.T) {
t.Parallel()
l := newLink("beelink", nil)
want := libvirt.Error{Code: uint32(libvirt.ErrNoDomain), Message: "gone"}
err := l.run(context.Background(), "DomainLookupByName", time.Second, func() error { return want })
if !hasCode(err, libvirt.ErrNoDomain) {
t.Errorf("run() error = %v, want the libvirt error through unchanged", err)
}
if l.isDead() {
t.Error("a backend error must not kill the link")
}
}
func TestCall_killedByAnotherCallerFailsTransient(t *testing.T) {
t.Parallel()
l := newLink("beelink", nil)
fn, _ := blockingFn(t)
done := make(chan error, 1)
go func() { done <- l.run(context.Background(), "DomainGetState", time.Minute, fn) }()
time.Sleep(10 * time.Millisecond)
l.markDead("keepalive failed")
select {
case err := <-done:
if !errors.Is(err, provider.ErrTransient) || !strings.Contains(err.Error(), "keepalive failed") {
t.Errorf("run() error = %v, want ErrTransient with the death reason", err)
}
case <-time.After(time.Second):
t.Fatal("run() did not return after the link was killed")
}
}
// TestMarkDead_slowKillDoesNotBlockCallers pins the guarantee the pooled
// connection rests on: a kill that takes its time — closing a transport can
// block for as long as the kernel retransmits to a host that went away —
// holds up neither the callers waiting on dead nor the next markDead. The
// call's own fn signals when it is in flight, so the parked caller is
// genuinely parked rather than refused by the dead check on the way in.
func TestMarkDead_slowKillDoesNotBlockCallers(t *testing.T) {
t.Parallel()
release := make(chan struct{})
t.Cleanup(func() { close(release) })
killing := make(chan struct{})
var killed atomic.Int32
l := newLink("beelink", func() {
killed.Add(1)
close(killing)
<-release
})
inflight := make(chan struct{})
waiting := make(chan error, 1)
go func() {
waiting <- l.run(context.Background(), "StorageVolUpload", time.Minute, func() error {
close(inflight)
<-release
return nil
})
}()
<-inflight
go l.markDead("keepalive failed")
<-killing
select {
case <-l.dead():
case <-time.After(time.Second):
t.Fatal("link not signalled dead while its kill is still running")
}
select {
case err := <-waiting:
if !errors.Is(err, provider.ErrTransient) || !strings.Contains(err.Error(), "keepalive failed") {
t.Errorf("parked call error = %v, want ErrTransient with the death reason", err)
}
case <-time.After(time.Second):
t.Fatal("parked call not released while the kill is still running")
}
second := make(chan struct{})
go func() {
l.markDead("closed")
close(second)
}()
select {
case <-second:
case <-time.After(time.Second):
t.Fatal("second markDead blocked behind the running kill")
}
if killed.Load() != 1 {
t.Errorf("kill hook called %d times, want 1", killed.Load())
}
if !strings.Contains(l.deadReason(), "keepalive failed") {
t.Errorf("deadReason() = %q, want the first reason", l.deadReason())
}
}
// TestEOFReader_sourceErrorNeverReachesTheStream pins the one property the
// upload seam needs: go-libvirt's stream sender must never see an error
// other than io.EOF, because it answers one by parking forever on a channel
// nothing reads. The real error has to survive for the caller.
func TestEOFReader_sourceErrorNeverReachesTheStream(t *testing.T) {
t.Parallel()
boom := errors.New("connection reset by peer")
tests := []struct {
name string
reader io.Reader
wantN int
wantErr error
}{
{name: "error after data", reader: io.MultiReader(strings.NewReader("abc"), errReader{boom}), wantN: 3, wantErr: boom},
{name: "error with no data", reader: errReader{boom}, wantErr: boom},
{name: "clean eof passes through", reader: strings.NewReader("abc"), wantN: 3},
}
for _, tc := range tests {
t.Run(tc.name, func(t *testing.T) {
t.Parallel()
e := &eofReader{r: tc.reader}
n, err := io.Copy(io.Discard, e)
if err != nil {
t.Fatalf("io.Copy through eofReader = %v, want nil: the stream sender must only ever see io.EOF", err)
}
if n != int64(tc.wantN) {
t.Errorf("copied %d bytes, want %d", n, tc.wantN)
}
if !errors.Is(e.sourceErr(), tc.wantErr) {
t.Errorf("sourceErr() = %v, want %v", e.sourceErr(), tc.wantErr)
}
})
}
}
func TestEOFReader_keepsTheFirstError(t *testing.T) {
t.Parallel()
first, second := errors.New("first"), errors.New("second")
e := &eofReader{r: io.MultiReader(errReader{first}, errReader{second})}
if _, err := io.Copy(io.Discard, e); err != nil {
t.Fatalf("io.Copy through eofReader = %v, want nil", err)
}
if !errors.Is(e.sourceErr(), first) {
t.Errorf("sourceErr() = %v, want the first error", e.sourceErr())
}
}
type errReader struct{ err error }
func (r errReader) Read([]byte) (int, error) { return 0, r.err }
func TestBudgets_upload(t *testing.T) {
t.Parallel()
tests := []struct {
name string
length uint64
want time.Duration
}{
{name: "seed sized", length: 400 << 10, want: defaultBudgets.seedUpload},
{name: "exactly the limit", length: smallUploadBytes, want: defaultBudgets.seedUpload},
{name: "one over the limit", length: smallUploadBytes + 1, want: defaultBudgets.transfer},
{name: "base image", length: 435 << 20, want: defaultBudgets.transfer},
}
for _, tc := range tests {
t.Run(tc.name, func(t *testing.T) {
t.Parallel()
if got := defaultBudgets.upload(tc.length); got != tc.want {
t.Errorf("upload(%d) = %v, want %v", tc.length, got, tc.want)
}
})
}
}
// mockAdapter wires a libvirtBackend to go-libvirt's own mock server over a
// net.Pipe, so the adapter is exercised with the real RPC client.
func mockAdapter(t *testing.T, b budgets) *libvirtBackend {
t.Helper()
m := libvirttest.New()
conn, err := m.Dial()
if err != nil {
t.Fatalf("mock dial: %v", err)
}
ad := newLibvirtBackend(libvirt.New(conn), "mock", conn, b)
if err := ad.connect(context.Background()); err != nil {
t.Fatalf("connect: %v", err)
}
t.Cleanup(func() { _ = ad.Close() })
return ad
}
func TestLibvirtBackend_roundTripThroughMock(t *testing.T) {
t.Parallel()
ad := mockAdapter(t, defaultBudgets)
v, err := ad.ConnectGetLibVersion(context.Background())
if err != nil {
t.Fatalf("ConnectGetLibVersion() error = %v", err)
}
if v == 0 {
t.Error("ConnectGetLibVersion() = 0, want the mock's version")
}
pool, err := ad.StoragePoolLookupByName(context.Background(), "default")
if err != nil {
t.Fatalf("StoragePoolLookupByName() error = %v", err)
}
if pool.Name == "" {
t.Error("StoragePoolLookupByName() returned an empty pool")
}
if isDead(ad) {
t.Error("adapter dead after successful calls")
}
}
func TestLibvirtBackend_unansweredCallIsAbandoned(t *testing.T) {
t.Parallel()
b := defaultBudgets
b.rpc = 100 * time.Millisecond
ad := mockAdapter(t, b)
// The mock does not implement ConnectGetHostname, so it never replies.
_, err := ad.ConnectGetHostname(context.Background())
if !errors.Is(err, provider.ErrTransient) || !errors.Is(err, context.DeadlineExceeded) {
t.Fatalf("ConnectGetHostname() error = %v, want ErrTransient wrapping DeadlineExceeded", err)
}
if !isDead(ad) {
t.Error("adapter not dead after an abandoned call")
}
select {
case <-ad.lv.Disconnected():
case <-time.After(time.Second):
t.Error("go-libvirt did not notice the closed socket")
}
if _, err := ad.ConnectGetLibVersion(context.Background()); !errors.Is(err, provider.ErrTransient) {
t.Errorf("call on dead adapter error = %v, want ErrTransient", err)
}
if err := ad.Close(); err != nil {
t.Errorf("Close() on dead adapter error = %v, want nil", err)
}
}
func TestLibvirtBackend_closeIsIdempotent(t *testing.T) {
t.Parallel()
ad := mockAdapter(t, defaultBudgets)
first := ad.Close()
second := ad.Close()
if first != nil || second != nil {
t.Errorf("Close() errors = %v, %v, want nil, nil", first, second)
}
if !isDead(ad) {
t.Error("adapter not dead after Close")
}
if _, err := ad.ConnectGetLibVersion(context.Background()); !errors.Is(err, provider.ErrTransient) {
t.Errorf("call after Close error = %v, want ErrTransient", err)
}
}
internal/provider/kvm2/check.go
+189
View File
@@ -0,0 +1,189 @@
package kvm2
import (
"context"
"errors"
"fmt"
"strings"
"sync"
"time"
"github.com/digitalocean/go-libvirt"
"golang.org/x/crypto/ssh/knownhosts"
"gitea.home.hrajfrisbee.cz/kacerr/egress-proxies-operator/internal/provider"
)
var _ provider.Checker = (*Provider)(nil)
// Reasons only this provider can tell apart (spec entry 16). Each names a
// failure an admin fixes somewhere other than where the shared reasons
// point.
const (
// reasonCredentialsUnavailable: the key or known_hosts file is missing
// or unparsable; fix the Secret in the cluster, not the host.
reasonCredentialsUnavailable = "CredentialsUnavailable"
// reasonPoolNotFound: the configured pool is not defined on the host;
// compare with `virsh pool-list --all`.
reasonPoolNotFound = "PoolNotFound"
// reasonPoolNotActive: the pool exists but is not running; `virsh
// pool-start <pool>` (and autostart) brings it back.
reasonPoolNotActive = "PoolNotActive"
)
// Check verifies every configured host concurrently, each on a fresh
// dedicated connection: dial, libvirt version, hostname, pool lookup, pool
// info, close. The pooled connection is never touched, so a check that runs
// out of time closes only its own connection and cannot fail a reconcile's
// call in flight. Every target gets a verdict, in config order.
func (p *Provider) Check(ctx context.Context) []provider.TargetStatus {
out := make([]provider.TargetStatus, len(p.cfg.Hosts))
var wg sync.WaitGroup
for i, host := range p.cfg.Hosts {
wg.Go(func() { out[i] = p.checkHost(ctx, host) })
}
wg.Wait()
return out
}
func (p *Provider) checkHost(ctx context.Context, host provider.KVMHost) provider.TargetStatus {
start := time.Now()
log := p.logger(ctx).WithValues("host", host.Name)
ts := provider.TargetStatus{
Name: host.Name,
Address: host.Address,
Details: map[string]string{"user": host.User, "pool": host.Pool},
}
err := p.probeHost(ctx, host, ts.Details)
ts.Latency = time.Since(start)
switch {
case err == nil:
ts.Ready = true
ts.Reason = provider.ReasonReachable
default:
var inactive *poolNotActiveError
switch {
case ctx.Err() != nil:
// The round ran out of time, whichever stage was running: the
// error's shape (a dial that saw the ctx end, an abandoned
// call) says nothing the admin should act on.
ts.Reason = provider.ReasonTimeout
case errors.As(err, &inactive):
ts.Reason = reasonPoolNotActive
default:
ts.Reason = checkReason(err)
}
ts.Message = err.Error()
}
log.V(1).Info("kvm2 host checked", "ready", ts.Ready, "reason", ts.Reason, "latency", ts.Latency)
return ts
}
// probeHost runs the check sequence on its own connection and fills details
// as facts arrive, so a failure late in the sequence still reports what the
// host said before it.
func (p *Provider) probeHost(ctx context.Context, host provider.KVMHost, details map[string]string) error {
b, err := p.hosts.dialFresh(ctx, host)
if err != nil {
return err
}
defer func() { _ = b.Close() }()
version, err := b.ConnectGetLibVersion(ctx)
if err != nil {
return err
}
details["libvirtVersion"] = formatLibVersion(version)
hostname, err := b.ConnectGetHostname(ctx)
if err != nil {
return err
}
details["hostname"] = hostname
pool, err := b.StoragePoolLookupByName(ctx, host.Pool)
if err != nil {
return err
}
state, capacity, _, available, err := b.StoragePoolGetInfo(ctx, pool)
if err != nil {
return err
}
details["poolState"] = poolStateName(state)
details["poolCapacityGiB"] = formatGiB(capacity)
details["poolAvailableGiB"] = formatGiB(available)
if libvirt.StoragePoolState(state) != libvirt.StoragePoolRunning {
return &poolNotActiveError{pool: host.Pool, state: poolStateName(state)}
}
return nil
}
// poolNotActiveError is the one verdict that is not an error from the
// backend: the pool answered, but is not usable.
type poolNotActiveError struct {
pool string
state string
}
func (e *poolNotActiveError) Error() string {
return fmt.Sprintf("storage pool %q is %s", e.pool, e.state)
}
// checkReason maps a failed probe to a reason once checkHost has ruled out
// the round's ctx ending. A dial failure is judged by its stage, so a TCP
// connect or an SSH handshake that ran out of the dial budget on its own is
// Unreachable, not Timeout: the host did not answer. Timeout is for a
// budget expiring once the connection was up.
func checkReason(err error) string {
var de *dialError
if errors.As(err, &de) {
switch de.stage {
case stageCredentials:
return reasonCredentialsUnavailable
case stageSSH:
var keyErr *knownhosts.KeyError
if errors.As(de.err, &keyErr) || strings.Contains(de.err.Error(), sshAuthFailed) {
return provider.ReasonUnauthorized
}
}
return provider.ReasonUnreachable
}
switch {
case errors.Is(err, context.DeadlineExceeded), errors.Is(err, context.Canceled):
return provider.ReasonTimeout
case hasCode(err, libvirt.ErrAccessDenied), hasCode(err, libvirt.ErrAuthFailed):
return provider.ReasonForbidden
case hasCode(err, libvirt.ErrNoStoragePool):
return reasonPoolNotFound
}
if _, ok := libvirtCode(err); ok {
return provider.ReasonAPIError
}
return provider.ReasonUnreachable
}
// formatLibVersion decodes ConnectGetLibVersion's major*1_000_000 +
// minor*1_000 + micro.
func formatLibVersion(v uint64) string {
return fmt.Sprintf("%d.%d.%d", v/1_000_000, v/1_000%1_000, v%1_000)
}
// poolStateName uses virsh's words for a pool state.
func poolStateName(state uint8) string {
switch libvirt.StoragePoolState(state) {
case libvirt.StoragePoolInactive:
return "inactive"
case libvirt.StoragePoolBuilding:
return "building"
case libvirt.StoragePoolRunning:
return "running"
case libvirt.StoragePoolDegraded:
return "degraded"
case libvirt.StoragePoolInaccessible:
return "inaccessible"
default:
return fmt.Sprintf("state %d", state)
}
}
func formatGiB(bytes uint64) string {
return fmt.Sprintf("%.1f", float64(bytes)/(1<<30))
}
internal/provider/kvm2/check_test.go
+579
View File
@@ -0,0 +1,579 @@
package kvm2
import (
"context"
"errors"
"fmt"
"os"
"path/filepath"
"slices"
"strings"
"sync/atomic"
"testing"
"time"
"github.com/digitalocean/go-libvirt"
"github.com/go-logr/logr/funcr"
"golang.org/x/crypto/ssh/knownhosts"
logf "sigs.k8s.io/controller-runtime/pkg/log"
"gitea.home.hrajfrisbee.cz/kacerr/egress-proxies-operator/internal/provider"
)
// checkContext is captureContext with a sink that tolerates the concurrent
// per-host log lines of Check.
func checkContext() (context.Context, *logCapture) {
logs := &logCapture{}
log := funcr.New(func(prefix, args string) {
logs.mu.Lock()
defer logs.mu.Unlock()
logs.lines = append(logs.lines, prefix+" "+args)
}, funcr.Options{Verbosity: 2})
return logf.IntoContext(context.Background(), log), logs
}
// checkDialer hands out a distinct fake per host name, fails the hosts in
// errs, counts every dial, and, with block set, waits for ctx to end and
// then fails the way dialSSHSocket does when the TCP connect sees the ctx
// end: a dialError at the tcp stage wrapping the ctx's error.
type checkDialer struct {
backends map[string]*fakeBackend
errs map[string]error
block bool
dials atomic.Int32
}
func (d *checkDialer) dial(ctx context.Context, host provider.KVMHost) (backend, error) {
d.dials.Add(1)
if d.block {
<-ctx.Done()
return nil, &dialError{stage: stageTCP, host: host.Name, err: fmt.Errorf("dial tcp %s: %w", host.Address, ctx.Err())}
}
if err := d.errs[host.Name]; err != nil {
return nil, err
}
b, ok := d.backends[host.Name]
if !ok {
return nil, fmt.Errorf("checkDialer: no backend for %s", host.Name)
}
return b, nil
}
func threeHosts() provider.KVMConfig {
cfg := twoHosts()
cfg.Hosts = append(cfg.Hosts, provider.KVMHost{Name: "lab3", Address: "lab3:22", User: "ops", Bridge: "br0", Pool: "ssd"})
return cfg
}
func fakesFor(cfg provider.KVMConfig) map[string]*fakeBackend {
out := make(map[string]*fakeBackend, len(cfg.Hosts))
for _, h := range cfg.Hosts {
f := newFakeBackend()
f.hostname = h.Name + ".home.lan"
if h.Pool != "" {
f.pools[h.Pool] = newFakePool()
}
out[h.Name] = f
}
return out
}
func detailKeys(d map[string]string) []string {
keys := make([]string, 0, len(d))
for k := range d {
keys = append(keys, k)
}
slices.Sort(keys)
return keys
}
func TestCheck_oneEntryPerHostInConfigOrder(t *testing.T) {
t.Parallel()
cfg := threeHosts()
d := &checkDialer{backends: fakesFor(cfg), errs: map[string]error{"lab2": errors.New("dial tcp 192.0.2.2:22: connect: no route to host")}}
p := newProvider(testProvider, cfg, d)
ctx, _ := checkContext()
got := p.Check(ctx)
if len(got) != len(cfg.Hosts) {
t.Fatalf("Check() returned %d targets, want %d", len(got), len(cfg.Hosts))
}
for i, h := range cfg.Hosts {
ts := got[i]
if ts.Name != h.Name || ts.Address != h.Address {
t.Errorf("target %d = %s (%s), want %s (%s)", i, ts.Name, ts.Address, h.Name, h.Address)
}
if ts.Name == "" || ts.Address == "" {
t.Errorf("target %d has an empty name or address: %+v", i, ts)
}
wantReady := h.Name != "lab2"
if ts.Ready != wantReady {
t.Errorf("target %s ready = %v, want %v (%s: %s)", ts.Name, ts.Ready, wantReady, ts.Reason, ts.Message)
}
}
if got[1].Reason != provider.ReasonUnreachable || got[0].Reason != provider.ReasonReachable {
t.Errorf("reasons = %s, %s, %s", got[0].Reason, got[1].Reason, got[2].Reason)
}
}
func TestCheck_hostsCheckedConcurrently(t *testing.T) {
t.Parallel()
cfg := twoHosts()
fakes := fakesFor(cfg)
for _, f := range fakes {
release := f.blockOn("ConnectGetLibVersion")
time.AfterFunc(200*time.Millisecond, func() { close(release) })
}
p := newProvider(testProvider, cfg, &checkDialer{backends: fakes})
ctx, _ := checkContext()
start := time.Now()
got := p.Check(ctx)
elapsed := time.Since(start)
if elapsed > 350*time.Millisecond {
t.Errorf("Check() took %v, want under 350ms for two 200ms hosts", elapsed)
}
for _, ts := range got {
if !ts.Ready {
t.Errorf("target %s not ready: %s: %s", ts.Name, ts.Reason, ts.Message)
}
if ts.Latency < 200*time.Millisecond {
t.Errorf("target %s latency = %v, want at least the 200ms block", ts.Name, ts.Latency)
}
}
}
// TestCheck_expiredContextReturnsPromptly drives the production shape of a
// round that ran out of time during the dial: the dialer fails at the tcp
// stage wrapping context.DeadlineExceeded, which checkReason alone would
// call Unreachable; the verdict must still be Timeout because the round's
// ctx ended.
func TestCheck_expiredContextReturnsPromptly(t *testing.T) {
t.Parallel()
cfg := twoHosts()
p := newProvider(testProvider, cfg, &checkDialer{block: true})
ctx, _ := checkContext()
ctx, cancel := context.WithTimeout(ctx, 0)
defer cancel()
start := time.Now()
got := p.Check(ctx)
if elapsed := time.Since(start); elapsed > time.Second {
t.Errorf("Check() took %v with an expired ctx, want under 1s", elapsed)
}
if len(got) != len(cfg.Hosts) {
t.Fatalf("Check() returned %d targets, want %d", len(got), len(cfg.Hosts))
}
for _, ts := range got {
if ts.Ready || ts.Reason != provider.ReasonTimeout {
t.Errorf("target %s = ready %v reason %s, want not ready, Timeout", ts.Name, ts.Ready, ts.Reason)
}
if !strings.Contains(ts.Message, context.DeadlineExceeded.Error()) || !strings.Contains(ts.Message, string(stageTCP)) {
t.Errorf("target %s message = %q, want the tcp stage and the deadline named", ts.Name, ts.Message)
}
}
}
// keyMaterial is the sentinel in the unusable key file of the credentials
// row; it must never reach a message.
const keyMaterial = "SENTINEL-PRIVATE-KEY-BYTES-9f3a"
func TestCheck_reasonsTable(t *testing.T) {
t.Parallel()
sshHandshake := func(cause error) error {
return &dialError{stage: stageSSH, host: testHost, err: fmt.Errorf("ssh: handshake failed: %w", cause)}
}
tests := []struct {
name string
// dialErr builds the dial failure; nil means the dial succeeds.
dialErr func(t *testing.T) error
prep func(f *fakeBackend)
timeout time.Duration
cancel bool
wantReady bool
wantReason string
wantMessage string
}{
{
name: "credentials unusable",
dialErr: func(t *testing.T) error {
path := filepath.Join(t.TempDir(), "id_ed25519")
if err := os.WriteFile(path, []byte(keyMaterial), 0o600); err != nil {
t.Fatal(err)
}
_, err := loadCredentials(provider.KVMConfig{SSHKeyFile: path, KnownHostsFile: "/nonexistent/known_hosts"})
if err == nil {
t.Fatal("loadCredentials accepted the sentinel file")
}
return &dialError{stage: stageCredentials, host: testHost, err: err}
},
wantReason: reasonCredentialsUnavailable,
},
{
name: "credentials missing",
dialErr: func(*testing.T) error {
_, err := loadCredentials(provider.KVMConfig{SSHKeyFile: "/nonexistent/kvm2/id_ed25519"})
return &dialError{stage: stageCredentials, host: testHost, err: err}
},
wantReason: reasonCredentialsUnavailable,
},
{
name: "tcp refused",
dialErr: func(*testing.T) error {
return &dialError{stage: stageTCP, host: testHost, err: errors.New("dial tcp 192.0.2.1:22: connect: connection refused")}
},
wantReason: provider.ReasonUnreachable,
},
{
name: "tcp timed out on the dial budget but ctx is fine",
dialErr: func(*testing.T) error {
return &dialError{stage: stageTCP, host: testHost, err: fmt.Errorf("dial tcp 192.0.2.1:22: %w", context.DeadlineExceeded)}
},
wantReason: provider.ReasonUnreachable,
},
{
name: "ssh host key unknown",
dialErr: func(*testing.T) error { return sshHandshake(&knownhosts.KeyError{}) },
wantReason: provider.ReasonUnauthorized,
},
{
name: "ssh host key mismatch",
dialErr: func(*testing.T) error {
return sshHandshake(&knownhosts.KeyError{Want: []knownhosts.KnownKey{{Filename: "known_hosts", Line: 3}}})
},
wantReason: provider.ReasonUnauthorized,
},
{
name: "ssh authentication refused",
dialErr: func(*testing.T) error {
return sshHandshake(errors.New("ssh: unable to authenticate, attempted methods [none publickey], no supported methods remain"))
},
wantReason: provider.ReasonUnauthorized,
},
{
name: "ssh handshake reset",
dialErr: func(*testing.T) error { return sshHandshake(errors.New("read tcp: connection reset by peer")) },
wantReason: provider.ReasonUnreachable,
},
{
name: "ssh handshake timed out on the dial budget but ctx is fine",
dialErr: func(*testing.T) error { return sshHandshake(context.DeadlineExceeded) },
wantReason: provider.ReasonUnreachable,
},
{
name: "socket refused",
dialErr: func(*testing.T) error {
return &dialError{stage: stageSocket, host: testHost, err: errors.New("opening /var/run/libvirt/libvirt-sock: ssh: rejected: connect failed (No such file or directory)")}
},
wantReason: provider.ReasonUnreachable,
wantMessage: "/var/run/libvirt/libvirt-sock",
},
{
name: "libvirt access denied",
prep: func(f *fakeBackend) {
f.failWith("ConnectGetLibVersion", fakeErr(libvirt.ErrAccessDenied, "access denied"))
},
wantReason: provider.ReasonForbidden,
},
{
name: "libvirt auth failed",
prep: func(f *fakeBackend) {
f.failWith("ConnectGetHostname", fakeErr(libvirt.ErrAuthFailed, "authentication failed"))
},
wantReason: provider.ReasonForbidden,
},
{
name: "pool not found",
prep: func(f *fakeBackend) {
f.mu.Lock()
delete(f.pools, fakePoolName)
f.mu.Unlock()
},
wantReason: reasonPoolNotFound,
wantMessage: fakePoolName,
},
{
name: "other libvirt error",
prep: func(f *fakeBackend) {
f.failWith("StoragePoolGetInfo", fakeErr(libvirt.ErrInternalError, "internal error: pool info"))
},
wantReason: provider.ReasonAPIError,
},
{
name: "rpc outlives the deadline",
prep: func(f *fakeBackend) { f.blockOn("ConnectGetLibVersion") },
timeout: 50 * time.Millisecond,
wantReason: provider.ReasonTimeout,
},
{
name: "ctx cancelled mid check",
prep: func(f *fakeBackend) { f.blockOn("StoragePoolLookupByName") },
cancel: true,
wantReason: provider.ReasonTimeout,
},
{
name: "link killed mid check",
prep: func(f *fakeBackend) {
f.failWith("StoragePoolLookupByName", transientf("libvirt call StoragePoolLookupByName on beelink: connection closed while waiting: keepalive: EOF"))
},
wantReason: provider.ReasonUnreachable,
},
{
name: "pool inactive",
prep: func(f *fakeBackend) { f.pools[fakePoolName].state = libvirt.StoragePoolInactive },
wantReason: reasonPoolNotActive,
wantMessage: `storage pool "default" is inactive`,
},
{
name: "ok",
wantReady: true,
wantReason: provider.ReasonReachable,
},
}
for _, tc := range tests {
t.Run(tc.name, func(t *testing.T) {
t.Parallel()
cfg := twoHosts()
cfg.Hosts = cfg.Hosts[:1]
f := newFakeBackend()
d := &checkDialer{backends: map[string]*fakeBackend{testHost: f}}
if tc.dialErr != nil {
d.errs = map[string]error{testHost: tc.dialErr(t)}
}
if tc.prep != nil {
tc.prep(f)
}
p := newProvider(testProvider, cfg, d)
ctx, _ := checkContext()
switch {
case tc.timeout > 0:
var cancel context.CancelFunc
ctx, cancel = context.WithTimeout(ctx, tc.timeout)
defer cancel()
case tc.cancel:
var cancel context.CancelFunc
ctx, cancel = context.WithCancel(ctx)
time.AfterFunc(50*time.Millisecond, cancel)
}
got := p.Check(ctx)
if len(got) != 1 {
t.Fatalf("Check() returned %d targets, want 1", len(got))
}
ts := got[0]
if ts.Ready != tc.wantReady || ts.Reason != tc.wantReason {
t.Errorf("verdict = ready %v reason %s (%s), want ready %v reason %s", ts.Ready, ts.Reason, ts.Message, tc.wantReady, tc.wantReason)
}
if !tc.wantReady && ts.Message == "" {
t.Error("failed check has no message")
}
if tc.wantReady && ts.Message != "" {
t.Errorf("successful check has a message: %q", ts.Message)
}
if strings.Contains(ts.Message, keyMaterial) {
t.Errorf("message leaks the key file content: %q", ts.Message)
}
if tc.wantMessage != "" && !strings.Contains(ts.Message, tc.wantMessage) {
t.Errorf("message = %q, want it to contain %q", ts.Message, tc.wantMessage)
}
if ts.Details["user"] != "virt" || ts.Details["pool"] != "default" {
t.Errorf("details = %v, want user and pool", ts.Details)
}
if tc.wantReady {
want := map[string]string{
"user": "virt", "pool": "default", "hostname": "fake-host", "libvirtVersion": "11.6.0",
"poolState": "running", "poolCapacityGiB": "100.0", "poolAvailableGiB": "50.0",
}
for k, v := range want {
if ts.Details[k] != v {
t.Errorf("details[%s] = %q, want %q", k, ts.Details[k], v)
}
}
}
})
}
}
func TestCheck_neverUsesPooledConnection(t *testing.T) {
t.Parallel()
cfg := threeHosts()
fakes := fakesFor(cfg)
d := &checkDialer{backends: fakes}
p := newProvider(testProvider, cfg, d)
ctx, _ := checkContext()
got := p.Check(ctx)
for _, ts := range got {
if !ts.Ready {
t.Errorf("target %s not ready: %s: %s", ts.Name, ts.Reason, ts.Message)
}
}
if n := d.dials.Load(); int(n) != len(cfg.Hosts) {
t.Errorf("dials = %d, want one fresh dial per host (%d)", n, len(cfg.Hosts))
}
p.hosts.mu.Lock()
pooled := len(p.hosts.hosts)
p.hosts.mu.Unlock()
if pooled != 0 {
t.Errorf("pooled table has %d entries after Check, want none", pooled)
}
for name, f := range fakes {
if f.closeCount() != 1 {
t.Errorf("fresh backend for %s closed %d times, want 1", name, f.closeCount())
}
}
}
func TestCheck_detailsKeys(t *testing.T) {
t.Parallel()
full := []string{"hostname", "libvirtVersion", "pool", "poolAvailableGiB", "poolCapacityGiB", "poolState", "user"}
tests := []struct {
name string
dialErr error
prep func(f *fakeBackend)
want []string
}{
{name: "success", want: full},
{name: "pool not active", prep: func(f *fakeBackend) { f.pools[fakePoolName].state = libvirt.StoragePoolDegraded }, want: full},
{
name: "dial failure",
dialErr: &dialError{stage: stageTCP, host: testHost, err: errors.New("dial tcp: connection refused")},
want: []string{"pool", "user"},
},
}
for _, tc := range tests {
t.Run(tc.name, func(t *testing.T) {
t.Parallel()
cfg := twoHosts()
cfg.Hosts = cfg.Hosts[:1]
f := newFakeBackend()
if tc.prep != nil {
tc.prep(f)
}
d := &checkDialer{backends: map[string]*fakeBackend{testHost: f}}
if tc.dialErr != nil {
d.errs = map[string]error{testHost: tc.dialErr}
}
p := newProvider(testProvider, cfg, d)
ctx, _ := checkContext()
got := p.Check(ctx)[0]
if keys := detailKeys(got.Details); !slices.Equal(keys, tc.want) {
t.Errorf("details keys = %v, want %v", keys, tc.want)
}
for k, v := range got.Details {
if v == "" {
t.Errorf("details[%s] is empty", k)
}
}
})
}
}
func TestCheck_logsVerdictPerHost(t *testing.T) {
t.Parallel()
cfg := twoHosts()
d := &checkDialer{backends: fakesFor(cfg), errs: map[string]error{"lab2": &dialError{stage: stageTCP, host: "lab2", err: errors.New("dial tcp: i/o timeout")}}}
p := newProvider(testProvider, cfg, d)
ctx, logs := checkContext()
p.Check(ctx)
checked := logs.matching("kvm2 host checked")
if len(checked) != 2 {
t.Fatalf("got %d verdict lines, want 2: %v", len(checked), checked)
}
for _, l := range checked {
switch {
case strings.Contains(l, `"host"="beelink"`):
if !strings.Contains(l, `"ready"=true`) || !strings.Contains(l, `"reason"="Reachable"`) || !strings.Contains(l, `"latency"=`) {
t.Errorf("beelink verdict line = %s", l)
}
case strings.Contains(l, `"host"="lab2"`):
if !strings.Contains(l, `"ready"=false`) || !strings.Contains(l, `"reason"="Unreachable"`) {
t.Errorf("lab2 verdict line = %s", l)
}
default:
t.Errorf("verdict line without a host: %s", l)
}
}
}
func TestCheckReason_table(t *testing.T) {
t.Parallel()
dial := func(stage dialStage, cause error) error {
return fmt.Errorf("wrapped: %w", &dialError{stage: stage, host: testHost, err: cause})
}
tests := []struct {
name string
err error
want string
}{
{name: "credentials stage", err: dial(stageCredentials, errors.New("reading ssh key file /k: no such file")), want: reasonCredentialsUnavailable},
{name: "credentials stage even when permanent", err: dial(stageCredentials, permanentf("parsing ssh key file /k: ssh: no key found")), want: reasonCredentialsUnavailable},
{name: "tcp stage", err: dial(stageTCP, errors.New("connect: connection refused")), want: provider.ReasonUnreachable},
{name: "tcp stage with a deadline", err: dial(stageTCP, context.DeadlineExceeded), want: provider.ReasonUnreachable},
{name: "ssh stage unknown host key", err: dial(stageSSH, &knownhosts.KeyError{}), want: provider.ReasonUnauthorized},
{name: "ssh stage wrapped key error", err: dial(stageSSH, fmt.Errorf("ssh: handshake failed: %w", &knownhosts.KeyError{Want: []knownhosts.KnownKey{{}}})), want: provider.ReasonUnauthorized},
{name: "ssh stage unable to authenticate", err: dial(stageSSH, errors.New("ssh: handshake failed: ssh: unable to authenticate, attempted methods [none publickey], no supported methods remain")), want: provider.ReasonUnauthorized},
{name: "ssh stage reset", err: dial(stageSSH, errors.New("read tcp: connection reset by peer")), want: provider.ReasonUnreachable},
{name: "ssh stage cancelled", err: dial(stageSSH, context.Canceled), want: provider.ReasonUnreachable},
{name: "socket stage", err: dial(stageSocket, errors.New("opening /run/libvirt/libvirt-sock: ssh: rejected")), want: provider.ReasonUnreachable},
{name: "deadline exceeded", err: fmt.Errorf("%w: libvirt call ConnectGetHostname on beelink exceeded its 30s budget: %w", provider.ErrTransient, context.DeadlineExceeded), want: provider.ReasonTimeout},
{name: "cancelled", err: fmt.Errorf("%w: libvirt call ConnectGetHostname on beelink abandoned: %w", provider.ErrTransient, context.Canceled), want: provider.ReasonTimeout},
{name: "access denied", err: fakeErr(libvirt.ErrAccessDenied, "access denied"), want: provider.ReasonForbidden},
{name: "auth failed wrapped", err: fmt.Errorf("connecting to libvirt on host beelink: %w", fakeErr(libvirt.ErrAuthFailed, "authentication failed")), want: provider.ReasonForbidden},
{name: "no storage pool", err: fakeErr(libvirt.ErrNoStoragePool, "Storage pool not found"), want: reasonPoolNotFound},
{name: "other libvirt error", err: fakeErr(libvirt.ErrInternalError, "internal error"), want: provider.ReasonAPIError},
{name: "link killed", err: transientf("libvirt call X on beelink: connection closed while waiting: keepalive: EOF"), want: provider.ReasonUnreachable},
{name: "plain error", err: errors.New("unexpected EOF"), want: provider.ReasonUnreachable},
{name: "nil", err: nil, want: provider.ReasonUnreachable},
}
for _, tc := range tests {
t.Run(tc.name, func(t *testing.T) {
t.Parallel()
if got := checkReason(tc.err); got != tc.want {
t.Errorf("checkReason(%v) = %s, want %s", tc.err, got, tc.want)
}
})
}
}
func TestFormatLibVersion(t *testing.T) {
t.Parallel()
tests := []struct {
in uint64
want string
}{
{0, "0.0.0"},
{11006000, "11.6.0"},
{10000003, "10.0.3"},
{9010012, "9.10.12"},
{100999999, "100.999.999"},
}
for _, tc := range tests {
t.Run(tc.want, func(t *testing.T) {
t.Parallel()
if got := formatLibVersion(tc.in); got != tc.want {
t.Errorf("formatLibVersion(%d) = %s, want %s", tc.in, got, tc.want)
}
})
}
}
func TestPoolStateName(t *testing.T) {
t.Parallel()
tests := []struct {
in uint8
want string
}{
{uint8(libvirt.StoragePoolInactive), "inactive"},
{uint8(libvirt.StoragePoolBuilding), "building"},
{uint8(libvirt.StoragePoolRunning), "running"},
{uint8(libvirt.StoragePoolDegraded), "degraded"},
{uint8(libvirt.StoragePoolInaccessible), "inaccessible"},
{42, "state 42"},
}
for _, tc := range tests {
t.Run(tc.want, func(t *testing.T) {
t.Parallel()
if got := poolStateName(tc.in); got != tc.want {
t.Errorf("poolStateName(%d) = %s, want %s", tc.in, got, tc.want)
}
})
}
}
internal/provider/kvm2/create.go
+280
View File
@@ -0,0 +1,280 @@
package kvm2
import (
"bytes"
"fmt"
"time"
"context"
"github.com/digitalocean/go-libvirt"
"github.com/go-logr/logr"
"gitea.home.hrajfrisbee.cz/kacerr/egress-proxies-operator/internal/provider"
)
// The wait after DomainCreate (spec entry 18): Create returns only once the
// domain is running, so Get never sees a domain kvm2 defined but did not
// start (spec entry 9). DomainCreate is synchronous, so the first read
// normally succeeds; the cap only covers state settling and must not pin a
// reconcile worker for long.
const (
defaultRunningWait = 15 * time.Second
defaultRunningPoll = 500 * time.Millisecond
)
// Create is spec entry 10: validate placement, then, under the per-name
// lock, find or build the domain step by step, every step idempotent, and
// return once it is running.
func (p *Provider) Create(ctx context.Context, req provider.CreateRequest) (string, error) {
host, err := p.host(req.Placement.Zone)
if err != nil {
if req.Placement.Zone != "" {
err = permanentf("placement.zone %q is not in kvm2.hosts", req.Placement.Zone)
}
return "", p.wrapErr("create", "", err)
}
mt, ok := p.cfg.MachineTypes[req.Placement.MachineType]
if !ok {
return "", p.wrapErr("create", "", permanentf("placement.machineType %q is not in kvm2.machineTypes", req.Placement.MachineType))
}
src, err := resolveImage(p.name, p.cfg, req.Placement.Image)
if err != nil {
return "", p.wrapErr("create", "", err)
}
id := formatProviderID(host.Name, req.Name)
log := p.logger(ctx).WithValues("host", host.Name, "name", req.Name)
log.V(2).Info("kvm2 create resolved",
"pool", host.Pool, "bridge", host.Bridge, "machineType", req.Placement.MachineType,
"vcpus", mt.VCPUs, "memoryMiB", mt.MemoryMiB, "diskSizeGb", p.cfg.DiskSizeGB,
"image", src.volumeName(), "firmware", src.Firmware, "cloudInitBytes", len(req.CloudInit))
unlock, err := p.lockName(ctx, req.Name)
if err != nil {
return "", p.wrapErr("create", id, err)
}
defer unlock()
be, err := p.hosts.get(ctx, host)
if err != nil {
logAPIError(log, "create", err)
return "", p.wrapErr("create", id, err)
}
if err := p.create(ctx, log, be, host, req, mt, src); err != nil {
logAPIError(log, "create", err)
return "", p.wrapErr("create", id, err)
}
log.V(1).Info("kvm2 domain ensured running", "id", id)
return id, nil
}
func (p *Provider) create(ctx context.Context, log logr.Logger, be backend, host provider.KVMHost, req provider.CreateRequest, mt provider.KVMMachineType, src imageSource) error {
dom, err := be.DomainLookupByName(ctx, req.Name)
switch {
case err == nil:
meta, err := readMetadata(ctx, be, dom)
if err != nil {
return err
}
if !meta.managed() {
return permanentf("domain %s exists but carries no kvm2 mark; it was not created by this provider", req.Name)
}
if meta.UID != "" && meta.UID != req.UID {
return permanentf("domain %s belongs to proxy uid %s, not %s", req.Name, meta.UID, req.UID)
}
log.V(1).Info("kvm2 domain already defined", "uid", meta.UID)
return p.ensureRunning(ctx, log, be, dom)
case !hasCode(err, libvirt.ErrNoDomain):
return fmt.Errorf("looking up domain %s: %w", req.Name, err)
}
pool, err := be.StoragePoolLookupByName(ctx, host.Pool)
if err != nil {
return fmt.Errorf("looking up storage pool %s: %w", host.Pool, err)
}
imageBytes, err := p.imageSizeIfUncached(ctx, be, pool, src)
if err != nil {
return err
}
if err := checkPoolSpace(ctx, be, pool, host.Pool, neededBytes(imageBytes, p.cfg.DiskSizeGB)); err != nil {
return err
}
basePath, err := p.ensureImage(ctx, host, pool, be, src)
if err != nil {
return err
}
overlay, err := ensureOverlay(ctx, log, be, pool, req.Name, basePath, src.Format, p.cfg.DiskSizeGB)
if err != nil {
return err
}
seed, err := p.writeSeed(ctx, log, be, pool, req)
if err != nil {
return err
}
diskPath, err := be.StorageVolGetPath(ctx, overlay)
if err != nil {
return fmt.Errorf("reading the path of overlay volume %s: %w", overlay.Name, err)
}
seedPath, err := be.StorageVolGetPath(ctx, seed)
if err != nil {
return fmt.Errorf("reading the path of seed volume %s: %w", seed.Name, err)
}
xmlDoc, err := buildDomainXML(domainParams{
Name: req.Name,
UID: req.UID,
CreatedAt: p.now(),
VCPUs: mt.VCPUs,
MemoryMiB: mt.MemoryMiB,
Firmware: src.Firmware,
Bridge: host.Bridge,
DiskPath: diskPath,
SeedPath: seedPath,
DiskFormat: provider.KVMFormatQcow2,
})
if err != nil {
return err
}
dom, err = be.DomainDefineXML(ctx, xmlDoc)
if err != nil {
return fmt.Errorf("defining domain %s: %w", req.Name, err)
}
kv := []any{"uid", req.UID, "firmware", src.Firmware}
if log.V(2).Enabled() {
kv = append(kv, "xml", xmlDoc)
}
log.V(1).Info("domain defined", kv...)
return p.ensureRunning(ctx, log, be, dom)
}
// ensureOverlay creates the per-VM qcow2 overlay backed by the base image,
// or reuses the one a previous attempt left: volume creation is atomic, so
// an existing overlay is complete.
func ensureOverlay(ctx context.Context, log logr.Logger, be backend, pool libvirt.StoragePool, name, basePath, baseFormat string, diskSizeGB int64) (libvirt.StorageVol, error) {
volName := overlayVolumeName(name)
xmlDoc, err := buildVolumeXML(volumeSpec{
Name: volName,
CapacityBytes: diskSizeGB << 30,
Format: provider.KVMFormatQcow2,
BackingPath: basePath,
BackingFormat: baseFormat,
})
if err != nil {
return libvirt.StorageVol{}, err
}
vol, err := be.StorageVolCreateXML(ctx, pool, xmlDoc, 0)
switch {
case err == nil:
log.V(1).Info("overlay volume created", "volume", volName)
return vol, nil
case hasCode(err, libvirt.ErrStorageVolExist):
vol, err = be.StorageVolLookupByName(ctx, pool, volName)
if err != nil {
return libvirt.StorageVol{}, fmt.Errorf("looking up overlay volume %s: %w", volName, err)
}
log.V(1).Info("overlay volume already exists", "volume", volName)
return vol, nil
default:
return libvirt.StorageVol{}, fmt.Errorf("creating overlay volume %s: %w", volName, err)
}
}
// writeSeed builds the NoCloud seed and uploads it as a raw volume,
// replacing whatever a previous attempt left (spec entry 5): a cut-off
// upload would boot a VM without cloud-init. The ISO is tens of KiB, so it
// is built in memory. Nothing about its content is logged or put in an
// error.
func (p *Provider) writeSeed(ctx context.Context, log logr.Logger, be backend, pool libvirt.StoragePool, req provider.CreateRequest) (libvirt.StorageVol, error) {
name := seedVolumeName(req.Name)
if err := deleteVolumeIfPresent(ctx, be, pool, name); err != nil {
return libvirt.StorageVol{}, err
}
installAgent := p.cfg.InstallGuestAgent == nil || *p.cfg.InstallGuestAgent
vendorData, err := buildVendorData(p.cfg.VendorData, installAgent)
if err != nil {
return libvirt.StorageVol{}, permanentf("building vendor-data for %s: %v", req.Name, err)
}
var iso bytes.Buffer
if err := buildSeedISO(req.Name, req.CloudInit, vendorData, &iso); err != nil {
return libvirt.StorageVol{}, permanentf("building seed ISO for %s: %v", req.Name, err)
}
xmlDoc, err := buildVolumeXML(volumeSpec{Name: name, CapacityBytes: int64(iso.Len()), Format: provider.KVMFormatRaw})
if err != nil {
return libvirt.StorageVol{}, err
}
vol, err := be.StorageVolCreateXML(ctx, pool, xmlDoc, 0)
if err != nil {
return libvirt.StorageVol{}, fmt.Errorf("creating seed volume %s: %w", name, err)
}
if err := be.StorageVolUpload(ctx, vol, bytes.NewReader(iso.Bytes()), 0, uint64(iso.Len()), 0); err != nil {
return libvirt.StorageVol{}, fmt.Errorf("uploading seed volume %s: %w", name, err)
}
log.V(1).Info("seed volume uploaded", "volume", name, "bytes", iso.Len())
return vol, nil
}
// ensureRunning starts the domain unless it is active already, then waits
// until libvirt reports it running. ErrOperationInvalid from DomainCreate
// means another caller started it between the state read and the call. A
// paused or pmsuspended domain is an admin's doing and DomainCreate would
// refuse it; the verdict is Permanent so the reconciler does not retry it.
func (p *Provider) ensureRunning(ctx context.Context, log logr.Logger, be backend, dom libvirt.Domain) error {
state, _, err := be.DomainGetState(ctx, dom, 0)
if err != nil {
return fmt.Errorf("reading state of domain %s: %w", dom.Name, err)
}
switch s := libvirt.DomainState(state); {
case isActive(s):
return nil
case s == libvirt.DomainPaused || s == libvirt.DomainPmsuspended:
return permanentf("domain %s is %s and cannot be started by the operator; resume or destroy it on the host", dom.Name, s)
}
switch err := be.DomainCreate(ctx, dom); {
case err == nil:
log.V(1).Info("domain started")
case hasCode(err, libvirt.ErrOperationInvalid):
log.V(1).Info("domain start reported it already running")
default:
return fmt.Errorf("starting domain %s: %w", dom.Name, err)
}
return p.waitRunning(ctx, be, dom)
}
// waitRunning polls DomainGetState until the domain is active, the cap
// passes (Transient: the next Create attempt finds the domain defined and
// tries again) or ctx ends. Only nostate is worth another poll; a domain
// that is shut off, crashed or paused right after its start is not going
// to come up on its own, so that is Transient at once.
func (p *Provider) waitRunning(ctx context.Context, be backend, dom libvirt.Domain) error {
deadline := time.NewTimer(p.runningWait)
defer deadline.Stop()
ticker := time.NewTicker(p.runningPoll)
defer ticker.Stop()
for {
state, _, err := be.DomainGetState(ctx, dom, 0)
if err != nil {
return fmt.Errorf("reading state of domain %s: %w", dom.Name, err)
}
switch s := libvirt.DomainState(state); {
case isActive(s):
return nil
case s != libvirt.DomainNostate:
return transientf("domain %s went %s right after start", dom.Name, s)
}
select {
case <-ticker.C:
case <-deadline.C:
return transientf("domain %s did not reach running within %s (state %s)", dom.Name, p.runningWait, libvirt.DomainState(state))
case <-ctx.Done():
return fmt.Errorf("%w: waiting for domain %s to run: %w", provider.ErrTransient, dom.Name, ctx.Err())
}
}
}
// logAPIError is the V(1) line per failed backend call of spec entry 20.
func logAPIError(log logr.Logger, op string, err error) {
kv := []any{"op", op, "error", err.Error()}
kv = append(kv, libvirtErrorFields(err)...)
log.V(1).Info("kvm2 backend call failed", kv...)
}
internal/provider/kvm2/create_test.go
+467
View File
@@ -0,0 +1,467 @@
package kvm2
import (
"bytes"
"errors"
"math"
"strconv"
"strings"
"testing"
"time"
"github.com/digitalocean/go-libvirt"
"gitea.home.hrajfrisbee.cz/kacerr/egress-proxies-operator/internal/provider"
)
func TestCreate_repeatReturnsSameIDAndCreatesNothing(t *testing.T) {
t.Parallel()
fx := newFixture(t, 1)
first := fx.create(testName)
before := len(fx.fake.callLog())
defines, vols, uploads := fx.fake.callCount("DomainDefineXML"), fx.fake.callCount("StorageVolCreateXML"), fx.fake.callCount("StorageVolUpload")
second := fx.create(testName)
if second != first {
t.Errorf("second Create = %q, want %q", second, first)
}
if got := fx.fake.callCount("DomainDefineXML"); got != defines {
t.Errorf("DomainDefineXML calls after repeat = %d, want %d", got, defines)
}
if got := fx.fake.callCount("StorageVolCreateXML"); got != vols {
t.Errorf("StorageVolCreateXML calls after repeat = %d, want %d", got, vols)
}
if got := fx.fake.callCount("StorageVolUpload"); got != uploads {
t.Errorf("StorageVolUpload calls after repeat = %d, want %d", got, uploads)
}
if got := fx.fake.callCount("DomainCreate"); got != 1 {
t.Errorf("DomainCreate calls = %d, want 1", got)
}
repeat := fx.fake.callLog()[before:]
for _, rpc := range repeat {
switch rpc {
case "DomainLookupByName", "DomainGetMetadata", "DomainGetState":
default:
t.Errorf("repeat Create made %s; want only lookup, metadata and state reads: %v", rpc, repeat)
}
}
if len(fx.logs.matching("kvm2 domain already defined")) != 1 {
t.Errorf("no 'already defined' line:\n%s", strings.Join(fx.logs.all(), "\n"))
}
}
func TestCreate_resumesAfterEachPartialStep(t *testing.T) {
t.Parallel()
overlay, seed := overlayVolumeName(testName), seedVolumeName(testName)
tests := []struct {
name string
setup func(fx *fixture)
check func(t *testing.T, fx *fixture)
}{
{
name: "base without marker is refetched",
setup: func(fx *fixture) {
fx.putVol(fx.baseName(), []byte("truncated"))
},
check: func(t *testing.T, fx *fixture) {
if got := fx.fake.vol(fx.baseName()); got == nil || !bytes.Equal(got.content, fx.image) {
t.Errorf("base volume not refetched")
}
if fx.fake.vol(fx.baseName()+".sha256") == nil {
t.Errorf("marker not written")
}
},
},
{
name: "overlay exists is reused",
setup: func(fx *fixture) {
fx.cacheImage()
fx.putVol(overlay, []byte("overlay"))
},
check: func(t *testing.T, fx *fixture) {
if got := fx.fake.vol(overlay); got == nil || string(got.content) != "overlay" {
t.Errorf("overlay volume was replaced")
}
if len(fx.logs.matching("overlay volume already exists")) != 1 {
t.Errorf("no reuse line:\n%s", strings.Join(fx.logs.all(), "\n"))
}
},
},
{
name: "truncated seed is replaced",
setup: func(fx *fixture) {
fx.cacheImage()
fx.putVol(seed, []byte("cut"))
},
check: func(t *testing.T, fx *fixture) {
got := fx.fake.vol(seed)
if got == nil || len(got.content) <= 3 || !bytes.Contains(got.content, []byte(seedLabel)) {
t.Errorf("seed volume not rebuilt")
}
},
},
{
name: "domain defined but shutoff is started",
setup: func(fx *fixture) {
fx.cacheImage()
fx.putVol(overlay, []byte("overlay"))
fx.putVol(seed, []byte("seed"))
fx.fake.addDomain(testName, libvirt.DomainShutoff, managedMetadata(testUID, time.Now()))
},
check: func(t *testing.T, fx *fixture) {
if got := fx.fake.callCount("DomainDefineXML"); got != 0 {
t.Errorf("DomainDefineXML calls = %d, want 0", got)
}
if got := fx.fake.callCount("DomainCreate"); got != 1 {
t.Errorf("DomainCreate calls = %d, want 1", got)
}
},
},
{
name: "domain running is returned as is",
setup: func(fx *fixture) {
fx.fake.addDomain(testName, libvirt.DomainRunning, managedMetadata(testUID, time.Now()))
},
check: func(t *testing.T, fx *fixture) {
for _, rpc := range []string{"DomainDefineXML", "DomainCreate", "StorageVolCreateXML"} {
if got := fx.fake.callCount(rpc); got != 0 {
t.Errorf("%s calls = %d, want 0", rpc, got)
}
}
},
},
}
for _, tc := range tests {
t.Run(tc.name, func(t *testing.T) {
t.Parallel()
fx := newFixture(t, 1)
tc.setup(fx)
fx.create(testName)
fx.withDomain(testName, func(d *fakeDomain) {
if d.state != libvirt.DomainRunning {
t.Errorf("domain state = %s, want running", d.state)
}
})
tc.check(t, fx)
})
}
}
func TestCreate_returnsOnlyWhenRunning(t *testing.T) {
t.Parallel()
t.Run("returns after the domain reports running", func(t *testing.T) {
t.Parallel()
fx := newFixture(t, 1)
fx.fake.startDelayPolls = 3
fx.create(testName)
fx.withDomain(testName, func(d *fakeDomain) {
if d.state != libvirt.DomainRunning {
t.Errorf("Create returned with domain state %s", d.state)
}
})
if got := fx.fake.callCount("DomainGetState"); got < 4 {
t.Errorf("DomainGetState calls = %d, want at least 4 (one before start, three polls)", got)
}
})
t.Run("never running is transient after the cap", func(t *testing.T) {
t.Parallel()
fx := newFixture(t, 1, withRunningWait(time.Millisecond, 30*time.Millisecond))
fx.fake.startDelayPolls = math.MaxInt32
start := time.Now()
id, err := fx.p.Create(fx.ctx, fx.req(testName))
if id != "" {
t.Errorf("Create = %q, want no ID", id)
}
assertProviderError(t, err, "create", provider.ErrTransient)
if !strings.Contains(err.Error(), "did not reach running") {
t.Errorf("error = %v, want the wait cap named", err)
}
if elapsed := time.Since(start); elapsed > 2*time.Second {
t.Errorf("Create took %v, want the shrunk cap to end it", elapsed)
}
})
}
func TestCreate_domainThatDiesAfterStartIsTransientAtOnce(t *testing.T) {
t.Parallel()
for _, state := range []libvirt.DomainState{libvirt.DomainCrashed, libvirt.DomainShutoff, libvirt.DomainPaused} {
t.Run(state.String(), func(t *testing.T) {
t.Parallel()
// The cap is well over the bound asserted below, so a poll that
// ran to the cap would fail the test.
fx := newFixture(t, 1, withRunningWait(time.Millisecond, 10*time.Second))
fx.fake.stateAfterStart = state
start := time.Now()
id, err := fx.p.Create(fx.ctx, fx.req(testName))
elapsed := time.Since(start)
if id != "" {
t.Errorf("Create = %q, want no ID", id)
}
assertProviderError(t, err, "create", provider.ErrTransient)
if !strings.Contains(err.Error(), "right after start") {
t.Errorf("error = %v, want the state after start named", err)
}
if elapsed > time.Second {
t.Errorf("Create took %v, want it to give up at once, not poll to the cap", elapsed)
}
if got := fx.fake.callCount("DomainCreate"); got != 1 {
t.Errorf("DomainCreate calls = %d, want 1", got)
}
})
}
}
func TestCreate_pausedDomainIsPermanent(t *testing.T) {
t.Parallel()
for _, state := range []libvirt.DomainState{libvirt.DomainPaused, libvirt.DomainPmsuspended} {
t.Run(state.String(), func(t *testing.T) {
t.Parallel()
fx := newFixture(t, 1)
fx.fake.addDomain(testName, state, managedMetadata(testUID, time.Now()))
id, err := fx.p.Create(fx.ctx, fx.req(testName))
if id != "" {
t.Errorf("Create = %q, want no ID", id)
}
assertProviderError(t, err, "create", provider.ErrPermanent)
if !strings.Contains(err.Error(), "cannot be started by the operator") {
t.Errorf("error = %v, want the admin action named", err)
}
for _, rpc := range []string{"DomainCreate", "DomainDefineXML", "DomainDestroy"} {
if got := fx.fake.callCount(rpc); got != 0 {
t.Errorf("%s calls = %d, want 0", rpc, got)
}
}
fx.withDomain(testName, func(d *fakeDomain) {
if d.state != state {
t.Errorf("domain state = %s, want it left %s", d.state, state)
}
})
})
}
}
func TestCreate_rejectsUnknownPlacement(t *testing.T) {
t.Parallel()
tests := []struct {
name string
mutate func(req *provider.CreateRequest)
wantMsg string
}{
{
name: "zone",
mutate: func(req *provider.CreateRequest) { req.Placement.Zone = "nas9" },
wantMsg: `placement.zone "nas9" is not in kvm2.hosts`,
},
{
name: "machineType",
mutate: func(req *provider.CreateRequest) { req.Placement.MachineType = "xlarge" },
wantMsg: `placement.machineType "xlarge" is not in kvm2.machineTypes`,
},
{
name: "image",
mutate: func(req *provider.CreateRequest) { req.Placement.Image = "ubuntu-24" },
wantMsg: `placement.image "ubuntu-24" is neither a key in kvm2.images nor an http(s) URL`,
},
}
for _, tc := range tests {
t.Run(tc.name, func(t *testing.T) {
t.Parallel()
fx := newFixture(t, 1)
req := fx.req(testName)
tc.mutate(&req)
id, err := fx.p.Create(fx.ctx, req)
if id != "" {
t.Errorf("Create = %q, want no ID", id)
}
assertProviderError(t, err, "create", provider.ErrPermanent)
if !strings.Contains(err.Error(), tc.wantMsg) {
t.Errorf("error = %q, want substring %q", err, tc.wantMsg)
}
if got := fx.fake.callLog(); len(got) != 0 {
t.Errorf("backend calls = %v, want none before validation", got)
}
if got := fx.dialer.dials.Load(); got != 0 {
t.Errorf("dials = %d, want 0", got)
}
})
}
}
func TestCreate_adoptionWithForeignUIDIsPermanent(t *testing.T) {
t.Parallel()
fx := newFixture(t, 1)
fx.fake.addDomain(testName, libvirt.DomainRunning, managedMetadata("other-uid", time.Now()))
id, err := fx.p.Create(fx.ctx, fx.req(testName))
if id != "" {
t.Errorf("Create = %q, want no ID", id)
}
assertProviderError(t, err, "create", provider.ErrPermanent)
if !strings.Contains(err.Error(), "other-uid") || !strings.Contains(err.Error(), testUID) {
t.Errorf("error = %q, want both UIDs named", err)
}
for _, rpc := range []string{"DomainCreate", "DomainDefineXML", "StorageVolCreateXML", "DomainDestroy"} {
if got := fx.fake.callCount(rpc); got != 0 {
t.Errorf("%s calls = %d, want 0", rpc, got)
}
}
}
func TestCreate_adoptsOnlyMarkedDomains(t *testing.T) {
t.Parallel()
tests := []struct {
name string
metadata string
wantAdopt bool
}{
{name: "matching uid", metadata: managedMetadata(testUID, time.Now()), wantAdopt: true},
{name: "managed, empty uid", metadata: managedMetadata("", time.Now()), wantAdopt: true},
{name: "no metadata element", metadata: ""},
{name: "element without the managed mark", metadata: `<instance><uid>` + testUID + `</uid></instance>`},
{name: "managed false", metadata: `<instance><managed>false</managed><uid>` + testUID + `</uid></instance>`},
}
for _, tc := range tests {
t.Run(tc.name, func(t *testing.T) {
t.Parallel()
fx := newFixture(t, 1)
fx.fake.addDomain(testName, libvirt.DomainRunning, tc.metadata)
id, err := fx.p.Create(fx.ctx, fx.req(testName))
if tc.wantAdopt {
if err != nil || id != formatProviderID(testHost, testName) {
t.Fatalf("Create = %q, %v; want the existing domain adopted", id, err)
}
} else {
if id != "" {
t.Errorf("Create = %q, want no ID", id)
}
assertProviderError(t, err, "create", provider.ErrPermanent)
if !strings.Contains(err.Error(), "carries no kvm2 mark") {
t.Errorf("error = %v, want the missing mark named", err)
}
}
for _, rpc := range []string{"DomainDefineXML", "DomainCreate", "StorageVolCreateXML", "DomainDestroy"} {
if got := fx.fake.callCount(rpc); got != 0 {
t.Errorf("%s calls = %d, want 0", rpc, got)
}
}
})
}
}
func TestCreate_poolOutOfSpaceIsQuotaExceeded(t *testing.T) {
t.Parallel()
fx := newFixture(t, 1)
fx.fake.mu.Lock()
fx.fake.pools[fakePoolName].available = 1 << 30
fx.fake.mu.Unlock()
_, err := fx.p.Create(fx.ctx, fx.req(testName))
assertProviderError(t, err, "create", provider.ErrQuotaExceeded)
if !strings.Contains(err.Error(), fakePoolName) || !strings.Contains(err.Error(), "needed") {
t.Errorf("error = %q, want the pool and the needed bytes named", err)
}
for _, rpc := range []string{"StorageVolCreateXML", "StorageVolUpload", "DomainDefineXML"} {
if got := fx.fake.callCount(rpc); got != 0 {
t.Errorf("%s calls = %d, want 0", rpc, got)
}
}
}
func TestCreate_libvirtNotFoundIsTransientNotNotFound(t *testing.T) {
t.Parallel()
tests := []struct {
name string
rpc string
err error
}{
{name: "volume vanished", rpc: "StorageVolGetPath", err: fakeErr(libvirt.ErrNoStorageVol, "Storage volume not found")},
{name: "domain vanished", rpc: "DomainGetState", err: fakeErr(libvirt.ErrNoDomain, "Domain not found")},
}
for _, tc := range tests {
t.Run(tc.name, func(t *testing.T) {
t.Parallel()
fx := newFixture(t, 1)
fx.fake.failWith(tc.rpc, tc.err)
_, err := fx.p.Create(fx.ctx, fx.req(testName))
assertProviderError(t, err, "create", provider.ErrTransient)
if errors.Is(err, provider.ErrNotFound) {
t.Errorf("Create error = %v, must never be NotFound", err)
}
})
}
}
func TestCreate_cloudInitSentinelNeverLoggedOrInErrors(t *testing.T) {
t.Parallel()
const sentinel = "SENTINEL-s3cr3t-passw0rd-7f3a"
fx := newFixture(t, 2)
fx.dialer.backend = observe(fx.fake, testProvider, testHost)
fx.fake.failWith("StorageVolUpload", fakeErr(libvirt.ErrInternalError, "internal error: upload failed"))
req := fx.req(testName)
req.CloudInit = "#cloud-config\nwrite_files:\n - content: " + sentinel + "\n"
_, err := fx.p.Create(fx.ctx, req)
assertProviderError(t, err, "create", provider.ErrTransient)
if strings.Contains(err.Error(), sentinel) {
t.Errorf("error carries the cloud-init: %v", err)
}
lines := fx.logs.all()
if len(lines) == 0 {
t.Fatal("no log lines captured at V(2)")
}
for _, l := range lines {
if strings.Contains(l, sentinel) {
t.Errorf("log line carries the cloud-init: %s", l)
}
}
if got := fx.logs.matching(`"cloudInitBytes"=` + strconv.Itoa(len(req.CloudInit))); len(got) != 1 {
t.Errorf("cloudInitBytes line = %d, want 1:\n%s", len(got), strings.Join(lines, "\n"))
}
}
func TestCreate_metadataWrittenInDefineCall(t *testing.T) {
t.Parallel()
fx := newFixture(t, 1)
fx.p.now = func() time.Time { return time.Date(2026, 9, 16, 12, 0, 0, 0, time.UTC) }
fx.create(testName)
fx.withDomain(testName, func(d *fakeDomain) {
for _, want := range []string{`xmlns="` + metadataURI + `"`, "<uid>" + testUID + "</uid>", "<managed>true</managed>", "<createdAt>2026-09-16T12:00:00Z</createdAt>"} {
if !strings.Contains(d.xml, want) {
t.Errorf("defined XML lacks %s:\n%s", want, d.xml)
}
}
meta, err := parseInstanceMetadata(d.metadata)
if err != nil {
t.Fatalf("metadata the define stored does not parse: %v", err)
}
if !meta.managed() || meta.UID != testUID || meta.createdAt().IsZero() {
t.Errorf("metadata = %+v, want managed, uid %s and a createdAt", meta, testUID)
}
})
}
func TestCreate_seedAlwaysRebuilt(t *testing.T) {
t.Parallel()
fx := newFixture(t, 1)
fx.create(testName)
seed := seedVolumeName(testName)
first := fx.fake.vol(seed)
if first == nil {
t.Fatal("no seed volume after Create")
}
uploads := fx.fake.callCount("StorageVolUpload")
// A crash between the seed upload and the define leaves the volumes
// without a domain; the next attempt must not trust the seed.
fx.fake.mu.Lock()
delete(fx.fake.domains, testName)
fx.fake.mu.Unlock()
fx.create(testName)
second := fx.fake.vol(seed)
if second == nil || second == first {
t.Errorf("seed volume reused across attempts")
}
if got := fx.fake.callCount("StorageVolUpload"); got != uploads+1 {
t.Errorf("StorageVolUpload calls = %d, want %d", got, uploads+1)
}
if len(second.content) != len(first.content) || !bytes.Contains(second.content, []byte(seedLabel)) {
t.Errorf("rebuilt seed is %d bytes, first was %d; want a complete seed of the same size", len(second.content), len(first.content))
}
}
internal/provider/kvm2/delete.go
+98
View File
@@ -0,0 +1,98 @@
package kvm2
import (
"context"
"fmt"
"github.com/digitalocean/go-libvirt"
"github.com/go-logr/logr"
"gitea.home.hrajfrisbee.cz/kacerr/egress-proxies-operator/internal/provider"
)
// undefineFlags removes everything libvirt keeps for a domain beside its
// definition; UEFI domains keep NVRAM and undefine refuses without the flag.
const undefineFlags = libvirt.DomainUndefineManagedSave | libvirt.DomainUndefineSnapshotsMetadata | libvirt.DomainUndefineNvram
// Delete is spec entry 11: under the per-name lock, destroy and undefine
// the domain, then remove its overlay and seed. Everything already gone is
// success, and nothing is left when nil is returned, so Get afterwards is
// NotFound. The base image stays: it is a shared cache.
func (p *Provider) Delete(ctx context.Context, providerID string) error {
hostName, name, err := parseProviderID(providerID)
if err != nil {
return provider.Wrap(provider.ErrPermanent, "delete", p.name, providerID, err)
}
log := p.logger(ctx).WithValues("host", hostName, "name", name)
host, err := p.host(hostName)
if err != nil {
log.V(1).Info("host is not configured; delete has nothing to clean up", "id", providerID)
return nil
}
unlock, err := p.lockName(ctx, name)
if err != nil {
return p.wrapErr("delete", providerID, err)
}
defer unlock()
be, err := p.hosts.get(ctx, host)
if err != nil {
logAPIError(log, "delete", err)
return p.wrapErr("delete", providerID, err)
}
if err := removeDomain(ctx, log, be, name); err != nil {
logAPIError(log, "delete", err)
return p.wrapErr("delete", providerID, err)
}
if err := removeVolumes(ctx, log, be, host.Pool, name); err != nil {
logAPIError(log, "delete", err)
return p.wrapErr("delete", providerID, err)
}
log.V(1).Info("kvm2 domain deleted")
return nil
}
func removeDomain(ctx context.Context, log logr.Logger, be backend, name string) error {
dom, err := be.DomainLookupByName(ctx, name)
if hasCode(err, libvirt.ErrNoDomain) {
log.V(1).Info("domain already gone")
return nil
}
if err != nil {
return fmt.Errorf("looking up domain %s: %w", name, err)
}
switch err := be.DomainDestroy(ctx, dom); {
case err == nil:
log.V(1).Info("domain destroyed")
case hasCode(err, libvirt.ErrOperationInvalid):
log.V(1).Info("domain not running, destroy skipped")
case hasCode(err, libvirt.ErrNoDomain):
log.V(1).Info("domain already gone")
return nil
default:
return fmt.Errorf("destroying domain %s: %w", name, err)
}
if err := be.DomainUndefineFlags(ctx, dom, undefineFlags); err != nil && !hasCode(err, libvirt.ErrNoDomain) {
return fmt.Errorf("undefining domain %s: %w", name, err)
}
log.V(1).Info("domain undefined")
return nil
}
func removeVolumes(ctx context.Context, log logr.Logger, be backend, poolName, name string) error {
pool, err := be.StoragePoolLookupByName(ctx, poolName)
if hasCode(err, libvirt.ErrNoStoragePool) {
log.V(1).Info("pool not found, no volumes to delete", "pool", poolName)
return nil
}
if err != nil {
return fmt.Errorf("looking up storage pool %s: %w", poolName, err)
}
for _, vol := range []string{overlayVolumeName(name), seedVolumeName(name)} {
if err := deleteVolumeIfPresent(ctx, be, pool, vol); err != nil {
return err
}
}
log.V(1).Info("per-VM volumes deleted")
return nil
}
internal/provider/kvm2/delete_test.go
+237
View File
@@ -0,0 +1,237 @@
package kvm2
import (
"errors"
"slices"
"strings"
"sync"
"testing"
"time"
"github.com/digitalocean/go-libvirt"
"gitea.home.hrajfrisbee.cz/kacerr/egress-proxies-operator/internal/provider"
)
func TestDelete_malformedIDIsPermanent(t *testing.T) {
t.Parallel()
for _, id := range []string{"", "proxy-x", "hosts/beelink/proxy-x", "hosts//domains/proxy-x", "hosts/beelink/domains/"} {
t.Run(id, func(t *testing.T) {
t.Parallel()
fx := newFixture(t, 1)
err := fx.p.Delete(fx.ctx, id)
assertProviderError(t, err, "delete", provider.ErrPermanent)
var pe *provider.Error
if errors.As(err, &pe) && pe.ID != id {
t.Errorf("error.ID = %q, want %q", pe.ID, id)
}
if got := fx.dialer.dials.Load(); got != 0 {
t.Errorf("dials = %d, want 0", got)
}
})
}
}
func TestDelete_twiceSucceeds(t *testing.T) {
t.Parallel()
fx := newFixture(t, 1)
id := fx.create(testName)
for i := range 2 {
if err := fx.p.Delete(fx.ctx, id); err != nil {
t.Fatalf("Delete #%d error = %v", i+1, err)
}
}
if fx.hasDomain(testName) {
t.Error("domain still defined after Delete")
}
if got := fx.logs.matching("domain already gone"); len(got) != 1 {
t.Errorf("'already gone' lines = %d, want 1 from the second Delete", len(got))
}
if got := fx.logs.matching("kvm2 domain deleted"); len(got) != 2 {
t.Errorf("'deleted' lines = %d, want 2", len(got))
}
}
func TestDelete_removesOverlayAndSeed(t *testing.T) {
t.Parallel()
fx := newFixture(t, 1)
id := fx.create(testName)
if err := fx.p.Delete(fx.ctx, id); err != nil {
t.Fatalf("Delete() error = %v", err)
}
for _, name := range []string{overlayVolumeName(testName), seedVolumeName(testName)} {
if fx.fake.vol(name) != nil {
t.Errorf("volume %s still present after Delete", name)
}
}
base := "base-" + testProvider + "-debian-13.qcow2"
for _, name := range []string{base, base + ".sha256"} {
if fx.fake.vol(name) == nil {
t.Errorf("shared base volume %s removed by Delete", name)
}
}
}
func TestDelete_runningDomainDestroyedFirst(t *testing.T) {
t.Parallel()
fx := newFixture(t, 1)
id := fx.create(testName)
before := len(fx.fake.callLog())
if err := fx.p.Delete(fx.ctx, id); err != nil {
t.Fatalf("Delete() error = %v", err)
}
calls := fx.fake.callLog()[before:]
destroy := slices.Index(calls, "DomainDestroy")
undefine := slices.Index(calls, "DomainUndefineFlags")
volDelete := slices.Index(calls, "StorageVolDelete")
if destroy < 0 || undefine < 0 || volDelete < 0 || !(destroy < undefine && undefine < volDelete) {
t.Errorf("call order = %v, want destroy before undefine before volume deletes", calls)
}
if n := strings.Count(strings.Join(calls, " "), "StorageVolDelete"); n != 2 {
t.Errorf("StorageVolDelete calls = %d, want 2 (overlay and seed)", n)
}
}
func TestDelete_stoppedDomainSkipsDestroy(t *testing.T) {
t.Parallel()
fx := newFixture(t, 1)
id := fx.create(testName)
fx.withDomain(testName, func(d *fakeDomain) { d.state = libvirt.DomainShutoff })
if err := fx.p.Delete(fx.ctx, id); err != nil {
t.Fatalf("Delete() error = %v", err)
}
if fx.hasDomain(testName) {
t.Error("domain still defined after Delete")
}
if got := fx.logs.matching("domain not running, destroy skipped"); len(got) != 1 {
t.Errorf("'destroy skipped' lines = %d, want 1:\n%s", len(got), strings.Join(fx.logs.all(), "\n"))
}
if got := fx.fake.callCount("DomainUndefineFlags"); got != 1 {
t.Errorf("DomainUndefineFlags calls = %d, want 1", got)
}
}
func TestDelete_toleratesMissingVolumesAndPool(t *testing.T) {
t.Parallel()
t.Run("domain without volumes", func(t *testing.T) {
t.Parallel()
fx := newFixture(t, 1)
fx.fake.addDomain(testName, libvirt.DomainRunning, managedMetadata(testUID, time.Now()))
if err := fx.p.Delete(fx.ctx, formatProviderID(testHost, testName)); err != nil {
t.Fatalf("Delete() error = %v", err)
}
if fx.hasDomain(testName) {
t.Error("domain still defined after Delete")
}
if got := fx.fake.callCount("StorageVolDelete"); got != 0 {
t.Errorf("StorageVolDelete calls = %d, want 0 for absent volumes", got)
}
})
t.Run("pool gone", func(t *testing.T) {
t.Parallel()
fx := newFixture(t, 1)
fx.p.cfg.Hosts[0].Pool = "retired"
fx.fake.addDomain(testName, libvirt.DomainShutoff, managedMetadata(testUID, time.Now()))
if err := fx.p.Delete(fx.ctx, formatProviderID(testHost, testName)); err != nil {
t.Fatalf("Delete() error = %v", err)
}
if fx.hasDomain(testName) {
t.Error("domain still defined after Delete")
}
if got := fx.logs.matching("pool not found, no volumes to delete"); len(got) != 1 || !strings.Contains(got[0], `"pool"="retired"`) {
t.Errorf("'pool not found' V(1) line = %v", got)
}
})
t.Run("nothing at all", func(t *testing.T) {
t.Parallel()
fx := newFixture(t, 1)
if err := fx.p.Delete(fx.ctx, formatProviderID(testHost, testName)); err != nil {
t.Fatalf("Delete() error = %v", err)
}
})
}
func TestDelete_unconfiguredHostIsNoOp(t *testing.T) {
t.Parallel()
fx := newFixture(t, 1)
id := formatProviderID("nas9", testName)
if err := fx.p.Delete(fx.ctx, id); err != nil {
t.Fatalf("Delete() error = %v, want nil", err)
}
if got := fx.dialer.dials.Load(); got != 0 {
t.Errorf("dials = %d, want 0", got)
}
if got := fx.fake.callLog(); len(got) != 0 {
t.Errorf("backend calls = %v, want none", got)
}
if got := fx.logs.matching("host is not configured; delete has nothing to clean up"); len(got) != 1 || !strings.Contains(got[0], `"host"="nas9"`) {
t.Errorf("V(1) line = %v, want one naming the host", got)
}
}
func TestDelete_noOpLinesAreSilentAtV0(t *testing.T) {
t.Parallel()
fx := newFixture(t, 0)
fx.p.cfg.Hosts[0].Pool = "retired"
for _, id := range []string{formatProviderID("nas9", testName), formatProviderID(testHost, testName)} {
if err := fx.p.Delete(fx.ctx, id); err != nil {
t.Fatalf("Delete(%s) error = %v, want nil", id, err)
}
}
if lines := fx.logs.all(); len(lines) != 0 {
t.Errorf("Delete logged %d lines at V(0), want none:\n%s", len(lines), strings.Join(lines, "\n"))
}
}
func TestDelete_concurrentSameNameSafe(t *testing.T) {
t.Parallel()
fx := newFixture(t, 1)
id := fx.create(testName)
const n = 10
errs := make([]error, n)
var wg sync.WaitGroup
for i := range n {
wg.Add(1)
go func() {
defer wg.Done()
errs[i] = fx.p.Delete(fx.ctx, id)
}()
}
wg.Wait()
for i, err := range errs {
if err != nil {
t.Errorf("Delete #%d error = %v, want nil", i, err)
}
}
if fx.hasDomain(testName) {
t.Error("domain still defined after concurrent Deletes")
}
for _, name := range []string{overlayVolumeName(testName), seedVolumeName(testName)} {
if fx.fake.vol(name) != nil {
t.Errorf("volume %s still present", name)
}
}
if got := fx.fake.callCount("DomainUndefineFlags"); got != 1 {
t.Errorf("DomainUndefineFlags calls = %d, want 1 (the lock serialises the rest)", got)
}
}
func TestDelete_undefineUsesNvramFlag(t *testing.T) {
t.Parallel()
fx := newFixture(t, 1)
id := fx.create(testName)
if err := fx.p.Delete(fx.ctx, id); err != nil {
t.Fatalf("Delete() error = %v", err)
}
fx.fake.mu.Lock()
flags, ok := fx.fake.undefineFlags[testName]
fx.fake.mu.Unlock()
if !ok {
t.Fatal("DomainUndefineFlags was not called")
}
for _, want := range []libvirt.DomainUndefineFlagsValues{libvirt.DomainUndefineNvram, libvirt.DomainUndefineManagedSave, libvirt.DomainUndefineSnapshotsMetadata} {
if flags&want == 0 {
t.Errorf("undefine flags = %d, lack %d", flags, want)
}
}
}
internal/provider/kvm2/errors.go
+97
View File
@@ -0,0 +1,97 @@
package kvm2
import (
"errors"
"fmt"
"strings"
"github.com/digitalocean/go-libvirt"
"golang.org/x/crypto/ssh/knownhosts"
"gitea.home.hrajfrisbee.cz/kacerr/egress-proxies-operator/internal/provider"
)
// permanentf and transientf classify an error at the source, so classify
// keeps that class no matter what wraps it later.
func permanentf(format string, args ...any) error {
return fmt.Errorf("%w: %s", provider.ErrPermanent, fmt.Sprintf(format, args...))
}
func transientf(format string, args ...any) error {
return fmt.Errorf("%w: %s", provider.ErrTransient, fmt.Sprintf(format, args...))
}
// sshAuthFailed is the substring x/crypto/ssh puts in every authentication
// failure ("ssh: handshake failed: ssh: unable to authenticate, attempted
// methods [...]").
const sshAuthFailed = "unable to authenticate"
// classify maps any error reaching the provider's boundary to one of the
// four contract sentinels, per spec entry 13. op is the contract operation:
// a vanished domain or volume inside Create is Transient, because the
// contract forbids NotFound from Create.
func classify(op string, err error) error {
if err == nil {
return provider.ErrTransient
}
for _, class := range []error{provider.ErrNotFound, provider.ErrQuotaExceeded, provider.ErrPermanent, provider.ErrTransient} {
if errors.Is(err, class) {
return class
}
}
if code, ok := libvirtCode(err); ok {
return classifyLibvirt(op, code)
}
var keyErr *knownhosts.KeyError
if errors.As(err, &keyErr) {
return provider.ErrPermanent
}
if strings.Contains(err.Error(), sshAuthFailed) {
return provider.ErrPermanent
}
// Everything else retries: a missing key or known_hosts file (the
// optional Secret has not landed yet), context cancellation and
// deadlines, net.OpError, io.EOF, and anything unrecognised.
return provider.ErrTransient
}
func classifyLibvirt(op string, code libvirt.ErrorNumber) error {
switch code {
case libvirt.ErrNoDomain, libvirt.ErrNoStorageVol:
if op == "create" {
return provider.ErrTransient
}
return provider.ErrNotFound
case libvirt.ErrNoStoragePool, libvirt.ErrXMLError, libvirt.ErrNoSupport, libvirt.ErrInvalidArg,
libvirt.ErrAuthFailed, libvirt.ErrAccessDenied:
return provider.ErrPermanent
default:
return provider.ErrTransient
}
}
// libvirtCode extracts the libvirt error number from err, if it is or wraps
// a libvirt.Error. That type has a value receiver, so errors.As needs a
// value target.
func libvirtCode(err error) (libvirt.ErrorNumber, bool) {
var le libvirt.Error
if !errors.As(err, &le) {
return 0, false
}
return libvirt.ErrorNumber(le.Code), true
}
func hasCode(err error, code libvirt.ErrorNumber) bool {
got, ok := libvirtCode(err)
return ok && got == code
}
// libvirtErrorFields returns the key/value pairs a log line carries for a
// libvirt error, or nil when err is not one.
func libvirtErrorFields(err error) []any {
code, ok := libvirtCode(err)
if !ok {
return nil
}
return []any{"libvirtCode", int(code), "libvirtError", code.String()}
}
internal/provider/kvm2/errors_test.go
+146
View File
@@ -0,0 +1,146 @@
package kvm2
import (
"context"
"errors"
"fmt"
"io"
"io/fs"
"net"
"os"
"testing"
"github.com/digitalocean/go-libvirt"
"golang.org/x/crypto/ssh"
"golang.org/x/crypto/ssh/knownhosts"
"gitea.home.hrajfrisbee.cz/kacerr/egress-proxies-operator/internal/provider"
)
func libvirtErr(code libvirt.ErrorNumber, msg string) error {
return libvirt.Error{Code: uint32(code), Message: msg}
}
// Captured from x/crypto/ssh when the offered key is not in authorized_keys.
const sshAuthFailedMsg = "ssh: handshake failed: ssh: unable to authenticate, attempted methods [none publickey], no supported methods remain"
func TestClassify_table(t *testing.T) {
t.Parallel()
tests := []struct {
name string
op string
err error
want error
}{
{name: "nil is transient", op: "get", err: nil, want: provider.ErrTransient},
{name: "ErrNoDomain on get", op: "get", err: libvirtErr(libvirt.ErrNoDomain, "Domain not found: no domain with matching name 'proxy-x'"), want: provider.ErrNotFound},
{name: "ErrNoDomain on delete", op: "delete", err: libvirtErr(libvirt.ErrNoDomain, "Domain not found"), want: provider.ErrNotFound},
{name: "ErrNoDomain on create is transient", op: "create", err: libvirtErr(libvirt.ErrNoDomain, "Domain not found"), want: provider.ErrTransient},
{name: "ErrNoStorageVol on get", op: "get", err: libvirtErr(libvirt.ErrNoStorageVol, "Storage volume not found: no storage vol with matching name 'proxy-x.qcow2'"), want: provider.ErrNotFound},
{name: "ErrNoStorageVol on create is transient", op: "create", err: libvirtErr(libvirt.ErrNoStorageVol, "Storage volume not found"), want: provider.ErrTransient},
{name: "ErrNoStoragePool", op: "create", err: libvirtErr(libvirt.ErrNoStoragePool, "Storage pool not found: no storage pool with matching name 'default'"), want: provider.ErrPermanent},
{name: "ErrXMLError", op: "create", err: libvirtErr(libvirt.ErrXMLError, "XML error: missing domain name"), want: provider.ErrPermanent},
{name: "ErrNoSupport", op: "create", err: libvirtErr(libvirt.ErrNoSupport, "this function is not supported by the connection driver"), want: provider.ErrPermanent},
{name: "ErrInvalidArg", op: "get", err: libvirtErr(libvirt.ErrInvalidArg, "invalid argument: unsupported flags"), want: provider.ErrPermanent},
{name: "ErrAuthFailed", op: "list", err: libvirtErr(libvirt.ErrAuthFailed, "authentication failed: access denied by policy"), want: provider.ErrPermanent},
{name: "ErrAccessDenied", op: "list", err: libvirtErr(libvirt.ErrAccessDenied, "access denied"), want: provider.ErrPermanent},
{name: "ErrOperationInvalid is transient", op: "create", err: libvirtErr(libvirt.ErrOperationInvalid, "Requested operation is not valid: domain is already running"), want: provider.ErrTransient},
{name: "ErrInternalError is transient", op: "get", err: libvirtErr(libvirt.ErrInternalError, "internal error: connection closed"), want: provider.ErrTransient},
{name: "unknown libvirt code is transient", op: "get", err: libvirtErr(libvirt.ErrorNumber(9999), "?"), want: provider.ErrTransient},
{name: "wrapped libvirt error keeps its code", op: "get", err: fmt.Errorf("looking up domain: %w", libvirtErr(libvirt.ErrNoDomain, "Domain not found")), want: provider.ErrNotFound},
{name: "knownhosts unknown host", op: "get", err: &knownhosts.KeyError{}, want: provider.ErrPermanent},
{name: "knownhosts mismatch", op: "get", err: &knownhosts.KeyError{Want: []knownhosts.KnownKey{{Filename: "/kh", Line: 1, Key: testPublicKey(t)}}}, want: provider.ErrPermanent},
{name: "wrapped knownhosts error", op: "create", err: fmt.Errorf("dialing beelink: %w", &knownhosts.KeyError{}), want: provider.ErrPermanent},
{name: "ssh unable to authenticate", op: "create", err: errors.New(sshAuthFailedMsg), want: provider.ErrPermanent},
{name: "wrapped ssh unable to authenticate", op: "list", err: fmt.Errorf("dialing beelink: %w", errors.New(sshAuthFailedMsg)), want: provider.ErrPermanent},
{name: "missing key file is transient", op: "create", err: fmt.Errorf("reading ssh key file /k: %w", fs.ErrNotExist), want: provider.ErrTransient},
{name: "os.PathError not exist is transient", op: "create", err: &os.PathError{Op: "open", Path: "/k", Err: fs.ErrNotExist}, want: provider.ErrTransient},
{name: "context canceled", op: "get", err: context.Canceled, want: provider.ErrTransient},
{name: "context deadline", op: "get", err: context.DeadlineExceeded, want: provider.ErrTransient},
{name: "net.OpError", op: "get", err: &net.OpError{Op: "dial", Net: "tcp", Err: errors.New("connection refused")}, want: provider.ErrTransient},
{name: "io.EOF", op: "list", err: io.EOF, want: provider.ErrTransient},
{name: "io.ErrUnexpectedEOF", op: "list", err: io.ErrUnexpectedEOF, want: provider.ErrTransient},
{name: "anything else is transient", op: "get", err: errors.New("anything"), want: provider.ErrTransient},
{name: "pre-classified NotFound stays", op: "create", err: fmt.Errorf("x: %w", provider.ErrNotFound), want: provider.ErrNotFound},
{name: "pre-classified Quota stays", op: "create", err: fmt.Errorf("x: %w", provider.ErrQuotaExceeded), want: provider.ErrQuotaExceeded},
{name: "pre-classified Permanent stays", op: "get", err: fmt.Errorf("x: %w", permanentf("bad %s", "thing")), want: provider.ErrPermanent},
{name: "pre-classified Transient stays", op: "get", err: fmt.Errorf("x: %w", transientf("blip %d", 1)), want: provider.ErrTransient},
{name: "pre-classified beats a libvirt code underneath", op: "get", err: fmt.Errorf("%w: %w", provider.ErrPermanent, libvirtErr(libvirt.ErrNoDomain, "x")), want: provider.ErrPermanent},
}
for _, tc := range tests {
t.Run(tc.name, func(t *testing.T) {
t.Parallel()
if got := classify(tc.op, tc.err); !errors.Is(got, tc.want) {
t.Errorf("classify(%q, %v) = %v, want %v", tc.op, tc.err, got, tc.want)
}
})
}
}
func TestHasCode(t *testing.T) {
t.Parallel()
tests := []struct {
name string
err error
code libvirt.ErrorNumber
want bool
}{
{name: "matching code", err: libvirtErr(libvirt.ErrNoDomain, "x"), code: libvirt.ErrNoDomain, want: true},
{name: "matching code wrapped", err: fmt.Errorf("w: %w", libvirtErr(libvirt.ErrStorageVolExist, "x")), code: libvirt.ErrStorageVolExist, want: true},
{name: "other code", err: libvirtErr(libvirt.ErrNoDomain, "x"), code: libvirt.ErrNoStorageVol, want: false},
{name: "not a libvirt error", err: errors.New("x"), code: libvirt.ErrNoDomain, want: false},
{name: "nil", err: nil, code: libvirt.ErrNoDomain, want: false},
}
for _, tc := range tests {
t.Run(tc.name, func(t *testing.T) {
t.Parallel()
if got := hasCode(tc.err, tc.code); got != tc.want {
t.Errorf("hasCode(%v, %v) = %v, want %v", tc.err, tc.code, got, tc.want)
}
})
}
}
func TestLibvirtErrorFields(t *testing.T) {
t.Parallel()
got := libvirtErrorFields(fmt.Errorf("w: %w", libvirtErr(libvirt.ErrNoDomain, "x")))
want := []any{"libvirtCode", 42, "libvirtError", "ErrNoDomain"}
if len(got) != len(want) {
t.Fatalf("libvirtErrorFields() = %v, want %v", got, want)
}
for i := range want {
if got[i] != want[i] {
t.Errorf("libvirtErrorFields()[%d] = %v (%T), want %v (%T)", i, got[i], got[i], want[i], want[i])
}
}
if fields := libvirtErrorFields(errors.New("x")); fields != nil {
t.Errorf("libvirtErrorFields(non-libvirt) = %v, want nil", fields)
}
if fields := libvirtErrorFields(libvirtErr(libvirt.ErrorNumber(9999), "x")); fields[3] != "ErrorNumber(9999)" {
t.Errorf("libvirtErrorFields(unknown code)[3] = %v, want ErrorNumber(9999)", fields[3])
}
}
func TestPermanentfAndTransientf(t *testing.T) {
t.Parallel()
perm := permanentf("host %q missing", "x")
if !errors.Is(perm, provider.ErrPermanent) || perm.Error() != `provider: permanent error: host "x" missing` {
t.Errorf("permanentf() = %q, want ErrPermanent with the message", perm)
}
trans := transientf("retry %d", 3)
if !errors.Is(trans, provider.ErrTransient) || trans.Error() != "provider: transient error: retry 3" {
t.Errorf("transientf() = %q, want ErrTransient with the message", trans)
}
}
// testPublicKey parses a fixed ed25519 public key so a knownhosts.KeyError
// with a Want entry can be built without generating keys.
func testPublicKey(t *testing.T) ssh.PublicKey {
t.Helper()
const line = "ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIGcZBhnMPTyDwFn9lBhwIutqLb2bSlhcZoqpuWv7VuKI test"
key, _, _, _, err := ssh.ParseAuthorizedKey([]byte(line))
if err != nil {
t.Fatalf("ParseAuthorizedKey: %v", err)
}
return key
}
internal/provider/kvm2/fake_test.go
+623
View File
@@ -0,0 +1,623 @@
package kvm2
import (
"context"
"crypto/sha256"
"encoding/hex"
"encoding/xml"
"fmt"
"io"
"slices"
"sync"
"sync/atomic"
"github.com/digitalocean/go-libvirt"
"gitea.home.hrajfrisbee.cz/kacerr/egress-proxies-operator/internal/provider"
)
// fakeBackend is the in-memory backend of spec entry 21: enough of a
// libvirt host for the provider's Create, Get, Delete and ListByTag, with
// per-method error hooks and blocking, call counters and an ordered log.
type fakeBackend struct {
// mu, domains and pools are the host's state; a twin shares them, so
// two connections to the same fake host see the same volumes.
mu *sync.Mutex
domains map[string]*fakeDomain
pools map[string]*fakePool
hostname string
version uint64
// hooks returns the error a method fails with before touching state.
hooks map[string]func() error
// block makes a method wait on its channel (or ctx) before running.
block map[string]chan struct{}
// discardUploads makes StorageVolUpload stream the reader through a
// sha256 in fixed chunks instead of holding the bytes, for tests that
// push more through the fake than they want in memory.
discardUploads bool
// startDelayPolls, when set, makes DomainCreate leave the domain in
// nostate; it turns running on the startDelayPolls-th DomainGetState
// after that, as a host whose QEMU is still settling would.
startDelayPolls int
// stateAfterStart, when not nostate, is what DomainCreate leaves the
// domain in instead of running, as a guest that dies at once would.
stateAfterStart libvirt.DomainState
// undefineFlags records the flags of the last DomainUndefineFlags per
// domain name.
undefineFlags map[string]libvirt.DomainUndefineFlagsValues
calls map[string]int
log []string
closed int
deadCh chan struct{}
deadMu sync.Once
}
type fakeDomain struct {
state libvirt.DomainState
xml string
metadata string
addrs map[uint32][]libvirt.DomainInterface
uuid libvirt.UUID
id int32
// pendingStart counts the DomainGetState calls left before a delayed
// start (see startDelayPolls) turns the domain running.
pendingStart int
}
type fakePool struct {
state libvirt.StoragePoolState
capacity uint64
available uint64
vols map[string]*fakeVol
}
type fakeVol struct {
content []byte
capacity uint64
// uploaded and sha256 record a discarded upload (see discardUploads).
uploaded int64
sha256 string
}
const fakePoolName = "default"
func newFakeBackend() *fakeBackend {
return &fakeBackend{
mu: &sync.Mutex{},
domains: make(map[string]*fakeDomain),
pools: map[string]*fakePool{fakePoolName: newFakePool()},
hostname: "fake-host",
version: 11006000,
hooks: make(map[string]func() error),
block: make(map[string]chan struct{}),
undefineFlags: make(map[string]libvirt.DomainUndefineFlagsValues),
calls: make(map[string]int),
deadCh: make(chan struct{}),
}
}
// twin returns a second connection to the same fake host: it shares the
// domains and pools (and their lock) but has its own hooks, counters, log,
// close count and liveness, so a test can tell which connection did what.
func (f *fakeBackend) twin() *fakeBackend {
return &fakeBackend{
mu: f.mu,
domains: f.domains,
pools: f.pools,
hostname: f.hostname,
version: f.version,
hooks: make(map[string]func() error),
block: make(map[string]chan struct{}),
discardUploads: f.discardUploads,
undefineFlags: make(map[string]libvirt.DomainUndefineFlagsValues),
calls: make(map[string]int),
deadCh: make(chan struct{}),
}
}
// vol returns the volume by name in the default pool, or nil.
func (f *fakeBackend) vol(name string) *fakeVol {
f.mu.Lock()
defer f.mu.Unlock()
return f.pools[fakePoolName].vols[name]
}
// closeCount is closed read under the lock, for twins closed by another
// goroutine.
func (f *fakeBackend) closeCount() int {
f.mu.Lock()
defer f.mu.Unlock()
return f.closed
}
func newFakePool() *fakePool {
return &fakePool{state: libvirt.StoragePoolRunning, capacity: 100 << 30, available: 50 << 30, vols: make(map[string]*fakeVol)}
}
func (f *fakeBackend) dead() <-chan struct{} { return f.deadCh }
func (f *fakeBackend) markDead() { f.deadMu.Do(func() { close(f.deadCh) }) }
// addDomain defines a domain directly, bypassing DomainDefineXML.
func (f *fakeBackend) addDomain(name string, state libvirt.DomainState, metadata string) *fakeDomain {
f.mu.Lock()
defer f.mu.Unlock()
d := &fakeDomain{state: state, metadata: metadata, addrs: make(map[uint32][]libvirt.DomainInterface), uuid: fakeUUID(name)}
f.domains[name] = d
return d
}
func (f *fakeBackend) failWith(method string, err error) {
f.mu.Lock()
defer f.mu.Unlock()
f.hooks[method] = func() error { return err }
}
func (f *fakeBackend) blockOn(method string) chan struct{} {
f.mu.Lock()
defer f.mu.Unlock()
ch := make(chan struct{})
f.block[method] = ch
return ch
}
func (f *fakeBackend) callCount(method string) int {
f.mu.Lock()
defer f.mu.Unlock()
return f.calls[method]
}
func (f *fakeBackend) callLog() []string {
f.mu.Lock()
defer f.mu.Unlock()
return slices.Clone(f.log)
}
func fakeUUID(name string) libvirt.UUID {
var u libvirt.UUID
copy(u[:], name)
return u
}
func fakeErr(code libvirt.ErrorNumber, format string, args ...any) error {
return libvirt.Error{Code: uint32(code), Message: fmt.Sprintf(format, args...)}
}
// enter records the call, honours a block and runs the hook. It is called
// without f.mu held; state access happens afterwards under the lock.
func (f *fakeBackend) enter(ctx context.Context, method string) error {
f.mu.Lock()
f.calls[method]++
f.log = append(f.log, method)
block := f.block[method]
hook := f.hooks[method]
f.mu.Unlock()
if block != nil {
select {
case <-block:
case <-ctx.Done():
return ctx.Err()
}
}
if hook != nil {
return hook()
}
return nil
}
func (f *fakeBackend) domain(name string) (*fakeDomain, error) {
d, ok := f.domains[name]
if !ok {
return nil, fakeErr(libvirt.ErrNoDomain, "Domain not found: no domain with matching name '%s'", name)
}
return d, nil
}
func (f *fakeBackend) pool(name string) (*fakePool, error) {
p, ok := f.pools[name]
if !ok {
return nil, fakeErr(libvirt.ErrNoStoragePool, "Storage pool not found: no storage pool with matching name '%s'", name)
}
return p, nil
}
func (f *fakeBackend) volume(vol libvirt.StorageVol) (*fakeVol, error) {
p, err := f.pool(vol.Pool)
if err != nil {
return nil, err
}
v, ok := p.vols[vol.Name]
if !ok {
return nil, fakeErr(libvirt.ErrNoStorageVol, "Storage volume not found: no storage vol with matching name '%s'", vol.Name)
}
return v, nil
}
func (f *fakeBackend) libvirtDomain(name string, d *fakeDomain) libvirt.Domain {
id := int32(-1)
if isActive(d.state) {
id = d.id
}
return libvirt.Domain{Name: name, UUID: d.uuid, ID: id}
}
func (f *fakeBackend) DomainLookupByName(ctx context.Context, name string) (libvirt.Domain, error) {
if err := f.enter(ctx, "DomainLookupByName"); err != nil {
return libvirt.Domain{}, err
}
f.mu.Lock()
defer f.mu.Unlock()
d, err := f.domain(name)
if err != nil {
return libvirt.Domain{}, err
}
return f.libvirtDomain(name, d), nil
}
func (f *fakeBackend) DomainDefineXML(ctx context.Context, doc string) (libvirt.Domain, error) {
if err := f.enter(ctx, "DomainDefineXML"); err != nil {
return libvirt.Domain{}, err
}
var parsed struct {
Name string `xml:"name"`
Metadata struct {
Inner string `xml:",innerxml"`
} `xml:"metadata"`
}
if err := xml.Unmarshal([]byte(doc), &parsed); err != nil || parsed.Name == "" {
return libvirt.Domain{}, fakeErr(libvirt.ErrXMLError, "XML error: %v", err)
}
f.mu.Lock()
defer f.mu.Unlock()
d, ok := f.domains[parsed.Name]
if !ok {
d = &fakeDomain{state: libvirt.DomainShutoff, addrs: make(map[uint32][]libvirt.DomainInterface), uuid: fakeUUID(parsed.Name)}
f.domains[parsed.Name] = d
}
d.xml = doc
d.metadata = parsed.Metadata.Inner
return f.libvirtDomain(parsed.Name, d), nil
}
func (f *fakeBackend) DomainCreate(ctx context.Context, dom libvirt.Domain) error {
if err := f.enter(ctx, "DomainCreate"); err != nil {
return err
}
f.mu.Lock()
defer f.mu.Unlock()
d, err := f.domain(dom.Name)
if err != nil {
return err
}
if isActive(d.state) {
return fakeErr(libvirt.ErrOperationInvalid, "Requested operation is not valid: domain is already running")
}
switch {
case f.startDelayPolls > 0:
d.state = libvirt.DomainNostate
d.pendingStart = f.startDelayPolls
return nil
case f.stateAfterStart != libvirt.DomainNostate:
d.state = f.stateAfterStart
return nil
}
d.state = libvirt.DomainRunning
d.id = int32(len(f.domains))
return nil
}
func (f *fakeBackend) DomainDestroy(ctx context.Context, dom libvirt.Domain) error {
if err := f.enter(ctx, "DomainDestroy"); err != nil {
return err
}
f.mu.Lock()
defer f.mu.Unlock()
d, err := f.domain(dom.Name)
if err != nil {
return err
}
if !isActive(d.state) {
return fakeErr(libvirt.ErrOperationInvalid, "Requested operation is not valid: domain is not running")
}
d.state = libvirt.DomainShutoff
return nil
}
func (f *fakeBackend) DomainUndefineFlags(ctx context.Context, dom libvirt.Domain, flags libvirt.DomainUndefineFlagsValues) error {
if err := f.enter(ctx, "DomainUndefineFlags"); err != nil {
return err
}
f.mu.Lock()
defer f.mu.Unlock()
if _, err := f.domain(dom.Name); err != nil {
return err
}
f.undefineFlags[dom.Name] = flags
delete(f.domains, dom.Name)
return nil
}
func (f *fakeBackend) DomainGetState(ctx context.Context, dom libvirt.Domain, _ uint32) (int32, int32, error) {
if err := f.enter(ctx, "DomainGetState"); err != nil {
return 0, 0, err
}
f.mu.Lock()
defer f.mu.Unlock()
d, err := f.domain(dom.Name)
if err != nil {
return 0, 0, err
}
if d.pendingStart > 0 {
d.pendingStart--
if d.pendingStart == 0 {
d.state = libvirt.DomainRunning
d.id = int32(len(f.domains))
}
}
return int32(d.state), 0, nil
}
func (f *fakeBackend) DomainGetMetadata(ctx context.Context, dom libvirt.Domain, typ int32, uri libvirt.OptString, _ libvirt.DomainModificationImpact) (string, error) {
if err := f.enter(ctx, "DomainGetMetadata"); err != nil {
return "", err
}
f.mu.Lock()
defer f.mu.Unlock()
d, err := f.domain(dom.Name)
if err != nil {
return "", err
}
if typ != int32(libvirt.DomainMetadataElement) || len(uri) != 1 || uri[0] != metadataURI || d.metadata == "" {
return "", fakeErr(libvirt.ErrNoDomainMetadata, "Requested metadata element is not present")
}
return d.metadata, nil
}
func (f *fakeBackend) DomainInterfaceAddresses(ctx context.Context, dom libvirt.Domain, source uint32, _ uint32) ([]libvirt.DomainInterface, error) {
if err := f.enter(ctx, "DomainInterfaceAddresses"); err != nil {
return nil, err
}
f.mu.Lock()
defer f.mu.Unlock()
d, err := f.domain(dom.Name)
if err != nil {
return nil, err
}
if !isActive(d.state) {
return nil, fakeErr(libvirt.ErrOperationInvalid, "Requested operation is not valid: domain is not running")
}
return slices.Clone(d.addrs[source]), nil
}
func (f *fakeBackend) ConnectListAllDomains(ctx context.Context, _ int32, _ libvirt.ConnectListAllDomainsFlags) ([]libvirt.Domain, uint32, error) {
if err := f.enter(ctx, "ConnectListAllDomains"); err != nil {
return nil, 0, err
}
f.mu.Lock()
defer f.mu.Unlock()
names := make([]string, 0, len(f.domains))
for name := range f.domains {
names = append(names, name)
}
slices.Sort(names)
out := make([]libvirt.Domain, 0, len(names))
for _, name := range names {
out = append(out, f.libvirtDomain(name, f.domains[name]))
}
return out, uint32(len(out)), nil
}
func (f *fakeBackend) StoragePoolLookupByName(ctx context.Context, name string) (libvirt.StoragePool, error) {
if err := f.enter(ctx, "StoragePoolLookupByName"); err != nil {
return libvirt.StoragePool{}, err
}
f.mu.Lock()
defer f.mu.Unlock()
if _, err := f.pool(name); err != nil {
return libvirt.StoragePool{}, err
}
return libvirt.StoragePool{Name: name, UUID: fakeUUID(name)}, nil
}
func (f *fakeBackend) StoragePoolRefresh(ctx context.Context, pool libvirt.StoragePool, _ uint32) error {
if err := f.enter(ctx, "StoragePoolRefresh"); err != nil {
return err
}
f.mu.Lock()
defer f.mu.Unlock()
_, err := f.pool(pool.Name)
return err
}
func (f *fakeBackend) StoragePoolGetInfo(ctx context.Context, pool libvirt.StoragePool) (uint8, uint64, uint64, uint64, error) {
if err := f.enter(ctx, "StoragePoolGetInfo"); err != nil {
return 0, 0, 0, 0, err
}
f.mu.Lock()
defer f.mu.Unlock()
p, err := f.pool(pool.Name)
if err != nil {
return 0, 0, 0, 0, err
}
return uint8(p.state), p.capacity, p.capacity - p.available, p.available, nil
}
func (f *fakeBackend) StorageVolLookupByName(ctx context.Context, pool libvirt.StoragePool, name string) (libvirt.StorageVol, error) {
if err := f.enter(ctx, "StorageVolLookupByName"); err != nil {
return libvirt.StorageVol{}, err
}
f.mu.Lock()
defer f.mu.Unlock()
vol := libvirt.StorageVol{Pool: pool.Name, Name: name, Key: fakeVolPath(pool.Name, name)}
if _, err := f.volume(vol); err != nil {
return libvirt.StorageVol{}, err
}
return vol, nil
}
func fakeVolPath(pool, name string) string { return "/pool/" + pool + "/" + name }
func (f *fakeBackend) StorageVolGetPath(ctx context.Context, vol libvirt.StorageVol) (string, error) {
if err := f.enter(ctx, "StorageVolGetPath"); err != nil {
return "", err
}
f.mu.Lock()
defer f.mu.Unlock()
if _, err := f.volume(vol); err != nil {
return "", err
}
return fakeVolPath(vol.Pool, vol.Name), nil
}
func (f *fakeBackend) StorageVolCreateXML(ctx context.Context, pool libvirt.StoragePool, doc string, _ libvirt.StorageVolCreateFlags) (libvirt.StorageVol, error) {
if err := f.enter(ctx, "StorageVolCreateXML"); err != nil {
return libvirt.StorageVol{}, err
}
var parsed struct {
Name string `xml:"name"`
Capacity uint64 `xml:"capacity"`
}
if err := xml.Unmarshal([]byte(doc), &parsed); err != nil || parsed.Name == "" {
return libvirt.StorageVol{}, fakeErr(libvirt.ErrXMLError, "XML error: %v", err)
}
f.mu.Lock()
defer f.mu.Unlock()
p, err := f.pool(pool.Name)
if err != nil {
return libvirt.StorageVol{}, err
}
if _, exists := p.vols[parsed.Name]; exists {
return libvirt.StorageVol{}, fakeErr(libvirt.ErrStorageVolExist, "storage volume '%s' exists already", parsed.Name)
}
p.vols[parsed.Name] = &fakeVol{capacity: parsed.Capacity}
return libvirt.StorageVol{Pool: pool.Name, Name: parsed.Name, Key: fakeVolPath(pool.Name, parsed.Name)}, nil
}
func (f *fakeBackend) StorageVolUpload(ctx context.Context, vol libvirt.StorageVol, r io.Reader, _, _ uint64, _ libvirt.StorageVolUploadFlags) error {
if err := f.enter(ctx, "StorageVolUpload"); err != nil {
return err
}
var (
content []byte
uploaded int64
digest string
)
if r != nil {
var err error
if f.discardUploads {
h := sha256.New()
if uploaded, err = io.CopyBuffer(h, r, make([]byte, 64<<10)); err != nil {
return err
}
digest = hex.EncodeToString(h.Sum(nil))
} else if content, err = io.ReadAll(r); err != nil {
return err
}
}
f.mu.Lock()
defer f.mu.Unlock()
v, err := f.volume(vol)
if err != nil {
return err
}
v.content = content
v.uploaded = uploaded
v.sha256 = digest
return nil
}
func (f *fakeBackend) StorageVolDownload(ctx context.Context, vol libvirt.StorageVol, w io.Writer, _, _ uint64, _ libvirt.StorageVolDownloadFlags) error {
if err := f.enter(ctx, "StorageVolDownload"); err != nil {
return err
}
f.mu.Lock()
v, err := f.volume(vol)
var content []byte
if v != nil {
content = slices.Clone(v.content)
}
f.mu.Unlock()
if err != nil {
return err
}
if w == nil {
return nil
}
_, err = w.Write(content)
return err
}
func (f *fakeBackend) StorageVolDelete(ctx context.Context, vol libvirt.StorageVol, _ libvirt.StorageVolDeleteFlags) error {
if err := f.enter(ctx, "StorageVolDelete"); err != nil {
return err
}
f.mu.Lock()
defer f.mu.Unlock()
if _, err := f.volume(vol); err != nil {
return err
}
delete(f.pools[vol.Pool].vols, vol.Name)
return nil
}
func (f *fakeBackend) ConnectGetLibVersion(ctx context.Context) (uint64, error) {
if err := f.enter(ctx, "ConnectGetLibVersion"); err != nil {
return 0, err
}
return f.version, nil
}
func (f *fakeBackend) ConnectGetHostname(ctx context.Context) (string, error) {
if err := f.enter(ctx, "ConnectGetHostname"); err != nil {
return "", err
}
return f.hostname, nil
}
func (f *fakeBackend) Close() error {
f.mu.Lock()
defer f.mu.Unlock()
f.closed++
return nil
}
// stubDialer covers every dialer a test needs: it counts dials, fails with
// err when set, waits on block (or ctx) when set, and hands out backend, or
// one from fresh per call, or a new fakeBackend when neither is given.
type stubDialer struct {
backend backend
fresh func() backend
err error
block chan struct{}
dials atomic.Int32
mu sync.Mutex
hosts []string
}
func (d *stubDialer) dial(ctx context.Context, host provider.KVMHost) (backend, error) {
d.dials.Add(1)
d.mu.Lock()
d.hosts = append(d.hosts, host.Name)
d.mu.Unlock()
if d.block != nil {
select {
case <-d.block:
case <-ctx.Done():
return nil, fmt.Errorf("%w: dialing %s: %w", provider.ErrTransient, host.Name, ctx.Err())
}
}
if d.err != nil {
return nil, d.err
}
switch {
case d.fresh != nil:
return d.fresh(), nil
case d.backend != nil:
return d.backend, nil
default:
return newFakeBackend(), nil
}
}
internal/provider/kvm2/get.go
+133
View File
@@ -0,0 +1,133 @@
package kvm2
import (
"context"
"errors"
"fmt"
"github.com/digitalocean/go-libvirt"
"github.com/go-logr/logr"
"gitea.home.hrajfrisbee.cz/kacerr/egress-proxies-operator/internal/provider"
)
// Get is spec entries 7 to 9: look the domain up by name, read the mark
// back, then state and address. Only ErrNoDomain becomes NotFound; every
// other failure is classified, never guessed.
func (p *Provider) Get(ctx context.Context, providerID string) (*provider.Instance, error) {
hostName, name, err := parseProviderID(providerID)
if err != nil {
return nil, provider.Wrap(provider.ErrPermanent, "get", p.name, providerID, err)
}
host, err := p.host(hostName)
if err != nil {
return nil, p.wrapErr("get", providerID, err)
}
log := p.logger(ctx).WithValues("host", host.Name, "name", name)
be, err := p.hosts.get(ctx, host)
if err != nil {
logAPIError(log, "get", err)
return nil, p.wrapErr("get", providerID, err)
}
inst, err := get(ctx, log, be, host.Name, name)
if err != nil {
if hasCode(err, libvirt.ErrNoDomain) {
log.V(1).Info("kvm2 domain not found")
} else {
logAPIError(log, "get", err)
}
return nil, p.wrapErr("get", providerID, err)
}
log.V(1).Info("kvm2 domain fetched", "state", inst.State, "ip", inst.IP, "uid", inst.UID)
return inst, nil
}
// get builds the Instance from the domain's metadata, its state and, for an
// active domain, its discovered address.
func get(ctx context.Context, log logr.Logger, be backend, hostName, name string) (*provider.Instance, error) {
dom, err := be.DomainLookupByName(ctx, name)
if err != nil {
return nil, fmt.Errorf("looking up domain %s: %w", name, err)
}
meta, err := readMetadata(ctx, be, dom)
if err != nil {
return nil, err
}
state, _, err := be.DomainGetState(ctx, dom, 0)
if err != nil {
return nil, fmt.Errorf("reading state of domain %s: %w", dom.Name, err)
}
s := libvirt.DomainState(state)
var ip string
if isActive(s) {
if ip, err = address(ctx, log, be, dom); err != nil {
return nil, err
}
}
return &provider.Instance{
ID: formatProviderID(hostName, dom.Name),
IP: ip,
State: mapState(s, ip),
UID: meta.UID,
CreatedAt: meta.createdAt(),
}, nil
}
// readMetadata returns the kvm2 mark of a domain, or the zero value (not
// managed, empty UID) when the domain carries none. A mark that does not
// parse is Permanent, from parseInstanceMetadata.
func readMetadata(ctx context.Context, be backend, dom libvirt.Domain) (instanceMetadata, error) {
raw, err := be.DomainGetMetadata(ctx, dom, int32(libvirt.DomainMetadataElement), libvirt.OptString{metadataURI}, libvirt.DomainAffectCurrent)
if hasCode(err, libvirt.ErrNoDomainMetadata) {
return instanceMetadata{}, nil
}
if err != nil {
return instanceMetadata{}, fmt.Errorf("reading metadata of domain %s: %w", dom.Name, err)
}
return parseInstanceMetadata(raw)
}
// addressSources is the discovery order of spec entry 7: the guest agent
// answers where the host's ARP table never sees guest traffic.
var addressSources = []struct {
name string
src libvirt.DomainInterfaceAddressesSource
}{
{"agent", libvirt.DomainInterfaceAddressesSrcAgent},
{"arp", libvirt.DomainInterfaceAddressesSrcArp},
}
// address returns the first usable IPv4 from the first source that yields
// one, or "" for "no address yet". One source failing is normal and is
// skipped: a booting VM has no agent yet, while the host's ARP table
// simply reports nothing, since it never sees traffic addressed
// elsewhere. A connection already gone, and every source failing, are
// errors instead — otherwise a link some other reconcile killed reads as
// a running VM that lost its address, and the controller clears its IP
// and the health engine drops its accumulated state.
func address(ctx context.Context, log logr.Logger, be backend, dom libvirt.Domain) (string, error) {
errs := make([]error, 0, len(addressSources))
for _, s := range addressSources {
ifaces, err := be.DomainInterfaceAddresses(ctx, dom, uint32(s.src), 0)
if err != nil {
if isDead(be) || errors.Is(err, context.Canceled) || errors.Is(err, context.DeadlineExceeded) {
return "", fmt.Errorf("reading addresses of domain %s from the %s source: %w", dom.Name, s.name, err)
}
log.V(2).Info("no addresses from source", "source", s.name, "error", err.Error())
errs = append(errs, fmt.Errorf("%s source: %w", s.name, err))
continue
}
if ip := firstIPv4(ifaces); ip != "" {
return ip, nil
}
}
// Transient explicitly: classify would otherwise pick a libvirt code
// out of the joined error, and ErrInvalidArg, ErrAccessDenied and
// ErrNoSupport are Permanent, which would latch a running proxy as
// Failed over an address read.
if len(errs) == len(addressSources) {
return "", fmt.Errorf("%w: reading addresses of domain %s: every source failed: %w",
provider.ErrTransient, dom.Name, errors.Join(errs...))
}
return "", nil
}
internal/provider/kvm2/get_test.go
+295
View File
@@ -0,0 +1,295 @@
package kvm2
import (
"context"
"errors"
"fmt"
"strings"
"testing"
"time"
"github.com/digitalocean/go-libvirt"
"gitea.home.hrajfrisbee.cz/kacerr/egress-proxies-operator/internal/provider"
)
func TestGet_afterDeleteReturnsNotFound(t *testing.T) {
t.Parallel()
fx := newFixture(t, 1)
id := fx.create(testName)
if err := fx.p.Delete(fx.ctx, id); err != nil {
t.Fatalf("Delete() error = %v", err)
}
inst, err := fx.p.Get(fx.ctx, id)
if inst != nil {
t.Errorf("Get after Delete = %+v, want nil", inst)
}
assertProviderError(t, err, "get", provider.ErrNotFound)
}
func TestGet_duringDeleteStillReturnsInstance(t *testing.T) {
t.Parallel()
t.Run("destroyed but still defined is stopped", func(t *testing.T) {
t.Parallel()
fx := newFixture(t, 1)
id := fx.create(testName)
fx.withDomain(testName, func(d *fakeDomain) { d.state = libvirt.DomainShutoff })
inst := fx.get(id)
if inst.State != provider.StateStopped || inst.UID != testUID {
t.Errorf("Get = %+v, want Stopped with uid %s", inst, testUID)
}
})
t.Run("undefine in flight still answers", func(t *testing.T) {
t.Parallel()
fx := newFixture(t, 1)
id := fx.create(testName)
release := fx.fake.blockOn("DomainUndefineFlags")
done := make(chan error, 1)
go func() { done <- fx.p.Delete(fx.ctx, id) }()
waitFor(t, "the undefine to block", func() bool { return fx.fake.callCount("DomainUndefineFlags") == 1 })
inst, err := fx.p.Get(fx.ctx, id)
if err != nil {
t.Fatalf("Get during Delete error = %v, want the instance", err)
}
if inst.State != provider.StateStopped || inst.ID != id {
t.Errorf("Get during Delete = %+v, want Stopped %s", inst, id)
}
close(release)
if err := <-done; err != nil {
t.Fatalf("Delete() error = %v", err)
}
_, err = fx.p.Get(fx.ctx, id)
assertProviderError(t, err, "get", provider.ErrNotFound)
})
}
func TestGet_backendBusyIsTransientNeverNotFound(t *testing.T) {
t.Parallel()
tests := []struct {
name string
err error
}{
{name: "operation invalid", err: fakeErr(libvirt.ErrOperationInvalid, "Requested operation is not valid: domain is being migrated")},
{name: "call timed out", err: fmt.Errorf("%w: libvirt call DomainGetState on beelink exceeded its 30s budget: %w", provider.ErrTransient, context.DeadlineExceeded)},
{name: "unknown libvirt code", err: fakeErr(libvirt.ErrInternalError, "internal error: qemu monitor unresponsive")},
{name: "not a libvirt error", err: errors.New("read tcp: connection reset by peer")},
}
for _, tc := range tests {
t.Run(tc.name, func(t *testing.T) {
t.Parallel()
fx := newFixture(t, 1)
id := fx.create(testName)
fx.fake.failWith("DomainGetState", tc.err)
inst, err := fx.p.Get(fx.ctx, id)
if inst != nil {
t.Errorf("Get = %+v, want nil", inst)
}
assertProviderError(t, err, "get", provider.ErrTransient)
if errors.Is(err, provider.ErrNotFound) {
t.Errorf("Get error = %v, must never be NotFound", err)
}
})
}
}
func TestGet_runningWithoutIPIsProvisioning(t *testing.T) {
t.Parallel()
fx := newFixture(t, 1)
inst := fx.get(fx.create(testName))
if inst.State != provider.StateProvisioning || inst.IP != "" {
t.Errorf("Get = %+v, want Provisioning with no IP", inst)
}
}
func TestGet_runningWithIPIsRunning(t *testing.T) {
t.Parallel()
fx := newFixture(t, 1)
id := fx.create(testName)
fx.setAddrs(testName, libvirt.DomainInterfaceAddressesSrcAgent, "127.0.0.1", "169.254.1.2", "10.0.0.5")
inst := fx.get(id)
if inst.State != provider.StateRunning || inst.IP != "10.0.0.5" {
t.Errorf("Get = %+v, want Running with 10.0.0.5", inst)
}
if got := fx.logs.matching("kvm2 domain fetched"); len(got) != 1 || !strings.Contains(got[0], `"ip"="10.0.0.5"`) || !strings.Contains(got[0], `"state"="Running"`) {
t.Errorf("fetched line = %v, want state and ip", got)
}
}
func TestGet_agentPreferredOverARP(t *testing.T) {
t.Parallel()
fx := newFixture(t, 1)
id := fx.create(testName)
fx.setAddrs(testName, libvirt.DomainInterfaceAddressesSrcAgent, "10.0.0.5")
fx.setAddrs(testName, libvirt.DomainInterfaceAddressesSrcArp, "10.0.0.6")
before := fx.fake.callCount("DomainInterfaceAddresses")
inst := fx.get(id)
if inst.IP != "10.0.0.5" {
t.Errorf("IP = %q, want the agent's 10.0.0.5", inst.IP)
}
if got := fx.fake.callCount("DomainInterfaceAddresses") - before; got != 1 {
t.Errorf("DomainInterfaceAddresses calls = %d, want 1 (ARP not consulted)", got)
}
}
func TestGet_arpFallbackWhenAgentFails(t *testing.T) {
t.Parallel()
fx := newFixture(t, 2)
id := fx.create(testName)
fx.setAddrs(testName, libvirt.DomainInterfaceAddressesSrcArp, "10.0.0.6")
var calls int
fx.fake.mu.Lock()
fx.fake.hooks["DomainInterfaceAddresses"] = func() error {
calls++
if calls%2 == 1 {
return fakeErr(libvirt.ErrAgentUnresponsive, "Guest agent is not responding")
}
return nil
}
fx.fake.mu.Unlock()
inst := fx.get(id)
if inst.State != provider.StateRunning || inst.IP != "10.0.0.6" {
t.Errorf("Get = %+v, want Running with ARP's 10.0.0.6", inst)
}
if got := fx.logs.matching("no addresses from source"); len(got) != 1 || !strings.Contains(got[0], `"source"="agent"`) {
t.Errorf("agent failure line = %v", got)
}
}
// TestGet_agentFailsAndARPEmptyIsProvisioning is a VM part way through
// booting on a plain bridge: the agent is not up yet, and the host's ARP
// table answers without error because it has nothing to report. One
// source failing must stay "no address yet", not an error.
func TestGet_agentFailsAndARPEmptyIsProvisioning(t *testing.T) {
t.Parallel()
fx := newFixture(t, 2)
id := fx.create(testName)
var calls int
fx.fake.mu.Lock()
fx.fake.hooks["DomainInterfaceAddresses"] = func() error {
calls++
if calls%2 == 1 {
return fakeErr(libvirt.ErrAgentUnresponsive, "Guest agent is not responding")
}
return nil
}
fx.fake.mu.Unlock()
inst := fx.get(id)
if inst.State != provider.StateProvisioning || inst.IP != "" {
t.Errorf("Get = %+v, want Provisioning with no IP", inst)
}
}
// TestGet_everyAddressSourceFailingIsTransient covers the other side of
// that line. A running VM whose address cannot be read is not a VM
// without an address: reporting Provisioning would make the controller
// clear its IP and the health engine drop its counters.
func TestGet_everyAddressSourceFailingIsTransient(t *testing.T) {
t.Parallel()
tests := []struct {
name string
setup func(fx *fixture)
}{
{
name: "both sources error",
setup: func(fx *fixture) {
fx.fake.failWith("DomainInterfaceAddresses", fakeErr(libvirt.ErrOperationInvalid, "domain is not running"))
},
},
{
name: "the pooled connection died under the read",
setup: func(fx *fixture) {
fx.fake.failWith("DomainInterfaceAddresses", transientf("libvirt call DomainInterfaceAddresses on beelink: connection is closed: keepalive failed"))
fx.fake.markDead()
},
},
}
for _, tc := range tests {
t.Run(tc.name, func(t *testing.T) {
t.Parallel()
fx := newFixture(t, 1)
id := fx.create(testName)
tc.setup(fx)
inst, err := fx.p.Get(fx.ctx, id)
if inst != nil {
t.Errorf("Get = %+v, want no instance", inst)
}
assertProviderError(t, err, "get", provider.ErrTransient)
})
}
}
func TestGet_readsUIDAndCreatedAtFromBackend(t *testing.T) {
t.Parallel()
fx := newFixture(t, 1)
created := time.Date(2026, 9, 16, 12, 0, 0, 0, time.UTC)
fx.p.now = func() time.Time { return created }
id := fx.create(testName)
inst := fx.get(id)
if inst.UID != testUID || !inst.CreatedAt.Equal(created) {
t.Errorf("Get = %+v, want uid %s created %s", inst, testUID, created)
}
later := created.Add(48 * time.Hour)
fx.withDomain(testName, func(d *fakeDomain) { d.metadata = managedMetadata("rewritten-uid", later) })
inst = fx.get(id)
if inst.UID != "rewritten-uid" || !inst.CreatedAt.Equal(later) {
t.Errorf("Get after metadata change = %+v, want uid rewritten-uid created %s (read back, not remembered)", inst, later)
}
}
func TestGet_unmanagedDomainHasEmptyUID(t *testing.T) {
t.Parallel()
fx := newFixture(t, 1)
fx.fake.addDomain("legacy-vm", libvirt.DomainRunning, "")
inst := fx.get(formatProviderID(testHost, "legacy-vm"))
if inst.UID != "" || !inst.CreatedAt.IsZero() || inst.State != provider.StateProvisioning {
t.Errorf("Get = %+v, want empty uid, zero createdAt, Provisioning", inst)
}
}
func TestGet_malformedIDIsPermanent(t *testing.T) {
t.Parallel()
ids := []string{"", "proxy-x", "beelink/proxy-x", "hosts/beelink/proxy-x", "hosts//domains/proxy-x", "hosts/beelink/domains/", "hosts/beelink/domains/a/b", "nodes/beelink/domains/proxy-x"}
for _, id := range ids {
t.Run(id, func(t *testing.T) {
t.Parallel()
fx := newFixture(t, 1)
inst, err := fx.p.Get(fx.ctx, id)
if inst != nil {
t.Errorf("Get(%q) = %+v, want nil", id, inst)
}
assertProviderError(t, err, "get", provider.ErrPermanent)
var pe *provider.Error
if errors.As(err, &pe) && pe.ID != id {
t.Errorf("error.ID = %q, want %q", pe.ID, id)
}
if got := fx.dialer.dials.Load(); got != 0 {
t.Errorf("dials = %d, want 0", got)
}
})
}
}
func TestGet_unconfiguredHostIsPermanent(t *testing.T) {
t.Parallel()
fx := newFixture(t, 1)
_, err := fx.p.Get(fx.ctx, formatProviderID("nas9", testName))
assertProviderError(t, err, "get", provider.ErrPermanent)
if !strings.Contains(err.Error(), `"nas9"`) {
t.Errorf("error = %v, want the host named", err)
}
if got := fx.dialer.dials.Load(); got != 0 {
t.Errorf("dials = %d, want 0", got)
}
}
func TestGet_unknownStateIsStopped(t *testing.T) {
t.Parallel()
fx := newFixture(t, 1)
fx.fake.addDomain(testName, libvirt.DomainState(42), managedMetadata(testUID, time.Now()))
inst := fx.get(formatProviderID(testHost, testName))
if inst.State != provider.StateStopped {
t.Errorf("State = %s, want Stopped for an unknown backend state", inst.State)
}
}
internal/provider/kvm2/hosts.go
+162
View File
@@ -0,0 +1,162 @@
package kvm2
import (
"context"
"fmt"
"sync"
"time"
"github.com/digitalocean/go-libvirt"
"go.opentelemetry.io/otel/attribute"
"go.opentelemetry.io/otel/codes"
"go.opentelemetry.io/otel/trace"
logf "sigs.k8s.io/controller-runtime/pkg/log"
"gitea.home.hrajfrisbee.cz/kacerr/egress-proxies-operator/internal/provider"
"gitea.home.hrajfrisbee.cz/kacerr/egress-proxies-operator/internal/tracing"
)
// dialer opens one connection to a host. The production dialer is
// libvirtDialer; tests inject one that hands out fakes.
type dialer interface {
dial(ctx context.Context, host provider.KVMHost) (backend, error)
}
// hostTable holds the pooled connection per host (spec entry 19). One dial
// per host at a time: concurrent callers wait for it and share the result.
// A dead backend (abandoned call, keepalive failure, socket lost) is dropped
// and redialled on the next get.
//
// What a cancelled or overlong call does to the shared connection (spec
// entry 18): the adapter kills it, so every other call in flight on that
// host fails once with Transient and its caller retries on a fresh
// connection. Callers that must not be exposed to that, the image transfer
// and the Checker, use dialFresh and get a connection of their own.
type hostTable struct {
dialer dialer
mu sync.Mutex
hosts map[string]*hostEntry
}
// hostEntry serialises dials for one host. The semaphore, not a mutex, so a
// waiter can give up when its ctx ends.
type hostEntry struct {
sem chan struct{}
backend backend
}
func newHostTable(d dialer) *hostTable {
return &hostTable{dialer: d, hosts: make(map[string]*hostEntry)}
}
func (t *hostTable) entry(name string) *hostEntry {
t.mu.Lock()
defer t.mu.Unlock()
e, ok := t.hosts[name]
if !ok {
e = &hostEntry{sem: make(chan struct{}, 1)}
t.hosts[name] = e
}
return e
}
// get returns the live pooled backend for host, dialling it when there is
// none. Waiting for another caller's dial ends with ctx.
func (t *hostTable) get(ctx context.Context, host provider.KVMHost) (backend, error) {
e := t.entry(host.Name)
select {
case e.sem <- struct{}{}:
case <-ctx.Done():
return nil, fmt.Errorf("%w: waiting for the connection to host %s: %w", provider.ErrTransient, host.Name, ctx.Err())
}
defer func() { <-e.sem }()
if e.backend != nil {
if !isDead(e.backend) {
return e.backend, nil
}
_ = e.backend.Close()
e.backend = nil
}
b, err := t.dialer.dial(ctx, host)
if err != nil {
return nil, err
}
e.backend = b
return b, nil
}
// dialFresh returns a dedicated connection that is never pooled. The
// caller owns it and must Close it.
func (t *hostTable) dialFresh(ctx context.Context, host provider.KVMHost) (backend, error) {
return t.dialer.dial(ctx, host)
}
// libvirtDialer is the production dialer: SSH transport, go-libvirt over
// the forwarded socket, the connect handshake, then the observe decorator,
// so provider code never holds an undecorated backend.
type libvirtDialer struct {
providerName string
cfg provider.KVMConfig
budgets budgets
keepalive time.Duration
tracing []tracing.Option
}
func newLibvirtDialer(providerName string, cfg provider.KVMConfig) *libvirtDialer {
return &libvirtDialer{
providerName: providerName,
cfg: cfg,
budgets: defaultBudgets,
keepalive: defaultKeepaliveInterval,
}
}
func (d *libvirtDialer) dial(ctx context.Context, host provider.KVMHost) (backend, error) {
ctx, span := tracing.StartSpan(ctx, tracing.Tracer(d.tracing...), "kvm2.dial",
trace.WithSpanKind(trace.SpanKindClient),
trace.WithAttributes(
attribute.String("kvm2.host", host.Name),
attribute.String("kvm2.address", host.Address),
attribute.String("kvm2.user", host.User),
))
defer span.End()
log := logf.FromContext(ctx).WithName("kvm2").WithValues("provider", d.providerName, "host", host.Name)
start := time.Now()
b, err := d.connect(ctx, host)
if err != nil {
span.SetStatus(codes.Error, err.Error())
kv := []any{"address", host.Address, "user", host.User, "duration", time.Since(start), "error", err.Error()}
if stage, ok := stageOf(err); ok {
kv = append(kv, "stage", string(stage))
}
kv = append(kv, libvirtErrorFields(err)...)
log.V(1).Info("libvirt host connect failed", kv...)
return nil, err
}
log.V(1).Info("libvirt host connected", "address", host.Address, "user", host.User, "duration", time.Since(start))
return observe(b, d.providerName, host.Name, d.tracing...), nil
}
// connect is the dial sequence under one budget: the SSH stages, then the
// libvirt handshake. On success the keepalive and the disconnect watcher
// run until the link dies.
func (d *libvirtDialer) connect(ctx context.Context, host provider.KVMHost) (*libvirtBackend, error) {
ctx, cancel := context.WithTimeout(ctx, d.budgets.dial)
defer cancel()
sock, err := dialSSHSocket(ctx, host, d.cfg, 0)
if err != nil {
return nil, err
}
b := newLibvirtBackend(libvirt.New(sock), host.Name, sock, d.budgets)
if err := b.connect(ctx); err != nil {
b.link.markDead("connect failed")
return nil, fmt.Errorf("connecting to libvirt on host %s: %w", host.Name, err)
}
go func() {
if err := sock.keepalive(d.keepalive, b.link.dead()); err != nil {
b.link.markDead(err.Error())
}
}()
return b, nil
}
internal/provider/kvm2/hosts_test.go
+195
View File
@@ -0,0 +1,195 @@
package kvm2
import (
"context"
"errors"
"sync"
"testing"
"time"
"gitea.home.hrajfrisbee.cz/kacerr/egress-proxies-operator/internal/provider"
)
var (
hostA = provider.KVMHost{Name: "beelink", Address: "beelink:22", User: "virt"}
hostB = provider.KVMHost{Name: "lab2", Address: "lab2:22", User: "virt"}
)
func TestHosts_oneDialPerHostUnderConcurrency(t *testing.T) {
t.Parallel()
shared := newFakeBackend()
d := &stubDialer{backend: shared}
table := newHostTable(d)
const n = 50
var wg sync.WaitGroup
got := make([]backend, n)
errs := make([]error, n)
for i := range n {
wg.Add(1)
go func() {
defer wg.Done()
got[i], errs[i] = table.get(context.Background(), hostA)
}()
}
wg.Wait()
for i := range n {
if errs[i] != nil {
t.Fatalf("get #%d error = %v", i, errs[i])
}
if got[i] != backend(shared) {
t.Fatalf("get #%d returned a different backend", i)
}
}
if d.dials.Load() != 1 {
t.Errorf("dials = %d, want 1", d.dials.Load())
}
}
func TestHosts_redialsAfterDrop(t *testing.T) {
t.Parallel()
first := newFakeBackend()
second := newFakeBackend()
backends := []backend{first, second}
d := &stubDialer{fresh: func() backend {
b := backends[0]
backends = backends[1:]
return b
}}
table := newHostTable(d)
b, err := table.get(context.Background(), hostA)
if err != nil || b != backend(first) {
t.Fatalf("first get = %v, %v, want the first backend", b, err)
}
if b, _ := table.get(context.Background(), hostA); b != backend(first) {
t.Fatal("second get did not reuse the live backend")
}
first.markDead()
b, err = table.get(context.Background(), hostA)
if err != nil || b != backend(second) {
t.Fatalf("get after drop = %v, %v, want the second backend", b, err)
}
if first.closed != 1 {
t.Errorf("dropped backend closed %d times, want 1", first.closed)
}
if d.dials.Load() != 2 {
t.Errorf("dials = %d, want 2", d.dials.Load())
}
}
func TestHosts_dialFreshIsNotCached(t *testing.T) {
t.Parallel()
d := &stubDialer{fresh: func() backend { return newFakeBackend() }}
table := newHostTable(d)
pooled, err := table.get(context.Background(), hostA)
if err != nil {
t.Fatalf("get error = %v", err)
}
fresh, err := table.dialFresh(context.Background(), hostA)
if err != nil {
t.Fatalf("dialFresh error = %v", err)
}
if fresh == pooled {
t.Fatal("dialFresh returned the pooled backend")
}
again, _ := table.get(context.Background(), hostA)
if again != pooled {
t.Error("get after dialFresh did not return the pooled backend")
}
if d.dials.Load() != 2 {
t.Errorf("dials = %d, want 2", d.dials.Load())
}
if fresh.(*fakeBackend).closed != 0 {
t.Error("dialFresh's backend was closed by the table; the caller owns it")
}
}
func TestHosts_waitingForDialEndsWithContext(t *testing.T) {
t.Parallel()
release := make(chan struct{})
d := &stubDialer{block: release}
table := newHostTable(d)
firstDone := make(chan error, 1)
go func() {
_, err := table.get(context.Background(), hostA)
firstDone <- err
}()
waitForDials(t, d, 1)
ctx, cancel := context.WithTimeout(context.Background(), 50*time.Millisecond)
defer cancel()
start := time.Now()
_, err := table.get(ctx, hostA)
if time.Since(start) > time.Second {
t.Errorf("second get took %v, want to give up with its ctx", time.Since(start))
}
if !errors.Is(err, provider.ErrTransient) || !errors.Is(err, context.DeadlineExceeded) {
t.Errorf("second get error = %v, want ErrTransient wrapping DeadlineExceeded", err)
}
close(release)
if err := <-firstDone; err != nil {
t.Errorf("first get error = %v, want nil once the dial completes", err)
}
if d.dials.Load() != 1 {
t.Errorf("dials = %d, want 1: the waiter must not dial itself", d.dials.Load())
}
}
func TestHosts_hostsDialIndependently(t *testing.T) {
t.Parallel()
release := make(chan struct{})
defer close(release)
dA := &stubDialer{block: release}
dB := &stubDialer{}
table := newHostTable(perHostDialer{hostA.Name: dA, hostB.Name: dB})
go func() { _, _ = table.get(context.Background(), hostA) }()
waitForDials(t, dA, 1)
ctx, cancel := context.WithTimeout(context.Background(), time.Second)
defer cancel()
start := time.Now()
if _, err := table.get(ctx, hostB); err != nil {
t.Fatalf("get(hostB) error = %v while hostA's dial is blocked", err)
}
if time.Since(start) > 500*time.Millisecond {
t.Errorf("get(hostB) took %v, want not to wait for hostA", time.Since(start))
}
}
func TestHosts_dialErrorIsNotCached(t *testing.T) {
t.Parallel()
d := &stubDialer{err: transientf("boom")}
table := newHostTable(d)
if _, err := table.get(context.Background(), hostA); !errors.Is(err, provider.ErrTransient) {
t.Fatalf("get error = %v, want the dial error", err)
}
d.err = nil
if _, err := table.get(context.Background(), hostA); err != nil {
t.Fatalf("get after the dialer recovered error = %v", err)
}
if d.dials.Load() != 2 {
t.Errorf("dials = %d, want 2", d.dials.Load())
}
}
// perHostDialer routes each host to its own stub.
type perHostDialer map[string]*stubDialer
func (p perHostDialer) dial(ctx context.Context, host provider.KVMHost) (backend, error) {
return p[host.Name].dial(ctx, host)
}
func waitForDials(t *testing.T, d *stubDialer, n int32) {
t.Helper()
deadline := time.Now().Add(2 * time.Second)
for d.dials.Load() < n {
if time.Now().After(deadline) {
t.Fatalf("dials = %d, want %d", d.dials.Load(), n)
}
time.Sleep(time.Millisecond)
}
}
internal/provider/kvm2/id.go
+28
View File
@@ -0,0 +1,28 @@
package kvm2
import (
"strings"
)
// The providerID is hosts/<host>/domains/<name>. Host names are validated at
// config load to contain no slash, and domain names come from NameFromUID,
// so a strict four-segment split is enough to parse it.
const (
idHostsSegment = "hosts"
idDomainsSegment = "domains"
)
func formatProviderID(host, name string) string {
return idHostsSegment + "/" + host + "/" + idDomainsSegment + "/" + name
}
// parseProviderID splits an ID back into host and domain name. Anything
// that is not exactly hosts/<host>/domains/<name> with both non-empty is
// ErrPermanent: no retry will make it resolve.
func parseProviderID(id string) (host, name string, err error) {
parts := strings.Split(id, "/")
if len(parts) != 4 || parts[0] != idHostsSegment || parts[2] != idDomainsSegment || parts[1] == "" || parts[3] == "" {
return "", "", permanentf("malformed providerID %q: want hosts/<host>/domains/<name>", id)
}
return parts[1], parts[3], nil
}
internal/provider/kvm2/id_test.go
+54
View File
@@ -0,0 +1,54 @@
package kvm2
import (
"errors"
"strings"
"testing"
"gitea.home.hrajfrisbee.cz/kacerr/egress-proxies-operator/internal/provider"
)
func TestFormatProviderID_roundTrip(t *testing.T) {
t.Parallel()
id := formatProviderID("beelink", "proxy-abc123def456ghij")
if id != "hosts/beelink/domains/proxy-abc123def456ghij" {
t.Fatalf("formatProviderID() = %q", id)
}
host, name, err := parseProviderID(id)
if err != nil || host != "beelink" || name != "proxy-abc123def456ghij" {
t.Errorf("parseProviderID(%q) = %q, %q, %v; want beelink, proxy-abc123def456ghij, nil", id, host, name, err)
}
}
func TestParseProviderID_malformed(t *testing.T) {
t.Parallel()
tests := []struct {
name string
id string
}{
{name: "empty", id: ""},
{name: "two segments", id: "hosts/h"},
{name: "three segments", id: "hosts/h/domains"},
{name: "empty host", id: "hosts//domains/n"},
{name: "empty name", id: "hosts/h/domains/"},
{name: "wrong first literal", id: "nodes/h/domains/n"},
{name: "wrong third literal", id: "hosts/h/vms/n"},
{name: "extra segment", id: "hosts/h/domains/n/extra"},
{name: "slash in name", id: "hosts/h/domains/a/b"},
}
for _, tc := range tests {
t.Run(tc.name, func(t *testing.T) {
t.Parallel()
host, name, err := parseProviderID(tc.id)
if err == nil {
t.Fatalf("parseProviderID(%q) = %q, %q, nil; want error", tc.id, host, name)
}
if !errors.Is(err, provider.ErrPermanent) {
t.Errorf("parseProviderID(%q) error = %v, want ErrPermanent", tc.id, err)
}
if !strings.Contains(err.Error(), tc.id) {
t.Errorf("parseProviderID(%q) error = %q, want it to name the ID", tc.id, err)
}
})
}
}
internal/provider/kvm2/image.go
+481
View File
@@ -0,0 +1,481 @@
package kvm2
import (
"context"
"crypto/sha256"
"encoding/hex"
"errors"
"fmt"
"io"
"net/http"
"net/url"
"regexp"
"strings"
"sync"
"time"
"github.com/digitalocean/go-libvirt"
"github.com/go-logr/logr"
"go.opentelemetry.io/otel/attribute"
"go.opentelemetry.io/otel/codes"
"go.opentelemetry.io/otel/trace"
"gitea.home.hrajfrisbee.cz/kacerr/egress-proxies-operator/internal/provider"
"gitea.home.hrajfrisbee.cz/kacerr/egress-proxies-operator/internal/tracing"
)
const (
// markerCapacityBytes is the marker volume's size: one lowercase hex
// sha256 digest, nothing else.
markerCapacityBytes = sha256.Size * 2
// markerReadCap bounds what is read back from a marker, so a wrong file
// under that name cannot grow memory.
markerReadCap = 1 << 10
// seedReserveBytes is what the quota precheck sets aside for the seed
// ISO next to the image and the overlay (spec entry 13).
seedReserveBytes = 16 << 20
)
var hexDigest = regexp.MustCompile(`^[0-9a-f]{64}$`)
// imageSource is a resolved placement.image (spec entries 3 and 4): the
// catalogue entry, or a URL given directly with the catalogue defaults, plus
// the key its base volume is named after.
type imageSource struct {
Entry string
Key string
provider.KVMImage
}
func (s imageSource) volumeName() string { return baseVolumeName(s.Entry, s.Key, s.Format) }
func (s imageSource) markerName() string { return markerVolumeName(s.volumeName()) }
// resolveImage maps placement.image to a source. cfg is already defaulted,
// so a catalogue entry carries its format and firmware; a URL gets qcow2
// and uefi. entry is the provider entry's name, which the base volume is
// named after so two entries never share one file.
func resolveImage(entry string, cfg provider.KVMConfig, image string) (imageSource, error) {
if img, ok := cfg.Images[image]; ok {
return imageSource{Entry: entry, Key: image, KVMImage: img}, nil
}
if provider.IsHTTPURL(image) {
return imageSource{
Entry: entry,
Key: urlImageKey(image),
KVMImage: provider.KVMImage{URL: image, Format: provider.KVMFormatQcow2, Firmware: provider.KVMFirmwareUEFI},
}, nil
}
return imageSource{}, permanentf("placement.image %q is neither a key in kvm2.images nor an http(s) URL", image)
}
// lockImage takes the per-(host, volume) lock of spec entry 19, or gives up
// with ctx: a caller whose ctx ends while another caller's download runs
// returns promptly (contract §1.5) instead of waiting out a 20-minute
// transfer.
func (p *Provider) lockImage(ctx context.Context, hostName, volume string) (func(), error) {
release, err := p.imageLocks.acquire(ctx, hostName+"/"+volume)
if err != nil {
return nil, fmt.Errorf("%w: waiting for image %s on host %s: %w", provider.ErrTransient, volume, hostName, err)
}
return release, nil
}
// ensureImage makes the base image present and complete in the host's pool
// and returns its path (spec entry 4). be is the caller's pooled connection
// and carries the cheap checks; the upload itself goes over a connection of
// its own. Concurrent callers for the same (host, volume) queue on the
// lock: the first fetches, the rest find the marker.
func (p *Provider) ensureImage(ctx context.Context, host provider.KVMHost, pool libvirt.StoragePool, be backend, src imageSource) (string, error) {
unlock, err := p.lockImage(ctx, host.Name, src.volumeName())
if err != nil {
return "", err
}
defer unlock()
log := p.logger(ctx).WithValues("host", host.Name, "volume", src.volumeName())
path, err := cachedImage(ctx, log, pool, be, src)
if err != nil || path != "" {
return path, err
}
vol, err := p.fetchImage(ctx, host, pool, be, src)
if err != nil {
return "", err
}
path, err = be.StorageVolGetPath(ctx, vol)
if err != nil {
return "", fmt.Errorf("reading the path of base volume %s: %w", vol.Name, err)
}
return path, nil
}
// cachedImage returns the base volume's path when the marker says the base
// is complete and its digest is the configured one (or none is configured).
// Anything else is cleared so the fetch starts from nothing, and "" is
// returned: a base without a marker is an upload that was cut off; a marker
// without a base, or one whose content is not a digest, is debris; a marker
// with a different digest means the catalogue entry changed.
func cachedImage(ctx context.Context, log logr.Logger, pool libvirt.StoragePool, be backend, src imageSource) (string, error) {
name, markerName := src.volumeName(), src.markerName()
marker, err := be.StorageVolLookupByName(ctx, pool, markerName)
switch {
case hasCode(err, libvirt.ErrNoStorageVol):
return "", deleteVolumeIfPresent(ctx, be, pool, name)
case err != nil:
return "", fmt.Errorf("looking up marker %s: %w", markerName, err)
}
digest, err := readMarker(ctx, be, marker)
if err != nil {
return "", err
}
switch {
case !hexDigest.MatchString(digest):
log.V(1).Info("base image marker holds no digest; refetching")
case src.SHA256 != "" && !strings.EqualFold(digest, src.SHA256):
// Refusing rather than replacing: every overlay on this host names
// the base as its backing file, and libvirt will unlink it under
// running domains without a word.
return "", permanentf("base image %s is cached with sha256 %s but %s is configured; add the image under a new key and move Proxies to it, or delete the Proxies using it and evict the base by hand",
name, digest, strings.ToLower(src.SHA256))
default:
base, err := be.StorageVolLookupByName(ctx, pool, name)
switch {
case hasCode(err, libvirt.ErrNoStorageVol):
return "", deleteVolumeIfPresent(ctx, be, pool, markerName)
case err != nil:
return "", fmt.Errorf("looking up base volume %s: %w", name, err)
}
path, err := be.StorageVolGetPath(ctx, base)
if err != nil {
return "", fmt.Errorf("reading the path of base volume %s: %w", name, err)
}
return path, nil
}
return "", errors.Join(deleteVolumeIfPresent(ctx, be, pool, markerName), deleteVolumeIfPresent(ctx, be, pool, name))
}
// readMarker returns the digest a marker volume holds, lowercased and
// stripped of padding. The volume is 64 bytes; the read is capped anyway.
func readMarker(ctx context.Context, be backend, marker libvirt.StorageVol) (string, error) {
w := &cappedWriter{limit: markerReadCap}
if err := be.StorageVolDownload(ctx, marker, w, 0, 0, 0); err != nil {
return "", fmt.Errorf("reading marker %s: %w", marker.Name, err)
}
return strings.ToLower(strings.TrimRight(string(w.buf), "\x00 \r\n\t")), nil
}
// cappedWriter keeps the first limit bytes and accepts the rest without
// storing them, so a download into it is bounded whatever its length.
type cappedWriter struct {
limit int
buf []byte
}
func (w *cappedWriter) Write(b []byte) (int, error) {
if room := w.limit - len(w.buf); room > 0 {
w.buf = append(w.buf, b[:min(room, len(b))]...)
}
return len(b), nil
}
// deleteVolumeIfPresent removes the named volume; one that is already gone
// is success.
func deleteVolumeIfPresent(ctx context.Context, be backend, pool libvirt.StoragePool, name string) error {
vol, err := be.StorageVolLookupByName(ctx, pool, name)
if hasCode(err, libvirt.ErrNoStorageVol) {
return nil
}
if err != nil {
return fmt.Errorf("looking up volume %s: %w", name, err)
}
if err := be.StorageVolDelete(ctx, vol, 0); err != nil && !hasCode(err, libvirt.ErrNoStorageVol) {
return fmt.Errorf("deleting volume %s: %w", name, err)
}
return nil
}
// fetchImage streams the image from its URL into a new base volume and then
// writes the marker, under the kvm2.download span (spec entry 17). The
// upload goes over a connection of its own (spec entry 18): abandoning a
// call kills the connection it ran on, and a 20-minute upload must not die
// because a poll for another Proxy was cancelled on the pooled one. The
// marker is written last, on the pooled connection, after
// StoragePoolRefresh made libvirt read the file's real format; its presence
// is what marks the base complete.
//
// Every exit between the base volume's creation and the marker deletes the
// base volume (best effort): the next attempt would delete it anyway on
// finding no marker, and removing it now frees the space at once.
func (p *Provider) fetchImage(ctx context.Context, host provider.KVMHost, pool libvirt.StoragePool, be backend, src imageSource) (libvirt.StorageVol, error) {
name := src.volumeName()
ctx, cancel := context.WithTimeout(ctx, defaultBudgets.transfer)
defer cancel()
ctx, span := tracing.StartSpan(ctx, tracing.Tracer(p.tracing...), "kvm2.download",
trace.WithSpanKind(trace.SpanKindClient),
trace.WithAttributes(
attribute.String("kvm2.host", host.Name),
attribute.String("kvm2.volume", name),
attribute.String("kvm2.url_host", urlHost(src.URL)),
attribute.Bool("kvm2.sha256_configured", src.SHA256 != ""),
))
defer span.End()
// Taken after the span starts, so the lines carry its ID.
log := p.logger(ctx).WithValues("host", host.Name, "volume", name)
start := time.Now()
// The body runs out of time before the surrounding budget does, so a
// mirror that stalls mid-image ends the upload through its reader
// rather than through the killed socket the budget would leave.
httpCtx, cancelHTTP := context.WithTimeout(ctx, defaultBudgets.transfer-transferGrace)
defer cancelHTTP()
resp, err := p.do(httpCtx, http.MethodGet, src.URL)
if err == nil {
err = checkResponse(src.URL, resp)
}
if err != nil {
span.SetStatus(codes.Error, err.Error())
return libvirt.StorageVol{}, err
}
defer func() { _ = resp.Body.Close() }()
size := resp.ContentLength
span.SetAttributes(attribute.Int64("kvm2.bytes", size))
log.V(1).Info("fetching base image into host pool", "url", src.URL, "bytes", size)
vol, digest, err := p.uploadImage(ctx, host, pool, src, resp.Body, size)
if err != nil {
// uploadImage has returned, so its dedicated connection is closed
// and libvirtd has let go of the stream before the file goes.
p.discardVolume(ctx, log, be, pool, name)
span.SetStatus(codes.Error, err.Error())
return libvirt.StorageVol{}, err
}
if err := be.StoragePoolRefresh(ctx, pool, 0); err != nil {
err = fmt.Errorf("refreshing storage pool %s after the upload: %w", pool.Name, err)
p.discardVolume(ctx, log, be, pool, name)
span.SetStatus(codes.Error, err.Error())
return libvirt.StorageVol{}, err
}
if err := writeMarker(ctx, be, pool, src.markerName(), digest); err != nil {
p.discardVolume(ctx, log, be, pool, name)
span.SetStatus(codes.Error, err.Error())
return libvirt.StorageVol{}, err
}
log.V(1).Info("base image cached", "bytes", size, "duration", time.Since(start), "sha256", digest)
return vol, nil
}
// uploadImage creates the base volume and streams body into it over a
// dedicated connection with a running sha256, then checks length and
// digest. The error says why: a body that ended early is Transient, a
// digest that differs from the configured one is Permanent. Discarding
// the volume is the caller's, once this connection is closed.
func (p *Provider) uploadImage(ctx context.Context, host provider.KVMHost, pool libvirt.StoragePool, src imageSource, body io.Reader, size int64) (libvirt.StorageVol, string, error) {
name := src.volumeName()
xmlDoc, err := buildVolumeXML(volumeSpec{Name: name, CapacityBytes: size, Format: src.Format})
if err != nil {
return libvirt.StorageVol{}, "", err
}
fresh, err := p.hosts.dialFresh(ctx, host)
if err != nil {
return libvirt.StorageVol{}, "", fmt.Errorf("dialling host %s for the image upload: %w", host.Name, err)
}
defer func() { _ = fresh.Close() }()
vol, err := fresh.StorageVolCreateXML(ctx, pool, xmlDoc, 0)
if err != nil {
return libvirt.StorageVol{}, "", fmt.Errorf("creating base volume %s: %w", name, err)
}
hash := sha256.New()
counted := &countingReader{r: io.TeeReader(body, hash)}
uploadErr := fresh.StorageVolUpload(ctx, vol, counted, 0, uint64(size), 0)
read, readErr := counted.state()
var digest string
if uploadErr == nil {
// Only then has the upload stopped reading; an abandoned call's
// goroutine may still be writing into the hash.
digest = hex.EncodeToString(hash.Sum(nil))
}
switch {
case readErr != nil:
err = fmt.Errorf("%w: image %s: the response ended after %d of %d bytes: %w", provider.ErrTransient, src.URL, read, size, readErr)
case uploadErr != nil:
err = fmt.Errorf("uploading base volume %s: %w", name, uploadErr)
case read != size:
err = transientf("image %s: short read, %d of %d bytes", src.URL, read, size)
case src.SHA256 != "" && !strings.EqualFold(digest, src.SHA256):
err = permanentf("image %s: sha256 %s does not match the configured %s", src.URL, digest, strings.ToLower(src.SHA256))
}
if err != nil {
return libvirt.StorageVol{}, "", err
}
return vol, digest, nil
}
// countingReader counts what the upload consumed and keeps the first
// non-EOF error the source returned. The adapter can return from an
// abandoned upload while go-libvirt still reads, so the fields are guarded.
type countingReader struct {
r io.Reader
mu sync.Mutex
n int64
err error
}
func (c *countingReader) Read(b []byte) (int, error) {
n, err := c.r.Read(b)
c.mu.Lock()
c.n += int64(n)
if err != nil && err != io.EOF && c.err == nil {
c.err = err
}
c.mu.Unlock()
return n, err
}
func (c *countingReader) state() (int64, error) {
c.mu.Lock()
defer c.mu.Unlock()
return c.n, c.err
}
// discardVolume removes a base volume a failed fetch left behind. It runs
// with ctx's values but not its cancellation: the fetch may have failed
// because ctx ended, and the adapter refuses calls on a finished ctx. The
// budget bounds it on its own. Failure is logged, not returned: the next
// attempt finds the volume without a marker and deletes it then.
func (p *Provider) discardVolume(ctx context.Context, log logr.Logger, be backend, pool libvirt.StoragePool, name string) {
ctx, cancel := context.WithTimeout(context.WithoutCancel(ctx), defaultBudgets.rpc)
defer cancel()
if err := deleteVolumeIfPresent(ctx, be, pool, name); err != nil {
log.V(1).Info("incomplete base volume left behind; the next attempt removes it", "error", err.Error())
}
}
// writeMarker creates the raw marker volume and uploads the digest into it.
// A marker that could not be filled is removed again: an empty marker next
// to a complete base would read as debris and cost a refetch.
func writeMarker(ctx context.Context, be backend, pool libvirt.StoragePool, name, digest string) error {
xmlDoc, err := buildVolumeXML(volumeSpec{Name: name, CapacityBytes: markerCapacityBytes, Format: provider.KVMFormatRaw})
if err != nil {
return err
}
vol, err := be.StorageVolCreateXML(ctx, pool, xmlDoc, 0)
if err != nil {
return fmt.Errorf("creating marker %s: %w", name, err)
}
if err := be.StorageVolUpload(ctx, vol, strings.NewReader(digest), 0, uint64(len(digest)), 0); err != nil {
_ = deleteVolumeIfPresent(ctx, be, pool, name)
return fmt.Errorf("writing marker %s: %w", name, err)
}
return nil
}
// do issues one request bound to ctx. Accept-Encoding is fixed to identity
// because a transparently decompressed body reports no Content-Length. A
// transport error is Transient.
func (p *Provider) do(ctx context.Context, method, rawURL string) (*http.Response, error) {
req, err := http.NewRequestWithContext(ctx, method, rawURL, nil)
if err != nil {
return nil, permanentf("image %s: %v", rawURL, err)
}
req.Header.Set("Accept-Encoding", "identity")
resp, err := p.http.Do(req)
if err != nil {
return nil, fmt.Errorf("%w: fetching image %s: %w", provider.ErrTransient, rawURL, err)
}
return resp, nil
}
// checkResponse classifies the response at the source (spec entry 13) and
// closes its body when it is unusable. 404, 403 and 410, and every other
// 4xx, mean the catalogue entry is wrong; 5xx and 429 are worth a retry. A
// missing Content-Length is Permanent: the volume's capacity must be
// declared before the upload, and a mirror that only chunks is one the
// catalogue must not point at.
func checkResponse(rawURL string, resp *http.Response) error {
var err error
code := resp.StatusCode
switch {
case code >= 200 && code < 300:
case code == http.StatusTooManyRequests || code >= 500:
err = transientf("image %s: HTTP %s", rawURL, resp.Status)
case code >= 400:
err = permanentf("image %s: HTTP %s", rawURL, resp.Status)
default:
err = transientf("image %s: unexpected HTTP %s", rawURL, resp.Status)
}
if err == nil && resp.ContentLength < 0 {
err = permanentf("image %s: response has no Content-Length; the volume capacity must be known before upload", rawURL)
}
if err == nil && resp.ContentLength == 0 {
err = permanentf("image %s: response is empty (Content-Length 0)", rawURL)
}
if err != nil {
_ = resp.Body.Close()
}
return err
}
// urlHost is the only part of the URL a span carries (contract §9).
func urlHost(rawURL string) string {
u, err := url.Parse(rawURL)
if err != nil {
return ""
}
return u.Host
}
// imageSizeIfUncached returns the image's Content-Length when the base is
// not cached on the host, and 0 when the marker is present, for the quota
// precheck of spec entry 13. It never downloads: HEAD first, and a GET whose
// body is closed at once when HEAD answers anything but 2xx, since servers
// refuse HEAD with 405, 403 or 501 alike; the GET's status is what gets
// classified. A marker is taken at face value here; ensureImage decides
// whether it still matches.
func (p *Provider) imageSizeIfUncached(ctx context.Context, be backend, pool libvirt.StoragePool, src imageSource) (int64, error) {
_, err := be.StorageVolLookupByName(ctx, pool, src.markerName())
switch {
case err == nil:
return 0, nil
case !hasCode(err, libvirt.ErrNoStorageVol):
return 0, fmt.Errorf("looking up marker %s: %w", src.markerName(), err)
}
ctx, cancel := context.WithTimeout(ctx, defaultBudgets.rpc)
defer cancel()
resp, err := p.do(ctx, http.MethodHead, src.URL)
if err != nil {
return 0, err
}
_ = resp.Body.Close()
if resp.StatusCode < 200 || resp.StatusCode >= 300 {
if resp, err = p.do(ctx, http.MethodGet, src.URL); err != nil {
return 0, err
}
_ = resp.Body.Close()
}
if err := checkResponse(src.URL, resp); err != nil {
return 0, err
}
return resp.ContentLength, nil
}
// checkPoolSpace is the precheck of spec entry 13: the pool must be running
// and have needed bytes available, or Create returns ErrQuotaExceeded and
// the reconciler waits QuotaRetry instead of failing the volume creation.
func checkPoolSpace(ctx context.Context, be backend, pool libvirt.StoragePool, poolName string, needed uint64) error {
state, _, _, available, err := be.StoragePoolGetInfo(ctx, pool)
if err != nil {
return fmt.Errorf("reading storage pool %s: %w", poolName, err)
}
if s := libvirt.StoragePoolState(state); s != libvirt.StoragePoolRunning {
return permanentf("storage pool %s is not running (state %s)", poolName, s)
}
if available < needed {
return fmt.Errorf("%w: storage pool %s has %d bytes available, %d needed", provider.ErrQuotaExceeded, poolName, available, needed)
}
return nil
}
// neededBytes is what one Create commits to the pool: the image when it is
// not cached yet (0 otherwise), the overlay's capacity and the seed reserve.
func neededBytes(imageBytes int64, diskSizeGB int64) uint64 {
return uint64(max(imageBytes, 0)) + uint64(max(diskSizeGB, 0))<<30 + seedReserveBytes
}
internal/provider/kvm2/image_test.go
+1007
View File
File diff suppressed because it is too large Load Diff
internal/provider/kvm2/kvm2.go
+246
View File
@@ -0,0 +1,246 @@
// Package kvm2 implements the provider contract on libvirt/KVM hypervisors
// reached over SSH: the host's libvirt unix socket is forwarded through the
// SSH connection and driven with go-libvirt's RPC client. The design is in
// docs/kvm2-provider-spec.md.
package kvm2
import (
"context"
"fmt"
"net/http"
"os"
"sync"
"time"
"github.com/go-logr/logr"
logf "sigs.k8s.io/controller-runtime/pkg/log"
"gitea.home.hrajfrisbee.cz/kacerr/egress-proxies-operator/internal/provider"
"gitea.home.hrajfrisbee.cz/kacerr/egress-proxies-operator/internal/tracing"
)
const (
defaultSocket = "/var/run/libvirt/libvirt-sock"
defaultPool = "default"
defaultDiskSizeGB = 10
)
// Provider is one configured kvm2 entry: a set of hosts sharing one SSH key
// and one image and machine-type catalogue.
type Provider struct {
name string
cfg provider.KVMConfig
now func() time.Time
hosts *hostTable
http *http.Client
tracing []tracing.Option
// runningWait caps the wait for a started domain to report running,
// polled every runningPoll (spec entry 18).
runningWait time.Duration
runningPoll time.Duration
// imageLocks serialises image fetches per (host, volume), spec entry 19.
imageLocks *keyedLocks
// nameLocks serialises Create and Delete per instance name, spec entry
// 19, so two Deletes never race on destroy and undefine and a Delete
// cannot interleave with a Create's steps.
nameLocks *keyedLocks
}
// providerOption adjusts what newProvider builds; tests inject an HTTP
// client bound to a test server, a span recorder and a shorter running
// wait.
type providerOption func(*Provider)
func withHTTPClient(c *http.Client) providerOption {
return func(p *Provider) { p.http = c }
}
func withTracing(opts ...tracing.Option) providerOption {
return func(p *Provider) { p.tracing = opts }
}
// withRunningWait shrinks the poll interval and the cap of the wait for a
// domain to run.
func withRunningWait(poll, cap time.Duration) providerOption {
return func(p *Provider) { p.runningPoll, p.runningWait = poll, cap }
}
var _ provider.Provider = (*Provider)(nil)
// New builds a Provider from pc.KVM2. It never dials a host: the credential
// files are only checked for existence, and a missing one is a warning, not
// an error, because the Secret mount is optional and may land later.
func New(ctx context.Context, pc provider.ProviderConfig) (provider.Provider, error) {
if !provider.IsKVMIdentifier(pc.Name) {
return nil, permanentf("provider name %q: kvm2 names its cached base images after the entry, so the name must be letters, digits, dots, dashes or underscores", pc.Name)
}
cfg := provider.KVMConfig{}
if pc.KVM2 != nil {
cfg = *pc.KVM2
}
cfg = withDefaults(cfg)
p := newProvider(pc.Name, cfg, newLibvirtDialer(pc.Name, cfg))
log := p.logger(ctx)
if _, err := os.Stat(p.cfg.SSHKeyFile); err != nil {
log.Info("WARNING: ssh key file is not readable; every call fails until it is", "path", p.cfg.SSHKeyFile, "reason", err.Error())
}
switch {
case p.cfg.InsecureIgnoreHostKey:
log.Info("WARNING: insecureIgnoreHostKey is set; host keys are not verified")
default:
if _, err := os.Stat(p.cfg.KnownHostsFile); err != nil {
log.Info("WARNING: known_hosts file is not readable; every call fails until it is", "path", p.cfg.KnownHostsFile, "reason", err.Error())
}
}
return p, nil
}
// newProvider is the constructor tests use: d hands out the backends, the
// production one being newLibvirtDialer. Without options the image
// downloads use http.DefaultClient (system CA roots, no client timeout: the
// per-request ctx carries the transfer budget).
func newProvider(name string, cfg provider.KVMConfig, d dialer, opts ...providerOption) *Provider {
p := &Provider{
name: name,
cfg: withDefaults(cfg),
now: time.Now,
hosts: newHostTable(d),
http: http.DefaultClient,
runningWait: defaultRunningWait,
runningPoll: defaultRunningPoll,
imageLocks: newKeyedLocks(),
nameLocks: newKeyedLocks(),
}
for _, opt := range opts {
opt(p)
}
if p.http == nil {
p.http = http.DefaultClient
}
return p
}
// withDefaults returns cfg with every optional field filled in, so the rest
// of the package never re-checks for an empty value. The hosts and images
// are copied, not shared with the caller.
func withDefaults(cfg provider.KVMConfig) provider.KVMConfig {
if cfg.DiskSizeGB == 0 {
cfg.DiskSizeGB = defaultDiskSizeGB
}
hosts := make([]provider.KVMHost, len(cfg.Hosts))
for i, h := range cfg.Hosts {
if h.Socket == "" {
h.Socket = defaultSocket
}
if h.Pool == "" {
h.Pool = defaultPool
}
hosts[i] = h
}
cfg.Hosts = hosts
images := make(map[string]provider.KVMImage, len(cfg.Images))
for k, img := range cfg.Images {
if img.Format == "" {
img.Format = provider.KVMFormatQcow2
}
if img.Firmware == "" {
img.Firmware = provider.KVMFirmwareUEFI
}
images[k] = img
}
cfg.Images = images
return cfg
}
func (p *Provider) logger(ctx context.Context) logr.Logger {
return logf.FromContext(ctx).WithName("kvm2").WithValues("provider", p.name)
}
// host resolves a placement.zone or a providerID's host segment to a
// configured host. An empty zone is allowed only when exactly one host is
// configured.
func (p *Provider) host(zone string) (provider.KVMHost, error) {
if zone == "" {
if len(p.cfg.Hosts) == 1 {
return p.cfg.Hosts[0], nil
}
return provider.KVMHost{}, permanentf("placement.zone is required when kvm2.hosts lists more than one host")
}
for _, h := range p.cfg.Hosts {
if h.Name == zone {
return h, nil
}
}
return provider.KVMHost{}, permanentf("host %q is not in kvm2.hosts", zone)
}
// wrapErr is the single exit for errors: it classifies err and wraps it with
// the configured provider name, op and id.
func (p *Provider) wrapErr(op, id string, err error) error {
return provider.Wrap(classify(op, err), op, p.name, id, err)
}
// lockName takes the per-name lock of spec entry 19, or gives up with ctx,
// so a caller whose ctx ends while another caller's Create is downloading
// an image returns promptly.
func (p *Provider) lockName(ctx context.Context, name string) (func(), error) {
release, err := p.nameLocks.acquire(ctx, name)
if err != nil {
return nil, fmt.Errorf("%w: waiting for the lock on %s: %w", provider.ErrTransient, name, err)
}
return release, nil
}
// keyedLocks is a set of one-slot semaphores by key (spec entry 19): a
// semaphore rather than a mutex so a waiter can give up with its ctx. Each
// entry counts the callers holding or waiting for it and is dropped from
// the map at zero, so the map does not grow with every name ever seen while
// two waiters on one key still always share one semaphore.
type keyedLocks struct {
mu sync.Mutex
locks map[string]*keyedLock
}
type keyedLock struct {
sem chan struct{}
refs int
}
func newKeyedLocks() *keyedLocks {
return &keyedLocks{locks: make(map[string]*keyedLock)}
}
// acquire takes the lock for key, or gives up with ctx's error. The
// returned func releases it and must be called exactly once.
func (k *keyedLocks) acquire(ctx context.Context, key string) (func(), error) {
k.mu.Lock()
l, ok := k.locks[key]
if !ok {
l = &keyedLock{sem: make(chan struct{}, 1)}
k.locks[key] = l
}
l.refs++
k.mu.Unlock()
select {
case l.sem <- struct{}{}:
return func() {
<-l.sem
k.unref(key, l)
}, nil
case <-ctx.Done():
k.unref(key, l)
return nil, ctx.Err()
}
}
func (k *keyedLocks) unref(key string, l *keyedLock) {
k.mu.Lock()
defer k.mu.Unlock()
l.refs--
if l.refs == 0 {
delete(k.locks, key)
}
}
internal/provider/kvm2/kvm2_integration_test.go
+590
View File
@@ -0,0 +1,590 @@
//go:build integration
package kvm2
import (
"context"
"crypto/rand"
"encoding/hex"
"errors"
"fmt"
"net"
"os"
"strconv"
"strings"
"sync"
"testing"
"time"
"github.com/digitalocean/go-libvirt"
"github.com/go-logr/logr"
"github.com/go-logr/logr/funcr"
sdktrace "go.opentelemetry.io/otel/sdk/trace"
"go.opentelemetry.io/otel/sdk/trace/tracetest"
"k8s.io/apimachinery/pkg/types"
logf "sigs.k8s.io/controller-runtime/pkg/log"
"gitea.home.hrajfrisbee.cz/kacerr/egress-proxies-operator/internal/provider"
"gitea.home.hrajfrisbee.cz/kacerr/egress-proxies-operator/internal/tracing"
)
// The real-hypervisor test of spec entry 21. It runs only with
// -tags=integration and skips itself unless KVM2_TEST_* names a host; see
// docs/testing.md for the variables and what one run asserts.
const (
integrationProvider = "kvm2-integration"
integrationHost = "kvm2-test"
integrationImage = "test-image"
integrationMachine = "small"
integrationPort = 3128
// integrationCloudInit is the user-data the VM boots with. Nothing
// asserts on the guest running it; it exists so the sentinel check can
// prove that no span attribute carries the cloud-init text.
integrationCloudInit = "#cloud-config\nruncmd:\n - echo kvm2-integration\n"
defaultIntegrationTimeout = 20 * time.Minute
defaultIntegrationLogLvl = 1
getPollInterval = 5 * time.Second
cleanupBudget = 5 * time.Minute
)
type integrationEnv struct {
host, user, sshKey, knownHosts, bridge, imageURL string
imageSHA256, pool, socket string
timeout time.Duration
keep bool
logLevel int
}
// loadIntegrationEnv reads the KVM2_TEST_* variables, skipping the test
// under -short or when a required one is missing.
func loadIntegrationEnv(t *testing.T) integrationEnv {
t.Helper()
if testing.Short() {
t.Skip("real-hypervisor test skipped under -short")
}
env := integrationEnv{timeout: defaultIntegrationTimeout, logLevel: defaultIntegrationLogLvl}
required := []struct {
name string
dst *string
}{
{"KVM2_TEST_HOST", &env.host},
{"KVM2_TEST_USER", &env.user},
{"KVM2_TEST_SSH_KEY", &env.sshKey},
{"KVM2_TEST_BRIDGE", &env.bridge},
{"KVM2_TEST_IMAGE_URL", &env.imageURL},
}
for _, r := range required {
v := os.Getenv(r.name)
if v == "" {
t.Skipf("%s is not set; the kvm2 integration test needs a real hypervisor (see docs/testing.md)", r.name)
}
*r.dst = v
}
env.knownHosts = os.Getenv("KVM2_TEST_KNOWN_HOSTS")
env.imageSHA256 = os.Getenv("KVM2_TEST_IMAGE_SHA256")
env.pool = os.Getenv("KVM2_TEST_POOL")
env.socket = os.Getenv("KVM2_TEST_SOCKET")
env.keep = os.Getenv("KVM2_TEST_KEEP") != ""
if v := os.Getenv("KVM2_TEST_TIMEOUT"); v != "" {
d, err := time.ParseDuration(v)
if err != nil {
t.Fatalf("KVM2_TEST_TIMEOUT=%q: %v", v, err)
}
env.timeout = d
}
if v := os.Getenv("KVM2_TEST_LOG_LEVEL"); v != "" {
n, err := strconv.Atoi(v)
if err != nil || n < 0 {
t.Fatalf("KVM2_TEST_LOG_LEVEL=%q: want a non-negative integer", v)
}
env.logLevel = n
}
return env
}
// providerConfig is what the operator would load for this host: one host,
// one image, one machine type. Host keys are verified only when
// KVM2_TEST_KNOWN_HOSTS is set.
func (e integrationEnv) providerConfig() provider.ProviderConfig {
return provider.ProviderConfig{
Name: integrationProvider,
Type: "kvm2",
KVM2: &provider.KVMConfig{
SSHKeyFile: e.sshKey,
KnownHostsFile: e.knownHosts,
InsecureIgnoreHostKey: e.knownHosts == "",
Images: map[string]provider.KVMImage{
integrationImage: {URL: e.imageURL, SHA256: e.imageSHA256},
},
MachineTypes: map[string]provider.KVMMachineType{
integrationMachine: {VCPUs: 1, MemoryMiB: 1024},
},
Hosts: []provider.KVMHost{{
Name: integrationHost,
Address: e.host,
User: e.user,
Socket: e.socket,
Pool: e.pool,
Bridge: e.bridge,
}},
},
}
}
// integrationLog routes the provider's logr output into t.Log through funcr
// at the chosen verbosity, keeps every line for assertions, and strips the
// key/value pairs that are constant for the run (stated once up front) so
// each line stays short. Lines arriving after the test ended are dropped:
// t.Log panics then.
type integrationLog struct {
t *testing.T
mu sync.Mutex
drop []string
lines []string
done bool
}
func newIntegrationLog(t *testing.T, level int, drop ...string) (*integrationLog, logr.Logger) {
t.Helper()
sink := &integrationLog{t: t, drop: drop}
t.Cleanup(func() {
sink.mu.Lock()
defer sink.mu.Unlock()
sink.done = true
})
log := funcr.New(func(prefix, args string) { sink.write(prefix, args) }, funcr.Options{Verbosity: level})
return sink, log
}
// dropValue adds a rendered pair (`"key"="value"`) to strip from later lines.
func (l *integrationLog) dropValue(key, value string) {
l.mu.Lock()
defer l.mu.Unlock()
l.drop = append(l.drop, fmt.Sprintf("%q=%q", key, value))
}
func (l *integrationLog) write(prefix, args string) {
l.mu.Lock()
defer l.mu.Unlock()
l.lines = append(l.lines, args)
if l.done {
return
}
for _, d := range l.drop {
args = strings.ReplaceAll(args, " "+d, "")
}
l.t.Logf("%s %s %s", time.Now().Format("15:04:05.000"), prefix, args)
}
func (l *integrationLog) matching(substr string) []string {
l.mu.Lock()
defer l.mu.Unlock()
var out []string
for _, line := range l.lines {
if strings.Contains(line, substr) {
out = append(out, line)
}
}
return out
}
// newIntegrationProvider builds the provider the way New does, plus the span
// recorder's tracer on both places the package starts spans: the Provider
// (kvm2.download) and the dialer (kvm2.dial and, through observe, every
// kvm2.<Method>). newLibvirtDialer has no option for it, so the field is set
// directly; that is what makes the kvm2.* spans land in the same recorder
// as provider.WithTracing's provider.* spans.
func newIntegrationProvider(pc provider.ProviderConfig, tp *sdktrace.TracerProvider) *Provider {
cfg := withDefaults(*pc.KVM2)
opt := tracing.WithTracerProvider(tp)
d := newLibvirtDialer(pc.Name, cfg)
d.tracing = []tracing.Option{opt}
return newProvider(pc.Name, cfg, d, withTracing(opt))
}
// randomUID is a UUID-shaped string, so the run's domain never collides with
// a previous run's leftovers or with a real Proxy's.
func randomUID(t *testing.T) string {
t.Helper()
var b [16]byte
if _, err := rand.Read(b[:]); err != nil {
t.Fatalf("reading random bytes: %v", err)
}
h := hex.EncodeToString(b[:])
return h[0:8] + "-" + h[8:12] + "-" + h[12:16] + "-" + h[16:20] + "-" + h[20:32]
}
// stepDone logs one lifecycle step with its duration; the log is the record
// of how long a cold image download or a boot took on this host.
func stepDone(t *testing.T, step string, start time.Time) {
t.Helper()
t.Logf("step %s: %s", step, time.Since(start).Round(time.Millisecond))
}
// waitUntil polls cond every interval until it is true or ctx ends.
func waitUntil(ctx context.Context, t *testing.T, interval time.Duration, what string, cond func() bool) {
t.Helper()
ticker := time.NewTicker(interval)
defer ticker.Stop()
for !cond() {
select {
case <-ticker.C:
case <-ctx.Done():
t.Fatalf("gave up waiting for %s: %v", what, ctx.Err())
}
}
}
// domainCount lists every domain on the host over a dedicated connection.
func domainCount(ctx context.Context, t *testing.T, p *Provider, host provider.KVMHost) int {
t.Helper()
be, err := p.hosts.dialFresh(ctx, host)
if err != nil {
t.Fatalf("dialling %s to count domains: %v", host.Name, err)
}
defer func() { _ = be.Close() }()
doms, _, err := be.ConnectListAllDomains(ctx, 1, libvirt.ConnectListDomainsActive|libvirt.ConnectListDomainsInactive)
if err != nil {
t.Fatalf("listing domains on %s: %v", host.Name, err)
}
return len(doms)
}
// poolVolumes reports, for each name, whether the host's pool holds a volume
// of that name, over a dedicated connection. Any error other than
// ErrNoStorageVol fails the test.
func poolVolumes(ctx context.Context, t *testing.T, p *Provider, host provider.KVMHost, names ...string) map[string]bool {
t.Helper()
be, err := p.hosts.dialFresh(ctx, host)
if err != nil {
t.Fatalf("dialling %s to inspect the pool: %v", host.Name, err)
}
defer func() { _ = be.Close() }()
pool, err := be.StoragePoolLookupByName(ctx, host.Pool)
if err != nil {
t.Fatalf("looking up pool %s: %v", host.Pool, err)
}
out := make(map[string]bool, len(names))
for _, name := range names {
_, err := be.StorageVolLookupByName(ctx, pool, name)
switch {
case err == nil:
out[name] = true
case hasCode(err, libvirt.ErrNoStorageVol):
out[name] = false
default:
t.Fatalf("looking up volume %s in pool %s: %v", name, host.Pool, err)
}
}
return out
}
func TestKVM2_lifecycle(t *testing.T) {
env := loadIntegrationEnv(t)
pc := env.providerConfig()
sink, log := newIntegrationLog(t, env.logLevel,
fmt.Sprintf("%q=%q", "provider", integrationProvider),
fmt.Sprintf("%q=%q", "host", integrationHost))
ctx, cancel := context.WithTimeout(logf.IntoContext(context.Background(), log), env.timeout)
defer cancel()
t.Logf("provider=%s host=%s address=%s user=%s bridge=%s image=%s budget=%s log level %d; provider, host and name are left off every provider log line",
integrationProvider, integrationHost, env.host, env.user, env.bridge, env.imageURL, env.timeout, env.logLevel)
if env.knownHosts == "" {
t.Log("KVM2_TEST_KNOWN_HOSTS is not set: host keys are NOT verified for this run")
}
// 1. New returns without dialling.
start := time.Now()
built, err := New(ctx, pc)
elapsed := time.Since(start)
if err != nil {
t.Fatalf("New() error = %v", err)
}
if elapsed > 100*time.Millisecond {
t.Errorf("New() took %s, want under 100ms: it must not dial", elapsed)
}
if lines := sink.matching("libvirt host connected"); len(lines) > 0 {
t.Errorf("New() dialled the host before any call: %q", lines)
}
stepDone(t, "1 New", start)
// 2. Check reports the host Ready with details.
checker, ok := built.(provider.Checker)
if !ok {
t.Fatalf("New() returned %T, which is not a provider.Checker", built)
}
start = time.Now()
targets := checker.Check(ctx)
if len(targets) != 1 {
t.Fatalf("Check() returned %d targets, want 1: %+v", len(targets), targets)
}
ts := targets[0]
if !ts.Ready || ts.Reason != provider.ReasonReachable {
t.Fatalf("Check() = ready %v reason %q message %q, want Ready/%s (details %v)", ts.Ready, ts.Reason, ts.Message, provider.ReasonReachable, ts.Details)
}
for _, key := range []string{"hostname", "libvirtVersion", "poolState", "poolAvailableGiB"} {
if ts.Details[key] == "" {
t.Errorf("Check() details lack %q: %v", key, ts.Details)
}
}
if ts.Details["poolState"] != "running" {
t.Errorf("Check() poolState = %q, want running", ts.Details["poolState"])
}
t.Logf("host %s: hostname=%s libvirt=%s pool=%s (%s) available=%s GiB capacity=%s GiB latency=%s",
ts.Name, ts.Details["hostname"], ts.Details["libvirtVersion"], ts.Details["pool"], ts.Details["poolState"],
ts.Details["poolAvailableGiB"], ts.Details["poolCapacityGiB"], ts.Latency.Round(time.Millisecond))
stepDone(t, "2 Check", start)
// 3. Every later call goes through the tracing decorator, with the
// package's own spans recorded alongside.
sr := tracetest.NewSpanRecorder()
tp := sdktrace.NewTracerProvider(sdktrace.WithSpanProcessor(sr))
t.Cleanup(func() { _ = tp.Shutdown(context.Background()) })
p := newIntegrationProvider(pc, tp)
traced := provider.WithTracing(pc.Name, p, tracing.WithTracerProvider(tp))
host := p.cfg.Hosts[0]
// 4. Create.
uid := randomUID(t)
name := provider.NameFromUID(types.UID(uid))
sink.dropValue("name", name)
id := formatProviderID(integrationHost, name)
req := provider.CreateRequest{
Name: name,
UID: uid,
Namespace: "integration",
ProxyName: "kvm2-lifecycle",
Placement: provider.Placement{Zone: integrationHost, MachineType: integrationMachine, Image: integrationImage},
CloudInit: integrationCloudInit,
Port: integrationPort,
}
t.Logf("domain name %s, uid %s, id %s", name, uid, id)
deleted := false
if env.keep {
t.Logf("KVM2_TEST_KEEP is set: the VM stays behind. Clean up on the host with:\n"+
" virsh undefine --nvram %s\n virsh vol-delete --pool %s %s\n virsh vol-delete --pool %s %s",
name, host.Pool, overlayVolumeName(name), host.Pool, seedVolumeName(name))
} else {
t.Cleanup(func() {
if deleted {
return
}
cctx, ccancel := context.WithTimeout(logf.IntoContext(context.Background(), log), cleanupBudget)
defer ccancel()
if err := p.Delete(cctx, id); err != nil {
t.Errorf("cleanup Delete(%s) error = %v; remove the domain and its volumes by hand", id, err)
}
})
}
start = time.Now()
gotID, err := traced.Create(ctx, req)
if err != nil {
t.Fatalf("Create() error = %v", err)
}
if gotID != id {
t.Fatalf("Create() = %q, want %q", gotID, id)
}
createReturned := time.Now()
stepDone(t, "4 Create (includes the image download on a cold pool)", start)
// 5. Create again: same ID, nothing new on the host.
start = time.Now()
before := domainCount(ctx, t, p, host)
again, err := traced.Create(ctx, req)
if err != nil {
t.Fatalf("second Create() error = %v", err)
}
if again != id {
t.Errorf("second Create() = %q, want %q", again, id)
}
if after := domainCount(ctx, t, p, host); after != before {
t.Errorf("second Create() changed the host's domain count from %d to %d", before, after)
}
stepDone(t, "5 Create again", start)
// 6. Get reaches Running with an IPv4.
start = time.Now()
var inst *provider.Instance
waitUntil(ctx, t, getPollInterval, "the domain to be Running with an IPv4", func() bool {
var err error
inst, err = traced.Get(ctx, id)
if err != nil {
t.Fatalf("Get(%s) error = %v", id, err)
}
return inst.State == provider.StateRunning && inst.IP != ""
})
if inst.UID != uid {
t.Errorf("Get() UID = %q, want %q", inst.UID, uid)
}
if inst.CreatedAt.IsZero() {
t.Error("Get() CreatedAt is zero; the metadata mark did not round-trip")
} else if skew := time.Since(inst.CreatedAt).Abs(); skew > 5*time.Minute {
t.Errorf("Get() CreatedAt = %s, %s from now; want within a few minutes", inst.CreatedAt, skew)
}
t.Logf("domain running at %s, %s after Create returned (createdAt %s)", inst.IP, time.Since(createReturned).Round(time.Millisecond), inst.CreatedAt.Format(time.RFC3339))
stepDone(t, "6 Get until Running", start)
// 7. ListByTag contains it.
start = time.Now()
list, err := traced.ListByTag(ctx)
if err != nil {
t.Fatalf("ListByTag() error = %v", err)
}
found := false
for _, li := range list {
if li.ID == id {
found = true
if li.UID != uid {
t.Errorf("ListByTag() entry %s has UID %q, want %q", id, li.UID, uid)
}
}
}
if !found {
t.Errorf("ListByTag() = %d instances, none with ID %s", len(list), id)
}
stepDone(t, "7 ListByTag", start)
// 8. Reachability, logged only: this machine may not route to the bridge.
start = time.Now()
addr := net.JoinHostPort(inst.IP, "22")
if conn, err := net.DialTimeout("tcp", addr, 3*time.Second); err != nil {
t.Logf("TCP connect to %s failed (informational; the test host may not route to the bridge): %v", addr, err)
} else {
_ = conn.Close()
t.Logf("TCP connect to %s succeeded", addr)
}
stepDone(t, "8 TCP connect", start)
if env.keep {
t.Log("KVM2_TEST_KEEP is set: skipping Delete and the pool inspection")
} else {
// 9. Delete, Get is NotFound, Delete again.
start = time.Now()
if err := traced.Delete(ctx, id); err != nil {
t.Fatalf("Delete() error = %v", err)
}
deleted = true
if _, err := traced.Get(ctx, id); !errors.Is(err, provider.ErrNotFound) {
t.Errorf("Get() after Delete error = %v, want ErrNotFound", err)
}
if err := traced.Delete(ctx, id); err != nil {
t.Errorf("second Delete() error = %v, want nil", err)
}
stepDone(t, "9 Delete, Get, Delete", start)
// 10. The pool keeps the shared base image and its marker, and holds
// no per-VM volume.
start = time.Now()
base := baseVolumeName(integrationProvider, integrationImage, provider.KVMFormatQcow2)
marker := markerVolumeName(base)
overlay, seed := overlayVolumeName(name), seedVolumeName(name)
present := poolVolumes(ctx, t, p, host, base, marker, overlay, seed)
for _, want := range []string{base, marker} {
if !present[want] {
t.Errorf("pool %s lacks %s after Delete; the base image cache must stay", host.Pool, want)
}
}
for _, gone := range []string{overlay, seed} {
if present[gone] {
t.Errorf("pool %s still holds %s after Delete", host.Pool, gone)
}
}
stepDone(t, "10 pool inspection", start)
}
// 11. Spans: the package's spans are children of the decorator's.
assertLifecycleSpans(t, sr.Ended())
}
// assertLifecycleSpans checks that at least one provider.create span has a
// kvm2.DomainDefineXML and a kvm2.dial child (the first Create dials the
// pooled connection and defines the domain), and that no attribute of any
// span carries the cloud-init text.
func assertLifecycleSpans(t *testing.T, spans []sdktrace.ReadOnlySpan) {
t.Helper()
var creates []sdktrace.ReadOnlySpan
for _, s := range spans {
if s.Name() == "provider.create" {
creates = append(creates, s)
}
}
if len(creates) == 0 {
t.Fatalf("no provider.create span among %d spans: %v", len(spans), spanNames(spans))
}
satisfied := false
for _, create := range creates {
parent := create.SpanContext().SpanID()
var children, define, dial int
for _, s := range spans {
if s.Parent().SpanID() != parent || !strings.HasPrefix(s.Name(), "kvm2.") {
continue
}
children++
switch s.Name() {
case "kvm2.DomainDefineXML":
define++
case "kvm2.dial":
dial++
}
}
t.Logf("provider.create span %s: %d kvm2.* children (%d DomainDefineXML, %d dial)", parent, children, define, dial)
if define > 0 && dial > 0 {
satisfied = true
}
}
if !satisfied {
t.Errorf("no provider.create span has both a kvm2.DomainDefineXML and a kvm2.dial child; spans: %v", spanNames(spans))
}
for _, s := range spans {
for _, a := range s.Attributes() {
v := a.Value.Emit()
if strings.Contains(v, integrationCloudInit) || strings.Contains(v, "#cloud-config") {
t.Errorf("span %s attribute %s carries the cloud-init text", s.Name(), a.Key)
}
}
}
}
// TestKVM2Check_realHost runs the health check against the host and creates
// nothing.
func TestKVM2Check_realHost(t *testing.T) {
env := loadIntegrationEnv(t)
pc := env.providerConfig()
_, log := newIntegrationLog(t, env.logLevel,
fmt.Sprintf("%q=%q", "provider", integrationProvider),
fmt.Sprintf("%q=%q", "host", integrationHost))
ctx, cancel := context.WithTimeout(logf.IntoContext(context.Background(), log), env.timeout)
defer cancel()
if env.knownHosts == "" {
t.Log("KVM2_TEST_KNOWN_HOSTS is not set: host keys are NOT verified for this run")
}
built, err := New(ctx, pc)
if err != nil {
t.Fatalf("New() error = %v", err)
}
checker, ok := built.(provider.Checker)
if !ok {
t.Fatalf("New() returned %T, which is not a provider.Checker", built)
}
targets := checker.Check(ctx)
if len(targets) != 1 {
t.Fatalf("Check() returned %d targets, want 1: %+v", len(targets), targets)
}
ts := targets[0]
if !ts.Ready || ts.Reason != provider.ReasonReachable {
t.Fatalf("Check() = ready %v reason %q message %q, want Ready/%s (details %v)", ts.Ready, ts.Reason, ts.Message, provider.ReasonReachable, ts.Details)
}
if ts.Details["poolState"] != "running" {
t.Errorf("Check() poolState = %q, want running", ts.Details["poolState"])
}
t.Logf("host %s (%s): hostname=%s libvirt=%s pool=%s (%s) available=%s GiB latency=%s",
ts.Name, ts.Address, ts.Details["hostname"], ts.Details["libvirtVersion"], ts.Details["pool"], ts.Details["poolState"],
ts.Details["poolAvailableGiB"], ts.Latency.Round(time.Millisecond))
}
internal/provider/kvm2/kvm2_test.go
+361
View File
@@ -0,0 +1,361 @@
package kvm2
import (
"context"
"errors"
"strings"
"sync"
"sync/atomic"
"testing"
"time"
"github.com/go-logr/logr/funcr"
logf "sigs.k8s.io/controller-runtime/pkg/log"
"gitea.home.hrajfrisbee.cz/kacerr/egress-proxies-operator/internal/provider"
)
// captureContext collects funcr output at V(2). The sink is locked because
// Check logs from one goroutine per host.
func captureContext() (context.Context, *[]string) {
lines := &[]string{}
var mu sync.Mutex
log := funcr.New(func(prefix, args string) {
mu.Lock()
defer mu.Unlock()
*lines = append(*lines, prefix+" "+args)
}, funcr.Options{Verbosity: 2})
return logf.IntoContext(context.Background(), log), lines
}
func twoHosts() provider.KVMConfig {
return provider.KVMConfig{
SSHKeyFile: "/nonexistent/kvm2/id_ed25519",
KnownHostsFile: "/nonexistent/kvm2/known_hosts",
MachineTypes: map[string]provider.KVMMachineType{"small": {VCPUs: 1, MemoryMiB: 1024}},
Hosts: []provider.KVMHost{
{Name: "beelink", Address: "beelink:22", User: "virt", Bridge: "br0"},
{Name: "lab2", Address: "lab2:22", User: "virt", Bridge: "br1", Socket: "/run/s", Pool: "fast"},
},
}
}
func TestNew_doesNotDialAndWarnsOnMissingFiles(t *testing.T) {
t.Parallel()
ctx, lines := captureContext()
cfg := twoHosts()
cfg.Hosts[0].Address = "192.0.2.1:22"
start := time.Now()
p, err := New(ctx, provider.ProviderConfig{Name: "kvm2-home", Type: "kvm2", KVM2: &cfg})
elapsed := time.Since(start)
if err != nil {
t.Fatalf("New() error = %v, want nil", err)
}
if p == nil {
t.Fatal("New() = nil provider")
}
if elapsed > 100*time.Millisecond {
t.Errorf("New() took %v, want under 100ms (it must not dial)", elapsed)
}
var warnings int
for _, l := range *lines {
if strings.Contains(l, "WARNING:") {
warnings++
}
if !strings.Contains(l, `"provider"="kvm2-home"`) {
t.Errorf("log line lacks the provider value: %s", l)
}
}
if warnings != 2 {
t.Errorf("got %d WARNING: lines, want 2 (key and known_hosts):\n%s", warnings, strings.Join(*lines, "\n"))
}
}
func TestNew_insecureIgnoreHostKeyWarns(t *testing.T) {
t.Parallel()
ctx, lines := captureContext()
cfg := twoHosts()
cfg.KnownHostsFile = ""
cfg.InsecureIgnoreHostKey = true
if _, err := New(ctx, provider.ProviderConfig{Name: "kvm2-home", Type: "kvm2", KVM2: &cfg}); err != nil {
t.Fatalf("New() error = %v, want nil", err)
}
found := false
for _, l := range *lines {
if strings.Contains(l, "WARNING:") && strings.Contains(l, "host keys are not verified") {
found = true
}
if strings.Contains(l, "known_hosts") {
t.Errorf("known_hosts warning emitted although host keys are ignored: %s", l)
}
}
if !found {
t.Errorf("no insecureIgnoreHostKey warning in:\n%s", strings.Join(*lines, "\n"))
}
}
func TestNew_nilBlockDoesNotPanic(t *testing.T) {
t.Parallel()
ctx, _ := captureContext()
p, err := New(ctx, provider.ProviderConfig{Name: "kvm2-empty", Type: "kvm2"})
if err != nil {
t.Fatalf("New() error = %v, want nil", err)
}
kp, ok := p.(*Provider)
if !ok {
t.Fatalf("New() = %T, want *Provider", p)
}
if kp.name != "kvm2-empty" || kp.cfg.DiskSizeGB != defaultDiskSizeGB || kp.now == nil {
t.Errorf("provider = %+v, want name kvm2-empty, default disk size and a clock", kp)
}
}
// TestNew_rejectsUnsafeEntryName guards the one config value that reaches a
// volume name without passing the kvm validator: the provider entry's own
// name, which is checked only for being non-empty and unique.
func TestNew_rejectsUnsafeEntryName(t *testing.T) {
t.Parallel()
ctx, _ := captureContext()
_, err := New(ctx, provider.ProviderConfig{Name: "kvm2/home", Type: "kvm2"})
if !errors.Is(err, provider.ErrPermanent) || !strings.Contains(err.Error(), "kvm2/home") {
t.Fatalf("New() error = %v, want ErrPermanent naming the entry", err)
}
}
func TestHost_table(t *testing.T) {
t.Parallel()
tests := []struct {
name string
hosts []provider.KVMHost
zone string
wantName string
wantErr string
}{
{
name: "one host and empty zone selects it",
hosts: twoHosts().Hosts[:1],
zone: "",
wantName: "beelink",
},
{
name: "several hosts and empty zone is permanent",
hosts: twoHosts().Hosts,
zone: "",
wantErr: "placement.zone is required when kvm2.hosts lists more than one host",
},
{
name: "known name",
hosts: twoHosts().Hosts,
zone: "lab2",
wantName: "lab2",
},
{
name: "unknown name is permanent",
hosts: twoHosts().Hosts,
zone: "nas9",
wantErr: `host "nas9" is not in kvm2.hosts`,
},
{
name: "no hosts at all is permanent",
hosts: nil,
zone: "",
wantErr: "placement.zone is required",
},
}
for _, tc := range tests {
t.Run(tc.name, func(t *testing.T) {
t.Parallel()
p := newProvider("kvm2-home", provider.KVMConfig{Hosts: tc.hosts}, &stubDialer{})
h, err := p.host(tc.zone)
if tc.wantErr != "" {
if err == nil {
t.Fatalf("host(%q) = %+v, want error containing %q", tc.zone, h, tc.wantErr)
}
if !errors.Is(err, provider.ErrPermanent) {
t.Errorf("host(%q) error = %v, want ErrPermanent", tc.zone, err)
}
if !strings.Contains(err.Error(), tc.wantErr) {
t.Errorf("host(%q) error = %q, want substring %q", tc.zone, err, tc.wantErr)
}
return
}
if err != nil {
t.Fatalf("host(%q) error = %v, want nil", tc.zone, err)
}
if h.Name != tc.wantName {
t.Errorf("host(%q).Name = %q, want %q", tc.zone, h.Name, tc.wantName)
}
})
}
}
func TestWithDefaults_table(t *testing.T) {
t.Parallel()
tests := []struct {
name string
in provider.KVMConfig
check func(t *testing.T, got provider.KVMConfig)
}{
{
name: "empty config gets disk size and empty maps",
in: provider.KVMConfig{},
check: func(t *testing.T, got provider.KVMConfig) {
if got.DiskSizeGB != 10 {
t.Errorf("DiskSizeGB = %d, want 10", got.DiskSizeGB)
}
if got.Hosts == nil || got.Images == nil {
t.Errorf("Hosts = %v, Images = %v, want non-nil", got.Hosts, got.Images)
}
},
},
{
name: "explicit disk size kept",
in: provider.KVMConfig{DiskSizeGB: 40},
check: func(t *testing.T, got provider.KVMConfig) {
if got.DiskSizeGB != 40 {
t.Errorf("DiskSizeGB = %d, want 40", got.DiskSizeGB)
}
},
},
{
name: "host socket and pool default",
in: provider.KVMConfig{Hosts: []provider.KVMHost{{Name: "a"}, {Name: "b", Socket: "/run/s", Pool: "fast"}}},
check: func(t *testing.T, got provider.KVMConfig) {
if got.Hosts[0].Socket != "/var/run/libvirt/libvirt-sock" || got.Hosts[0].Pool != "default" {
t.Errorf("hosts[0] = %+v, want default socket and pool", got.Hosts[0])
}
if got.Hosts[1].Socket != "/run/s" || got.Hosts[1].Pool != "fast" {
t.Errorf("hosts[1] = %+v, want explicit socket and pool kept", got.Hosts[1])
}
},
},
{
name: "image format and firmware default",
in: provider.KVMConfig{Images: map[string]provider.KVMImage{
"d": {URL: "https://x/d.qcow2"},
"u": {URL: "https://x/u.img", Format: provider.KVMFormatRaw, Firmware: provider.KVMFirmwareBIOS},
}},
check: func(t *testing.T, got provider.KVMConfig) {
if got.Images["d"].Format != provider.KVMFormatQcow2 || got.Images["d"].Firmware != provider.KVMFirmwareUEFI {
t.Errorf("images[d] = %+v, want qcow2 and uefi", got.Images["d"])
}
if got.Images["u"].Format != provider.KVMFormatRaw || got.Images["u"].Firmware != provider.KVMFirmwareBIOS {
t.Errorf("images[u] = %+v, want raw and bios kept", got.Images["u"])
}
},
},
}
for _, tc := range tests {
t.Run(tc.name, func(t *testing.T) {
t.Parallel()
tc.check(t, withDefaults(tc.in))
})
}
}
func TestWithDefaults_doesNotAliasInput(t *testing.T) {
t.Parallel()
in := provider.KVMConfig{
Hosts: []provider.KVMHost{{Name: "a"}},
Images: map[string]provider.KVMImage{"d": {URL: "https://x/d.qcow2"}},
}
got := withDefaults(in)
got.Hosts[0].Name = "changed"
got.Images["d"] = provider.KVMImage{URL: "changed"}
if in.Hosts[0].Name != "a" || in.Images["d"].URL != "https://x/d.qcow2" {
t.Errorf("input mutated through the defaulted copy: %+v", in)
}
}
func lockCount(k *keyedLocks) int {
k.mu.Lock()
defer k.mu.Unlock()
return len(k.locks)
}
func TestKeyedLocks_entryRemovedWhenUncontended(t *testing.T) {
t.Parallel()
k := newKeyedLocks()
release, err := k.acquire(context.Background(), "a")
if err != nil {
t.Fatalf("acquire error = %v", err)
}
if lockCount(k) != 1 {
t.Errorf("entries while held = %d, want 1", lockCount(k))
}
release()
if lockCount(k) != 0 {
t.Errorf("entries after release = %d, want 0", lockCount(k))
}
if _, err := k.acquire(context.Background(), "a"); err != nil {
t.Fatalf("acquire after release error = %v", err)
}
}
func TestKeyedLocks_waitersShareOneLock(t *testing.T) {
t.Parallel()
k := newKeyedLocks()
const n = 20
var inside, overlaps atomic.Int32
var wg sync.WaitGroup
for range n {
wg.Go(func() {
release, err := k.acquire(context.Background(), "shared")
if err != nil {
t.Errorf("acquire error = %v", err)
return
}
if inside.Add(1) > 1 {
overlaps.Add(1)
}
time.Sleep(time.Millisecond)
inside.Add(-1)
release()
})
}
wg.Wait()
if overlaps.Load() != 0 {
t.Errorf("%d holders overlapped on one key", overlaps.Load())
}
if lockCount(k) != 0 {
t.Errorf("entries after every holder released = %d, want 0", lockCount(k))
}
}
func TestKeyedLocks_waiterGivesUpWithContext(t *testing.T) {
t.Parallel()
k := newKeyedLocks()
release, err := k.acquire(context.Background(), "a")
if err != nil {
t.Fatalf("acquire error = %v", err)
}
ctx, cancel := context.WithTimeout(context.Background(), 20*time.Millisecond)
defer cancel()
start := time.Now()
if _, err := k.acquire(ctx, "a"); !errors.Is(err, context.DeadlineExceeded) {
t.Fatalf("waiter error = %v, want DeadlineExceeded", err)
}
if time.Since(start) > time.Second {
t.Errorf("waiter took %v, want to give up with its ctx", time.Since(start))
}
if lockCount(k) != 1 {
t.Errorf("entries with the holder still in = %d, want 1: the waiter's exit must not drop the holder's entry", lockCount(k))
}
release()
if lockCount(k) != 0 {
t.Errorf("entries after the holder released = %d, want 0", lockCount(k))
}
}
func TestWrapErr_carriesProviderNameOpAndID(t *testing.T) {
t.Parallel()
p := newProvider("kvm2-home", twoHosts(), &stubDialer{})
err := p.wrapErr("get", "hosts/beelink/domains/proxy-x", permanentf("boom"))
var pe *provider.Error
if !errors.As(err, &pe) {
t.Fatalf("wrapErr() = %T, want *provider.Error", err)
}
if pe.Provider != "kvm2-home" || pe.Op != "get" || pe.ID != "hosts/beelink/domains/proxy-x" || pe.Class != provider.ErrPermanent {
t.Errorf("wrapErr() = %+v, want provider kvm2-home, op get, the id and ErrPermanent", pe)
}
}
internal/provider/kvm2/list.go
+56
View File
@@ -0,0 +1,56 @@
package kvm2
import (
"context"
"fmt"
"github.com/digitalocean/go-libvirt"
"gitea.home.hrajfrisbee.cz/kacerr/egress-proxies-operator/internal/provider"
)
// ListByTag is spec entry 12: every domain on every configured host, in
// config order, filtered client-side on the kvm2 mark. GC uses only ID, UID
// and CreatedAt (contract §2.4), all of which the name and the metadata
// element give, so no state or address RPC is made and State and IP stay
// empty. A host that cannot be reached or listed fails the whole call
// (fail-whole); a single domain whose metadata cannot be read is skipped,
// unless ctx ended, which fails the call rather than pass off an empty view
// as the truth.
func (p *Provider) ListByTag(ctx context.Context) ([]provider.Instance, error) {
log := p.logger(ctx)
var out []provider.Instance
for _, host := range p.cfg.Hosts {
hlog := log.WithValues("host", host.Name)
be, err := p.hosts.get(ctx, host)
if err != nil {
logAPIError(hlog, "list", err)
return nil, p.wrapErr("list", "", err)
}
doms, _, err := be.ConnectListAllDomains(ctx, 1, libvirt.ConnectListDomainsActive|libvirt.ConnectListDomainsInactive)
if err != nil {
err = fmt.Errorf("listing domains on host %s: %w", host.Name, err)
logAPIError(hlog, "list", err)
return nil, p.wrapErr("list", "", err)
}
for _, dom := range doms {
dlog := hlog.WithValues("name", dom.Name)
meta, err := readMetadata(ctx, be, dom)
if err != nil {
if ctx.Err() != nil {
return nil, p.wrapErr("list", "", err)
}
dlog.V(1).Info("skipping domain with unreadable metadata", "error", err.Error())
continue
}
if !meta.managed() {
continue
}
inst := provider.Instance{ID: formatProviderID(host.Name, dom.Name), UID: meta.UID, CreatedAt: meta.createdAt()}
dlog.V(2).Info("kvm2 listed instance", "id", inst.ID, "uid", inst.UID, "createdAt", inst.CreatedAt)
out = append(out, inst)
}
}
log.V(1).Info("kvm2 domains listed", "hosts", len(p.cfg.Hosts), "managed", len(out))
return out, nil
}
internal/provider/kvm2/list_test.go
+213
View File
@@ -0,0 +1,213 @@
package kvm2
import (
"errors"
"strings"
"testing"
"time"
"github.com/digitalocean/go-libvirt"
"gitea.home.hrajfrisbee.cz/kacerr/egress-proxies-operator/internal/provider"
)
func listIDs(list []provider.Instance) []string {
ids := make([]string, len(list))
for i, inst := range list {
ids[i] = inst.ID
}
return ids
}
func TestListByTag_excludesUnmarkedDomains(t *testing.T) {
t.Parallel()
fx := newFixture(t, 1)
id := fx.create(testName)
fx.fake.addDomain("legacy-vm", libvirt.DomainRunning, "")
fx.fake.addDomain("unmarked", libvirt.DomainRunning, `<instance><managed>false</managed><uid>x</uid></instance>`)
fx.fake.addDomain("unrelated", libvirt.DomainShutoff, `<instance><note>hand made</note></instance>`)
list, err := fx.p.ListByTag(fx.ctx)
if err != nil {
t.Fatalf("ListByTag() error = %v", err)
}
if got := listIDs(list); len(got) != 1 || got[0] != id {
t.Errorf("ListByTag = %v, want only %s", got, id)
}
if list[0].UID != testUID || list[0].CreatedAt.IsZero() {
t.Errorf("listed = %+v, want uid and createdAt", list[0])
}
if got := fx.logs.matching("kvm2 domains listed"); len(got) != 1 || !strings.Contains(got[0], `"hosts"=1`) || !strings.Contains(got[0], `"managed"=1`) {
t.Errorf("listed line = %v", got)
}
}
func TestListByTag_emptyUIDStillListed(t *testing.T) {
t.Parallel()
fx := newFixture(t, 1)
fx.fake.addDomain("proxy-nouid", libvirt.DomainRunning, managedMetadata("", time.Now()))
list, err := fx.p.ListByTag(fx.ctx)
if err != nil {
t.Fatalf("ListByTag() error = %v", err)
}
if len(list) != 1 || list[0].UID != "" || list[0].ID != formatProviderID(testHost, "proxy-nouid") {
t.Errorf("ListByTag = %+v, want the domain listed with an empty UID", list)
}
}
func TestListByTag_acrossHostsInConfigOrder(t *testing.T) {
t.Parallel()
cfg := twoHosts()
cfg.Hosts[1].Pool = ""
first, second := newFakeBackend(), newFakeBackend()
first.addDomain("proxy-z", libvirt.DomainRunning, managedMetadata("uid-z", time.Now()))
first.addDomain("proxy-a", libvirt.DomainShutoff, managedMetadata("uid-a", time.Now()))
second.addDomain("proxy-b", libvirt.DomainRunning, managedMetadata("uid-b", time.Now()))
p := newProvider(testProvider, cfg, &hostDialer{backends: map[string]backend{"beelink": first, "lab2": second}})
ctx, _ := captureContext()
list, err := p.ListByTag(ctx)
if err != nil {
t.Fatalf("ListByTag() error = %v", err)
}
want := []string{formatProviderID("beelink", "proxy-a"), formatProviderID("beelink", "proxy-z"), formatProviderID("lab2", "proxy-b")}
if got := listIDs(list); strings.Join(got, ",") != strings.Join(want, ",") {
t.Errorf("ListByTag = %v, want %v (config order, then name order)", got, want)
}
if list[0].UID != "uid-a" || list[1].UID != "uid-z" || list[2].UID != "uid-b" {
t.Errorf("uids = %s, %s, %s, want uid-a, uid-z, uid-b", list[0].UID, list[1].UID, list[2].UID)
}
}
func TestListByTag_doesNotReadStateOrAddresses(t *testing.T) {
t.Parallel()
fx := newFixture(t, 2)
id := fx.create(testName)
fx.setAddrs(testName, libvirt.DomainInterfaceAddressesSrcAgent, "10.0.0.5")
fx.fake.addDomain("proxy-off", libvirt.DomainShutoff, managedMetadata("uid-off", time.Now()))
before := len(fx.fake.callLog())
list, err := fx.p.ListByTag(fx.ctx)
if err != nil {
t.Fatalf("ListByTag() error = %v", err)
}
if len(list) != 2 || list[0].ID != id || list[1].ID != formatProviderID(testHost, "proxy-off") {
t.Fatalf("ListByTag = %v, want the two managed domains", listIDs(list))
}
for _, inst := range list {
if inst.State != "" || inst.IP != "" {
t.Errorf("listed %s = state %q ip %q, want both empty: GC does not use them", inst.ID, inst.State, inst.IP)
}
if inst.UID == "" || inst.CreatedAt.IsZero() {
t.Errorf("listed %s lacks uid or createdAt: %+v", inst.ID, inst)
}
}
for _, rpc := range fx.fake.callLog()[before:] {
switch rpc {
case "ConnectListAllDomains", "DomainGetMetadata":
default:
t.Errorf("ListByTag made %s; want only the listing and one metadata read per domain", rpc)
}
}
if got := fx.logs.matching("kvm2 listed instance"); len(got) != 2 || strings.Contains(got[0], `"state"=`) {
t.Errorf("listed instance lines = %v, want two without a state", got)
}
}
func TestListByTag_unreachableHostFailsWhole(t *testing.T) {
t.Parallel()
first := newFakeBackend()
first.addDomain("proxy-a", libvirt.DomainRunning, managedMetadata("uid-a", time.Now()))
d := &hostDialer{
backends: map[string]backend{"beelink": first},
errs: map[string]error{"lab2": errors.New("dial tcp 192.0.2.2:22: i/o timeout")},
}
p := newProvider(testProvider, twoHosts(), d)
ctx, _ := captureContext()
list, err := p.ListByTag(ctx)
if list != nil {
t.Errorf("ListByTag = %v, want nil with the error", list)
}
assertProviderError(t, err, "list", provider.ErrTransient)
}
func TestListByTag_unreadableDomainSkipped(t *testing.T) {
t.Parallel()
tests := []struct {
name string
setup func(fx *fixture)
line string
}{
{
name: "corrupt metadata on one domain",
setup: func(fx *fixture) {
fx.fake.addDomain("proxy-bad", libvirt.DomainRunning, "<bogus>not an instance</bogus>")
},
line: "skipping domain with unreadable metadata",
},
{
name: "metadata rpc fails for the first domain only",
setup: func(fx *fixture) {
fx.fake.addDomain("proxy-bad", libvirt.DomainRunning, managedMetadata("uid-bad", time.Now()))
var calls int
fx.fake.mu.Lock()
fx.fake.hooks["DomainGetMetadata"] = func() error {
calls++
if calls == 1 {
return fakeErr(libvirt.ErrInternalError, "internal error: metadata read failed")
}
return nil
}
fx.fake.mu.Unlock()
},
line: "skipping domain with unreadable metadata",
},
{
name: "domain gone between listing and metadata read",
setup: func(fx *fixture) {
fx.fake.addDomain("proxy-bad", libvirt.DomainRunning, managedMetadata("uid-bad", time.Now()))
var calls int
fx.fake.mu.Lock()
fx.fake.hooks["DomainGetMetadata"] = func() error {
calls++
if calls == 1 {
return fakeErr(libvirt.ErrNoDomain, "Domain not found")
}
return nil
}
fx.fake.mu.Unlock()
},
line: "skipping domain with unreadable metadata",
},
}
for _, tc := range tests {
t.Run(tc.name, func(t *testing.T) {
t.Parallel()
fx := newFixture(t, 1)
fx.fake.addDomain("proxy-good", libvirt.DomainRunning, managedMetadata("uid-good", time.Now()))
tc.setup(fx)
list, err := fx.p.ListByTag(fx.ctx)
if err != nil {
t.Fatalf("ListByTag() error = %v", err)
}
if got := listIDs(list); len(got) != 1 || got[0] != formatProviderID(testHost, "proxy-good") {
t.Errorf("ListByTag = %v, want only proxy-good", got)
}
if got := fx.logs.matching(tc.line); len(got) != 1 || !strings.Contains(got[0], `"name"="proxy-bad"`) {
t.Errorf("skip line = %v, want one naming proxy-bad", got)
}
})
}
}
func TestListByTag_includesInactiveDomains(t *testing.T) {
t.Parallel()
fx := newFixture(t, 1)
id := fx.create(testName)
fx.withDomain(testName, func(d *fakeDomain) { d.state = libvirt.DomainShutoff })
list, err := fx.p.ListByTag(fx.ctx)
if err != nil {
t.Fatalf("ListByTag() error = %v", err)
}
if len(list) != 1 || list[0].ID != id || list[0].UID != testUID {
t.Errorf("ListByTag = %+v, want the shut-off domain listed with its uid", list)
}
}
internal/provider/kvm2/metadata.go
+78
View File
@@ -0,0 +1,78 @@
package kvm2
import (
"encoding/xml"
"strings"
"time"
"gitea.home.hrajfrisbee.cz/kacerr/egress-proxies-operator/internal/provider"
)
// The GC mark lives in the domain's <metadata> as one element in kvm2's own
// namespace (spec entry 8). The namespace is what selects it in
// DomainGetMetadata; libvirt 11.6 hands the element back without the xmlns
// declaration, so parsing matches on the local name only.
const (
metadataURI = "https://gitea.home.hrajfrisbee.cz/kacerr/egress-proxies-operator/kvm2"
metadataLocalName = "instance"
)
// instanceMetadata is the mark as it is written and read back. The fields
// are the element's text as is; managed and createdAt interpret them.
type instanceMetadata struct {
XMLName xml.Name
Managed string `xml:"managed"`
UID string `xml:"uid"`
CreatedAt string `xml:"createdAt"`
}
func newInstanceMetadata(uid string, createdAt time.Time) instanceMetadata {
return instanceMetadata{
XMLName: xml.Name{Space: metadataURI, Local: metadataLocalName},
Managed: provider.LabelManagedYes,
UID: uid,
CreatedAt: createdAt.UTC().Format(time.RFC3339),
}
}
func (m instanceMetadata) managed() bool {
return m.Managed == provider.LabelManagedYes
}
// createdAt returns the zero time when the element carries no parsable
// timestamp; GC treats that as old, which the spec accepts.
func (m instanceMetadata) createdAt() time.Time {
t, err := time.Parse(time.RFC3339, m.CreatedAt)
if err != nil {
return time.Time{}
}
return t
}
// marshalInstanceMetadata renders the namespaced element that goes under
// the domain's <metadata>.
func marshalInstanceMetadata(m instanceMetadata) (string, error) {
m.XMLName = xml.Name{Space: metadataURI, Local: metadataLocalName}
out, err := xml.Marshal(m)
if err != nil {
return "", permanentf("marshalling instance metadata: %v", err)
}
return string(out), nil
}
// parseInstanceMetadata reads the element back, with or without its xmlns
// declaration. Anything that is not an <instance> element is ErrPermanent:
// the domain's metadata is corrupt and no retry fixes it.
func parseInstanceMetadata(s string) (instanceMetadata, error) {
var m instanceMetadata
if strings.TrimSpace(s) == "" {
return instanceMetadata{}, permanentf("parsing instance metadata: empty document")
}
if err := xml.Unmarshal([]byte(s), &m); err != nil {
return instanceMetadata{}, permanentf("parsing instance metadata: %v", err)
}
if m.XMLName.Local != metadataLocalName {
return instanceMetadata{}, permanentf("parsing instance metadata: root element is <%s>, want <%s>", m.XMLName.Local, metadataLocalName)
}
return m, nil
}
internal/provider/kvm2/metadata_test.go
+162
View File
@@ -0,0 +1,162 @@
package kvm2
import (
"errors"
"strings"
"testing"
"time"
"gitea.home.hrajfrisbee.cz/kacerr/egress-proxies-operator/internal/provider"
)
func TestMarshalInstanceMetadata_namespacedElement(t *testing.T) {
t.Parallel()
created := time.Date(2026, 9, 16, 10, 30, 0, 0, time.FixedZone("CEST", 2*3600))
got, err := marshalInstanceMetadata(newInstanceMetadata("uid-1", created))
if err != nil {
t.Fatal(err)
}
for _, want := range []string{
`<instance xmlns="` + metadataURI + `">`,
"<managed>true</managed>",
"<uid>uid-1</uid>",
"<createdAt>2026-09-16T08:30:00Z</createdAt>",
"</instance>",
} {
if !strings.Contains(got, want) {
t.Errorf("marshalled metadata %q lacks %q", got, want)
}
}
}
func TestParseMetadata_withNamespace(t *testing.T) {
t.Parallel()
in := `<instance xmlns="` + metadataURI + `"><managed>true</managed><uid>abc</uid><createdAt>2026-09-16T08:30:00Z</createdAt></instance>`
m, err := parseInstanceMetadata(in)
if err != nil {
t.Fatal(err)
}
if !m.managed() {
t.Error("managed() = false, want true")
}
if m.UID != "abc" {
t.Errorf("UID = %q, want abc", m.UID)
}
if want := time.Date(2026, 9, 16, 8, 30, 0, 0, time.UTC); !m.createdAt().Equal(want) {
t.Errorf("createdAt() = %v, want %v", m.createdAt(), want)
}
}
func TestParseMetadata_withoutNamespaceDeclaration(t *testing.T) {
t.Parallel()
in := "<instance>\n <managed>true</managed>\n <uid>abc</uid>\n <createdAt>2026-09-16T08:30:00Z</createdAt>\n</instance>"
m, err := parseInstanceMetadata(in)
if err != nil {
t.Fatal(err)
}
if !m.managed() || m.UID != "abc" {
t.Errorf("got managed=%v uid=%q, want true/abc", m.managed(), m.UID)
}
if m.createdAt().IsZero() {
t.Error("createdAt() is zero, want parsed timestamp")
}
}
func TestParseMetadata_absentGivesEmptyUID(t *testing.T) {
t.Parallel()
tests := []struct {
name string
in string
}{
{name: "empty element", in: "<instance/>"},
{name: "empty element with namespace", in: `<instance xmlns="` + metadataURI + `"></instance>`},
{name: "empty children", in: "<instance><managed></managed><uid></uid><createdAt></createdAt></instance>"},
}
for _, tc := range tests {
t.Run(tc.name, func(t *testing.T) {
t.Parallel()
m, err := parseInstanceMetadata(tc.in)
if err != nil {
t.Fatal(err)
}
if m.managed() {
t.Error("managed() = true, want false")
}
if m.UID != "" {
t.Errorf("UID = %q, want empty", m.UID)
}
if !m.createdAt().IsZero() {
t.Errorf("createdAt() = %v, want zero", m.createdAt())
}
})
}
}
func TestParseMetadata_badCreatedAtIsZeroTime(t *testing.T) {
t.Parallel()
tests := []struct {
name string
in string
}{
{name: "garbage", in: "<instance><managed>true</managed><uid>u</uid><createdAt>yesterday</createdAt></instance>"},
{name: "date only", in: "<instance><managed>true</managed><uid>u</uid><createdAt>2026-09-16</createdAt></instance>"},
{name: "missing", in: "<instance><managed>true</managed><uid>u</uid></instance>"},
}
for _, tc := range tests {
t.Run(tc.name, func(t *testing.T) {
t.Parallel()
m, err := parseInstanceMetadata(tc.in)
if err != nil {
t.Fatal(err)
}
if !m.createdAt().IsZero() {
t.Errorf("createdAt() = %v, want zero", m.createdAt())
}
if !m.managed() || m.UID != "u" {
t.Errorf("other fields lost: managed=%v uid=%q", m.managed(), m.UID)
}
})
}
}
func TestParseMetadata_notXMLIsPermanent(t *testing.T) {
t.Parallel()
tests := []struct {
name string
in string
}{
{name: "empty string", in: ""},
{name: "whitespace", in: " \n"},
{name: "plain text", in: "managed: true"},
{name: "unclosed element", in: "<instance><managed>true"},
}
for _, tc := range tests {
t.Run(tc.name, func(t *testing.T) {
t.Parallel()
_, err := parseInstanceMetadata(tc.in)
if !errors.Is(err, provider.ErrPermanent) {
t.Errorf("err = %v, want ErrPermanent", err)
}
})
}
}
func TestParseMetadata_wrongRootIsPermanent(t *testing.T) {
t.Parallel()
tests := []struct {
name string
in string
}{
{name: "other local name", in: "<proxy><managed>true</managed></proxy>"},
{name: "other local name with our namespace", in: `<vm xmlns="` + metadataURI + `"><uid>x</uid></vm>`},
}
for _, tc := range tests {
t.Run(tc.name, func(t *testing.T) {
t.Parallel()
_, err := parseInstanceMetadata(tc.in)
if !errors.Is(err, provider.ErrPermanent) {
t.Errorf("err = %v, want ErrPermanent", err)
}
})
}
}
internal/provider/kvm2/observe.go
+263
View File
@@ -0,0 +1,263 @@
package kvm2
import (
"context"
"io"
"time"
"github.com/digitalocean/go-libvirt"
"go.opentelemetry.io/otel/attribute"
"go.opentelemetry.io/otel/codes"
"go.opentelemetry.io/otel/trace"
logf "sigs.k8s.io/controller-runtime/pkg/log"
"gitea.home.hrajfrisbee.cz/kacerr/egress-proxies-operator/internal/tracing"
)
// observed is the span-decorator of spec entries 17 and 20: one child span
// kvm2.<Method> and one V(1) "libvirt call" line per call, with request and
// response detail at V(2). Applied once, in the dialer, over every backend
// that leaves it.
type observed struct {
inner backend
provider string
host string
tracer trace.Tracer
}
var (
_ backend = (*observed)(nil)
_ liveness = (*observed)(nil)
)
func observe(inner backend, providerName, hostName string, opts ...tracing.Option) backend {
return &observed{inner: inner, provider: providerName, host: hostName, tracer: tracing.Tracer(opts...)}
}
// dead forwards the inner backend's liveness; a backend without one is
// never dropped by the host table.
func (o *observed) dead() <-chan struct{} {
if l, ok := o.inner.(liveness); ok {
return l.dead()
}
return nil
}
// id is one identifier a call names: a domain, pool or volume. It becomes
// both the span attribute kvm2.<key> and the log key.
type id struct{ key, val string }
func domainID(name string) id { return id{"domain", name} }
func poolID(name string) id { return id{"pool", name} }
func volumeID(name string) id { return id{"volume", name} }
// detail returns V(2)-only key/value pairs. It runs only when V(2) is on,
// so building the list of a hundred domain names costs nothing otherwise.
type detail func() []any
// call wraps one backend call: span, timing, status and the log line. fn
// returns the V(2) detail for the line, or nil.
func (o *observed) call(ctx context.Context, rpc string, ids []id, fn func(ctx context.Context) (detail, error)) error {
attrs := make([]attribute.KeyValue, 0, len(ids)+1)
attrs = append(attrs, attribute.String("kvm2.host", o.host))
for _, i := range ids {
attrs = append(attrs, attribute.String("kvm2."+i.key, i.val))
}
ctx, span := tracing.StartSpan(ctx, o.tracer, "kvm2."+rpc, trace.WithSpanKind(trace.SpanKindClient), trace.WithAttributes(attrs...))
defer span.End()
log := logf.FromContext(ctx).WithName("kvm2").WithValues("provider", o.provider, "host", o.host)
start := time.Now()
more, err := fn(ctx)
kv := make([]any, 0, 8+2*len(ids))
kv = append(kv, "rpc", rpc)
for _, i := range ids {
kv = append(kv, i.key, i.val)
}
kv = append(kv, "duration", time.Since(start))
if err != nil {
span.SetStatus(codes.Error, err.Error())
if code, ok := libvirtCode(err); ok {
span.SetAttributes(attribute.Int("kvm2.libvirt_code", int(code)))
}
kv = append(kv, "error", err.Error())
kv = append(kv, libvirtErrorFields(err)...)
}
if more != nil && log.V(2).Enabled() {
kv = append(kv, more()...)
}
log.V(1).Info("libvirt call", kv...)
return err
}
func (o *observed) DomainLookupByName(ctx context.Context, name string) (dom libvirt.Domain, err error) {
err = o.call(ctx, "DomainLookupByName", []id{domainID(name)}, func(ctx context.Context) (detail, error) {
dom, err = o.inner.DomainLookupByName(ctx, name)
return nil, err
})
return dom, err
}
func (o *observed) DomainDefineXML(ctx context.Context, xml string) (dom libvirt.Domain, err error) {
err = o.call(ctx, "DomainDefineXML", nil, func(ctx context.Context) (detail, error) {
dom, err = o.inner.DomainDefineXML(ctx, xml)
return func() []any { return []any{"domain", dom.Name, "xml", xml} }, err
})
return dom, err
}
func (o *observed) DomainCreate(ctx context.Context, dom libvirt.Domain) error {
return o.call(ctx, "DomainCreate", []id{domainID(dom.Name)}, func(ctx context.Context) (detail, error) {
return nil, o.inner.DomainCreate(ctx, dom)
})
}
func (o *observed) DomainDestroy(ctx context.Context, dom libvirt.Domain) error {
return o.call(ctx, "DomainDestroy", []id{domainID(dom.Name)}, func(ctx context.Context) (detail, error) {
return nil, o.inner.DomainDestroy(ctx, dom)
})
}
func (o *observed) DomainUndefineFlags(ctx context.Context, dom libvirt.Domain, flags libvirt.DomainUndefineFlagsValues) error {
return o.call(ctx, "DomainUndefineFlags", []id{domainID(dom.Name)}, func(ctx context.Context) (detail, error) {
return nil, o.inner.DomainUndefineFlags(ctx, dom, flags)
})
}
func (o *observed) DomainGetState(ctx context.Context, dom libvirt.Domain, flags uint32) (state int32, reason int32, err error) {
err = o.call(ctx, "DomainGetState", []id{domainID(dom.Name)}, func(ctx context.Context) (detail, error) {
state, reason, err = o.inner.DomainGetState(ctx, dom, flags)
return func() []any { return []any{"state", libvirt.DomainState(state).String(), "reason", reason} }, err
})
return state, reason, err
}
func (o *observed) DomainGetMetadata(ctx context.Context, dom libvirt.Domain, typ int32, uri libvirt.OptString, flags libvirt.DomainModificationImpact) (md string, err error) {
err = o.call(ctx, "DomainGetMetadata", []id{domainID(dom.Name)}, func(ctx context.Context) (detail, error) {
md, err = o.inner.DomainGetMetadata(ctx, dom, typ, uri, flags)
return func() []any { return []any{"metadata", md} }, err
})
return md, err
}
func (o *observed) DomainInterfaceAddresses(ctx context.Context, dom libvirt.Domain, source uint32, flags uint32) (ifaces []libvirt.DomainInterface, err error) {
err = o.call(ctx, "DomainInterfaceAddresses", []id{domainID(dom.Name)}, func(ctx context.Context) (detail, error) {
ifaces, err = o.inner.DomainInterfaceAddresses(ctx, dom, source, flags)
return func() []any { return []any{"source", source, "addresses", interfaceAddresses(ifaces)} }, err
})
return ifaces, err
}
func (o *observed) ConnectListAllDomains(ctx context.Context, needResults int32, flags libvirt.ConnectListAllDomainsFlags) (doms []libvirt.Domain, n uint32, err error) {
err = o.call(ctx, "ConnectListAllDomains", nil, func(ctx context.Context) (detail, error) {
doms, n, err = o.inner.ConnectListAllDomains(ctx, needResults, flags)
return func() []any { return []any{"count", len(doms), "domains", domainNames(doms)} }, err
})
return doms, n, err
}
func (o *observed) StoragePoolLookupByName(ctx context.Context, name string) (pool libvirt.StoragePool, err error) {
err = o.call(ctx, "StoragePoolLookupByName", []id{poolID(name)}, func(ctx context.Context) (detail, error) {
pool, err = o.inner.StoragePoolLookupByName(ctx, name)
return nil, err
})
return pool, err
}
func (o *observed) StoragePoolRefresh(ctx context.Context, pool libvirt.StoragePool, flags uint32) error {
return o.call(ctx, "StoragePoolRefresh", []id{poolID(pool.Name)}, func(ctx context.Context) (detail, error) {
return nil, o.inner.StoragePoolRefresh(ctx, pool, flags)
})
}
func (o *observed) StoragePoolGetInfo(ctx context.Context, pool libvirt.StoragePool) (state uint8, capacity, allocation, available uint64, err error) {
err = o.call(ctx, "StoragePoolGetInfo", []id{poolID(pool.Name)}, func(ctx context.Context) (detail, error) {
state, capacity, allocation, available, err = o.inner.StoragePoolGetInfo(ctx, pool)
return func() []any {
return []any{"state", libvirt.StoragePoolState(state).String(), "capacity", capacity, "allocation", allocation, "available", available}
}, err
})
return state, capacity, allocation, available, err
}
func (o *observed) StorageVolLookupByName(ctx context.Context, pool libvirt.StoragePool, name string) (vol libvirt.StorageVol, err error) {
err = o.call(ctx, "StorageVolLookupByName", []id{poolID(pool.Name), volumeID(name)}, func(ctx context.Context) (detail, error) {
vol, err = o.inner.StorageVolLookupByName(ctx, pool, name)
return nil, err
})
return vol, err
}
func (o *observed) StorageVolGetPath(ctx context.Context, vol libvirt.StorageVol) (path string, err error) {
err = o.call(ctx, "StorageVolGetPath", []id{poolID(vol.Pool), volumeID(vol.Name)}, func(ctx context.Context) (detail, error) {
path, err = o.inner.StorageVolGetPath(ctx, vol)
return func() []any { return []any{"path", path} }, err
})
return path, err
}
func (o *observed) StorageVolCreateXML(ctx context.Context, pool libvirt.StoragePool, xml string, flags libvirt.StorageVolCreateFlags) (vol libvirt.StorageVol, err error) {
err = o.call(ctx, "StorageVolCreateXML", []id{poolID(pool.Name)}, func(ctx context.Context) (detail, error) {
vol, err = o.inner.StorageVolCreateXML(ctx, pool, xml, flags)
return func() []any { return []any{"volume", vol.Name, "xml", xml} }, err
})
return vol, err
}
// StorageVolUpload and StorageVolDownload log the length only; the bodies
// are a seed ISO carrying CloudInit or a base image.
func (o *observed) StorageVolUpload(ctx context.Context, vol libvirt.StorageVol, r io.Reader, offset, length uint64, flags libvirt.StorageVolUploadFlags) error {
return o.call(ctx, "StorageVolUpload", []id{poolID(vol.Pool), volumeID(vol.Name)}, func(ctx context.Context) (detail, error) {
err := o.inner.StorageVolUpload(ctx, vol, r, offset, length, flags)
return func() []any { return []any{"offset", offset, "length", length} }, err
})
}
func (o *observed) StorageVolDownload(ctx context.Context, vol libvirt.StorageVol, w io.Writer, offset, length uint64, flags libvirt.StorageVolDownloadFlags) error {
return o.call(ctx, "StorageVolDownload", []id{poolID(vol.Pool), volumeID(vol.Name)}, func(ctx context.Context) (detail, error) {
err := o.inner.StorageVolDownload(ctx, vol, w, offset, length, flags)
return func() []any { return []any{"offset", offset, "length", length} }, err
})
}
func (o *observed) StorageVolDelete(ctx context.Context, vol libvirt.StorageVol, flags libvirt.StorageVolDeleteFlags) error {
return o.call(ctx, "StorageVolDelete", []id{poolID(vol.Pool), volumeID(vol.Name)}, func(ctx context.Context) (detail, error) {
return nil, o.inner.StorageVolDelete(ctx, vol, flags)
})
}
func (o *observed) ConnectGetLibVersion(ctx context.Context) (v uint64, err error) {
err = o.call(ctx, "ConnectGetLibVersion", nil, func(ctx context.Context) (detail, error) {
v, err = o.inner.ConnectGetLibVersion(ctx)
return func() []any { return []any{"libvirtVersion", v} }, err
})
return v, err
}
func (o *observed) ConnectGetHostname(ctx context.Context) (name string, err error) {
err = o.call(ctx, "ConnectGetHostname", nil, func(ctx context.Context) (detail, error) {
name, err = o.inner.ConnectGetHostname(ctx)
return func() []any { return []any{"hostname", name} }, err
})
return name, err
}
// Close is passed through: it is not a libvirt call and gets no span.
func (o *observed) Close() error { return o.inner.Close() }
func domainNames(doms []libvirt.Domain) []string {
names := make([]string, len(doms))
for i, d := range doms {
names[i] = d.Name
}
return names
}
func interfaceAddresses(ifaces []libvirt.DomainInterface) []string {
var out []string
for _, iface := range ifaces {
for _, a := range iface.Addrs {
out = append(out, iface.Name+"="+a.Addr)
}
}
return out
}
internal/provider/kvm2/observe_test.go
+334
View File
@@ -0,0 +1,334 @@
package kvm2
import (
"context"
"errors"
"reflect"
"strconv"
"strings"
"sync"
"testing"
"github.com/digitalocean/go-libvirt"
"github.com/go-logr/logr/funcr"
"go.opentelemetry.io/otel/attribute"
"go.opentelemetry.io/otel/codes"
sdktrace "go.opentelemetry.io/otel/sdk/trace"
"go.opentelemetry.io/otel/sdk/trace/tracetest"
logf "sigs.k8s.io/controller-runtime/pkg/log"
"gitea.home.hrajfrisbee.cz/kacerr/egress-proxies-operator/internal/tracing"
)
// logCapture collects funcr output at a verbosity, safe for concurrent use.
type logCapture struct {
mu sync.Mutex
lines []string
}
func (c *logCapture) all() []string {
c.mu.Lock()
defer c.mu.Unlock()
return append([]string(nil), c.lines...)
}
func (c *logCapture) matching(substr string) []string {
var out []string
for _, l := range c.all() {
if strings.Contains(l, substr) {
out = append(out, l)
}
}
return out
}
// observeFixture is a decorated fake with a span recorder and a log capture
// wired into ctx.
type observeFixture struct {
fake *fakeBackend
b backend
sr *tracetest.SpanRecorder
tp *sdktrace.TracerProvider
logs *logCapture
}
func newObserveFixture(t *testing.T, verbosity int) (*observeFixture, context.Context) {
t.Helper()
sr := tracetest.NewSpanRecorder()
tp := sdktrace.NewTracerProvider(sdktrace.WithSpanProcessor(sr))
t.Cleanup(func() { _ = tp.Shutdown(context.Background()) })
logs := &logCapture{}
log := funcr.New(func(prefix, args string) {
logs.mu.Lock()
defer logs.mu.Unlock()
logs.lines = append(logs.lines, prefix+" "+args)
}, funcr.Options{Verbosity: verbosity})
fake := newFakeBackend()
fx := &observeFixture{
fake: fake,
b: observe(fake, "kvm2-home", "beelink", tracing.WithTracerProvider(tp)),
sr: sr,
tp: tp,
logs: logs,
}
return fx, logf.IntoContext(context.Background(), log)
}
func TestObserve_everySeamMethodInstrumented(t *testing.T) {
t.Parallel()
seam := reflect.TypeOf((*backend)(nil)).Elem()
for i := range seam.NumMethod() {
m := seam.Method(i)
if m.Name == "Close" {
continue
}
t.Run(m.Name, func(t *testing.T) {
t.Parallel()
fx, ctx := newObserveFixture(t, 1)
args := make([]reflect.Value, m.Type.NumIn())
args[0] = reflect.ValueOf(ctx)
for j := 1; j < m.Type.NumIn(); j++ {
args[j] = reflect.Zero(m.Type.In(j))
}
reflect.ValueOf(fx.b).MethodByName(m.Name).Call(args)
ended := fx.sr.Ended()
if len(ended) != 1 {
t.Fatalf("got %d spans, want 1", len(ended))
}
if got, want := ended[0].Name(), "kvm2."+m.Name; got != want {
t.Errorf("span name = %q, want %q", got, want)
}
if !hasAttr(ended[0].Attributes(), attribute.String("kvm2.host", "beelink")) {
t.Errorf("span attributes %v lack kvm2.host", ended[0].Attributes())
}
lines := fx.logs.matching(`"msg"="libvirt call"`)
if len(lines) != 1 {
t.Fatalf("got %d 'libvirt call' lines, want 1:\n%s", len(lines), strings.Join(fx.logs.all(), "\n"))
}
for _, want := range []string{`"rpc"="` + m.Name + `"`, `"provider"="kvm2-home"`, `"host"="beelink"`, `"duration"=`, `"traceID"=`} {
if !strings.Contains(lines[0], want) {
t.Errorf("log line lacks %s: %s", want, lines[0])
}
}
if fx.fake.callCount(m.Name) != 1 {
t.Errorf("inner %s called %d times, want 1", m.Name, fx.fake.callCount(m.Name))
}
})
}
}
func TestObserve_childSpansUnderParent(t *testing.T) {
t.Parallel()
fx, ctx := newObserveFixture(t, 1)
ctx, parent := tracing.StartSpan(ctx, fx.tp.Tracer("test"), "provider.get")
_, _ = fx.b.DomainLookupByName(ctx, "proxy-x")
parent.End()
var child sdktrace.ReadOnlySpan
for _, s := range fx.sr.Ended() {
if s.Name() == "kvm2.DomainLookupByName" {
child = s
}
}
if child == nil {
t.Fatal("no kvm2.DomainLookupByName span recorded")
}
if child.Parent().SpanID() != parent.SpanContext().SpanID() {
t.Errorf("child parent = %s, want %s", child.Parent().SpanID(), parent.SpanContext().SpanID())
}
if child.SpanContext().TraceID() != parent.SpanContext().TraceID() {
t.Error("child is in a different trace than its parent")
}
if !hasAttr(child.Attributes(), attribute.String("kvm2.domain", "proxy-x")) {
t.Errorf("child attributes %v lack kvm2.domain", child.Attributes())
}
lines := fx.logs.matching(`"msg"="libvirt call"`)
if len(lines) != 1 || !strings.Contains(lines[0], `"spanID"="`+child.SpanContext().SpanID().String()+`"`) {
t.Errorf("log line does not carry the child's spanID: %v", lines)
}
}
func TestObserve_logsOneLinePerCallAtV1(t *testing.T) {
t.Parallel()
fx, ctx := newObserveFixture(t, 1)
fx.fake.addDomain("proxy-x", libvirt.DomainRunning, "")
pool, _ := fx.b.StoragePoolLookupByName(ctx, fakePoolName)
_, _ = fx.b.DomainLookupByName(ctx, "proxy-x")
_, _ = fx.b.StorageVolLookupByName(ctx, pool, "proxy-x.qcow2")
lines := fx.logs.matching(`"msg"="libvirt call"`)
if len(lines) != 3 {
t.Fatalf("got %d lines, want 3:\n%s", len(lines), strings.Join(fx.logs.all(), "\n"))
}
if len(fx.logs.all()) != 3 {
t.Errorf("got %d lines in total, want only the 3 call lines:\n%s", len(fx.logs.all()), strings.Join(fx.logs.all(), "\n"))
}
wants := []string{
`"rpc"="StoragePoolLookupByName" "pool"="default"`,
`"rpc"="DomainLookupByName" "domain"="proxy-x"`,
`"rpc"="StorageVolLookupByName" "pool"="default" "volume"="proxy-x.qcow2"`,
}
for i, want := range wants {
if !strings.Contains(lines[i], want) {
t.Errorf("line %d = %s, want %s in it", i, lines[i], want)
}
if !strings.Contains(lines[i], `"level"=1`) {
t.Errorf("line %d is not V(1): %s", i, lines[i])
}
}
}
func TestObserve_detailOnlyAtV2(t *testing.T) {
t.Parallel()
const xmlDoc = "<domain><name>proxy-x</name><metadata><instance><uid>u1</uid></instance></metadata></domain>"
tests := []struct {
name string
verbosity int
wantXML bool
}{
{name: "V1 has no xml", verbosity: 1, wantXML: false},
{name: "V2 has xml", verbosity: 2, wantXML: true},
}
for _, tc := range tests {
t.Run(tc.name, func(t *testing.T) {
t.Parallel()
fx, ctx := newObserveFixture(t, tc.verbosity)
dom, err := fx.b.DomainDefineXML(ctx, xmlDoc)
if err != nil {
t.Fatalf("DomainDefineXML error = %v", err)
}
_, _ = fx.b.DomainGetMetadata(ctx, dom, int32(libvirt.DomainMetadataElement), libvirt.OptString{metadataURI}, 0)
_, _, _ = fx.b.ConnectListAllDomains(ctx, 1, 0)
all := strings.Join(fx.logs.all(), "\n")
if got := strings.Contains(all, "<domain>"); got != tc.wantXML {
t.Errorf("xml logged = %v, want %v:\n%s", got, tc.wantXML, all)
}
if got := strings.Contains(all, `"metadata"="<instance>`); got != tc.wantXML {
t.Errorf("metadata logged = %v, want %v:\n%s", got, tc.wantXML, all)
}
if got := strings.Contains(all, `"domains"=["proxy-x"]`); got != tc.wantXML {
t.Errorf("domain names logged = %v, want %v:\n%s", got, tc.wantXML, all)
}
if got := len(fx.logs.matching(`"msg"="libvirt call"`)); got != 3 {
t.Errorf("got %d call lines, want 3", got)
}
})
}
}
func TestObserve_uploadBodyNeverLogged(t *testing.T) {
t.Parallel()
const sentinel = "SENTINEL-CLOUDINIT-3c7a"
fx, ctx := newObserveFixture(t, 2)
pool, _ := fx.b.StoragePoolLookupByName(ctx, fakePoolName)
vol, err := fx.b.StorageVolCreateXML(ctx, pool, "<volume><name>proxy-x-seed.iso</name><capacity>1024</capacity></volume>", 0)
if err != nil {
t.Fatalf("StorageVolCreateXML error = %v", err)
}
body := "#cloud-config\npassword: " + sentinel + "\n"
if err := fx.b.StorageVolUpload(ctx, vol, strings.NewReader(body), 0, uint64(len(body)), 0); err != nil {
t.Fatalf("StorageVolUpload error = %v", err)
}
var sink strings.Builder
if err := fx.b.StorageVolDownload(ctx, vol, &sink, 0, 0, 0); err != nil {
t.Fatalf("StorageVolDownload error = %v", err)
}
if sink.String() != body {
t.Errorf("downloaded %q, want the uploaded body", sink.String())
}
all := strings.Join(fx.logs.all(), "\n")
if strings.Contains(all, sentinel) {
t.Errorf("upload body leaked into logs:\n%s", all)
}
uploads := fx.logs.matching(`"rpc"="StorageVolUpload"`)
if len(uploads) != 1 || !strings.Contains(uploads[0], `"length"=`+strconv.Itoa(len(body))) {
t.Errorf("upload line lacks length: %v", uploads)
}
for _, s := range fx.sr.Ended() {
for _, a := range s.Attributes() {
if strings.Contains(a.Value.Emit(), sentinel) {
t.Errorf("span %s attribute %s carries the body", s.Name(), a.Key)
}
}
}
}
func TestObserve_errorFieldsOnFailure(t *testing.T) {
t.Parallel()
fx, ctx := newObserveFixture(t, 1)
_, err := fx.b.DomainLookupByName(ctx, "missing")
if !hasCode(err, libvirt.ErrNoDomain) {
t.Fatalf("error = %v, want ErrNoDomain through the decorator", err)
}
lines := fx.logs.matching(`"msg"="libvirt call"`)
if len(lines) != 1 {
t.Fatalf("got %d lines, want 1", len(lines))
}
for _, want := range []string{`"error"="Domain not found`, `"libvirtCode"=42`, `"libvirtError"="ErrNoDomain"`} {
if !strings.Contains(lines[0], want) {
t.Errorf("line lacks %s: %s", want, lines[0])
}
}
ended := fx.sr.Ended()
if len(ended) != 1 {
t.Fatalf("got %d spans, want 1", len(ended))
}
if ended[0].Status().Code != codes.Error {
t.Errorf("span status = %v, want Error", ended[0].Status().Code)
}
if !hasAttr(ended[0].Attributes(), attribute.Int("kvm2.libvirt_code", 42)) {
t.Errorf("span attributes %v lack kvm2.libvirt_code=42", ended[0].Attributes())
}
fx.fake.failWith("ConnectGetHostname", errors.New("plain failure"))
_, _ = fx.b.ConnectGetHostname(ctx)
lines = fx.logs.matching(`"rpc"="ConnectGetHostname"`)
if len(lines) != 1 || strings.Contains(lines[0], "libvirtCode") || !strings.Contains(lines[0], `"error"="plain failure"`) {
t.Errorf("non-libvirt failure line = %v, want error without libvirt fields", lines)
}
}
func TestObserve_v0Silent(t *testing.T) {
t.Parallel()
fx, ctx := newObserveFixture(t, 0)
fx.fake.addDomain("proxy-x", libvirt.DomainRunning, "")
_, _ = fx.b.DomainLookupByName(ctx, "proxy-x")
_, _ = fx.b.DomainLookupByName(ctx, "missing")
if lines := fx.logs.all(); len(lines) != 0 {
t.Errorf("got %d lines at V(0), want none:\n%s", len(lines), strings.Join(lines, "\n"))
}
if got := len(fx.sr.Ended()); got != 2 {
t.Errorf("got %d spans, want 2: spans do not depend on verbosity", got)
}
}
func TestObserve_closeAndLivenessPassThrough(t *testing.T) {
t.Parallel()
fx, _ := newObserveFixture(t, 1)
if isDead(fx.b) {
t.Error("decorated live backend reported dead")
}
fx.fake.markDead()
if !isDead(fx.b) {
t.Error("decorated dead backend reported live")
}
if err := fx.b.Close(); err != nil || fx.fake.closed != 1 {
t.Errorf("Close() = %v, closed %d times, want nil and 1", err, fx.fake.closed)
}
if got := len(fx.sr.Ended()); got != 0 {
t.Errorf("Close produced %d spans, want none", got)
}
if lines := fx.logs.all(); len(lines) != 0 {
t.Errorf("Close produced log lines: %v", lines)
}
}
func hasAttr(attrs []attribute.KeyValue, want attribute.KeyValue) bool {
for _, a := range attrs {
if a.Key == want.Key && a.Value == want.Value {
return true
}
}
return false
}
internal/provider/kvm2/provider_test.go
+426
View File
@@ -0,0 +1,426 @@
package kvm2
import (
"context"
"errors"
"fmt"
"net/http/httptest"
"strings"
"testing"
"time"
"github.com/digitalocean/go-libvirt"
"github.com/go-logr/logr/funcr"
logf "sigs.k8s.io/controller-runtime/pkg/log"
"gitea.home.hrajfrisbee.cz/kacerr/egress-proxies-operator/internal/provider"
)
const (
testProvider = "kvm2-home"
testHost = "beelink"
testUID = "0f1e2d3c-uid-1"
testName = "proxy-abcdefghijklmnop"
)
// fixture is one configured host backed by a fake, an image server the
// catalogue points at, a log capture at a chosen verbosity and a provider
// with a fast running-wait, so a Create runs end to end in memory.
type fixture struct {
t *testing.T
p *Provider
fake *fakeBackend
dialer *stubDialer
logs *logCapture
ctx context.Context
image []byte
}
func newFixture(t *testing.T, verbosity int, opts ...providerOption) *fixture {
t.Helper()
fx := &fixture{t: t, fake: newFakeBackend(), logs: &logCapture{}, image: []byte("QFI\xfb fake base image bytes")}
srv := httptest.NewServer(serveBytes(fx.image))
t.Cleanup(srv.Close)
fx.dialer = &stubDialer{backend: fx.fake}
cfg := provider.KVMConfig{
Images: map[string]provider.KVMImage{imageKey: {URL: srv.URL + "/debian-13-generic-amd64.qcow2"}},
MachineTypes: map[string]provider.KVMMachineType{"small": {VCPUs: 1, MemoryMiB: 1024}},
Hosts: []provider.KVMHost{{Name: testHost, Address: "beelink:22", User: "virt", Bridge: "br0"}},
}
opts = append([]providerOption{withHTTPClient(srv.Client()), withRunningWait(time.Millisecond, 500*time.Millisecond)}, opts...)
fx.p = newProvider(testProvider, cfg, fx.dialer, opts...)
log := funcr.New(func(prefix, args string) {
fx.logs.mu.Lock()
defer fx.logs.mu.Unlock()
fx.logs.lines = append(fx.logs.lines, prefix+" "+args)
}, funcr.Options{Verbosity: verbosity})
fx.ctx = logf.IntoContext(context.Background(), log)
return fx
}
func (fx *fixture) req(name string) provider.CreateRequest {
return provider.CreateRequest{
Name: name,
UID: testUID,
Namespace: "default",
ProxyName: "egress",
Placement: provider.Placement{Zone: testHost, MachineType: "small", Image: imageKey},
CloudInit: "#cloud-config\npackages: [squid]\n",
Port: 3128,
}
}
func (fx *fixture) create(name string) string {
fx.t.Helper()
id, err := fx.p.Create(fx.ctx, fx.req(name))
if err != nil {
fx.t.Fatalf("Create(%s) error = %v", name, err)
}
if want := formatProviderID(testHost, name); id != want {
fx.t.Fatalf("Create(%s) = %q, want %q", name, id, want)
}
return id
}
func (fx *fixture) get(id string) *provider.Instance {
fx.t.Helper()
inst, err := fx.p.Get(fx.ctx, id)
if err != nil {
fx.t.Fatalf("Get(%s) error = %v", id, err)
}
return inst
}
// withDomain runs fn on the fake's domain under its lock.
func (fx *fixture) withDomain(name string, fn func(d *fakeDomain)) {
fx.t.Helper()
fx.fake.mu.Lock()
defer fx.fake.mu.Unlock()
d, ok := fx.fake.domains[name]
if !ok {
fx.t.Fatalf("fake has no domain %s", name)
}
fn(d)
}
func (fx *fixture) hasDomain(name string) bool {
fx.fake.mu.Lock()
defer fx.fake.mu.Unlock()
_, ok := fx.fake.domains[name]
return ok
}
func (fx *fixture) setAddrs(name string, src libvirt.DomainInterfaceAddressesSource, ips ...string) {
fx.withDomain(name, func(d *fakeDomain) {
var addrs []libvirt.DomainIPAddr
for _, ip := range ips {
addrs = append(addrs, libvirt.DomainIPAddr{Type: int32(libvirt.IPAddrTypeIpv4), Addr: ip, Prefix: 24})
}
d.addrs[uint32(src)] = []libvirt.DomainInterface{{Name: "enp1s0", Addrs: addrs}}
})
}
// putVol places a volume in the fake's pool directly.
func (fx *fixture) putVol(name string, content []byte) *fakeVol {
fx.fake.mu.Lock()
defer fx.fake.mu.Unlock()
v := &fakeVol{content: content, capacity: uint64(len(content))}
fx.fake.pools[fakePoolName].vols[name] = v
return v
}
// baseName is the cached base image's volume name for this fixture's
// provider entry and image key.
func (fx *fixture) baseName() string {
fx.t.Helper()
src, err := resolveImage(fx.p.name, fx.p.cfg, imageKey)
if err != nil {
fx.t.Fatal(err)
}
return src.volumeName()
}
// cacheImage places a complete base image and its marker, as a previous
// Create on the host would have.
func (fx *fixture) cacheImage() {
src, err := resolveImage(fx.p.name, fx.p.cfg, imageKey)
if err != nil {
fx.t.Fatal(err)
}
fx.putVol(src.volumeName(), fx.image)
fx.putVol(src.markerName(), []byte(digestOf(fx.image)))
}
func managedMetadata(uid string, createdAt time.Time) string {
s, err := marshalInstanceMetadata(newInstanceMetadata(uid, createdAt))
if err != nil {
panic(err)
}
return s
}
// assertProviderError checks err is a *provider.Error from the configured
// provider with the given op and class.
func assertProviderError(t *testing.T, err error, op string, class error) {
t.Helper()
if err == nil {
t.Fatalf("error = nil, want op %s class %v", op, class)
}
var pe *provider.Error
if !errors.As(err, &pe) {
t.Fatalf("error = %T (%v), want *provider.Error", err, err)
}
if pe.Provider != testProvider {
t.Errorf("error.Provider = %q, want %q", pe.Provider, testProvider)
}
if pe.Op != op {
t.Errorf("error.Op = %q, want %q", pe.Op, op)
}
if !errors.Is(err, class) {
t.Errorf("error = %v, want class %v", err, class)
}
for _, other := range []error{provider.ErrNotFound, provider.ErrQuotaExceeded, provider.ErrPermanent, provider.ErrTransient} {
if other != class && errors.Is(err, other) {
t.Errorf("error = %v, also matches %v", err, other)
}
}
}
// hostDialer hands out one backend per host name and fails hosts in errs.
type hostDialer struct {
backends map[string]backend
errs map[string]error
}
func (d *hostDialer) dial(_ context.Context, host provider.KVMHost) (backend, error) {
if err := d.errs[host.Name]; err != nil {
return nil, err
}
b, ok := d.backends[host.Name]
if !ok {
return nil, fmt.Errorf("hostDialer: no backend for %s", host.Name)
}
return b, nil
}
// waitFor polls cond until it holds or the deadline passes.
func waitFor(t *testing.T, what string, cond func() bool) {
t.Helper()
deadline := time.Now().Add(5 * time.Second)
for !cond() {
if time.Now().After(deadline) {
t.Fatalf("timed out waiting for %s", what)
}
time.Sleep(time.Millisecond)
}
}
func TestProvider_cancelledContextReturnsPromptly(t *testing.T) {
t.Parallel()
tests := []struct {
name string
firstRPC string
op string
call func(ctx context.Context, fx *fixture) error
}{
{
name: "create", firstRPC: "DomainLookupByName", op: "create",
call: func(ctx context.Context, fx *fixture) error { _, err := fx.p.Create(ctx, fx.req(testName)); return err },
},
{
name: "get", firstRPC: "DomainLookupByName", op: "get",
call: func(ctx context.Context, fx *fixture) error {
_, err := fx.p.Get(ctx, formatProviderID(testHost, testName))
return err
},
},
{
name: "delete", firstRPC: "DomainLookupByName", op: "delete",
call: func(ctx context.Context, fx *fixture) error {
return fx.p.Delete(ctx, formatProviderID(testHost, testName))
},
},
{
name: "list", firstRPC: "ConnectListAllDomains", op: "list",
call: func(ctx context.Context, fx *fixture) error { _, err := fx.p.ListByTag(ctx); return err },
},
}
for _, tc := range tests {
t.Run(tc.name, func(t *testing.T) {
t.Parallel()
fx := newFixture(t, 1)
release := fx.fake.blockOn(tc.firstRPC)
defer close(release)
ctx, cancel := context.WithCancel(fx.ctx)
done := make(chan error, 1)
go func() { done <- tc.call(ctx, fx) }()
waitFor(t, "the first RPC to block", func() bool { return fx.fake.callCount(tc.firstRPC) == 1 })
cancel()
select {
case err := <-done:
assertProviderError(t, err, tc.op, provider.ErrTransient)
case <-time.After(time.Second):
t.Fatalf("%s did not return within 1s of cancellation", tc.name)
}
})
}
}
func TestProvider_everyErrorIsWrappedWithConfiguredName(t *testing.T) {
t.Parallel()
dialErr := errors.New("dial tcp 192.0.2.1:22: connect: connection refused")
tests := []struct {
name string
setup func(fx *fixture)
call func(fx *fixture) error
op string
class error
}{
{
name: "create unknown zone",
call: func(fx *fixture) error {
req := fx.req(testName)
req.Placement.Zone = "nas9"
_, err := fx.p.Create(fx.ctx, req)
return err
},
op: "create", class: provider.ErrPermanent,
},
{
name: "create dial failure",
setup: func(fx *fixture) { fx.dialer.err = dialErr },
call: func(fx *fixture) error { _, err := fx.p.Create(fx.ctx, fx.req(testName)); return err },
op: "create", class: provider.ErrTransient,
},
{
name: "create pool out of space",
setup: func(fx *fixture) {
fx.fake.mu.Lock()
fx.fake.pools[fakePoolName].available = 1
fx.fake.mu.Unlock()
},
call: func(fx *fixture) error { _, err := fx.p.Create(fx.ctx, fx.req(testName)); return err },
op: "create", class: provider.ErrQuotaExceeded,
},
{
name: "create vanished volume",
setup: func(fx *fixture) { fx.fake.failWith("StorageVolGetPath", fakeErr(libvirt.ErrNoStorageVol, "gone")) },
call: func(fx *fixture) error { _, err := fx.p.Create(fx.ctx, fx.req(testName)); return err },
op: "create", class: provider.ErrTransient,
},
{
name: "get malformed id",
call: func(fx *fixture) error { _, err := fx.p.Get(fx.ctx, "beelink/"+testName); return err },
op: "get", class: provider.ErrPermanent,
},
{
name: "get missing domain",
call: func(fx *fixture) error { _, err := fx.p.Get(fx.ctx, formatProviderID(testHost, testName)); return err },
op: "get", class: provider.ErrNotFound,
},
{
name: "get busy backend",
setup: func(fx *fixture) {
fx.create(testName)
fx.fake.failWith("DomainGetState", fakeErr(libvirt.ErrOperationInvalid, "busy"))
},
call: func(fx *fixture) error { _, err := fx.p.Get(fx.ctx, formatProviderID(testHost, testName)); return err },
op: "get", class: provider.ErrTransient,
},
{
name: "delete malformed id",
call: func(fx *fixture) error { return fx.p.Delete(fx.ctx, "hosts//domains/"+testName) },
op: "delete", class: provider.ErrPermanent,
},
{
name: "delete dial failure",
setup: func(fx *fixture) { fx.dialer.err = dialErr },
call: func(fx *fixture) error { return fx.p.Delete(fx.ctx, formatProviderID(testHost, testName)) },
op: "delete", class: provider.ErrTransient,
},
{
name: "delete access denied",
setup: func(fx *fixture) { fx.fake.failWith("DomainLookupByName", fakeErr(libvirt.ErrAccessDenied, "denied")) },
call: func(fx *fixture) error { return fx.p.Delete(fx.ctx, formatProviderID(testHost, testName)) },
op: "delete", class: provider.ErrPermanent,
},
{
name: "list dial failure",
setup: func(fx *fixture) { fx.dialer.err = dialErr },
call: func(fx *fixture) error { _, err := fx.p.ListByTag(fx.ctx); return err },
op: "list", class: provider.ErrTransient,
},
{
name: "list rpc failure",
setup: func(fx *fixture) {
fx.fake.failWith("ConnectListAllDomains", fakeErr(libvirt.ErrInternalError, "boom"))
},
call: func(fx *fixture) error { _, err := fx.p.ListByTag(fx.ctx); return err },
op: "list", class: provider.ErrTransient,
},
}
for _, tc := range tests {
t.Run(tc.name, func(t *testing.T) {
t.Parallel()
fx := newFixture(t, 1)
if tc.setup != nil {
tc.setup(fx)
}
assertProviderError(t, tc.call(fx), tc.op, tc.class)
})
}
}
func TestLogging_v0SilentV1OutcomeV2Detail(t *testing.T) {
t.Parallel()
outcomes := []string{"overlay volume created", "seed volume uploaded", "domain defined", "kvm2 domain ensured running"}
t.Run("v0 is silent", func(t *testing.T) {
t.Parallel()
fx := newFixture(t, 0)
fx.create(testName)
if lines := fx.logs.all(); len(lines) != 0 {
t.Errorf("Create logged %d lines at V(0), want none:\n%s", len(lines), strings.Join(lines, "\n"))
}
})
t.Run("v1 has outcomes without xml", func(t *testing.T) {
t.Parallel()
fx := newFixture(t, 1)
fx.create(testName)
for _, want := range outcomes {
if len(fx.logs.matching(want)) != 1 {
t.Errorf("V(1) lines matching %q = %d, want 1:\n%s", want, len(fx.logs.matching(want)), strings.Join(fx.logs.all(), "\n"))
}
}
if got := fx.logs.matching("<domain"); len(got) != 0 {
t.Errorf("V(1) carries domain XML:\n%s", strings.Join(got, "\n"))
}
if got := fx.logs.matching("kvm2 create resolved"); len(got) != 0 {
t.Errorf("V(1) carries the V(2) detail line:\n%s", strings.Join(got, "\n"))
}
for _, l := range fx.logs.all() {
if !strings.Contains(l, `"provider"="kvm2-home"`) || !strings.Contains(l, `"host"="beelink"`) {
t.Errorf("line lacks provider or host: %s", l)
}
if !strings.Contains(l, `"name"="`+testName+`"`) && !strings.Contains(l, `"volume"="base-`) {
t.Errorf("instance line lacks the name: %s", l)
}
}
})
t.Run("v2 has detail", func(t *testing.T) {
t.Parallel()
fx := newFixture(t, 2)
fx.create(testName)
if got := fx.logs.matching("<domain"); len(got) == 0 {
t.Errorf("V(2) lacks domain XML:\n%s", strings.Join(fx.logs.all(), "\n"))
}
resolved := fx.logs.matching("kvm2 create resolved")
if len(resolved) != 1 {
t.Fatalf("V(2) lines matching resolved = %d, want 1", len(resolved))
}
for _, key := range []string{`"cloudInitBytes"=`, `"image"="base-` + testProvider + `-debian-13.qcow2"`, `"firmware"="uefi"`, `"vcpus"=1`, `"memoryMiB"=1024`, `"machineType"="small"`, `"bridge"="br0"`, `"pool"="default"`} {
if !strings.Contains(resolved[0], key) {
t.Errorf("resolved line lacks %s: %s", key, resolved[0])
}
}
})
}
internal/provider/kvm2/seed.go
+124
View File
@@ -0,0 +1,124 @@
package kvm2
import (
"bytes"
"errors"
"fmt"
"io"
"mime/multipart"
"net/textproto"
"strings"
"github.com/kdomanski/iso9660"
)
// seedLabel is the volume label cloud-init's NoCloud datasource looks for.
const seedLabel = "cidata"
// guestAgentScript is the vendor-data payload that gets qemu-guest-agent
// running so the provider can read the VM's address through libvirt. It is
// a shell script and not a cloud-config on purpose: cloud-init merges a
// vendor-data cloud-config under the user-data with mergers that keep the
// user-data's value for every top-level key and never append lists, so a
// Proxy with its own packages: would silently lose the agent. Scripts
// bypass the merge.
const guestAgentScript = `#!/bin/sh
# Added by egress-proxies-operator: installs and starts qemu-guest-agent so
# the operator can learn this VM's IP address through libvirt.
# To leave this out, set installGuestAgent: false in the provider's config.
set -e
if command -v qemu-ga >/dev/null 2>&1 || [ -x /usr/sbin/qemu-ga ]; then
echo "egress-proxies-operator: qemu-guest-agent already installed"
elif command -v apt-get >/dev/null 2>&1; then
export DEBIAN_FRONTEND=noninteractive
apt-get update
apt-get install -y qemu-guest-agent
elif command -v dnf >/dev/null 2>&1; then
dnf install -y qemu-guest-agent
elif command -v yum >/dev/null 2>&1; then
yum install -y qemu-guest-agent
elif command -v zypper >/dev/null 2>&1; then
zypper --non-interactive install qemu-guest-agent
elif command -v apk >/dev/null 2>&1; then
apk add qemu-guest-agent
else
echo "egress-proxies-operator: no supported package manager found; install qemu-guest-agent by hand" >&2
exit 1
fi
if command -v systemctl >/dev/null 2>&1; then
systemctl enable --now qemu-guest-agent
else
rc-update add qemu-guest-agent default
rc-service qemu-guest-agent start
fi
`
// buildSeedISO writes a NoCloud seed to w: user-data byte for byte,
// meta-data naming the instance, and vendor-data (an empty file when there
// is none, so cloud-init does not look elsewhere). iso9660.NewWriter stages
// the files in os.MkdirTemp("", ""), that is under os.TempDir(): $TMPDIR
// when set, else /tmp, the manager's 64Mi emptyDir. The staging dir is
// removed before returning. Nothing about the payload is logged here.
func buildSeedISO(name, userData, vendorData string, w io.Writer) (err error) {
if name == "" {
return permanentf("seed ISO: instance name is empty")
}
iw, err := iso9660.NewWriter()
if err != nil {
return fmt.Errorf("creating seed ISO staging dir: %w", err)
}
defer func() {
if cerr := iw.Cleanup(); cerr != nil {
err = errors.Join(err, fmt.Errorf("removing seed ISO staging dir: %w", cerr))
}
}()
metaData := "instance-id: " + name + "\nlocal-hostname: " + name + "\n"
files := []struct{ name, content string }{
{"user-data", userData},
{"meta-data", metaData},
{"vendor-data", vendorData},
}
for _, f := range files {
if err := iw.AddFile(strings.NewReader(f.content), f.name); err != nil {
return fmt.Errorf("adding %s to seed ISO: %w", f.name, err)
}
}
if err := iw.WriteTo(w, seedLabel); err != nil {
return fmt.Errorf("writing seed ISO: %w", err)
}
return nil
}
// buildVendorData composes the seed's vendor-data from the configured
// document and the guest agent script. With both, they travel as separate
// parts of a MIME multipart, so cloud-init handles each by its own type; the
// MIME-Version header comes first because cloud-init detects multipart by
// finding "mime-version:" within the first 4096 bytes.
func buildVendorData(configured string, installAgent bool) (string, error) {
switch {
case !installAgent:
return configured, nil
case strings.TrimSpace(configured) == "":
return guestAgentScript, nil
}
var buf bytes.Buffer
mw := multipart.NewWriter(&buf)
fmt.Fprintf(&buf, "MIME-Version: 1.0\r\nContent-Type: multipart/mixed; boundary=%q\r\n\r\n", mw.Boundary())
parts := []struct{ contentType, body string }{
{"text/plain", configured},
{"text/x-shellscript", guestAgentScript},
}
for _, p := range parts {
pw, err := mw.CreatePart(textproto.MIMEHeader{"Content-Type": {p.contentType}})
if err != nil {
return "", fmt.Errorf("creating vendor-data part %s: %w", p.contentType, err)
}
if _, err := io.WriteString(pw, p.body); err != nil {
return "", fmt.Errorf("writing vendor-data part %s: %w", p.contentType, err)
}
}
if err := mw.Close(); err != nil {
return "", fmt.Errorf("closing vendor-data multipart: %w", err)
}
return buf.String(), nil
}
internal/provider/kvm2/seed_test.go
+264
View File
@@ -0,0 +1,264 @@
package kvm2
import (
"bufio"
"bytes"
"io"
"mime"
"mime/multipart"
"net/textproto"
"os"
"strings"
"testing"
"github.com/kdomanski/iso9660"
)
// readSeed builds a seed and returns its files by name plus the label.
func readSeed(t *testing.T, name, userData, vendorData string) (map[string]string, string) {
t.Helper()
var buf bytes.Buffer
if err := buildSeedISO(name, userData, vendorData, &buf); err != nil {
t.Fatalf("buildSeedISO: %v", err)
}
img, err := iso9660.OpenImage(bytes.NewReader(buf.Bytes()))
if err != nil {
t.Fatalf("OpenImage: %v", err)
}
root, err := img.RootDir()
if err != nil {
t.Fatalf("RootDir: %v", err)
}
children, err := root.GetChildren()
if err != nil {
t.Fatalf("GetChildren: %v", err)
}
files := make(map[string]string, len(children))
for _, c := range children {
if c.IsDir() {
t.Errorf("unexpected directory %q in seed", c.Name())
continue
}
data, err := io.ReadAll(c.Reader())
if err != nil {
t.Fatalf("reading %s: %v", c.Name(), err)
}
files[c.Name()] = string(data)
}
label, err := img.Label()
if err != nil {
t.Fatalf("Label: %v", err)
}
return files, label
}
func TestSeedISO_filesAndLabel(t *testing.T) {
t.Parallel()
userData := "#cloud-config\npackages: [squid]\n"
vendorData := "#!/bin/sh\necho hi\n"
files, label := readSeed(t, "proxy-abc", userData, vendorData)
if label != seedLabel {
t.Errorf("label = %q, want %q", label, seedLabel)
}
want := map[string]string{
"user-data": userData,
"meta-data": "instance-id: proxy-abc\nlocal-hostname: proxy-abc\n",
"vendor-data": vendorData,
}
if len(files) != len(want) {
t.Errorf("seed has %d files %v, want exactly %d", len(files), keys(files), len(want))
}
for name, content := range want {
if got, ok := files[name]; !ok {
t.Errorf("seed lacks %s", name)
} else if got != content {
t.Errorf("%s = %q, want %q", name, got, content)
}
}
}
func keys(m map[string]string) []string {
out := make([]string, 0, len(m))
for k := range m {
out = append(out, k)
}
return out
}
func TestSeedISO_userDataByteForByte(t *testing.T) {
t.Parallel()
tests := []struct {
name string
userData string
}{
{name: "mime multipart with CRLF", userData: "MIME-Version: 1.0\r\nContent-Type: multipart/mixed; boundary=\"b\"\r\n\r\n--b\r\nContent-Type: text/cloud-config\r\n\r\n#cloud-config\r\npackages: [squid]\r\n--b--\r\n"},
{name: "jinja header", userData: "## template: jinja\n#cloud-config\nhostname: {{ v1.local_hostname }}\n"},
{name: "no trailing newline", userData: "#cloud-config\nruncmd:\n - echo done"},
{name: "bytes above 0x7f", userData: "#cloud-config\nwrite_files:\n - content: \"caf\xc3\xa9 \xff\xfe\x80\"\n"},
{name: "empty", userData: ""},
{name: "trailing whitespace preserved", userData: "#!/bin/sh\necho x \n\n\n"},
}
for _, tc := range tests {
t.Run(tc.name, func(t *testing.T) {
t.Parallel()
files, _ := readSeed(t, "proxy-abc", tc.userData, "")
if got := files["user-data"]; got != tc.userData {
t.Errorf("user-data changed:\n got %q\nwant %q", got, tc.userData)
}
})
}
}
// Not parallel: it changes TMPDIR for the process.
func TestSeedISO_tempDirRemoved(t *testing.T) {
tmp := t.TempDir()
t.Setenv("TMPDIR", tmp)
if got := os.TempDir(); got != tmp {
t.Fatalf("os.TempDir() = %q, want %q", got, tmp)
}
var buf bytes.Buffer
if err := buildSeedISO("proxy-abc", "#cloud-config\n", guestAgentScript, &buf); err != nil {
t.Fatalf("buildSeedISO: %v", err)
}
if buf.Len() == 0 {
t.Fatal("seed ISO is empty")
}
entries, err := os.ReadDir(tmp)
if err != nil {
t.Fatal(err)
}
if len(entries) != 0 {
names := make([]string, 0, len(entries))
for _, e := range entries {
names = append(names, e.Name())
}
t.Errorf("staging left behind in TMPDIR: %v", names)
}
}
func TestSeedISO_emptyVendorDataStillHasFile(t *testing.T) {
t.Parallel()
files, _ := readSeed(t, "proxy-abc", "#cloud-config\n", "")
got, ok := files["vendor-data"]
if !ok {
t.Fatalf("seed lacks vendor-data; files: %v", keys(files))
}
if got != "" {
t.Errorf("vendor-data = %q, want empty", got)
}
}
func TestSeedISO_rejectsEmptyName(t *testing.T) {
t.Parallel()
var buf bytes.Buffer
if err := buildSeedISO("", "#cloud-config\n", "", &buf); err == nil {
t.Fatal("buildSeedISO with empty name succeeded")
}
if buf.Len() != 0 {
t.Errorf("wrote %d bytes alongside the error", buf.Len())
}
}
func TestVendorData_isShellScriptNotCloudConfig(t *testing.T) {
t.Parallel()
got, err := buildVendorData("", true)
if err != nil {
t.Fatal(err)
}
if !strings.HasPrefix(got, "#!/bin/sh\n") {
t.Errorf("vendor-data starts with %q, want #!/bin/sh", firstLine(got))
}
if strings.HasPrefix(got, "#cloud-config") {
t.Error("vendor-data is a cloud-config; cloud-init would merge it away")
}
for _, want := range []string{"qemu-guest-agent", "set -e", "apt-get", "dnf", "yum", "zypper", "apk", "systemctl enable --now qemu-guest-agent", "rc-update", "installGuestAgent: false", "DEBIAN_FRONTEND=noninteractive"} {
if !strings.Contains(got, want) {
t.Errorf("script lacks %q", want)
}
}
if strings.Contains(got, "MIME-Version") {
t.Error("script alone should not be wrapped in a multipart")
}
}
func firstLine(s string) string {
line, _, _ := strings.Cut(s, "\n")
return line
}
func bufioReader(s string) *bufio.Reader {
return bufio.NewReader(strings.NewReader(s))
}
func TestVendorData_multipartWhenExtraProvided(t *testing.T) {
t.Parallel()
configured := "#cloud-config\nntp:\n enabled: true\n"
got, err := buildVendorData(configured, true)
if err != nil {
t.Fatal(err)
}
if !strings.HasPrefix(got, "MIME-Version: 1.0\r\n") {
t.Errorf("vendor-data starts with %q, want MIME-Version: 1.0", firstLine(got))
}
r := textproto.NewReader(bufioReader(got))
hdr, err := r.ReadMIMEHeader()
if err != nil {
t.Fatalf("reading top-level headers: %v", err)
}
mediaType, params, err := mime.ParseMediaType(hdr.Get("Content-Type"))
if err != nil {
t.Fatalf("parsing Content-Type %q: %v", hdr.Get("Content-Type"), err)
}
if mediaType != "multipart/mixed" {
t.Errorf("media type = %q, want multipart/mixed", mediaType)
}
mr := multipart.NewReader(r.R, params["boundary"])
type part struct{ contentType, body string }
var parts []part
for {
p, err := mr.NextPart()
if err == io.EOF {
break
}
if err != nil {
t.Fatalf("NextPart: %v", err)
}
body, err := io.ReadAll(p)
if err != nil {
t.Fatalf("reading part: %v", err)
}
parts = append(parts, part{p.Header.Get("Content-Type"), string(body)})
}
want := []part{{"text/plain", configured}, {"text/x-shellscript", guestAgentScript}}
if len(parts) != len(want) {
t.Fatalf("got %d parts, want %d", len(parts), len(want))
}
for i := range want {
if parts[i] != want[i] {
t.Errorf("part %d = {%q, %q}, want {%q, %q}", i, parts[i].contentType, firstLine(parts[i].body), want[i].contentType, firstLine(want[i].body))
}
}
}
func TestVendorData_offWithoutConfiguredIsEmpty(t *testing.T) {
t.Parallel()
got, err := buildVendorData("", false)
if err != nil {
t.Fatal(err)
}
if got != "" {
t.Errorf("vendor-data = %q, want empty", got)
}
}
func TestVendorData_offWithConfiguredPassesThrough(t *testing.T) {
t.Parallel()
configured := "#cloud-config\nntp:\n enabled: true"
got, err := buildVendorData(configured, false)
if err != nil {
t.Fatal(err)
}
if got != configured {
t.Errorf("vendor-data = %q, want the configured document unchanged", got)
}
}
internal/provider/kvm2/ssh.go
+218
View File
@@ -0,0 +1,218 @@
package kvm2
import (
"context"
"errors"
"fmt"
"io/fs"
"net"
"os"
"sync"
"time"
"golang.org/x/crypto/ssh"
"golang.org/x/crypto/ssh/knownhosts"
"gitea.home.hrajfrisbee.cz/kacerr/egress-proxies-operator/internal/provider"
)
// dialStage names the step of a dial that failed, so the Checker (spec
// entry 16) can pick a reason without parsing messages.
type dialStage string
const (
// stageCredentials: the key or known_hosts file is missing or unparsable.
stageCredentials dialStage = "credentials"
// stageTCP: the TCP connection to host.Address failed.
stageTCP dialStage = "tcp"
// stageSSH: the SSH handshake failed, including host key and auth.
stageSSH dialStage = "ssh"
// stageSocket: the forwarded channel to the libvirt socket was refused.
stageSocket dialStage = "socket"
)
// dialError tags a dial failure with its stage and host. Unwrap keeps the
// cause visible, so classify still sees *knownhosts.KeyError, fs.ErrNotExist,
// the "unable to authenticate" text and *net.OpError.
type dialError struct {
stage dialStage
host string
err error
}
func (e *dialError) Error() string {
return fmt.Sprintf("dialing host %s (%s): %v", e.host, e.stage, e.err)
}
func (e *dialError) Unwrap() error { return e.err }
// stageOf returns the failed stage of a dial error, if err is one.
func stageOf(err error) (dialStage, bool) {
var de *dialError
if !errors.As(err, &de) {
return "", false
}
return de.stage, true
}
const (
defaultKeepaliveInterval = 15 * time.Second
keepaliveRequest = "keepalive@openssh.com"
)
// credentials is what one dial needs from the mounted files. They are read
// on every dial and never cached (spec entry 14), so rotating the Secret
// takes effect on the next connection.
type credentials struct {
signer ssh.Signer
hostKey ssh.HostKeyCallback
}
// loadCredentials reads and parses the key and known_hosts files. A missing
// file keeps fs.ErrNotExist (Transient: the Secret has not landed); a file
// that does not parse is Permanent. Messages name the path and carry the
// parser's own error, never the file's content.
func loadCredentials(cfg provider.KVMConfig) (credentials, error) {
keyBytes, err := os.ReadFile(cfg.SSHKeyFile)
if err != nil {
return credentials{}, fmt.Errorf("reading ssh key file %s: %w", cfg.SSHKeyFile, err)
}
signer, err := ssh.ParsePrivateKey(keyBytes)
if err != nil {
return credentials{}, permanentf("parsing ssh key file %s: %v", cfg.SSHKeyFile, err)
}
if cfg.InsecureIgnoreHostKey {
return credentials{signer: signer, hostKey: ssh.InsecureIgnoreHostKey()}, nil
}
hostKey, err := knownhosts.New(cfg.KnownHostsFile)
if err != nil {
var pathErr *fs.PathError
if errors.As(err, &pathErr) {
return credentials{}, fmt.Errorf("reading known_hosts file %s: %w", cfg.KnownHostsFile, err)
}
return credentials{}, permanentf("parsing known_hosts file %s: %v", cfg.KnownHostsFile, err)
}
return credentials{signer: signer, hostKey: hostKey}, nil
}
// sshSocket is the libvirt socket forwarded through one SSH connection. The
// embedded Conn is the channel; Close closes the SSH client, whose TCP
// connection carries it, and that alone unblocks the channel's readers and
// writers. Closing the channel as well would be worse than useless: its
// Close writes a close message under the channel's write lock, so it would
// queue behind a transfer stuck in a TCP write, which is the case Close
// exists to survive.
type sshSocket struct {
net.Conn
client *ssh.Client
closeOnce sync.Once
closeErr error
}
func (s *sshSocket) Close() error {
s.closeOnce.Do(func() { s.closeErr = s.client.Close() })
return s.closeErr
}
// keepalive sends keepalive@openssh.com every interval until stop is closed
// or a request fails or takes longer than the interval, and returns why it
// stopped (nil for stop). The caller marks its link dead on an error.
func (s *sshSocket) keepalive(interval time.Duration, stop <-chan struct{}) error {
ticker := time.NewTicker(interval)
defer ticker.Stop()
for {
select {
case <-stop:
return nil
case <-ticker.C:
}
done := make(chan error, 1)
go func() {
_, _, err := s.client.SendRequest(keepaliveRequest, true, nil)
done <- err
}()
timer := time.NewTimer(interval)
select {
case err := <-done:
timer.Stop()
if err != nil {
return fmt.Errorf("keepalive: %w", err)
}
case <-timer.C:
return fmt.Errorf("keepalive: no reply within %s", interval)
case <-stop:
timer.Stop()
return nil
}
}
}
// dialSSHSocket opens host's libvirt socket through SSH: credentials, TCP,
// handshake, channel. timeout bounds the whole sequence on top of ctx. Each
// failure is a *dialError naming its stage. Nothing here logs.
func dialSSHSocket(ctx context.Context, host provider.KVMHost, cfg provider.KVMConfig, timeout time.Duration) (*sshSocket, error) {
if timeout > 0 {
var cancel context.CancelFunc
ctx, cancel = context.WithTimeout(ctx, timeout)
defer cancel()
}
creds, err := loadCredentials(cfg)
if err != nil {
return nil, &dialError{stage: stageCredentials, host: host.Name, err: err}
}
var nd net.Dialer
tcp, err := nd.DialContext(ctx, "tcp", host.Address)
if err != nil {
return nil, &dialError{stage: stageTCP, host: host.Name, err: err}
}
clientCfg := &ssh.ClientConfig{
User: host.User,
Auth: []ssh.AuthMethod{ssh.PublicKeys(creds.signer)},
HostKeyCallback: creds.hostKey,
Timeout: timeout,
}
client, err := await(ctx, tcp.Close, func() (*ssh.Client, error) {
c, chans, reqs, err := ssh.NewClientConn(tcp, host.Address, clientCfg)
if err != nil {
return nil, err
}
return ssh.NewClient(c, chans, reqs), nil
})
if err != nil {
_ = tcp.Close()
return nil, &dialError{stage: stageSSH, host: host.Name, err: err}
}
conn, err := await(ctx, func() error { return client.Close() }, func() (net.Conn, error) {
return client.Dial("unix", host.Socket)
})
if err != nil {
_ = client.Close()
return nil, &dialError{stage: stageSocket, host: host.Name, err: fmt.Errorf("opening %s: %w", host.Socket, err)}
}
return &sshSocket{Conn: conn, client: client}, nil
}
// await runs a blocking step in a goroutine. When ctx ends first it calls
// abort, which closes the transport the step is blocked on, waits for the
// step to give up, and returns ctx's error instead.
func await[T any](ctx context.Context, abort func() error, step func() (T, error)) (T, error) {
type result struct {
v T
err error
}
done := make(chan result, 1)
go func() {
v, err := step()
done <- result{v, err}
}()
select {
case r := <-done:
return r.v, r.err
case <-ctx.Done():
_ = abort()
<-done
var zero T
return zero, ctx.Err()
}
}
internal/provider/kvm2/ssh_test.go
+343
View File
@@ -0,0 +1,343 @@
package kvm2
import (
"context"
"errors"
"io/fs"
"net"
"os"
"strings"
"testing"
"time"
"golang.org/x/crypto/ssh/knownhosts"
"gitea.home.hrajfrisbee.cz/kacerr/egress-proxies-operator/internal/provider"
)
// sshFixture is one in-process server with a client key it accepts and a
// known_hosts file naming its host key.
type sshFixture struct {
server *testSSHServer
socket string
cfg provider.KVMConfig
host provider.KVMHost
}
func newSSHFixture(t *testing.T) *sshFixture {
t.Helper()
dir := t.TempDir()
socket := startEchoSocket(t)
keyPath, pub := writeKeyFile(t, dir, "id_ed25519")
server := startSSHServer(t, pub, socket)
return &sshFixture{
server: server,
socket: socket,
cfg: provider.KVMConfig{
SSHKeyFile: keyPath,
KnownHostsFile: writeKnownHosts(t, dir, "known_hosts", server.addr, server.hostKey),
},
host: provider.KVMHost{Name: "beelink", Address: server.addr, User: "virt", Socket: socket},
}
}
const dialTestTimeout = 5 * time.Second
func TestDial_forwardsUnixSocket(t *testing.T) {
t.Parallel()
fx := newSSHFixture(t)
for round := range 2 {
sock, err := dialSSHSocket(context.Background(), fx.host, fx.cfg, dialTestTimeout)
if err != nil {
t.Fatalf("round %d: dial error = %v", round, err)
}
msg := []byte("ping over libvirt-sock")
if _, err := sock.Write(msg); err != nil {
t.Fatalf("round %d: write: %v", round, err)
}
buf := make([]byte, len(msg))
_ = sock.SetReadDeadline(time.Now().Add(dialTestTimeout))
if _, err := readFull(sock, buf); err != nil {
t.Fatalf("round %d: read: %v", round, err)
}
if string(buf) != string(msg) {
t.Errorf("round %d: echoed %q, want %q", round, buf, msg)
}
if err := sock.Close(); err != nil {
t.Errorf("round %d: close error = %v", round, err)
}
if err := sock.Close(); err != nil {
t.Errorf("round %d: second close error = %v, want nil", round, err)
}
}
for _, p := range fx.server.requestedPaths() {
if p != fx.socket {
t.Errorf("server saw channel for %q, want %q", p, fx.socket)
}
}
}
func readFull(sock *sshSocket, buf []byte) (int, error) {
n := 0
for n < len(buf) {
m, err := sock.Read(buf[n:])
n += m
if err != nil {
return n, err
}
}
return n, nil
}
func TestDial_unknownHostKeyRejected(t *testing.T) {
t.Parallel()
fx := newSSHFixture(t)
fx.cfg.KnownHostsFile = writeKnownHosts(t, t.TempDir(), "known_hosts", "other.example:22", fx.server.hostKey)
_, err := dialSSHSocket(context.Background(), fx.host, fx.cfg, dialTestTimeout)
assertDialFailure(t, err, stageSSH, provider.ErrPermanent)
var keyErr *knownhosts.KeyError
if !errors.As(err, &keyErr) {
t.Fatalf("error = %v, want *knownhosts.KeyError", err)
}
if len(keyErr.Want) != 0 {
t.Errorf("KeyError.Want = %v, want empty (host unknown)", keyErr.Want)
}
}
func TestDial_mismatchedHostKeyRejected(t *testing.T) {
t.Parallel()
fx := newSSHFixture(t)
fx.cfg.KnownHostsFile = writeKnownHosts(t, t.TempDir(), "known_hosts", fx.server.addr, otherPublicKey(t))
_, err := dialSSHSocket(context.Background(), fx.host, fx.cfg, dialTestTimeout)
assertDialFailure(t, err, stageSSH, provider.ErrPermanent)
var keyErr *knownhosts.KeyError
if !errors.As(err, &keyErr) {
t.Fatalf("error = %v, want *knownhosts.KeyError", err)
}
if len(keyErr.Want) == 0 {
t.Error("KeyError.Want empty, want the mismatching known key")
}
}
func TestDial_insecureIgnoreHostKeyAccepts(t *testing.T) {
t.Parallel()
fx := newSSHFixture(t)
fx.cfg.KnownHostsFile = ""
fx.cfg.InsecureIgnoreHostKey = true
sock, err := dialSSHSocket(context.Background(), fx.host, fx.cfg, dialTestTimeout)
if err != nil {
t.Fatalf("dial error = %v, want nil with host keys ignored", err)
}
_ = sock.Close()
}
func TestDial_missingKeyFileIsTransient(t *testing.T) {
t.Parallel()
fx := newSSHFixture(t)
fx.cfg.SSHKeyFile = fx.cfg.SSHKeyFile + ".missing"
_, err := dialSSHSocket(context.Background(), fx.host, fx.cfg, dialTestTimeout)
assertDialFailure(t, err, stageCredentials, provider.ErrTransient)
if !errors.Is(err, fs.ErrNotExist) {
t.Errorf("error = %v, want fs.ErrNotExist", err)
}
if !strings.Contains(err.Error(), fx.cfg.SSHKeyFile) {
t.Errorf("error = %q, want the path in it", err)
}
}
func TestDial_missingKnownHostsIsTransient(t *testing.T) {
t.Parallel()
fx := newSSHFixture(t)
fx.cfg.KnownHostsFile = fx.cfg.KnownHostsFile + ".missing"
_, err := dialSSHSocket(context.Background(), fx.host, fx.cfg, dialTestTimeout)
assertDialFailure(t, err, stageCredentials, provider.ErrTransient)
if !errors.Is(err, fs.ErrNotExist) {
t.Errorf("error = %v, want fs.ErrNotExist", err)
}
}
func TestDial_unparsableKeyIsPermanent(t *testing.T) {
t.Parallel()
fx := newSSHFixture(t)
fx.cfg.SSHKeyFile = writeFile(t, t.TempDir(), "id_bad", "this is not a key\n")
_, err := dialSSHSocket(context.Background(), fx.host, fx.cfg, dialTestTimeout)
assertDialFailure(t, err, stageCredentials, provider.ErrPermanent)
if !strings.Contains(err.Error(), "parsing ssh key file "+fx.cfg.SSHKeyFile) {
t.Errorf("error = %q, want 'parsing ssh key file <path>'", err)
}
}
func TestDial_unparsableKnownHostsIsPermanent(t *testing.T) {
t.Parallel()
fx := newSSHFixture(t)
fx.cfg.KnownHostsFile = writeFile(t, t.TempDir(), "known_hosts", "host.example ssh-ed25519 not-base64!\n")
_, err := dialSSHSocket(context.Background(), fx.host, fx.cfg, dialTestTimeout)
assertDialFailure(t, err, stageCredentials, provider.ErrPermanent)
if !strings.Contains(err.Error(), "parsing known_hosts file "+fx.cfg.KnownHostsFile) {
t.Errorf("error = %q, want 'parsing known_hosts file <path>'", err)
}
}
func TestDial_wrongKeyIsUnauthorized(t *testing.T) {
t.Parallel()
fx := newSSHFixture(t)
fx.cfg.SSHKeyFile, _ = writeKeyFile(t, t.TempDir(), "id_other")
_, err := dialSSHSocket(context.Background(), fx.host, fx.cfg, dialTestTimeout)
assertDialFailure(t, err, stageSSH, provider.ErrPermanent)
if !strings.Contains(err.Error(), sshAuthFailed) {
t.Errorf("error = %q, want %q in it", err, sshAuthFailed)
}
}
func TestDial_handshakeTimeout(t *testing.T) {
t.Parallel()
fx := newSSHFixture(t)
fx.host.Address = silentListener(t)
const budget = 200 * time.Millisecond
start := time.Now()
_, err := dialSSHSocket(context.Background(), fx.host, fx.cfg, budget)
elapsed := time.Since(start)
assertDialFailure(t, err, stageSSH, provider.ErrTransient)
if !errors.Is(err, context.DeadlineExceeded) {
t.Errorf("error = %v, want context.DeadlineExceeded", err)
}
if elapsed > budget+time.Second {
t.Errorf("dial took %v, want about the %v budget", elapsed, budget)
}
}
func TestDial_cancelledContext(t *testing.T) {
t.Parallel()
fx := newSSHFixture(t)
fx.host.Address = silentListener(t)
ctx, cancel := context.WithCancel(context.Background())
go func() {
time.Sleep(20 * time.Millisecond)
cancel()
}()
start := time.Now()
_, err := dialSSHSocket(ctx, fx.host, fx.cfg, dialTestTimeout)
if time.Since(start) > time.Second {
t.Errorf("dial took %v after cancel", time.Since(start))
}
assertDialFailure(t, err, stageSSH, provider.ErrTransient)
if !errors.Is(err, context.Canceled) {
t.Errorf("error = %v, want context.Canceled", err)
}
}
func TestDial_refusedIsTransient(t *testing.T) {
t.Parallel()
fx := newSSHFixture(t)
fx.host.Address = closedPort(t)
_, err := dialSSHSocket(context.Background(), fx.host, fx.cfg, dialTestTimeout)
assertDialFailure(t, err, stageTCP, provider.ErrTransient)
var opErr *net.OpError
if !errors.As(err, &opErr) {
t.Errorf("error = %v, want *net.OpError", err)
}
}
func TestDial_socketRefusedIsSocketStage(t *testing.T) {
t.Parallel()
fx := newSSHFixture(t)
fx.host.Socket = "/run/nonexistent/libvirt-sock"
_, err := dialSSHSocket(context.Background(), fx.host, fx.cfg, dialTestTimeout)
assertDialFailure(t, err, stageSocket, provider.ErrTransient)
if !strings.Contains(err.Error(), fx.host.Socket) {
t.Errorf("error = %q, want the socket path in it", err)
}
}
func TestSSH_keyMaterialNeverInErrorsOrLogs(t *testing.T) {
t.Parallel()
const keySentinel = "SENTINEL-KEY-4f9c2b"
const hostsSentinel = "SENTINEL-HOSTS-8a1d7e"
fx := newSSHFixture(t)
dir := t.TempDir()
tests := []struct {
name string
cfg func(c *provider.KVMConfig)
}{
{
name: "bad key file",
cfg: func(c *provider.KVMConfig) {
c.SSHKeyFile = writeFile(t, dir, "id_bad", "-----BEGIN NOT A KEY-----\n"+keySentinel+"\n-----END NOT A KEY-----\n")
},
},
{
name: "bad known_hosts file",
cfg: func(c *provider.KVMConfig) {
c.KnownHostsFile = writeFile(t, dir, "known_hosts_bad", "beelink ssh-ed25519 "+hostsSentinel+"\n")
},
},
{
name: "good key, host unknown",
cfg: func(c *provider.KVMConfig) {
c.KnownHostsFile = writeFile(t, dir, "known_hosts_other", knownhosts.Line([]string{"other.example:22"}, fx.server.hostKey)+"\n")
},
},
}
keyBytes := readFileT(t, fx.cfg.SSHKeyFile)
for _, tc := range tests {
t.Run(tc.name, func(t *testing.T) {
t.Parallel()
cfg := fx.cfg
tc.cfg(&cfg)
_, err := dialSSHSocket(context.Background(), fx.host, cfg, dialTestTimeout)
if err == nil {
t.Fatal("dial succeeded, want an error")
}
msg := err.Error()
for _, s := range []string{keySentinel, hostsSentinel} {
if strings.Contains(msg, s) {
t.Errorf("error %q contains sentinel %q", msg, s)
}
}
if strings.Contains(msg, keyBytes) || strings.Contains(msg, "PRIVATE KEY") {
t.Errorf("error %q contains key material", msg)
}
})
}
}
func readFileT(t *testing.T, path string) string {
t.Helper()
b, err := os.ReadFile(path)
if err != nil {
t.Fatalf("reading %s: %v", path, err)
}
return strings.TrimSpace(string(b))
}
func assertDialFailure(t *testing.T, err error, wantStage dialStage, wantClass error) {
t.Helper()
if err == nil {
t.Fatal("dial succeeded, want an error")
}
stage, ok := stageOf(err)
if !ok {
t.Fatalf("error %v is not a *dialError", err)
}
if stage != wantStage {
t.Errorf("stage = %q, want %q (error: %v)", stage, wantStage, err)
}
if got := classify("get", err); !errors.Is(got, wantClass) {
t.Errorf("classify() = %v, want %v (error: %v)", got, wantClass, err)
}
}
func TestDialError_unwrapsToCause(t *testing.T) {
t.Parallel()
cause := &knownhosts.KeyError{}
err := &dialError{stage: stageSSH, host: "beelink", err: cause}
if !errors.Is(err, cause) {
t.Error("errors.Is(dialError, cause) = false")
}
if !strings.Contains(err.Error(), "beelink") || !strings.Contains(err.Error(), "(ssh)") {
t.Errorf("Error() = %q, want the host and the stage", err)
}
if _, ok := stageOf(errors.New("plain")); ok {
t.Error("stageOf(plain error) reported a stage")
}
}
internal/provider/kvm2/sshserver_test.go
+289
View File
@@ -0,0 +1,289 @@
package kvm2
import (
"bytes"
"crypto/ed25519"
"crypto/rand"
"encoding/pem"
"errors"
"io"
"net"
"os"
"path/filepath"
"sync"
"testing"
"golang.org/x/crypto/ssh"
"golang.org/x/crypto/ssh/knownhosts"
)
// testSSHServer is an in-process x/crypto/ssh server that accepts one
// authorised public key and forwards direct-streamlocal@openssh.com channels
// for socketPath to that unix socket, rejecting every other channel.
type testSSHServer struct {
addr string
hostKey ssh.PublicKey
socketPath string
listener net.Listener
mu sync.Mutex
conns []net.Conn
wg sync.WaitGroup
stopOnce sync.Once
reqMu sync.Mutex
requested []string
}
func startSSHServer(t *testing.T, authorized ssh.PublicKey, socketPath string) *testSSHServer {
t.Helper()
_, hostPriv, err := ed25519.GenerateKey(rand.Reader)
if err != nil {
t.Fatalf("generating host key: %v", err)
}
hostSigner, err := ssh.NewSignerFromKey(hostPriv)
if err != nil {
t.Fatalf("host signer: %v", err)
}
cfg := &ssh.ServerConfig{
PublicKeyCallback: func(_ ssh.ConnMetadata, key ssh.PublicKey) (*ssh.Permissions, error) {
if bytes.Equal(key.Marshal(), authorized.Marshal()) {
return &ssh.Permissions{}, nil
}
return nil, errors.New("key not authorised")
},
}
cfg.AddHostKey(hostSigner)
ln, err := net.Listen("tcp", "127.0.0.1:0")
if err != nil {
t.Fatalf("listen: %v", err)
}
s := &testSSHServer{addr: ln.Addr().String(), hostKey: hostSigner.PublicKey(), socketPath: socketPath, listener: ln}
s.wg.Add(1)
go s.serve(cfg)
t.Cleanup(s.stop)
return s
}
func (s *testSSHServer) serve(cfg *ssh.ServerConfig) {
defer s.wg.Done()
for {
conn, err := s.listener.Accept()
if err != nil {
return
}
s.mu.Lock()
s.conns = append(s.conns, conn)
s.mu.Unlock()
s.wg.Add(1)
go s.handle(conn, cfg)
}
}
func (s *testSSHServer) handle(conn net.Conn, cfg *ssh.ServerConfig) {
defer s.wg.Done()
sc, chans, reqs, err := ssh.NewServerConn(conn, cfg)
if err != nil {
_ = conn.Close()
return
}
defer sc.Close()
go ssh.DiscardRequests(reqs)
for nc := range chans {
if nc.ChannelType() != "direct-streamlocal@openssh.com" {
_ = nc.Reject(ssh.UnknownChannelType, "only direct-streamlocal@openssh.com is served")
continue
}
var open struct {
SocketPath string
Reserved0 string
Reserved1 uint32
}
if err := ssh.Unmarshal(nc.ExtraData(), &open); err != nil {
_ = nc.Reject(ssh.ConnectionFailed, "bad direct-streamlocal payload")
continue
}
s.reqMu.Lock()
s.requested = append(s.requested, open.SocketPath)
s.reqMu.Unlock()
if open.SocketPath != s.socketPath {
_ = nc.Reject(ssh.ConnectionFailed, "no such socket")
continue
}
target, err := net.Dial("unix", s.socketPath)
if err != nil {
_ = nc.Reject(ssh.ConnectionFailed, "socket unavailable")
continue
}
ch, chReqs, err := nc.Accept()
if err != nil {
_ = target.Close()
continue
}
go ssh.DiscardRequests(chReqs)
s.wg.Add(1)
go func() {
defer s.wg.Done()
proxy(ch, target)
}()
}
}
func proxy(ch ssh.Channel, target net.Conn) {
var wg sync.WaitGroup
wg.Add(2)
go func() {
defer wg.Done()
_, _ = io.Copy(target, ch)
_ = target.Close()
}()
go func() {
defer wg.Done()
_, _ = io.Copy(ch, target)
_ = ch.Close()
}()
wg.Wait()
}
func (s *testSSHServer) stop() {
s.stopOnce.Do(func() {
_ = s.listener.Close()
s.mu.Lock()
for _, c := range s.conns {
_ = c.Close()
}
s.mu.Unlock()
s.wg.Wait()
})
}
func (s *testSSHServer) requestedPaths() []string {
s.reqMu.Lock()
defer s.reqMu.Unlock()
return append([]string(nil), s.requested...)
}
// startEchoSocket listens on a unix socket that echoes what it reads. The
// path is kept short: unix socket paths are limited to about 100 bytes and
// t.TempDir() names are long.
func startEchoSocket(t *testing.T) string {
t.Helper()
dir, err := os.MkdirTemp("", "kvm2")
if err != nil {
t.Fatalf("temp dir: %v", err)
}
t.Cleanup(func() { _ = os.RemoveAll(dir) })
path := filepath.Join(dir, "libvirt-sock")
ln, err := net.Listen("unix", path)
if err != nil {
t.Fatalf("listen unix: %v", err)
}
t.Cleanup(func() { _ = ln.Close() })
go func() {
for {
c, err := ln.Accept()
if err != nil {
return
}
go func() {
_, _ = io.Copy(c, c)
_ = c.Close()
}()
}
}()
return path
}
// silentListener accepts TCP connections and never speaks, for handshake
// timeouts. It returns the address.
func silentListener(t *testing.T) string {
t.Helper()
ln, err := net.Listen("tcp", "127.0.0.1:0")
if err != nil {
t.Fatalf("listen: %v", err)
}
var conns []net.Conn
var mu sync.Mutex
go func() {
for {
c, err := ln.Accept()
if err != nil {
return
}
mu.Lock()
conns = append(conns, c)
mu.Unlock()
}
}()
t.Cleanup(func() {
_ = ln.Close()
mu.Lock()
defer mu.Unlock()
for _, c := range conns {
_ = c.Close()
}
})
return ln.Addr().String()
}
// closedPort returns an address nothing listens on.
func closedPort(t *testing.T) string {
t.Helper()
ln, err := net.Listen("tcp", "127.0.0.1:0")
if err != nil {
t.Fatalf("listen: %v", err)
}
addr := ln.Addr().String()
_ = ln.Close()
return addr
}
// writeKeyFile generates an ed25519 key, writes it in OpenSSH format and
// returns the path and the public half.
func writeKeyFile(t *testing.T, dir, name string) (string, ssh.PublicKey) {
t.Helper()
pub, priv, err := ed25519.GenerateKey(rand.Reader)
if err != nil {
t.Fatalf("generating key: %v", err)
}
block, err := ssh.MarshalPrivateKey(priv, "kvm2 test")
if err != nil {
t.Fatalf("marshalling key: %v", err)
}
path := writeFile(t, dir, name, string(pem.EncodeToMemory(block)))
sshPub, err := ssh.NewPublicKey(pub)
if err != nil {
t.Fatalf("public key: %v", err)
}
return path, sshPub
}
// writeKnownHosts writes one known_hosts line for addr and key.
func writeKnownHosts(t *testing.T, dir, name, addr string, key ssh.PublicKey) string {
t.Helper()
return writeFile(t, dir, name, knownhosts.Line([]string{addr}, key)+"\n")
}
func writeFile(t *testing.T, dir, name, content string) string {
t.Helper()
path := filepath.Join(dir, name)
if err := os.WriteFile(path, []byte(content), 0o600); err != nil {
t.Fatalf("writing %s: %v", path, err)
}
return path
}
// otherPublicKey returns a public key unrelated to any file, for mismatch
// cases.
func otherPublicKey(t *testing.T) ssh.PublicKey {
t.Helper()
pub, _, err := ed25519.GenerateKey(rand.Reader)
if err != nil {
t.Fatalf("generating key: %v", err)
}
sshPub, err := ssh.NewPublicKey(pub)
if err != nil {
t.Fatalf("public key: %v", err)
}
return sshPub
}
internal/provider/kvm2/state.go
+49
View File
@@ -0,0 +1,49 @@
package kvm2
import (
"net/netip"
"github.com/digitalocean/go-libvirt"
"gitea.home.hrajfrisbee.cz/kacerr/egress-proxies-operator/internal/provider"
)
// mapState turns a libvirt domain state plus the discovered address into the
// contract's state. nostate means libvirt could not tell, so the next poll
// decides; an active domain is Running only once it has an address; every
// inactive state, and anything unknown, is Stopped, which the reconciler
// answers by replacing the instance.
func mapState(s libvirt.DomainState, ip string) provider.InstanceState {
switch {
case s == libvirt.DomainNostate:
return provider.StateProvisioning
case isActive(s) && ip == "":
return provider.StateProvisioning
case isActive(s):
return provider.StateRunning
default:
return provider.StateStopped
}
}
func isActive(s libvirt.DomainState) bool {
return s == libvirt.DomainRunning || s == libvirt.DomainBlocked
}
// firstIPv4 picks the first usable IPv4 address from the guest's interfaces:
// not loopback, not link-local, not unspecified. Empty means no address yet.
func firstIPv4(ifaces []libvirt.DomainInterface) string {
for _, iface := range ifaces {
for _, a := range iface.Addrs {
if libvirt.IPAddrType(a.Type) != libvirt.IPAddrTypeIpv4 {
continue
}
addr, err := netip.ParseAddr(a.Addr)
if err != nil || !addr.Is4() || addr.IsLoopback() || addr.IsLinkLocalUnicast() || addr.IsUnspecified() {
continue
}
return addr.String()
}
}
return ""
}
internal/provider/kvm2/state_test.go
+92
View File
@@ -0,0 +1,92 @@
package kvm2
import (
"testing"
"github.com/digitalocean/go-libvirt"
"gitea.home.hrajfrisbee.cz/kacerr/egress-proxies-operator/internal/provider"
)
func TestMapState_table(t *testing.T) {
t.Parallel()
tests := []struct {
state libvirt.DomainState
noIP provider.InstanceState
withIP provider.InstanceState
display string
}{
{libvirt.DomainNostate, provider.StateProvisioning, provider.StateProvisioning, "nostate"},
{libvirt.DomainRunning, provider.StateProvisioning, provider.StateRunning, "running"},
{libvirt.DomainBlocked, provider.StateProvisioning, provider.StateRunning, "blocked"},
{libvirt.DomainPaused, provider.StateStopped, provider.StateStopped, "paused"},
{libvirt.DomainShutdown, provider.StateStopped, provider.StateStopped, "shutdown"},
{libvirt.DomainShutoff, provider.StateStopped, provider.StateStopped, "shutoff"},
{libvirt.DomainCrashed, provider.StateStopped, provider.StateStopped, "crashed"},
{libvirt.DomainPmsuspended, provider.StateStopped, provider.StateStopped, "pmsuspended"},
{libvirt.DomainState(99), provider.StateStopped, provider.StateStopped, "unknown code 99"},
}
for _, tc := range tests {
t.Run(tc.display, func(t *testing.T) {
t.Parallel()
if got := mapState(tc.state, ""); got != tc.noIP {
t.Errorf("mapState(%s, \"\") = %s, want %s", tc.display, got, tc.noIP)
}
if got := mapState(tc.state, "10.0.0.5"); got != tc.withIP {
t.Errorf("mapState(%s, ip) = %s, want %s", tc.display, got, tc.withIP)
}
})
}
}
func TestIsActive(t *testing.T) {
t.Parallel()
for s, want := range map[libvirt.DomainState]bool{
libvirt.DomainRunning: true, libvirt.DomainBlocked: true,
libvirt.DomainNostate: false, libvirt.DomainPaused: false, libvirt.DomainShutoff: false, libvirt.DomainState(99): false,
} {
if got := isActive(s); got != want {
t.Errorf("isActive(%d) = %v, want %v", s, got, want)
}
}
}
func addr(typ libvirt.IPAddrType, a string) libvirt.DomainIPAddr {
return libvirt.DomainIPAddr{Type: int32(typ), Addr: a}
}
func TestFirstIPv4_filters(t *testing.T) {
t.Parallel()
v4 := libvirt.IPAddrTypeIpv4
v6 := libvirt.IPAddrTypeIpv6
tests := []struct {
name string
ifaces []libvirt.DomainInterface
want string
}{
{name: "empty list", ifaces: nil, want: ""},
{name: "loopback only", ifaces: []libvirt.DomainInterface{{Name: "lo", Addrs: []libvirt.DomainIPAddr{addr(v4, "127.0.0.1")}}}, want: ""},
{name: "link-local only", ifaces: []libvirt.DomainInterface{{Name: "eth0", Addrs: []libvirt.DomainIPAddr{addr(v4, "169.254.10.1")}}}, want: ""},
{name: "ipv6 only", ifaces: []libvirt.DomainInterface{{Name: "eth0", Addrs: []libvirt.DomainIPAddr{addr(v6, "2001:db8::1"), addr(v6, "fe80::1")}}}, want: ""},
{name: "unspecified", ifaces: []libvirt.DomainInterface{{Name: "eth0", Addrs: []libvirt.DomainIPAddr{addr(v4, "0.0.0.0")}}}, want: ""},
{name: "unparsable", ifaces: []libvirt.DomainInterface{{Name: "eth0", Addrs: []libvirt.DomainIPAddr{addr(v4, "not-an-ip")}}}, want: ""},
{name: "ipv4 literal tagged as ipv6 is skipped", ifaces: []libvirt.DomainInterface{{Name: "eth0", Addrs: []libvirt.DomainIPAddr{addr(v6, "10.0.0.7")}}}, want: ""},
{name: "ipv6 literal tagged as ipv4 is skipped", ifaces: []libvirt.DomainInterface{{Name: "eth0", Addrs: []libvirt.DomainIPAddr{addr(v4, "2001:db8::1")}}}, want: ""},
{
name: "good one after bad ones across interfaces",
ifaces: []libvirt.DomainInterface{
{Name: "lo", Addrs: []libvirt.DomainIPAddr{addr(v4, "127.0.0.1")}},
{Name: "eth0", Addrs: []libvirt.DomainIPAddr{addr(v6, "fe80::5054:ff:fe12:3456"), addr(v4, "169.254.1.1"), addr(v4, "bogus"), addr(v4, "192.168.1.42"), addr(v4, "192.168.1.43")}},
},
want: "192.168.1.42",
},
}
for _, tc := range tests {
t.Run(tc.name, func(t *testing.T) {
t.Parallel()
if got := firstIPv4(tc.ifaces); got != tc.want {
t.Errorf("firstIPv4() = %q, want %q", got, tc.want)
}
})
}
}

Some files were not shown because too many files have changed in this diff Show More