kacerr/egress-proxies-operator
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

Compare commits

merge into: kacerr/egress-proxies-operator:feat/provider-blueprint
Branches Tags
kacerr/egress-proxies-operator:feat/proxy-operator kacerr/egress-proxies-operator:main kacerr/egress-proxies-operator:feat/proxmox-provider kacerr/egress-proxies-operator:feat/kvm2-provider kacerr/egress-proxies-operator:feat/provider-blueprint kacerr/egress-proxies-operator:feat/provider-health kacerr/egress-proxies-operator:feat/kvm-provider kacerr/egress-proxies-operator:feat/otel-tracing kacerr/egress-proxies-operator:feat/demo-scripts kacerr/egress-proxies-operator:feat/gitea-build-workflow
kacerr/egress-proxies-operator:0.01
...
pull from: kacerr/egress-proxies-operator:feat/proxmox-provider
Branches Tags
kacerr/egress-proxies-operator:main kacerr/egress-proxies-operator:feat/proxmox-provider kacerr/egress-proxies-operator:feat/kvm2-provider kacerr/egress-proxies-operator:feat/provider-blueprint kacerr/egress-proxies-operator:feat/provider-health kacerr/egress-proxies-operator:feat/kvm-provider kacerr/egress-proxies-operator:feat/proxy-operator kacerr/egress-proxies-operator:feat/otel-tracing kacerr/egress-proxies-operator:feat/demo-scripts kacerr/egress-proxies-operator:feat/gitea-build-workflow
kacerr/egress-proxies-operator:0.01
40 Commits
feat/provider-blueprint ... feat/proxmox-provider
Author SHA1 Message Date
kacerrandClaude Opus 5.5 3d72507894 proxmox: review fixes, real-host validation record, changelog 
Phase 10. Two review passes (go-operator-reviewer, /code-review high)
over the branch, recorded in docs/reviews/2026-09-23-proxmox-provider-review/:
9 findings fixed, 3 accepted as documented limits, 7 rejected with a
reason.

- Delete removes the seed between the stop and the destroy, so a failed
  seed delete leaves the VM for the next Delete instead of leaking the
  ISO; a Permanent refusal on the every-node path fails the call.
- Create skips a probe VMID whose listed name is someone else's without
  waiting on its lock.
- The three answers Phase 0 never captured are fixtures now: a missing
  storage is a 400 (the Checker reports StorageNotFound), a hibernated
  VM is destroyable (Delete destroys ours), a paused VM stops.
- TestProxmox_lifecycle and TestProxmoxCheck_realHost passed on pv1-1
  (PVE 9.2.20); the docs record the run and drop "not yet run".

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
 2026-09-24 10:28:28 +00:00
kacerrandClaude Opus 5.5 7f2073f94b proxmox: deployment files, sample, user guide and doc rows 
The optional proxmox-token Secret (token, ca.pem) mounted at
/etc/proxy-operator/proxmox, make proxmox-token-secret with content
checks that never print the token, a commented proxmox-home entry in
both providers configs, and config/samples/proxy_proxmox.yaml.
TestLoadConfig_proxmoxSampleEntry loads the commented entry from both
files, and a second test ties the sample Proxy's placement to it.

docs/proxmox-provider.md is the user guide: prerequisites and privileges,
cluster setup, config reference, placement, what lands on Proxmox,
logging, tracing, timeouts, health check, error classes, known gaps and
testing. providers-existing.md gains the proxmox column and section;
architecture.md, the README, providers.md and CLAUDE.md gain their rows.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
 2026-09-23 21:14:18 +00:00
kacerrandClaude Opus 5.5 6e9ef26dcb proxmox: contract and observability tests, integration test, testing docs 
The contract sweep in provider_test.go: every method and Check returns
within a second of a cancel, every wrapErr path carries the configured
name, op and id, the V0/V1/V2 logging tiers and the no-op lines, and a
Get over the real client whose HTTP spans are children of provider.get.
proxmox_integration_test.go (-tags=integration) holds TestProxmox_lifecycle
and TestProxmoxCheck_realHost; both compile and skip without
PROXMOX_TEST_*, and have not run against a host yet. make
test-integration-proxmox runs them. docs/testing.md gains the proxmox,
nocloud and keyedlock rows, the Live Proxmox gaps and the integration
section; spec entry 21's section 11 list is ticked.

Waits that could hang the package under a mutation are bounded: the
concurrent-Create loop and eight direct blocking calls in older tests.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
 2026-09-23 20:43:00 +00:00
kacerrandClaude Opus 5.5 2f89c1fffb proxmox: Delete with seed cleanup, health Checker, CRD type comment 
Delete stops and destroys a VM of ours under the per-name lock, then
deletes its seed on the VM's node, or on every node when no VM of ours
was left. A hibernated VM of ours is Permanent, as in Create. The
Checker verifies each node concurrently (version, node status, ISO
storage, templates) and adds CertificateInvalid, CredentialsUnavailable,
StorageNotFound and TemplateNotFound to the shared reasons. The
ProviderHealth type comment gains proxmox.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
 2026-09-23 20:14:14 +00:00
kacerrandClaude Opus 5.5 34f132d421 proxmox: Create by cloning a template, seed upload, config, start 
Create validates placement, then under the per-name lock adopts the VM a
previous attempt left (by name, or at a taken VMID after the ownership
check of spec entry 2) or full-clones the template onto the UID's
deterministic VMID with the mark in the clone call. It then uploads the
seed over any existing file, writes cores, memory, agent, tag, ide2 and
net0 when configured, grows the boot disk when diskGiB is larger, starts,
and waits for running/running.

waitUnlocked no longer waits on a suspended lock: a hibernated VM is
judged at once, so ours is Permanent and a foreign one is probed past.
The template is checked only on the allocation path, and its shape
(net0, boot disk size) is refused before any clone exists. Spec entries
2, 4 and 10 updated to match.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
 2026-09-23 15:32:31 +00:00
kacerrandClaude Opus 5.5 803716459a proxmox: Get, ListByTag and task polling 
Get follows spec entry 9's check order (lock and unknown status before
the name, suspended is Stopped after the name and mark checks), reads
status/current so paused is Stopped, and maps the three agent messages
of entry 7. ListByTag reads one config per candidate and fails whole
only on the resources call or an ended ctx. waitTask polls the UPID's
node every budgets.poll, logs failed tasks at V(1) and counts
"WARNINGS: <n>" as success; waitUnlocked serves Create's ownership check.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
 2026-09-23 15:11:20 +00:00
kacerrandClaude Opus 5.5 e05d994974 proxmox: thin HTTP client, seam, log decorator and in-memory fake 
httpAPI implements the sixteen seam methods over net/http: the token is
read per request, non-2xx answers become apiError from the body message or
the reason phrase, echoed secrets are redacted, the CA loads lazily, HTTP/2
is off, and uploads stream with an exact Content-Length. The observed
decorator logs one V(1) line per call and starts no span; New wires both.
fakeAPI simulates clone locks, tasks and ISOs with the captured error texts.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
 2026-09-23 14:47:50 +00:00
kacerrandClaude Opus 5.5 0a6c8624f3 provider: extract nocloud seed builder and keyedlock from kvm2 
kvm2's NoCloud seed builder moves to internal/provider/nocloud (exported,
with an ErrEmptyName sentinel instead of permanentf) and its keyed locks
to internal/provider/keyedlock, tests moved under their existing names.
kvm2 behaviour is unchanged. proxmox gains the per-name lock and the pure
seed helpers (seedISOName, seedVolID, buildSeed); unlike kvm2 (review
finding S5.1), a seed staging failure there is Transient, not Permanent.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
 2026-09-23 14:01:27 +00:00
kacerrandClaude Opus 5.5 6e996ae2c7 proxmox: config block, IDs, marks, state mapping, classify, registration 
Phase 2 of the proxmox provider plan. Adds the proxmox: config block with
its validation, registers the type in the config allow-list and the
constructor map, and starts internal/provider/proxmox: the api seam and
its data types, providerIDs (vms/<vmid>/<name>) with the UID-derived VMID,
the description mark, state mapping, and error classification.

Tests read the responses captured from pv1-1 in Phase 0 rather than
repeating Proxmox's wording, and every matcher is run against every
captured error so none fires on the wrong message. The contract methods
are Transient stubs until Phases 5 to 7.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
 2026-09-23 09:42:46 +00:00
kacerrandClaude Opus 5.5 4e974f30f6 docs: proxmox provider spec 
Phase 1 of the proxmox provider plan: docs/proxmox-provider-spec.md
answers all 24 entries of the provider spec template against the
contract, citing the Phase 0 facts from pv1-1 as evidence.

It folds in the ten corrections recon found. The two that matter most:
Create inspects a taken VMID, waiting out a clone lock, before probing
past it, so a crash-retry adopts its own half-cloned VM instead of cloning
a duplicate; and Get checks the lock before the name, because a VM being
cloned has no name yet.

The plan now opens with a note that the spec wins where they disagree.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
 2026-09-23 09:18:29 +00:00
kacerrandClaude Opus 5.5 ff94fe70bb proxmox: captured API fixtures and host facts from pv1-1 (PVE 9.2.20) 
Phase 0 of the proxmox provider plan. 55 fixtures under
internal/provider/proxmox/testdata, each the status, reason phrase and
raw body of a real response from the user's host, and the execution log
answering the plan's facts F1 to F23.

Recon ran a clone of the Debian 13 template through the whole path the
provider will take, twice: untagged and on VLAN 4. On both, the NoCloud
seed on ide2 installed the guest agent and an address came back about 50
seconds after start. It also found two bugs in the plan: a VM being
cloned has no name in /cluster/resources, so name-first lookups would read
it as gone, and a crash-retry would clone a duplicate. Those and eight
smaller corrections are listed for the spec.

Decisions made with the user: seeds go to a dedicated proxy-seeds
storage where the token can delete them; proxies land on a VLAN set in
config (vlanTag); the API URL is https://192.168.0.8:8006, the address in
the node certificate.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
 2026-09-23 08:22:41 +00:00
kacerrandClaude Opus 5 2167819157 docs: retarget the proxmox plan at Proxmox VE 9.2 
The host is 9.2.20, not 8.x. Three corrections that would have failed
at runtime:

- VM.Monitor was removed in PVE 9.0, so it is dropped from the role and
  the agent read rests on VM.GuestAgent.Audit alone.
- The GC-assist tag becomes proxy_operator: current tag validation takes
  only lowercase letters, digits and underscore, so proxy-operator would
  have been rejected on every Create. The description mark keeps its
  hyphenated keys; it is free text, not a tag.
- HTTP/2 is documented as absent from pveproxy on every release, so the
  client disabling it is a fact rather than a guess.

Two new Phase 0 facts: F22 for the 9.2 unified authentication-failure
message, which invalidates any 401 text remembered from 8.x, and F23 for
the node /tmp tmpfs move. F12 records that the upload endpoint takes
iso, vztmpl or import on 9.x and still refuses snippets, which is what
keeps the seed ISO route the only API-only path for user-data.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
 2026-09-18 15:49:14 +00:00
kacerrandClaude Fable 5.1 708989c40f docs: plan the proxmox provider 
Eleven self-contained phases (host recon and fixtures, spec, config and
skeleton, nocloud/keyedlock extraction from kvm2, HTTP client and fake,
Get/List, Create, Delete/Checker, test sweep and integration test,
deployment and docs, review and MR), with the Proxmox host setup
checklist and the backend facts Phase 0 must verify on 192.168.2.8.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
 2026-09-18 15:02:53 +00:00
kacerrandClaude Opus 5 8fa9e8aad9 Housekeeping: review docs into docs/, refreshed agent guide and settings 
Four unrelated working-tree changes, none of them touching operator
behaviour:

- review/*.md -> docs/reviews/2026-09-17-kvm2-provider-review/, byte for
  byte, so the kvm2 review sits with the rest of the docs under a name
  that says which review it was.
- AGENTS.md: replaced the generic kubebuilder scaffold guide with what is
  actually true of this repo — the real component layout, the lint/test
  commands that work here, and the behavioural gotchas (replace-on-change
  hashing, the cloud-init Secret label, in-memory leases).
- .claude/settings.json: the allow list still carried /Users/jan.novak
  paths from the macOS checkout; re-pointed at /home/novakj and collapsed
  the one-off entries into prefixes.
- config/samples/proxy_kvm2_crawl_only_one.yaml: a one-VM cut of the crawl
  pool, the rest of the members left commented out.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
 2026-09-18 08:12:27 +00:00
kacerrandClaude Opus 5 1fa3e55b9c docs: start docs/operating.md with the libvirt-RPC log query 
A place for helpers against a running operator, as opposed to the design
docs. First entry is the jq filter for timestamp + rpc: both libvirt
providers log the same "libvirt call" msg, so one select covers kvm and
kvm2, and a fromjson? variant survives a stream with non-JSON lines in it.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
 2026-09-18 07:53:12 +00:00
kacerrandClaude Opus 5 c3a581af95 samples: a kvm2 crawl pool, not just the single-proxy sample 
proxy_kvm2.yaml shows one VM; nothing showed the shape a crawler actually
leases from. Add a five-Proxy pool on the kvm2-home entry: four share one
labelled cloud-init Secret, and the fifth carries its own inline block
because it runs squid on 8080 and spec.port has to match the http_port a
shared Secret would pin.

It stays out of the samples kustomization so `apply -k config/samples`
doesn't boot a pool along with the one-per-provider samples. All six
objects validate against the installed CRD under --dry-run=server.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
 2026-09-18 07:45:16 +00:00
kacerr 5928c3fe75 Merge pull request 'kvm2 provider: clean-room libvirt/KVM provider, code review, and the four high findings' (#8) from feat/kvm2-provider into main 
Reviewed-on: #8
 2026-09-17 23:12:46 +02:00
kacerrandClaude Opus 5 54525bf67a kvm2: gofmt volume.go 
Making baseVolumeName multi-line in the H4 commit collapsed the
alignment gofmt gives a run of single-line funcs, and the commit went out
with the old spacing. Caught by make test, whose fmt target rewrites the
file and leaves the tree dirty.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
 2026-09-17 21:07:10 +00:00
kacerrandClaude Opus 5 41fbd5e7e6 kvm2: record the review validation, enable the provider entry, changelog 
Closes out the review-fix campaign. The kvm2 provider was deployed
in-cluster and validated against beelink -- the first time any of this
code had run against real libvirt rather than the in-memory fake.

config/manager/providers_config.yaml ships the kvm2-home entry enabled,
since the operator is now running it for real.

review/ carries the code review the four H-commits reference, so H1..H4
resolve to something rather than dangling: scope, the seven-chunk sweep,
the escalations, the ten deep dives and the final report. .claude/agents
carries the two reviewer definitions that produced them.

What the validation did not prove is recorded next to what it did: H2's
error paths need a body that fails part way through, and a healthy 433 MB
download never finishes short, so the unverified libvirtd short-stream
behaviour is still open. The kvm2 integration test has also still never
been run.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
 2026-09-17 20:50:45 +00:00
kacerrandClaude Opus 5 4114d43e87 Route klog and net/http's own errors into the structured logger 
Some lines ignored the zap encoder entirely, so `kubectl logs | jq` could
never work:

  I0917 19:50:02.580097  1 leaderelection.go:258] "Attempting to acquire
  leader lease..." lock="egress-proxies-operator-system/b47711d1..."

That is klog's own stderr writer, not ours. client-go's leader elector
calls klog.FromContext(ctx), and controller-runtime hands the elector a
context with no logr in it -- mgr.Start receives ctrl.SetupSignalHandler(),
which carries no logger. FromContext falls back to Background(), and with
no logger registered that is klog's text sink. Nothing here imported or
configured klog; it was an indirect dependency that around 60 linked
packages use, including client-go/tools/{leaderelection,cache,record} and
client-go/transport. controller-runtime does not route it either -- its
only klog.SetLogger call is in its own test suite.

Plain klog.SetLogger, deliberately not SetLoggerWithOptions with
ContextualLogger(true). The contextual form is what klog's docs recommend
because it skips a klogr round trip, but it also disables klog's verbosity
gate and hands the decision to zap, whose level under Development is Debug
-- so client-go's V(1) chatter, suppressed today by klog's own -v=0, would
have started appearing. Plain SetLogger keeps klog's check in front and
still redirects what it does emit. Same fix to the format, no new volume,
which matters because volume is what prompted this.

The discovery API's http.Server set no ErrorLog, so net/http's own faults
(a failed Accept, a panic escaping a handler) went to the stdlib logger on
stderr. It now drains into logr through logr.ToSlogHandler, the pattern
internal/provider/gcp/wirelog.go already uses for the same purpose.

The deployment gets --zap-encoder=json and --zap-time-encoding=rfc3339.
The flag beats the in-code Development default -- addDefaults fills
NewEncoder only when it is nil -- and the time encoding is not optional,
since the JSON encoder defaults to epoch seconds. Verbosity stays at debug
on purpose; --zap-log-level=info drops the per-RPC lines.

zap.Options{Development: true} stays as the in-code default: flipping it
would change the format for every deployment of this operator, and the
flags get the same result here without that reach.

k8s.io/klog/v2 moves from indirect to direct in go.mod. It was already in
the module graph, so nothing new is downloaded.

No tests: three changes, all wiring, in a file with no test coverage, and
pinning klog.SetLogger means mutating process-global state that leaks into
every other test in the package. Verified in-cluster instead --
`kubectl logs | jq -e -s 'length > 0'` now passes, which fails if any
single line is not JSON.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
 2026-09-17 20:24:37 +00:00
kacerrandClaude Opus 5 82d01efb96 Plan routing klog into the structured logger (Step 6) 
Found while watching the Step 5 validation, not a review finding: the
leader election lines ignore the zap encoder because they are not ours.
client-go calls klog.FromContext, controller-runtime hands the elector a
context with no logr in it, and klog falls back to its own stderr writer.
Nothing in the repo configures klog; it is an indirect dependency that
~60 linked packages use.

Plain klog.SetLogger rather than the contextual form the klog docs
recommend: the contextual option disables klog verbosity checking and
hands the decision to zap, which at Development level would let
client-go V(1) chatter through that klog suppresses today. Same fix to
the format, no new volume.

Also the one other in-process stderr bypass (the discovery http.Server
has no ErrorLog) and json/rfc3339 encoder flags on the deployment.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
 2026-09-17 20:05:02 +00:00
kacerrandClaude Opus 5 966807bf57 Plan the in-cluster kvm2 validation (Step 5) 
First contact between the kvm2 provider and a real hypervisor: it has
never run against libvirt, so H1-H4 landed on code no host has executed.

The run happens in the existing kind cluster on purpose. That cluster
already owns two live kvm-home VMs on beelink, and orphan GC deletes any
instance whose UID is missing from its own cluster live set after
gc-min-age. A second operator carrying the kvm-home entry would therefore
destroy both within ten minutes; using the existing cluster makes its
live set the thing that protects them.

The plan also records what the run cannot prove: H2 error paths need a
body that fails part way through, and a healthy download never finishes
short, so the unverified libvirtd short-stream behaviour stays open.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
 2026-09-17 19:23:44 +00:00
kacerrandClaude Opus 5 c10f98a352 kvm2: name the cached base image after the provider entry (H4) 
baseVolumeName was base-<key>.<format>, byte-identical to what the kvm
provider builds in the same pool -- for URL images too, since kvm takes
hex(sum[:8]) and kvm2 takes hex(sum[:])[:16], the same eight bytes -- with
a completely separate lock table. The shipped config has a live kvm-home
entry and a commented-out kvm2-home entry on the same host, the same key
debian-13 and the same default pool, and the comment invites the operator
to enable it. Both integration tests use key test-image. On beelink both
live overlays back onto kvm's base-debian-13.qcow2.

Since cachedImage accepts any 64-hex marker when no sha256 is configured
and never re-hashes the base, that meant two entries pointing one key at
different URLs silently shared a file; a concurrent cold fetch let one
entry delete the other's half-uploaded base and boot a guest from the
remains; and a sha256 edit deleted the base in place under live overlays,
which libvirt permits without complaint.

The name now carries the provider entry: base-<entry>-<key>.<format>.
imageSource gained an Entry field and resolveImage takes the provider
name, so every production call site follows from imageSource.volumeName().
That closes the cross-entry sharing outright, kvm<->kvm2 and kvm2<->kvm2
alike, and turns the per-Provider imageLocks from a latent bug into a
correct one: with names unique per entry, two entries cannot contend for a
file, so no process-wide lock table is needed.

The entry name now reaches a file name, so New checks it against the
existing kvmIdentifier rule, exported as provider.IsKVMIdentifier rather
than duplicated. Provider names are otherwise validated for all types at
once and only for being non-empty and unique, so a name with a slash would
have produced a volume nothing could address.

One companion fix, because the naming alone does not stop it:
cachedImage's digest-mismatch branch deleted the marker and the base and
refetched. It now returns ErrPermanent naming both digests and telling the
operator what to do. That branch was the remaining path that unlinks a
good file under running domains, and the contract already said "add a new
key instead" -- the code was contradicting it. The no-marker and
non-hex-marker branches keep deleting, since there the base is debris.

Not fixed, and said so in the docs rather than left to look closed:
editing images.<key>.url is still a silent no-op, since nothing records
the URL; and cross-process fencing still needs the deferred atomic publish
(upload to a temp name, verify, publish), which lands where H2 put the
discard.

Migration is a non-event. The base-debian-13.qcow2 on beelink is kvm's,
kvm keeps using it untouched, and kvm2 downloads its own copy once per
host. Nothing is replaced under a running guest; the pool needs room for
both, which the prerequisites now say.

The eviction recipe in the guide named base-debian-13.qcow2, which after
this change is kvm's file -- following it would have deleted the wrong
provider's base. It now greps for the prefix and deletes the marker first.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
 2026-09-17 17:57:02 +00:00
kacerrandClaude Opus 5 9682737c69 kvm2: an unreadable address is not a missing one (H3) 
address() tried the guest agent then the host's ARP table and skipped any
error that was not a context error, returning ("", nil) after both. An
active domain with no IP maps to StateProvisioning; the controller clears
status.IP on that, and the health engine prunes the proxy's counters and
calls ForgetProxy in one tick once EffectiveHost() goes empty.

The damaging trigger was not the guest. The libvirt connection is pooled
per host, so when another reconcile's cancel or budget overrun killed it,
call returned "connection is closed" -- carrying no context sentinel, so
the check did not recognise it -- and Get returned a *successful*
instance in Provisioning with an empty IP. An unrelated proxy's timeout
dropped a healthy proxy out of rotation.

The discriminator was already in the spec without the conclusion being
drawn: during a normal boot on beelink exactly one source fails. The
agent errors because it is not up yet, while ARP answers without error
and with nothing, since the host never sees the guest's traffic. So one
source failing stays "no address yet"; every source failing, or a
connection already dead, is ErrTransient.

The explicit provider.ErrTransient wrap is load-bearing, not decoration.
Without it classify picks the first libvirt code out of the joined error,
and ErrInvalidArg, ErrAccessDenied and ErrNoSupport are Permanent -- an
address read failing would latch a running proxy as Failed, a worse
version of the harm being fixed.

Half of the finding is deliberately still open, and it is the half that
fires more often: a guest agent restarting inside a running VM still
demotes it, because that is indistinguishable from a VM still booting.
The discriminator is the previous IP, which the provider does not hold by
design and the controller already has. Follow-up finding to file against
proxy_controller.go:208-211 -- on StateProvisioning, keep an existing IP
and Provisioned condition for a few polls instead of clearing on the
first. The kvm2-local alternative (use the mark's createdAt as a
provisioning window) was rejected: createdAt is creation, not boot, and
it invents a deadline that belongs to the controller.

TestGet_addressErrorIsProvisioning failed against the new rule, correctly
-- it fails both sources with ErrAgentUnresponsive, and code 86 is
agent-specific, so the ARP source cannot produce it. Re-pointed as
TestGet_agentFailsAndARPEmptyIsProvisioning, the beelink boot it was
reaching for, which nothing covered before.
TestGet_everyAddressSourceFailingIsTransient covers the other side, both
sources erroring and the pooled link dying under the read; against the
unfixed code both return a Provisioning instance instead of an error.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
 2026-09-17 17:48:57 +00:00
kacerrandClaude Opus 5 62561c21fe kvm2: stop handing go-libvirt a reader that can error (H2) 
uploadImage streamed the HTTP body straight into StorageVolUpload. When
that body failed part way through, go-libvirt deadlocked: requestStream
starts its stream sender on two unbuffered channels and, on the error
path, returns without ever reading outErr, so SendStream emits a
StatusError packet and parks on `outErr <-` forever while the main
goroutine parks on `abort <- true` forever. StorageVolUpload never
returned. The 20-minute transfer budget rescued the caller only, and the
socket kill did nothing, because both ends are blocked on channel sends
rather than on I/O -- which is what the comment at backend.go:152 got
wrong. Each failed upload leaked kvm2's call goroutine, go-libvirt's
sender goroutine and a 4 MiB buffer, permanently, against a 128Mi pod
limit and a reconciler that retries on backoff.

The defect is upstream and stays there: no fork, no replace. Instead
eofReader, at the go-libvirt seam so it covers the seed and marker
uploads too, converts any source error to io.EOF -- the one outcome that
ends a stream the normal way -- and keeps the real error for the caller.
countingReader sits underneath untouched, so the existing length and
digest checks report exactly what they did before.

The download also gets its own deadline a minute inside the transfer
budget. fetchImage already bounded the whole fetch at 20 minutes, so the
body's death and the budget's socket kill were racing and the socket
usually won; with the body a minute ahead, a stalled mirror ends through
its reader and unwinds cleanly instead of leaking.

discardVolume moves from uploadImage into fetchImage's upload-error
branch, so the dedicated connection is closed and libvirtd has let go of
the stream's fd before the file is unlinked. Moving it rather than adding
a second Close keeps image_test's closeCount() == 1 true. That discard is
still unfenced against a second provider entry sharing the volume name;
H4 closes that by making the name per-entry.

What remains: a failure originating in libvirtd mid-upload still leaks,
since SendStream has already returned by then. Confirmed unfixable from
here. Known gaps quotes the two-character upstream fix
(make(chan bool, 1) / make(chan error, 1)) and the replace escape hatch.

Unverified at ~85%: that libvirtd answers a short-but-clean stream finish
with OK rather than an error. virFDStreamWrite enforces length only on
writes past it and nothing compares offset to length on the close path,
which is also why virsh vol-upload of a small file into a large volume
works. If it errors we are back to a deadlock by a different door, which
is exactly today's behaviour, so this cannot be worse. The execution file
carries a probe recipe to settle it.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
 2026-09-17 17:46:15 +00:00
kacerrandClaude Opus 5 d826114fd3 Plan the kvm2 H2, H3 and H4 fixes 
Extends the campaign plan with the three remaining high findings, each its
own step and commit, in the order H2 -> H3 -> H4 (H2 and H4 both touch
image.go, and H2 restructures the failure path H4 would otherwise be
written against).

H2 is a workaround, not a repair: the deadlock is upstream in go-libvirt,
whose requestStream never drains an unbuffered outErr on its error path.
Per the decision taken while planning, no fork and no replace directive;
we stop handing it a reader that can error, move the transfer deadline
onto the HTTP source so a stalled mirror unwinds cleanly, and document the
residual with the two-character upstream patch.

H4 names the base image per provider entry, which is what was chosen over
a content fingerprint. That fixes cross-entry sharing but not the silent
no-op on a url edit, so the plan says so rather than implying the finding
is closed.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
 2026-09-17 17:42:34 +00:00
kacerrandClaude Opus 5 0239fb7e69 kvm2: do not let a stalled write delay the connection teardown (H1) 
sshSocket.Close was errors.Join(s.Conn.Close(), s.client.Close()). Go
evaluates those left to right, so the blocking close ran before the
unblocking one. s.Conn is the forwarded libvirt socket, and closing an SSH
channel writes a close message under ch.writeMu — which go-libvirt already
holds while streaming a base image through that same channel. Against a
host that vanished without RST or FIN, that write blocks until the kernel
stops retransmitting (tcp_retries2, ~15 min), and s.client.Close(), a bare
net.Conn.Close() that takes no lock and unblocks the stalled writer through
netpoll, was never reached. Close the client only; nothing wants the polite
channel-close message, and reordering instead would leave an error to
filter in a memoised closeErr.

link.markDead compounded it by running kill() before close(l.closed), both
inside a sync.Once. Reordering alone was not enough: sync.Once.Do blocks
concurrent callers until the first body returns, so every other markDead —
including the one in call is own ctx-cancel branch — queued behind the
stalled kill, and cancelling the reconcile did not help. The once field is
gone; the mutex the struct already had guards the close-exactly-once
invariant, reason is set before the signal, and kill runs outside the lock.

Blast radius was the whole operator, not one host: MaxConcurrentReconciles
is 3 and ensureImage holds a per-image lock whose acquire is ctx-aware but
reconcile contexts carry no deadline, so three Proxies on one host with an
uncached image parked every worker.

The same edits close a second route to the same pin: go-libvirt is
Disconnect closes the net.Conn it was handed (libvirt.go:331 ->
socket.go:164), and that conn is our sshSocket, so libvirtBackend.Close is
Disconnect goroutine entered closeOnce and hung; the 5 s closeBudget
released Close, which called markDead -> kill -> sshSocket.Close and
blocked in the other goroutine is sync.Once.

Verified against the pinned x/crypto v0.54.0 rather than assumed: the
flow-control park (remoteWin.reserve, channel.go:268) is outside writeMu
and only writePacket holds it across the TCP write, so a writer parked in
SSH flow control never blocks the close and a writer inside the TCP write
always does. To reproduce by hand, shrink SO_SNDBUF well under the 2 MiB
channel window (channelWindowSize = 64 * channelMaxPacket) so the socket
buffer fills before the window does.

The kvm provider does not have this bug: it closes client only and never
touches the forwarded channel.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
 2026-09-17 15:30:41 +00:00
kacerrandClaude Opus 5 85ea319cc2 Plan the kvm2 code-review fixes 
The review of feat/kvm2-provider confirmed 52 findings (4 high, 20 medium,
28 low); review/REPORT.md has them. This is the campaign plan: one step and
one commit per finding, highest severity first, with only the step in hand
detailed so the plan is not a fiction about work not yet designed. Same
shape as the kvm campaign in 2026-09-10-2222-kvm-review-fixes.md.

Step 1 (H1) is detailed: sshSocket.Close runs the blocking close before the
unblocking one, so a transfer stuck in a TCP write pins the teardown for as
long as the kernel retransmits, and markDead compounds it by killing inside
a sync.Once before signalling dead.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
 2026-09-17 15:27:04 +00:00
kacerrandClaude Fable 5.1 2510842396 kvm2: apply the milestone review 
Shorter start wait with immediate verdicts for dead or paused domains,
ListByTag without state or address RPCs, Timeout for an expired check
round, no adoption of unmarked domains, reference-counted keyed locks,
dead options removed, HEAD fallback on any non-2xx.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
 2026-09-16 21:19:50 +00:00
kacerrandClaude Fable 5.1 fcbf8ddc91 kvm2: user guide, appendix column and section, architecture and README entries 
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
 2026-09-16 19:54:25 +00:00
kacerrandClaude Fable 5.1 fb6f59090b kvm2: integration test, sample Proxy, providers-config entries and Makefile target 
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
 2026-09-16 19:40:50 +00:00
kacerrandClaude Fable 5.1 ce62fe57f6 kvm2: health checker on dedicated connections; kvm2 in the ProviderHealth type comment 
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
 2026-09-16 19:26:51 +00:00
kacerrandClaude Fable 5.1 794a2aef3d kvm2: Create, Get, Delete and ListByTag with the contract tests 
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
 2026-09-16 19:23:49 +00:00
kacerrandClaude Fable 5.1 6b3c985b4a kvm2: base image cache with a streamed upload and the pool-space precheck 
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
 2026-09-16 18:19:05 +00:00
kacerrandClaude Fable 5.1 b45f83327a kvm2: SSH transport, ctx-first libvirt seam, host table and span decorator 
Every go-libvirt call runs under its own budget and a cancelled or overlong
call kills the connection; credentials are read on every dial; one span and
one log line per RPC come from a decorator applied where connections are
made.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
 2026-09-16 17:43:27 +00:00
kacerrandClaude Fable 5.1 1f10a51f82 kvm2: domain and volume XML, metadata element and NoCloud seed builders 
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
 2026-09-16 16:57:30 +00:00
kacerrandClaude Fable 5.1 95768281be kvm2: config block, constructor registration, IDs, state mapping and classify 
The kvm2 block reuses KVMConfig; validateKVM gains a block-name variant so
messages say kvm2. The package skeleton has a non-dialing New, strict
providerID parsing, the state table and a 34-row classify test.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
 2026-09-16 16:38:48 +00:00
kacerrandClaude Fable 5.1 4bd9aae03b Write the kvm2 provider spec from the template 
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
 2026-09-16 13:38:29 +00:00
kacerrandClaude Fable 5.1 854856cf0f Plan the clean-room kvm2 provider built from the provider contract 
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
 2026-09-16 13:33:56 +00:00
kacerr 0dc25a1637 Merge pull request 'Provider docs: generic contract, spec template and appendix on the existing providers' (#7) from feat/provider-blueprint into main 
Reviewed-on: #7
 2026-09-16 13:48:40 +02:00
185 changed files with 32424 additions and 524 deletions
Expand all files Collapse all files
.claude/agents/deep-reviewer.md
+11
View File
@@ -0,0 +1,11 @@
---
name: deep-reviewer
description: Deep-dive verification of a single escalated hypothesis from the sweep.
model: fable
effort: high
tools: Read, Grep, Glob, Bash
---
You get ONE hypothesis with entry points. Follow it across the whole repo: callers,
callees, tests, configs. Verdict: CONFIRMED / REFUTED / PLAUSIBLE, with the exact
path/interleaving that breaks, or the reason it can't. Propose a minimal fix.
Don't re-review the package generally. Stay on the hypothesis.
.claude/agents/sweep-reviewer.md
+23
View File
@@ -0,0 +1,23 @@
---
name: sweep-reviewer
description: First-pass broad code review of one package/module. Use for fan-out sweeps.
model: opus
effort: high
tools: Read, Grep, Glob, Bash
---
You review ONE package/module given in the prompt. Read-only, never edit.
Output two sections:
## Findings
Concrete defects you verified. Per item: file:line, severity, one-line defect,
failure scenario (inputs/state -> wrong behavior). No style nits unless they hide a bug.
## Escalate
Things you could NOT confirm or refute within this package, where being wrong is costly.
Escalate when: shared mutable state across goroutines, lock ordering, context/cancel
propagation, retry+timeout+idempotency interactions, reconcile races, authn/authz
boundaries, invariants that depend on code outside this package, state machines.
Per item: entry-point files, the specific hypothesis ("X may be read without mu
on path A->B->C"), what you already checked, what's still unknown.
Do NOT escalate things you can resolve yourself. Go resolve them.
.claude/settings.json
+174 -102
View File
@@ -1,113 +1,185 @@
{
"permissions": {
"allow": [
"Bash(go version *)",
"Bash(go mod *)",
"Read(//Users/jan.novak/.claude/skills/go-project-bootstrap/references/**)",
"Bash(ls -la /Users/jan.novak/.claude/skills/go-project-bootstrap/assets/ /Users/jan.novak/.claude/skills/go-project-bootstrap/assets/commands/ /Users/jan.novak/.claude/skills/go-project-bootstrap/assets/agents/ 2>&1)",
"Read(//Users/jan.novak/.claude/skills/go-project-bootstrap/assets/**)",
"Read(//Users/jan.novak/.claude/skills/go-project-bootstrap/assets/agents/**)",
"Read(//Users/jan.novak/.claude/skills/go-project-bootstrap/assets/commands/**)",
"Bash(mkdir -p /Users/jan.novak/srv/go/egress-proxies-operator/docs/plans && touch /Users/jan.novak/srv/go/egress-proxies-operator/docs/plans/.gitkeep && mkdir -p /Users/jan.novak/srv/go/egress-proxies-operator/.claude/commands /Users/jan.novak/srv/go/egress-proxies-operator/.claude/agents && ls -la /Users/jan.novak/srv/go/egress-proxies-operator)",
"Bash(git status)",
"Bash(git status:*)",
"Bash(git diff:*)",
"Bash(git log:*)",
"Bash(git branch:*)",
"Bash(git add:*)",
"Bash(git checkout:*)",
"Bash(git pull:*)",
"Bash(git fetch:*)",
"Bash(date:*)",
"Read(//tmp/**)",
"Read(//private/tmp/**)",
"Bash(go build:*)",
"Bash(go test:*)",
"Bash(go vet:*)",
"Bash(go mod:*)",
"Bash(go run:*)",
"Bash(go version:*)",
"Bash(go get:*)",
"Bash(golangci-lint run:*)",
"Bash(golangci-lint --version)",
"Bash(gofumpt:*)",
"Bash(tea pr create:*)",
"Bash(curl -sL https://go.dev/dl/?mode=json)",
"Bash(find / -iname \"docs-conventions.md\" -maxdepth 6)",
"Read(//Users/jan.novak/.claude/**)",
"Read(//Users/jan.novak/.claude/skills/**)",
"Bash(mkdir -p docs/plans && date \"+%Y-%m-%d-%H%M\")",
"Bash(go install *)",
"Bash(go env *)",
"Bash(git rm *)",
"Bash(mkdir -p /Users/jan.novak/srv/go/egress-proxies-operator/docs/plans-executions)",
"Bash(cd /Users/jan.novak/srv/go/egress-proxies-operator *)",
"Bash(echo \"build: $?\")",
"Bash(echo \"vet: $?\")",
"Bash(perl -i -pe 's{^\\\\t\\\\t\\\\t\\\\t\\\\t// TODO\\\\\\(user\\\\\\): Specify other spec details if needed\\\\.\\\\n}{\\\\t\\\\t\\\\t\\\\t\\\\t// A minimal, schema-valid spec so this placeholder test survives the\\\\n\\\\t\\\\t\\\\t\\\\t\\\\t// CEL/CRD validation added in Step 1. Rewritten wholesale in Step 4\\\\n\\\\t\\\\t\\\\t\\\\t\\\\t// alongside the real reconciler and envtest suite.\\\\n\\\\t\\\\t\\\\t\\\\t\\\\tSpec: crawlv1alpha1.ProxySpec{\\\\n\\\\t\\\\t\\\\t\\\\t\\\\t\\\\tMode: crawlv1alpha1.ModeExternal,\\\\n\\\\t\\\\t\\\\t\\\\t\\\\t\\\\tEndpoint: &crawlv1alpha1.EndpointSpec{Host: \"10.0.0.1\"},\\\\n\\\\t\\\\t\\\\t\\\\t\\\\t},\\\\n}' internal/controller/proxy_controller_test.go)",
"Bash(go tool *)",
"Bash(docker version *)",
"Bash(curl -sI --max-time 5 https://hub.docker.com)",
"Bash(kind get *)",
"Bash(curl -s --max-time 5 \"https://hub.docker.com/v2/repositories/ubuntu/squid/tags?page_size=10\")",
"Bash(python3 -c ' *)",
"Bash(curl -s --max-time 5 \"https://hub.docker.com/v2/repositories/ubuntu/squid/tags?page_size=100\")",
"Bash(go doc *)",
"Bash(go list *)",
"Bash(gofmt -w internal/provider/config.go)",
"Bash(gofmt -l .)",
"Bash(git restore *)",
"Bash(make manifests *)",
"Bash(make test *)",
"Bash(KUBEBUILDER_ASSETS=\"/Users/jan.novak/srv/go/egress-proxies-operator/bin/k8s/1.36.2-darwin-arm64\" go test -race ./internal/controller/)",
"Bash(grep -n 'func Channel' -A8 __CMDSUB_OUTPUT__/sigs.k8s.io/controller-runtime@v0.24.1/pkg/source/source.go)",
"Bash(grep -n 'type GenericEvent' __CMDSUB_OUTPUT__/sigs.k8s.io/controller-runtime@v0.24.1/pkg/event/event.go)",
"Bash(KUBEBUILDER_ASSETS=__TRACKED_VAR__/bin/k8s/1.36.2-darwin-arm64 go test -race ./...)",
"Bash(cat >> *)",
"Bash(make run-dev *)",
"Bash(kubectl get *)",
"Bash(kubectl delete *)",
"Bash(make docker-build *)",
"Bash(kind load *)",
"Bash(make deploy *)",
"Bash(kubectl -n egress-proxies-operator-system rollout status deploy/egress-proxies-operator-controller-manager --timeout=120s)",
"Bash(kubectl -n egress-proxies-operator-system rollout restart deploy/egress-proxies-operator-controller-manager)",
"Bash(git -C /Users/jan.novak/srv/go/egress-proxies-operator add docs/plans/2026-08-11-1742-gcp-provider-verbose-logging.md)",
"Bash(git -C /Users/jan.novak/srv/go/egress-proxies-operator commit -m 'Add plan: verbose V-level logging in the GCP provider *)",
"Bash(echo \"exit: $?\")",
"Bash(echo \"tests exit: $?\")",
"Bash(./bin/manager --version)",
"Bash(./bin/manager-stamped --version)",
"Bash(./bin/manager-pkg --version)",
"Bash(docker run *)",
"Bash(kubectl -n egress-proxies-operator-system get pods -o wide)",
"Bash(kubectl -n egress-proxies-operator-system get deploy egress-proxies-operator-controller-manager -o jsonpath='{.spec.template.spec.containers[0].args}')",
"Bash(kubectl -n egress-proxies-operator-system logs deploy/egress-proxies-operator-controller-manager)",
"Bash(python3 -c \"import json; d=json.load\\(open\\('docs/deploy/sa_key.json'\\)\\); print\\(d.get\\('type'\\), d.get\\('client_email'\\)\\)\")",
"Bash(tea pr *)",
"Bash(git worktree *)",
"Bash(python3 -c \"import yaml; yaml.safe_load\\(open\\('.gitea/workflows/build.yaml'\\)\\); print\\('YAML OK'\\)\")",
"Bash(ruby -ryaml -e \"YAML.load_file\\('.gitea/workflows/build.yaml'\\); puts 'YAML OK'\")",
"Bash(git -C /Users/jan.novak/srv/go/egress-proxies-operator status --short --branch)",
"Bash(git -C /Users/jan.novak/srv/go/egress-proxies-operator log --oneline -1)",
"Bash(git -C /Users/jan.novak/srv/go/egress-proxies-operator tag 0.01)",
"Bash(git -C /Users/jan.novak/srv/go/egress-proxies-operator push origin 0.01)",
"Bash(chmod +x docs/demo/run-demo.sh)",
"Bash(bash -n docs/demo/run-demo.sh)",
"Bash(git add *)",
"Bash(git commit -q -m 'Add plan: KVM/libvirt provider *)",
"Bash(go get *)",
"Bash(go build *)",
"Bash(go test *)",
"Bash(git commit *)"
"Bash(go vet *)",
"Bash(go run *)",
"Bash(go mod *)",
"Bash(go get *)",
"Bash(go install *)",
"Bash(go env *)",
"Bash(go tool *)",
"Bash(go doc *)",
"Bash(go list *)",
"Bash(go version *)",
"Bash(gofmt *)",
"Bash(gofumpt *)",
"Bash(golangci-lint run *)",
"Bash(golangci-lint --version)",
"Bash(./bin/manager* --version)",
"Bash(make -n *)",
"Bash(make manifests *)",
"Bash(make test *)",
"Bash(make test-integration-kvm *)",
"Bash(make run-dev *)",
"Bash(make docker-build *)",
"Bash(make deploy *)",
"Bash(make kvm-ssh-secret *)",
"Bash(git status *)",
"Bash(git diff *)",
"Bash(git log *)",
"Bash(git branch)",
"Bash(git branch -a)",
"Bash(git branch --show-current)",
"Bash(git add *)",
"Bash(git commit *)",
"Bash(git fetch *)",
"Bash(git pull *)",
"Bash(git worktree list *)",
"Bash(git config --get *)",
"Bash(git config --list *)",
"Bash(git merge-tree *)",
"Bash(git ls-remote *)",
"Bash(git -c credential.helper='!f\\(\\) { echo username=kacerr; echo \"password=$\\(tr -d \"[:space:]\" < ~/.config/gitea-token\\)\"; }; f' push origin feat/kvm-provider)",
"Bash(git -c credential.helper='!f\\(\\) { echo username=kacerr; echo \"password=$\\(tr -d \"[:space:]\" < ~/.config/gitea-token\\)\"; }; f' push -q origin feat/kvm-provider)",
"Bash(git -c credential.helper='!f\\(\\) { echo username=kacerr; echo \"password=$\\(tr -d \"[:space:]\" < ~/.config/gitea-token\\)\"; }; f' push -u origin feat/kvm-provider)",
"Bash(tea pr create *)",
"Bash(tea pr list *)",
"Bash(curl -sL https://go.dev/dl/?mode=json)",
"Bash(curl -s --max-time 10 https://gitea.home.hrajfrisbee.cz/api/v1/repos/kacerr/egress-proxies-operator/*)",
"Bash(curl -s --max-time 10 'https://gitea.home.hrajfrisbee.cz/api/v1/repos/kacerr/egress-proxies-operator/*')",
"Bash(curl -sI --max-time 10 https://gitea.home.hrajfrisbee.cz/v2/kacerr/egress-proxies-operator/*)",
"Bash(curl -s -H \"Authorization: token $\\(tr -d '[:space:]' < ~/.config/gitea-token\\)\" https://gitea.home.hrajfrisbee.cz/api/v1/repos/kacerr/egress-proxies-operator/pulls/5)",
"Bash(kubectl get *)",
"Bash(kubectl wait *)",
"Bash(kubectl kustomize *)",
"Bash(kubectl apply *)",
"Bash(kubectl run *)",
"Bash(kubectl config current-context)",
"Bash(kubectl config get-contexts *)",
"Bash(kubectl config view *)",
"Bash(kubectl -n egress-proxies-operator-system get *)",
"Bash(kubectl -n egress-proxies-operator-system logs *)",
"Bash(kubectl -n egress-proxies-operator-system rollout status *)",
"Bash(kubectl -n egress-proxies-operator-system rollout restart *)",
"Bash(kind get *)",
"Bash(kind load *)",
"Bash(kind version *)",
"Bash(docker version *)",
"Bash(docker info *)",
"Bash(docker image ls *)",
"Bash(docker image inspect *)",
"Bash(docker manifest inspect *)",
"Bash(docker history *)",
"Bash(docker save *)",
"Bash(docker run *)",
"Bash(docker exec *)",
"Bash(virsh -c qemu:///system list *)",
"Bash(virsh -c qemu:///system net-list *)",
"Bash(virsh -c qemu:///system net-dumpxml *)",
"Bash(virsh -c qemu:///system pool-list *)",
"Bash(virsh -c qemu:///system pool-dumpxml *)",
"Bash(virsh -c qemu:///system vol-list *)",
"Bash(ssh -i ~/.ssh/proxy-operator_ed25519 * novakj@beelink.tail8c69c.ts.net *)",
"Bash(ip -br *)",
"Bash(getent hosts *)",
"Bash(tailscale status *)",
"Bash(python3 *)",
"Bash(grep *)",
"Bash(echo \"*$?\")",
"Bash(date *)",
"Bash(env)",
"Bash(command -v *)",
"Bash(cat >> *)",
"Bash(tar -tf *)",
"Bash(bash -n *)",
"Bash(mkdir -p docs/*)",
"Read(//home/novakj/**)",
"Read(//Users/jan.novak/.claude/**)",
"Read(//tmp/**)",
"Read(//private/tmp/**)",
"Read(//etc/**)",
"Read(//dev/**)",
"Read(//run/libvirt/**)",
"WebSearch",
"WebFetch(domain:gitlab.com)",
"WebFetch(domain:bugs.debian.org)",
"WebFetch(domain:tracker.debian.org)",
"WebFetch(domain:cloud.debian.org)",
"WebFetch(domain:alioth-lists.debian.net)",
"WebFetch(domain:www.mail-archive.com)",
"WebFetch(domain:forum.proxmox.com)",
"Bash(curl -s --max-time 10 'https://gitea.home.hrajfrisbee.cz/api/v1/repos/kacerr/egress-proxies-operator/pulls?state=open')",
"Bash(python3 -c 'import json,sys; [print\\(p[\"number\"], p[\"head\"][\"ref\"], \"->\", p[\"base\"][\"ref\"], \"|\", p[\"title\"]\\) for p in json.load\\(sys.stdin\\)]')",
"Bash(curl -s --max-time 10 'https://gitea.home.hrajfrisbee.cz/api/v1/repos/kacerr/egress-proxies-operator/pulls/5')",
"Bash(python3 -c 'import json,sys; p=json.load\\(sys.stdin\\); print\\(p[\"state\"], \"merged=\", p[\"merged\"], p.get\\(\"merge_commit_sha\"\\)\\)')",
"Read(//home/novakj/srv/egress-proxies-operator-simplify/origin/**)",
"Bash(echo \"exit=$?\")",
"Read(//home/novakj/srv/egress-proxies-operator-simplify/**)",
"Bash(git -C /home/novakj/srv/egress-proxies-operator-simplify diff 82eeeef -- internal/controller/proxy_controller.go internal/provider/kvm/errors.go)",
"Bash(git switch *)",
"Bash(git -C /home/novakj/srv/egress-proxies-operator-simplify add -A)",
"Bash(git -C /home/novakj/srv/egress-proxies-operator-simplify commit -q -m 'WIP: uncommitted simplify-pass edits preserved before worktree removal *)",
"Bash(git -C /home/novakj/srv/egress-proxies-operator-simplify log --oneline -1)",
"Bash(git worktree *)",
"Bash(git branch *)",
"Bash(git stash *)",
"Bash(make generate *)",
"Bash(bin/kustomize build *)",
"Bash(grep -Ev \"^\\\\s*$\")",
"Bash(make lint *)",
"Bash(command -v kind docker)",
"Bash(bin/golangci-lint version *)",
"Bash(make test-e2e *)",
"Bash(echo \"make test-e2e exit=$?\")",
"Bash(ls \"$\\(go env GOMODCACHE\\)/golang.org/\")",
"Bash(ls \"$\\(go env GOMODCACHE\\)/golang.org/toolchain@\"* -d)",
"Bash(ps -eo pid,ppid,etime,args)",
"Bash(kubectl config *)",
"Bash(git commit -q -m 'Plan the provider blueprint document *)",
"Bash(grep -nE '^#{1,3} ' docs/architecture.md; echo ---; sed -n '/Implemented providers/,/^## /p' README.md | head -30; echo ---; sed -n 140,180p config/manager/manager.yaml; echo ---; cat Dockerfile; echo ---; ls -a | grep -i golangci; echo ---; grep -nE 'Placement|placement|Region|Zone|MachineType|Image ' api/v1alpha1/proxy_types.go | head -40; echo ---; grep -nE 'Type|type' api/v1alpha1/providerhealth_types.go | head -20; echo ---; grep -n -i 'placement' docs/api.md | head; echo ---; grep -nE 'WithTimeout|Deadline|timeout' internal/controller/proxy_controller.go internal/gc/gc.go internal/providerhealth/monitor.go | head -20; echo ---; cat .gitea/workflows/build.yaml | sed -n 1,60p)",
"WebFetch(domain:pve.proxmox.com)",
"Bash(git -c credential.helper='!f\\(\\) { echo username=kacerr; echo \"password=$\\(tr -d \"[:space:]\" < ~/.config/gitea-token\\)\"; }; f' push -u origin feat/provider-blueprint)",
"Bash(curl -s -H \"Authorization: token $\\(tr -d '[:space:]' < ~/.config/gitea-token\\)\" \"https://gitea.home.hrajfrisbee.cz/api/v1/repos/kacerr/egress-proxies-operator/pulls?state=open\")",
"Bash(curl -s -H \"Authorization: token $\\(tr -d '[:space:]' < ~/.config/gitea-token\\)\" \"https://gitea.home.hrajfrisbee.cz/api/v1/repos/kacerr/egress-proxies-operator/pulls/7\")",
"Bash(curl -s -X PATCH -H \"Authorization: token $\\(tr -d '[:space:]' < ~/.config/gitea-token\\)\" -H 'Content-Type: application/json' -d @/tmp/claude-1000/-home-novakj-srv-egress-proxies-operator/7662ab9d-4540-4676-a943-0874ac6b081d/scratchpad/pr7.json \"https://gitea.home.hrajfrisbee.cz/api/v1/repos/kacerr/egress-proxies-operator/pulls/7\")",
"Bash(bin/golangci-lint run *)",
"Bash(awk '/^## The image cache/,/^## [^T]/' docs/kvm-provider.md)",
"Bash(awk '/^#+ .*Step 11/,0' docs/plans/2026-09-16-1333-kvm2-provider.md)",
"Bash(sort -t: -k2 -u)",
"Bash(awk '/kvm2/&&/^#/{f=1} f' docs/testing.md)",
"Bash(awk -F'\"' '{print $2}')",
"Bash(sed -n '/targets:/,/type: object/p' config/crd/bases/*providerhealths.yaml | sed -n 1,45p)",
"Bash(ssh -i ~/.ssh/proxy-operator_ed25519 -o IdentitiesOnly=yes -o BatchMode=yes -o ConnectTimeout=10 -o UserKnownHostsFile=~/.ssh/kvm_known_hosts -o StrictHostKeyChecking=yes novakj@beelink 'hostname; id -un; virsh -c qemu:///system version; virsh -c qemu:///system pool-list --all; virsh -c qemu:///system list --all')",
"Bash(awk '{print $1, $2}' ~/.ssh/kvm_known_hosts)",
"Bash(ssh-keygen -F beelink -f ~/.ssh/kvm_known_hosts)",
"Bash(ssh -i ~/.ssh/proxy-operator_ed25519 -o IdentitiesOnly=yes -o BatchMode=yes -o ConnectTimeout=10 -o UserKnownHostsFile=~/.ssh/kvm_known_hosts -o StrictHostKeyChecking=yes novakj@beelink.tail8c69c.ts.net 'hostname; id -un; id -Gn; virsh -c qemu:///system version; virsh -c qemu:///system pool-list --all --details; virsh -c qemu:///system list --all; command -v qemu-img')",
"Bash(qemu-img --version)",
"Bash(./probe)",
"Bash(./probe -proxyrcv 8192)",
"Bash(./probe -small)",
"Bash(sysctl net.ipv4.tcp_wmem net.ipv4.tcp_retries2)",
"Bash(GOFLAGS=-mod=mod GOPROXY=off go build -o probe .)",
"Bash(./probe -clientsnd 131072 -proxyrcv 8192)",
"Bash(truncate -s 64M gpt.raw)",
"Bash(sfdisk -l gptshort.raw)",
"Bash(sgdisk -v gptshort.raw)",
"Bash(mkdir -p /tmp/claude-1000/-home-novakj-srv-egress-proxies-operator/8db56daf-938c-48b8-8a17-084f3e1dea44/scratchpad)",
"Bash(curl -sS -m 30 -o storage_driver.c https://raw.githubusercontent.com/libvirt/libvirt/v11.6.0/src/storage/storage_driver.c)",
"Bash(GOFLAGS=-mod=mod GOPROXY=off go vet ./...)",
"Bash(GOFLAGS=-mod=mod GOPROXY=off go test -count=1 -run TestLeak -v . -args -n 1000 -transport unix)",
"Bash(curl -sS -m 30 -o storage_util.c https://raw.githubusercontent.com/libvirt/libvirt/v11.6.0/src/storage/storage_util.c)",
"Bash(awk '/^virStorageBackendVolUploadLocal/,/^}/' storage_util.c)",
"Bash(export GOFLAGS=-mod=mod GOPROXY=off)",
"Bash(awk '/WARNING: DATA RACE/{c++} c==1' race_ssh.log)",
"Bash(kubectl version *)",
"Bash(timeout 5 kubectl cluster-info)"
],
"additionalDirectories": [
"/Users/jan.novak/srv/go/egress-proxies-operator/.claude",
"/Users/jan.novak/srv/go/egress-proxies-operator/docs/plans",
"/Users/jan.novak/srv/go/egress-proxies-operator/docs",
"/Users/jan.novak/srv/go/egress-proxies-operator/docs/prompts",
"/tmp",
"/Users/jan.novak/srv/go/egress-proxies-operator/docs/plans-executions"
"/home/novakj/srv/egress-proxies-operator",
"/home/novakj/srv/egress-proxies-operator/config/crd/bases"
]
}
}
AGENTS.md
+42 -320
View File
@@ -1,320 +1,42 @@
# egress-proxies-operator - AI Agent Guide
## Project Structure
**Single-group layout (default):**
```
cmd/main.go Manager entry (registers controllers/webhooks)
api/<version>/*_types.go CRD schemas (+kubebuilder markers)
api/<version>/zz_generated.* Auto-generated (DO NOT EDIT)
internal/controller/* Reconciliation logic
internal/webhook/* Validation/defaulting (if present)
config/crd/bases/* Generated CRDs (DO NOT EDIT)
config/rbac/role.yaml Generated RBAC (DO NOT EDIT)
config/samples/* Example CRs (edit these)
Makefile Build/test/deploy commands
PROJECT Kubebuilder metadata Auto-generated (DO NOT EDIT)
```
**Multi-group layout** (for projects with multiple API groups):
```
api/<group>/<version>/*_types.go CRD schemas by group
internal/controller/<group>/* Controllers by group
internal/webhook/<group>/<version>/* Webhooks by group and version (if present)
```
Multi-group layout organizes APIs by group name (e.g., `batch`, `apps`). Check the `PROJECT` file for `multigroup: true`.
**To convert to multi-group layout:**
1. Run: `kubebuilder edit --multigroup=true`
2. Move APIs: `mkdir -p api/<group> && mv api/<version> api/<group>/`
3. Move controllers: `mkdir -p internal/controller/<group> && mv internal/controller/*.go internal/controller/<group>/`
4. Move webhooks (if present): `mkdir -p internal/webhook/<group> && mv internal/webhook/<version> internal/webhook/<group>/`
5. Update import paths in all files
6. Fix `path` in `PROJECT` file for each resource
7. Update test suite CRD paths (add one more `..` to relative paths)
## Critical Rules
### Never Edit These (Auto-Generated)
- `config/crd/bases/*.yaml` - from `make manifests`
- `config/rbac/role.yaml` - from `make manifests`
- `config/webhook/manifests.yaml` - from `make manifests`
- `**/zz_generated.*.go` - from `make generate`
- `PROJECT` - from `kubebuilder [OPTIONS]`
### Never Remove Scaffold Markers
Do NOT delete `// +kubebuilder:scaffold:*` comments. CLI injects code at these markers.
### Keep Project Structure
Do not move files around. The CLI expects files in specific locations.
### Always Use CLI Commands
Always use `kubebuilder create api` and `kubebuilder create webhook` to scaffold. Do NOT create files manually.
### E2E Tests Require an Isolated Kind Cluster
The e2e tests are designed to validate the solution in an isolated environment (similar to GitHub Actions CI).
Ensure you run them against a dedicated [Kind](https://kind.sigs.k8s.io/) cluster (not your “real” dev/prod cluster).
## After Making Changes
**After editing `*_types.go` or markers:**
```
make manifests # Regenerate CRDs/RBAC from markers
make generate # Regenerate DeepCopy methods
```
**After editing `*.go` files:**
```
make lint-fix # Auto-fix code style
make test # Run unit tests
```
## CLI Commands Cheat Sheet
### Create API (your own types)
```bash
kubebuilder create api --group <group> --version <version> --kind <Kind>
```
### Deploy Image Plugin (scaffold to deploy/manage ANY container image)
Generate a controller that deploys and manages a container image (nginx, redis, memcached, your app, etc.):
```bash
# Example: deploying memcached
kubebuilder create api --group example.com --version v1alpha1 --kind Memcached \
--image=memcached:alpine \
--plugins=deploy-image.go.kubebuilder.io/v1-alpha
```
Scaffolds good-practice code: reconciliation logic, status conditions, finalizers, RBAC. Use as a reference implementation.
### Create Webhooks
```bash
# Validation + defaulting
kubebuilder create webhook --group <group> --version <version> --kind <Kind> \
--defaulting --programmatic-validation
# Conversion webhook (for multi-version APIs)
kubebuilder create webhook --group <group> --version v1 --kind <Kind> \
--conversion --spoke v2
```
### Controller for Core Kubernetes Types
```bash
# Watch Pods
kubebuilder create api --group core --version v1 --kind Pod \
--controller=true --resource=false
# Watch Deployments
kubebuilder create api --group apps --version v1 --kind Deployment \
--controller=true --resource=false
```
### Controller for External Types (e.g., from other operators)
Watch resources from external APIs (cert-manager, Argo CD, Istio, etc.):
```bash
# Example: watching cert-manager Certificate resources
kubebuilder create api \
--group cert-manager --version v1 --kind Certificate \
--controller=true --resource=false \
--external-api-path=github.com/cert-manager/cert-manager/pkg/apis/certmanager/v1 \
--external-api-domain=io \
--external-api-module=github.com/cert-manager/cert-manager
```
**Note:** Use `--external-api-module=<module>@<version>` only if you need a specific version. Otherwise, omit `@<version>` to use what's in go.mod.
### Webhook for External Types
```bash
# Example: validating external resources
kubebuilder create webhook \
--group cert-manager --version v1 --kind Issuer \
--defaulting \
--external-api-path=github.com/cert-manager/cert-manager/pkg/apis/certmanager/v1 \
--external-api-domain=io \
--external-api-module=github.com/cert-manager/cert-manager
```
## Testing & Development
```bash
make test # Run unit tests (uses envtest: real K8s API + etcd)
make run # Run locally (uses current kubeconfig context)
```
Tests use **Ginkgo + Gomega** (BDD style). Check `suite_test.go` for setup.
## Deployment Workflow
```bash
# 1. Regenerate manifests
make manifests generate
# 2. Build & deploy
export IMG=<registry>/<project>:tag
make docker-build docker-push IMG=$IMG # Or: kind load docker-image $IMG --name <cluster>
make deploy IMG=$IMG
# 3. Test
kubectl apply -k config/samples/
# 4. Debug
kubectl logs -n <project>-system deployment/<project>-controller-manager -c manager -f
```
### API Design
**Key markers for** `api/<version>/*_types.go`:
```go
// +kubebuilder:object:root=true
// +kubebuilder:subresource:status
// +kubebuilder:resource:scope=Namespaced
// +kubebuilder:printcolumn:name="Status",type=string,JSONPath=".status.conditions[?(@.type=='Ready')].status"
// On fields:
// +kubebuilder:validation:Required
// +kubebuilder:validation:Minimum=1
// +kubebuilder:validation:MaxLength=100
// +kubebuilder:validation:Pattern="^[a-z]+$"
// +kubebuilder:default="value"
```
- **Use** `metav1.Condition` for status (not custom string fields)
- **Use predefined types**: `metav1.Time` instead of `string` for dates
- **Follow K8s API conventions**: Standard field names (`spec`, `status`, `metadata`)
### Controller Design
**RBAC markers in** `internal/controller/*_controller.go`:
```go
// +kubebuilder:rbac:groups=mygroup.example.com,resources=mykinds,verbs=get;list;watch;create;update;patch;delete
// +kubebuilder:rbac:groups=mygroup.example.com,resources=mykinds/status,verbs=get;update;patch
// +kubebuilder:rbac:groups=mygroup.example.com,resources=mykinds/finalizers,verbs=update
// +kubebuilder:rbac:groups=events.k8s.io,resources=events,verbs=create;patch
// +kubebuilder:rbac:groups=apps,resources=deployments,verbs=get;list;watch;create;update;patch;delete
```
**Implementation rules:**
- **Idempotent reconciliation**: Safe to run multiple times
- **Re-fetch before updates**: `r.Get(ctx, req.NamespacedName, obj)` before `r.Update` to avoid conflicts
- **Structured logging**: `log := log.FromContext(ctx); log.Info("msg", "key", val)`
- **Owner references**: Enable automatic garbage collection (`SetControllerReference`)
- **Watch secondary resources**: Use `.Owns()` or `.Watches()`, not just `RequeueAfter`
- **Finalizers**: Clean up external resources (buckets, VMs, DNS entries)
### Logging
**Follow Kubernetes logging message style guidelines:**
- Start from a capital letter
- Do not end the message with a period
- Active voice: subject present (`"Deployment could not create Pod"`) or omitted (`"Could not create Pod"`)
- Past tense: `"Could not delete Pod"` not `"Cannot delete Pod"`
- Specify object type: `"Deleted Pod"` not `"Deleted"`
- Balanced key-value pairs
```go
log.Info("Starting reconciliation")
log.Info("Created Deployment", "name", deploy.Name)
log.Error(err, "Failed to create Pod", "name", name)
```
**Reference:** https://github.com/kubernetes/community/blob/master/contributors/devel/sig-instrumentation/logging.md#message-style-guidelines
### Webhooks
- **Create all types together**: `--defaulting --programmatic-validation --conversion`
- **When`--force`is used**: Backup custom logic first, then restore after scaffolding
- **For multi-version APIs**: Use hub-and-spoke pattern (`--conversion --spoke v2`)
- Hub version: Usually oldest stable version (v1)
- Spoke versions: Newer versions that convert to/from hub (v2, v3)
- Example: `--group crew --version v1 --kind Captain --conversion --spoke v2` (v1 is hub, v2 is spoke)
### Learning from Examples
The **deploy-image plugin** scaffolds a complete controller following good practices. Use it as a reference implementation:
```bash
kubebuilder create api --group example --version v1alpha1 --kind MyApp \
--image=<your-image> --plugins=deploy-image.go.kubebuilder.io/v1-alpha
```
Generated code includes: status conditions (`metav1.Condition`), finalizers, owner references, events, idempotent reconciliation.
## Distribution Options
### Option 1: YAML Bundle (Kustomize)
```bash
# Generate dist/install.yaml from Kustomize manifests
make build-installer IMG=<registry>/<project>:tag
```
**Key points:**
- The `dist/install.yaml` is generated from Kustomize manifests (CRDs, RBAC, Deployment)
- Commit this file to your repository for easy distribution
- Users only need `kubectl` to install (no additional tools required)
**Example:** Users install with a single command:
```bash
kubectl apply -f https://raw.githubusercontent.com/<org>/<repo>/<tag>/dist/install.yaml
```
### Option 2: Helm Chart
```bash
kubebuilder edit --plugins=helm/v2-alpha # Generates dist/chart/ (default)
kubebuilder edit --plugins=helm/v2-alpha --output-dir=charts # Generates charts/chart/
```
**For development:**
```bash
make helm-deploy IMG=<registry>/<project>:<tag> # Deploy manager via Helm
make helm-deploy IMG=$IMG HELM_EXTRA_ARGS="--set ..." # Deploy with custom values
make helm-status # Show release status
make helm-uninstall # Remove release
make helm-history # View release history
make helm-rollback # Rollback to previous version
```
**For end users/production:**
```bash
helm install my-release ./<output-dir>/chart/ --namespace <ns> --create-namespace
```
**Important:** If you add webhooks or modify manifests after initial chart generation:
1. Backup any customizations in `<output-dir>/chart/values.yaml` and `<output-dir>/chart/manager/manager.yaml`
2. Re-run: `kubebuilder edit --plugins=helm/v2-alpha --force` (use same `--output-dir` if customized)
3. Manually restore your custom values from the backup
### Publish Container Image
```bash
export IMG=<registry>/<project>:<version>
make docker-build docker-push IMG=$IMG
```
## References
### Essential Reading
- **Kubebuilder Book**: https://book.kubebuilder.io (comprehensive guide)
- **controller-runtime FAQ**: https://github.com/kubernetes-sigs/controller-runtime/blob/main/FAQ.md (common patterns and questions)
- **Good Practices**: https://book.kubebuilder.io/reference/good-practices.html (why reconciliation is idempotent, status conditions, etc.)
- **Logging Conventions**: https://github.com/kubernetes/community/blob/master/contributors/devel/sig-instrumentation/logging.md#message-style-guidelines (message style, verbosity levels)
### API Design & Implementation
- **API Conventions**: https://github.com/kubernetes/community/blob/master/contributors/devel/sig-architecture/api-conventions.md
- **Operator Pattern**: https://kubernetes.io/docs/concepts/extend-kubernetes/operator/
- **Markers Reference**: https://book.kubebuilder.io/reference/markers.html
### Tools & Libraries
- **controller-runtime**: https://github.com/kubernetes-sigs/controller-runtime
- **controller-tools**: https://github.com/kubernetes-sigs/controller-tools
- **Kubebuilder Repo**: https://github.com/kubernetes-sigs/kubebuilder
# egress-proxies-operator
## Sources and architecture
- Trust `go.mod`, `Makefile`, and current code over version tables or bootstrap prose. `CLAUDE.md`'s opening architecture/commands sections are stale; this is a working operator, not an empty scaffold.
- `cmd/main.go` assembles the provider registry, reconciler, proxy health engine, in-memory lease store, discovery HTTP API, orphan GC, and provider-health monitor on one controller-runtime manager.
- `api/v1alpha1` contains namespaced `Proxy` and cluster-scoped `ProviderHealth` resources (`crawl.example.com`). Validation/defaulting uses CRD markers/CEL; there are no admission webhooks.
- `internal/provider` defines the backend contract independently of CRD types. Constructors are registered in `cmd/main.go`; `spec.provider` selects a configured **instance name**, not a provider type. `kvm` and `kvm2` are separate implementations and do not adopt each other's VMs.
- Before changing a provider, read `docs/provider-contract.md`; use `docs/provider-spec-template.md` for a new one. Callers rely on self-contained provider IDs, truthful `ErrNotFound`, idempotent create/delete, and persisted UID/creation time for orphan GC. See `docs/providers-existing.md` for implementation-specific exceptions.
- `docs/architecture.md` explains component ownership and state transitions; `docs/api.md` documents discovery/leases; `docs/testing.md` covers test boundaries and real-infrastructure setup.
## Commands and verification
- Go version comes from `go.mod` (currently 1.26.0). Make installs development tools into `bin/`.
- After Go changes: `make lint-fix`, then `make test`. The latter regenerates manifests/DeepCopy, formats, vets, sets up envtest, then runs non-e2e tests with `-race` and writes `cover.out`.
- Fast checks: `go test -short -race ./...` skips envtest. Focused package/test: `go test -race ./internal/lease/` or `go test -race -run TestAcquire ./internal/lease/`.
- Envtest needs local API-server/etcd binaries; use `make setup-envtest` before targeted runs. For the controller suite with the current Kubernetes minor:
```sh
KUBEBUILDER_ASSETS="$(./bin/setup-envtest use 1.36 --bin-dir "$PWD/bin" -p path)" go test -race ./internal/controller/ -run TestControllers -ginkgo.focus='<spec regex>'
```
The Makefile derives the envtest **minor** from `go.mod`; do not replace it with a dependency patch version that may have no published binary bundle.
- Tests are mostly stdlib `testing`; controller envtest and e2e use Ginkgo/Gomega. Fake clients do not execute CEL or schema defaults: test those with envtest, and use the API's `*OrDefault` helpers in unit tests. Envtest has no kubelet, so it cannot prove Squid starts or serves traffic.
- `make lint` / `make lint-fix` builds a custom golangci-lint with the `logcheck` plugin from `.custom-gcl.yml`; a stock global binary is insufficient. `.golangci.yml` forbids `sort` (use `slices`). Logging uses controller-runtime/logr, with Kubernetes-style messages.
- Real-host tests are opt-in: `make test-integration-kvm` / `make test-integration-kvm2` require the corresponding `KVM_TEST_*` / `KVM2_TEST_*` variables from `docs/testing.md`. After provider refactors, compile-check these with `go vet -tags=integration ./...`.
- E2E requires Docker, kind, kubectl, and an isolated cluster. `make test-e2e` uses `KIND_CLUSTER=egress-proxies-operator-test-e2e`; ensure kubectl targets that cluster, especially if it already exists. On failure, run `make cleanup-test-e2e` explicitly. Compile-check with `go vet -tags=e2e ./...`. Tracing specs additionally require both `TEMPO_URL` and `OTLP_ENDPOINT`.
- `.gitea/workflows/build.yaml` runs only on tags/manual dispatch; its gate is `go vet ./...`, `go build ./...`, `go test -short ./...`. It does not replace local lint, race, or envtest verification.
## Code generation
- After API types or Kubebuilder markers change, run `make manifests generate`. Edit source markers, never generated `config/crd/bases/*.yaml`, `config/rbac/role.yaml`, or `**/zz_generated.*.go`.
- Keep `+kubebuilder:rbac` markers in standalone package-level comment blocks, outside function/type documentation; otherwise controller-gen can silently ignore them. Inspect `git diff config/rbac/role.yaml` after generation.
- Preserve `+kubebuilder:scaffold:*` markers and the single-group layout. Use `kubebuilder create api` / `kubebuilder create webhook` for new scaffolding; `PROJECT` is CLI-managed.
## Behaviour and operational gotchas
- Managed proxies are replace-on-change: `internal/controller/spechash.go` hashes placement, resolved cloud-init content, and effective port. Do not hash all of `ProxySpec`: attributes, lease limits, and health settings must not churn VMs.
- Cloud-init Secrets must carry `crawl.example.com/cloud-init: "true"`; the manager's Secret cache excludes unlabelled Secrets. Rotating labelled Secret content triggers VM replacement.
- Proxy health checks fetch **through** the proxy. The health engine supplies snapshots/events and the reconciler writes status; status-affecting transitions drive writes, so `lastHealthCheckTime` is not a last-probe heartbeat.
- Leases/cooldowns are in-memory and disappear on restart. Discovery is not leader-elected; the deployment uses one replica. Adding replicas alone does not provide shared lease capacity.
- Use `make run-dev` locally: plain `make run` omits the required `--providers-config`. Host-side probes generally cannot route to kind pod IPs; deploy in-cluster using the README quickstart to verify kubernetes-provider health.
- `make deploy IMG=...` and `make build-installer IMG=...` modify `config/manager/kustomization.yaml` in place. Check that diff before including it in a change.
- Namespace-restricted mode disables orphan GC by default. `--gc-allow-namespaced` bypasses that guard and can classify instances owned outside the visible namespace as orphans.
CHANGELOG.md
+80
View File
@@ -1,5 +1,85 @@
# Changelog
## 2026-09-24 10:28 UTC — proxmox provider: reviewed, fixed, proven on a real Proxmox VE 9.2 host
The `proxmox` provider (VMs cloned from a template on Proxmox VE through its REST API)
finished its last phase: a two-pass review, a real-host run on pv1-1, and captures of the
three API answers the design had only guessed.
- Delete now removes the seed ISO **before** destroying the VM. Before, a seed delete that
failed after the destroy was never retried (the next Get is NotFound), so the ISO leaked.
A 403 for `Datastore.Allocate` now fails Delete on every path instead of being skipped.
- Create skips a probe VMID held by someone else's VM at once, instead of waiting up to 10
minutes on that VM's lock on every retry.
- The captures corrected two assumptions: a missing ISO storage is a 400 (the health check
now says `StorageNotFound`, not `APIError`), and a hibernated VM can be destroyed (Delete
destroys ours instead of latching the Proxy).
- `TestProxmox_lifecycle` and `TestProxmoxCheck_realHost` passed on PVE 9.2.20: Create 6.2 s,
an address 50 s later. A kind pass took a sample Proxy to Ready and back.
- Review record: `docs/reviews/2026-09-23-proxmox-provider-review/REPORT.md`. Timings:
`docs/testing.md`. Code: `internal/provider/proxmox/{delete,create,check,errors}.go`.
## 2026-09-18 07:52 UTC — docs/operating.md, starting with the libvirt-RPC log query
New home for day-to-day helpers against a running operator, seeded with the `jq` query
that pulls timestamp + rpc out of the manager's JSON logs (both libvirt providers log the
same `"libvirt call"` msg, so one filter covers `kvm` and `kvm2`), plus the `fromjson?`
variant for streams carrying non-JSON lines. Linked from the README's doc index.
## 2026-09-18 07:44 UTC — a kvm2 crawl-pool sample
Until now the only kvm2 manifest was the one-VM `config/samples/proxy_kvm2.yaml`, which
left the multi-proxy shape — the one a crawler actually leases from — undocumented.
- Added `config/samples/proxy_kvm2_crawl_pool.yaml`: five Managed Proxies on the `kvm2-home`
entry, four sharing one labelled cloud-init Secret and one inline (it runs squid on 8080,
so it cannot share a Secret that pins `http_port 3128`). Covers the `tier` attribute the
discovery API selects on, a `medium`/`ubuntu-24.04` member with a raised `maxLeases`, and
a member with a tightened `healthCheck`.
- Kept out of `config/samples/kustomization.yaml` on purpose so `apply -k` doesn't boot a
whole pool; the exclusion is commented there next to the existing `providers-config.yaml`
one. Validated with `kubectl apply --dry-run=server`.
## 2026-09-17 20:36 UTC — kvm2: the four high findings from the code review, and its first run on a real hypervisor
A scoped review of the kvm2 provider (`review/REPORT.md`: 52 findings, 4 high, 20 medium,
28 low) produced four fixes, one commit each, then the provider was deployed in-cluster and
validated against beelink — the first time any of this code had run against real libvirt.
- **H1 — a stalled write no longer delays a connection teardown.** `sshSocket.Close` ran the
blocking close before the unblocking one, so a transfer stuck in a TCP write pinned the
teardown for as long as the kernel retransmitted (~15 min). `link.markDead` compounded it by
killing inside a `sync.Once` before signalling dead, which made even cancelling the reconcile
useless. With `MaxConcurrentReconciles: 3`, one silent host could stall every Proxy on every
provider. `internal/provider/kvm2/{ssh,backend}.go`.
- **H2 — a source error no longer reaches go-libvirt's stream sender.** A failing HTTP body
deadlocked `StorageVolUpload` outright and leaked two goroutines plus 4 MiB per occurrence,
against a 128Mi pod limit. The defect is upstream (`requestStream` never drains an unbuffered
`outErr`); the workaround wraps the reader so it can only ever end the stream cleanly, and
moves the transfer deadline onto the HTTP source so a stalled mirror unwinds instead of
leaking. A daemon-side failure mid-upload still leaks — documented in *Known gaps* with the
two-character upstream patch. `internal/provider/kvm2/{backend,image}.go`.
- **H3 — an unreadable address is no longer reported as a missing one.** Any error from both
address sources read as "no address yet", so a pooled connection killed by an unrelated
reconcile demoted a healthy proxy to Provisioning, cleared its IP and made the health engine
drop its counters. One source failing stays "no address yet"; every source failing, or a dead
connection, is now Transient. `internal/provider/kvm2/get.go`.
- **H4 — the base image cache is no longer shared between provider entries.** kvm2 built
byte-identical volume names to the `kvm` provider in the same pool, so two entries could share
one file, a concurrent fetch could boot a guest from a half-written base, and a `sha256` edit
deleted the base under live overlays. Names now carry the entry
(`base-<entry>-<key>.<format>`), and a digest mismatch is refused rather than resolved by
deleting. `internal/provider/kvm2/{volume,image}.go`, `internal/provider/config_kvm.go`.
- **klog and `net/http` errors now join the structured log stream.** client-go's leader election
wrote `I0917 ...` lines straight to stderr because nothing routed klog into logr, and the
discovery server's `http.Server` had no `ErrorLog`. The deployment also gained
`--zap-encoder=json` and `--zap-time-encoding=rfc3339`, so `kubectl logs | jq` works.
`cmd/main.go`, `internal/discovery/server.go`, `config/manager/manager.yaml`.
- **The `kvm2-home` provider entry is enabled** in `config/manager/providers_config.yaml`.
Validated end to end: health check Ready, a 433 MB image cached in 8.3 s under the new name
beside kvm's untouched one, a second Proxy reusing the cache, Squid answering a real CONNECT
probe, and Delete leaving the base image while removing the domain and its volumes.
## 2026-09-16 11:46 UTC — Provider blueprint split into contract, spec template and appendix
- **`docs/provider-contract.md`** is the generic provider specification: MUST/SHOULD/MAY
CLAUDE.md
+3
View File
@@ -226,3 +226,6 @@ A lightweight `check` job (`go vet` / `go build` / `go test -short`) gates the b
- `+kubebuilder:rbac` markers are package-level. Keep them in their own comment block,
never inside a type's or function's doc comment: controller-gen silently ignores
them there. After `make manifests`, check `git diff config/rbac/role.yaml`.
- `make test-integration-proxmox` runs the proxmox provider's integration tests against
a real Proxmox VE host. They skip unless `PROXMOX_TEST_URL`, `_TOKEN_FILE`, `_NODE`,
`_TEMPLATE_VMID` and `_ISO_STORAGE` are set; see docs/testing.md for the full env block.
Makefile
+25
View File
@@ -70,6 +70,14 @@ test: manifests generate fmt vet setup-envtest ## Run tests.
test-integration-kvm: ## Run the KVM provider test against a real hypervisor. Needs KVM_TEST_HOST/USER/SSH_KEY/BRIDGE/IMAGE_URL; see docs/testing.md.
go test -race -tags=integration -count=1 -v -timeout 40m -run TestKVM ./internal/provider/kvm/
.PHONY: test-integration-kvm2
test-integration-kvm2: ## Run the kvm2 provider test against a real hypervisor. Needs KVM2_TEST_HOST/USER/SSH_KEY/BRIDGE/IMAGE_URL; see docs/testing.md.
go test -race -tags=integration -count=1 -v -timeout 40m -run TestKVM2 ./internal/provider/kvm2/
.PHONY: test-integration-proxmox
test-integration-proxmox: ## Run the proxmox provider test against a real Proxmox VE host. Needs PROXMOX_TEST_URL/TOKEN_FILE/NODE/TEMPLATE_VMID/ISO_STORAGE; see docs/testing.md.
go test -race -tags=integration -count=1 -v -timeout 30m -run TestProxmox ./internal/provider/proxmox/
# TODO(user): To use a different vendor for e2e tests, modify the setup under 'tests/e2e'.
# The default setup assumes Kind is pre-installed and builds/loads the Manager Docker image locally.
# kubectl kuberc is disabled by default for test isolation; enable with:
@@ -192,6 +200,23 @@ kvm-ssh-secret: ## Create or update the kvm-ssh Secret from KVM_SSH_KEY (private
--from-file=known_hosts="$(KVM_KNOWN_HOSTS)" \
--dry-run=client -o yaml | "$(KUBECTL)" apply -f -
PROXMOX_TOKEN_FILE ?= $(HOME)/.config/proxmox-token
PROXMOX_CA_FILE ?= $(HOME)/.config/proxmox-ca.pem
PROXMOX_NAMESPACE ?= egress-proxies-operator-system
.PHONY: proxmox-token-secret
proxmox-token-secret: ## Create or update the proxmox-token Secret from PROXMOX_TOKEN_FILE (user@realm!tokenid=secret) and PROXMOX_CA_FILE.
@test -f "$(PROXMOX_TOKEN_FILE)" || { echo "PROXMOX_TOKEN_FILE=$(PROXMOX_TOKEN_FILE) not found"; exit 1; }
@test -f "$(PROXMOX_CA_FILE)" || { echo "PROXMOX_CA_FILE=$(PROXMOX_CA_FILE) not found"; exit 1; }
@test "$$(grep -cv '^[[:space:]]*$$' "$(PROXMOX_TOKEN_FILE)")" = 1 && \
grep -qxE '[[:space:]]*[^@[:space:]]+@[^![:space:]]+![^=[:space:]]+=[^[:space:]]+[[:space:]]*' "$(PROXMOX_TOKEN_FILE)" || \
{ echo "PROXMOX_TOKEN_FILE=$(PROXMOX_TOKEN_FILE) is not one line of user@realm!tokenid=secret"; exit 1; }
@grep -q 'BEGIN CERTIFICATE' "$(PROXMOX_CA_FILE)" || { echo "PROXMOX_CA_FILE=$(PROXMOX_CA_FILE) holds no PEM certificate"; exit 1; }
"$(KUBECTL)" -n $(PROXMOX_NAMESPACE) create secret generic proxmox-token \
--from-file=token="$(PROXMOX_TOKEN_FILE)" \
--from-file=ca.pem="$(PROXMOX_CA_FILE)" \
--dry-run=client -o yaml | "$(KUBECTL)" apply -f -
.PHONY: deploy
deploy: manifests kustomize ## Deploy controller to the K8s cluster specified in ~/.kube/config.
cd config/manager && "$(KUSTOMIZE)" edit set image controller=${IMG}
README.md
+9 -1
View File
@@ -160,6 +160,10 @@ Managed proxies as VMs on libvirt/KVM hosts you own. Full guide:
size (`placement.machineType`) and an image (`placement.image`) — see
[config/samples/proxy_kvm.yaml](config/samples/proxy_kvm.yaml).
The `kvm2` provider uses the same Secret and the same block shape (`type: kvm2`,
block `kvm2:`), so the steps above apply to it unchanged; its guide is
[docs/kvm2-provider.md](docs/kvm2-provider.md).
Cloud-init from a Secret: the Secret **must** carry the label
`crawl.example.com/cloud-init: "true"` — the operator's cache only holds
labelled Secrets, so an unlabelled one is invisible (the Proxy reports
@@ -187,6 +191,8 @@ can be two differently-configured instances of the same `gcp` type (see
| `kubernetes` | A real Squid pod (`ubuntu/squid:6.6-24.04_edge` by default) in the same cluster the operator runs in | `image` (optional) | Needs no cloud account — local dev, CI, and the kind quickstart. Pods share the cluster's egress IP, so it exercises the full lifecycle but not distinct egress paths. |
| `gcp` | A Compute Engine VM with an ephemeral external IP | `project` (required), `network`, `networkTag`, `diskSizeGb` | The real egress fleet. Deliberately uses only four API calls (Insert / Get / Delete / AggregatedList), all fire-and-forget: `Create` returns as soon as the operation is submitted and the reconciler discovers progress by polling `Get`. Auth is Application Default Credentials — workload identity in-cluster, `gcloud` ADC locally; no key-file plumbing. |
| `kvm` | A libvirt VM on one of your own hypervisors, reached over SSH | `sshKeyFile`, `knownHostsFile` (unless `insecureIgnoreHostKey`), `machineTypes`, `hosts` (required); `images`, `diskSizeGb`, `vendorData`, `installGuestAgent` | Home-lab or on-prem egress. The base image is downloaded into each host's pool on first use; `Create` waits for the VM to start. See [docs/kvm-provider.md](docs/kvm-provider.md). |
| `kvm2` | The same: a libvirt VM on one of your own hypervisors, reached over SSH | the same block shape as `kvm`, under `kvm2:` | The second-generation libvirt provider, built from the provider contract. Adds a child span per libvirt RPC, a pool-space precheck reported as quota (the reconciler waits instead of failing), SSH credentials re-read on every dial, and image downloads and health checks on connections of their own. It does not adopt `kvm`'s VMs. See [docs/kvm2-provider.md](docs/kvm2-provider.md). |
| `proxmox` | A VM on a Proxmox VE cluster: a full clone of a template you prepare, reached through the Proxmox REST API with an API token | `url`, `tokenFile`, `nodes`, `isoStorage`, `templates`, `machineTypes` (required); `caFile` or `insecureSkipTLSVerify`, `storage`, `bridge`, `vlanTag`, `vmidRange`, `requestTimeoutSeconds`, `vendorData`, `installGuestAgent` | Home-lab or on-prem egress without SSH. Cloud-init goes on a seed ISO uploaded through the API; the address comes from the QEMU guest agent, so the proxy network needs a route to a package mirror unless the template ships the agent. `Create` waits for the VM to run. The token is read on every request, from the optional `proxmox-token` Secret (`make proxmox-token-secret`). Built from the provider contract and run end to end against Proxmox VE 9.2.20. See [docs/proxmox-provider.md](docs/proxmox-provider.md). |
### Adding a provider
@@ -305,4 +311,6 @@ and the manual kind verification procedure — is in
Project layout, reconcile-loop diagrams, and the decision log are in
[docs/architecture.md](docs/architecture.md); the build history is in
[docs/plans-executions/](docs/plans-executions/).
[docs/plans-executions/](docs/plans-executions/). Day-to-day helpers for a
running operator — log queries and one-off inspections — are collected in
[docs/operating.md](docs/operating.md).
api/v1alpha1/providerhealth_types.go
+1 -1
View File
@@ -68,7 +68,7 @@ type ProviderTarget struct {
// ProviderHealthStatus is the result of the latest check of one provider.
type ProviderHealthStatus struct {
// type is the provider's type from the providers config: kubernetes,
// gcp or kvm.
// gcp, kvm, kvm2 or proxmox.
// +optional
Type string `json:"type,omitempty"`
cmd/main.go
+15 -2
View File
@@ -38,6 +38,7 @@ import (
"k8s.io/apimachinery/pkg/runtime"
utilruntime "k8s.io/apimachinery/pkg/util/runtime"
clientgoscheme "k8s.io/client-go/kubernetes/scheme"
"k8s.io/klog/v2"
ctrl "sigs.k8s.io/controller-runtime"
"sigs.k8s.io/controller-runtime/pkg/cache"
"sigs.k8s.io/controller-runtime/pkg/client"
@@ -59,6 +60,8 @@ import (
"gitea.home.hrajfrisbee.cz/kacerr/egress-proxies-operator/internal/provider/gcp"
"gitea.home.hrajfrisbee.cz/kacerr/egress-proxies-operator/internal/provider/kubernetes"
"gitea.home.hrajfrisbee.cz/kacerr/egress-proxies-operator/internal/provider/kvm"
"gitea.home.hrajfrisbee.cz/kacerr/egress-proxies-operator/internal/provider/kvm2"
"gitea.home.hrajfrisbee.cz/kacerr/egress-proxies-operator/internal/provider/proxmox"
"gitea.home.hrajfrisbee.cz/kacerr/egress-proxies-operator/internal/provider/registry"
"gitea.home.hrajfrisbee.cz/kacerr/egress-proxies-operator/internal/providerhealth"
"gitea.home.hrajfrisbee.cz/kacerr/egress-proxies-operator/internal/tracing"
@@ -161,7 +164,15 @@ func main() {
os.Exit(0)
}
ctrl.SetLogger(zap.New(zap.UseFlagOptions(&opts)))
logger := zap.New(zap.UseFlagOptions(&opts))
ctrl.SetLogger(logger)
// client-go logs through klog, and controller-runtime hands the leader
// elector a context with no logr in it, so klog.FromContext falls back
// to klog's own stderr writer and those lines ignore the encoder
// entirely. Plain SetLogger rather than the contextual form: it keeps
// klog's verbosity check in front, so this changes the format of what
// klog already emits without letting client-go's V(1) chatter through.
klog.SetLogger(logger)
setupLog.Info("Starting egress-proxies-operator",
"commit", version.Resolve(), "goVersion", goruntime.Version())
@@ -201,7 +212,9 @@ func main() {
"gcp": func(ctx context.Context, pc provider.ProviderConfig) (provider.Provider, error) {
return gcp.NewWithWireOptions(ctx, pc, gcp.WireLogOptions{FullPayloads: gcpWireFullPayloads})
},
"kvm": kvm.New,
"kvm": kvm.New,
"kvm2": kvm2.New,
"proxmox": proxmox.New,
})
if err != nil {
setupLog.Error(err, "Failed to build providers")
config/crd/bases/crawl.example.com_providerhealths.yaml
+1 -1
View File
@@ -183,7 +183,7 @@ spec:
type:
description: |-
type is the provider's type from the providers config: kubernetes,
gcp or kvm.
gcp, kvm, kvm2 or proxmox.
type: string
type: object
type: object
config/manager/manager.yaml
+29 -3
View File
@@ -64,6 +64,13 @@ spec:
- --leader-elect
- --health-probe-bind-address=:8081
- --providers-config=/etc/proxy-operator/providers.yaml
# Structured logs: every line is one JSON object, so `kubectl logs
# | jq` works. The time encoding is not optional — the JSON encoder
# defaults to epoch seconds. Verbosity stays at the binary's
# default (debug, one line per provider API call); add
# --zap-log-level=info for outcomes only.
- --zap-encoder=json
- --zap-time-encoding=rfc3339
# Add --trace-health-probes to emit one span per health probe
# (high volume: ~1/s per proxy).
# Provider health checks (kubectl get providerhealth) run every
@@ -149,15 +156,28 @@ spec:
- name: providers-config
mountPath: /etc/proxy-operator
readOnly: true
# SSH material for kvm providers (sshKeyFile / knownHostsFile in the
# providers config point here). Optional: the mount is empty when the
# Secret does not exist, and only kvm entries need it. Create it with:
# SSH material for the kvm and kvm2 providers (sshKeyFile /
# knownHostsFile in the providers config point here). One Secret
# serves both types. Optional: the mount is empty when the Secret
# does not exist, and only kvm and kvm2 entries need it. Create it with:
# kubectl -n egress-proxies-operator-system create secret generic kvm-ssh \
# --from-file=id_ed25519=<private key> \
# --from-file=known_hosts=<known_hosts with every host's key>
- name: kvm-ssh
mountPath: /etc/proxy-operator/kvm
readOnly: true
# API token and CA for the proxmox provider (tokenFile / caFile in
# the providers config point here). Optional: the mount is empty when
# the Secret does not exist, and only proxmox entries need it. The
# token file is read on every request; the CA once, on first use.
# Create it with:
# kubectl -n egress-proxies-operator-system create secret generic proxmox-token \
# --from-file=token=<file holding user@realm!tokenid=secret> \
# --from-file=ca.pem=<the node's /etc/pve/pve-root-ca.pem>
# or `make proxmox-token-secret`.
- name: proxmox-token
mountPath: /etc/proxy-operator/proxmox
readOnly: true
# Scratch space: the root filesystem is read-only, and the kvm
# provider builds each VM's cloud-init seed ISO through a temp
# directory (a few hundred KB, deleted right after upload).
@@ -167,10 +187,16 @@ spec:
- name: providers-config
configMap:
name: providers-config
# Shared by the kvm and kvm2 providers; see the mount above.
- name: kvm-ssh
secret:
secretName: kvm-ssh
optional: true
# The proxmox provider's token and CA; see the mount above.
- name: proxmox-token
secret:
secretName: proxmox-token
optional: true
- name: tmp
emptyDir:
sizeLimit: 64Mi
config/manager/providers_config.yaml
+61
View File
@@ -51,4 +51,65 @@ data:
bridge: br0 # to a LAN with DHCP; the VM's IP comes from there
# socket: /var/run/libvirt/libvirt-sock
# pool: default
# kvm2 takes the same block shape as kvm: an entry ports by changing
# type to kvm2 and the block name to kvm2. kvm2 does not adopt domains
# made by kvm, so delete those Proxies first. It caches its base images
# under base-<entry>-<key>.<format>, so it downloads its own copy rather
# than sharing kvm's; budget the pool space. See docs/kvm2-provider.md.
- name: kvm2-home
type: kvm2
kvm2:
sshKeyFile: /etc/proxy-operator/kvm/id_ed25519
knownHostsFile: /etc/proxy-operator/kvm/known_hosts
# insecureIgnoreHostKey: true
# diskSizeGb: 10
# installGuestAgent: false
# vendorData: |
# #cloud-config
# ...
images:
debian-13:
url: https://cloud.debian.org/images/cloud/trixie/latest/debian-13-generic-amd64.qcow2
# sha256: "<64 hex>"
# firmware: uefi
ubuntu-24.04:
url: https://cloud-images.ubuntu.com/noble/current/noble-server-cloudimg-amd64.img
machineTypes:
small: {vcpus: 1, memoryMiB: 1024}
medium: {vcpus: 2, memoryMiB: 2048}
hosts:
- name: beelink
address: beelink.tail8c69c.ts.net:22
user: novakj
bridge: br0
# socket: /var/run/libvirt/libvirt-sock
# pool: default
# proxmox clones a prepared template on a Proxmox VE cluster through its
# REST API and seeds it with a NoCloud ISO uploaded to isoStorage; it needs
# no SSH. Both files come from the optional proxmox-token Secret mounted by
# config/manager/manager.yaml. The url's host must be a name or address in
# the node certificate. See docs/proxmox-provider.md.
# - name: proxmox-home
# type: proxmox
# proxmox:
# url: https://192.168.0.8:8006
# tokenFile: /etc/proxy-operator/proxmox/token
# caFile: /etc/proxy-operator/proxmox/ca.pem
# # insecureSkipTLSVerify: true # instead of caFile; logged as a warning
# nodes: [pv1-1] # placement.zone names a node; optional with one node
# isoStorage: proxy-seeds # iso content; the token needs Datastore.Allocate here
# # storage: local-lvm # clone target; default is the template's disk storage
# # bridge: vmbr0 # replaces the cloned net0's bridge
# vlanTag: 4 # set on the cloned net0, MAC kept
# # vmidRange: {min: 100000, max: 199999}
# # requestTimeoutSeconds: 30
# # installGuestAgent: false
# # vendorData: |
# # #cloud-config
# # ...
# templates: # placement.image names a key; add keys, never repoint one
# debian-13: {vmid: 9001}
# machineTypes: # placement.machineType names a key here
# small: {cores: 1, memoryMiB: 1024}
# medium: {cores: 2, memoryMiB: 2048, diskGiB: 10}
config/samples/kustomization.yaml
+5
View File
@@ -1,9 +1,14 @@
## Append samples of your project ##
# providers-config.yaml is deliberately absent: it is a sample
# --providers-config file, not a Kubernetes manifest.
# proxy_kvm2_crawl_pool.yaml is deliberately absent too: it is a whole pool
# of VMs, so it stays an explicit `kubectl apply -f` rather than something
# `apply -k` boots along with the one-per-provider samples.
resources:
- proxy_kubernetes.yaml
- proxy_gcp.yaml
- proxy_kvm.yaml
- proxy_kvm2.yaml
- proxy_proxmox.yaml
- proxy_external.yaml
# +kubebuilder:scaffold:manifestskustomizesamples
config/samples/providers-config.yaml
+61
View File
@@ -55,3 +55,64 @@ providers:
bridge: br0 # to a LAN with DHCP; the VM's IP comes from there
# socket: /var/run/libvirt/libvirt-sock
# pool: default
# kvm2 takes the same block shape as kvm: an entry ports by changing type
# to kvm2 and the block name to kvm2. kvm2 does not adopt domains made by
# kvm, so delete those Proxies first. It caches its base images under
# base-<entry>-<key>.<format>, so it downloads its own copy rather than
# sharing kvm's; budget the pool space. See docs/kvm2-provider.md.
# - name: kvm2-home
# type: kvm2
# kvm2:
# sshKeyFile: /etc/proxy-operator/kvm/id_ed25519
# knownHostsFile: /etc/proxy-operator/kvm/known_hosts
# # insecureIgnoreHostKey: true
# # diskSizeGb: 10
# # installGuestAgent: false
# # vendorData: |
# # #cloud-config
# # ...
# images:
# debian-13:
# url: https://cloud.debian.org/images/cloud/trixie/latest/debian-13-generic-amd64.qcow2
# # sha256: "<64 hex>"
# # firmware: uefi
# ubuntu-24.04:
# url: https://cloud-images.ubuntu.com/noble/current/noble-server-cloudimg-amd64.img
# machineTypes:
# small: {vcpus: 1, memoryMiB: 1024}
# medium: {vcpus: 2, memoryMiB: 2048}
# hosts:
# - name: beelink
# address: beelink:22
# user: novakj
# bridge: br0
# # socket: /var/run/libvirt/libvirt-sock
# # pool: default
# proxmox clones a prepared template on a Proxmox VE cluster through its
# REST API and seeds it with a NoCloud ISO uploaded to isoStorage; it needs
# no SSH. Both files come from the optional proxmox-token Secret mounted by
# config/manager/manager.yaml. The url's host must be a name or address in
# the node certificate. See docs/proxmox-provider.md.
# - name: proxmox-home
# type: proxmox
# proxmox:
# url: https://192.168.0.8:8006
# tokenFile: /etc/proxy-operator/proxmox/token
# caFile: /etc/proxy-operator/proxmox/ca.pem
# # insecureSkipTLSVerify: true # instead of caFile; logged as a warning
# nodes: [pv1-1] # placement.zone names a node; optional with one node
# isoStorage: proxy-seeds # iso content; the token needs Datastore.Allocate here
# # storage: local-lvm # clone target; default is the template's disk storage
# # bridge: vmbr0 # replaces the cloned net0's bridge
# vlanTag: 4 # set on the cloned net0, MAC kept
# # vmidRange: {min: 100000, max: 199999}
# # requestTimeoutSeconds: 30
# # installGuestAgent: false
# # vendorData: |
# # #cloud-config
# # ...
# templates: # placement.image names a key; add keys, never repoint one
# debian-13: {vmid: 9001}
# machineTypes: # placement.machineType names a key here
# small: {cores: 1, memoryMiB: 1024}
# medium: {cores: 2, memoryMiB: 2048, diskGiB: 10}
config/samples/proxy_kvm2.yaml
+44
View File
@@ -0,0 +1,44 @@
# A Managed proxy backed by a libvirt/KVM hypervisor through the kvm2
# provider: the operator boots a stock Debian cloud image as a VM on the
# named host, with a NIC on the host's bridge (IP from the LAN's DHCP), and
# installs Squid via cloud-init. Requires a providers-config entry named
# "kvm2-home" (see providers-config.yaml) and the kvm-ssh Secret, which the
# kvm and kvm2 providers share (docs/kvm2-provider.md).
#
# placement.zone is the host name from kvm2.hosts (optional when only one
# host is configured); machineType and image are keys in the provider's
# machineTypes and images maps. The first Proxy for an image on a host
# waits for the image download; later ones reuse the cached copy.
apiVersion: crawl.example.com/v1alpha1
kind: Proxy
metadata:
name: proxy-kvm2-sample
spec:
mode: Managed
provider: kvm2-home
placement:
zone: beelink
machineType: small
image: debian-13
port: 3128
cloudInit:
inline: |
#cloud-config
packages:
- squid
write_files:
- path: /etc/squid/conf.d/proxy-operator.conf
content: |
http_port 3128
http_access allow all
via off
forwarded_for off
runcmd:
- systemctl restart squid
# Optional: log in with `ssh debian@<ip>` (the Debian image's default
# user) to look around; `virsh console <name>` on the host works too.
# ssh_authorized_keys:
# - ssh-ed25519 AAAA... you@laptop
attributes:
geo: home
purpose: crawl
config/samples/proxy_kvm2_crawl_only_one.yaml
+103
View File
@@ -0,0 +1,103 @@
# A pool of Managed crawl proxies on the kvm2 provider — the multi-proxy
# counterpart to the single proxy_kvm2.yaml sample. Everything here boots on
# the one host the kvm2 entry configures (beelink); the proxies differ in the
# attributes crawlers select on, in size, and in image.
#
# Requires the same prerequisites as proxy_kvm2.yaml: a providers-config
# entry named "kvm2-home" and the kvm-ssh Secret that the kvm and kvm2
# providers share (docs/kvm2-provider.md).
#
# Deliberately NOT in kustomization.yaml: `kubectl apply -k config/samples`
# would boot every VM in here at once. Apply it on its own when you want the
# pool:
#
# kubectl apply -f config/samples/proxy_kvm2_crawl_pool.yaml
#
# Crawlers reach these through the discovery API by attribute, e.g.
# `GET /v1/proxies?attr.purpose=crawl&healthy=true`, or by leasing one with
# the same selector. Attribute matching is exact equality, so keep the keys
# consistent across the pool and vary only the values you want to select on.
#
# The first Proxy for an image on the host waits for that image to download;
# the rest reuse the cached copy. debian-13 and ubuntu-24.04 are separate
# downloads, so the ubuntu member below is the slow one on a cold host.
---
# One cloud-init Secret shared by the whole pool: the proxies below point at
# it with secretRef instead of each carrying its own copy of the squid
# config. The crawl.example.com/cloud-init label is mandatory — the manager's
# cache only holds Secrets that carry it, so an unlabelled Secret is
# invisible to the operator and the Proxy never resolves.
#
# Editing this Secret changes the resolved cloud-init of every Proxy that
# references it, and that replaces each of their VMs. That is the trade for
# sharing one definition: convenient for a config rollout, disruptive if you
# only meant to touch one member. Give a member its own Secret (or an inline
# block) when it needs to diverge.
apiVersion: v1
kind: Secret
metadata:
name: crawl-squid-cloud-init
labels:
crawl.example.com/cloud-init: "true"
type: Opaque
stringData:
# The default key name; SecretKeySelector.key defaults to user-data, so
# the proxies below omit it.
user-data: |
#cloud-config
packages:
- squid
write_files:
- path: /etc/squid/conf.d/proxy-operator.conf
content: |
http_port 3128
http_access allow all
via off
forwarded_for off
runcmd:
- systemctl restart squid
# Optional: log in with `ssh debian@<ip>` (or `ubuntu@<ip>` on the
# ubuntu-24.04 member) to look around; `virsh console <name>` on the
# host works too.
# ssh_authorized_keys:
# - ssh-ed25519 AAAA... you@laptop
---
apiVersion: crawl.example.com/v1alpha1
kind: Proxy
metadata:
name: crawl-home-1
spec:
mode: Managed
provider: kvm2-home
placement:
zone: beelink
machineType: small
image: debian-13
port: 3128
cloudInit:
secretRef:
name: crawl-squid-cloud-init
attributes:
geo: home
purpose: crawl
tier: general
# ---
# apiVersion: crawl.example.com/v1alpha1
# kind: Proxy
# metadata:
# name: crawl-home-2
# spec:
# mode: Managed
# provider: kvm2-home
# placement:
# zone: beelink
# machineType: small
# image: debian-13
# port: 3128
# cloudInit:
# secretRef:
# name: crawl-squid-cloud-init
# attributes:
# geo: home
# purpose: crawl
# tier: general
config/samples/proxy_kvm2_crawl_pool.yaml
+190
View File
@@ -0,0 +1,190 @@
# A pool of Managed crawl proxies on the kvm2 provider — the multi-proxy
# counterpart to the single proxy_kvm2.yaml sample. Everything here boots on
# the one host the kvm2 entry configures (beelink); the proxies differ in the
# attributes crawlers select on, in size, and in image.
#
# Requires the same prerequisites as proxy_kvm2.yaml: a providers-config
# entry named "kvm2-home" and the kvm-ssh Secret that the kvm and kvm2
# providers share (docs/kvm2-provider.md).
#
# Deliberately NOT in kustomization.yaml: `kubectl apply -k config/samples`
# would boot every VM in here at once. Apply it on its own when you want the
# pool:
#
# kubectl apply -f config/samples/proxy_kvm2_crawl_pool.yaml
#
# Crawlers reach these through the discovery API by attribute, e.g.
# `GET /v1/proxies?attr.purpose=crawl&healthy=true`, or by leasing one with
# the same selector. Attribute matching is exact equality, so keep the keys
# consistent across the pool and vary only the values you want to select on.
#
# The first Proxy for an image on the host waits for that image to download;
# the rest reuse the cached copy. debian-13 and ubuntu-24.04 are separate
# downloads, so the ubuntu member below is the slow one on a cold host.
---
# One cloud-init Secret shared by the whole pool: the proxies below point at
# it with secretRef instead of each carrying its own copy of the squid
# config. The crawl.example.com/cloud-init label is mandatory — the manager's
# cache only holds Secrets that carry it, so an unlabelled Secret is
# invisible to the operator and the Proxy never resolves.
#
# Editing this Secret changes the resolved cloud-init of every Proxy that
# references it, and that replaces each of their VMs. That is the trade for
# sharing one definition: convenient for a config rollout, disruptive if you
# only meant to touch one member. Give a member its own Secret (or an inline
# block) when it needs to diverge.
apiVersion: v1
kind: Secret
metadata:
name: crawl-squid-cloud-init
labels:
crawl.example.com/cloud-init: "true"
type: Opaque
stringData:
# The default key name; SecretKeySelector.key defaults to user-data, so
# the proxies below omit it.
user-data: |
#cloud-config
packages:
- squid
write_files:
- path: /etc/squid/conf.d/proxy-operator.conf
content: |
http_port 3128
http_access allow all
via off
forwarded_for off
runcmd:
- systemctl restart squid
# Optional: log in with `ssh debian@<ip>` (or `ubuntu@<ip>` on the
# ubuntu-24.04 member) to look around; `virsh console <name>` on the
# host works too.
# ssh_authorized_keys:
# - ssh-ed25519 AAAA... you@laptop
---
apiVersion: crawl.example.com/v1alpha1
kind: Proxy
metadata:
name: crawl-home-1
spec:
mode: Managed
provider: kvm2-home
placement:
zone: beelink
machineType: small
image: debian-13
port: 3128
cloudInit:
secretRef:
name: crawl-squid-cloud-init
attributes:
geo: home
purpose: crawl
tier: general
---
apiVersion: crawl.example.com/v1alpha1
kind: Proxy
metadata:
name: crawl-home-2
spec:
mode: Managed
provider: kvm2-home
placement:
zone: beelink
machineType: small
image: debian-13
port: 3128
cloudInit:
secretRef:
name: crawl-squid-cloud-init
attributes:
geo: home
purpose: crawl
tier: general
---
apiVersion: crawl.example.com/v1alpha1
kind: Proxy
metadata:
name: crawl-home-3
spec:
mode: Managed
provider: kvm2-home
placement:
zone: beelink
machineType: small
image: debian-13
port: 3128
cloudInit:
secretRef:
name: crawl-squid-cloud-init
attributes:
geo: home
purpose: crawl
tier: general
---
# A heavier member: two vCPUs and 2 GiB from the medium machine type, a
# different base image, and room for more concurrent leases. tier=heavy is
# what a crawler selects on to land here rather than on the general members.
apiVersion: crawl.example.com/v1alpha1
kind: Proxy
metadata:
name: crawl-home-heavy-1
spec:
mode: Managed
provider: kvm2-home
placement:
zone: beelink
machineType: medium
image: ubuntu-24.04
port: 3128
cloudInit:
secretRef:
name: crawl-squid-cloud-init
attributes:
geo: home
purpose: crawl
tier: heavy
maxLeases: 10
---
# A member that probes harder than the pool default (30s interval, 3 failures
# to go unhealthy): a tighter loop takes a flapping proxy out of rotation
# sooner, at the cost of more probe traffic. Everything under healthCheck is
# defaulted, so set only what you want to change.
#
# This one also runs squid on a non-default port, which is why it carries its
# own inline cloud-init: spec.port and the http_port in the config have to
# agree, and the shared Secret above pins 3128.
apiVersion: crawl.example.com/v1alpha1
kind: Proxy
metadata:
name: crawl-home-probe-fast-1
spec:
mode: Managed
provider: kvm2-home
placement:
zone: beelink
machineType: small
image: debian-13
port: 8080
cloudInit:
inline: |
#cloud-config
packages:
- squid
write_files:
- path: /etc/squid/conf.d/proxy-operator.conf
content: |
http_port 8080
http_access allow all
via off
forwarded_for off
runcmd:
- systemctl restart squid
healthCheck:
intervalSeconds: 10
timeoutSeconds: 3
failureThreshold: 2
attributes:
geo: home
purpose: crawl
tier: general
config/samples/proxy_proxmox.yaml
+46
View File
@@ -0,0 +1,46 @@
# A Managed proxy backed by a Proxmox VE cluster through the proxmox
# provider: the operator full-clones a prepared template on the named node,
# attaches a NoCloud seed ISO carrying this cloud-init, and starts the VM on
# the configured network (here VLAN 4 on vmbr0, IP from that network's
# DHCP), where cloud-init installs Squid. Requires a providers-config entry
# named "proxmox-home" (see providers-config.yaml) and the proxmox-token
# Secret (docs/proxmox-provider.md).
#
# placement.zone is a node name from proxmox.nodes (optional when only one
# node is configured); machineType and image are keys in the provider's
# machineTypes and templates maps. The address comes from the guest agent,
# which a vendor-data script installs at first boot, so the VM's network
# needs a route to a package mirror.
apiVersion: crawl.example.com/v1alpha1
kind: Proxy
metadata:
name: proxy-proxmox-sample
spec:
mode: Managed
provider: proxmox-home
placement:
zone: pv1-1
machineType: small
image: debian-13
port: 3128
cloudInit:
inline: |
#cloud-config
packages:
- squid
write_files:
- path: /etc/squid/conf.d/proxy-operator.conf
content: |
http_port 3128
http_access allow all
via off
forwarded_for off
runcmd:
- systemctl restart squid
# Optional: log in with `ssh debian@<ip>` (the Debian image's default
# user) to look around; `qm terminal <vmid>` on the node works too.
# ssh_authorized_keys:
# - ssh-ed25519 AAAA... you@laptop
attributes:
geo: home
purpose: crawl
docs/architecture.md
+158 -1
View File
@@ -10,6 +10,9 @@
| kubernetes provider | `internal/provider/kubernetes` | library | — | real Squid pods in this cluster (local dev/CI) |
| gcp provider | `internal/provider/gcp` | library | — | Compute Engine VMs, four API calls, fire-and-forget ops |
| kvm provider | `internal/provider/kvm` | library | — | VMs on libvirt hosts over SSH-forwarded sockets; image cache, overlay disks, NoCloud seeds |
| kvm2 provider | `internal/provider/kvm2` | library | — | the same backend, built clean-room from the provider contract as a second type: per-RPC child spans, pool-space quota, per-call budgets, its own metadata namespace, credentials re-read on every dial |
| proxmox provider | `internal/provider/proxmox` | library | — | VMs on a Proxmox VE cluster through its REST API: full clones of a template, a NoCloud seed uploaded through the API, the VMID derived from the Proxy UID, every task waited on, HTTP child spans |
| Shared provider helpers | `internal/provider/nocloud`, `internal/provider/keyedlock` | library | — | the NoCloud seed ISO and vendor-data builder, and one-slot semaphores by key; used by kvm2 and proxmox |
| Health engine | `internal/health` | Runnable | yes | through-the-proxy probes, thresholds, transition events |
| Lease store | `internal/lease` | Runnable (expiry sweep) | no | in-memory leases + cooldowns, single mutex |
| Discovery API | `internal/discovery` | Runnable | no | HTTP list/lease/release/report on `:8090` |
@@ -168,6 +171,72 @@ providerID = hosts/<host>/domains/<name>; GC identity lives in the domain's
<metadata> element (libvirt has no labels). All RPCs run under a context-
bounded call: go-libvirt never times out on its own, so a cancelled reconcile
closes the SSH connection to unblock it.
kvm2 provider (internal/provider/kvm2/) - the same transport, a second type built from the contract
prov.Create ──► placement checked against the catalogue (Permanent before any dial)
DomainLookupByName: exists → mark+uid guard (no mark or foreign ─┐ every step
uid = Permanent; paused = Permanent), │
start if needed, wait for running, done │ idempotent,
else: StoragePoolGetInfo precheck: image (if uncached) + diskSizeGb │ under a
+ 16 MiB must fit, else ErrQuotaExceeded (reconciler │ per-name lock
waits 5m) │ shared with
ensure base image: marker present = cached; otherwise GET │ Delete
streamed into StorageVolUpload on a DEDICATED connection, │
StoragePoolRefresh, marker written last │
overlay volume (exists = reuse) │
seed ISO (always rebuilt) │
DomainDefineXML (<metadata> in kvm2's own namespace: │
managed, uid, createdAt) + DomainCreate, poll until │
running (15s cap; a domain that dies at once = Transient) ─┘
prov.Get ──► DomainLookupByName + DomainGetMetadata + DomainGetState
+ DomainInterfaceAddresses (guest agent, then ARP; first routable IPv4)
running without IP = Provisioning; nostate = Provisioning;
only ErrNoDomain is NotFound
prov.Delete ──► destroy (not running = fine) → undefine (nvram, managed save, snapshots)
→ delete overlay + seed by name; unconfigured host = logged no-op
prov.ListByTag ─► ConnectListAllDomains on every host, keep domains whose kvm2
<metadata> says managed=true; one host down fails the sweep,
one unreadable domain is skipped
Check ──► per host, on a FRESH connection: version, hostname, pool lookup,
pool info, close; never the pooled one
providerID = hosts/<host>/domains/<name>. Every RPC has its own budget (dial 10s,
RPC 30s, DomainCreate 90s, seed upload 60s, image transfer 20m): a call that
outlives its ctx or budget kills the pooled connection (the only way to unblock
go-libvirt), so other calls in flight on that host fail once with Transient and
retry; the next call redials. Each RPC is a child span kvm2.<Method> under
provider.*, plus kvm2.dial and kvm2.download.
proxmox provider (internal/provider/proxmox/) - the Proxmox VE REST API, one URL per cluster
prov.Create ──► placement checked against the catalogue (Permanent before any request)
GET /cluster/resources: a VM with the name → ownership check ─┐ every step
else: template config checked (not a template, cloud-init drive, │ idempotent,
no net0 or boot disk it needs = Permanent) │ under a
VMID = hash(UID) in vmidRange, probing on; a taken VMID → │ per-name lock
ownership check: wait out a lock, adopt our name + mark + │ shared with
UID, probe past another name, Permanent otherwise │ Delete; each
POST .../clone (full, description = the mark) → wait task │ task's UPID
upload seed ISO to isoStorage (overwrites) → wait task │ polled every
PUT .../config: cores, memory, agent=1, tag, ide2 = seed, │ 1s to its end
net0 bridge/tag when configured │
resize if diskGiB is larger → wait task │
start → wait task → poll status/current until running (60s) ─┘
prov.Get ──► /cluster/resources (absent = NotFound; lock or node unknown = Transient)
+ config (name and mark must match, else NotFound: VMID reused)
+ status/current (paused and hibernated = Stopped)
+ agent network-get-interfaces (first routable IPv4; agent not
up yet = Provisioning)
prov.Delete ──► resources + config: gone, not ours or unmarked = leave it;
stop (unless stopped or hibernated) → delete the seed on the
VM's node → destroy (purge); with no VM of ours left, the seed
is deleted on every node
prov.ListByTag ─► one /cluster/resources for the whole cluster, one config read per
candidate (tag or proxy- name); the description's mark decides
Check ──► per node: /version, node status, isoStorage status, templates in
/cluster/resources
providerID = vms/<vmid>/<name>: no node, because VMs migrate; the node is looked
up in /cluster/resources on every call. Token read from its file on every
request. Each request is an HTTP <METHOD> child span from the parent-gated
transport wrapper; a cancelled request ends only itself.
```
The reconciler never watches provider-side resources (Pods or GCP VMs).
@@ -343,6 +412,21 @@ incoming trace context):
- `provider.create|get|delete|list` (client spans, `provider.WithTracing`
decorator wired outermost around `WithMetrics`); the GCP SDK's own
otelhttp transport contributes HTTP child spans automatically.
- kvm2 child spans: under `provider.*`, one `kvm2.<Method>` client span per
libvirt RPC (for example `kvm2.DomainDefineXML`, attributes `kvm2.host`,
`kvm2.domain`, `kvm2.pool`, `kvm2.volume`, `kvm2.libvirt_code` on
failure), `kvm2.dial` around the SSH handshake and libvirt connect, and
`kvm2.download` around an image fetch; under `providerhealth.check`,
`kvm2.dial` and the four `kvm2.<Method>` spans of a check. They come
from a span decorator over the seam, applied once in the dialer, so no
RPC is unspanned. The `libvirt call` log line of an RPC is written
inside its span. The `kvm` provider has no child spans.
- proxmox child spans: one `HTTP <METHOD>` client span per API request,
from `tracing.RestConfigWrapper()` on the provider's own HTTP transport,
under `provider.*` or `providerhealth.check`. Parent-gated like the
Kubernetes API spans, so a request without a parent starts none. The
provider's logging decorator starts no span; its `proxmox api call` line
carries the operation span's IDs.
- Kubernetes API calls become client child spans via a wrapped
`rest.Config` transport — gated on an existing parent span, so informer
list/watch long-polls and leader-election renewals never create root
@@ -368,11 +452,13 @@ the slog handler's ctx.
Answers "can this cluster reach each provider's backend right now?"
without waiting for a Proxy to fail. A provider opts in by implementing
`provider.Checker`: `Check(ctx) []TargetStatus` returns one verdict per
target and never fails as a whole. All three providers do:
target and never fails as a whole. All five providers do:
| Provider | Target | What one check does |
|---|---|---|
| kvm | each `kvm.hosts` entry | over the pooled SSH+libvirt connection: libvirt version and hostname, then the configured pool must exist and be running; reports pool capacity and free space ([details](kvm-provider.md#health-check)) |
| kvm2 | each `kvm2.hosts` entry | on a fresh SSH+libvirt connection dialled for the check and closed after it: libvirt version and hostname, then the configured pool must exist and be running; `details` carry `user`, `pool`, `hostname`, `libvirtVersion`, `poolState`, `poolCapacityGiB` and `poolAvailableGiB` ([details](kvm2-provider.md#health-check)) |
| proxmox | each `proxmox.nodes` entry | four requests on the shared HTTP client: API version, node status, the `isoStorage` status (it must be active and hold `iso` content), and the cluster's resources (every configured template must exist and be a template); `details` carry `isoStorage`, `tokenID`, `pveVersion`, `kernel`, `isoStorageAvailGiB` and `templates` ([details](proxmox-provider.md#health-check)) |
| kubernetes | `apiserver` | lists at most one managed Pod cluster-wide: the verb and scope `ListByTag` and GC rely on |
| gcp | the project | one page of an aggregated instance list with `maxResults=1`: credentials, Compute API enabled, `compute.instances.list` |
@@ -613,3 +699,74 @@ they read as choices, not accidents. Chronological by build step.
- **A provider name that is not a valid object name** gets metrics but no
`ProviderHealth` object, with a warning at start. Tightening config
validation instead would crash-loop existing deployments on upgrade.
- **kvm2 is a second provider type built clean-room from the contract,**
not a rewrite of `kvm` in place. Its spec
(`docs/kvm2-provider-spec.md`) and code were written from
[provider-contract.md](provider-contract.md), the kvm user guide and
go-libvirt alone, never from `internal/provider/kvm/`, as a test of
whether the contract is enough to build a provider. Both types ship,
share the `KVMConfig` block shape and the `kvm-ssh` Secret, and may
share a host.
- **kvm2 image downloads run on a dedicated connection.** A cancelled or
overlong call kills the connection it ran on (the only way to unblock
go-libvirt), so a 435 MB upload on the pooled connection could die
because a poll for another Proxy was cancelled. The transfer dials its
own connection and closes it afterwards; the cheap checks and the marker
stay on the pooled one.
- **kvm2 health checks dial fresh connections**, the opposite of the kvm
decision above. With credentials re-read on every dial a fresh
connection does pick up a rotated Secret, and a check that runs out of
time then closes only its own connection, so it cannot disrupt a
reconcile. The cost is one sshd session per host per round.
- **A cancelled kvm2 call kills the pooled connection.** Every RPC has a
budget; when a call outlives its ctx or budget the adapter closes the
socket and the SSH client, returns Transient, and marks the connection
dead so the next call redials. Other calls in flight on that host fail
once with Transient and their callers retry. Documented in the guide
rather than avoided: the alternative, leaving a hung RPC blocked
forever, pins a worker.
- **kvm2 marks domains in its own metadata namespace**, so it never lists
or adopts domains the `kvm` provider made. Switching an entry from `kvm`
to `kvm2` means deleting its Proxies first; the two types can share a
host without one collecting the other's VMs.
- **kvm2 turns pool free space into the quota signal.** libvirt reports a
full pool as an ordinary error, which would retry on the fast backoff.
Before creating any volume, `Create` compares `StoragePoolGetInfo`'s
available bytes with the image (if uncached) plus `diskSizeGb` plus 16
MiB for the seed, and returns `ErrQuotaExceeded` below that, so the
reconciler waits 5 minutes instead.
- **kvm2 classifies SSH authentication and host-key failures as
Permanent.** A key missing from `authorized_keys` or a host key missing
from or different in known_hosts needs an admin; kvm retried these
forever. A missing key or known_hosts file stays Transient, because the
optional Secret may land later.
- **kvm2 re-reads the SSH key and known_hosts on every dial**, so rotating
the Secret needs no restart; nothing is cached. The constructor only
checks that the files exist, to warn early without failing the provider
set.
- **proxmox derives the VMID from the Proxy UID and checks ownership
before using a taken one.** Proxmox addresses VMs by VMID, and names are
not unique, so the name cannot be the key. A hash of the UID placed in
`vmidRange` makes a retry land on the same VMID; a taken one is waited
on while locked, then adopted when it has our name, mark and UID,
probed past when it has another name, and refused otherwise. The wait
is what stops a retry after a crash mid-clone from cloning a duplicate:
a clone in progress has no name yet, so the by-name lookup misses it.
- **proxmox writes its mark into the VM's description**, in the clone
call itself, so no VM of ours exists unmarked. Tags cannot be passed to
the clone, and the clone's own creation time is copied from the
template. The `proxy_operator` tag is set afterwards for the Proxmox UI
and to narrow GC's candidates; the mark decides.
- **proxmox delivers cloud-init as a seed ISO uploaded through the API**,
not as Proxmox's own cloud-init drive or a `cicustom` snippet. The
upload endpoint does not accept snippets, and the operator has no shell
on the nodes; the seed code is kvm2's, extracted into
`internal/provider/nocloud`. Templates must therefore carry no Proxmox
cloud-init drive, and seeds of VMs destroyed outside the operator are a
documented GC blind spot.
- **proxmox resolves a VM's node on every call** from
`/cluster/resources`, and its providerID names none, because VMs migrate
between nodes. The cost is one extra request per operation; the gain is
that a migrated VM stays addressable and the listing, which includes the
guests of an unreachable node as `status: unknown`, never mistakes a
down node for a deleted VM.
docs/kvm2-provider-spec.md
+665
View File
@@ -0,0 +1,665 @@
# Provider spec: kvm2
Filled from [provider-spec-template.md](provider-spec-template.md) against
[provider-contract.md](provider-contract.md). This is the design the `kvm2`
package is built from. Where an entry names an axis in backticks, the answer is
one of that axis's profiles in contract §4. Nothing in this spec was taken from
the code of the existing `kvm` provider; backend facts come from
[kvm-provider.md](kvm-provider.md), the execution log of the first build, the
go-libvirt module in the module cache, and the beelink host.
## Header
- **Type**: `kvm2`
- **Example configured name**: `kvm2-home`
- **Backend and API version this spec was written against**: libvirt 11.6.0 and
QEMU 10.1.0 on Ubuntu (host `beelink`), reached with go-libvirt
`github.com/digitalocean/go-libvirt v0.0.0-20260814190004-1a83157e1858` over an
SSH-forwarded unix socket.
- **Client**: `rpc-over-ssh`. go-libvirt's RPC client (`libvirt.New(conn)`
then `Connect()`) over a `direct-streamlocal` channel opened with
`golang.org/x/crypto/ssh` to the host's libvirt socket. Both modules are
already direct requires in `go.mod`; no new dependency.
- **Author and date**: Claude (orchestrator session), 2026-09-16.
## 1. Identity and providerID
**Answer:** `hosts/<host>/domains/<name>`, where `<host>` is `hosts[].name`
from the config block and `<name>` is `req.Name`. Parsing is strict: exactly
four segments, literal `hosts` and `domains`, non-empty host and name, no
further slashes; anything else is `ErrPermanent` from `Get` and `Delete`. Host
names are validated at config load by the shared `kvmIdentifier` regexp in
`internal/provider/config_kvm.go` (letters, digits, dots, dashes, underscores),
so a host name can never contain the separator. Instances do not migrate
between hosts (libvirt domains are per host), so the location belongs in the
ID.
**Evidence:** contract §6; `kvmIdentifier` in `internal/provider/config_kvm.go`.
## 2. Primary key (`primary-key`)
**Answer:** `name-is-key`. libvirt addresses domains by name
(`DomainLookupByName`); `req.Name` is used verbatim as the domain name and as
the base of the per-VM volume names. Create looks the name up first and adopts
an existing domain only if it carries the kvm2 mark (`managed` true, entry 8)
and its metadata `uid` is empty or equals `req.UID`. A domain without the
mark returns `ErrPermanent` saying it was not created by this provider: kvm2
never starts or takes over a VM someone else defined under that name. A
different non-empty `uid` returns `ErrPermanent` naming both UIDs. This
closes the contract's "adoption is not verified" limit for this provider.
**Evidence:** `DomainLookupByName(Name string) (Domain, error)` in the
go-libvirt module; contract §2.1 idempotency and §13.
## 3. Placement mapping
**Answer:** three fields are catalogue keys checked before any backend call, one is ignored:
| Field | Meaning | Required | Unknown value |
|---|---|---|---|
| `Zone` | a `hosts[].name` | only when more than one host is configured; with exactly one host an empty zone selects it | `ErrPermanent`: `placement.zone "x" is not in kvm2.hosts` |
| `MachineType` | a key of `machineTypes` (`vcpus`, `memoryMiB`) | yes | `ErrPermanent`: `placement.machineType "x" is not in kvm2.machineTypes` |
| `Image` | a key of `images`, or an absolute http(s) URL used directly | yes | `ErrPermanent`: `placement.image "x" is neither a key in kvm2.images nor an http(s) URL` |
| `Region` | ignored | | still part of the spec hash: changing it replaces the VM |
All three checks run before any backend call. The guide documents the ignored
field and that catalogue edits affect only new VMs.
**Evidence:** contract §2.1 (reject early, MAY ignore with documentation);
`IsHTTPURL` in `internal/provider/config_kvm.go`.
## 4. Image source
**Answer:** a catalogue in the config block (`images.<key>.url`, optional
`sha256`, `format` qcow2 or raw, `firmware` uefi or bios) or a URL given
directly in `placement.image`. The provider downloads the image into the
host's storage pool on first use and caches it there as a base volume named
`base-<entry>-<key>.<format>` (for a URL: `<key>` is the first 16 hex of
sha256(url)), where `<entry>` is the provider entry's name, with a sidecar
marker volume `base-<entry>-<key>.<format>.sha256` holding the digest of
the verified upload, written last. The entry is in the name because the `kvm`
provider builds `base-<key>.<format>` in the same pool from the same catalogue
shape, and two entries pointing one key at different URLs would otherwise share
a file; the cost is one copy per entry per host, and the entry's name is checked
against `kvmIdentifier` at construction. A base volume without a marker is
incomplete and is deleted and redone. When `sha256` is configured and differs
from the marker's content, the image is **not** replaced: that is `ErrPermanent`
naming both digests, because overlays back onto the base file and libvirt
unlinks it under running domains without complaint. The download streams
from the URL through the operator into `StorageVolUpload` with a running
sha256; nothing is buffered. `Content-Length` is required because the volume's
capacity must be declared before the upload; a response without it is
`ErrPermanent` naming the URL. Per `(host, volume)` a keyed mutex makes
concurrent Creates download once. The download runs on a dedicated
connection to the host, not the pooled one, so a cancelled poll on another
Proxy cannot cut it off. After a successful upload `StoragePoolRefresh` makes
the file usable as a backing image.
Catalogue edit warning for the guide: overlays back onto the base file, so
evicting a base volume while Proxies use it corrupts those VMs. A `sha256`
change is refused for that reason; a `url` change is a silent no-op, since the
cache is keyed by entry and key. Add a new key for either.
**Evidence:** kvm-provider.md "The image cache" and "Known gaps"; execution log
of 2026-09-04 (raw qcow2 upload plus pool refresh accepted by libvirt 11.6;
435 MB trixie image); contract §1.4 (stream, no buffering) and §2.1
(catalogue changes).
## 5. Cloud-init delivery
**Answer:** a NoCloud seed ISO built in-process with
`github.com/kdomanski/iso9660`, volume label `cidata`, files `user-data` (the
Proxy's `CloudInit`, byte for byte, never parsed), `meta-data`
(`instance-id: <name>` and `local-hostname: <name>`, so a reboot is never taken
for a new instance) and `vendor-data`. The ISO is written into a temp
directory created under the default temp dir (`/tmp`, a 64Mi emptyDir in the
manager), uploaded as a raw volume `<name>-seed.iso` and attached as a CD-ROM.
The seed is rebuilt on every Create attempt: a previous attempt may have been
cut off mid-upload, and a truncated seed would boot a VM without cloud-init.
`vendor-data` carries a shell script that installs and starts
`qemu-guest-agent` with whichever of apt-get, dnf, yum, zypper or apk the image
has. It is a script and not a cloud-config on purpose: cloud-init merges a
vendor-data cloud-config under the user-data with mergers that keep the
user-data's value for every top-level key both define and never append lists,
so a Proxy with its own `packages:` would silently lose the agent. Scripts
bypass the merge. When the config's `vendorData` is set, the two travel as
separate parts of a MIME multipart (`text/plain` for the configured document,
`text/x-shellscript` for the agent script), headed by `MIME-Version` within
the first bytes. With `installGuestAgent: false` and no `vendorData` the file
is empty. Size: cloud-init has no practical limit for NoCloud; the ISO is a
few hundred KB.
**Evidence:** kvm-provider.md "What lands on the host"; execution log run 7
(vendor-data as YAML silently dropped) and run 8 (script survives); contract
§2.1 (byte for byte, never logged) and §11 (sentinel test).
## 6. Machine type mapping
**Answer:** a catalogue: `machineTypes.<key>` is `{vcpus, memoryMiB}`,
validated at load (both greater than zero, at least one profile). Disk size is
provider-wide: `diskSizeGb` (default 10) is the capacity of every VM's overlay
volume, thin-provisioned; cloud-init's growpart expands the root filesystem
into it.
**Evidence:** `KVMMachineType` and `validateKVM` in
`internal/provider/config_kvm.go`.
## 7. Network and IP discovery (`readiness`, `ip-source`)
**Answer:** `readiness`: `provisioning-window` (boot plus cloud-init, plus the
image download on a cold pool). `ip-source`: `discovered`. The VM gets one
virtio NIC on `hosts[].bridge`, which must lead to a LAN with DHCP that the
cluster can route to. The address comes from `DomainInterfaceAddresses`, guest
agent source first, then the ARP table (on beelink the agent answered about
20 s after start where ARP never did, because the host itself does not talk to
the guest). The first IPv4 that is not loopback, link-local or unspecified is
reported. One source failing means "no address yet" and `Get` returns
`StateProvisioning`, which is the ordinary boot on beelink: the agent is not up
while ARP answers with nothing. A connection already gone, or every source
failing, is `ErrTransient` instead — a running VM whose address cannot be read
is not a VM without an address, and reporting it as one would make the
reconciler clear its IP and the health engine drop its counters. The guest needs the agent: it is installed by the
vendor-data script (entry 5) unless `installGuestAgent: false`, which needs an
image that ships it or a network where ARP sees guest traffic. `Port` is not
used by the provider; nothing on the host filters it. Reachability is a
prerequisite in the guide: the operator's pod network and the clients must
route to the bridge's LAN.
**Evidence:** kvm-provider.md "Hypervisor prerequisites" and "Known gaps"
(IP discovery); execution log run 6 (agent 20 s, ARP never); contract §1.3
and §2.2 (Running only with IP).
## 8. GC marks and CreatedAt (`gc-mark`, `created-at`)
**Answer:** `gc-mark`: `written-metadata`. `created-at`: `written-at-create`.
The domain XML passed to `DomainDefineXML` carries, under `<metadata>`, an
element `<instance xmlns="https://gitea.home.hrajfrisbee.cz/kacerr/egress-proxies-operator/kvm2">`
with children `<managed>true</managed>`, `<uid>req.UID</uid>` and
`<createdAt>RFC3339 UTC</createdAt>`. The mark lands in the same call that
defines the domain, so there is no mark window. Read back with
`DomainGetMetadata(dom, DomainMetadataElement, uri, DomainAffectCurrent)`;
libvirt 11.6 returns the element without its namespace declaration, so the
parser matches on the local name `instance` only (the URI was already used to
select it). A domain without the element is not managed and reports an empty
UID. A `createdAt` that does not parse gives the zero time, which GC treats as
old; that is accepted. The volumes carry no mark (documented GC blind spot,
entry 24). The namespace is kvm2's own, so kvm2 never lists or adopts domains
made by the `kvm` provider; the two types may share a host.
**Evidence:** execution log run 3 (namespace asymmetry);
`DomainGetMetadata(Dom Domain, Type int32, Uri OptString, Flags DomainModificationImpact)`
in the go-libvirt module; contract §2.1 (mark in the create call) and §4.5.
## 9. State mapping (`provisioning-state`)
**Answer:** `provisioning-state`: `create-blocks`. Create defines and starts the
domain in one call and returns only once `DomainGetState` reports running, so
`Get` never sees a domain kvm2 defined but did not start. A domain that is
shut off later stopped on its own or was stopped by someone, and Stopped is
the right answer.
| Backend state (`DomainGetState`) | Lock or task | Provider state |
|---|---|---|
| nostate (0) | none | Provisioning (libvirt could not determine it; next poll decides) |
| running (1), blocked (2) | no IP found | Provisioning |
| running (1), blocked (2) | IP found | Running |
| paused (3), shutdown (4), shutoff (5), crashed (6), pmsuspended (7) | none | Stopped |
| any other code | none | Stopped |
libvirt has no locks or tasks; every RPC is synchronous.
**Evidence:** go-libvirt `DomainState` constants; contract §2.2 and §4.11.
## 10. Create: blocking and task tracking (`create-return`, `async-ops`)
**Answer:** `create-return`: `blocking`. `async-ops`: `none`. Create returns
when the domain is running, holding one reconcile worker for the duration,
including the first image download on a host (minutes on a cold pool; the
guide says so). The wait after `DomainCreate` is capped at 15 seconds:
`DomainCreate` is synchronous in libvirt, so the first `DomainGetState`
normally answers running and the cap only covers state settling. Two rules
keep the wait from pinning a worker: a state after the start that is neither
active nor `nostate` (shutoff, crashed, paused, pmsuspended, shutdown) is
`ErrTransient` at once, since polling will not bring that guest up; and a
domain found paused or pmsuspended before the start is `ErrPermanent`
(`cannot be started by the operator; resume or destroy it on the host`), an
admin's doing that `DomainCreate` would refuse and the reconciler must not
retry. Every RPC has its own deadline (entry 18).
Steps, each idempotent, in order, all under the per-name lock (entry 19):
1. Validate placement (entry 3). No backend call yet.
2. `DomainLookupByName`. Found: adoption guard (entry 2), then start it if it
is not running (paused or pmsuspended is `ErrPermanent`; `DomainCreate`;
`ErrOperationInvalid` means already running), poll as in step 9, return
the ID.
3. Pool space precheck (entry 13, quota).
4. Ensure the base image (entry 4): marker present and matching means done.
5. Overlay volume `<name>.qcow2`, qcow2 backed by the base file with
`diskSizeGb` capacity; `ErrStorageVolExist` means reuse (creation is
atomic, so an existing overlay is complete).
6. Seed volume `<name>-seed.iso`: delete if it exists, create, upload
(entry 5). Always rebuilt.
7. `StorageVolGetPath` for the overlay and the seed.
8. `DomainDefineXML` with file-type disks (entry 15's AppArmor note), the
metadata element, firmware per image.
9. `DomainCreate`, then poll `DomainGetState` until running: `nostate` is
polled again, any other inactive state is `ErrTransient` at once, and the
15-second cap is `ErrTransient` too.
10. Return `hosts/<host>/domains/<name>`.
A retry after a crash finds whatever the previous attempt got to: an existing
overlay is reused, a seed is rebuilt, a defined domain is started, a running
domain is returned. `ErrNoDomain` and `ErrNoStorageVol` inside Create (a
volume vanishing between steps) are `ErrTransient`, never `ErrNotFound`. The
log line for a Create records `cloudInitBytes`, never the content.
**Evidence:** contract §2.1, §4.2, §5 ("do not return NotFound from Create");
execution log run 1 (AppArmor).
## 11. Delete and secondary resources (`delete-return`, `secondary-resources`)
**Answer:** `delete-return`: `waits`. `secondary-resources`: `explicit`. Under
the per-name lock:
1. Parse the ID (`ErrPermanent` if malformed). A host no longer in the config
is a logged no-op success: the provider cannot reach it and refusing would
pin the finalizer; the guide tells admins to delete Proxies before retiring
a host.
2. `DomainLookupByName`; `ErrNoDomain` means the domain is already gone, skip
to step 5.
3. `DomainDestroy`; `ErrOperationInvalid` (not running) and `ErrNoDomain` are
success.
4. `DomainUndefineFlags` with `DomainUndefineNvram`,
`DomainUndefineManagedSave` and `DomainUndefineSnapshotsMetadata`
(UEFI domains keep NVRAM; undefine refuses without the flag); `ErrNoDomain`
is success.
5. Delete `<name>.qcow2` and `<name>-seed.iso` by name from the host's pool;
`ErrNoStorageVol` is success, and `ErrNoStoragePool` means nothing is left
to clean.
6. Return nil. The base image and its marker stay (shared cache).
Everything is gone when Delete returns nil, so `Get` afterwards is NotFound.
A second Delete during the first waits on the lock and then finds nothing.
Volumes that exist without a domain (a crash between step 5 and 8 of Create)
are removed by the same Proxy's next Delete but are invisible to GC (entry 24).
**Evidence:** kvm-provider.md "Known gaps"; execution log run 4 (NVRAM);
contract §2.3.
## 12. ListByTag scope and partial failure (`list-partial`)
**Answer:** `fail-whole`, per host, with single unreadable domains skipped.
Scope: every host in `hosts`, in config order, `ConnectListAllDomains` with
active and inactive domains. For each domain, read the metadata element; keep
those with `managed` true and build the `Instance` from the domain name and
the element alone: `ID`, `UID` and `CreatedAt`, which is all GC reads
(contract §2.4). `State` and `IP` are left empty; no state or address RPC is
made. A domain whose metadata cannot be read (mid teardown, corrupt element)
is skipped with a V(1) line, unless ctx ended, which fails the call; a host
that cannot be dialled or whose listing fails fails the whole call, so GC
skips this provider for the sweep rather than act on a partial view.
Filtering is client-side; libvirt has no label index. Cost: exactly one
`DomainGetMetadata` per domain per sweep, about half a millisecond each over
the tunnel.
**Evidence:** execution log run 6 (per-domain metadata cost); contract §2.4
and §4.7.
## 13. Error classification (`not-found`, `quota`)
**Answer:** `not-found`: `error-code`. `quota`: `detected`, from a pool-space
precheck rather than a backend error: before creating any volume, Create reads
`StoragePoolGetInfo` and compares `available` with the bytes it is about to
commit: the image's `Content-Length` when the base is not cached, plus
`diskSizeGb` in bytes, plus 16 MiB for the seed. Below that it returns
`ErrQuotaExceeded` naming the pool, available and needed, and the reconciler
waits `QuotaRetry`.
One `classify(op string, err error)`:
| Backend response | Class | Why |
|---|---|---|
| `libvirt.Error` code `ErrNoDomain`, `ErrNoStorageVol` | NotFound (Transient when `op == "create"`) | the object is gone; inside Create it vanished mid-call |
| `ErrNoStoragePool` | Permanent | config names a pool that does not exist |
| `ErrXMLError`, `ErrNoSupport`, `ErrInvalidArg` | Permanent | the request itself is wrong |
| `ErrAuthFailed`, `ErrAccessDenied` | Permanent | the SSH user may not use the socket |
| `ErrOperationInvalid` | Transient (Delete and Create treat the specific cases as success before classify) | state race; the next poll decides |
| any other `libvirt.Error` | Transient | unknown means retry |
| SSH: `ssh: unable to authenticate` (from `x/crypto/ssh`) | Permanent | the key is not in `authorized_keys` |
| SSH: `knownhosts.KeyError` with a known key that differs, or with no known key | Permanent | host key changed or missing from known_hosts; the admin must act |
| SSH key or known_hosts file missing (`fs.ErrNotExist`) | Transient | the Secret has not landed yet |
| SSH key present but does not parse | Permanent | wrong file in the Secret |
| `context.Canceled`, `context.DeadlineExceeded`, `net.OpError`, `io.EOF`, `io.ErrUnexpectedEOF` | Transient | network or cancellation |
| HTTP 404, 403, 410 on the image URL; missing `Content-Length`; sha256 mismatch | Permanent | the catalogue entry is wrong |
| HTTP 5xx, 429, network errors on the image URL | Transient | retry |
| an error already wrapped with a class (`errors.Is` any sentinel) | that class | validation and builders classify at the source |
| anything else | Transient | contract default |
The unit test table uses a `libvirt.Error` per code, the real SSH error
strings produced by `x/crypto/ssh` against the in-process server, and a real
`knownhosts.KeyError`.
**Evidence:** go-libvirt `ErrorNumber` constants and `Error{Code uint32, Message}`
(value receiver, so `errors.As` uses a `libvirt.Error` value); contract §5;
kvm-provider.md "Known gaps" (SSH auth retried forever in v1: closed here).
## 14. Credentials and TLS (`credentials`)
**Answer:** `mounted-files`. The config block names `sshKeyFile` (private key,
OpenSSH or PEM, parsed with `ssh.ParsePrivateKey`) and `knownHostsFile`
(OpenSSH `known_hosts`, parsed with `knownhosts.New`), both expected under the
existing optional `kvm-ssh` Secret mount at `/etc/proxy-operator/kvm`
(`config/manager/manager.yaml`; the mount is shared with the `kvm` provider and
its comment says so). `insecureIgnoreHostKey: true` replaces known_hosts with
`ssh.InsecureIgnoreHostKey()` and is logged as a warning by `New`. Both files
are read on every dial, so rotating the Secret takes effect on the next
connection without a restart. `New` tries to read them once to log a warning
when they are unusable, but never fails on them and never dials.
Minimum privilege: the SSH user must be able to open the libvirt socket
(member of the `libvirt` group); libvirt itself is used as `qemu:///system`
through the socket, so no further host permissions.
TLS: none (SSH provides transport security); `system-roots` for the image
download's HTTPS, which uses the distroless image's CA bundle.
Hygiene: the private key bytes and the known_hosts content never appear in
logs, spans or error strings; errors from parsing say "parsing ssh key file
<path>" plus the parser's message. Tested with a sentinel placed in a bad key
file.
**Evidence:** kvm-provider.md "Cluster-side setup"; execution log run 8
(lazy load: a missing Secret must not abort the provider set); contract §7.
## 15. Config block
**Answer:** the block is `kvm2:` and reuses the Go type `provider.KVMConfig`
from `internal/provider/config_kvm.go` unchanged, so an existing `kvm-home`
entry ports by changing `type: kvm` to `type: kvm2` and `kvm:` to `kvm2:`.
Validation calls the shared validator with the block name `kvm2` so messages
read `kvm2.hosts` and not `kvm.hosts`.
| Field | Required | Default | Validation | Ends up in |
|---|---|---|---|---|
| `sshKeyFile` | yes | | non-empty | dial |
| `knownHostsFile` | unless `insecureIgnoreHostKey` | | non-empty | dial |
| `insecureIgnoreHostKey` | | false | | dial, warning at `New` |
| `diskSizeGb` | | 10 | not negative | overlay capacity, quota precheck |
| `vendorData` | | | | seed `vendor-data` |
| `installGuestAgent` | | true | | seed `vendor-data` |
| `images.<key>` | | | key matches `kvmIdentifier` | volume name `base-<entry>-<key>.<format>` |
| `images.<key>.url` | yes per entry | | http(s) URL | download |
| `images.<key>.sha256` | | | 64 hex | marker comparison |
| `images.<key>.format` | | qcow2 | qcow2 or raw | base volume XML, overlay backing format |
| `images.<key>.firmware` | | uefi | uefi or bios | domain XML `<os>` |
| `machineTypes.<key>` | at least one | | vcpus and memoryMiB greater than zero | domain XML |
| `hosts[].name` | yes | | unique, matches `kvmIdentifier` | providerID segment |
| `hosts[].address` | yes | | non-empty `host:port` | TCP dial |
| `hosts[].user` | yes | | non-empty | SSH |
| `hosts[].socket` | | `/var/run/libvirt/libvirt-sock` | | SSH channel target |
| `hosts[].pool` | | `default` | | every storage RPC |
| `hosts[].bridge` | yes | | non-empty | domain XML NIC |
Wiring: `ProviderConfig.KVM2 *KVMConfig` with tag `kvm2,omitempty`;
`setBlocks` reports `kvm2`; the allow-list in `Config.validate` and the
constructor map in `cmd/main.go` both gain `kvm2` in the same commit.
**Evidence:** `KVMConfig`, `validateKVM`, `kvmIdentifier` in
`internal/provider/config_kvm.go`; contract §10.
## 16. Checker
**Answer:** one target per `hosts[]` entry, `Name` the host name, `Address`
the SSH address, checked concurrently in config order on a fresh dedicated
connection each time (dial, `ConnectGetLibVersion`, `ConnectGetHostname`,
`StoragePoolLookupByName`, `StoragePoolGetInfo`, close). It never touches the
pooled connection, so a check that runs out of time closes only its own
connection and cannot disrupt a reconcile. Cost: one SSH handshake and four
cheap RPCs per host per round.
Reasons: `Reachable`; `Timeout` (the round's ctx ended before the check
finished, whichever stage was running, the dial included, or an RPC's own
budget ran out once the connection was up); `Unreachable` (TCP connect or the
SSH transport failed, including on the dial's own 10 s budget while the round
still had time, or the link died mid-check); `Unauthorized` (SSH authentication refused, host key unknown or
changed); `Forbidden` (`ErrAccessDenied` or `ErrAuthFailed` from libvirt: the
socket refused the user); `APIError` (any other libvirt error);
`CredentialsUnavailable` (the key or known_hosts file is missing or
unparsable, i.e. the Secret is absent or wrong: fixed in the cluster, not on
the host); `PoolNotFound`; `PoolNotActive`.
Details on success: `user`, `hostname`, `libvirtVersion` (decoded
major.minor.micro), `pool`, `poolState`, `poolCapacityGiB`, `poolAvailableGiB`
(the closest thing to a capacity signal, and the same number the quota
precheck reads). The `status.type` comment in
`api/v1alpha1/providerhealth_types.go` gains `kvm2` and the CRD is
regenerated.
**Evidence:** contract §8; `StoragePoolGetInfo` returning state, capacity,
allocation, available; `ConnectGetLibVersion` returning major*1e6 + minor*1e3
+ micro.
## 17. Client and child spans (`client-kind`, `child-spans`)
**Answer:** `client-kind`: `rpc-over-ssh`. `child-spans`: `span-decorator`.
The seam is an interface over exactly the go-libvirt methods the provider
uses, with `ctx` as the first parameter of every method (go-libvirt's own
methods take none and block): `DomainLookupByName`, `DomainDefineXML`,
`DomainCreate`, `DomainDestroy`, `DomainUndefineFlags`, `DomainGetState`,
`DomainGetMetadata`, `DomainInterfaceAddresses`, `ConnectListAllDomains`,
`StoragePoolLookupByName`, `StoragePoolRefresh`, `StoragePoolGetInfo`,
`StorageVolLookupByName`, `StorageVolGetPath`, `StorageVolCreateXML`,
`StorageVolUpload`, `StorageVolDownload`, `StorageVolDelete`,
`ConnectGetLibVersion`, `ConnectGetHostname`, plus `Close`. One production
adapter wraps `*libvirt.Libvirt` and applies the per-call deadline
(entry 18); one in-memory fake implements the seam for unit tests. A dialer
seam (`dial(ctx, host) (backend, error)`) separates the SSH transport so tests
inject fakes and an in-process SSH server test covers the real dialer.
The decorator is applied once, inside the real dialer, at the point where a
connection is created, so provider code never sees an undecorated backend. Per
call it starts a span `kvm2.<Method>` (for example `kvm2.DomainDefineXML`) with
attributes `kvm2.host` and the identifiers passed (`kvm2.domain`, `kvm2.pool`,
`kvm2.volume`), records the libvirt error code on failure, and inside the span
writes the V(1) log line, so the line carries that span's ID. Two more spans:
`kvm2.dial` around the SSH handshake and libvirt connect (attributes address
and user) and `kvm2.download` around an image fetch (attributes URL host,
bytes, whether a sha256 was configured). All spans are children of the
decorator's `provider.create`, `provider.get`, `provider.delete`,
`provider.list` or the monitor's `providerhealth.check`. No CloudInit, key
material or upload bodies in attributes. A reflection test walks the seam
interface and asserts every method produces exactly one span and one log
line.
**Evidence:** contract §4.15, §4.18, §9; `tracing.Start` in
`internal/tracing/logger.go`.
## 18. Timeouts (`timeouts`)
**Answer:** `per-call`. The adapter runs each blocking go-libvirt call in a
goroutine and waits on the result, `ctx.Done()` and a timer:
| Call | Budget |
|---|---|
| TCP connect plus SSH handshake plus libvirt connect | 10 s |
| ordinary RPC | 30 s |
| `DomainCreate` | 90 s |
| `StorageVolUpload` (seed) | 60 s |
| `StorageVolUpload` (image), HTTP download | ctx, capped at 20 m |
| wait for running after `DomainCreate` | 15 s (entry 10) |
| HTTP client for images | no client timeout; the per-request ctx carries the cap |
When a call is abandoned (ctx cancelled or budget exceeded), the adapter
closes the underlying connection, because that is the only way to unblock
go-libvirt, and marks it dead so the next call redials; the abandoned call
returns `ErrTransient` wrapping the ctx or deadline error. Consequence,
documented in the guide: other RPCs in flight on that host's pooled connection
fail once with Transient and their callers retry. The image download uses a
dedicated connection so the expensive case (a cancelled poll killing a
435 MB upload) cannot happen. The keepalive (`keepalive@openssh.com` request
every 15 s) marks the connection dead on failure.
**Evidence:** contract §1.5 (no caller deadline; a hung ListByTag stalls GC),
§4.16; go-libvirt methods take no ctx.
## 19. Concurrency
**Answer:** shared state is the host connection table (one pooled connection
per host, dialled lazily, under a mutex with a per-host dial mutex so
concurrent callers dial once and share the result), the per-`(host, volume)`
download locks, and a lock per instance name. Both lock sets are one instance
each of `keyedlock.Locks` (`internal/provider/keyedlock`): a one-slot semaphore per key, so a waiter gives up when
its ctx ends, reference-counted so an entry is removed from the map once no
caller holds or waits for it and the map does not grow with every name ever
seen, while two waiters on one key always share one semaphore. Create and
Delete for the same name run under that name's lock, so two Deletes (GC and
the reconciler) never race on destroy and undefine, and a Delete cannot
interleave with a Create's steps. Get and ListByTag take no name lock. go-libvirt multiplexes
RPCs on one connection, so calls for different names proceed concurrently
on the shared connection. Three reconciles, a GC sweep and a check round at
once are safe; the check uses its own connections.
**Evidence:** contract §1.2 (concurrency), §4.15.
## 20. Logging (`logging`)
**Answer:** `tiered`. Logger from `logf.FromContext(ctx)` on every call,
named `kvm2`, with `provider` on every line and `host` and `name` on every
line about an instance.
- V(1): one line per operation outcome (`kvm2 domain ensured running`,
`kvm2 domain fetched` with state, ip and uid, `kvm2 domain deleted`,
`kvm2 domains listed` with counts), one `libvirt call` line per RPC from the
decorator (`rpc`, identifiers, `duration`, on failure `error`, `libvirtCode`
and `libvirtError` name), connection events (dialled, dropped with reason),
and a line per failed backend call with the raw error.
- V(2): the same lines also carry domain and volume XML, the metadata element,
interface addresses, listed domain names, and the resolved Create parameters
(`cloudInitBytes`, never the content).
- No V(5) wire tier; go-libvirt exposes no hook for it.
- Never logged at any level: `CloudInit`, key material, known_hosts content,
upload or download bodies.
**Evidence:** contract §4.17, §9.
## 21. Tests (`backend-test`)
**Answer:** `backend-test`: `integration-tag`. Unit tests run over the
in-memory fake behind the seam (and an in-process `x/crypto/ssh` server for
the dialer); the integration test is `kvm2_integration_test.go` behind
`//go:build integration`, skipped unless `KVM2_TEST_HOST`, `KVM2_TEST_USER`,
`KVM2_TEST_SSH_KEY`, `KVM2_TEST_BRIDGE` and `KVM2_TEST_IMAGE_URL` are set,
optional `KVM2_TEST_KNOWN_HOSTS` (without it host keys are not verified and the
log says so), `KVM2_TEST_IMAGE_SHA256`, `KVM2_TEST_POOL`, `KVM2_TEST_SOCKET`,
`KVM2_TEST_TIMEOUT` (default 20m), `KVM2_TEST_KEEP`, `KVM2_TEST_LOG_LEVEL`;
run with `make test-integration-kvm2`. It asserts, in order: `New` returns
without dialling; `Check` reports the host Ready with details; `Create`
returns `hosts/<host>/domains/<name>`; a second `Create` returns the same ID
and the host's domain count is unchanged; `Get` reaches Running with an IPv4
within the budget and its `UID` and `CreatedAt` equal what Create wrote;
`ListByTag` contains it; `Delete` returns nil; `Get` is `ErrNotFound`;
`Delete` again returns nil; the pool holds the base image and its marker and
no `<name>.qcow2` or `<name>-seed.iso`; with an in-memory span recorder,
`kvm2.DomainDefineXML` and `kvm2.dial` spans are children of
`provider.create`.
Contract §11 list, with the test that covers each item:
- [x] repeat `Create`: `TestCreate_repeatReturnsSameIDAndCreatesNothing`
- [x] `Get` after `Delete`: `TestGet_afterDeleteReturnsNotFound`
- [x] `Delete` twice: `TestDelete_twiceSucceeds`
- [x] malformed ID: `TestGet_malformedIDIsPermanent`, `TestDelete_malformedIDIsPermanent`, `TestParseProviderID_malformed`
- [x] unknown state maps to `StateStopped`: `TestMapState_table`
- [x] `StateRunning` requires an IP: `TestGet_runningWithoutIPIsProvisioning`, `TestGet_runningWithIPIsRunning`
- [x] `UID` and `CreatedAt` read back: `TestGet_readsUIDAndCreatedAtFromBackend`
- [x] `ListByTag` excludes unmarked instances: `TestListByTag_excludesUnmarkedDomains`, `TestListByTag_emptyUIDStillListed`
- [x] `Get` during an in-progress delete: `TestGet_duringDeleteStillReturnsInstance`
- [x] `Get` during an asynchronous operation or lock: `TestGet_backendBusyIsTransientNeverNotFound`
- [x] never started is never `StateStopped`: `TestCreate_returnsOnlyWhenRunning`
- [x] cancelled `ctx` returns promptly: `TestProvider_cancelledContextReturnsPromptly`, `TestCall_cancelUnblocksRPC`
- [x] `classify` table with captured responses: `TestClassify_table`
- [x] Checker reasons, expired `ctx`, stable order: `TestCheck_reasonsTable`, `TestCheck_expiredContextReturnsPromptly`, `TestCheck_oneEntryPerHostInConfigOrder`
- [x] sentinel tests for `CloudInit` and the credential: `TestCreate_cloudInitSentinelNeverLoggedOrInErrors`, `TestSSH_keyMaterialNeverInErrorsOrLogs`
- [x] child spans under the operation span: `TestObserve_childSpansUnderParent`, `TestObserve_everySeamMethodInstrumented`
- [x] real-backend lifecycle test: `TestKVM2_lifecycle` and `TestKVM2Check_realHost` in `kvm2_integration_test.go`
Further tests per file are listed in the plan
(`docs/plans/2026-09-16-1333-kvm2-provider.md`).
**Evidence:** contract §11; docs/testing.md conventions.
## 22. Deployment
**Answer:** the existing `kvm-ssh` mount is reused; the rest is samples and a Makefile target:
- `config/manager/manager.yaml`: no new mount; the existing optional `kvm-ssh`
Secret at `/etc/proxy-operator/kvm` serves both `kvm` and `kvm2`, and its
comment says so. `/tmp` is already an emptyDir.
- `config/manager/providers_config.yaml` and
`config/samples/providers-config.yaml`: a commented `kvm2-home` entry, the
same block as `kvm-home` under `kvm2:`.
- `config/samples/proxy_kvm2.yaml`, listed in `config/samples/kustomization.yaml`:
provider `kvm2-home`, zone `beelink` (matching the shipped config, unlike the
`kvm` sample's `nas1`), machine type `small`, image `debian-13`, a cloud-init
that installs Squid.
- Makefile: `test-integration-kvm2` (`go test -race -tags=integration -count=1
-v -timeout 40m -run TestKVM2 ./internal/provider/kvm2/`). The Secret helper
`make kvm-ssh-secret` already exists and is reused.
- RBAC: none; the provider does not talk to the cluster's API.
- `hack/providers-dev.yaml`: no entry (needs credentials).
**Evidence:** contract §12; `config/manager/manager.yaml`.
## 23. Docs
**Answer:** `docs/kvm2-provider.md` (hypervisor prerequisites including OVMF
and the `libvirt` group and a bridge with DHCP; cluster-side setup reusing the
`kvm-ssh` Secret; config reference; placement mapping; what lands on the host
including the metadata namespace; the image cache and the quota precheck; the
health check reason table; logging tiers and span names; migrating an entry
from `kvm`, which requires deleting its Proxies first because kvm2 does not
adopt v1 domains; known gaps). A fourth column in the three tables of
`docs/providers-existing.md` and a kvm2 section. `docs/architecture.md`:
components row, a `kvm2` block in §5, the span topology in §10, a §11 row, and
Decisions bullets (dedicated download connection, checks on fresh connections,
own metadata namespace, quota precheck, SSH auth failures Permanent). A README
row. `docs/testing.md` rows and a kvm2 integration section. A `CHANGELOG.md`
entry after the user confirms.
**Evidence:** contract §12.
## 24. Known limits accepted at design time
**Answer:** nine limits, each with its workaround:
- **Create blocks**, including the first image download per host, holding one
of three reconcile workers. Workaround: warm the cache with one Proxy per
image per host before creating many.
- **Volumes without a domain are invisible to GC** (crash between volume
creation and define). The same Proxy's next reconcile removes them; a Proxy
deleted for good leaves them for `virsh vol-list`.
- **A host removed from the config orphans its VMs**: Delete is a no-op
success for it and GC does not sweep it. Delete the Proxies first.
- **No adoption of `kvm` (v1) domains**: switching an entry's type requires
deleting its Proxies first, or the old domains stay behind.
- **Cancelling one call drops the host's pooled connection**, so other RPCs
in flight on that host fail once with Transient and retry. The download is
isolated on its own connection.
- **`Content-Length` is required** for image downloads; a chunked mirror is
rejected as Permanent.
- **A paused or pmsuspended domain is not started.** A repeat `Create` for
a domain in that state is `ErrPermanent` (entry 10), so the Proxy latches
as Failed until its spec changes, rather than retrying a `DomainCreate`
that libvirt refuses. Resume the domain on the host (`virsh resume`) and
touch the spec, or delete the Proxy.
- **No v1 log-format compatibility**: the `libvirt call` lines have the same
keys where the same facts exist, but dashboards keyed on `kvm` logger names
need `kvm2`.
- Contract §13 limits that apply unchanged: config read once per process
(adding a host needs a restart; credentials do not), no installation identity
in GC, an unconfigured provider latches the Proxy.
**Evidence:** contract §13; entries 4, 8, 11, 18 above.
docs/kvm2-provider.md
+680
View File
@@ -0,0 +1,680 @@
# The kvm2 provider
Runs Managed proxies as VMs on your own libvirt/KVM hypervisors. It is the
second-generation libvirt provider, built clean-room from
[provider-contract.md](provider-contract.md) and the design in
[kvm2-provider-spec.md](kvm2-provider-spec.md), and it ships side by side
with the first-generation `kvm` provider: both types can be configured at
once, both take the same config block shape, and both read the same
`kvm-ssh` Secret. What kvm2 adds over kvm: a child span per libvirt RPC,
a pool-space precheck that maps to the reconciler's quota wait, SSH
credentials re-read on every dial, image downloads and health checks on
connections of their own, an adoption guard on the owning UID, and SSH
authentication failures that stop retrying.
Like kvm, the operator reaches each host over SSH, forwards the libvirt
unix socket through that connection and speaks libvirt's own RPC over it,
so the hosts need nothing beyond `sshd` and `libvirtd` and the operator
image carries no libvirt libraries. VMs boot a stock distro cloud image
that the operator downloads into the host's storage pool on first use,
with the Proxy's cloud-init delivered on a NoCloud seed ISO built
in-process.
## Hypervisor prerequisites
Per host, once:
1. **libvirt and KVM** (`libvirtd` running, `/dev/kvm` present).
2. **An SSH user in the `libvirt` group**, so it may open
`/var/run/libvirt/libvirt-sock`. The operator authenticates with a
private key; nothing else on the host is touched. The socket is used
as `qemu:///system`, so no further host permissions are needed.
3. **A bridge to a network with DHCP** (for example `br0` bridged onto
the LAN). Each VM gets one virtio NIC on it and takes its IP from that
network's DHCP. The operator's pod network and the crawler clients
must route to those IPs: the health engine probes `http://<ip>:<port>`
through the proxy and clients get the same address from the discovery
API. Nothing on the host filters the port; the provider does not use
`spec.port` at all.
4. **A storage pool** (default `default`, usually `/var/lib/libvirt/images`)
with room for one base image per distro and provider entry (a few
hundred MB each) plus the
per-VM overlays (`diskSizeGb` each, thin-provisioned). The pool must be
running: a stopped pool fails the health check with `PoolNotActive` and
fails `Create` permanently.
5. **UEFI firmware for guests**: the `ovmf` package on Debian/Ubuntu,
`edk2-ovmf` on Fedora/RHEL. VMs boot UEFI by default with secure boot
off (libvirt picks the OVMF binary itself); a domain start fails with
"no firmware found" without it. Only `firmware: bios` images skip this.
6. **Outbound internet from the VMs**, in practice: a vendor-data script
installs `qemu-guest-agent` from the distro mirrors on first boot, and
the agent is how the operator learns the VM's address. Without it the
operator falls back to the host's ARP table, which only sees traffic
addressed to the host itself, so on a plain bridge a VM without the
agent stays `Provisioning`.
Check from the operator's point of view before wiring it in:
```sh
ssh novakj@beelink virsh -c qemu:///system list --all # user can drive libvirt
ssh novakj@beelink virsh -c qemu:///system pool-info default
ssh novakj@beelink ip link show br0
```
## Cluster-side setup
1. Create the SSH Secret from a key pair dedicated to the operator and a
`known_hosts` file holding every host's key. It is the same `kvm-ssh`
Secret the `kvm` provider uses; one Secret serves both types, and if
you already have it there is nothing to do here.
```sh
# once: a key pair for the operator; the .pub goes on every host
ssh-keygen -t ed25519 -f ~/.ssh/proxy-operator_ed25519 -N '' -C proxy-operator
ssh-copy-id -i ~/.ssh/proxy-operator_ed25519.pub novakj@beelink
# once per host: its host key, under the address the *operator* will
# dial (hosts[].address), which may differ from what your laptop uses
ssh-keyscan -H beelink.tail8c69c.ts.net >> ~/.ssh/kvm_known_hosts
# the Secret: the PRIVATE key (not the .pub) plus known_hosts
kubectl -n egress-proxies-operator-system create secret generic kvm-ssh \
--from-file=id_ed25519=$HOME/.ssh/proxy-operator_ed25519 \
--from-file=known_hosts=$HOME/.ssh/kvm_known_hosts
```
`make kvm-ssh-secret` runs that last command as an apply, so it also
updates an existing Secret; override the paths with
`KVM_SSH_KEY=... KVM_KNOWN_HOSTS=...`. Putting the `.pub` in the
Secret is the classic mistake: the target's `kvm-ssh-secret` check
refuses a file without `PRIVATE KEY` in it, and the provider itself
would fail every call permanently with a "parsing ssh key file" error.
Without `known_hosts`, set `insecureIgnoreHostKey: true` and accept
that a MITM could hand you a fake hypervisor.
`config/manager/manager.yaml` mounts the Secret (optional) at
`/etc/proxy-operator/kvm` as two files, `id_ed25519` and
`known_hosts`, which is what `sshKeyFile` and `knownHostsFile` in the
provider config point at. The deployment starts fine without the
Secret, even with a kvm2 provider configured: `New` only checks that
the files exist and logs a `WARNING:` line when one does not; the
provider's proxies fail transiently and recover once the Secret lands.
**Rotating the key or a host key needs no restart.** Both files are
read and parsed on every dial (`loadCredentials` in
`internal/provider/kvm2/ssh.go`); nothing is cached. Re-run the
command or the make target, and the next connection, health check
included, uses the new material. Connections that are already open
stay open until they drop.
2. Add a provider entry to the providers config
(`config/manager/providers_config.yaml` in-cluster; full sample in
`config/samples/providers-config.yaml`). The type is `kvm2` and the
block is `kvm2:`; its fields are those of the `kvm` block:
```yaml
- name: kvm2-home
type: kvm2
kvm2:
sshKeyFile: /etc/proxy-operator/kvm/id_ed25519
knownHostsFile: /etc/proxy-operator/kvm/known_hosts
images:
debian-13:
url: https://cloud.debian.org/images/cloud/trixie/latest/debian-13-generic-amd64.qcow2
machineTypes:
small: {vcpus: 1, memoryMiB: 1024}
hosts:
- name: beelink
address: beelink.tail8c69c.ts.net:22
user: novakj
bridge: br0
```
Validation messages name the block, so a mistake reads
`kvm2.hosts[0]: bridge is required`, not `kvm.hosts`. The providers
config is read once per process: adding a host or an image entry needs
a manager restart; credentials do not.
3. Apply a Proxy (`config/samples/proxy_kvm2.yaml`) and watch it:
```sh
kubectl get px -w
# NAME MODE PROVIDER PHASE IP HEALTHY
# proxy-kvm2-sample Managed kvm2-home Provisioning
# proxy-kvm2-sample Managed kvm2-home Running 192.168.1.73
# proxy-kvm2-sample Managed kvm2-home Ready 192.168.1.73 True
```
The first Proxy for a given image on a given host sits in
`Provisioning` for the length of the image download; the operator log
shows `fetching base image into host pool` with the URL and size and
`base image cached` with the size, duration and digest.
## Config reference
The block is `kvm2:` and its Go type is the shared `KVMConfig` in
`internal/provider/config_kvm.go`, so the fields, defaults and validation
are the same as for `kvm`. Defaults are applied by `withDefaults` in
`internal/provider/kvm2/kvm2.go`.
| Field | Required | Default | Meaning |
|---|---|---|---|
| `sshKeyFile` | yes | | private key used for every host; OpenSSH or PEM, parsed on every dial |
| `knownHostsFile` | unless `insecureIgnoreHostKey` | | OpenSSH `known_hosts` to verify host keys against, read on every dial |
| `insecureIgnoreHostKey` | | `false` | skip host-key verification entirely; `New` logs a `WARNING:` line when it is on |
| `diskSizeGb` | | `10` | capacity of each VM's overlay disk; cloud-init's growpart expands the root fs into it. Also what the pool-space precheck reserves per VM |
| `vendorData` | | | extra cloud-init vendor-data placed on every seed next to the Proxy's user-data. Beware that cloud-init drops any top-level key of a vendor-data **cloud-config** that the user-data also defines. Shipped as one part of a MIME multipart alongside the agent script when `installGuestAgent` is on |
| `installGuestAgent` | | `true` | add the vendor-data script that installs and starts `qemu-guest-agent`. Turn off for images that ship it already or cannot reach a package mirror; IP discovery then depends on the host's ARP table |
| `images.<key>` | | | the key names the image in `placement.image`. Letters, digits, dots, dashes and underscores only: it becomes the cached volume name `base-<key>.<format>` |
| `images.<key>.url` | yes per entry | | http(s) URL of a cloud image (qcow2 or raw). The server must send `Content-Length` |
| `images.<key>.sha256` | | | expected digest, 64 hex characters; verified after download, a change triggers a re-download |
| `images.<key>.format` | | `qcow2` | `qcow2` or `raw` |
| `images.<key>.firmware` | | `uefi` | `uefi` (OVMF, secure boot off) or `bios` (SeaBIOS) for images without an EFI system partition |
| `machineTypes.<key>` | at least one | | `{vcpus, memoryMiB}` size profile, both greater than zero |
| `hosts[].name` | yes | | the `placement.zone` value that selects this host; unique. Letters, digits, dots, dashes and underscores only: it becomes part of the providerID `hosts/<name>/domains/<vm>` |
| `hosts[].address` | yes | | SSH endpoint, `host:port` |
| `hosts[].user` | yes | | SSH user, member of the `libvirt` group |
| `hosts[].socket` | | `/var/run/libvirt/libvirt-sock` | libvirt socket path on the host |
| `hosts[].pool` | | `default` | storage pool for base images, overlays and seeds |
| `hosts[].bridge` | yes | | host bridge for the VM's NIC |
## Placement
All three fields are checked before any host is dialled. An unknown value
is `ErrPermanent`, which the reconciler latches as `Failed` with the
message below until the spec changes.
| Proxy field | Meaning for kvm2 | Error when unknown |
|---|---|---|
| `placement.zone` | host name from `hosts`. May be omitted when exactly one host is configured; otherwise required | `placement.zone "x" is not in kvm2.hosts`; with several hosts and no zone, `placement.zone is required when kvm2.hosts lists more than one host` |
| `placement.machineType` | key in `machineTypes` | `placement.machineType "x" is not in kvm2.machineTypes` |
| `placement.image` | key in `images`, or an `http(s)://` URL used directly with the defaults (`qcow2`, `uefi`) | `placement.image "x" is neither a key in kvm2.images nor an http(s) URL` |
| `placement.region` | ignored. It is still part of the spec hash, so changing it replaces the VM and does nothing else | |
The messages come from `Create` in `internal/provider/kvm2/create.go`,
`host` in `kvm2.go` and `resolveImage` in `image.go`. Fix the spec (which
replaces the VM) or the config. Catalogue edits affect only new VMs: the
spec hash covers the placement values, not what the config maps them to.
## What lands on the host
For a Proxy whose deterministic name is `proxy-abc…`, on the chosen
host's pool:
| Object | Name | Lifetime |
|---|---|---|
| base image | `base-<entry>-<key>.<format>`, where `<entry>` is the provider entry's name (for a URL image, `<key>` is the first 16 hex characters of the URL's sha256) | cached; survives Proxy deletion |
| image marker | `base-<entry>-<key>.<format>.sha256`, a 64-byte raw volume holding the digest | with the base image |
| overlay disk | `proxy-abc….qcow2`, qcow2 backed by the base file, `diskSizeGb` capacity | with the Proxy |
| seed ISO | `proxy-abc…-seed.iso`, raw | with the Proxy |
| domain | `proxy-abc…` | with the Proxy |
The names are built in `internal/provider/kvm2/volume.go`. The base image
carries the **provider entry's name** because the file is shared by every
Proxy on that host but must not be shared by two entries: the `kvm`
provider builds `base-<key>.<format>` in the same pool from the same
catalogue shape, and two kvm2 entries pointing one key at different URLs
would otherwise agree on the name and disagree on the contents. The cost
is one copy of each image per entry per host. For that reason the entry's
name must itself be a usable file name — letters, digits, dots, dashes and
underscores — which `New` checks at startup, since provider names are
otherwise only required to be non-empty and unique.
The domain (`buildDomainXML` in `xml.go`) is a `q35` machine with a
host-passthrough CPU, ACPI and APIC, memory in MiB and vcpus from the
machine type, the overlay on `vda` (virtio) and the seed as a read-only
SATA cdrom on `sda`, one virtio NIC on the bridge, a serial console on a
pty, the guest-agent channel `org.qemu.guest_agent.0`, a virtio RNG fed
from `/dev/urandom`, and no graphics. Both disks are `type="file"` with
the paths `StorageVolGetPath` returned: AppArmor derives QEMU's profile
from those paths, and volume-type disks fail with permission denied on
stock Debian and Ubuntu hosts. UEFI images get `<os firmware="efi">` with
the `secure-boot` feature disabled; BIOS images get neither.
Its `<metadata>` carries one element in kvm2's own namespace,
`https://gitea.home.hrajfrisbee.cz/kacerr/egress-proxies-operator/kvm2`
(`metadataURI` in `metadata.go`):
```xml
<instance xmlns="https://gitea.home.hrajfrisbee.cz/kacerr/egress-proxies-operator/kvm2">
<managed>true</managed>
<uid>0f3a…</uid>
<createdAt>2026-09-16T17:42:11Z</createdAt>
</instance>
```
That is how orphan GC recognises its own VMs, since libvirt has no
labels, and how a repeat `Create` knows whether an existing domain of the
same name belongs to this Proxy: a domain without the element is a
permanent error (`carries no kvm2 mark; it was not created by this
provider`), a different non-empty `uid` is a permanent error naming both
UIDs, and a marked domain with a matching or empty `uid` is adopted. The
namespace is kvm2's own, so the `kvm` provider's domains are invisible to
it (see *Migrating from kvm*).
The seed (`nocloud.BuildSeedISO`, shared with the proxmox provider in
`internal/provider/nocloud`) is an ISO 9660 image labelled
`cidata` with three files. `user-data` is the Proxy's `cloudInit`, **byte
for byte**: any format cloud-init accepts works, including a Jinja
template, a MIME archive, a shell script or a gzipped payload; the
operator never parses or re-renders it. `meta-data` holds
`instance-id: <name>` and `local-hostname: <name>`, so a reboot is never
taken for a new instance. `vendor-data` carries the guest-agent
bootstrap. The ISO is staged in a temp directory under `/tmp` (the
manager's 64Mi emptyDir), assembled in memory (tens of KiB) and uploaded
as a raw volume; the staging directory is removed before the call
returns. The seed is rebuilt on every `Create` attempt, because a
previous attempt may have been cut off mid-upload and a truncated seed
would boot a VM without cloud-init.
The guest agent rides along in `vendor-data` as a **shell script**
(`nocloud.GuestAgentScript`), not a cloud-config, on purpose:
cloud-init merges a vendor-data cloud-config under the user-data with
mergers that keep the user-data's value for every top-level key both
define and never append lists, so a Proxy carrying its own `packages:`
would silently lose the agent. Scripts bypass the merge. The script is
idempotent, picks whichever of `apt-get`, `dnf`, `yum`, `zypper` or `apk`
the image has, enables the service with `systemctl` or OpenRC, and logs
into `/var/log/cloud-init-output.log`. With `vendorData` also configured,
the two travel as separate parts of a MIME multipart (`text/plain` for
your document, `text/x-shellscript` for the script), with the
`MIME-Version` header first so cloud-init detects it. With
`installGuestAgent: false` the file is your `vendorData` alone, or empty.
## The image cache
The base image is downloaded **by the operator, through libvirt**, into
the host's pool the first time a Proxy asks for it on that host. Nothing
is pre-placed on the hypervisor and the operator needs no filesystem
access there. The bytes stream from the URL through the operator into
`StorageVolUpload` with a running sha256; nothing is buffered
(`fetchImage` and `uploadImage` in `internal/provider/kvm2/image.go`).
The upload runs on a **dedicated connection** to the host, dialled for
the transfer and closed after it, not on the pooled connection the other
operations share. A cancelled or overlong call kills the connection it
ran on (see *Timeouts and connections*), and a poll for some other Proxy
being cancelled must not cut off a 20-minute upload.
A body that fails part way through is reported to the caller **out of
band**: the reader handed to `StorageVolUpload` turns any source error
into a clean end of stream and keeps the real error for afterwards, so
the short upload is caught by the length and digest checks rather than
by the stream. This is not tidiness. go-libvirt answers a reader error
by parking its stream sender forever on a channel it no longer reads, so
handing it one wedges the call and leaks the sender with its 4 MiB
buffer; see *Known gaps*. For the same reason the download carries its
own deadline, a minute inside the transfer budget, so a stalled mirror
ends through its body rather than through the killed socket the budget
would otherwise leave behind.
The download needs a `Content-Length`: the volume's capacity is declared
before the upload starts. The request sends `Accept-Encoding: identity`
so a transparently gzipped body cannot hide the length. A response
without it, or with length 0, is a permanent error naming the URL, and so
is any 4xx status; 429 and 5xx are transient and retried.
The marker volume is written **last**, on the pooled connection, after
`StoragePoolRefresh` has made libvirt read the file's real format. Its
presence is what marks a base complete: a crash mid-download leaves a
base volume without a marker, which the next attempt deletes and redoes
(`cachedImage`). A re-download also happens when the marker's content is
not a 64-character hex digest, or when `sha256` is configured and differs
from it. Otherwise the cached copy is used for as long as it exists, even
if the URL's content changed upstream: a `latest` URL is pinned at the
moment of first fetch, per host. Concurrent Creates for the same image
on the same host queue on a per-(host, volume) lock; the first fetches,
the rest find the marker.
To evict manually:
```sh
ssh novakj@beelink virsh -c qemu:///system vol-list --pool default | grep base-kvm2-home-
ssh novakj@beelink virsh -c qemu:///system vol-delete --pool default base-kvm2-home-debian-13.qcow2.sha256
ssh novakj@beelink virsh -c qemu:///system vol-delete --pool default base-kvm2-home-debian-13.qcow2
```
Marker first: a base without its marker is refetched, a marker without its
base is cleaned up, and neither leaves a Proxy pointing at nothing.
**Warning: do not evict a base volume while Proxies using that image exist
on the host.** The overlays back onto the base file, and libvirt will
unlink it under running domains without complaint; the guests keep running
on the open file descriptor and fail at their next start. Delete those
Proxies first.
Changing `sha256` on an entry whose base is already cached is **refused**,
permanently and by name, rather than resolved by replacing the file — that
replacement was the same corruption, just automated. Add the image under a
new key and move Proxies over by changing their `placement.image` (which
replaces the VM), or delete the Proxies and evict the base by hand.
Changing `url` alone does nothing at all: the cache is keyed by entry and
key, so the new URL is never fetched. Add a new key for that too.
## Quota: the pool-space precheck
Before creating any volume, `Create` reads `StoragePoolGetInfo` and
compares the pool's free space with what it is about to commit
(`checkPoolSpace` and `neededBytes` in `image.go`): the image's
`Content-Length` when the base is not cached yet (found with a `HEAD`
request, or a `GET` whose body is closed at once when `HEAD` answers
anything but 2xx, since mirrors refuse it with 405, 403 or 501 alike; the
`GET`'s status is then what counts), plus `diskSizeGb` in bytes for the
overlay, plus 16 MiB for the
seed. Below that it returns `ErrQuotaExceeded` with a message naming the
pool, the bytes available and the bytes needed. The reconciler answers
quota with a wait, not a failure: the Proxy shows `Provisioned=False`
with reason `QuotaExceeded` and is retried after `QuotaRetry` (5
minutes), off the error backoff curve and never `Failed`. Free up the
pool or lower `diskSizeGb` and the next attempt goes through.
The precheck is the only quota signal; libvirt itself reports a full pool
as an ordinary error, which would be transient. The health check exposes
the same number as `poolAvailableGiB`, so a pool that is filling up shows
there before Creates start waiting.
A pool that is not running fails the precheck permanently (`storage pool
<pool> is not running`): start it with `virsh pool-start` and
`virsh pool-autostart`.
## Migrating from `kvm`
The `kvm2` block is the `kvm` block under another name, so an entry ports
by changing `type: kvm` to `type: kvm2` and `kvm:` to `kvm2:`. The
`kvm-ssh` Secret is shared; nothing changes on the hosts.
What does not port is the VMs. kvm2 marks its domains in its own metadata
namespace and neither lists nor adopts domains the `kvm` provider made:
`ListByTag` skips them, so orphan GC never sees them, and `Create` refuses
a domain of the same name that lacks the kvm2 mark with a permanent error
rather than start it (a Proxy recreated under the new entry gets the same
deterministic name, so this is what happens when step 1 below is skipped).
A Proxy's `spec.provider` is immutable, so an existing Proxy cannot be
moved to the new entry anyway. Do it in this order:
1. Delete the Proxies on the `kvm` entry and wait for them to go; the
`kvm` provider removes their domains and volumes.
2. Change the entry (or add a second one under a new name and keep the
old one for a while; both types may share a host).
3. Restart the manager, then recreate the Proxies with the new
`spec.provider`.
A `kvm` domain left behind on a host is invisible to kvm2 forever; clean
it up on the host with `virsh undefine --nvram <name>` and
`virsh vol-delete` for its `<name>.qcow2` and `<name>-seed.iso`.
The base image cache is **not** shared between the two types, and that is
deliberate: kvm names its copy `base-<key>.<format>` while kvm2 names its
own `base-<entry>-<key>.<format>`, so kvm2 downloads the image once more
per host rather than adopting a file another provider maintains. Nothing
needs deleting; the `kvm` base stays where it is and keeps serving the
overlays that back onto it. Budget the extra few hundred MB in the pool.
## Logging
The provider logs through controller-runtime's logger, named `kvm2`, with
`provider` (the configured name) on every line, and `host` and `name`
(the domain) on every line about an instance. Every line inside a traced
operation also carries `traceID` and `spanID`.
- `--zap-log-level=debug` (logr V(1)):
- one line per operation outcome: `kvm2 domain ensured running` with the
`id`, `kvm2 domain fetched` with `state`, `ip` and `uid`,
`kvm2 domain not found`, `kvm2 domain deleted`, `kvm2 domains listed`
with `hosts` and `managed` counts, `kvm2 host checked` with `ready`,
`reason` and `latency`; plus the steps inside a Create
(`kvm2 domain already defined`, `overlay volume created`,
`seed volume uploaded` with `bytes`, `domain defined` with `uid` and
`firmware`, `domain started`, `fetching base image into host pool`,
`base image cached`);
- **one `libvirt call` line per libvirt RPC**, written by the decorator
in `internal/provider/kvm2/observe.go` inside that RPC's span: `rpc`
(the go-libvirt method name, for example `DomainLookupByName`), the
identifiers it was given (`domain`, `pool`, `volume`), `duration`, and
on failure `error` plus `libvirtCode` and `libvirtError` (the code's
name, for example `ErrNoDomain`). A Get is four or five RPCs; a
Create with a cached image is around twenty;
- connection events from `internal/provider/kvm2/hosts.go`:
`libvirt host connected` with `address`, `user` and `duration`, and
`libvirt host connect failed` with the same plus `error` and `stage`
(`credentials`, `tcp`, `ssh` or `socket`);
- `kvm2 backend call failed` with `op` and the raw `error` once per
failed operation, before the error is returned;
- a line per domain that `ListByTag` skips, and a line when the image
marker forces a refetch.
- `--zap-log-level=2`: the `libvirt call` lines also carry the domain and
volume `xml`, the `metadata` element, the interface `addresses`, the
listed `domains`, the pool `state` and sizes and the volume `path`; the
`domain defined` line carries the `xml`; `kvm2 create resolved` logs the
resolved parameters (pool, bridge, machine type, sizes, image volume,
firmware) and `cloudInitBytes`; `kvm2 listed instance` logs each listed
domain.
- Never logged at any level: the cloud-init content, the SSH key, the
known_hosts content, and upload or download bodies. The seed and image
uploads report `offset` and `length` only. There is no raw wire tier
like the GCP provider's V(5); go-libvirt exposes no hook for it.
## Tracing
With an OTLP exporter configured ([architecture.md
§10](architecture.md#10-tracing-internaltracing)), every backend call is
a child span of the operation span the decorators open
(`provider.create`, `provider.get`, `provider.delete`, `provider.list`)
or of the monitor's `providerhealth.check`:
| Span | Around | Attributes |
|---|---|---|
| `kvm2.<Method>`, for example `kvm2.DomainDefineXML` | one libvirt RPC (`observe.go`) | `kvm2.host`, and the identifiers passed: `kvm2.domain`, `kvm2.pool`, `kvm2.volume`; on failure the status is Error and `kvm2.libvirt_code` carries the libvirt error number |
| `kvm2.dial` | the SSH handshake and the libvirt connect (`hosts.go`) | `kvm2.host`, `kvm2.address`, `kvm2.user` |
| `kvm2.download` | one image fetch, from the HTTP request to the marker (`image.go`) | `kvm2.host`, `kvm2.volume`, `kvm2.url_host`, `kvm2.sha256_configured`, and `kvm2.bytes` once the response arrived |
All spans are client spans. No attribute carries cloud-init, key material
or a URL beyond its host. The `libvirt call` log line for an RPC is
written inside that RPC's span, so its `spanID` points at the span.
## Timeouts and connections
No caller puts a deadline on the context, so every backend call carries
its own budget (`defaultBudgets` in `internal/provider/kvm2/backend.go`,
applied by `call`):
| Call | Budget |
|---|---|
| TCP connect plus SSH handshake plus libvirt connect | 10 s |
| ordinary RPC | 30 s |
| `DomainCreate` | 90 s |
| `StorageVolUpload` of at most 16 MiB (the seed, the marker) | 60 s |
| `StorageVolUpload` of an image, `StorageVolDownload`, and the whole image fetch from HTTP request to marker | 20 m; the caller's ctx may end it earlier |
| wait for running after `DomainCreate` | 15 s, polled every 500 ms (`defaultRunningWait` and `defaultRunningPoll` in `create.go`); `DomainCreate` is synchronous, so the first poll normally answers running, and a domain that is shut off, crashed or paused right after the start fails at once rather than waiting out the cap |
| the `HEAD` request of the pool-space precheck | 30 s |
| the courtesy `ConnectClose` when a connection is closed | 5 s |
go-libvirt's calls take no context and block, so the only way to abandon
one is to close the connection under it. That is what a cancelled or
overlong call does: the adapter marks the connection dead so the next call
redials, kills it by closing the SSH client's TCP connection, which takes
the forwarded socket with it, and returns `ErrTransient` wrapping the
context or deadline error. It marks dead before it kills, and closes the
TCP connection rather than the forwarded channel, because both of those are
lock-free: closing the channel writes a close message under the channel's
write lock, so it would queue behind a transfer stuck in a TCP write to a
host that has gone away — for as long as the kernel retransmits, which is
the one case the kill exists to survive. The consequence:
**other RPCs in flight on that host's pooled connection fail once with
`ErrTransient`** and their callers retry on a fresh connection. In the
reconciler that is one requeue with backoff; in GC it is one skipped
sweep.
Two callers are kept off the pooled connection for that reason
(`dialFresh` in `hosts.go`): the image upload, because a 435 MB transfer
must not die because a poll for another Proxy was cancelled, and the
health check, because a check that runs out of the monitor's timeout must
close only its own connection.
Otherwise there is one pooled SSH plus libvirt connection per host,
dialled lazily on the first call that needs it and shared by every
operation for that host (go-libvirt multiplexes RPCs on one connection).
Concurrent callers wait for one dial and share the result; waiting ends
with the caller's context. A `keepalive@openssh.com` request every 15 s
marks the connection dead when it fails or gets no reply within another
15 s, as does go-libvirt noticing the socket is gone; the next call
redials. Create and Delete for the same domain name are serialised on a
per-name lock, so two Deletes (GC and the reconciler) never race on
destroy and undefine, and a Delete cannot interleave with a Create's
steps; Get and ListByTag take no lock.
## Health check
The operator checks every configured host every `--provider-check-interval`
(default 1m) and publishes the result in the `ProviderHealth` object named
after the provider (`kubectl get ph kvm2-home -o yaml`) and in the
`proxy_operator_provider_target_up{provider,target}` gauge. One target per
`hosts[]` entry, in config order, all checked concurrently.
Each check dials a **fresh connection** to the host and closes it when
done (`probeHost` in `internal/provider/kvm2/check.go`); the pooled
connection is never touched, so a check that runs out of time cannot fail
a reconcile's call in flight, and a fresh dial picks up a rotated Secret
at once. The sequence is the SSH and libvirt handshakes, then four RPCs:
`ConnectGetLibVersion`, `ConnectGetHostname`, `StoragePoolLookupByName`
and `StoragePoolGetInfo` on the host's `pool`. The cost per host per
round is one sshd session and those four calls.
Every target's `details` carry `user` and `pool`. As facts arrive they
gain `libvirtVersion` (decoded `major.minor.micro`), `hostname`,
`poolState` (virsh's word: `running`, `inactive`, ...),
`poolCapacityGiB` and `poolAvailableGiB`, so a target that fails late in
the sequence still says what the host answered before. The free space is
the number the quota precheck compares against.
A failing target says where the check broke (`checkReason` in `check.go`):
| Reason | Meaning | Look at |
|---|---|---|
| `Reachable` | ready | |
| `CredentialsUnavailable` | the key or known_hosts file is missing or does not parse | the `kvm-ssh` Secret (see *Cluster-side setup*); fixed in the cluster, not on the host |
| `Unreachable` | the TCP connect failed or ran out of the 10 s dial budget while the round still had time, the SSH transport failed for a reason other than authentication, the forwarded libvirt socket was refused, or the link died mid-check | DNS, Tailscale or a firewall between the cluster and `address`; `libvirtd` running and the `socket` path |
| `Unauthorized` | the SSH server rejected the key, or the host key is missing from known_hosts or has changed | `user`'s `authorized_keys`; the known_hosts entry (re-run `ssh-keyscan` and update the Secret) |
| `Forbidden` | SSH worked, but libvirt refused the user (`ErrAccessDenied` or `ErrAuthFailed`) | `user` in the `libvirt` group |
| `Timeout` | the monitor's `--provider-check-timeout` (default 15s) ran out before the check finished, whichever stage was running, the dial included; or the connection was up and an RPC's own budget ran out | a hung `libvirtd`, a saturated link, or a round timeout shorter than the dial takes |
| `PoolNotFound` | there is no pool named `pool` | `virsh pool-list --all` |
| `PoolNotActive` | the pool exists but is not running | `virsh pool-start <pool>` and `virsh pool-autostart <pool>` |
| `APIError` | any other libvirt error | the message carries libvirt's own text |
## Error classes
Every error is classified once, in `classify` in
`internal/provider/kvm2/errors.go`, with errors that carry a class from
their source (validation, the image fetch, the dial) keeping it. The
reconciler latches `ErrPermanent` as `Failed` until the spec changes,
waits 5 minutes on `ErrQuotaExceeded`, recreates on `ErrNotFound` from
`Get`, and retries everything else with backoff.
| Class | What lands there |
|---|---|
| `ErrPermanent` | placement values not in the catalogue; a malformed providerID; a host no longer in the config (from `Get`; `Delete` treats it as done); libvirt `ErrXMLError`, `ErrNoSupport`, `ErrInvalidArg`, `ErrAuthFailed`, `ErrAccessDenied` and `ErrNoStoragePool`; a pool that is not running; SSH authentication refused (`unable to authenticate`); a host key missing from known_hosts or changed (`knownhosts.KeyError`); a key or known_hosts file that exists but does not parse; a domain of the same name owned by another UID or carrying no kvm2 mark; a domain of the same name that is paused or pmsuspended; corrupt metadata on a domain; an image URL answering 4xx, a response without `Content-Length` or with length 0, a sha256 that does not match the configured one |
| `ErrQuotaExceeded` | the pool-space precheck |
| `ErrNotFound` | libvirt `ErrNoDomain` or `ErrNoStorageVol` from `Get`, `Delete` or `ListByTag`; inside `Create` the same codes are `ErrTransient`, because the contract forbids NotFound from Create |
| `ErrTransient` | everything else: a missing key or known_hosts file (the Secret has not landed yet), TCP and network errors, context cancellation, an abandoned or budget-exceeded call, a connection killed by another caller, libvirt `ErrOperationInvalid` and every unrecognised libvirt code, an image URL answering 429 or 5xx or ending the body early, every address source failing on an active domain, the wait for running expiring or the domain going shut off or crashed right after its start |
Error strings name paths, hosts, volumes and libvirt's own text. They
never contain key bytes, the known_hosts content or cloud-init.
## Known gaps
- **An upload that libvirtd itself fails leaks a goroutine, upstream.**
go-libvirt's `requestStream` hands its stream sender an unbuffered
channel and then, on the error path, returns without ever reading it:
```go
abort := make(chan bool) // rpc.go
outErr := make(chan error)
go func() { outErr <- l.socket.SendStream(...) }()
resp, err = l.processIncomingStream(c, in)
if err != nil {
abort <- true
return resp, err // outErr is never drained
}
```
Whenever the daemon errors mid-upload — a pool that fills, a volume
removed underneath — the sender parks on `outErr` for good, taking its
4 MiB buffer with it. Closing the socket does not help: both ends are
blocked on channel sends, not on I/O. kvm2 avoids the two cases it
controls (a failing body, and its own transfer deadline; see *The image
cache*), so what is left needs a server-side failure. The fix is two
characters upstream, `make(chan bool, 1)` and `make(chan error, 1)`; if
it ever bites hard enough, a `replace` carrying that patch is the
escape hatch.
- **Create blocks**, including the first image download on each host,
holding one of the three reconcile workers for its duration (LAN
speed, a few hundred MB, once per image per host). Other Proxies keep
reconciling on the remaining workers. Warm the cache with one Proxy per
image per host before creating many.
- **Volumes without a domain are invisible to GC.** A crash between
creating the overlay or seed and defining the domain leaves volumes
that `ListByTag` cannot see (it lists domains). The same Proxy's next
Create reuses the overlay and rebuilds the seed, and its Delete removes
both by name even with no domain, but a Proxy CR that is gone for good
leaves them for `virsh vol-list` to find.
- **A host removed from the config orphans its VMs.** `Delete` for a
providerID naming an unconfigured host is a logged no-op success, so
the Proxy finishes deleting instead of pinning its finalizer, but the
domain and its volumes stay on the hypervisor and orphan GC cannot see
them either (it only sweeps configured hosts). Delete the Proxies before
retiring a host, or clean up afterwards with `virsh undefine --nvram`
plus `virsh vol-delete`. `Get` fails permanently for such a providerID
on purpose: a typo in `hosts` must not look like a missing VM and
trigger a rebuild elsewhere.
- **No adoption of `kvm` domains.** See *Migrating from kvm*.
- **A cancelled call drops the host's pooled connection once**, so other
RPCs in flight on that host fail once with `ErrTransient` and retry.
The image upload and the health check are isolated on their own
connections.
- **`Content-Length` is required** for image downloads; a mirror that
only chunks is rejected permanently. Point the catalogue at one that
sends the length.
- **A paused or pmsuspended domain is not started.** A repeat `Create`
for a domain in that state fails permanently (`is DomainPaused and
cannot be started by the operator; resume or destroy it on the host`),
so the Proxy shows `Failed` until its spec changes; the operator never
resumes a VM an admin paused. Resume it on the host
(`virsh resume <name>`) and touch the spec, or delete the Proxy.
- **Parser text in credential errors.** A key or known_hosts file that
does not parse fails with the parser's own message, which can quote a
PEM header type or a malformed known_hosts token. Key bytes are never
quoted.
- **The sample cloud-init is an open proxy.**
`config/samples/proxy_kvm2.yaml` installs Squid with
`http_access allow all`; restrict it before exposing the bridge network
to anything but the cluster.
- **IP discovery needs the agent or guest traffic to the host.** A VM on
a bridge with no DHCP never gets an address and stays `Provisioning`
forever; check `virsh console <name>` on the host. The same symptom
with a booted, addressed VM means the guest agent never came up: check
`systemctl status qemu-guest-agent`, `cloud-init status --long` and the
vendor script's output in `/var/log/cloud-init-output.log` (a VM without
outbound internet cannot install it). A user-data that disables
vendor-data (`vendor_data: {enabled: false}`) also removes the agent, by
design. A host where the ARP source *errors* rather than answering with
nothing is the one case this reads differently: with no source left to
answer, a booting VM retries with `ErrTransient` instead of resting in
`Provisioning`. It still converges; the giveaway is two "no addresses
from source" lines per poll at `-v 2`.
`virsh domifaddr <name> --source agent` and `--source arp` show
what the operator sees, in that order.
- **Debian 13 cloud images hang in GRUB under BIOS on a headless VM**
([Debian bug #1111240](https://bugs.debian.org/1111240)), which is why
`firmware` defaults to `uefi`; the same image boots under UEFI with the
serial console intact. Images that need `firmware: bios` are the
exception.
## Testing
Unit tests run over an in-memory fake of the libvirt seam and an
in-process SSH server. The optional integration test against a real host
is described in
[testing.md](testing.md#the-kvm2-integration-test-real-hypervisor); run it
with `make test-integration-kvm2`. `TestKVM2Check_realHost` runs the
health check against the same host and creates nothing:
`go test -tags=integration -count=1 -v -run TestKVM2Check ./internal/provider/kvm2/`.
docs/operating.md
+58
View File
@@ -0,0 +1,58 @@
# Operating the operator
Helpers for running the operator day to day: log queries, one-off
inspections, and the small recipes that are easier to look up than to
reconstruct. Not a runbook — it assumes you already know what you are
looking for.
## Log queries
The manager logs one JSON object per line, so `jq` is the tool. The
`.msg` field is the discriminator: it is a fixed string per event, and
the variable parts live in their own fields rather than being
interpolated into the message.
### Libvirt RPCs, with timestamps
Both libvirt providers log one `"libvirt call"` per RPC, so this catches
`kvm` and `kvm2` alike:
```sh
jq -r 'select(.msg == "libvirt call") | "\(.ts) \(.rpc)"'
```
```text
2026-09-18T07:48:29Z virStoragePoolGetInfo
2026-09-18T07:48:29Z ConnectGetLibVersion
2026-09-18T07:48:29Z ConnectGetHostname
2026-09-18T07:48:29Z StoragePoolLookupByName
2026-09-18T07:48:29Z StoragePoolGetInfo
```
These lines are debug level, which is the binary's default
(`zap.Options{Development: true}` in `cmd/main.go`) — nothing to turn on.
They disappear if the Deployment has been given `--zap-log-level=info`,
which is the switch for outcomes only.
Piping from a live manager:
```sh
kubectl logs -n egress-proxies-operator-system \
deploy/egress-proxies-operator-controller-manager -f \
| jq -r 'select(.msg == "libvirt call") | "\(.ts) \(.rpc)"'
```
`jq` aborts on the first line that is not JSON, which happens when
something bypasses the structured logger — a panic trace, or output from
a tool wrapping the manager. Reading line-by-line with `fromjson?` skips
those instead:
```sh
jq -rR 'fromjson? | select(.msg == "libvirt call") | [.ts, .rpc] | @tsv'
```
Note that the connection-level events use their own `.msg` values
(`"libvirt host connected"`, `"kvm2 host checked"`), so they drop out of
the filters above. Widen to `select(.msg | startswith("libvirt"))` to
keep the connect lines — usually what you want when chasing a latency
spike rather than a call count.
docs/plans-executions/2026-09-16-1333-kvm2-provider.md
+382
View File
@@ -0,0 +1,382 @@
# Execution: kvm2, a clean-room libvirt/KVM provider built from the provider contract
Plan: [docs/plans/2026-09-16-1333-kvm2-provider.md](../plans/2026-09-16-1333-kvm2-provider.md)
- [x] Step 0 — Plan file and branch
- [x] Step 1 — The spec, `docs/kvm2-provider-spec.md`
- [x] Step 2 — Config wiring, IDs, state mapping, classify
- [x] Step 3 — Pure builders: domain and volume XML, metadata, seed
- [x] Step 4 — Transport, seam, decorator, fakes
- [x] Step 5 — Image cache and pool-space precheck
- [x] Step 6 — The four contract methods
- [x] Step 7 — Checker and the CRD comment
- [x] Step 8 — Deployment files, Makefile, integration test
- [x] Step 9 — Documentation
- [x] Step 10 — Milestone review
- [ ] Step 11 — Real-host validation, changelog, MR
## Step 0 — Plan file and branch
`main` was fast-forwarded to the merge of PR #7 (`0dc25a1`), which carries the
three provider documents, and `feat/kvm2-provider` was cut from it. The plan
was committed alone as `854856c`. The pre-existing local modification of
`.claude/settings.json` stays uncommitted throughout.
## Step 1 — The spec, `docs/kvm2-provider-spec.md`
2026-09-16 13:37. Written by the orchestrator, not a subagent, since the spec
is the "specifics" a user hands to the implementer. All 24 entries answered;
every Evidence line cites the v1 user guide, the 2026-09-04 execution log, a
go-libvirt symbol, or the contract. Nothing was taken from
`internal/provider/kvm/`.
Design choices that go beyond the user's profile answers, all recorded in the
spec: an adoption guard on `Create` (foreign UID is Permanent), a pool-space
precheck mapped to `ErrQuotaExceeded`, a dedicated connection for image
downloads so a cancelled poll cannot cut one off, the Checker on fresh
connections rather than the pool, SSH authentication and host-key failures
classified Permanent (v1 retried them forever), SSH material re-read on every
dial so rotation needs no restart, kvm2's own metadata namespace (no adoption
of v1 domains), and the shared `validateKVM` refactored to take the block name
so messages say `kvm2.`.
Checks run on the spec:
```sh
grep -c '^\*\*Answer:\*\*$' docs/kvm2-provider-spec.md # 0 blank answers
grep -n 'internal/provider/kvm/' docs/kvm2-provider-spec.md # empty
```
plus the axis-code check against contract §4 headings and the path-existence
loop from the docs-split log. All clean.
Worth noting: the spec's entry 21 lists the concrete test name for every
contract §11 item, so the later steps' briefs can quote it rather than
re-derive the mapping.
## Step 2 — Config wiring, IDs, state mapping, classify
2026-09-16 16:38. First subagent step. The brief named the allowed files,
the spec entries, the test names and the prohibition on
`internal/provider/kvm/`; the agent reported nothing opened there, with one
honest caveat: a `golangci-lint` run printed seven single source lines from
v1 files among its findings, which it did not act on. The review greps
(`grep -rn 'provider/kvm"' internal/provider/kvm2/`, the v1 identifier
smell list) are clean.
What landed: `ProviderConfig.KVM2` reusing `KVMConfig`, `setBlocks`, the
allow-list and dispatch in `Config.validate`, `validateKVM` refactored into
`validateKVMBlock(prefix, block, c)` with `validateKVM` kept as a wrapper so
v1 messages and tests are untouched, `"kvm2": kvm2.New` in `cmd/main.go`
(both type lists in one commit), and the package skeleton: `kvm2.go`
(`New` never dials, warns on unreadable files, `withDefaults`, `host`,
`wrapErr`, stubbed methods), `id.go`, `state.go`, `errors.go` with 34
`classify` rows.
Decisions the agent made that the spec left open, all accepted: `New`
checks file existence only (parsing belongs to the dialer); `withDefaults`
copies hosts and images; `firstIPv4` also drops an address whose type tag and
literal disagree; there is no explicit `fs.ErrNotExist` branch in `classify`
because it falls to the Transient default anyway.
Commands as run by the agent and re-run by the orchestrator:
```sh
go vet ./... && go build ./... && go test -race -count=1 ./internal/provider/... ./cmd/...
go test -short ./...
```
All packages `ok`.
Worth noting: `bin/golangci-lint` panics inside staticcheck's IR builder on
this machine (`unexpected expr: *ast.KeyValueExpr` on stdlib `internal/poll`)
because the local toolchain is Go 1.27.0 while `go.mod` says 1.26 and the
bundled staticcheck predates it; it reproduces on untouched packages, so lint
is deferred to Step 10 with a toolchain note. The `.claude/settings.json`
modification is still local and uncommitted.
## Step 3 — Pure builders: domain and volume XML, metadata, seed
2026-09-16 16:55. Second subagent step, new files only: `xml.go`,
`metadata.go`, `volume.go`, `seed.go` and their tests (26 tests). The
domain XML is built from `encoding/xml` structs, both disks are
`type="file"` with resolved paths (the AppArmor lesson), UEFI is the default
with the secure-boot feature disabled, the metadata element parses with or
without its namespace declaration, and the seed ISO is staged under
`os.TempDir()` (the library uses `os.MkdirTemp("", "")`, so `TMPDIR` is
honoured; the test points it at a `t.TempDir()` and asserts nothing is left
behind).
Decisions the agent made, all accepted: `buildDomainXML` defaults an empty
disk format to qcow2 and rejects anything but qcow2 or raw;
`buildVolumeXML` rejects a non-positive capacity; `buildSeedISO` rejects an
empty name; the guest-agent script still runs the enable-and-start step when
the agent is already installed; the multipart boundary is random, so
vendor-data is not byte-deterministic across calls (the seed is rebuilt every
Create anyway).
Commands:
```sh
gofmt -l ./internal/provider/kvm2 ; go vet ./internal/provider/kvm2/ && go test -race -count=1 ./internal/provider/kvm2/
go build ./... && go test -short ./...
```
All `ok`. Review greps for a v1 import or v1 identifiers are empty; the only
`type="volume"` in the package is the comment explaining why it is not used.
The agent reported "files read outside the allowed set: none".
## Step 4 — Transport, seam, decorator, fakes
2026-09-16 17:42. Third subagent step, the largest: `backend.go` (the
ctx-first seam `backend`, the production adapter `libvirtBackend`, a
generic `call[T]` helper with per-call budgets and the abandon policy),
`ssh.go` (credentials read on every dial, stage-tagged `dialError`, the
forwarded socket, keepalive), `hosts.go` (`hostTable` with a per-host
channel semaphore so waiting for a dial ends with the caller's ctx,
`dialFresh` for dedicated connections, the production `libvirtDialer`
starting the `kvm2.dial` span and applying the decorator once),
`observe.go` (one span `kvm2.<Method>` and one `libvirt call` line per
RPC), plus an in-memory `fakeBackend`, an in-process SSH server and 41
tests. The agent also drove the real adapter over go-libvirt's own
`libvirttest` mock through a `net.Pipe`, which the brief did not ask for.
Abandon policy as built: a call that outlives its ctx or budget kills the
link (closes the socket and the SSH client), returns Transient wrapping the
ctx or deadline error, and other in-flight calls on that connection fail once
with go-libvirt's `ErrInterrupted`; the next `get` redials. Results travel
through a channel so a late-finishing abandoned call never writes into its
caller's frame; the agent reported that the first `-race` run caught exactly
that bug in an earlier version.
Decisions the agent made, accepted: liveness is a separate unexported
interface so the reflection test over `backend` stays at the 20 RPCs plus
`Close`; `Close` sends a bounded `ConnectClose` before killing the link;
`observe` and the dialer take `...tracing.Option` so tests can inject a
recording tracer provider, as `provider.WithTracing` does.
Clean-room note: three type names coincide with names v1 uses
(`dialError`, `dialStage`, `hostEntry`). The brief described each concept
("an error type that records which stage failed", "a per-host entry") so the
names follow from the description; the greps for a v1 import and for the
listed v1 identifiers are empty and the agent attested to reading nothing
under `internal/provider/kvm/`.
Commands (agent and orchestrator):
```sh
gofmt -l ./internal/provider/kvm2 ; go vet ./internal/provider/kvm2/ && go test -race -count=2 ./internal/provider/kvm2/
go build ./... && go test -short ./...
grep -n 'context.Background()' internal/provider/kvm2/*.go | grep -v _test.go # empty
```
All `ok`; the agent also ran the timing-sensitive tests at `-count=8`.
## Step 5 — Image cache and pool-space precheck
2026-09-16 18:11. Fourth subagent step: `image.go` and `image_test.go`
(27 tests), plus additive changes to `fake_test.go` (a `twin()` sharing
state with its own call log, so a test can tell the pooled connection from
the dedicated one) and `kvm2.go` (`http` client, budgets and tracing
options behind functional options, so `New` is unchanged). The pipeline:
marker lookup and bounded read on the pooled connection, then a GET bound to
ctx plus the 20-minute cap under a `kvm2.download` span, `StorageVolCreateXML`
and a streamed `StorageVolUpload` through sha256 on a fresh connection,
`StoragePoolRefresh`, and the marker written last; every failure after the
volume exists discards it with the cancellation stripped from ctx so the
cleanup RPC still runs.
Streaming is asserted by counting: a hook on the dedicated fake's upload
snapshots how many bytes the 64 MiB test server had written when the upload
began, and asserts under 16 MiB. The agent chose that over a process-wide
`MemStats` delta because the tests run in parallel.
Decisions the agent made, accepted: the base volume is created on the
dedicated connection (nothing exists if the dial fails); `Accept-Encoding:
identity` is sent so transparent gunzip cannot hide the Content-Length; a
marker whose content is not 64 hex characters counts as debris;
`Content-Length: 0` is its own Permanent error; the image lock is a
one-slot channel so a waiter gives up with its ctx instead of sitting out a
transfer; the fetch logger is derived after the span starts so its lines
carry the span ID (the agent's first test run caught that).
Commands (agent and orchestrator):
```sh
gofmt -l ./internal/provider/kvm2 ; go vet ./internal/provider/kvm2/ && go test -race -count=2 ./internal/provider/kvm2/
go build ./... && go test -short ./...
grep -n 'io.ReadAll\|bytes.Buffer' internal/provider/kvm2/image.go # empty
grep -n 'context.Background()' internal/provider/kvm2/*.go | grep -v _test.go # empty
```
All `ok`. Attestation: "files read outside the allowed set: none".
## Step 6 — The four contract methods
2026-09-16 18:36. Fifth subagent step: `create.go`, `get.go`, `delete.go`,
`list.go` and five test files (the package now has 150 top-level tests).
Create follows spec entry 10 step for step under a per-name lock, with the
adoption guard, the quota precheck, the image cache, an overlay that is
reused and a seed that is always rebuilt, file-type disks from resolved
paths, the metadata element in the define call, and a wait for running
capped at two minutes. Get maps only `ErrNoDomain` to NotFound, prefers the
guest agent over ARP, and reads UID and CreatedAt back every time. Delete is
idempotent at every step and a no-op success for an unconfigured host.
ListByTag fails whole per host and skips single unreadable domains.
Decisions the agent made, accepted: the adoption path also waits for
running, so a repeat Create on a shut-off domain blocks like a first one;
`address` and `ListByTag` propagate ctx cancellation instead of reading it as
"no address" or "skip", so a cancelled Get is Transient rather than a false
Provisioning; NotFound from Get gets its own V(1) line because polling to
NotFound is expected; the domain XML rides on the `domain defined` line at
V(2). Flagged, not solved: adopting a paused domain fails Transient on every
retry (`DomainCreate` refuses and the wait expires), which the spec does not
cover; recorded as a known gap for the docs step.
The agent found a one-word conflict between the brief and the spec
(`DomainAffectCurrent` versus `DomainAffectConfig` for `DomainGetMetadata`);
the code uses `DomainAffectCurrent`, and spec entry 8 was corrected to
match in this commit.
Commands (agent and orchestrator):
```sh
gofmt -l ./internal/provider/kvm2 ; go vet ./internal/provider/kvm2/ && go test -race -count=2 ./internal/provider/kvm2/
go build ./... && go test -short ./...
grep -n 'CloudInit' internal/provider/kvm2/create.go # only len(req.CloudInit) and the seed builder call
grep -n 'not implemented' internal/provider/kvm2/*.go # empty
grep -n 'provider.Wrap(' internal/provider/kvm2/*.go | grep -v _test.go | grep -v 'p.name' # empty
```
All `ok`. Attestation: "files read outside the allowed set: none".
## Step 7 — Checker and the CRD comment
2026-09-16 18:52. Sixth subagent step: `check.go` and `check_test.go`
(ten tests, the reasons table has 19 rows). Every host is checked
concurrently on a fresh dedicated connection that is closed when the probe
ends; the pooled table is never consulted, which the test proves with a
counting dialer. Details fill in as facts arrive, so a `PoolNotFound`
verdict still says which libvirt version answered. The `status.type` comment
in `api/v1alpha1/providerhealth_types.go` gained `kvm2` and `make manifests`
changed only the CRD's description line:
```sh
make manifests && git diff --stat config/ api/
# api/v1alpha1/providerhealth_types.go | 2 +-
# config/crd/bases/crawl.example.com_providerhealths.yaml | 2 +-
```
Decisions the agent made, accepted: a dial that runs out of time is
`Unreachable` (the host did not answer), and `Timeout` is reserved for ctx or
budget expiry on an established connection; `Close` runs synchronously in
the probe so the close-count assertion is deterministic, at the cost of up
to five seconds past ctx if a disconnect hangs on a live link.
The agent also reported that the `captureContext` helper in `kvm2_test.go`
appends to an unlocked slice, which the race detector flagged once `Check`
logged from several goroutines. The helper was outside its allowed file set,
so it built its own locked capture; the orchestrator fixed `captureContext`
with a mutex in this commit and re-ran the package at `-race -count=3`.
Attestation: "files read outside the allowed set: none".
## Step 8 — Deployment files, Makefile, integration test
2026-09-16 19:39. Seventh subagent step. `kvm2_integration_test.go`
(`//go:build integration`, `KVM2_TEST_*` variables, the eleven ordered
assertions of spec entry 21 plus `TestKVM2Check_realHost`),
`config/samples/proxy_kvm2.yaml` (zone `beelink`, so the sample matches the
shipped config), a commented `kvm2-home` entry in both providers configs
with the porting note, comment-only edits on the shared `kvm-ssh` mount, the
`test-integration-kvm2` Makefile target and the kvm2 rows and section in
`docs/testing.md`. The integration test was not run; that is Step 11 with
the user's go-ahead.
The agent found that `New` builds the production dialer without tracing
options, so in production the dialer and decorator use the global tracer
provider (correct: `cmd/main.go` sets it up), while a test that wants every
span family in one recorder sets the dialer's option itself. The test does
so and asserts that `kvm2.DomainDefineXML` and `kvm2.dial` are children of
`provider.create`.
Decisions the agent made, accepted: the UID is generated from `crypto/rand`
rather than promoting `google/uuid` to a direct dependency; the sentinel
check looks for the full cloud-init string because the provider name also
appears in attributes; `-run TestKVM2` matches both tests, as the kvm
target's pattern does. Forward links to `docs/kvm2-provider.md` dangle until
Step 9.
Commands:
```sh
go vet ./... && go vet -tags=integration ./... && go vet -tags=e2e ./...
bin/kustomize build config/default > /dev/null && bin/kustomize build config/samples > /dev/null
make -n test-integration-kvm2
# go test -race -tags=integration -count=1 -v -timeout 40m -run TestKVM2 ./internal/provider/kvm2/
```
All clean. Attestation: "files read outside the allowed set: none".
## Step 9 — Documentation
2026-09-16 19:54. Eighth subagent step: `docs/kvm2-provider.md` (sixteen
sections modelled on the kvm guide), a `kvm2` column in the three tables of
`docs/providers-existing.md` plus a kvm2 section and renumbering, the
components row, §5 block, §10 span bullet, §11 row and eight Decisions
bullets in `docs/architecture.md`, a README row and paragraph, and the
guide in the `docs/providers.md` index. The agent verified each claim
against the package rather than the spec and dropped a draft sentence
claiming kvm and kvm2 share the base-image cache, because checking it would
have meant reading v1's volume names.
Checks re-run by the orchestrator: no em-dash in the guide, no line
references, each of the 19 axis codes appears once in the appendix table
with seven cells, every link and anchor across the seven touched documents
resolves. The one added em-dash is the `Leader-elected` cell of the new
components-table row, where the table's own convention wins.
Attestation: "files read outside the allowed set: none".
## Step 10 — Milestone review
2026-09-16 21:19. The `go-operator-reviewer` agent reviewed the package
against the contract and the spec (v1 code off limits to it as well) and
returned seven findings, all accepted and applied by a fresh fix agent in
one commit with regression tests:
1. `waitRunning`'s two-minute cap only ever elapsed in the bad cases and
pinned a reconcile worker; now 15 s, a domain that dies right after
start is Transient at once, and a paused or pmsuspended domain is
Permanent before `DomainCreate` (an admin's doing).
2. `ListByTag` read state and addresses that GC never uses, one agent RPC
per active domain per sweep; it now builds instances from the name and
the metadata only (`TestListByTag_doesNotReadStateOrAddresses`).
3. A health round that ran out of time was blamed on the host
(`Unreachable`); `checkHost` now reports `Timeout` whenever the round's
ctx ended, whichever dial stage was running, and the promptness test
uses the production error shape.
4. `Create` adopted an unmarked domain and never wrote the mark, so GC could
not see it; an unmarked domain under the name is now Permanent
(`TestCreate_adoptsOnlyMarkedDomains`).
5. `hostTable.closeAll` had no production caller and `withBudgets` could not
work (the dialer captured the defaults first); both removed.
6. The per-name and per-image lock maps grew without bound; replaced by a
reference-counted `keyedLocks` whose entries vanish when uncontended.
7. `imageSizeIfUncached` falls back to GET on any non-2xx HEAD, and two
Delete no-op lines moved from V(0) to V(1).
The spec (entries 2, 10, 12, 16, 18, 19, 24), the guide, the appendix and
the architecture flow were updated in the same commit. The reviewer also
listed what it checked and found correct: the error boundary, the state
rules, Delete's idempotency and the replacement window, ListByTag's
partial-failure policy, the abandon policy and double-close guards,
resource handling on every path, and secret hygiene in the format strings.
Lint: `bin/golangci-lint` still panics on this toolchain (Go 1.27 against
a staticcheck built for 1.26), reproducible on untouched packages, so
`make lint` could not gate this step; `go vet` with every build tag is
clean. Tests: `go test -race -count=3 ./internal/provider/kvm2/` (168
top-level tests) and `go test -short ./...` pass; `make test` with envtest
runs next.
docs/plans-executions/2026-09-17-1515-kvm2-review-fixes.md
+531
View File
@@ -0,0 +1,531 @@
# Execution: fix the kvm2 code-review findings
Plan: [docs/plans/2026-09-17-1515-kvm2-review-fixes.md](../plans/2026-09-17-1515-kvm2-review-fixes.md)
- [x] Step 0 — Save plan into the repo
- [x] Step 1 — H1: a stalled write must not delay the teardown
- [x] Step 2 — H2: a source error must not reach go-libvirt's stream sender
- [x] Step 3 — H3: an unreadable address must not look like a missing one
- [x] Step 4 — H4: the base image cache must not be shared across provider entries
- [x] Step 5 — Validate H1-H4 in-cluster against beelink
- [x] Step 6 — Route klog into the structured logger
Remaining after those: the 48 mediums and lows in `review/REPORT.md`. Each gets its own
checklist line here when it is planned, so this list stays a record of decided work rather
than a wish.
## Step 0 — plan committed
Copied out of `~/.claude/plans/` and committed alone before touching code, as the repo's
plan rules require.
## Step 1 — H1: close the SSH client, and signal dead before killing
Two edits, both small, in the order the bug unfolds.
`sshSocket.Close` was `errors.Join(s.Conn.Close(), s.client.Close())`. Go evaluates those
arguments left to right, so the blocking close ran before the unblocking one. `s.Conn` is
the forwarded libvirt socket, and closing an SSH channel writes a close message under
`ch.writeMu`, which go-libvirt already holds while streaming a base image through the same
channel. Against a host that vanished without RST or FIN, that write blocks until the
kernel stops retransmitting — `tcp_retries2`, about 15 minutes. `s.client.Close()`, a bare
`net.Conn.Close()` on the TCP socket that takes no lock and unblocks the stalled writer
through netpoll, was never reached. It now closes the client only.
**Dropping the channel close rather than reordering it** was the judgment call. Reordering
works, but afterwards the channel close returns an error — `net.ErrClosed`, a
`*net.OpError`, the transport's sticky write error, or `io.EOF` once `mux.loop` has run
`ch.close()` — and `closeErr` is memoised, so a filter that missed a case would poison
every later `Close` and break `ssh_test.go:66,69`, which assert both closes return nil.
Nothing wants the polite channel-close message: `dialSSHSocket` builds the client solely
for this one channel, the `sshSocket` owns it exclusively, and `dialSSHSocket`'s own error
paths already tear down by closing the client.
`link.markDead` then compounded it by running `kill()` before `close(l.closed)`, and doing
both inside a `sync.Once`. Reordering alone would not have been enough: `sync.Once.Do`
blocks concurrent callers until the first body returns, so every other `markDead` — including
the one in `call`'s own ctx-cancel branch — queued behind the stalled kill. **Cancelling the
reconcile did not help.** The `once` field is gone; the mutex the struct already had now
guards the "close it exactly once" invariant, `reason` is set before the signal so a
released caller always reads a populated `deadReason()`, and `kill` runs outside the lock.
**A second route to the same pin, found while planning and fixed by the same edits.**
go-libvirt's `Disconnect` closes the `net.Conn` it was handed (`libvirt.go:331` →
`socket.go:164`) — and that conn *is* our `*sshSocket`. So `libvirtBackend.Close`'s
`Disconnect` goroutine entered `sshSocket.closeOnce` and hung there; the 5 s `closeBudget`
released `Close`, which called `markDead("closed")` → `kill()` → `sshSocket.Close()` →
blocked in the other goroutine's `sync.Once`. The budget did not protect it.
The precondition was verified in the pinned x/crypto v0.54.0 rather than assumed: the
flow-control park (`remoteWin.reserve`, `channel.go:268`) happens *outside* `writeMu`, and
only `writePacket` holds it across the TCP write. So a writer parked in SSH flow control
never blocks the close; a writer inside the TCP write always does. Since the channel window
is 2 MiB (`channelWindowSize = 64 * channelMaxPacket`), reproducing it by hand needs a
kernel send buffer that fills first — shrink `SO_SNDBUF` well under 2 MiB.
The older kvm provider does not have this bug: it closes `client` only
(`internal/provider/kvm/conn.go:429,456`) and never touches the forwarded channel. kvm2's
rewrite introduced it.
### Tests: planned four, shipped one
The first draft had three hermetic tests plus a `//go:build integration && linux`
stalled-write test with a TCP proxy and shrunken socket buffers. The user pushed back —
once the bug is fixed the connection closes properly, so what is left to test? — and they
were right about most of it. `TestDial_forwardsUnixSocket` already covers "Close tears the
connection down", and after the fix `Close` is a single call with no ordering to get wrong.
The stalled-write test was the worst of the four: ~150 lines of Linux-only,
buffer-autotuning-dependent scaffolding whose own false-green guard made it *skip* whenever
the kernel did not cooperate, to demonstrate a bug that reading x/crypto settles and that
the fix deletes the code path for.
What survived is `TestMarkDead_slowKillDoesNotBlockCallers`, because the `markDead` change
*adds* a guarantee that did not exist before and nothing else touches it —
`TestCall_killedByAnotherCallerFailsTransient` passes a nil kill, so it cannot see the
property. It is also the plausible regression: restoring the `sync.Once` looks like a
correctness improvement. The call's own `fn` signals when it is in flight, so the parked
caller is genuinely parked rather than refused by the dead check on the way in.
The step that actually validates a regression test is running it against the unfixed code.
`markDead` was reverted in place, the test run, and the fixed version restored from a copy:
```bash
cp internal/provider/kvm2/backend.go "$SCRATCH/backend.go.fixed"
# ... revert markDead to the sync.Once form and re-add the once field ...
go test -race -count=1 -run 'TestMarkDead_slowKillDoesNotBlockCallers' ./internal/provider/kvm2/
# --- FAIL: TestMarkDead_slowKillDoesNotBlockCallers (1.00s)
# backend_test.go:186: link not signalled dead while its kill is still running
cp "$SCRATCH/backend.go.fixed" internal/provider/kvm2/backend.go
```
It fails in a second with the right diagnostic rather than hanging the suite, because the
blocked kill is released by `t.Cleanup`.
```bash
go build ./... && go vet ./... && go test -race -count=1 ./internal/provider/kvm2/
```
### Worth noting
**An accepted risk, deliberately not covered by a test.** Nothing now stops someone
re-adding `s.Conn.Close()` to `sshSocket.Close` as tidiness; it would look harmless and
every test would still pass, because in a healthy test the channel close returns instantly.
The guard is the doc comment on `sshSocket`, which says outright that closing the channel
would queue behind a stalled transfer. That is in idiom for this package — `hosts.go:25-34`
and `create.go:151-153` carry the same kind of hard-won decision — but it is a comment, not
a test, and the failure it prevents is a 15-minute operator-wide stall.
**Left for later, on purpose.** `ReconciliationTimeout` in `SetupWithManager` does not exist
in this repo at all; adding a per-reconcile deadline would change the contract for every
provider, since `hosts.go` and `image.go` both cite "no caller deadline", so it needs its own
change and its own spec entry. `sshSocket.keepalive` still has no test (sweep finding S1.2).
**Docs.** `docs/kvm2-provider.md` said the adapter "kills the connection (socket and SSH
client)", which is no longer true and was the description that made the bug look correct.
It now states the order (mark dead, then kill) and why both steps are lock-free.
`docs/kvm2-provider-spec.md` entries 18 and 19 were checked and are not falsified.
**No CHANGELOG entry yet** — the repo's rule is to add one once the user confirms the change
works, so it lands with a later "validate and record" step, as in the kvm campaign.
## Step 2 — H2: a source error must not reach go-libvirt's stream sender
This one is a workaround, not a repair. The defect is upstream, and the entry says so in
three places (code comment, `docs/kvm2-provider.md` *Known gaps*, commit message) so nobody
later reads the wrapper as ours to delete.
`requestStream` in the pinned go-libvirt starts its stream sender on two **unbuffered**
channels and then, on the error path, returns without ever reading `outErr`. A non-EOF error
from the reader makes `SendStream` emit a StatusError packet and return, whereupon it parks on
`outErr <-` forever; libvirtd answers the StatusError with an error, so the main goroutine
parks on `abort <- true` forever. `StorageVolUpload` never returns. The 20-minute budget
rescues only the caller, and the socket kill does nothing at all here, because both ends are
blocked on channel sends rather than on I/O — which is what the comment at `backend.go:152`
got wrong. Every failed upload leaked two goroutines and 4 MiB against a 128Mi pod limit,
with the reconciler retrying on backoff.
Three changes.
`eofReader` sits at the go-libvirt seam in `backend.go`, not in `image.go`, so it also covers
the seed and marker uploads, and so it stays out of the file Step 4 rewrites. It converts any
source error to `io.EOF` — the one outcome that ends a stream the normal way — and keeps the
real error for `StorageVolUpload` to return afterwards. `countingReader` is untouched
underneath it and still sees the true error, so `uploadImage`'s existing length and digest
checks produce exactly the messages they did before.
The download now carries its own deadline, one minute inside the transfer budget.
`fetchImage` already bounded the whole fetch at 20 minutes, so the body's death and the
budget's socket kill were racing, and the socket usually won. With the body a minute ahead, a
stalled mirror ends through the reader and unwinds cleanly. This is the difference between a
slow mirror costing a Transient error and costing a permanent leak, and slow mirrors are the
realistic repeat offender.
`discardVolume` moved from `uploadImage` up into `fetchImage`'s upload-error branch, so the
dedicated connection is closed — and libvirtd has let go of the stream's fd — before the file
is unlinked. Moving it rather than adding a second `Close` is what keeps
`image_test.go`'s `closeCount() == 1` assertion true. `uploadImage` lost its `be` parameter
and its private logger in the process.
**What is left, deliberately.** A failure that originates in libvirtd mid-upload (ENOSPC, or
the volume deleted underneath) still leaks, because by then `SendStream` has already returned
and nothing we pass, cancel or close can reach it. Confirmed unfixable from this side. The
upstream fix is two characters and is quoted in *Known gaps* along with the `replace` escape
hatch, should it ever bite hard enough to be worth the maintenance.
**Unverified, ~85%.** That libvirtd answers a short-but-clean stream finish with OK rather
than an error. `virFDStreamWrite` enforces `length` only on writes past it and nothing
compares `offset` to `length` on the close path, which is also why `virsh vol-upload` of a
small file into a large volume works. If it answers with an error we land back in a deadlock
by a different door — but that is precisely today's behaviour, so the change cannot make
things worse. To settle it, on any box with a libvirtd (not beelink, which is read-only
here), against the pinned go-libvirt:
```sh
virsh -c qemu:///session pool-define-as probe dir --target ~/vmprobe
virsh -c qemu:///session pool-start probe
virsh -c qemu:///session vol-create-as probe short.raw 10M --format raw
# then a ~40-line main.go: StorageVolUpload(vol, r, 0, 10<<20, 0) where r
# yields 1 MiB then io.EOF; print the error, then runtime.NumGoroutine()
# and a goroutine profile after a two-second settle.
```
Expected: `err == nil` and the goroutine count back at baseline. The same program with a
reader that returns a non-EOF error should wedge against the *unpatched* seam, which also
confirms the diagnosis end to end.
The negative check that matters was run on the unit test instead — `eofReader.Read` was
changed back to returning the real error, and both new tests failed with
`io.Copy through eofReader = connection reset by peer, want nil`, then passed again once
restored.
```bash
go build ./... && go vet ./... && go test -race -count=1 ./internal/provider/kvm2/
```
### Worth noting
The `fetchImage` discard is still unfenced against a second provider entry that shares the
volume name — Step 4 (H4) is what closes that, by making the name per-entry. The deferred
atomic-publish work then lands exactly where this step put the discard.
**Deliberately not tested:** the abort protocol end to end. `libvirttest` reads a fixed
28-byte header per packet and implements no stream procedures, so pinning this hermetically
would mean a 150-250 line libvirt RPC server stub whose only assertion is about *upstream's*
goroutines. Same call as H1's: the test would cost more than it protects. The two tests that
did land pin the one property the seam actually needs.
## Step 3 — H3: an unreadable address must not look like a missing one
`address()` tried the guest agent then the host's ARP table and skipped any error that was
not a context error, returning `("", nil)` after both. An active domain with no IP maps to
`StateProvisioning`, the controller clears `status.IP` on that, and the health engine prunes
the proxy's counters and calls `ForgetProxy` in one tick once `EffectiveHost()` is empty.
The damaging trigger was not the guest. The libvirt connection is pooled per host, so when
another reconcile's cancel or budget overrun killed it, `call` returned
`"connection is closed"` — carrying no context sentinel, so the check did not recognise it —
and `Get` returned a *successful* instance in Provisioning with an empty IP. An unrelated
proxy's timeout dropped a healthy proxy out of rotation.
The fix rests on a discriminator the spec had already recorded without drawing the
conclusion: during a normal boot on beelink exactly one source fails. The agent errors
because it is not up yet, while ARP answers *without* error and with nothing, since the host
never sees the guest's traffic. So one source failing is "no address yet"; every source
failing, or a connection already dead, is `ErrTransient`.
The explicit `provider.ErrTransient` wrap on the joined error is load-bearing rather than
decoration. Without it `classify` picks the first libvirt code out of the join tree, and
`ErrInvalidArg`, `ErrAccessDenied` and `ErrNoSupport` are Permanent — an address read failing
would have latched a *running* proxy as Failed, a worse version of the harm being fixed.
`isDead(be)` is checked first in the error branch. It is not strictly needed, since a dead
link fails both sources and the count rule catches it anyway, but it fails on the first
source instead of the second, attributes the error better, and is the same predicate M15 will
want for `ListByTag`.
### Tests
`TestGet_addressErrorIsProvisioning` failed against the new rule, which is correct: it fails
*both* sources with `ErrAgentUnresponsive`, and code 86 is agent-specific — the ARP source
cannot produce it. Re-pointed as `TestGet_agentFailsAndARPEmptyIsProvisioning`, the beelink
boot it was really reaching for: agent errors, ARP returns no addresses and no error, result
is Provisioning. Nothing covered that before, and it is the case the new rule must not break.
`TestGet_everyAddressSourceFailingIsTransient` covers the other side, two subtests: both
sources erroring, and the pooled connection dying under the read. Against the unfixed code
both return a Provisioning instance instead of an error, which is exactly the bug:
```bash
# with isDead() and the every-source-failed arm removed from address():
go test -count=1 -run TestGet_everyAddressSourceFailingIsTransient ./internal/provider/kvm2/
# get_test.go:215: Get = &{... IP: State:Provisioning ...}, want no instance
```
`TestGet_arpFallbackWhenAgentFails` passes untouched — one source of two.
### Worth noting
**Half of H3 is deliberately still open, and it is the half that fires more often.** A guest
agent restarting inside a running VM still demotes it to Provisioning and clears its IP: the
agent errors, ARP answers empty, and that is indistinguishable from a VM still booting. The
discriminator is the *previous* IP, which the provider does not hold by design
(`TestGet_readsUIDAndCreatedAtFromBackend` pins "read back, not remembered") and the
controller already has. Follow-up finding, to be filed against
`internal/controller/proxy_controller.go:208-211`: on `StateProvisioning`, keep an existing
IP and `Provisioned` condition for a few polls rather than clearing on the first, or give the
health engine a grace period before pruning.
The one kvm2-local alternative was considered and rejected: the mark carries `createdAt`, so
`get` could treat "active, no address from any source, domain older than a provisioning
window" as an error. It uses state the provider genuinely has, but `createdAt` is creation
rather than boot, it fires on every genuinely stuck VM as a retry-forever error, and it
invents a provisioning deadline that belongs to the controller. A design change wearing a bug
fix's clothes.
**L6 not folded in.** A domain that vanishes between `DomainGetState` and the address read
now returns Transient rather than a fake Provisioning — better, and self-correcting on the
next poll. Returning NotFound one poll earlier is a two-line addition at the same spot, left
for when L6 comes up so this commit stays one idea.
## Step 4 — H4: the base image cache must not be shared across provider entries
`baseVolumeName` was `base-<key>.<format>` — byte-identical to what
`internal/provider/kvm/image.go` builds in the same pool, for URL images too (kvm takes
`hex(sum[:8])`, kvm2 takes `hex(sum[:])[:16]`, the same eight bytes), with a completely
separate lock table. The shipped config has a live `kvm-home` entry and a commented-out
`kvm2-home` entry on the same host, the same key `debian-13` and the same default pool, and
the comment invites the operator to enable it. Both integration tests use key `test-image`.
On beelink both live overlays back onto kvm's `base-debian-13.qcow2`.
Since `cachedImage` accepts any 64-hex marker when no `sha256` is configured and never
re-hashes the base, that meant: two entries pointing one key at different URLs silently share
a file; a concurrent cold fetch lets one entry delete the other's half-uploaded base and boot
a guest from the remains; and a `sha256` edit deletes the base in place under live overlays,
which libvirt permits without complaint.
The name now carries the provider entry: `base-<entry>-<key>.<format>`. `imageSource` gained
an `Entry` field and `resolveImage` takes the provider name, so the four production call
sites all follow from `imageSource.volumeName()`. That closes the cross-entry sharing
outright, kvm↔kvm2 and kvm2↔kvm2 alike, and it turns the existing per-`Provider` `imageLocks`
from a latent bug into a correct one — with names unique per entry, no two entries can
contend for a file, so a process-wide lock table is not needed.
**The entry name reaches a file name, so it now has to be a usable one.** Provider names are
validated for all types at once and only for being non-empty and unique
(`internal/provider/config.go`), so `New` checks its own against the existing `kvmIdentifier`
rule, exported as `provider.IsKVMIdentifier` rather than duplicated. A name with a slash
fails at startup instead of producing a volume nothing can address.
**One companion fix, because the naming alone does not stop it.** `cachedImage`'s
digest-mismatch branch deleted the marker and the base and refetched. It now returns
`ErrPermanent` naming both digests and telling the operator what to do. That branch was the
remaining path that unlinks a good file under running domains, and the contract already said
"add a new key instead" — the code was contradicting it. The no-marker and non-hex-marker
branches keep deleting, since there the base is genuine debris.
### What this does NOT fix
Stated plainly rather than left to look closed:
- **Editing `images.<key>.url` is still a silent no-op.** The cache is keyed by entry and
key, and nothing records the URL, so the new one is never fetched. Pre-existing, already
documented as "add a new key"; the eviction section now says it outright.
- **Cross-process fencing.** Two managers, or a manager plus a hand-run `virsh`, can still
meet on one name. E4 refuted the rollout and leader-handover triggers for the shipped
manifests, so this is narrow, and its fix is the deferred atomic publish — upload to a temp
name, verify, publish — which lands where H2 put the discard.
### Migration
Nothing to do. The `base-debian-13.qcow2` on beelink is kvm's, kvm keeps using it untouched,
and kvm2 downloads its own copy once per host when `kvm2-home` is enabled. No window in which
anything is replaced under a running guest. The pool needs room for both, which the
prerequisites now say.
### Tests
Mostly literals becoming helpers, plus a new `fx.baseName()` in `provider_test.go` for the
three fixtures that asserted `"base-debian-13.qcow2"` by hand. The log and span assertions in
`TestEnsureImage_downloadsAndWritesMarkerLast` keep asserting a full name rather than loosening
to a prefix, so they still pin that the key stays readable in operator-facing output.
One test changed behaviour rather than spelling: `TestEnsureImage_sha256ChangeRedownloads`
became `TestEnsureImage_sha256ChangeRefusesInsteadOfReplacing`, asserting Permanent, zero
server hits, zero deletes, and both volumes untouched. Two added:
`TestNew_rejectsUnsafeEntryName`, and a `TestVolumeNames` row for a second entry with the same
key.
Both negative checks were run:
```bash
# baseVolumeName reverted to base-<key>.<format>:
go test -count=1 -run TestVolumeNames ./internal/provider/kvm2/
# volume_test.go:135: got "base-debian-13.qcow2", want "base-kvm2-lab-debian-13.qcow2"
# the digest branch reverted to delete-and-refetch:
go test -count=1 -run TestEnsureImage_sha256ChangeRefuses ./internal/provider/kvm2/
# image_test.go:412: ensureImage() error = <nil>, want ErrPermanent telling the operator what to do
```
`go vet -tags=integration` was run too, since `kvm2_integration_test.go:484` builds the base
name and only compiles under that tag.
### Worth noting
**The integration test has still never run on beelink.** Step 11 is unchecked in
`docs/plans-executions/2026-09-16-1333-kvm2-provider.md`, every `base-*` on the host predates
kvm2, and both live domains carry the kvm v1 metadata namespace. After this change it is also
the first end-to-end exercise of the new name, and it will do a real cold download under
`base-kvm2-integration-test-image.qcow2` beside kvm's untouched `base-test-image.qcow2`.
**Docs touched beyond the obvious:** the eviction recipe named `base-debian-13.qcow2`, which
after this change is *kvm's* file — following it would have deleted the wrong provider's base.
It now greps for the prefix and deletes the marker first.
## Step 6 — Route klog into the structured logger
Not a review finding. Found while watching the Step 5 validation: some lines ignored the zap
encoder no matter what, so `kubectl logs | jq` could never work.
```
I0917 19:50:02.580097 1 leaderelection.go:258] "Attempting to acquire leader lease..." lock="..."
```
That is klog's own stderr writer, not ours. client-go's leader elector calls
`klog.FromContext(ctx)` (`leaderelection.go:257`), and controller-runtime hands the elector a
context with no logr in it — `mgr.Start` receives `ctrl.SetupSignalHandler()`, which carries no
logger (`pkg/manager/internal.go:461-473`). `FromContext` falls back to `Background()`, and with
no logger registered that is klog's text sink. Nothing in this repo imported or configured klog;
it was an indirect dependency that around 60 linked packages use, including
`client-go/tools/{leaderelection,cache,record}` and `client-go/transport`. controller-runtime
does not route it either — its only `klog.SetLogger` call is in its own test suite.
**`klog.SetLogger(logger)`, deliberately not `SetLoggerWithOptions(logger, ContextualLogger(true))`.**
The contextual form is what klog's docs recommend, because it skips a klogr round trip. It also
disables klog's verbosity gate ("Such a logger cannot rely on verbosity checking in klog",
`contextual.go:83-86`), handing the decision to zap — whose level under `Development: true` is
Debug, which enables V(1). client-go's V(1) chatter, suppressed today by klog's own `-v=0`,
would have started appearing. Plain `SetLogger` keeps klog's check in front and still redirects
what it does emit ("all log lines will be suppressed from the regular output, and redirected to
the logr implementation", `contextual.go:47-48`). Same fix to the format, no new volume — which
matters, because the complaint that started this was volume.
`k8s.io/klog/v2` moves from indirect to direct in `go.mod`. It was already in the module graph;
nothing new is downloaded.
**The other bypass.** The discovery API's `http.Server` set no `ErrorLog`, so `net/http`'s own
faults — a failed `Accept`, a panic escaping a handler — went to the stdlib logger on stderr.
It now drains into logr through `logr.ToSlogHandler`, the pattern
`internal/provider/gcp/wirelog.go:31-36` already uses. This server is plain HTTP with no TLS, so
`ErrorLog` carries real faults rather than handshake noise, and Error level is right.
**Deployment.** `--zap-encoder=json` and `--zap-time-encoding=rfc3339` on the manager.
Verified the flag beats the in-code default rather than assuming it: `addDefaults` fills
`NewEncoder` only `if o.NewEncoder == nil` (`zap.go:176-191`) and `BindFlags` sets it from the
flag. The time encoding is not optional — the JSON encoder defaults to epoch seconds.
Verbosity was left at debug on purpose; `--zap-log-level=info` drops the per-RPC lines.
`opts := zap.Options{Development: true}` stays. Flipping it would change the format for every
deployment of this operator, and the flags get the same result here without that reach.
### Tests
None. Three changes, all wiring: two struct fields and a manifest key. `cmd/main.go` has no test
coverage, and pinning `klog.SetLogger` means mutating process-global state that leaks into every
other test in the package. Verified by observation instead, which for this change is stronger
than a unit test would be.
### Verification
```bash
kubectl -n egress-proxies-operator-system logs deploy/... | grep -i leader
# {"level":"info","ts":"2026-09-17T20:10:00Z","msg":"Attempting to acquire leader lease...",
# "lock":"egress-proxies-operator-system/b47711d1.example.com"}
kubectl -n egress-proxies-operator-system logs deploy/... | jq -e -s 'length > 0'
# YES - every line is valid JSON
```
The `jq -s` check is the one that counts: it fails if *any* line is not JSON, which is exactly
the property being fixed. All four existing proxies stayed Ready across the rollout and all
three ProviderHealth entries stayed Ready.
### Worth noting
**The JSON encoder renders durations as float seconds**, so `"duration":"566.12µs"` became
`"duration":0.000336664`. Machine-friendlier, worse to read. Either format in the query —
`(.duration*1000000|round)` for µs — or pass an encoder config option in code, which
controller-runtime exposes (`zap.Options.EncoderConfigOptions`, `zap.go:140-144`):
```go
opts := zap.Options{
Development: true,
EncoderConfigOptions: []zap.EncoderConfigOption{
func(c *zapcore.EncoderConfig) { c.EncodeDuration = zapcore.StringDurationEncoder },
},
}
```
Left out of this step as scope the plan did not cover.
## Step 5 — Validation against beelink
First contact between the kvm2 provider and real libvirt. Everything before this ran against
an in-memory fake of the twenty libvirt calls the provider uses.
Run in the existing `kind-proxy-operator-demo` cluster rather than a fresh one, deliberately.
That cluster already owned two live `kvm-home` VMs on beelink, and orphan GC deletes any
instance whose UID is missing from *its own* cluster's live set after `gc-min-age`. A second
operator carrying the `kvm-home` entry would have destroyed both within ten minutes. Using the
existing cluster made its live set the thing that protected them — the isolation argument
pointed the opposite way from instinct.
Baseline recorded first (4 Ready proxies, 3 ProviderHealth entries, beelink's domains, volumes
and pool space), so "did we break something" stayed answerable.
The image tag had to change from the running `kvm-dev`. `make deploy` sets the image through
kustomize, so an identical tag leaves the Deployment spec untouched, no rollout happens, the
new process never starts, and the providers config — which is read once per process — is never
re-read. The ConfigMap carries no generator hash either, so nothing else would have forced it.
Results, in the order they were checked:
| Check | Outcome |
|---|---|
| `kvm2-home` ProviderHealth | `Ready 1/1`, `Reachable`, 189 ms — hostname, libvirt 11.6.0, pool running, 247.8 GiB free |
| First Create, cold cache | `base image cached`, 433,651,712 bytes in 8.3 s, digest verified |
| Volume naming (H4) | `base-kvm2-home-debian-13.qcow2` created **beside** kvm's untouched `base-debian-13.qcow2` |
| Second Proxy | no `fetching` line at all — cache hit under the new name |
| State reporting (H3) | `Running` held steady across every poll, no flapping to Provisioning |
| Squid | `Healthy=True` on a real CONNECT probe |
| Delete | domain and both per-VM volumes gone; base image survived |
| Existing infrastructure | all four pre-existing proxies stayed Ready across two rollouts |
The health check alone is worth calling out as a gate: it dials SSH, verifies the host key,
completes the libvirt handshake, runs four RPCs and tears down a dedicated connection — the
exact path H1 changed — without creating anything. Putting the stop gate there meant the
riskiest unknown was settled before a single byte was written to the hypervisor.
### What this run did NOT prove
**H2's error paths.** The unverified claim — that libvirtd answers a short stream finish with
OK rather than an error — only arises when a body *fails* part way through, and a healthy
433 MB download never finishes short. The 19-minute HTTP deadline never fired either. What was
proved is narrower: the wrapper does not break normal uploads. Settling the rest still needs
the scratch-pool probe in the Step 2 entry, and it would be easy to mistake this green run for
more than it is.
**The `kvm2` integration test still has not run.** `make test-integration-kvm2` remains
unexecuted; this was a cluster-level validation, not that suite.
### Worth noting
`config/manager/providers_config.yaml` now ships with the `kvm2-home` entry **enabled**, since
the operator is running it for real. `config/manager/kustomization.yaml` was left at its
committed tag: `make deploy` rewrites it to whatever local image was last deployed, which is
churn rather than work.
Two host-side leftovers were *not* touched, both pre-existing and unrelated to kvm2: five
orphan `proxy-*` volumes from earlier kvm runs, and the `base-test-image.qcow2` pair from the
kvm integration test. Every write to beelink in this step came from the operator itself; no
`virsh` write was issued by hand.
docs/plans-executions/2026-09-18-1502-proxmox-provider.md
+1441
View File
File diff suppressed because it is too large Load Diff
docs/plans/2026-09-16-1333-kvm2-provider.md
+301
View File
@@ -0,0 +1,301 @@
# Plan: kvm2, a clean-room libvirt/KVM provider built from the provider contract
**Created:** 2026-09-16 13:33
## Context
The provider documentation was just split into a generic contract
(`docs/provider-contract.md`), a spec template (`docs/provider-spec-template.md`)
and an appendix on the existing providers (`docs/providers-existing.md`), so that
a new provider can be specified and built from those files alone. Before using
them for Proxmox, the user wants a dry run on a known backend: rebuild the
libvirt/KVM provider from scratch as a second type, `kvm2`, following the
documents, without touching or reading the existing `internal/provider/kvm/`
code. The result is a validated workflow (spec first, then implementation by
fresh agents against the contract) and a v2 provider that can replace v1 once it
passes on the real hypervisor.
User decisions:
- **Placement:** new package `internal/provider/kvm2`, type `kvm2`, config block
`kvm2:`, side by side with `kvm`. The Go config struct reuses
`provider.KVMConfig` (`internal/provider/config_kvm.go` is shared code, not
the kvm package), so an entry ports by changing `type` and the block name.
- **Clean room:** implementers never open, grep or import
`internal/provider/kvm/`. Docs (`docs/kvm-provider.md`) and the operational
lessons from the first build's execution log are allowed.
- **Implementer:** the orchestrator (this session) writes the spec and the step
briefs; each implementation step runs in a fresh general-purpose subagent
given only the three docs, the filled spec, the backend facts and its brief.
The orchestrator reviews each result against the contract, runs the tests and
commits.
- **Profiles:** `create-return` blocking, `provisioning-state` create-blocks,
`child-spans` span-decorator, `quota` detected (pool free space precheck),
`credentials` mounted-files (the existing `kvm-ssh` Secret), `client-kind`
rpc-over-ssh, `gc-mark` written-metadata, `created-at` written-at-create,
`ip-source` discovered (agent, then ARP), `list-partial` fail-whole per host
with unreadable domains skipped, `not-found` error-code, `delete-return`
waits, `secondary-resources` explicit, `timeouts` per-call, `logging` tiered,
`backend-test` integration-tag.
## Backend facts the spec carries (allowed lessons)
- Real host `beelink`: libvirt 11.6, QEMU 10.1, pool `default` at
`/srv/vms/images`, bridge `br0`, `beelink.tail8c69c.ts.net:22`, user `novakj`,
key `~/.ssh/proxy-operator_ed25519`, known_hosts `~/.ssh/kvm_known_hosts`.
The Debian trixie qcow2 is about 435 MB.
- AppArmor confines QEMU from the domain XML: only `<disk type="file">` with a
`<source file>` path gets a rule, so volume paths are resolved with
`StorageVolGetPath` and emitted as file disks.
- Debian 13 hangs in GRUB under BIOS with a serial console and no video: UEFI
by default (`<os firmware="efi">`, secure boot feature disabled), `firmware:
bios` per image; OVMF is a host prerequisite; undefine needs the NVRAM flag.
- libvirt hands the metadata element back without its namespace declaration:
parse by local name.
- Vendor-data must be a shell script, not cloud-config, to survive cloud-init's
merge with user-data; it installs `qemu-guest-agent`. The agent answered after
about 20 s where ARP never did.
- Read-only rootfs: `iso9660.NewWriter` needs a temp dir under `/tmp` (64Mi).
- go-libvirt (`github.com/digitalocean/go-libvirt`, in go.mod): every method
blocks and takes no ctx; `libvirt.Error{Code uint32, Message}` has a value
receiver; needed `ErrorNumber` constants exist (`ErrNoDomain`,
`ErrNoStorageVol`, `ErrNoStoragePool`, `ErrOperationInvalid`,
`ErrNoDomainMetadata`, `ErrStorageVolExist`, `ErrXMLError`, `ErrNoSupport`,
`ErrInvalidArg`, `ErrAuthFailed`, `ErrAccessDenied`). `libvirt.New(net.Conn)`
then `Connect()`. No new dependencies: `x/crypto/ssh`, `kdomanski/iso9660`
are already direct requires; use `slices`, never `sort` (depguard).
## Design decisions recorded in the spec (step 1)
- **Package layout** (`internal/provider/kvm2/`): `kvm2.go` (type, `New`,
logger, per-name keyed lock), `id.go`, `state.go`, `errors.go`, `xml.go`,
`seed.go`, `backend.go` (ctx-first seam over the 20 go-libvirt calls, adapter
with a per-RPC deadline), `ssh.go`, `hosts.go` (one cached connection per
host, lazy dial, dialer seam), `observe.go` (the single decorator: one span
`kvm2.<Call>` per RPC plus the V(1)/V(2) log line), `image.go`, `create.go`,
`get.go`, `delete.go`, `list.go`, `check.go`, plus `fake_test.go`,
`sshserver_test.go`, `kvm2_integration_test.go`. Names are suggestions for
the briefs; the implementer may choose others by responsibility.
- **Seam:** every seam method takes `ctx`; the adapter runs the blocking call in
a goroutine and selects on result, `ctx.Done()` and a timer (dial 10s, RPC
30s, `DomainCreate` 90s, upload/download ctx plus a 20m cap). On abandon it
closes the connection and marks it dead; the next call redials. The image
download runs on a dedicated connection so a cancelled poll cannot cut it off.
- **Decorator wrapped once:** applied inside the real dialer where a connection
is born, so provider code never sees an undecorated backend. Spans `kvm2.dial`
(in the dialer) and `kvm2.download` (in `image.go`) are the two non-RPC spans.
A reflection test asserts every seam method is instrumented.
- **Metadata:** own namespace
`https://gitea.home.hrajfrisbee.cz/kacerr/egress-proxies-operator/kvm2`,
element `<instance>` with `managed`, `uid`, `createdAt`. kvm2 never lists or
adopts v1 domains; a `kvm` and a `kvm2` entry may share a host.
- **Adoption guard:** an existing domain whose metadata `uid` is non-empty and
differs from `req.UID` makes `Create` return `ErrPermanent`.
- **Quota precheck:** before creating volumes, `StoragePoolGetInfo`; needed =
image `Content-Length` (if not cached) + `diskSizeGb` + 16 MiB; below that
returns `ErrQuotaExceeded` naming pool, available and needed.
- **Download** requires `Content-Length` (Permanent otherwise), streams through
`sha256` into `StorageVolUpload`, refreshes the pool, writes the marker
volume last; a failed attempt deletes the partial volume.
- **Classify:** `ErrNoDomain`, `ErrNoStorageVol` NotFound, but Transient inside
`Create`; `ErrNoStoragePool`, `ErrXMLError`, `ErrNoSupport`, `ErrInvalidArg`,
`ErrAuthFailed`, `ErrAccessDenied` Permanent; SSH authentication failures and
host-key mismatches or unknown host keys Permanent; missing key or
known_hosts file Transient (Secret pending), unparsable key Permanent;
cancellation, deadlines, `net.OpError`, `io.EOF` and anything else Transient.
- **Same-name concurrency:** a keyed mutex per instance name serialises Create
and Delete for one name.
- **State:** nostate Provisioning; running or blocked without IP Provisioning,
with IP Running; everything else, including unknown codes, Stopped. IP from
the guest agent first, then ARP, first non-loopback non-link-local IPv4.
- **Checker:** per host concurrently on a fresh dedicated connection (dial,
`ConnectGetLibVersion`, `ConnectGetHostname`, pool lookup and info, close), so
it never touches the pooled connection. Reasons: shared ones plus
`PoolNotFound`, `PoolNotActive`, `CredentialsUnavailable`. Details `user`,
`hostname`, `libvirtVersion`, `pool`, `poolState`, `poolCapacityGiB`,
`poolAvailableGiB`.
- **Rotation:** SSH material is read on every dial, so Secret rotation needs no
restart.
- **Config validation messages:** `validateKVM` is refactored into
`validateKVMBlock(prefix, block string, c *KVMConfig)` so messages say
`kvm2.` for the new block; `validateKVM` stays as a wrapper for `kvm`.
## Steps (one commit each, on `feat/kvm2-provider` off `main`)
PR #7 is merged, so `main` carries the three docs. Step 0 begins with
`git fetch origin && git switch main && git pull --ff-only`.
Every subagent brief opens and closes with: "Do not open, grep, list or import
anything under `internal/provider/kvm/`. If a detail seems to need that code,
stop and report the question instead." Each brief names the files it may create
or edit and nothing else, lists the inputs (the three docs, the spec,
`docs/kvm-provider.md` for backend facts, CLAUDE.md conventions), the
acceptance tests by name and the commands to run, and asks the agent to end
with "files read outside the allowed set: none".
- **Step 0 (orchestrator):** copy this plan to `docs/plans/<ts>-kvm2-provider.md`
with the real timestamp, commit alone, create the branch. Execution summary
`docs/plans-executions/<ts>-kvm2-provider.md` with the checklist after step 1.
- **Step 1 (orchestrator): the spec.** Copy `docs/provider-spec-template.md` to
`docs/kvm2-provider-spec.md` and answer all 24 entries with the profiles and
decisions above; every Evidence line cites the user guide, a go-libvirt symbol,
or "verified on beelink in step 11". Entry 21's checklist lists the concrete
test names from steps 4 to 7. Verify: no blank Answer lines, every axis code
used matches contract §4.20, no reference to `internal/provider/kvm/`.
- **Step 2 (subagent): config wiring, IDs, state, classify.** Edits
`internal/provider/config.go` (`KVM2 *KVMConfig` field, `setBlocks`,
allow-list, dispatch), `config_kvm.go` (`validateKVMBlock`),
`config_test.go` and `config_kvm_test.go` rows. Creates `kvm2.go` (type,
assertions, `New` that never dials and warns on unreadable SSH files, stub
methods), `id.go`, `state.go`, `errors.go` with tests:
`TestFormatProviderID_roundTrip`, `TestParseProviderID_malformed`,
`TestMapState_table`, `TestFirstIPv4_filters`, `TestClassify_table`. Also
registers `"kvm2": kvm2.New` in `cmd/main.go` so the two type lists change in
one commit. Verify: three `"kvm2"` sites in `config.go`, no import of
`provider/kvm`, `go test -race ./internal/provider/... && go vet ./...`.
- **Step 3 (subagent): pure builders.** `xml.go`, `seed.go` and tests:
`TestDomainXML_fileDisksWithResolvedPaths`, `TestDomainXML_uefiDefaultSecureBootOff`,
`TestDomainXML_biosEscape`, `TestDomainXML_metadataElement`,
`TestDomainXML_guestAgentChannel`, `TestParseMetadata_withNamespace`,
`TestParseMetadata_withoutNamespaceDeclaration`,
`TestParseMetadata_absentGivesEmptyUID`, `TestVolumeXML_overlayBackingStore`,
`TestSeedISO_filesAndLabel`, `TestSeedISO_userDataByteForByte`,
`TestSeedISO_tempDirRemoved`, `TestVendorData_isShellScriptNotCloudConfig`,
`TestVendorData_multipartWhenExtraProvided`. Verify: no `type="volume"` in
`xml.go`; temp dir under `/tmp` with a deferred removal.
- **Step 4 (subagent): transport, seam, decorator, fakes.** `backend.go`,
`ssh.go`, `hosts.go`, `observe.go`, `fake_test.go`, `sshserver_test.go` and
tests: `TestDial_forwardsUnixSocket`, `TestDial_unknownHostKeyRejected`,
`TestDial_mismatchedHostKeyRejected`, `TestDial_insecureIgnoreHostKeyAccepts`,
`TestDial_missingKeyFileIsTransient`, `TestDial_unparsableKeyIsPermanent`,
`TestDial_handshakeTimeout`, `TestCall_deadlineClosesConnectionAndReturnsTransient`,
`TestCall_cancelUnblocksRPC`, `TestHosts_oneDialPerHostUnderConcurrency`,
`TestHosts_redialsAfterDrop`, `TestHosts_unconfiguredHostIsPermanent`,
`TestObserve_everySeamMethodInstrumented`, `TestObserve_childSpansUnderParent`,
`TestObserve_logsOneLinePerCallAtV1`, `TestObserve_detailOnlyAtV2`,
`TestObserve_uploadBodyNeverLogged`, `TestSSH_keyMaterialNeverInErrorsOrLogs`.
Verify: no `context.Background()` outside tests and the keepalive goroutine;
`tracing.Start` in the decorator and the dialer; `go test -race -count=2`.
- **Step 5 (subagent): image cache and pool-space precheck.** `image.go` and
tests: `TestEnsureImage_downloadsAndWritesMarkerLast`,
`TestEnsureImage_cachedWithMarkerSkipsDownload`,
`TestEnsureImage_missingMarkerRedownloads`,
`TestEnsureImage_sha256ChangeRedownloads`,
`TestEnsureImage_sha256MismatchIsPermanentAndRemovesVolume`,
`TestEnsureImage_noContentLengthIsPermanent`, `TestEnsureImage_http404IsPermanent`,
`TestEnsureImage_http503IsTransient`, `TestEnsureImage_concurrentCallersFetchOnce`,
`TestEnsureImage_cancelledMidStreamReturnsPromptly`,
`TestEnsureImage_streamsWithoutBuffering`, `TestCheckPoolSpace_belowNeededIsQuota`.
Verify: no `io.ReadAll` or `bytes.Buffer` on the download path; download span.
- **Step 6 (subagent): the four contract methods.** `create.go`, `get.go`,
`delete.go`, `list.go`, `kvm2_test.go`, finishing `kvm2.go`. Tests map 1:1 to
contract §11: `TestCreate_repeatReturnsSameIDAndCreatesNothing`,
`TestCreate_resumesAfterEachPartialStep`, `TestCreate_returnsOnlyWhenRunning`,
`TestCreate_rejectsUnknownPlacement`, `TestCreate_adoptionWithForeignUIDIsPermanent`,
`TestCreate_poolOutOfSpaceIsQuotaExceeded`,
`TestCreate_libvirtNotFoundIsTransientNotNotFound`,
`TestCreate_cloudInitSentinelNeverLoggedOrInErrors`,
`TestGet_afterDeleteReturnsNotFound`, `TestGet_duringDeleteStillReturnsInstance`,
`TestGet_backendBusyIsTransientNeverNotFound`, `TestGet_runningWithoutIPIsProvisioning`,
`TestGet_runningWithIPIsRunning`, `TestGet_agentPreferredOverARP`,
`TestGet_addressErrorIsProvisioning`, `TestGet_readsUIDAndCreatedAtFromBackend`,
`TestGet_malformedIDIsPermanent`, `TestGet_unconfiguredHostIsPermanent`,
`TestDelete_malformedIDIsPermanent`, `TestDelete_twiceSucceeds`,
`TestDelete_removesOverlayAndSeed`, `TestDelete_runningDomainDestroyedFirst`,
`TestDelete_toleratesMissingVolumesAndPool`, `TestDelete_unconfiguredHostIsNoOp`,
`TestDelete_concurrentSameNameSafe`, `TestListByTag_excludesUnmarkedDomains`,
`TestListByTag_emptyUIDStillListed`, `TestListByTag_acrossHostsInConfigOrder`,
`TestListByTag_unreachableHostFailsWhole`, `TestListByTag_unreadableDomainSkipped`,
`TestProvider_cancelledContextReturnsPromptly`,
`TestProvider_everyErrorIsWrappedWithConfiguredName`,
`TestLogging_v0SilentV1OutcomeV2Detail`. Verify: `CloudInit` appears in
`create.go` only as `len(req.CloudInit)`; stub strings gone; `Wrap` always
gets the configured name; `go test -race -count=2 ./internal/provider/kvm2/`.
- **Step 7 (subagent): Checker and the CRD comment.** `check.go`,
`check_test.go`: `TestCheck_oneEntryPerHostInConfigOrder`,
`TestCheck_hostsCheckedConcurrently`, `TestCheck_expiredContextReturnsPromptly`,
`TestCheck_reasonsTable`, `TestCheck_neverUsesPooledConnection`,
`TestCheck_detailsKeys`. Edit `api/v1alpha1/providerhealth_types.go`
(`status.type` comment) and run `make manifests`. Verify: only the CRD
description line changes under `config/`; `go test ./internal/providerhealth/`.
- **Step 8 (subagent): deployment, Makefile, integration test.** Edit
`config/manager/manager.yaml` (comment: the `kvm-ssh` mount serves kvm and
kvm2), `config/manager/providers_config.yaml` and
`config/samples/providers-config.yaml` (commented `kvm2` entry),
`config/samples/kustomization.yaml`, `Makefile` (`test-integration-kvm2`,
`-run TestKVM2`, `-timeout 40m`), `docs/testing.md` (suite rows and a kvm2
integration section). Create `config/samples/proxy_kvm2.yaml` (zone
`beelink`, provider `kvm2-home`) and `kvm2_integration_test.go`
(`KVM2_TEST_HOST`, `_USER`, `_SSH_KEY`, `_KNOWN_HOSTS`, `_BRIDGE`,
`_IMAGE_URL`; optional `_IMAGE_SHA256`, `_POOL`, `_SOCKET`, `_TIMEOUT`,
`_KEEP`, `_LOG_LEVEL`) asserting in order: `New` does not dial; `Check`
Ready with details; `Create` returns `hosts/<host>/domains/<name>`; second
`Create` same ID, domain count unchanged; `Get` reaches Running with an IPv4
and the written UID and CreatedAt; `ListByTag` contains it; `Delete` nil;
`Get` NotFound; `Delete` again nil; pool holds base and marker, no per-VM
volumes; with an in-memory span recorder, `kvm2.DomainDefineXML` and
`kvm2.dial` spans are children of `provider.create`. Verify:
`go vet -tags=integration ./... && go vet -tags=e2e ./...`,
`bin/kustomize build config/default` and `config/samples` parse,
`make -n test-integration-kvm2`.
- **Step 9 (subagent): documentation.** Create `docs/kvm2-provider.md`
(prerequisites incl. OVMF, cluster setup reusing `kvm-ssh`, config reference
pointing at the shared `KVMConfig` table, placement mapping and the
add-a-key-never-edit warning, what lands on the host incl. the metadata
namespace, image cache and quota precheck, health-check reasons, logging and
span names, migration from `kvm`, known gaps). Edit
`docs/providers-existing.md` (fourth column in the three tables and a kvm2
section), `docs/architecture.md` (components row, §5 block, §10 span
topology, §11 row, Decisions bullets), `README.md` row, `docs/providers.md`
guide link. Verify with the link and symbol sweep from the docs-split
execution log; no em-dashes in new prose.
- **Step 10 (orchestrator): milestone review.** `go-operator-reviewer` on the
branch with the spec and contract as inputs, then `/code-review high`,
`make lint`, `make test` (full, with envtest). Fix accepted findings with
regression tests in one commit; record rejected ones in the summary.
- **Step 11 (orchestrator with the user): real-host validation, changelog, MR.**
Hand the user a host checklist (leftover `proxy-*` domains and volumes from
earlier runs, OVMF installed, pool running with free space, `br0` up, the
known_hosts entry) and the `KVM2_TEST_*` export lines; do not run remote
probes unprompted. With the user's go-ahead run `make test-integration-kvm2`
(a permission prompt is expected; the allow-list entry
`make test-integration-kvm *` does not match) and, if wanted, the in-cluster
pass on kind: build and load the image, deploy, a scratch ConfigMap with the
`kvm2-home` entry, apply `proxy_kvm2.yaml`, wait for `Ready`,
`kubectl get ph kvm2-home -o yaml`, a `curl -x` through the proxy, delete
and confirm the domain and volumes are gone; scratch overrides are not
committed. Record cold download duration and time to IP. CHANGELOG entry
after the user confirms; push with the token helper; MR via the Gitea API
with base `main`; never merge.
## Verification
- Per step: the checks listed above, plus `go build ./... && go test -short ./...`
before every commit and `git diff --stat` reviewed for files outside the
step's allowed set.
- Clean-room check on every step's diff: `grep -rn 'provider/kvm"' internal/provider/kvm2/`
empty; the agent's closing line "files read outside the allowed set: none";
identifiers that coincide with v1's names from the appendix (`libvirtAPI`,
`hostPool`, `imageCache`, `tracingAPI`, `lazySSH`) are treated as a smell and
questioned.
- Contract mapping: after step 7, every rule of contract §2 and §8 has a named
test in the package; the §11 list maps 1:1 to test names (recorded in the
spec's entry 21).
- End to end: the integration run's ordered assertions (step 8) on beelink, and
the optional kind pass (step 11).
## Risks
- Subagent scope creep into `internal/provider/kvm/`: instruction-only guard,
mitigated by the closing attestation and the diff greps.
- `.claude/settings.json` is locally modified and uncommitted; it stays out of
feature commits, and the permission prompt for the new make target is
accepted rather than pre-approved.
- `Content-Length` is required for downloads; the Debian and Ubuntu mirrors send
it.
- The cancelled-call policy still drops other in-flight RPCs on that host once;
the dedicated download connection removes the expensive case and the spec
states the rest.
- `go-operator-reviewer` hunts over-abstraction; the seam, the decorator and
the dialer interface each have one production implementation and one fake,
which the contract requires (§4.15, §11).
docs/plans/2026-09-17-1515-kvm2-review-fixes.md
+640
View File
@@ -0,0 +1,640 @@
# Plan: fix the kvm2 code-review findings
**Created:** 2026-09-17 15:15
## Context
The review of `feat/kvm2-provider` (`review/REPORT.md`) confirmed 52 findings: 4 high,
20 medium, 28 low. We fix them one at a time, highest severity first, each as its own step and
its own commit. This plan is the campaign; the execution file's checklist is the authoritative
progress record. Steps are detailed as we reach them, so the plan is never a fiction about work
not yet designed.
Precedent for this shape: `docs/plans/2026-09-10-2222-kvm-review-fixes.md`.
## Steps
- [x] **Step 0** — Save this plan into the repo, commit alone, before any code.
- [x] **Step 1** — H1: `sshSocket.Close` ran the blocking close before the unblocking one, and
`markDead` killed inside a `sync.Once` before signalling dead. Shipped in `0239fb7`; the
reasoning is in the execution file.
- [ ] **Step 2** — H2: an error during `StorageVolUpload` wedges go-libvirt's stream sender.
- [ ] **Step 3** — H3: `address()` swallows every error and demotes healthy proxies.
- [ ] **Step 4** — H4: the base image cache is shared across provider entries.
- [ ] **Step 5** — Validate H1-H4 in-cluster against the real hypervisor (detailed below).
- [ ] **Step 6** — Route klog into the structured logger (found during Step 5; detailed below).
- [ ] **Step 7+** — the mediums, from `review/REPORT.md`. Detailed when reached.
---
## Step 2 — H2: a source error must not reach go-libvirt's stream sender
### The bug
`uploadImage` (`image.go:255-296`) hands a reader over `resp.Body` to `StorageVolUpload`. I
verified the mechanism in the pinned `go-libvirt@v0.0.0-20260814190004`:
```go
// rpc.go requestStream
abort := make(chan bool) // UNBUFFERED
outErr := make(chan error) // UNBUFFERED
go func() { outErr <- l.socket.SendStream(serial, proc, program, out, abort) }()
resp, err = l.processIncomingStream(c, in)
if err != nil {
abort <- true
return resp, err // outErr is NEVER drained on this path
}
err = <-outErr
```
`SendStream` (`socket/socket.go`) holds a 4 MiB buffer, polls `abort` once per loop, and on a
**non-EOF reader error** sends a `StatusError` packet and returns. It then blocks forever on
`outErr <-`, because the main goroutine is still inside `processIncomingStream`. libvirtd
answers the `StatusError` with an error, so `processIncomingStream` returns an error and the
main goroutine blocks forever on `abort <- true`. **Deadlock: `StorageVolUpload` never returns.**
kvm2's `call` rescues only the caller, after `budgets.transfer` = 20 minutes, and the socket
kill does **not** unblock either goroutine — both are parked on channel sends, not on socket
I/O, which is what the comment at `backend.go:152-156` gets wrong. So every failed upload leaks
kvm2's call goroutine, go-libvirt's sender goroutine and 4 MiB, permanently. The pod limit is
128Mi and the reconciler retries on backoff.
Three shapes, only the first of which is ours to fix:
- **(A) reader error** — a truncated or failing HTTP body. The common case.
- **(B) server-side error mid-upload** — ENOSPC, or the volume deleted underneath. After
`SendStream` has returned there is nothing any reader, ctx or socket close can do: confirmed
unfixable from our side.
- **(C) our own abandonment** — ctx cancel or the 20 m budget kills the socket, which makes
`processIncomingStream` error. Same tail as (B).
### The fix (workaround; the defect stays upstream)
No fork, no `replace`. Two changes that between them close (A) and the realistic half of (C).
**2a. Never hand go-libvirt a reader that errors** — `internal/provider/kvm2/backend.go`, at the
seam rather than in `image.go`, so it covers the seed and marker uploads too and stays out of
the file Step 4 rewrites:
```go
type eofReader struct {
r io.Reader
mu sync.Mutex
err error
}
func (e *eofReader) Read(b []byte) (int, error) {
n, err := e.r.Read(b)
if err == nil || err == io.EOF {
return n, err
}
e.mu.Lock()
if e.err == nil {
e.err = err
}
e.mu.Unlock()
return n, io.EOF
}
func (e *eofReader) sourceErr() error { e.mu.Lock(); defer e.mu.Unlock(); return e.err }
```
`libvirtBackend.StorageVolUpload` wraps `r` in one, and after a nil-error return converts a
recorded `sourceErr()` into the returned error. `io.EOF` is the one outcome that ends the stream
the normal way: libvirtd gets `StatusOK`, replies OK, `processIncomingStream` returns nil, the
main goroutine reaches `err = <-outErr` and both goroutines unwind.
`countingReader` is untouched — it sits under the wrapper, still sees the real error, so
`uploadImage`'s `readErr != nil` branch and `TestEnsureImage_shortReadRemovesVolumeAndIsTransient`
keep working verbatim.
**Unverified, ~85%:** that libvirtd answers a short-but-clean stream finish with OK rather than
an error. `virFDStreamWrite` enforces `length` only on writes *past* it and nothing compares
`offset` to `length` on the close path, which is also why `virsh vol-upload` of a small file
into a large volume works. If it answers with an error instead, we land back in a deadlock by a
different door — but that is exactly today's behaviour, so the change is non-worsening either
way. Probe recipe goes in the commit message; it needs a real libvirtd and a write, so it is
for the user to run, not an agent.
**2b. Make the transfer deadline land on the HTTP source, not on the libvirt socket.**
`p.do` already builds the request with `http.NewRequestWithContext` (`image.go:360`), so the
body is ctx-bound, and `http.DefaultClient` has no timeout of its own by deliberate choice
(`kvm2.go:100`). Give the image GET in `fetchImage` a deadline of `defaultBudgets.transfer`
minus a grace of one minute (precedent for reaching for `defaultBudgets` directly:
`discardVolume`, `image.go:331`).
A stalled mirror then ends by the *body* failing, which 2a turns into `io.EOF` and a clean
unwind, roughly a minute before the libvirt budget would have killed the socket and leaked.
That converts the realistic repeated (C) — a slow or stalling mirror on a 415 MB image — from a
permanent leak into an ordinary Transient error. The socket kill stays as the backstop for
libvirtd itself hanging, which is the residual we cannot close.
**2c. Close the dedicated connection before unlinking the file.** Move the `discardVolume` call
out of `uploadImage` (`image.go:291-294`) up into `fetchImage`'s upload-error branch, where the
deferred `fresh.Close()` has already run, so libvirtd has let go of the stream's fd before the
volume is deleted. `uploadImage` then drops its `be` parameter and its private logger — a net
deletion. Doing it this way rather than adding an explicit `Close` keeps
`image_test.go:573`'s `closeCount() == 1` assertion true.
### Tests, docs, scope
One test: `TestEOFReader_sourceErrorNeverReachesTheStream` in `backend_test.go` — a table with
"error after data", "error with no data", "plain io.EOF passes through", "first error wins".
~35 lines.
**Deliberately not tested:** the abort protocol end to end. `libvirttest` reads a fixed 28-byte
header per packet and implements no stream procedures, so pinning this hermetically would mean
writing a libvirt RPC server stub — 150-250 lines whose only assertion is about *upstream's*
goroutines. That is the bloat we cut for H1.
Docs: a paragraph in `docs/kvm2-provider.md`'s *The image cache* section saying a truncated
response is reported out of band and why, and a *Known gaps* bullet naming shapes (B) and (C)
as an upstream defect with the two-character patch quoted, so a future reader knows it is not
ours and what the escape hatch is:
```go
abort := make(chan bool, 1)
outErr := make(chan error, 1)
```
File that as an upstream issue/PR against `digitalocean/go-libvirt` and link it from the bullet.
---
## Step 3 — H3: an unreadable address must not look like a missing one
### The bug
`address()` (`get.go:104-119`) tries the guest agent then the host's ARP table. Any error that
is not `context.Canceled`/`DeadlineExceeded` is logged at V(2) and skipped; after both it
returns `("", nil)`. `mapState(active, "")` is `StateProvisioning` (`state.go:20-21`), the
controller clears `p.Status.IP` on Provisioning (`proxy_controller.go:208-211`), and the health
engine prunes the proxy's accumulated state and calls `Metrics.ForgetProxy` in one tick once
`EffectiveHost()` is empty.
The trigger that matters: another reconcile's cancel or budget overrun kills the **shared**
pooled connection, and `call` then returns `transientf("... connection is closed: ...")` with no
ctx sentinel, which `get.go:108` does not recognise. `Get` returns a *successful* Instance in
Provisioning with `IP: ""`, and a healthy proxy is pulled out of rotation by an unrelated
reconcile's timeout.
### The fix
A booting VM and a dead link are distinguishable: on beelink the agent source errors while the
ARP source returns success-with-no-addresses (`docs/kvm2-provider-spec.md:146` — the host never
sees the guest's traffic). So *one* source failing is normal; *every* source failing, or a
connection already gone, is not.
```go
func address(ctx context.Context, log logr.Logger, be backend, dom libvirt.Domain) (string, error) {
errs := make([]error, 0, len(addressSources))
for _, s := range addressSources {
ifaces, err := be.DomainInterfaceAddresses(ctx, dom, uint32(s.src), 0)
if err != nil {
if isDead(be) || errors.Is(err, context.Canceled) || errors.Is(err, context.DeadlineExceeded) {
return "", fmt.Errorf("reading addresses of domain %s from the %s source: %w", dom.Name, s.name, err)
}
log.V(2).Info("no addresses from source", "source", s.name, "error", err.Error())
errs = append(errs, fmt.Errorf("%s source: %w", s.name, err))
continue
}
if ip := firstIPv4(ifaces); ip != "" {
return ip, nil
}
}
if len(errs) == len(addressSources) {
return "", fmt.Errorf("%w: reading addresses of domain %s: every source failed: %w",
provider.ErrTransient, dom.Name, errors.Join(errs...))
}
return "", nil
}
```
The explicit `provider.ErrTransient` wrap is load-bearing, not decoration: without it `classify`
picks the first libvirt code out of the join tree, and `ErrInvalidArg`/`ErrAccessDenied`/
`ErrNoSupport` map to **Permanent** — an address read failing would latch a *running* proxy as
Failed, a worse version of the harm being fixed. `isDead(be)` (`backend.go:50-61`, already
works through the `observed` decorator and the test fake) is not strictly needed — a dead link
fails both sources anyway — but it fails faster, attributes better, and is the same predicate
M15 will need for `ListByTag`.
Also fix the now-wrong comment at `get.go:100-103` ("only ctx ending is an error").
**What this does not fix, on purpose.** A guest agent restarting inside a running VM still
demotes it: the agent errors, ARP returns empty, and the provider cannot tell that from a VM
that is still booting. The discriminator is the *previous* IP, which the provider deliberately
does not hold (`TestGet_readsUIDAndCreatedAtFromBackend` pins "read back, not remembered") and
the controller already has. File as a follow-up finding against
`proxy_controller.go:208-211`: on Provisioning, keep a Proxy's existing IP and `Provisioned`
condition for N polls before clearing, or give the health engine a grace period before pruning.
The one kvm2-local alternative — treating "active, no address, domain older than a provisioning
window" as an error using the mark's `createdAt` — is a design change wearing a bug fix's
clothes, and it invents a provisioning deadline that belongs to the controller. Rejected.
### Tests
- `TestGet_arpFallbackWhenAgentFails` (`get_test.go:135`) passes unchanged — one source of two.
- `TestGet_addressErrorIsProvisioning` (`get_test.go:159`) must be **re-pointed**. It fails both
sources with `ErrAgentUnresponsive`, which is not realistic: code 86 is agent-specific and the
ARP source never produces it. Rewrite as `TestGet_agentFailsAndARPEmptyIsProvisioning` — agent
errors, ARP returns no addresses and no error — which is the beelink boot and the behaviour
the new rule must not break. Nothing covers it today.
- One new test, `TestGet_everyAddressSourceFailingIsTransient`, two subtests: both sources error;
and the pooled link is dead before the address read (the fake implements `liveness`). Assert
Transient, not NotFound, not Permanent.
Docs: `docs/kvm2-provider-spec.md:146-151` says "errors from either source mean 'no address yet'",
now false. The `ErrTransient` row of the error-class table in `docs/kvm2-provider.md`.
---
## Step 4 — H4: the base image cache must not be shared across provider entries
### The bug
`baseVolumeName` (`volume.go:17`) is `base-<key>.<format>`, with a `.sha256` marker beside it.
`internal/provider/kvm/image.go:44-56` builds **byte-identical** names in the same pool — for
URL images too (kvm takes `hex(sum[:8])`, kvm2 takes `hex(sum[:])[:16]`, the same 8 bytes) —
with a completely separate lock table. The shipped config has a live `kvm-home` entry and a
commented-out `kvm2-home` entry on the same host, the same key `debian-13` and the same default
pool, and the comment tells the operator to enable it. Both integration tests use key
`test-image`. On beelink both live overlays back onto `base-debian-13.qcow2`, which kvm created.
Consequences, given `cachedImage` accepts *any* 64-hex marker when no `sha256` is configured and
never re-hashes the base:
- Two entries, one key, different URLs: whichever fetched first owns the file and the other
silently boots the wrong image.
- A concurrent cold fetch: B finds A's base-without-marker, deletes it, recreates and uploads
its own; A finishes into the unlinked inode, writes the marker, and `StorageVolGetPath`
resolves A's handle to B's partial file — a guest boots from a half-written base.
- A `sha256` edit deletes the base in place under live overlays (`cachedImage` branch E), and
libvirt does not refuse to delete a volume that running domains use as a backing store.
### The fix — the provider entry's name in the volume name
`base-<entry>-<key>.<format>`, e.g. `base-kvm2-home-debian-13.qcow2`, with the marker following
automatically. Every production call site already goes through `imageSource.volumeName()`
(`image.go:49`), so this is a small change: give `resolveImage` the provider name and add an
`Entry` field to `imageSource`.
`Provider.name` is already carried (`kvm2.go:104`). But provider names are validated only as
non-empty and unique (`config.go:96-105`) — no charset rule — and a name with a `/` would
produce an unusable volume name, so **`New` must reject an entry name that is not a safe volume
name component**, using the existing `kvmIdentifier` shape. Fail fast at startup, in kvm2 only:
tightening the rule for every provider type is a broader change than this fix.
**What this buys, and what it does not.** It fixes the cross-entry sharing outright — kvm↔kvm2
and kvm2↔kvm2 — which is the wrong-image bug and the concurrent-fetch corruption, and it makes
the existing per-`Provider` `imageLocks` correct rather than a latent bug, since no two entries
can now contend for one name. It does **not** make the name mean the content, so two things
stay open and must be said plainly rather than quietly skipped:
- Editing `images.<key>.url` without changing the key is still a silent no-op. Already
documented behaviour ("add a new key instead"); left as is.
- Cross-*process* fencing still depends on the atomic-publish work (upload to a temp name,
verify, then publish). E4 refuted the rollout and leader-handover triggers for the shipped
manifests, so this is a narrow residual. Its own later commit.
**One companion fix, in this commit**, because the naming alone does not stop it and it is the
remaining path that unlinks a good file under a running domain: `cachedImage` branch E
(`image.go:129-130`, digest differs from the configured `sha256`) currently deletes the marker
and the base. Make it return Permanent instead — "this key is cached with digest X but Y is
configured; add the image under a new key, or delete the Proxies using it and evict the base."
That enforces the rule the contract already states instead of contradicting it. Keep the delete
for the no-marker and non-hex-marker branches, where the base is genuine debris.
### Migration
Nothing to do. The existing `base-debian-13.qcow2` on beelink is kvm's, kvm keeps using it
untouched, and kvm2 downloads its own copy once per host (~415 MB) when `kvm2-home` is enabled.
No window in which anything is replaced under a running guest. The pool needs room for both.
### Tests and docs
Tests are mostly literals becoming helpers: `volume_test.go:122-125`, `image_test.go:184,190,196`
(`wantVolume`), `create_test.go:64-70`, `delete_test.go:67`, `provider_test.go:409`; a small
`fx.baseName()` helper covers the last three. `image_test.go:281,298` should assert the log
field and the `kvm2.volume` span attribute **by prefix**, which keeps pinning that the key stays
readable in logs. `kvm2_integration_test.go:481-484` switches from `baseVolumeName(...)` to
`resolveImage(...).volumeName()`; its assertion that the base survives `Delete` is unchanged.
Two new tests:
- `TestBaseVolumeName_isPerProviderEntry` — two entries, same key, different names; the produced
name never equals the kvm provider's `base-<key>.<format>`.
- `TestNew_rejectsUnsafeEntryName` — an entry name with a `/` fails at construction.
**Not tested:** that libvirt permits deleting an in-use backing file (established from libvirt
source by the review; re-proving it needs a write on a real host), and the cross-process race
(no hermetic way, and its fix is deferred).
Docs: the *What lands on the host* table (`docs/kvm2-provider.md:198-212`), the eviction recipe
(`:305-318`, which currently names `base-debian-13.qcow2` and would delete kvm's file), the
migration section (`:346-370` — kvm2 never reuses a kvm base, one extra download, nothing to
delete), the prerequisites note that the pool needs one base per distro **per entry**, the
naming contract in `docs/kvm2-provider-spec.md` entries 4 and 10, the shared validator's message
at `internal/provider/config_kvm.go:139-141` ("it becomes the cached volume name
`base-<key>.<format>`", now wrong for kvm2), and the kvm2 block's comment in both
`providers-config.yaml` files. Leave `docs/kvm-provider.md` alone; kvm is unchanged.
---
## Step 5 — Validate in-cluster against beelink
### Why, and what this run can and cannot prove
The kvm2 provider has **never run against real libvirt**: Step 11 is unchecked in
`docs/plans-executions/2026-09-16-1333-kvm2-provider.md`, every `base-*` volume on beelink
predates it, and both live `proxy-*` domains carry the kvm v1 metadata namespace. Four fixes
have landed on code no hypervisor has executed. This run is first contact, and it is also the
gate the campaign's `CHANGELOG` entry waits on.
It proves the happy paths: the dial and health probe, a real 415 MB image download through the
reader H2 now wraps, H4's new volume name appearing beside kvm's untouched one, a second Proxy
hitting the cache, H3's address path reaching Running with an IP, and Squid answering a real
CONNECT probe.
It does **not** prove H2's error paths. The unverified ~85% claim — that libvirtd answers a
short stream finish with OK — only arises when a body *fails*, and a healthy download never
finishes short. The 19-minute HTTP deadline will not fire either. A green run means "the
wrapper did not break normal uploads", not "the deadlock is settled". That still needs the
scratch-pool probe recorded in the Step 2 execution entry.
### The cluster this runs in, and the one hazard
`kind-proxy-operator-demo` is **not a throwaway**. It runs the operator that currently owns
four Ready proxies, two of them real VMs on beelink:
```
proxy-beelink-1 Managed kvm-home Ready 192.168.0.166 True
proxy-beelink-2 Managed kvm-home Ready 192.168.0.94 True
proxy-kubernetes-demo-1/2 Managed kubernetes Ready 10.244.0.7/.8 True
```
Validating kvm2 means rolling the operator that owns them. That is low risk — the controller
and the whole `kvm` provider are untouched by H1-H4 — and rollback is one `kubectl set image`.
**Do not run a second operator against beelink with `kvm-home` in its config.** Orphan GC
(`internal/gc/gc.go:86-131`) lists every Proxy in *its own* cluster, and deletes any instance
whose UID is not in that set and is older than `--gc-min-age` (10 m, swept every 10 m). A
second cluster's live set would not contain `proxy-beelink-1/2`'s UIDs, so its GC would destroy
them within ten minutes. Using the existing cluster removes the hazard entirely: its live set
is the one that protects them.
### Steps
**0. Baseline** — record what is there before touching anything, so "did we break it" is
answerable: `kubectl get px -A`, `kubectl get ph`, and a read-only `virsh vol-list default`
plus `virsh list --all` on beelink.
**1. Build and load, under a new tag.**
```bash
make docker-build IMG=egress-proxies-operator:kvm2-dev
kind load docker-image egress-proxies-operator:kvm2-dev --name proxy-operator-demo
```
The tag **must** differ from the running `kvm-dev`. `make deploy` sets the image through
kustomize; an identical tag leaves the Deployment spec unchanged, so no rollout happens, so the
new process never starts, so the new providers config is never read. The ConfigMap carries no
generator hash, so nothing else forces a restart either.
The working tree is dirty (`review/`, `.claude/`), so `GIT_COMMIT` bakes in as `c10f98a-dirty`
and `--version` will say so. Cosmetic; commit or stash `review/` first to avoid it.
**2. Enable kvm2 in the deployed config.** Uncomment the `kvm2-home` block in
`config/manager/providers_config.yaml` (it is the shipped default, commented out). Keep its
`address: beelink.tail8c69c.ts.net:22` — the Tailscale FQDN, which is what
`~/.ssh/kvm_known_hosts` pins. `config/samples/providers-config.yaml` uses the short
`beelink:22`, which **fails host-key verification**; I hit exactly that when validating SSH
earlier in this session.
**3. Deploy.**
```bash
make deploy IMG=egress-proxies-operator:kvm2-dev
kubectl -n egress-proxies-operator-system rollout status deploy/egress-proxies-operator-controller-manager
```
`make deploy` rewrites `config/manager/kustomization.yaml`'s image block as a side effect; it
already carries `kvm-dev` from a previous run.
**4. STOP GATE — provider health, before anything is created.**
```bash
kubectl get ph
kubectl get ph kvm2-home -o yaml
```
`kvm2-home` must be `Ready 1/1` with reason `Reachable` and details carrying `hostname`,
`libvirtVersion`, `poolState: running` and `poolAvailableGiB`. This proves SSH, the host key,
the libvirt handshake and the pool **without creating a thing**, and it exercises the
dedicated-connection dial and close that H1 touched. A failure here is free to diagnose:
`CredentialsUnavailable`, `Unauthorized`, `Forbidden`, `PoolNotActive` each name their own
cause. Report and wait for a go before step 5.
**5. Two Proxies.** Two manifests in the scratchpad (not committed), modelled on
`config/samples/proxy_kvm2.yaml` — `proxy-kvm2-1` and `proxy-kvm2-2` in `default`, matching the
existing `proxy-beelink-*` convention, `provider: kvm2-home`, `zone: beelink`,
`machineType: small`, `image: debian-13`, port 3128 and the sample's Squid cloud-init. Apply
the first, let it finish, then the second — sequencing them is what makes the cache hit
observable.
```bash
kubectl get px -w
kubectl -n egress-proxies-operator-system logs -f deploy/egress-proxies-operator-controller-manager
```
**6. What to check, and which finding it exercises.**
| Check | Finding |
|---|---|
| `fetching base image into host pool` then `base image cached` with size, duration, digest | H2's wrapped reader on a real 415 MB upload |
| `virsh vol-list default` shows `base-kvm2-home-debian-13.qcow2` **and** `base-debian-13.qcow2` still there | H4: per-entry naming, kvm's cache untouched |
| Second Proxy logs no `fetching`, and the server is not hit again | H4 + the image lock: cache hit under the new name |
| `px` goes Provisioning → Running with an IP, without flapping back to Provisioning | H3: the address path, and no spurious demotion |
| `Healthy=True` | Squid answers a real CONNECT probe end to end |
| `virsh list --all` shows two new `proxy-*` domains | Create |
| After deleting both: domains and their `.qcow2`/`-seed.iso` gone, base image still present | Delete, and the cache surviving |
Optionally `curl -x http://<ip>:3128 https://example.com -I` from this box, if it routes to
192.168.0.0/24 — the existing kvm proxies live on that LAN.
**7. Cleanup.** `kubectl delete px proxy-kvm2-1 proxy-kvm2-2` and let the finalizer do the
work, then verify on beelink read-only. **Any leftover domain or volume: I print the exact
`virsh undefine --nvram` / `vol-delete` lines and you run them** — no write reaches the shared
host from me. Then decide whether the `kvm2-home` ConfigMap entry stays (keep it if you want
kvm2 running for real; `git checkout config/manager/providers_config.yaml` otherwise).
**Rollback at any point:**
```bash
kubectl -n egress-proxies-operator-system set image \
deploy/egress-proxies-operator-controller-manager manager=egress-proxies-operator:kvm-dev
```
### Who runs what
I drive steps 0-6 and stop at gate 4 for a go. Every beelink write is yours. Nothing here
touches `proxy-beelink-1/2`, `base-debian-13.qcow2`, your unrelated VMs, or the five
pre-existing orphan volumes the review found.
---
## Step 6 — Route klog into the structured logger
**Not a review finding.** Found while watching the Step 5 validation: some lines ignore the
zap encoder entirely, so no amount of `--zap-encoder=json` makes the stream parseable.
```
I0917 19:50:02.580097 1 leaderelection.go:258] "Attempting to acquire leader lease..." lock="..."
```
That is klog's own text sink. client-go's leader elector does `logger := klog.FromContext(ctx)`
(`leaderelection.go:257`), and controller-runtime hands the elector a context with no logr in it
(`pkg/manager/internal.go:461-473` — `mgr.Start` gets `ctrl.SetupSignalHandler()`, which carries
no logger). `klog.FromContext` falls back to `Background()`, and with no logger registered that
is klog's stderr writer. Nothing in this repo imports or configures klog; it is an indirect
dependency (`k8s.io/klog/v2 v2.140.0`) that ~60 linked packages use, including
`client-go/tools/{leaderelection,cache,record}`, `client-go/transport` and `client-go/rest`.
controller-runtime does not route it either — its only `klog.SetLogger` call is in its own test
suite.
### 6a. `cmd/main.go` — one line and an import
```go
logger := zap.New(zap.UseFlagOptions(&opts))
ctrl.SetLogger(logger)
klog.SetLogger(logger)
```
**Plain `SetLogger`, deliberately not `SetLoggerWithOptions(logger, klog.ContextualLogger(true))`.**
The contextual form is what klog's own docs recommend, because it skips a klogr round trip — but
it also disables klog's verbosity gate ("Such a logger cannot rely on verbosity checking in
klog", `contextual.go:83-86`), handing the decision to zap's level. With `Development: true`
that level is Debug, which enables V(1), so client-go's V(1) chatter that klog currently
suppresses at its default `-v=0` would start appearing. Plain `SetLogger` keeps klog's own check
in front ("klog will do its own verbosity checks before calling logger.V().Info",
`contextual.go:43-44`) and still redirects everything it does emit ("all log lines will be
suppressed from the regular output, and redirected to the logr implementation", `:47-48`).
Same fix, no new volume. Performance is irrelevant at this scale.
`go.mod` promotes `k8s.io/klog/v2` from indirect to direct. It is already in the module graph,
so nothing new is downloaded.
### 6b. `internal/discovery/server.go` — the other stderr bypass
The discovery API's `http.Server` (`:113-119`) sets no `ErrorLog`, so `net/http`'s own errors —
`Accept` failures, handler panics that escape — go to `log.Default()` and out to stderr in
stdlib format. `s.log` is already in scope two lines above:
```go
srv := &http.Server{
Handler: s.handler(),
ErrorLog: slog.NewLogLogger(logr.ToSlogHandler(s.log), slog.LevelError),
ReadHeaderTimeout: readHeadTimeout,
...
}
```
`logr.ToSlogHandler` is the pattern the repo already uses for the same purpose in
`internal/provider/gcp/wirelog.go:31-36`. This server is plain HTTP with no TLS, so `ErrorLog`
carries genuine faults rather than handshake noise from scanners — Error level is right.
### 6c. `config/manager/manager.yaml` — make the deployed stream parseable
```yaml
args:
- --leader-elect
- --health-probe-bind-address=:8081
- --providers-config=/etc/proxy-operator/providers.yaml
- --zap-encoder=json
- --zap-time-encoding=rfc3339
```
Verified that the flag wins over the in-code default: `addDefaults` only fills `NewEncoder`
`if o.NewEncoder == nil` (`zap.go:176-191`), and `BindFlags` sets it from `--zap-encoder`.
`--zap-time-encoding=rfc3339` matters because the JSON encoder defaults to `epoch`, which would
give `1758138422.31` instead of a timestamp anyone can read. Verbosity is left at debug, so the
per-RPC lines stay — that was the choice.
### Deliberately not doing
`opts := zap.Options{Development: true}` (`cmd/main.go:154-155`) stays. Flipping it to a
production default would change the format for every existing deployment, not just this one, and
the flags in 6c get the same result here without that blast radius.
### What stays unformatted regardless, and why that is fine
Flag-parse errors and `--version` (`cmd/main.go:160-163`) print before any logger exists, and
`utilruntime.Must` panics in `init()` run before `main`. controller-runtime's own metrics and
health servers also set no `ErrorLog` (`pkg/internal/httpserver/server.go:11-19`) and expose no
option to set one. All three are outside reach; none can fire during steady-state operation.
### Tests
None. All three changes are wiring: two struct fields and one manifest key. `cmd/main.go` has no
test coverage at all, and pinning `klog.SetLogger` hermetically means mutating global state that
would leak into every other test in the package. Verified by observation instead — see below.
Same call as H1's: a test that costs more than it protects does not earn its place.
### Verification
Rebuild, redeploy, and watch the rollout produce the line that started this:
```bash
kubectl -n egress-proxies-operator-system logs deploy/egress-proxies-operator-controller-manager \
| grep -i 'leader' | head -3
```
It must be JSON with an rfc3339 `ts`, not `I0917 ...`. Then confirm the whole stream parses:
```bash
kubectl -n egress-proxies-operator-system logs deploy/egress-proxies-operator-controller-manager \
| jq -e -s 'length > 0' >/dev/null && echo "every line is JSON"
```
That second command is the real check — it fails if *any* line is not JSON, which is exactly the
property being fixed.
---
## Ordering
**H2 → H3 → H4**, three commits.
H3 is orthogonal: it touches only `get.go` and `get_test.go`.
H2 and H4 both touch `image.go` but at different heights, and the order matters. H2's `image.go`
edit is a deletion — `discardVolume` moves out of `uploadImage` and up into `fetchImage` — while
H4 edits `imageSource.volumeName()`, `resolveImage`, and `cachedImage`'s digest branch ten lines
below. Landing H2 first means H4 rebases cleanly; the reverse means H4 is written against a
failure path H2 is about to restructure. Keeping H2's guard in `backend.go` rather than in
`countingReader` is part of the same separation.
Semantically, H2 makes `fetchImage` delete the base after the transfer connection is closed, and
that delete is only unambiguously safe once names are per-entry — so H2's commit message should
say the discard is still unfenced against a second entry sharing the name and that H4 closes it,
and H4's should point back. The deferred atomic-publish work then lands exactly where H2 put the
discard, on a volume this call created under a name no other entry can produce.
## Verification
```bash
go build ./... && go vet ./...
go test -race -count=1 ./internal/provider/kvm2/
make test
```
Per step, the check that actually matters is that the new test fails against the unfixed code —
stash the source edit, keep the test, watch it fail, restore. For H2 that means asserting
`eofReader` converts before the fix exists; for H3, that the dead-link subtest returns a
successful Provisioning instance today; for H4, that two entries produce one name today.
After H4, run the kvm2 integration test on beelink. **It has never run** — Step 11 is unchecked
in `docs/plans-executions/2026-09-16-1333-kvm2-provider.md`, every `base-*` on the host predates
kvm2, and both live domains carry the kvm v1 metadata namespace. It is the first end-to-end
exercise of the new name and does a real cold download beside kvm's untouched base.
Each step: its own commit on `feat/kvm2-provider` with the execution-file entry and its checklist
line ticked. `CHANGELOG.md` lands with a "validate and record" step once the user confirms the
work, as in the kvm campaign. Trailer:
`Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>`.
docs/plans/2026-09-18-1502-proxmox-provider.md
+884
View File
@@ -0,0 +1,884 @@
# Plan: proxmox, a Proxmox VE provider built from the provider contract
**Created:** 2026-09-18 15:02
## Context
> **Superseded in places by the spec.** Phase 0 recon found ten corrections
> to the design below (listed in the execution log), and Phase 1 folded them
> into [docs/proxmox-provider-spec.md](../proxmox-provider-spec.md). Where a
> phase step here disagrees with the spec (the Create allocation loop, Get's
> check order, resize and seed delete as tasks, no delete before upload,
> `isoStorage` required, `vlanTag`, the agent messages), **the spec wins**.
> Test names in the phases below remain the minimum; spec entry 21 adds the
> ones for the corrections.
The provider contract (`docs/provider-contract.md`), the spec template
(`docs/provider-spec-template.md`) and the kvm2 build proved the workflow:
spec first, then implementation against the contract, one commit per step.
This plan adds a third VM-style backend, `proxmox`, on the Proxmox VE 9.x
REST API at `https://<host>:8006/api2/json`. The user has a real host at
192.168.2.8 running **9.2.20** (Debian 13 Trixie, QEMU 11) and prepares the
API token and a cloud-init-capable template before Phase 0 (checklist
below). Everything below is written against 9.2; where 8.x differed the
difference is called out, because the appendix and the guide must name the
version they were proven on.
The implementation runs in sessions whose context gets compacted, so every
phase is self-contained: it names what to read first, what it creates or
edits, the test names it must add, a `Verify:` clause, a "Done when" line
and a commit subject. A fresh session reading only this plan, the spec
(`docs/proxmox-provider-spec.md`, from Phase 1 on) and the execution log's
checklist (`docs/plans-executions/2026-09-18-1502-proxmox-provider.md`) must be able to
pick up where the last one stopped.
**Resumption protocol, run at the start of every phase.** Read this plan's
Context and the phase entry; read the spec; read the execution log's
checklist; run `git branch --show-current` (must be `feat/proxmox-provider`
from Phase 0 on), `git status --short` and `git log --oneline -8`. The first
unchecked box is the phase to do. If its files already exist partially, its
"Done when" line says whether it counts as done; finish it, never redo it. A
phase is finished only when its "Done when" holds and its execution-log entry
(a few sentences, exact commands, "worth noting", the box ticked) is
committed together with the code. Follow `CLAUDE.md` for the log format.
## Before Phase 0: what the user prepares on 192.168.2.8
Run as root on the Proxmox node. **`VM.Monitor` was removed in PVE 9.0**, so
the agent read rests on `VM.GuestAgent.Audit` alone and the role below must
not mention `VM.Monitor`. Drop any privilege `pveum` still rejects and note
it for F17.
```bash
pveum user add proxy-operator@pve --comment "egress-proxies-operator"
pveum role add ProxyOperator -privs "VM.Allocate VM.Clone VM.Audit VM.Config.CDROM VM.Config.CPU VM.Config.Memory VM.Config.Disk VM.Config.Network VM.Config.Options VM.PowerMgmt VM.GuestAgent.Audit Datastore.Audit Datastore.AllocateSpace Datastore.AllocateTemplate Sys.Audit SDN.Use"
pveum aclmod / -user proxy-operator@pve -role ProxyOperator
pveum user token add proxy-operator@pve operator --privsep 0
# The secret is printed once. Save it on the workstation as one line:
# proxy-operator@pve!operator=<secret>
# in ~/.config/proxmox-token (mode 0600). The same file becomes the
# Kubernetes Secret later.
cp /etc/pve/pve-root-ca.pem /tmp/ # copy to the workstation as ~/.config/proxmox-ca.pem, or use insecureSkipTLSVerify
```
A cloud-init template without a Proxmox cloud-init drive (the provider puts
its own NoCloud ISO on `ide2`). Debian 13's BIOS boot path did not boot
under kvm2, so the template boots UEFI:
```bash
wget https://cloud.debian.org/images/cloud/trixie/latest/debian-13-generic-amd64.qcow2
qm create 9000 --name debian-13-template --memory 2048 --cores 2 --ostype l26 \
--machine q35 --bios ovmf --efidisk0 local-lvm:1,efitype=4m,pre-enrolled-keys=0 \
--scsihw virtio-scsi-pci --net0 virtio,bridge=vmbr0 \
--serial0 socket --vga serial0 --agent 1
qm set 9000 --scsi0 local-lvm:0,import-from=/root/debian-13-generic-amd64.qcow2
qm set 9000 --boot order=scsi0
qm template 9000
```
Facts the user hands over at Phase 0: the token file path, the CA file path
(or "insecure"), the node name, the template VMID, the ISO-content storage
id (usually `local`), the VM disk storage (`local-lvm`), the bridge, and
whether the operator's cluster can route to that bridge's LAN. Everything
else Phase 0 reads from the API.
## Design decisions made with the user (not re-opened)
- **Backend and client.** Proxmox VE 9.x REST API, JSON, HTTP/1.1 only
(pveproxy offers no HTTP/2 on any release; only Proxmox Backup Server
does).
Thin client on `net/http` (`client-kind: thin-client`), no SDK, no new
dependency. Auth header
`Authorization: PVEAPIToken=<user>@<realm>!<tokenid>=<secret>`, read from a
mounted token file on every request (rotation free). TLS: `caFile` or
`insecureSkipTLSVerify` (warned at startup). Child spans from a parent-gated
otelhttp transport: `tracing.RestConfigWrapper()(base)` returns a plain
`http.RoundTripper` and works as is (`child-spans: transport-wrapper`).
- **Image source.** Clone a prepared template. Config catalogue
`templates: {<key>: {vmid: 9000}}`; `Placement.Image` is a key.
`Placement.Zone` is a node name from `nodes: [..]` (optional with one node).
`Placement.MachineType` is a key of `machineTypes: {<key>: {cores, memoryMiB,
diskGiB}}`; `diskGiB` (optional) grows the boot disk via `PUT .../resize`.
`Placement.Region` is ignored (documented).
- **Identity.** `primary-key: allocated-id`. VMID derived deterministically
from the Proxy UID (FNV-1a 64 into `vmidRange: {min, max}`, default
100000 to 199999) with forward probing on collision; a clone whose `newid`
is taken fails with a 500, which is the allocate-then-verify step. Create
first looks the name up in `GET /cluster/resources?type=vm`; an existing VM
with that name must carry our mark and a matching or empty UID, else
Permanent. providerID is location-free: `vms/<vmid>/<name>` (VMs migrate;
the node is resolved from `/cluster/resources` on every call). Get and
Delete verify the VM at that VMID has that name and our mark; a different
name means our VM is gone and the VMID was reused: Get returns NotFound,
Delete returns nil without touching it.
- **Cloud-init.** NoCloud seed ISO built in-process by reusing kvm2's seed
code, extracted into a shared package `internal/provider/nocloud` (its own
phase). Uploaded to `isoStorage` (default `local`) on the target node via
`POST /nodes/{node}/storage/{storage}/upload` (multipart, `content=iso`,
`filename=<name>-seed.iso`, streamed through `io.Pipe`, returns a UPID),
attached as `ide2: <storage>:iso/<name>-seed.iso,media=cdrom`. The template
must have no Proxmox cloudinit drive on ide2 (documented prerequisite).
Vendor-data installs qemu-guest-agent like kvm2 (`installGuestAgent`,
`vendorData`).
- **GC mark.** `gc-mark: written-metadata` in the VM `description`, which the
clone call accepts (`tags` is not a clone parameter, so tags would be a
mark window). Content, one per line: `proxy-operator-managed=true`,
`proxy-operator-uid=<uid>`, `proxy-operator-created-at=<RFC3339>`.
`created-at: written-at-create`. The config step after the clone also sets
tag `proxy_operator` for UI filtering only. **Underscore, not hyphen:**
current PVE tag validation accepts only lowercase letters, digits and
underscore, and a hyphen is rejected; the underscore form is valid under
both the strict and the permissive reading, so it is used regardless of
what F9 finds. ListByTag: one
`GET /cluster/resources?type=vm`; candidates are entries with tag
`proxy_operator` or name prefix `proxy-`; a `GET /nodes/{node}/qemu/{vmid}/config`
per candidate for the description; unmarked skipped; one unreadable VM
skipped with a V(1) line unless ctx ended; the resources call failing fails
the whole list (`list-partial: fail-whole`).
- **Create** (`create-return: blocking`, `provisioning-state: create-blocks`,
`async-ops: task-handle` and `backend-lock`), every step idempotent, in
this order: validate placement (Permanent before any call); resources
lookup by name (adopt or allocate); `POST /nodes/{tmplnode}/qemu/{tmplvmid}/clone`
`{newid, name, description, full=1, target?, storage?}`; wait the task
(`exitstatus != OK` is Transient with the message; a VM with `lock=clone`
on retry is waited on); upload the seed (always rebuilt, delete then
upload; before the config step because `ide2` may refuse a volume that
does not exist yet); config (`cores`, `memory`, `net0` bridge override if
configured, `agent=1`, `tags`, `ide2` seed); optional resize;
`POST .../status/start` and wait; poll `status/current` until `running`
(capped); return `vms/<vmid>/<name>`. Retry: name found running returns
the ID; found stopped with our mark continues from the seed step; found
locked waits.
- **Get.** Resources lookup by vmid: absent is NotFound; name mismatch or no
mark is NotFound (logged); `lock` set is Transient; `running` reads
`GET /nodes/{node}/qemu/{vmid}/agent/network-get-interfaces`, first
routable IPv4 excluding loopback and link-local gives Running, else
Provisioning (the agent not running is a 500 "QEMU guest agent is not
running", which is Provisioning, not an error); `stopped` is Stopped;
`unknown` is Transient; `qmpstatus=paused` is Stopped; anything else is
Stopped. UID and CreatedAt come from the description on every call, so a
Get is three calls: resources, config, agent.
- **Delete** (`delete-return: waits`, `secondary-resources: explicit`):
resolve by vmid via resources; absent or foreign is nil (logged); running
gets `POST status/stop` and a task wait; then
`DELETE /nodes/{node}/qemu/{vmid}?purge=1&destroy-unreferenced-disks=1` and
a task wait; then delete the seed ISO
`DELETE /nodes/{node}/storage/{storage}/content/{storage}:iso/<name>-seed.iso`
tolerating absence; a locked VM is Transient. Called again mid-delete it
repeats the same tolerant steps. GC blind spot: seed ISOs without a VM
(documented; the ISO name carries the instance name).
- **Errors**, one `classify(op, err)` over `*apiError{Status, Message, Errors}`:
401, 403, 400 and 404 are Permanent (404 means the path never exists;
NotFound comes only from the list-and-miss on resources and from the 500
"does not exist" message in Get and Delete, matched against captured
fixtures); 500 "does not exist" is NotFound (Transient inside Create); 500
"VM is locked" is Transient; 500 "already exists" is handled by the
allocator, otherwise Transient; 500 "not enough free space" or "no space
left" is QuotaExceeded once a real message is captured, else `quota:
absent`; other 5xx, 595, 596, network and ctx errors are Transient;
anything not an apiError is Transient. Error text: Proxmox puts most of it
in the HTTP status reason phrase (`HTTP/1.1 500 Configuration file '...'
does not exist`) and newer releases also in the body's `message`; the
client reads both, body first.
- **Checker.** One target per configured node, in config order, concurrently,
on the same http client: `GET /version` (Details `pveVersion`),
`GET /nodes/{node}/status` (Details `kernel`),
`GET /nodes/{node}/storage/{isoStorage}/status` (Details `isoStorage`,
`isoStorageAvailGiB`), template existence from `GET /cluster/resources?type=vm`
(the `template` flag, any node; Details `templates` as `<found>/<configured>`).
Reasons: the shared ones plus `TemplateNotFound` and `StorageNotFound`.
The `status.type` comment in `api/v1alpha1/providerhealth_types.go` gains
`proxmox`; `make manifests`.
- **Timeouts** (`timeouts: per-call`): request timeout 30s
(`requestTimeoutSeconds`); task-wait caps clone 10m, upload 2m, start and
stop 2m, destroy 5m, running wait 60s; all in a `budgets` struct
overridable in tests. A cancelled ctx cancels that request only; the
connection pool is unaffected. Per-name keyed lock for Create and Delete
(`internal/provider/keyedlock`, extracted from kvm2); Get and ListByTag
lock-free.
- **Logging** (`logging: tiered`): `logf.FromContext(ctx).WithName("proxmox").WithValues("provider", name)`,
`vmid` and `name` on instance lines; V(1) outcomes and one
`proxmox api call` line per seam call from the `observed` decorator; V(2)
request params and response summaries; never the token, never cloud-init,
never the multipart body, never `description` at V(1); no V(5). The
decorator logs, the transport traces: no `proxmox.<Method>` spans, so
nothing is double-counted.
- **Config writes** use `PUT .../config` (synchronous; the VM is stopped at
that point). `resize` is skipped when the boot disk's current `size=` is
at or above the target. Boot disk: first disk key in `boot: order=...`
that is not `net*` and not the ide2 cdrom, else `bootdisk`, else the first
of `scsi0`, `virtio0`, `sata0`, `ide0`; none is Permanent.
- **A UPID names its node** (`UPID:<node>:...`); task polling uses the node
parsed from the UPID.
- **Two VMs with our name** (Proxmox does not enforce unique names) make
Create and Get return Permanent naming both VMIDs; an admin resolves it.
- **Seam.** `api` interface with one method per endpoint (table below),
`httpAPI` production adapter, `fakeAPI` in `fake_test.go` keeping VMs, tasks
and ISOs in memory with lock simulation, task polls, per-method errors and
block channels, and a call log.
- **Integration test** `proxmox_integration_test.go` behind
`//go:build integration`, `PROXMOX_TEST_URL`, `PROXMOX_TEST_TOKEN_FILE`,
`PROXMOX_TEST_NODE`, `PROXMOX_TEST_TEMPLATE_VMID`, `PROXMOX_TEST_ISO_STORAGE`,
optional `_CA_FILE`, `_INSECURE`, `_BRIDGE`, `_STORAGE`, `_VMID_MIN`,
`_VMID_MAX`, `_TIMEOUT`, `_KEEP`, `_LOG_LEVEL`; `make test-integration-proxmox`.
- **Deployment.** Secret `proxmox-token` mounted optional at
`/etc/proxy-operator/proxmox/token`, `make proxmox-token-secret`, commented
entries in `config/manager/providers_config.yaml` and
`config/samples/providers-config.yaml`, `config/samples/proxy_proxmox.yaml`,
`docs/proxmox-provider.md`, `providers-existing.md` column and section,
`architecture.md` rows, README row, `testing.md` section, CHANGELOG after
confirmation. Branch `feat/proxmox-provider`, MR via the Gitea API (no `tea`).
## Backend facts Phase 0 must verify on 192.168.2.8 (numbered for citation)
- F1. Where error text lives: status reason phrase vs. JSON `message`; the
`errors` map shape on a 400.
- F2. Exact 500 message for a missing VM (`GET /nodes/{node}/qemu/999999/config`).
- F3. Exact 500 message for a locked VM (`POST status/start` on a VM with
`lock` set), and whether `PUT config` with `lock=backup` sets the lock and
`delete=lock` clears it through the API, or `qm unlock` is needed.
- F4. Exact 500 message when `newid` is in use on clone.
- F5. Exact 500 message for the agent not running, and for the agent call
on a stopped VM.
- F6. Exact message for no space, if it can be produced cheaply; otherwise
`quota: absent`.
- F7. Whether `PUT .../config` is refused while `lock=clone`, and with which
message.
- F8. `/cluster/resources?type=vm` fields present: `vmid`, `name`, `node`,
`status` values (`running`, `stopped`, `unknown`), `type` (`qemu`, `lxc`),
`lock`, `tags` (separator), `template`; whether `lock` appears there during
a clone.
- F9. Tag rules on 9.2: confirm a hyphen is rejected and `proxy_operator`
accepted, plus case folding, length, and the `tags` separator on
`PUT config` and in `/cluster/resources`. The underscore form stays either
way; this fact only decides what the guide tells users about their own
tags.
- F10. `description` on `POST .../clone`: accepted, survives verbatim
(newlines), max length; whether `name` accepts the 22-character
`proxy-<base32>` form.
- F11. `meta` in the cloned config: is `ctime` the clone time (evidence only;
`written-at-create` stays the profile).
- F12. Upload endpoint: field names (`content`, `filename`, file part name
`filename`), the filename rule (`.iso` suffix, no slashes), the response
(UPID), the privilege it needs, and whether re-uploading an existing name
is refused (then delete-then-upload) or overwrites. On 9.x `content`
accepts `iso`, `vztmpl` or `import` and still refuses `snippets`, which is
why the seed travels as an ISO; confirm `iso` is what an `.iso` needs.
- F13. `DELETE /nodes/{node}/storage/{storage}/content/{volid}`: response
shape (UPID or null) and the message for a missing volume.
- F14. `status/current` on a stopped VM: is `qmpstatus` present; on a paused
VM: `status=running`, `qmpstatus=paused`.
- F15. `POST .../clone`: whether a failed clone destroys the partial VM by
itself, and what a VM interrupted mid-clone looks like (config present,
`lock=clone`).
- F16. Task status shape: `status` (`running`, `stopped`), `exitstatus`
(`OK` or the error string), and how long a full clone of the template
takes.
- F17. Privileges: which of the prepared list the token really needs, in
particular the exact ACL path that satisfies `SDN.Use` for the `net0`
write (the 9.x docs describe it as "access SDN vnets and local network
bridges" without spelling out the path for a plain bridge), and
`VM.GuestAgent.Audit` for the agent read. Also whether the config step
needs `VM.Config.Cloudinit` even though only `ide2` is written.
- F18. HTTP/2 is not offered by pveproxy on any release (documented), so the
client disables it; confirm with `curl --http2 -I` that the connection
falls back to HTTP/1.1.
- F19. The template's disk key (`scsi0` or `virtio0`), its `boot` line, that
ide2 is free and no `cloudinit` drive exists.
- F20. `resize` on a stopped VM: request shape, the message when the target
is smaller or equal, and that it is synchronous.
- F21. Agent response shape: `result[].name`, `result[]["ip-addresses"][]`
with `ip-address`, `ip-address-type`.
- F22. The 401 body and reason phrase on 9.2. PVE 9.2 unified the
authentication-failure message so a failed login no longer leaks whether
the user exists, so any 401 text remembered from 8.x is wrong. Capture it
fresh into `err_unauthorized.json`.
- F23. Where the upload endpoint stages the request body, given that PVE 9
moved the node's `/tmp` to a tmpfs capped at half of RAM and cleans `/tmp`
and `/var/tmp` periodically. A seed ISO is tens of KiB so this cannot
bite in practice, but the observation belongs in the guide next to the
size limit.
## Proxmox endpoint map for the seam
| `api` method | HTTP |
|---|---|
| `Version(ctx)` | `GET /version` |
| `NodeStatus(ctx, node)` | `GET /nodes/{node}/status` |
| `StorageStatus(ctx, node, storage)` | `GET /nodes/{node}/storage/{storage}/status` |
| `ClusterResources(ctx)` | `GET /cluster/resources?type=vm` |
| `VMConfig(ctx, node, vmid)` | `GET /nodes/{node}/qemu/{vmid}/config` |
| `SetVMConfig(ctx, node, vmid, params)` | `PUT /nodes/{node}/qemu/{vmid}/config` |
| `Clone(ctx, node, vmid, cloneParams)` | `POST /nodes/{node}/qemu/{vmid}/clone` |
| `Resize(ctx, node, vmid, disk, size)` | `PUT /nodes/{node}/qemu/{vmid}/resize` |
| `Start(ctx, node, vmid)` | `POST /nodes/{node}/qemu/{vmid}/status/start` |
| `Stop(ctx, node, vmid)` | `POST /nodes/{node}/qemu/{vmid}/status/stop` |
| `VMStatus(ctx, node, vmid)` | `GET /nodes/{node}/qemu/{vmid}/status/current` |
| `Destroy(ctx, node, vmid)` | `DELETE /nodes/{node}/qemu/{vmid}?purge=1&destroy-unreferenced-disks=1` |
| `TaskStatus(ctx, node, upid)` | `GET /nodes/{node}/tasks/{upid}/status` |
| `AgentNetworkInterfaces(ctx, node, vmid)` | `GET /nodes/{node}/qemu/{vmid}/agent/network-get-interfaces` |
| `UploadISO(ctx, node, storage, filename, r, size)` | `POST /nodes/{node}/storage/{storage}/upload` (multipart) |
| `DeleteVolume(ctx, node, storage, volid)` | `DELETE /nodes/{node}/storage/{storage}/content/{volid}` |
## Package layout (`internal/provider/proxmox/`)
`proxmox.go` (`Provider`, `New`, `newProvider`, `withDefaults`, `logger`,
`node`, `wrapErr`, `lockName`, `providerOption`, `budgets`), `api.go` (the
`api` interface and DTOs `vmResource`, `vmConfig`, `vmStatus`, `taskStatus`,
`agentInterface`, `storageStatus`, `nodeStatus`, `versionInfo`,
`cloneParams`), `client.go` (`httpAPI`, `newHTTPAPI`, `do`, `readToken`,
`decodeConfig`), `errors.go` (`apiError`, `classify`, `permanentf`,
`transientf`, `isNotExist`, `isLocked`, `isAlreadyExists`,
`isAgentNotRunning`, `isNoSpace`), `id.go` (`formatProviderID`,
`parseProviderID`, `vmidFor`), `mark.go` (`instanceMark`,
`formatDescription`, `parseDescription`), `state.go` (`mapState`,
`firstIPv4`), `observe.go` (`observed`, `observe`), `task.go` (`parseUPID`,
`waitTask`, `waitUnlocked`, `waitRunning`), `seed.go` (`seedISOName`,
`seedVolID`, `uploadSeed`), `create.go`, `get.go`, `delete.go`, `list.go`,
`check.go`, plus `fake_test.go`, `provider_test.go`,
`proxmox_integration_test.go`, `testdata/*.json`. Shared:
`internal/provider/config_proxmox.go`, `internal/provider/nocloud/`,
`internal/provider/keyedlock/`.
Reused as-is: `provider.NameFromUID`, `provider.Wrap`, the `Err*` sentinels,
`provider.Label*` constants, `provider.Reason*` constants,
`tracing.RestConfigWrapper`, `tracing.WithTracerProvider`, kvm2's test
idioms (`fixture`, `assertProviderError`, `logCapture`).
## Steps (one commit per phase, on `feat/proxmox-provider` off `main`)
- **Phase 0 (orchestrator with the user): plan file, branch, host recon,
fixtures.** Read first: contract §5, §4.9, §7;
`docs/plans-executions/2026-09-16-1333-kvm2-provider.md` Step 0 entry.
Do, in this order: `git switch main && git pull --ff-only`; copy this plan
to `docs/plans/2026-09-18-1502-proxmox-provider.md` with the real timestamp
(`date "+%Y-%m-%d-%H%M"`), commit it alone on `main`
(`docs: plan the proxmox provider`); create `feat/proxmox-provider`. Ask
the user for the hand-over facts listed above. Then, with the user's
go-ahead (each `curl` is a permission prompt; read-only ones first, one at
a time), capture F1 to F21 with
`curl -s --cacert "$CA" -D /tmp/h.txt -o /tmp/b.json -w '%{http_code}\n' -H "Authorization: PVEAPIToken=$(cat "$TOKEN_FILE")" https://192.168.2.8:8006/api2/json/<path>`,
storing each response as `internal/provider/proxmox/testdata/<name>.json`
in the shape `{"status": <code>, "reason": "<status line after the
code>", "body": "<raw body>"}`: `version.json`, `node_status.json`,
`storage_status.json`, `cluster_resources.json`,
`vm_config_template.json`, `vm_config_clone.json`,
`status_current_running.json`, `status_current_stopped.json`,
`agent_network_get_interfaces.json`, `task_status_running.json`,
`task_status_stopped_ok.json`, `task_status_stopped_error.json`,
`upload_response.json`, `content_delete_response.json`,
`err_vm_missing.json`, `err_locked.json`, `err_newid_in_use.json`,
`err_agent_not_running.json`, `err_agent_vm_stopped.json`,
`err_unauthorized.json` (a wrong secret), `err_forbidden.json` (a path
the token lacks), `err_bad_param.json` (`newid=abc`), `err_no_space.json`
(only if F6 is cheap), `err_volume_missing.json` (F13). The write
experiments: one clone of the template into the top of the VMID range
(`newid=199999`, a multi-line `description`, a `name` of the `proxy-`
form), a config read of the clone (F10, F11, F19), a `PUT config` while
it is locked if timing allows (F7), an upload of a 1 KiB
`recon-seed.iso` (F12) and its delete (F13), a resize attempt (F20), a
start, an agent read (F5, F21), a stop, then destroy with
`purge=1&destroy-unreferenced-disks=1`; a final resources read shows
nothing remains. Redact nothing but the token. Write the answers to F1 to
F21 and the privilege outcome into the execution log's Phase 0 entry, one
line per fact; Phase 1 cites them as Evidence. Create the execution log
`docs/plans-executions/2026-09-18-1502-proxmox-provider.md` with the checklist (Phase
0 to Phase 10) and this entry.
Verify: `ls internal/provider/proxmox/testdata | wc -l` is at least 22;
`grep -rl 'PVEAPIToken' internal/provider/proxmox/testdata` is empty;
every fixture parses as JSON; the host shows no VM in the range and no
`recon-seed.iso`.
Done when: the plan commit is on `main`, the branch exists, the fixtures
and the execution log with F1 to F21 are committed. Commit: `proxmox:
captured API fixtures and host facts from 192.168.2.8`.
- **Phase 1 (orchestrator): the spec.** Read first: contract §2, §4, §5,
§6, §11, §12; `docs/provider-spec-template.md` end to end;
`docs/kvm2-provider-spec.md` for the level of detail; this plan's
decisions; the Phase 0 log entry. Create `docs/proxmox-provider-spec.md`
answering all 24 entries: 1 `vms/<vmid>/<name>`, strict parsing, the
node-name and storage-id regexps; 2 `allocated-id`, `vmidFor(uid, rng,
attempt)`, the two-VMs-with-our-name rule, the adoption guard; 3 the
placement table (Region ignored); 4 the template catalogue and the "add a
key, never edit" warning; 5 the seed ISO via `nocloud` and the ide2
prerequisite; 6 `machineTypes` with `diskGiB`; 7 `provisioning-window`,
`discovered` via the guest agent, the IPv4 filter, `Port` unused; 8
`written-metadata` in `description` plus the tag, `written-at-create`, F9
to F11 as Evidence; 9 the state table, `create-blocks`; 10 the ordered
Create steps with what a retry finds at each, `task-handle` plus
`backend-lock`, the budgets; 11 the Delete order, the ISO blind spot,
node-local ISO storage after a migration; 12 `fail-whole`, the candidate
rule, one config read per candidate; 13 the classify table with the F1 to
F6 messages, `not-found: error-code` plus list-and-miss on resources
(allowed because offline nodes' guests appear as `unknown`, not absent);
14 `mounted-files` (Secret `proxmox-token`, key `token`, path
`/etc/proxy-operator/proxmox/token`), `ca-file` or
`insecureSkipTLSVerify`, the privilege list from F17, rotation free; 15
the config table (Phase 2's struct); 16 the Checker; 17 `thin-client`,
`transport-wrapper`, the seam's method list; 18 the budgets table; 19 the
keyed lock per name; 20 the log keys; 21 the §11 list with the test names
from Phases 2 to 8 (copy from this plan); 22 the deployment list; 23 the
docs list; 24 the accepted limits (blocking Create up to the clone budget,
ISO blind spot, VMID range shared with humans, node-local ISO storage, no
adoption of foreign VMs, two-installs GC caveat, `unknown` status while a
node is down).
Verify: `grep -c '^\*\*Answer:\*\*$' docs/proxmox-provider-spec.md` is 0;
every axis code used appears in contract §4.20; Evidence lines cite F
numbers; `grep -c '—' docs/proxmox-provider-spec.md` is 0.
Done when: the spec is committed with all 24 entries and the Phase 1 box
ticked. Commit: `docs: proxmox provider spec`.
- **Phase 2 (implementer): config block, registration, package skeleton.**
Read first: spec entries 1, 2, 3, 6, 8, 9, 13, 15; contract §10, §5, §6;
`internal/provider/config.go`, `config_kvm.go`, `config_test.go`,
`config_kvm_test.go`; `internal/provider/kvm2/{kvm2.go,id.go,metadata.go,state.go,errors.go}`
and their tests as the pattern; `cmd/main.go`; `testdata/err_*.json`.
Create `internal/provider/config_proxmox.go`: `ProxmoxConfig{URL,
TokenFile, CAFile string; InsecureSkipTLSVerify bool; Nodes []string;
ISOStorage, Storage, Bridge string; VMIDRange *ProxmoxVMIDRange; Templates
map[string]ProxmoxTemplate; MachineTypes map[string]ProxmoxMachineType;
RequestTimeoutSeconds int; VendorData string; InstallGuestAgent *bool}`,
`ProxmoxVMIDRange{Min, Max int}`, `ProxmoxTemplate{VMID int}`,
`ProxmoxMachineType{Cores int32; MemoryMiB int64; DiskGiB int64}`,
`validateProxmox(prefix string, c *ProxmoxConfig) error` (URL http(s)
with a host; `tokenFile` non-empty; at least one node, unique, matching
`proxmoxNodeName = ^[a-z0-9][a-z0-9-]*$`; `isoStorage` and `storage`
matching `proxmoxStorageID = ^[A-Za-z][A-Za-z0-9_.-]*$`; `vmidRange` min
at least 100, max at most 999999999, min below max; at least one template
with `vmid` at least 100; at least one machine type with cores and
memoryMiB above zero and diskGiB not negative; `requestTimeoutSeconds`
not negative). Edit `config.go`: `Proxmox *ProxmoxConfig` with tag
`proxmox,omitempty`, `setBlocks`, the allow-list, the dispatch. Edit
`cmd/main.go`: `"proxmox": proxmox.New`. Create in
`internal/provider/proxmox/`: `proxmox.go` (`Provider{name, cfg, api,
now, budgets, nameLocks, tracing}`; until Phase 3 a local `keyedLocks`
copy in `locks.go` that Phase 3 deletes; `New` never dials: builds the
client, `os.Stat`s the token file and warns, warns on
`insecureSkipTLSVerify`; `withDefaults`: `isoStorage` `local`, `vmidRange`
100000 to 199999, `requestTimeoutSeconds` 30, `installGuestAgent` true;
`node(zone)` mirrors kvm2's `host`; the four methods stubbed with
`transientf("not implemented")`), `api.go` (the interface and DTOs;
`vmConfig` is `map[string]string` with scalars rendered as strings by
`decodeConfig`), `id.go`, `mark.go` (keys from `provider.LabelManaged`,
`provider.LabelManagedYes`, `provider.LabelUID` and
`proxy-operator-created-at`; `parseDescription` tolerates extra lines
and CRLF), `state.go`, `errors.go` (`apiError{Status int; Message
string; Errors map[string]string}` whose `Error()` never includes headers
or URLs; `classify` per spec entry 13; matcher helpers keyed on the exact
substrings captured in F2 to F6). Tests: `config_proxmox_test.go`
`TestValidateProxmox_table` (missing url, non-http scheme, missing
tokenFile, no nodes, duplicate node, node with a slash, bad isoStorage
id, vmidRange min below 100, min above max, no templates, template vmid
below 100, no machineTypes, cores zero, negative diskGiB, negative
timeout, valid full, valid minimal) with a `validProxmoxYAML` const;
`config_test.go` rows `proxmox type with kvm block`, `kvm type with
proxmox block`, `proxmox minimal entry loads`. `id_test.go`:
`TestFormatProviderID_roundTrip`, `TestParseProviderID_malformed`,
`TestVMIDFor_deterministicAndInRange`, `TestVMIDFor_probesForwardAndWraps`.
`mark_test.go`: `TestFormatDescription_keys`, `TestParseDescription_table`.
`state_test.go`: `TestMapState_table`, `TestFirstIPv4_filters`.
`errors_test.go`: `TestClassify_table` (every class, every matched
message loaded from `testdata/err_*.json` through `loadFixture(t, name)
*apiError`, `op == "create"` turning NotFound into Transient,
`context.Canceled`, a `*net.OpError`, `errors.New("x")`,
already-classified errors keep their class), `TestAPIError_matchers`,
`TestPermanentfAndTransientf`. `proxmox_test.go`:
`TestNew_doesNotDialAndWarnsOnMissingToken`,
`TestNew_insecureSkipTLSVerifyWarns`, `TestNew_nilBlockDoesNotPanic`,
`TestWithDefaults_table`, `TestNode_table`,
`TestWrapErr_carriesProviderNameOpAndID`.
Verify: `grep -c '"proxmox"' internal/provider/config.go` is 3;
`go build ./... && go vet ./... && go test -race -count=1 ./internal/provider/... ./cmd/... && go test -short ./...`;
the fixture loader fails loudly on a missing file.
Done when: `go test -race ./internal/provider/proxmox/` passes with the
stubs in place and the constructor is registered. Commit: `proxmox:
config block, IDs, marks, state mapping, classify, registration`.
- **Phase 3 (implementer): extract `nocloud` and `keyedlock` from kvm2.**
Read first: `internal/provider/kvm2/{seed.go,seed_test.go,kvm2.go,kvm2_test.go,create.go}`;
contract §1.4, §2.1; spec entries 5 and 19;
`grep -n 'seed.go\|keyedLocks\|buildSeedISO' docs/*.md`. Create
`internal/provider/nocloud/seed.go` exporting `BuildSeedISO(name,
userData, vendorData string, w io.Writer) error` (plain errors, an
`ErrEmptyName` sentinel instead of kvm2's `permanentf`),
`BuildVendorData(configured string, installAgent bool) (string, error)`,
`GuestAgentScript`, `SeedLabel`; move the seed tests to
`nocloud/seed_test.go` under their existing names. Create
`internal/provider/keyedlock/keyedlock.go` exporting `Locks`, `New()
*Locks`, `(*Locks).Acquire(ctx, key) (release func(), err error)`; move
the `TestKeyedLocks_*` tests. Edit kvm2: delete `seed.go`, `seed_test.go`,
the `keyedLocks` type and its tests; `create.go` calls
`nocloud.BuildVendorData` and `nocloud.BuildSeedISO` and wraps their
errors with `permanentf`; `kvm2.go` uses `*keyedlock.Locks`. Edit
`internal/provider/proxmox/proxmox.go` to use `keyedlock.Locks` and delete
`locks.go`. Fix the doc mentions so they name the new packages.
Verify: `go test -race -count=1 ./internal/provider/...`; kvm2's test
count drops by exactly the moved tests and the two new packages carry
them; `grep -rn 'iso9660' internal/provider/kvm2/` is empty;
`go vet -tags=integration ./...` still compiles.
Done when: kvm2 and proxmox import both packages, all tests pass, the docs
name the new packages. Commit: `provider: extract nocloud seed builder
and keyedlock from kvm2`.
- **Phase 4 (implementer): seam, HTTP client, decorator, fake.** Read first:
spec entries 14, 17, 18, 20; contract §4.15, §7, §9;
`internal/tracing/transport.go`; `internal/provider/kvm2/{observe.go,observe_test.go,fake_test.go}`;
`internal/provider/proxmox/{api.go,errors.go}`; `testdata/*.json`. Create
`client.go`: `httpAPI{base *url.URL; http *http.Client; tokenFile
string}`; `newHTTPAPI(cfg provider.ProxmoxConfig, opts ...tracing.Option)
(*httpAPI, error)` builds a transport with `DialContext` timeout 10s,
`TLSHandshakeTimeout` 10s, `ResponseHeaderTimeout` from the request
timeout, `IdleConnTimeout` 90s, `MaxIdleConnsPerHost` 4,
`ForceAttemptHTTP2: false`, an empty `TLSNextProto` map (F18), `RootCAs`
from `caFile` or `InsecureSkipVerify` from the flag, wrapped with
`tracing.RestConfigWrapper(opts...)(transport)`; `http.Client.Timeout` is
the request timeout; `do(ctx, method, path string, query, form
url.Values, out any) error` reads the token file on every call (missing
file `transientf`, a file without `!` and `=` `permanentf`), sets the
header, decodes `{"data": ...}` into `out`, turns any non-2xx into
`*apiError` per F1 with the error body read capped at 64 KiB;
`UploadISO` streams a multipart body through `io.Pipe` with a goroutine
writing `content=iso`, `filename` and the file part. Create `observe.go`:
`observed{inner api; provider string}` writing one V(1) `proxmox api
call` line per seam call with `call`, `node`, `vmid`, `storage`, `upid`,
`duration`, `httpStatus` and `error` on failure, V(2) adding request
params and a response summary; `observe(inner api, providerName string)
api`; `New` wraps `httpAPI` in it. Create `fake_test.go`: `fakeAPI{mu; vms
map[int]*fakeVM; tasks map[string]*fakeTask; isos
map[string]map[string][]byte; hooks; block; calls; log; taskPolls int;
cloneFailsWith, startFailsWith string; agentNotRunning, cloneLeavesLock
bool}`, `fakeVM{node, name, status, qmpstatus, lock string; template
bool; config map[string]string; ifaces []agentInterface}`,
`fakeTask{node string; polls int; exit string; onDone func()}`; `Clone`
creates the VM with `lock=clone` and a task that clears the lock or, per
`cloneFailsWith`, removes the VM (F15); `Start` and `Stop` flip status
through a task; `Destroy` removes the VM through a task; `UploadISO`
records bytes; `DeleteVolume` errors with the F13 message when absent;
`enter(ctx, method)` for blocks and hooks like kvm2; helpers `addVM`,
`addTemplate`, `failWith`, `blockOn`, `callCount`, `callLog`,
`fakeErr(status int, msg string) *apiError`. Tests: `client_test.go` over
`httptest.NewTLSServer`: `TestClient_sendsTokenHeaderFromFileOnEveryRequest`
(rotate the file between two calls), `TestClient_missingTokenFileIsTransient`,
`TestClient_unparsableTokenFileIsPermanent`,
`TestClient_apiErrorFromStatusReasonAndBody` (serve each
`testdata/err_*.json` verbatim), `TestClient_decodesDataEnvelope`,
`TestClient_http1Only`, `TestClient_requestTimeoutIsTransient`,
`TestClient_cancelledContextReturnsPromptly`,
`TestClient_childSpanUnderParent`, `TestClient_noSpanWithoutParent`,
`TestClient_tokenNeverInErrorsOrLogs` (sentinel secret echoed by the
server in a 401 body), `TestClient_caFileTrustsPrivateCA`,
`TestClient_insecureSkipsVerification`, `TestClient_uploadStreamsMultipart`
(the server parses the multipart; the client never `io.ReadAll`s the
reader), `TestClient_destroyQueryFlags`. `observe_test.go`:
`TestObserve_everySeamMethodLogsOneLine` (reflection over `api`),
`TestObserve_detailOnlyAtV2`, `TestObserve_uploadBodyNeverLogged`,
`TestObserve_errorFieldsOnFailure`, `TestObserve_v0Silent`.
Verify: `go test -race -count=2 ./internal/provider/proxmox/`;
`grep -n 'ReadAll' internal/provider/proxmox/client.go` shows only the
bounded error-body read; `grep -n 'context.Background()' internal/provider/proxmox/*.go`
is empty outside tests.
Done when: the client passes its tests against the fixtures and the fake
compiles against the full `api` interface. Commit: `proxmox: thin HTTP
client, seam, log decorator and in-memory fake`.
- **Phase 5 (implementer): Get, ListByTag, task helpers.** Read first: spec
entries 7, 8, 9, 12, 13; contract §2.2, §2.4, §4.7, §4.12;
`internal/provider/kvm2/{get.go,get_test.go,list.go,list_test.go}`; the
resources, config, status, agent and task fixtures. Create `task.go`:
`parseUPID(upid) (node string, err error)` (Permanent when malformed),
`waitTask(ctx, log, api, upid, budget) error` polling `TaskStatus` every
`budgets.taskPoll` (2s; tests shrink it) until `status == "stopped"`,
`exitstatus != "OK"` giving `transientf` with the message, the budget
giving `transientf`, ctx giving Transient; `waitUnlocked(ctx, api, node,
vmid, budget)` polling `VMConfig` until no `lock`. Create `get.go`:
`resolve(ctx, api, vmid) (vmResource, bool, error)` (one resources call,
`Type == "qemu"` and the vmid), the name check, `VMConfig` for the mark,
`lock` Transient, `unknown` Transient, `running` reads the agent
(`isAgentNotRunning` gives no IP, other errors Transient), `mapState`.
Create `list.go` per spec entry 12; `Instance` carries ID, UID, CreatedAt
only. Replace the two stubs. Tests: `task_test.go` `TestParseUPID_table`,
`TestWaitTask_table` (ok, error exitstatus, budget exceeded, ctx
cancelled, poll count), `TestWaitUnlocked_table`. `get_test.go`:
`TestGet_missingVMIsNotFound`, `TestGet_nameMismatchIsNotFound`,
`TestGet_unmarkedIsNotFound`, `TestGet_runningWithoutIPIsProvisioning`,
`TestGet_runningWithIPIsRunning`, `TestGet_agentNotRunningIsProvisioning`,
`TestGet_agentOtherErrorIsTransient`, `TestGet_ipFilterSkipsLoopbackAndLinkLocal`,
`TestGet_stoppedIsStopped`, `TestGet_pausedIsStopped`,
`TestGet_unknownStatusIsTransient`, `TestGet_lockedIsTransientNeverNotFound`,
`TestGet_unknownStateIsStopped`, `TestGet_readsUIDAndCreatedAtFromBackend`,
`TestGet_unmanagedDescriptionHasEmptyUID`, `TestGet_malformedIDIsPermanent`,
`TestGet_resourcesFailureIsTransient`, `TestGet_twoVMsWithOurNameIsPermanent`.
`list_test.go`: `TestListByTag_excludesUnmarkedVMs`,
`TestListByTag_emptyUIDStillListed`, `TestListByTag_candidatesByTagOrNamePrefix`,
`TestListByTag_skipsLXCAndTemplates`, `TestListByTag_resourcesFailureFailsWhole`,
`TestListByTag_unreadableVMSkipped`, `TestListByTag_unreadableVMWithEndedContextFails`,
`TestListByTag_unknownStatusStillAttempted`, `TestListByTag_doesNotReadStatusOrAgent`,
`TestListByTag_idsAcceptedByParse`.
Verify: `go test -race -count=2 ./internal/provider/proxmox/`;
`grep -n 'not implemented' internal/provider/proxmox/*.go` lists only
Create and Delete.
Done when: Get and ListByTag are real and every listed test passes.
Commit: `proxmox: Get, ListByTag and task polling`.
- **Phase 6 (implementer): Create.** Read first: spec entries 2, 3, 4, 5, 6,
10, 19; contract §2.1, §4.2, §4.10, §4.11, §5;
`internal/provider/kvm2/{create.go,create_test.go}`;
`internal/provider/nocloud`; F4, F7, F10, F12, F15, F16, F19, F20 in the
Phase 0 log. Create `seed.go`: `seedISOName(name)` is `<name>-seed.iso`,
`seedVolID(storage, name)` is `<storage>:iso/<name>-seed.iso`,
`uploadSeed(ctx, log, api, node, storage, req, cfg)` builds vendor-data
and the ISO into a `bytes.Buffer` (tens of KiB), deletes an existing
volume tolerating absence, uploads with the buffer's length, waits the
task with `budgets.upload`, logs `cloudInitBytes` only. Create `create.go`
implementing the ordered steps: placement validation (`node`,
`MachineTypes`, `Templates`, all Permanent naming field and value), the
per-name lock, `ClusterResources` once for both the name lookup (adoption
guard, then `waitUnlocked` if locked, then continue from the seed step if
stopped, return if running) and the template's node (absent template is
Permanent); allocation loop `for attempt := 0; attempt < maxProbes (32)`
calling `Clone` with `vmidFor(req.UID, rng, attempt)`, `isAlreadyExists`
moving to the next attempt, exhaustion Transient; `waitTask` with
`budgets.clone`; `uploadSeed`; `SetVMConfig` with `cores`, `memory`,
`agent=1`, `tags=proxy_operator`, `ide2=<volid>,media=cdrom`, and `net0`
rewritten from the cloned config with `bridge=<cfg.Bridge>` only when
configured (MAC kept); `bootDisk(cfg)` and `Resize` when `diskGiB` is
above the current `size=`; `Start` and `waitTask` with `budgets.start`;
`waitRunning` on `VMStatus` with `budgets.running`; return the ID.
`isNotExist` inside Create is Transient (classify with `op == "create"`).
Tests `create_test.go`: `TestCreate_repeatReturnsSameIDAndCreatesNothing`,
`TestCreate_resumesAfterEachPartialStep` (after clone task, after seed
upload, after config, after resize, after start, after running),
`TestCreate_returnsOnlyWhenRunning`, `TestCreate_runningWaitCapIsTransient`,
`TestCreate_rejectsUnknownPlacement`, `TestCreate_zoneOptionalWithOneNode`,
`TestCreate_regionIgnored`, `TestCreate_adoptionWithForeignUIDIsPermanent`,
`TestCreate_adoptsOnlyMarkedVMs`, `TestCreate_duplicateNameIsPermanent`,
`TestCreate_vmidCollisionProbesForward`, `TestCreate_vmidRangeExhaustedIsTransient`,
`TestCreate_waitsForCloneLockOnRetry`, `TestCreate_cloneTaskFailureIsTransient`,
`TestCreate_cloneTaskTimeoutIsTransient`, `TestCreate_markWrittenInCloneCall`,
`TestCreate_tagSetInConfigStep`, `TestCreate_seedAlwaysRebuilt`,
`TestCreate_seedUploadedBeforeConfig`, `TestCreate_configSetsCoresMemoryAgentAndSeed`,
`TestCreate_bridgeOverrideKeepsMAC`, `TestCreate_noBridgeOverrideLeavesNet0`,
`TestCreate_resizeOnlyWhenLarger`, `TestBootDisk_table`,
`TestCreate_startTaskFailureIsTransient`, `TestCreate_templateMissingIsPermanent`,
`TestCreate_notExistInsideCreateIsTransientNotNotFound`,
`TestCreate_cloudInitSentinelNeverLoggedOrInErrors` (V(2) capture, every
error path), `TestCreate_concurrentSameNameSerialised`,
`TestCreate_cancelledMidWaitReturnsPromptly`.
Verify: `grep -n 'CloudInit' internal/provider/proxmox/create.go internal/provider/proxmox/seed.go`
shows only `len(req.CloudInit)` and the pass-through into
`nocloud.BuildSeedISO`; `go test -race -count=2 ./internal/provider/proxmox/`;
`grep -n 'not implemented'` lists only Delete.
Done when: Create is real and every listed test passes. Commit:
`proxmox: Create by cloning a template, seed upload, config, start`.
- **Phase 7 (implementer): Delete, Checker, CRD comment.** Read first: spec
entries 11, 16; contract §2.3, §4.3, §4.13, §8;
`internal/provider/kvm2/{delete.go,delete_test.go,check.go,check_test.go}`;
`internal/provider/check.go`; `api/v1alpha1/providerhealth_types.go`; F13,
F17 in the Phase 0 log. Create `delete.go`: parse the ID (Permanent), the
per-name lock, `resolve`; absent is a logged no-op; name mismatch or
unmarked is a logged no-op; `lock` Transient; `unknown` Transient;
`running` gets `Stop` and `waitTask` (`budgets.stop`); `Destroy` and
`waitTask` (`budgets.destroy`), `isNotExist` from `Destroy` being
success; `DeleteVolume` of the seed on the VM's node tolerating the F13
absent message; nil. Create `check.go`: `var _ provider.Checker =
(*Provider)(nil)`; `reasonTemplateNotFound = "TemplateNotFound"`,
`reasonStorageNotFound = "StorageNotFound"`; `Check` fans out one
goroutine per node in config order (`wg.Go`), each `checkNode` running
`Version`, `NodeStatus`, `StorageStatus` (a 500 "does not exist" is
`StorageNotFound`; a storage whose `content` lacks `iso` is
`StorageNotFound` with a message saying so), `ClusterResources` for the
templates; `checkReason(err)`: ctx ended `Timeout`, 401 `Unauthorized`,
403 `Forbidden`, other `*apiError` `APIError`, network `Unreachable`;
Details `pveVersion`, `kernel`, `isoStorage`, `isoStorageAvailGiB`,
`templates`; `Name` is the node, `Address` the base URL. Edit
`api/v1alpha1/providerhealth_types.go` (`kubernetes, gcp, kvm, kvm2 or
proxmox`) and run `make manifests`. Tests `delete_test.go`:
`TestDelete_malformedIDIsPermanent`, `TestDelete_twiceSucceeds`,
`TestDelete_absentIsNoOp`, `TestDelete_foreignNameIsNoOp`,
`TestDelete_unmarkedIsNoOp`, `TestDelete_runningVMStoppedFirst`,
`TestDelete_stoppedVMSkipsStop`, `TestDelete_lockedIsTransient`,
`TestDelete_unknownStatusIsTransient`, `TestDelete_removesSeedISO`,
`TestDelete_toleratesMissingISO`, `TestDelete_destroyTaskFailureIsTransient`,
`TestDelete_usesPurgeAndDestroyUnreferencedDisks`,
`TestDelete_concurrentSameNameSafe`, `TestDelete_noOpLinesAreSilentAtV0`;
in `get_test.go`: `TestGet_afterDeleteReturnsNotFound`,
`TestGet_duringDeleteStillReturnsInstance` (block the fake's `Destroy`).
`check_test.go`: `TestCheck_oneEntryPerNodeInConfigOrder`,
`TestCheck_nodesCheckedConcurrently`, `TestCheck_expiredContextReturnsPromptly`,
`TestCheck_reasonsTable`, `TestCheck_detailsKeys`, `TestCheck_logsVerdictPerNode`,
`TestCheckReason_table`.
Verify: `go test -race -count=2 ./internal/provider/proxmox/ ./internal/providerhealth/`;
`git diff --stat config/` shows only the CRD description line;
`grep -rn 'not implemented' internal/provider/proxmox/` is empty.
Done when: all four contract methods and `Check` are real. Commit:
`proxmox: Delete with seed cleanup, health Checker, CRD type comment`.
- **Phase 8 (implementer): contract sweep, observability sweep, integration
test, Makefile, testing.md.** Read first: contract §11 and §9; spec
entries 17, 18, 20, 21; `internal/provider/kvm2/{provider_test.go,kvm2_integration_test.go}`;
`internal/provider/tracing_test.go`; `Makefile` integration targets;
`docs/testing.md` kvm2 rows and section. Create `provider_test.go`: the
shared `fixture` (fake, log capture, budgets shrunk), `assertProviderError`,
`TestProvider_cancelledContextReturnsPromptly` (each method blocked on
its first seam call), `TestProvider_everyErrorIsWrappedWithConfiguredName`,
`TestLogging_v0SilentV1OutcomeV2Detail` (V(1) lines `proxmox vm ensured
running`, `proxmox vm fetched`, `proxmox vm deleted`, `proxmox vms
listed`, `proxmox node checked`, every line with `provider`, instance
lines with `vmid` and `name`; V(2) has `cloudInitBytes` and never the
text), `TestTracing_httpSpansUnderOperationSpan` (a real `httpAPI`
against an `httptest` server playing the fixtures for a Get, wrapped in
`provider.WithTracing`, with a span recorder: every HTTP span is a child
of `provider.get` and no attribute carries the token). Create
`proxmox_integration_test.go` (`//go:build integration`):
`loadIntegrationEnv` with the `PROXMOX_TEST_*` variables, `providerConfig()`,
the `integrationLog` pattern from kvm2; `TestProxmox_lifecycle` asserting
in order: `New` returns within 100 ms without a request; `Check` Ready
with `pveVersion`, `isoStorageAvailGiB` and `templates`; `Create` returns
`vms/<vmid>/<name>` with the vmid in range; a second `Create` returns the
same ID and the resources count is unchanged; `Get` reaches Running with
an IPv4 within the budget, UID and CreatedAt as written; `ListByTag`
contains it; a TCP connect to the guest's port is logged only; `Delete`
nil; `Get` NotFound; `Delete` again nil; the ISO storage has no
`<name>-seed.iso` and resources has no VM with the name; an HTTP span is
a child of `provider.create` and no attribute carries the cloud-init
text; `PROXMOX_TEST_KEEP` skips the teardown and prints the `qm destroy`
and ISO delete commands. `TestProxmoxCheck_realHost` creates nothing. Edit
`Makefile`: `test-integration-proxmox` (`go test -race -tags=integration
-count=1 -v -timeout 30m -run TestProxmox ./internal/provider/proxmox/`).
Edit `docs/testing.md`: rows for the package in the suites table and a
"The proxmox integration test" section. Update spec entry 21's tick list
with the final test names.
Verify: `go test -race -count=2 ./internal/provider/proxmox/`;
`go vet -tags=integration ./... && go vet -tags=e2e ./...`;
`make -n test-integration-proxmox`; every §11 item has a test name in
spec entry 21 that exists.
Done when: the package covers every §11 item by name, the integration
test compiles, the target exists. Commit: `proxmox: contract and
observability tests, integration test, testing docs`.
- **Phase 9 (implementer): deployment and documentation.** Read first:
contract §12 items 11 to 13; spec entries 22, 23, 24;
`config/manager/manager.yaml`, `config/manager/providers_config.yaml`,
`config/samples/{providers-config.yaml,proxy_kvm2.yaml,kustomization.yaml}`;
`Makefile` `kvm-ssh-secret`; `docs/kvm2-provider.md` headings;
`docs/providers-existing.md`; `docs/architecture.md` lines mentioning
kvm2; `README.md`; `docs/providers.md`; `CLAUDE.md` Gotchas. Edit
`config/manager/manager.yaml`: an optional Secret mount `proxmox-token` at
`/etc/proxy-operator/proxmox` (key `token`). Edit `Makefile`:
`PROXMOX_TOKEN_FILE ?= $(HOME)/.config/proxmox-token`,
`proxmox-token-secret` creating `proxmox-token` from `--from-file=token=`.
Edit both providers-config files: a commented `proxmox-home` entry (`url:
https://192.168.2.8:8006`, `tokenFile`, `caFile` or
`insecureSkipTLSVerify`, `nodes`, `isoStorage`, `storage`, `bridge`,
`vmidRange`, `templates`, `machineTypes`). Create
`config/samples/proxy_proxmox.yaml` (provider `proxmox-home`, zone the
node, machine type `small`, image the template key, the Squid cloud-init
from `proxy_kvm2.yaml`) and list it in `config/samples/kustomization.yaml`.
Create `docs/proxmox-provider.md` with sections: Proxmox prerequisites
(the template commands from this plan, no cloudinit drive on ide2, the
guest agent or outbound access, the bridge, the privilege list from F17,
reserving the VMID range), Cluster-side setup (the Secret, the config
entry), Config reference (spec entry 15's table), Placement (Region
ignored, catalogue-edit warning), What lands on Proxmox (VMID scheme,
description mark, tag, seed ISO, config keys, `qm unlock` for a stuck
clone), Logging, Tracing, Timeouts, Health check, Error classes, Known
gaps (spec entry 24), Testing. Edit `docs/providers-existing.md`: a fifth
column in the three tables and a proxmox section mirroring kvm2's. Edit
`docs/architecture.md`: components row, a proxmox block in §5, the span
note in §10, a §11 row (and the "All four providers" count), Decisions
bullets. Edit `README.md`, `docs/providers.md`, `CLAUDE.md` Gotchas
(`make test-integration-proxmox`).
Verify: `bin/kustomize build config/default >/dev/null && bin/kustomize build config/samples >/dev/null`;
a test `TestLoadConfig_proxmoxSampleEntry` loads the uncommented block
from `config/samples/providers-config.yaml` through `provider.LoadConfig`;
a link sweep over the edited docs finds no missing target;
`grep -c '—' docs/proxmox-provider.md` is 0.
Done when: manifests build, the sample loads, docs are linked and
link-clean. Commit: `proxmox: deployment files, sample, user guide and
doc rows`.
- **Phase 10 (orchestrator with the user): review, real-host validation,
changelog, MR.** Read first: `docs/plans-executions/2026-09-17-1515-kvm2-review-fixes.md`;
`.claude/agents/go-operator-reviewer.md`; the latest report under
`docs/reviews/` for the shape; the spec; this plan's Risks. Do: run
`go-operator-reviewer` on the branch with the spec and the contract as
inputs, then `/code-review high`, `make lint` (if the local toolchain
panics in staticcheck as it did during kvm2, record it and run
`bin/golangci-lint run ./internal/provider/proxmox/...` with staticcheck
disabled), `make test`. Fix accepted findings with regression tests in
one commit; record rejected ones under `docs/reviews/<date>-proxmox-provider-review/`.
Hand the user the host checklist (template ready, no leftover VMs in the
range, ISO storage has space, the token's privileges per F17) and the
`PROXMOX_TEST_*` export lines; with the go-ahead run
`make test-integration-proxmox` and record the clone duration and time to
IP. Optionally the in-cluster pass on kind: build and load the image,
`make proxmox-token-secret`, a scratch ConfigMap with the `proxmox-home`
entry, apply `proxy_proxmox.yaml`, wait for `Ready`, `kubectl get
providerhealth proxmox-home -o yaml`, a `curl -x` through the proxy,
delete and confirm the VM and ISO are gone; scratch overrides are not
committed. After the user confirms: CHANGELOG entry
(`date "+%Y-%m-%d %H:%M %Z"`), push with `-u` using the token helper from
memory (`~/.config/gitea-token`), open the MR with the Gitea API
(`POST /api/v1/repos/kacerr/egress-proxies-operator/pulls` with `head:
feat/proxmox-provider`, `base: main`, the body ending with the
attribution line; `pr.json` in the scratchpad, never committed). Never
merge.
Verify: the integration run's ordered assertions pass on 192.168.2.8;
`make test` green; the review report lists every finding as accepted or
rejected.
Done when: the MR URL is printed and the execution log's last box is
ticked. Commit: `proxmox: review fixes, real-host validation record,
changelog`.
## Verification
- Per phase: the `Verify:` clause, plus `go build ./... && go test -short ./...`
before every commit and `git diff --stat` reviewed for files outside the
phase's list.
- Contract mapping: after Phase 8 every rule of contract §2 and §8 has a
named test in the package, and spec entry 21 maps the §11 list 1:1 to
existing test names.
- Fixture discipline: every message `classify` or a matcher helper compares
against is loaded from `testdata/` in `TestClassify_table` or
`TestAPIError_matchers`; a string literal in `errors.go` without a fixture
is a review finding.
- Secrets: `TestClient_tokenNeverInErrorsOrLogs` and
`TestCreate_cloudInitSentinelNeverLoggedOrInErrors` run at V(2), and the
integration test asserts no span attribute carries the cloud-init text.
- End to end: the integration run (Phase 8's assertions) on 192.168.2.8 and
the optional kind pass in Phase 10.
## Risks
- **Error text location and exact wording (F1 to F6)** decide the whole
NotFound path; if messages vary by storage type or PVE minor version, the
matchers get one fixture per variant and the guide names the tested
version. This is not hypothetical: 9.2 rewrote the authentication-failure
message (F22), so fixtures captured from an 8.x host would already be
wrong. Every matched string comes from 192.168.2.8 at 9.2.20, and the
guide says so.
- **Clone cleanup (F15).** A failed or interrupted clone may leave a VM with
`lock=clone` that never clears; Create then waits out `budgets.clone` and
returns Transient every time. The guide documents `qm unlock`; the
reconciler keeps retrying rather than latching.
- **`SDN.Use` (F17).** PVE 9 refuses `net0` writes to non-root users without
it; a missing privilege is a 403 Permanent that latches the Proxy until
its spec changes. The guide's privilege list is the mitigation.
- **The ide2 seed and a template with a cloudinit drive** both claim
cloud-init; the Proxy's user-data would be ignored. Documented
prerequisite; the Checker cannot see it.
- **VMID range shared with humans.** A human VM inside the range costs a
probe; a human VM carrying our name is Permanent. The guide says to
reserve the range.
- **Node-local ISO storage after a migration** leaves the seed on the
original node; Delete tolerates its absence and the ISO is a documented
blind spot. Shared ISO storage avoids it.
- **Blocking Create** holds a reconcile worker for up to the clone budget;
the guide sets expectations.
- **The 30s request timeout on the seed upload** is fine for tens of KiB; a
multi-megabyte `vendorData` would trip it. Documented.
- **Two operator installs** on one Proxmox cluster delete each other's VMs
(contract §13); documented in entry 24.
- **kvm2 edits in Phase 3** touch a merged provider; the moved tests are the
guard, and `go vet -tags=integration` keeps its integration test compiling.
docs/providers-existing.md
+469 -68
View File
@@ -1,13 +1,20 @@
# The existing providers
This is the appendix to [provider-contract.md](provider-contract.md): how the
three providers that exist today (kubernetes, gcp, kvm) meet the contract,
which profile each one chose on every axis, and what is still missing. It is
reference material for whoever writes the next provider, not a user guide;
the user guides are [kvm-provider.md](kvm-provider.md),
five providers that exist today (kubernetes, gcp, kvm, kvm2, proxmox) meet
the contract, which profile each one chose on every axis, and what is still
missing. It is reference material for whoever writes the next provider, not
a user guide; the user guides are [kvm-provider.md](kvm-provider.md),
[kvm2-provider.md](kvm2-provider.md),
[proxmox-provider.md](proxmox-provider.md),
[gcp-vm-validation.md](gcp-vm-validation.md) and
[gcp-in-specific-project.md](gcp-in-specific-project.md). The README's
"Implemented providers" table stays the short public summary.
"Implemented providers" table stays the short public summary. kvm2 and
proxmox are the two providers built from the contract itself (kvm2
clean-room from [kvm2-provider-spec.md](kvm2-provider-spec.md) without
reading kvm's code, proxmox from
[proxmox-provider-spec.md](proxmox-provider-spec.md)), so their columns are
also a record of what the contract produced.
The per-provider call flows are drawn in
[architecture.md §5](architecture.md#5-what-provider-calls-do-in-the-outside-world),
@@ -23,59 +30,59 @@ Code references are a file path plus a function or symbol name.
One row per axis of contract §4, in the same order, with the profile name
and the symbol that shows it.
| Axis | kubernetes | gcp | kvm |
|---|---|---|---|
| `readiness` | immediate: a Pod is scheduled within seconds (`stateFromPod`) | provisioning-window: minutes of boot (`mapState`) | provisioning-window: boot plus, on first use, an image download (`imageCache.ensure`) |
| `create-return` | submit: returns after `client.Create` | submit: `Insert` is never awaited, `Operation.Wait` is not called | **blocking**: waits through `DomainCreate` (`ensureRunning`), and the first image fetch runs inside (`imageCache.fetch`) |
| `delete-return` | returns-early: `client.Delete`, Pod terminates on its own | returns-early: `Delete` submits an operation each time | waits: destroy, undefine and volume removal complete before return (`removeDomain`, `removeVolumes`) |
| `ip-source` | assigned-at-create: `PodIP` (`instanceFromPod`) | assigned-at-create: the external NAT IP of the first NIC (`toInstance`) | discovered: ARP table first, then the guest agent, first routable IPv4 (`address`, `firstIPv4`) |
| `gc-mark` | native-labels: Pod labels (`buildPod`) | native-labels: instance labels (`buildInsertRequest`) | written-metadata: a namespaced `<metadata>` element written in `DomainDefineXML` (`metadata.go`) |
| `created-at` | backend-timestamp: `CreationTimestamp` | backend-timestamp: `CreationTimestamp`, parse error gives zero | written-at-create: `createdAt` in the metadata element (`newInstanceMetadata`) |
| `list-partial` | n/a: one call, all namespaces | omit-unreachable: `ReturnPartialSuccess` on `AggregatedList` | fail-whole per host, one unreadable domain skipped (`ListByTag`, `managedInstance`) |
| `quota` | absent: a quota 403 is Permanent (`classify`) | detected: 429, and 403 with reason `quotaExceeded` or `rateLimitExceeded` (`classify`, `hasReason`) | absent (`classify`) |
| `not-found` | status-404: `apierrors.IsNotFound` | status-404: `isNotFound` | error-code: `ErrNoDomain`, `ErrNoStorageVol` (`hasCode`) |
| `primary-key` | name-is-key: `namespace/name` | name-is-key: `zones/<zone>/instances/<name>` | name-is-key: `hosts/<host>/domains/<name>` |
| `provisioning-state` | native: `PodPending` | native: `PROVISIONING`, `STAGING`, `REPAIRING` | create-blocks: a domain is defined and started in one `Create`; `nostate` maps to Provisioning |
| `async-ops` | none | task-handle, but the handle is discarded: an insert that fails after acceptance is never read | none |
| `secondary-resources` | none | cascade: `AutoDelete` on the boot disk (`buildInsertRequest`) | explicit: overlay and seed removed by name (`removeVolumes`); base image kept |
| `credentials` | workload-identity: in-cluster config or kubeconfig (`NewWithTransportWrapper`) | environment: Application Default Credentials | mounted-files: SSH key and known_hosts from the optional `kvm-ssh` Secret, read lazily (`newLazySSH`) |
| `client-kind` | vendor-sdk: controller-runtime `client.Client`, no seam interface | vendor-sdk: `cloud.google.com/go/compute` behind `instancesAPI` | rpc-over-ssh: `go-libvirt` over an SSH-forwarded unix socket, behind `libvirtAPI` (`conn.go`) |
| `timeouts` | ctx-only: no `rest.Config.Timeout` is set | ctx-only: SDK defaults, with SDK retries on Get and List | per-call for the dial only (`defaultHandshakeTimeout` in `dialSSH`); RPCs are ctx-only and a cancelled `call` closes the connection |
| `logging` | none | tiered: V(1) outcomes and `logAPIError`, V(2) detail, V(5) wire log (`wirelog.go`) | tiered: V(1) outcomes and one `libvirt call` line per RPC (`trace.go`), V(2) XML and addresses; no wire tier |
| `child-spans` | transport-wrapper: `tracing.RestConfigWrapper()` on its own `rest.Config`, parent-gated | sdk-instrumentation: the SDK adds `otelhttp` itself; not parent-gated, but every call site has a parent | **none**: `trace.go` is a log decorator, not spans |
| `backend-test` | cluster-e2e: kind (`test/e2e`) | manual-runbook: [gcp-vm-validation.md](gcp-vm-validation.md) | integration-tag: `kvm_integration_test.go`, `make test-integration-kvm` |
| Axis | kubernetes | gcp | kvm | kvm2 | proxmox |
|---|---|---|---|---|---|
| `readiness` | immediate: a Pod is scheduled within seconds (`stateFromPod`) | provisioning-window: minutes of boot (`mapState`) | provisioning-window: boot plus, on first use, an image download (`imageCache.ensure`) | provisioning-window: boot plus, on first use, an image download (`ensureImage`) | provisioning-window: boot plus the guest agent's install at first boot, about 50 s to an address on the test host (`agentAddress`) |
| `create-return` | submit: returns after `client.Create` | submit: `Insert` is never awaited, `Operation.Wait` is not called | **blocking**: waits through `DomainCreate` (`ensureRunning`), and the first image fetch runs inside (`imageCache.fetch`) | **blocking**: waits through `DomainCreate` and a 15-second poll for running (`ensureRunning`, `waitRunning`), and the first image fetch runs inside (`fetchImage`) | **blocking**: waits through the clone, upload, resize and start tasks and a 60-second poll for running (`provision`, `waitRunning`) |
| `delete-return` | returns-early: `client.Delete`, Pod terminates on its own | returns-early: `Delete` submits an operation each time | waits: destroy, undefine and volume removal complete before return (`removeDomain`, `removeVolumes`) | waits: destroy, undefine and volume removal complete before return (`removeDomain`, `removeVolumes`) | waits: the stop, seed delete and destroy tasks complete before return (`destroy`, `deleteSeed`) |
| `ip-source` | assigned-at-create: `PodIP` (`instanceFromPod`) | assigned-at-create: the external NAT IP of the first NIC (`toInstance`) | discovered: ARP table first, then the guest agent, first routable IPv4 (`address`, `firstIPv4`) | discovered: the guest agent first, then the ARP table, first routable IPv4 (`addressSources`, `address`, `firstIPv4`) | discovered: the guest agent only, first routable IPv4 in the agent's order (`agentAddress`, `firstIPv4`) |
| `gc-mark` | native-labels: Pod labels (`buildPod`) | native-labels: instance labels (`buildInsertRequest`) | written-metadata: a namespaced `<metadata>` element written in `DomainDefineXML` (`metadata.go`) | written-metadata: an element in kvm2's own namespace written in `DomainDefineXML` (`metadataURI`, `newInstanceMetadata`) | written-metadata: three `key=value` lines in the VM's description, passed in the clone call itself (`formatDescription`, `parseDescription`) |
| `created-at` | backend-timestamp: `CreationTimestamp` | backend-timestamp: `CreationTimestamp`, parse error gives zero | written-at-create: `createdAt` in the metadata element (`newInstanceMetadata`) | written-at-create: `createdAt` in the metadata element, parse error gives zero (`newInstanceMetadata`, `createdAt`) | written-at-create: `proxy-operator-created-at` in the description, parse error gives zero (`parseDescription`) |
| `list-partial` | n/a: one call, all namespaces | omit-unreachable: `ReturnPartialSuccess` on `AggregatedList` | fail-whole per host, one unreadable domain skipped (`ListByTag`, `managedInstance`) | fail-whole per host, one unreadable domain skipped unless ctx ended; `ID`, `UID` and `CreatedAt` from the name and the mark only, no state or address RPC (`ListByTag`) | fail-whole for the one cluster-wide `/cluster/resources` call, one unreadable VM skipped unless ctx ended; `ID`, `UID` and `CreatedAt` from the description only, no status or agent call (`ListByTag`) |
| `quota` | absent: a quota 403 is Permanent (`classify`) | detected: 429, and 403 with reason `quotaExceeded` or `rateLimitExceeded` (`classify`, `hasReason`) | absent (`classify`) | detected: a pool-space precheck before any volume is created (`checkPoolSpace`, `neededBytes`) | absent: thin storage overprovisions, so no out-of-space answer exists to match; free space goes to the Checker's details (`classify`) |
| `not-found` | status-404: `apierrors.IsNotFound` | status-404: `isNotFound` | error-code: `ErrNoDomain`, `ErrNoStorageVol` (`hasCode`) | error-code: `ErrNoDomain`, `ErrNoStorageVol`, Transient inside Create (`classifyLibvirt`) | error-code: the 500 `Configuration file '…' does not exist`, plus list-and-miss on `/cluster/resources`; Transient inside Create (`isNotExist`, `classifyAPI`) |
| `primary-key` | name-is-key: `namespace/name` | name-is-key: `zones/<zone>/instances/<name>` | name-is-key: `hosts/<host>/domains/<name>` | name-is-key: `hosts/<host>/domains/<name>`, with a UID guard on adoption (`create`) | **allocated-id**: `vms/<vmid>/<name>`, the VMID from an FNV-1a hash of the UID with forward probing, each taken VMID put through an ownership check with a UID guard (`vmidFor`, `allocate`, `inspect`) |
| `provisioning-state` | native: `PodPending` | native: `PROVISIONING`, `STAGING`, `REPAIRING` | create-blocks: a domain is defined and started in one `Create`; `nostate` maps to Provisioning | create-blocks: defined, started and polled to running in one `Create`; `nostate` maps to Provisioning (`mapState`) | create-blocks: cloned, configured, started and polled to running in one `Create`; `qmpstatus: prelaunch` maps to Provisioning (`mapState`) |
| `async-ops` | none | task-handle, but the handle is discarded: an insert that fails after acceptance is never read | none | none | task-handle and backend-lock: every clone, upload, resize, start, stop, destroy and volume delete returns a UPID polled to its end within the call (`waitTask`); a VM's lock is waited on in Create and Transient in Get and Delete (`waitUnlocked`) |
| `secondary-resources` | none | cascade: `AutoDelete` on the boot disk (`buildInsertRequest`) | explicit: overlay and seed removed by name (`removeVolumes`); base image kept | explicit: overlay and seed removed by name (`removeVolumes`); base image kept | explicit: the seed ISO deleted by name (`deleteSeed`, `deleteSeedEverywhere`); the disks go with the destroy's `purge=1&destroy-unreferenced-disks=1` |
| `credentials` | workload-identity: in-cluster config or kubeconfig (`NewWithTransportWrapper`) | environment: Application Default Credentials | mounted-files: SSH key and known_hosts from the optional `kvm-ssh` Secret, read lazily (`newLazySSH`) | mounted-files: the same `kvm-ssh` Secret, read on every dial and never cached (`loadCredentials`) | mounted-files: the token from the optional `proxmox-token` Secret, read on every request (`readToken`); the CA from the same Secret, loaded once and cached (`httpClient`) |
| `client-kind` | vendor-sdk: controller-runtime `client.Client`, no seam interface | vendor-sdk: `cloud.google.com/go/compute` behind `instancesAPI` | rpc-over-ssh: `go-libvirt` over an SSH-forwarded unix socket, behind `libvirtAPI` (`conn.go`) | rpc-over-ssh: `go-libvirt` over an SSH-forwarded unix socket, behind the ctx-first `backend` seam (`backend.go`) | thin-client: `net/http` and `encoding/json` behind the ctx-first `api` seam, one method per endpoint (`api.go`, `httpAPI`) |
| `timeouts` | ctx-only: no `rest.Config.Timeout` is set | ctx-only: SDK defaults, with SDK retries on Get and List | per-call for the dial only (`defaultHandshakeTimeout` in `dialSSH`); RPCs are ctx-only and a cancelled `call` closes the connection | per-call: a budget around the dial, every RPC, the uploads and the download (`defaultBudgets`, `call`); a cancelled or overlong call kills the pooled connection | per-call: dial and TLS handshake 10 s, each request `requestTimeoutSeconds`, a budget per task wait (`defaultBudgets`); a cancelled request ends only itself |
| `logging` | none | tiered: V(1) outcomes and `logAPIError`, V(2) detail, V(5) wire log (`wirelog.go`) | tiered: V(1) outcomes and one `libvirt call` line per RPC (`trace.go`), V(2) XML and addresses; no wire tier | tiered: V(1) outcomes and one `libvirt call` line per RPC (`observe.go`), V(2) XML and addresses; no wire tier | tiered: V(1) outcomes and one `proxmox api call` line per request (`observe.go`), V(2) parameters, config summaries and addresses; no wire tier |
| `child-spans` | transport-wrapper: `tracing.RestConfigWrapper()` on its own `rest.Config`, parent-gated | sdk-instrumentation: the SDK adds `otelhttp` itself; not parent-gated, but every call site has a parent | **none**: `trace.go` is a log decorator, not spans | span-decorator: `kvm2.<Method>` per RPC (`observe`), plus `kvm2.dial` (`libvirtDialer`) and `kvm2.download` (`fetchImage`) | transport-wrapper: `tracing.RestConfigWrapper()` on the client's transport, parent-gated, one `HTTP <METHOD>` span per request (`newHTTPAPI`) |
| `backend-test` | cluster-e2e: kind (`test/e2e`) | manual-runbook: [gcp-vm-validation.md](gcp-vm-validation.md) | integration-tag: `kvm_integration_test.go`, `make test-integration-kvm` | integration-tag: `kvm2_integration_test.go`, `make test-integration-kvm2` | integration-tag: `proxmox_integration_test.go`, `make test-integration-proxmox`; passed on a PVE 9.2.20 host |
## 2. Other differences
Rows that are not profile axes but that a new provider author compares
against.
| Feature | kubernetes | gcp | kvm |
|---|---|---|---|
| Create verifies the adopted instance's UID | no | no | no |
| Placement used | none | Zone, MachineType, Image | Zone (host), MachineType, Image |
| CloudInit delivered | **no**: ignored | yes: `user-data` metadata | yes: NoCloud seed ISO built in-process (`buildSeedISO`) |
| Port used | yes: Squid config and containerPort (`squidConf`) | no: firewall is manual | no |
| States produced | Provisioning, Running, Terminated | all four | Provisioning, Running, Stopped |
| Deleting instance still reports its state | yes: a Pod being deleted reports its phase | n/a | n/a |
| Error.Provider is the configured name | **no**: hardcoded `"kubernetes"` | yes | yes |
| Checker target | `apiserver` | the project | each host, concurrently |
| Checker Details | none | none | `user`, `pool`, `hostname`, `libvirtVersion`, `poolState`, `poolCapacity`, `poolAvailable` |
| Credential rotation without restart | n/a | via ADC | no: cached after first successful load |
| Connection handling | client-go | SDK client, never closed | one pooled SSH plus libvirt connection per host, 15s keepalive (`hostPool`) |
| Unit-test double | controller-runtime fake client | `fakeAPI` behind `instancesAPI`, plus `httptest` for the real SDK client in `check_test.go` | `fakeAPI` behind `libvirtAPI`, plus an in-process `x/crypto/ssh` server in `conn_test.go` |
| Proxy software | generated Squid config (`pod.go`) | from `cloudInit`; the image must run cloud-init | from `cloudInit`, plus a vendor-data script that installs the guest agent (`seed.go`) |
| IP reachable from the operator | Pod network, so in-cluster only (not from `make run-dev` on a laptop) | external IP, once a firewall rule allows the cluster's egress IP | LAN address on the host's bridge; the operator needs a route to that LAN |
| Distinct egress IP per proxy | no: the cluster's egress | yes: ephemeral external IP | whatever the host's LAN routes through; not per VM by itself |
| Feature | kubernetes | gcp | kvm | kvm2 | proxmox |
|---|---|---|---|---|---|
| Create verifies the adopted instance's UID | no | no | no | **yes**: a domain without the kvm2 mark or with a different non-empty UID is Permanent; a marked one with a matching or empty UID is adopted (`create`) | **yes**: a VM with the proxy's name and no mark, or with a different non-empty UID, is Permanent; a marked one with a matching or empty UID is adopted (`inspect`) |
| Placement used | none | Zone, MachineType, Image | Zone (host), MachineType, Image | Zone (host), MachineType, Image | Zone (node), MachineType, Image (a template key) |
| CloudInit delivered | **no**: ignored | yes: `user-data` metadata | yes: NoCloud seed ISO built in-process (`buildSeedISO`) | yes: NoCloud seed ISO built in-process (`nocloud.BuildSeedISO`) | yes: NoCloud seed ISO built in-process (`nocloud.BuildSeedISO`) and uploaded through the API (`uploadSeed`) |
| Port used | yes: Squid config and containerPort (`squidConf`) | no: firewall is manual | no | no | no |
| States produced | Provisioning, Running, Terminated | all four | Provisioning, Running, Stopped | Provisioning, Running, Stopped | Provisioning, Running, Stopped |
| Deleting instance still reports its state | yes: a Pod being deleted reports its phase | n/a | n/a | n/a: Delete waits, and a concurrent Get sees the domain until it is undefined | n/a: Delete waits, and a concurrent Get sees the VM (Stopped once stopped) until the destroy removes its config |
| Error.Provider is the configured name | **no**: hardcoded `"kubernetes"` | yes | yes | yes (`wrapErr`) | yes (`wrapErr`) |
| Checker target | `apiserver` | the project | each host, concurrently | each host, concurrently, on a fresh connection | each node, concurrently, on the shared HTTP client |
| Checker Details | none | none | `user`, `pool`, `hostname`, `libvirtVersion`, `poolState`, `poolCapacity`, `poolAvailable` | `user`, `pool`, `hostname`, `libvirtVersion`, `poolState`, `poolCapacityGiB`, `poolAvailableGiB`, filled in as far as the check got | `isoStorage`, `tokenID`, `pveVersion`, `kernel`, `isoStorageAvailGiB`, `templates`, filled in as far as the check got |
| Credential rotation without restart | n/a | via ADC | no: cached after first successful load | yes: both files read on every dial (`loadCredentials`) | the token yes, read on every request (`readToken`); the CA no, cached after first use |
| Connection handling | client-go | SDK client, never closed | one pooled SSH plus libvirt connection per host, 15s keepalive (`hostPool`) | one pooled SSH plus libvirt connection per host, 15s keepalive (`hostTable`); dedicated connections for the image upload and the check (`dialFresh`) | one `net/http` client, 4 idle connections per host, HTTP/1.1 only; a cancelled request ends only itself |
| Unit-test double | controller-runtime fake client | `fakeAPI` behind `instancesAPI`, plus `httptest` for the real SDK client in `check_test.go` | `fakeAPI` behind `libvirtAPI`, plus an in-process `x/crypto/ssh` server in `conn_test.go` | `fakeBackend` behind `backend` (`fake_test.go`), plus an in-process `x/crypto/ssh` server (`sshserver_test.go`) and go-libvirt's own mock for the adapter | `fakeAPI` behind `api` (`fake_test.go`), plus `httptest.NewTLSServer` replaying the Phase 0 captures for the real client (`client_test.go`) |
| Proxy software | generated Squid config (`pod.go`) | from `cloudInit`; the image must run cloud-init | from `cloudInit`, plus a vendor-data script that installs the guest agent (`seed.go`) | from `cloudInit`, plus a vendor-data script that installs the guest agent (`nocloud.GuestAgentScript`) | from `cloudInit`, plus a vendor-data script that installs the guest agent (`nocloud.GuestAgentScript`) |
| IP reachable from the operator | Pod network, so in-cluster only (not from `make run-dev` on a laptop) | external IP, once a firewall rule allows the cluster's egress IP | LAN address on the host's bridge; the operator needs a route to that LAN | LAN address on the host's bridge; the operator needs a route to that LAN | address on the proxy network (bridge or VLAN from the config); the operator needs a route to it |
| Distinct egress IP per proxy | no: the cluster's egress | yes: ephemeral external IP | whatever the host's LAN routes through; not per VM by itself | whatever the host's LAN routes through; not per VM by itself | whatever the proxy network routes through; not per VM by itself |
## 3. Observability status
| Layer | kubernetes | gcp | kvm |
|---|---|---|---|
| 1. Operation spans and metrics | yes: decorators | yes: decorators | yes: decorators |
| 2. Child spans per backend call | yes: `tracing.RestConfigWrapper()`, parent-gated (`cmd/main.go`, `NewWithTransportWrapper`) | yes: the SDK's own `otelhttp`; not parent-gated | **no** |
| 3. Logs carry `traceID` and `spanID` | n/a: the package does not log | yes, including the V(5) wire log | yes: every `libvirt call` line at V(1) |
| Tested against a real collector | yes: the kind e2e checks `provider.create` and `provider.delete` in Tempo (`test/e2e/tracing_test.go`), operation spans only | no | no |
| Layer | kubernetes | gcp | kvm | kvm2 | proxmox |
|---|---|---|---|---|---|
| 1. Operation spans and metrics | yes: decorators | yes: decorators | yes: decorators | yes: decorators | yes: decorators |
| 2. Child spans per backend call | yes: `tracing.RestConfigWrapper()`, parent-gated (`cmd/main.go`, `NewWithTransportWrapper`) | yes: the SDK's own `otelhttp`; not parent-gated | **no** | yes: `kvm2.<Method>` from the `observe` decorator, `kvm2.dial`, `kvm2.download`; always under the caller's span, since every call carries the caller's ctx | yes: `tracing.RestConfigWrapper()` on the transport, parent-gated, one `HTTP <METHOD>` per request |
| 3. Logs carry `traceID` and `spanID` | n/a: the package does not log | yes, including the V(5) wire log | yes: every `libvirt call` line at V(1) | yes: every `libvirt call` line at V(1), written inside its RPC's span | yes: every `proxmox api call` line at V(1), with the operation span's `spanID` |
| Tested against a real collector | yes: the kind e2e checks `provider.create` and `provider.delete` in Tempo (`test/e2e/tracing_test.go`), operation spans only | no | no | no; the integration test asserts the parent-child topology in an in-memory recorder (`assertLifecycleSpans`) | no; `TestTracing_httpSpansUnderOperationSpan` asserts the parent-child topology in an in-memory recorder |
## 4. kubernetes (`internal/provider/kubernetes`)
@@ -329,18 +336,404 @@ the author's view.
tries ARP first; it says `host` and `name` are on every log line, but
the check and list paths lack them.
## 7. Gaps shared by the existing providers
## 7. kvm2 (`internal/provider/kvm2`)
- **Adoption is not verified.** None of the three checks `LabelUID` when
`Create` finds an existing instance with the right name.
- **The Create comment does not match kvm.** The interface comment in
`internal/provider/provider.go` says `Create` returns as soon as the
request is submitted; kvm blocks until the domain has started.
The full user guide is [kvm2-provider.md](kvm2-provider.md) and the design
it was built from is [kvm2-provider-spec.md](kvm2-provider-spec.md); this
section is the author's view. Nothing in the package was taken from
`internal/provider/kvm`; where the two agree it is because both follow the
same libvirt.
- **Package and seam.** `kvm2.go` (`Provider`, `New`, `withDefaults`,
`host`, `wrapErr`, the per-name lock), `create.go`, `get.go`,
`delete.go`, `list.go` (the four methods), `backend.go` (the seam, the
production adapter and the budgets), `ssh.go` (credentials, the forwarded
socket, keepalive), `hosts.go` (the pooled table and the production
dialer), `observe.go` (the span and log decorator), `image.go` (cache,
download, quota precheck), `xml.go`, `volume.go`, `metadata.go`,
`id.go`, `state.go`, `errors.go`, `check.go`; the seed ISO comes from
the shared `internal/provider/nocloud` and both lock sets are
`internal/provider/keyedlock`. The seam is
`backend`: the twenty go-libvirt calls the provider uses, each with `ctx`
first (go-libvirt's own methods take none and block), plus `Close`.
`libvirtBackend` adapts `*libvirt.Libvirt` and runs every call through
`call`, which applies the budget and the abandon policy. The `dialer`
seam separates the SSH transport: `newProvider` injects one for tests,
`newLibvirtDialer` is production. The `observe` decorator is applied
once, in `libvirtDialer.dial`, so provider code never holds an
undecorated backend.
- **Config.** `ProviderConfig.KVM2`, the same `KVMConfig` type as kvm,
validated by `validateKVMBlock` with block name `kvm2` so messages read
`kvm2.hosts`. Defaults (`diskSizeGb` 10, socket, pool, image format and
firmware) are filled by `withDefaults`. `New` never dials; it checks that
the credential files exist and warns when one does not, or when
`insecureIgnoreHostKey` is on.
- **ID.** `formatProviderID(host, name)` gives `hosts/<host>/domains/<name>`;
`parseProviderID` accepts exactly four segments with the literal
`hosts` and `domains` and non-empty host and name, else `ErrPermanent`.
- **Create.** `Create` validates `Zone` (`host`; optional with one host),
`MachineType` (a catalogue key) and `Image` (`resolveImage`: a catalogue
key or an http(s) URL with qcow2 and uefi defaults) before any dial, and
logs the resolved parameters with `cloudInitBytes` at V(2). Then, under
the per-name lock (`lockName`) and on the pooled connection
(`hostTable.get`), `create`: `DomainLookupByName`; found means the
adoption guard (no kvm2 mark, or a non-empty metadata UID that differs
from `req.UID`, is Permanent) and `ensureRunning`. Otherwise
`StoragePoolLookupByName`, `imageSizeIfUncached` (marker lookup, then
`HEAD`, or a closed `GET` when `HEAD` is not 2xx) and
`checkPoolSpace` with `neededBytes` (image if uncached, plus
`diskSizeGb`, plus 16 MiB), then `ensureImage` (`cachedImage` on the
pooled connection, `fetchImage` streaming into `uploadImage` on a
`dialFresh` connection, `StoragePoolRefresh`, `writeMarker` last),
`ensureOverlay` (`ErrStorageVolExist` means reuse), `writeSeed` (delete,
build, upload; always rebuilt), two `StorageVolGetPath`, `buildDomainXML`
with file-type disks and the metadata element, `DomainDefineXML`,
`ensureRunning` (`DomainGetState`, where paused or pmsuspended is
Permanent; `DomainCreate` with `ErrOperationInvalid` as already running;
`waitRunning` polls every 500 ms for up to 15 seconds, Transient on
expiry and at once for any inactive state other than `nostate`).
`Create` returns only when the domain is running.
- **Get and state.** `mapState(state, ip)`:
| libvirt state | IP | Provider state |
|---|---|---|
| nostate | any | Provisioning |
| running, blocked | empty | Provisioning |
| running, blocked | set | Running |
| paused, shutdown, shutoff, crashed, pmsuspended, anything else | any | Stopped |
`get`: `DomainLookupByName` (only `ErrNoDomain` reaches NotFound),
`readMetadata` (`DomainGetMetadata` with `metadataURI`;
`ErrNoDomainMetadata` means unmanaged with an empty UID; an element that
does not parse is Permanent), then `DomainGetState` and, for an active
domain, `address`. `addressSources` asks the guest agent first,
then ARP; a source that fails is skipped at V(2), only ctx ending is an
error, so a cancelled Get is Transient and not a false Provisioning.
`firstIPv4` drops loopback, link-local and unspecified addresses and an
address whose type tag and literal disagree. `UID` and `CreatedAt` are
read back on every call; an unparsable `createdAt` gives the zero time.
- **Delete.** Under the per-name lock: `removeDomain` (`DomainDestroy`,
tolerating not running and gone; `DomainUndefineFlags` with NVRAM,
managed save and snapshot metadata) and `removeVolumes` (overlay and seed
by name through `deleteVolumeIfPresent`; a missing pool is a no-op). A
host no longer in the config is a logged no-op success. Nothing is left
when nil is returned.
- **ListByTag.** `ConnectListAllDomains` (active and inactive) on every
host in config order; `readMetadata` and `managed` keep kvm2's own
domains, and the `Instance` is `ID`, `UID` and `CreatedAt` from the name
and the element, with `State` and `IP` empty: GC reads nothing else
(contract §2.4), so no state or address RPC is made. A domain whose
metadata cannot be read is skipped at V(1), unless ctx ended, which
fails the call. A host that cannot be dialled or listed fails the whole
call.
- **Errors.** `classify(op, err)`: an error already carrying a class keeps
it; libvirt codes through `classifyLibvirt` (`ErrNoDomain` and
`ErrNoStorageVol` NotFound, Transient when `op == "create"`;
`ErrNoStoragePool`, `ErrXMLError`, `ErrNoSupport`, `ErrInvalidArg`,
`ErrAuthFailed`, `ErrAccessDenied` Permanent; the rest Transient);
`knownhosts.KeyError` and the `unable to authenticate` text from
`x/crypto/ssh` Permanent; everything else Transient, including a missing
credential file. `permanentf` and `transientf` classify at the source:
placement, XML builders, a parse failure of the key or known_hosts, an
image 4xx, a missing or zero `Content-Length`, a sha256 mismatch, a pool
that is not running (Permanent); an image 429 or 5xx, a short body, the
running wait expiring (Transient). `checkPoolSpace` is the only source of
`ErrQuotaExceeded`. `logAPIError` writes `kvm2 backend call failed` with
`libvirtErrorFields`.
- **Check.** `Check` runs `checkHost` per host concurrently, each through
`probeHost` on a `dialFresh` connection that is closed when the probe
ends; the pooled table is never consulted. `ConnectGetLibVersion`,
`ConnectGetHostname`, `StoragePoolLookupByName`, `StoragePoolGetInfo`;
a pool that is not running is `poolNotActiveError`. Details are filled
as facts arrive (`user` and `pool` always; `libvirtVersion`, `hostname`,
`poolState`, `poolCapacityGiB`, `poolAvailableGiB`). `checkHost` answers
`Timeout` whenever the round's ctx ended, whatever stage was running and
whatever shape the error has; otherwise `checkReason`: a `dialError` is
judged by its stage (`credentials` gives `CredentialsUnavailable`; `ssh`
with a `knownhosts.KeyError` or the auth text gives `Unauthorized`; any
other stage, including a dial that ran out of its own budget, gives
`Unreachable`); then `context.DeadlineExceeded` or `Canceled` `Timeout`,
`ErrAccessDenied` or `ErrAuthFailed` `Forbidden`, `ErrNoStoragePool`
`PoolNotFound`, any other libvirt code `APIError`, anything else
`Unreachable`; `PoolNotActive` from the pool state.
- **Connections.** `hostTable` keeps one pooled connection per host,
dialled lazily under a per-host one-slot semaphore so waiting ends with
the caller's ctx; a dead backend (`liveness`) is closed and redialled on
the next `get`. `dialSSHSocket` runs credentials, TCP, the SSH handshake
and the `direct-streamlocal` channel to the socket, each failure a
`dialError` with its stage; `libvirtDialer.connect` bounds the whole
sequence plus the libvirt `Connect` with the dial budget and starts the
`keepalive` (15 s, `keepalive@openssh.com`) and a watcher on go-libvirt's
`Disconnected`. `call` runs each blocking go-libvirt call in a goroutine
and waits on the result, ctx and a budget timer; abandoning a call
(`link.markDead`) closes the socket and the SSH client, returns Transient
wrapping the ctx or deadline error, and every other call waiting on that
link fails once with Transient. `loadCredentials` runs on every dial.
- **Logging.** Logger from the context, named `kvm2`, with `provider`,
and `host` and `name` on lines about an instance. V(1): one line per
operation outcome (`kvm2 domain ensured running`, `kvm2 domain fetched`,
`kvm2 domain deleted`, `kvm2 domains listed`, `kvm2 host checked`), one
`libvirt call` line per RPC from `observe` (rpc name, identifiers,
duration, error fields), the connection events `libvirt host connected`
and `libvirt host connect failed` with `stage`, and `kvm2 backend call
failed`. V(2): XML, metadata, addresses, listed names, pool sizes, the
resolved Create parameters. Upload and download bodies, cloud-init and
key material are never logged.
- **Tracing.** `observe.call` starts `kvm2.<Method>` per RPC with
`kvm2.host` and the identifiers (`kvm2.domain`, `kvm2.pool`,
`kvm2.volume`), sets `kvm2.libvirt_code` on failure and writes the
`libvirt call` line inside the span; `libvirtDialer.dial` starts
`kvm2.dial` (`kvm2.address`, `kvm2.user`); `fetchImage` starts
`kvm2.download` (`kvm2.url_host`, `kvm2.sha256_configured`,
`kvm2.bytes`). Both the decorator and the dialer take `tracing.Option`
so tests inject a recorder; `New` leaves them on the global provider.
- **Tests.** Pure builders (`xml_test.go`, `volume_test.go`,
`seed_test.go`, `id_test.go`, `metadata_test.go`, `state_test.go`);
`errors_test.go` (the classify table with real SSH error strings and a
real `knownhosts.KeyError`); `backend_test.go` (the adapter over
go-libvirt's own mock through a `net.Pipe`: deadline kills the
connection, cancel unblocks, killed by another caller); `hosts_test.go`;
`ssh_test.go` over `sshserver_test.go` (every dial stage, host-key
policies, key material never in errors or logs); `image_test.go`
(marker rules, streaming asserted by counting bytes, dedicated upload
connection); `create_test.go`, `get_test.go`, `delete_test.go`,
`list_test.go`, `kvm2_test.go`, `provider_test.go` (the contract §11
list, named test by test in the spec's entry 21); `observe_test.go`
(every seam method spanned and logged exactly once, by reflection);
`check_test.go` (the reason table, never touches the pool). The double is
`fakeBackend` in `fake_test.go`, whose `twin` shares state under its own
call log so a test can tell the pooled connection from a dedicated one.
Real backend: `kvm2_integration_test.go` behind `//go:build integration`,
driven by `KVM2_TEST_*` variables and `make test-integration-kvm2`,
described in
[testing.md](testing.md#the-kvm2-integration-test-real-hypervisor).
- **Deployment.** The `kvm-ssh` Secret and its mount are shared with kvm
(`config/manager/manager.yaml` says so), `make kvm-ssh-secret` is reused,
a commented `kvm2-home` entry in both providers configs,
`config/samples/proxy_kvm2.yaml` (zone `beelink`, matching the shipped
config), the `kvm2` type in the `status.type` comment of
`api/v1alpha1/providerhealth_types.go`. No RBAC.
- **Known gaps** (see also [kvm2-provider.md § Known gaps](kvm2-provider.md#known-gaps)).
- `Create` blocks, including the first image download on each host.
- GC blind spots: volumes without a domain are invisible to GC, and so are
VMs on a host removed from the config, which `Delete` treats as done.
- No adoption of kvm's domains: a switched entry needs its Proxies
deleted first.
- Cancelling one operation cuts off the others on that host once: a
cancelled `call` kills the pooled connection.
- `Content-Length` is required for image downloads.
- A paused or pmsuspended domain is not started by a repeat `Create`:
`ensureRunning` returns Permanent, so the Proxy latches until its spec
changes; the admin resumes the VM on the host.
- Credential parse errors carry the parser's own text, which can quote a
PEM header type or a malformed known_hosts token (never key bytes).
- No reused-overlay check: an existing overlay is reused without checking
its backing image or size.
- The sample cloud-init installs Squid with `http_access allow all`.
## 8. proxmox (`internal/provider/proxmox`)
The full user guide is [proxmox-provider.md](proxmox-provider.md) and the
design it was built from is
[proxmox-provider-spec.md](proxmox-provider-spec.md); this section is the
author's view. It is the second provider built from the contract, and the
first on a REST API of the hypervisor's own: no SSH, no libvirt. Every
error text it matches was captured from a PVE 9.2.20 host before the code
was written.
- **Package and seam.** `proxmox.go` (`Provider`, `New`, `withDefaults`,
`node`, `wrapErr`, the per-name lock, the budgets), `create.go`,
`get.go`, `delete.go`, `list.go` (the four methods), `api.go` (the
seam), `client.go` (the production adapter `httpAPI`, the token and CA
loading), `observe.go` (the logging decorator), `task.go` (task, lock
and running waits), `seed.go`, `mark.go`, `id.go`, `state.go`,
`errors.go`, `check.go`; the seed ISO comes from the shared
`internal/provider/nocloud` and the name lock from
`internal/provider/keyedlock`. The seam is `api`: one method per
endpoint, sixteen in all, each with `ctx` first. `New` wraps `httpAPI` in
`observe` once, so provider code never holds an undecorated seam;
`newProvider` is the test constructor.
- **Config.** `ProviderConfig.Proxmox`, the `ProxmoxConfig` type in
`internal/provider/config_proxmox.go`, validated by `validateProxmox`
(the URL shape, node names and storage ids by regexp because they end
up in request paths, `vlanTag`, `vmidRange`, the two catalogues).
`isoStorage` has no default on purpose: the token needs
`Datastore.Allocate` there. Defaults (`vmidRange` 100000 to 199999,
`requestTimeoutSeconds` 30, `installGuestAgent` true) are filled by
`withDefaults`. `New` never makes a request; it warns when the token
file does not exist, or when `insecureSkipTLSVerify` is on.
- **ID.** `formatProviderID(vmid, name)` gives `vms/<vmid>/<name>`, with no
node, because VMs migrate; the node is resolved from
`/cluster/resources` on every call (`resolve`). `parseProviderID`
accepts exactly three segments, the literal `vms`, a decimal VMID
without leading zeros between 100 and 999999999 and a non-empty name,
else `ErrPermanent`. The name is there so `Get` and `Delete` can tell
our VM from another that later got the same VMID.
- **Create.** `placement` checks `Zone` (`node`; optional with one node),
`MachineType` and `Image` (both catalogue keys) before any request, and
logs the resolved parameters with `cloudInitBytes` at V(2). Then, under
the per-name lock, `create`: one `ClusterResources` read;
`adoptByName` (two VMs with the name are Permanent, one goes to
`inspect`); with none, `template` reads the template's config and
refuses a non-template, a cloud-init drive, a missing `net0` when the
config rewrites it, and an unsized boot disk when `diskGiB` is set, all
before any clone; `allocate` probes `vmidFor(uid, attempt)` up to 32
times, cloning onto a free VMID and running `inspect` on a taken one or
on the clone's `config file already exists` answer. `inspect` waits out
a lock (`waitUnlocked`; a clone in progress is also nameless), then
adopts our name with our mark and a matching or empty UID, probes past
another name, and is Permanent for our name without the mark, with
another UID, or hibernated. An adopted VM gets one `VMStatus`: running
returns at once. Otherwise `provision`: `uploadSeed` (rebuilt and
overwritten every attempt), `SetVMConfig` (`configParams`: cores,
memory, `agent=1`, the tag, `ide2`, and `net0` through `rewriteNet0`
when a bridge or VLAN is configured), the resize task when
`resizeTarget` says the boot disk is smaller than `diskGiB`, the start
task, and `waitRunning` (every second up to 60 s, Transient on expiry
or on a stop). `Create` returns only when the VM runs.
- **Get and state.** `get` checks in order: absent from the resources is
NotFound; a lock other than `suspended` is Transient (checked before the
name, because a mid-clone entry has none); `status: unknown` is
Transient; the config's `does not exist` is NotFound, a config lock
Transient; another name or no mark is NotFound; hibernated is Stopped
without further calls. Then `VMStatus` and `mapState(status,
qmpstatus, ip)`:
| `status` | `qmpstatus` | agent | Provider state |
|---|---|---|---|
| running | running | an IPv4 | Running |
| running | running | 200 without an IPv4, or `QEMU guest agent is not running` | Provisioning |
| running | running | `No QEMU guest agent configured` | Stopped |
| running | prelaunch | not asked | Provisioning |
| running | paused, suspended, io-error, anything else | not asked | Stopped |
| stopped, anything else | any | not asked | Stopped |
Any other agent answer, including `VM <vmid> is not running`, is
Transient. `UID` and `CreatedAt` are read from the description on every
call. A running `Get` is four calls: resources, config, status, agent.
- **Delete.** Under the per-name lock, `destroy`: `resolve`; a lock other
than `suspended` or `status: unknown` is Transient; the config's `does
not exist`, another name or no mark leave the VM alone; our VM
hibernated is destroyed like a stopped one, since the destroy takes it.
Otherwise stop (unless stopped), `deleteSeed` on the
VM's node, where a failure fails the call and leaves the VM for the
next Delete (a 403 for `Datastore.Allocate` is Permanent), then destroy
with `purge=1&destroy-unreferenced-disks=1`. Stop and destroy are tasks
waited on through `runTask`, where `does not exist` means gone. The
seed goes first because a VM already destroyed is NotFound to `Get`,
and nothing would call Delete again. With no VM of ours,
`deleteSeedEverywhere` tries every configured node, where a node that
fails is logged and skipped unless ctx ended or the failure is
Permanent. A missing volume still ends in a successful task, so no
message is matched. Nothing is left when nil is returned.
- **ListByTag.** One `ClusterResources` covers every node, unreachable
ones included (their guests are listed with `status: unknown`).
`isCandidate` keeps QEMU VMs that are not templates and carry the tag
`proxy_operator` or a name starting `proxy-`; a nameless entry (a clone
in progress) is not a candidate. One `VMConfig` per candidate; the mark
in the description decides, and the `Instance` is `ID`, `UID` and
`CreatedAt` only. A candidate whose config cannot be read is skipped at
V(1), unless ctx ended, which fails the call; the resources call failing
fails the whole call.
- **Errors.** `classify(op, err)`: an error already carrying a class keeps
it; then `apiError`s by `classifyAPI` (every 4xx Permanent; the 500
`does not exist` NotFound, Transient when `op == "create"`; every other
status Transient, including the lock, the taken VMID and pveproxy's 595
and 596); TLS verification failures (`tls.CertificateVerificationError`
and the three `x509` types) Permanent; everything else Transient. The
500 matchers (`isNotExist`, `isAlreadyExists`, the two agent ones)
are anchored to the 500 status and each is backed by a captured
response. `permanentf` and `transientf` classify at the
source: placement, the template checks, the ownership check, a
malformed token file or CA file (Permanent); a missing token or CA file,
an expired wait, a failed task (Transient). Nothing is Quota.
- **Check.** `Check` runs `checkNode` per node concurrently on the shared
client. `probeNode` reads the token file for `tokenID` (no request when
it is unusable), then `Version`, `NodeStatus`, `checkStorage` (a 500
whatever its text, inactive, or no `iso` content is `StorageNotFound`)
and `checkTemplates` (each catalogue VMID exists and is a template,
else `TemplateNotFound`). `checkNode` answers `Timeout` whenever the
round's ctx ended; otherwise `checkReason`: the two failures above,
`errCredentials` `CredentialsUnavailable`, `errCAFile` or a TLS
verification failure `CertificateInvalid`, 401 `Unauthorized`, 403
`Forbidden`, 595 and 596 `Unreachable`, any other status `APIError`, a
`net.OpError` `Unreachable` (before the deadline, so a dial out of its
own budget is a node that did not answer), a deadline or cancellation
`Timeout`, a cut connection `Unreachable`.
- **Connections.** One `http.Client`, built on first use once the CA
loads (`httpClient`), shared by every call and the Checker: dial and
TLS handshake 10 s each, `Client.Timeout` `requestTimeoutSeconds`, 4
idle connections per host kept 90 s, HTTP/2 disabled. A cancelled
request ends only itself. The token file is read by `readToken` on
every request.
- **Logging.** Logger from the context, named `proxmox`, with `provider`,
and `vmid`, `name` and `node` on lines about an instance. V(1): one line
per operation outcome (`proxmox vm ensured running`, `proxmox vm
fetched`, `proxmox vm deleted` with `destroyed`, `proxmox vms listed`,
`proxmox node checked`), one `proxmox api call` line per seam call from
`observe` (call, identifiers, UPID, duration, `httpStatus` and `error`),
`proxmox task failed`, and the no-op lines. V(2): request parameters,
config summaries, agent addresses, the resolved Create parameters.
Cloud-init, the token, the CA and the upload body are never logged.
- **Tracing.** No span of its own: `newHTTPAPI` wraps the transport in
`tracing.RestConfigWrapper(opts...)`, so each request is an
`HTTP <METHOD>` client span under the caller's span, and none without
one. `observe` only logs, so nothing is counted twice.
- **Tests.** Pure pieces (`id_test.go`, `mark_test.go`, `state_test.go`,
`seed_test.go`, `errors_test.go` with the classify table over the
captures, `api_test.go`); `client_test.go` (the real client over
`httptest.NewTLSServer`: the token never in errors or logs, x509
failures Permanent, cancellation, the child span); `create_test.go`,
`get_test.go`, `delete_test.go`, `list_test.go`, `task_test.go`,
`proxmox_test.go`, `check_test.go`, `observe_test.go` (every seam
method logs exactly one line, by reflection) and `provider_test.go`
(the contract §11 sweep, named test by test in the spec's entry 21).
The double is `fakeAPI` in `fake_test.go`: VMs, tasks and ISOs in
memory, a clone that leaves a nameless locked entry until its task
ends, per-method errors and blocking, a call log; its error texts are
checked against the captures. Real backend:
`proxmox_integration_test.go` behind `//go:build integration`, driven
by `PROXMOX_TEST_*` variables and `make test-integration-proxmox`,
described in
[testing.md](testing.md#the-proxmox-integration-test-real-proxmox-ve-host);
it passed against PVE 9.2.20 on 2026-09-24.
- **Deployment.** The optional `proxmox-token` Secret (keys `token` and
`ca.pem`) mounted at `/etc/proxy-operator/proxmox`
(`config/manager/manager.yaml`), `make proxmox-token-secret`, a
commented `proxmox-home` entry in both providers configs (loaded by
`TestLoadConfig_proxmoxSampleEntry`), `config/samples/proxy_proxmox.yaml`,
the `proxmox` type in the `status.type` comment of
`api/v1alpha1/providerhealth_types.go`. No RBAC, and no
`hack/providers-dev.yaml` entry, since it needs credentials.
- **Known gaps** (see also [proxmox-provider.md § Known gaps](proxmox-provider.md#known-gaps)).
- `Create` blocks until the VM runs.
- A clone lock that never clears makes every Create for that Proxy wait
out the 10-minute clone budget; `qm unlock` by hand.
- GC blind spot: a seed ISO whose VM was destroyed outside the operator
is invisible to GC, and a seed on node-local storage stays behind
when its VM migrated.
- The VMID range is shared with humans; a hand-made VM with a proxy's
name is Permanent.
- Cross-node clones (`target`) need shared storage and are untested.
- A hibernated VM of ours latches its Proxy when Create's adoption
finds it; a recorded one is replaced.
- The CA is cached after first use; the token is not.
## 9. Gaps shared by the existing providers
- **Adoption is not verified in three of the five.** kubernetes, gcp and
kvm adopt any existing instance with the right name without checking
`LabelUID`; kvm2 and proxmox return `ErrPermanent` when the UID differs.
- **The Create comment does not match kvm, kvm2 and proxmox.** The
interface comment in `internal/provider/provider.go` says `Create`
returns as soon as the request is submitted; the three VM providers on
your own hypervisors block until the VM has started.
- **`Stopped` and `Terminated` are produced inconsistently.** kubernetes
never produces Stopped, kvm never produces Terminated, and the reconciler
treats them the same.
never produces Stopped, kvm, kvm2 and proxmox never produce Terminated,
and the reconciler treats them the same.
- **Different `ListByTag` partial-failure policies.** gcp leaves failed
parts out, kvm fails the whole list. Both are safe.
parts out, kvm and kvm2 fail the whole list, and proxmox fails it when
its one cluster-wide listing fails. Both are safe.
- **Stale `mock` comments** in `internal/provider/provider.go`,
`internal/provider/registry/registry.go` and `api/v1alpha1/proxy_types.go`
(and so the generated CRD's `spec.provider` description).
@@ -351,33 +744,41 @@ the author's view.
is still open.
- **Tracing.** kvm has no child spans, so a kvm trace shows `provider.create`
and nothing under it; correlation works only through `traceID` on its V(1)
log lines. Only kubernetes is tested against a real collector, and that
test checks only the operation spans; nothing checks that gcp's SDK spans
or the kubernetes API spans nest under `provider.*`.
log lines. kvm2 has them, but only an in-memory recorder checks their
topology. Only kubernetes is tested against a real collector, and that
test checks only the operation spans; nothing checks that gcp's SDK spans,
the kubernetes API spans, kvm2's spans or proxmox's HTTP spans nest under
`provider.*` in a real collector.
## 8. Proposed features
## 10. Proposed features
Listed only. Nothing below is designed or implemented.
Listed only. Nothing below is designed or implemented, except where a
bullet says kvm2 already does it.
- **Child spans in kvm:** extend the `tracingAPI` decorator in
`kvm/trace.go`, which already wraps every libvirt call, to also start a
span per RPC, and add spans for the SSH dial and the image download. The
seam's methods take no `ctx` today (`hostPool.call` hands `fn` only the
API), so the span context has to be plumbed in.
- **Tempo e2e checks for child spans**, and for gcp and kvm through their
real backend runbooks.
API), so the span context has to be plumbed in. kvm2's `observe.go` over
its ctx-first seam is the shape this would take.
- **Tempo e2e checks for child spans**, and for gcp, kvm and kvm2 through
their real backend runbooks.
- **A shared conformance suite:** for example `internal/provider/providertest`,
which runs the contract §11 tests against any provider plus a fake
backend, so a new provider gets the contract tests for free.
- **Verify the UID when `Create` adopts an instance.** Return `ErrPermanent`
on a mismatch instead of adopting an instance that belongs to someone else.
kvm2 and proxmox do this; the other three do not.
- **Report asynchronous create failures:** gcp keeps the operation name, and
`Get` reports the failure instead of NotFound.
- **Capacity or quota signals** for kvm, such as pool free space and host
memory, mapped to `ErrQuotaExceeded`.
memory, mapped to `ErrQuotaExceeded`. kvm2 has the pool-space precheck;
host memory is still unchecked in both.
- **GC for secondary resources:** mark volumes and similar resources, and
let `ListByTag` or a second listing method cover them.
- **Reload credentials and config without a restart.**
- **Reload credentials and config without a restart.** kvm2 re-reads its
SSH material on every dial; the providers config is still read once per
process for every type.
- **Checker `Details`** for kubernetes (server version) and gcp (quota
headroom).
- **Internal-IP mode** for gcp, and a firewall rule managed per provider.
docs/providers.md
+6 -3
View File
@@ -13,11 +13,14 @@ the existing ones happen to work.
a new provider's design answers, one entry per decision the contract leaves
open. Copy it to `docs/<type>-provider-spec.md`, fill it in, and hand it to
the implementer with the other two files.
- **[providers-existing.md](providers-existing.md)**: how kubernetes, gcp and
kvm meet the contract, a table mapping each of them to every profile, their
per-provider reference notes, shared gaps and proposed features.
- **[providers-existing.md](providers-existing.md)**: how kubernetes, gcp,
kvm, kvm2 and proxmox meet the contract, a table mapping each of them to every
profile, their per-provider reference notes, shared gaps and proposed
features.
Per-provider user guides: [kvm-provider.md](kvm-provider.md),
[kvm2-provider.md](kvm2-provider.md),
[proxmox-provider.md](proxmox-provider.md),
[gcp-vm-validation.md](gcp-vm-validation.md) and
[gcp-in-specific-project.md](gcp-in-specific-project.md).
docs/proxmox-provider-spec.md
+961
View File
@@ -0,0 +1,961 @@
# Provider spec: proxmox
Filled from [provider-spec-template.md](provider-spec-template.md) against
[provider-contract.md](provider-contract.md). This is the design the
`proxmox` package is built from. Where an entry names an axis in backticks,
the answer is one of that axis's profiles in contract §4.
Backend facts come from the Phase 0 recon on the user's host, recorded in
[the execution log](plans-executions/2026-09-18-1502-proxmox-provider.md) as
facts F1 to F23, with the raw responses under
`internal/provider/proxmox/testdata/`. An Evidence line citing `F<n>` points
there. Where this spec and the plan
([docs/plans/2026-09-18-1502-proxmox-provider.md](plans/2026-09-18-1502-proxmox-provider.md))
disagree, this spec wins: it folds in the ten corrections Phase 0 found.
## Header
- **Type**: `proxmox`
- **Example configured name**: `proxmox-home`
- **Backend and API version this spec was written against**: Proxmox VE
9.2.20 (`pve-manager/9.2.20/49318c671b82f31e`, kernel `7.0.14-17-pve`,
QEMU 11.0.3) on the single node `pv1-1`, API at
`https://192.168.0.8:8006/api2/json`.
- **Client**: `thin-client`: `net/http` and `encoding/json` from the standard
library, with the transport wrapped by the already-required
`go.opentelemetry.io/contrib/instrumentation/net/http/otelhttp`. No new
dependency. The seed ISO uses `github.com/kdomanski/iso9660`, already
required by kvm2.
- **Author and date**: Claude (orchestrator session), 2026-09-23.
## 1. Identity and providerID
**Answer:** `vms/<vmid>/<name>`, where `<vmid>` is the VM's numeric id and
`<name>` is `req.Name`. The ID is location-free: the node is looked up in
`GET /cluster/resources?type=vm` on every call, so a VM migrated between
nodes stays addressable. Parsing is strict: exactly three segments, the
literal `vms`, a decimal VMID without leading zeros between 100 and
999999999, and a non-empty name without `/`. Anything else is `ErrPermanent`
from `Get` and `Delete`. The name is carried so that `Get` and `Delete` can
tell our VM from another VM that later received the same VMID (entry 2). No
config value is pasted into the ID, so no ID-safety regexp is needed; node
names and storage ids are still validated at load (entry 15) because they
end up in request paths.
**Evidence:** contract §6 and §4.10 (the ID must not depend on the location
when instances can migrate); F8 (`node` is a field of every resource entry).
## 2. Primary key (`primary-key`)
**Answer:** `allocated-id`. Proxmox addresses VMs by VMID; `name` is only a
label and is not unique. The VMID is derived from the Proxy UID so a retry
lands on the same number: `vmidFor(uid, attempt)` is
`min + (fnv1a64(uid) + attempt) mod (max - min + 1)` over `vmidRange`
(entry 15). Create, under the per-name lock (entry 19):
1. Reads `/cluster/resources` once. Entries with `type: qemu`, `template: 0`
and `name == req.Name` are the by-name candidates. Two or more is
`ErrPermanent` naming every VMID: an admin must remove the duplicate.
Exactly one goes to the ownership check below and is adopted or refused.
2. With none, probes `attempt = 0, 1, …` up to 32:
- VMID free in the listing: clone onto it. If the clone fails with
`unable to create VM <vmid>: config file already exists` (F4), someone
took it between the listing and the clone; run the ownership check on
it.
- VMID taken: run the ownership check.
- Exhausted: `ErrTransient` naming the range.
3. The **ownership check** on a VMID: if its resource entry carries a
`lock` or has no `name` (a clone in progress lists neither, F8), wait
until it is unlocked (entry 10, clone budget). Then read its config. It
is **ours** when `name == req.Name` and the description carries the mark
with `uid` equal to `req.UID` or empty: adopt it and resume Create at the
seed step (entry 10). It is **foreign** when the name differs: move to
the next attempt. The same name with no mark, or with a different
non-empty `uid`, is `ErrPermanent` naming the VMID and both UIDs; the
operator never takes over a VM it did not create. A `suspended` lock (a
VM hibernated to disk) is not waited on: it is not a task and never
clears by itself. Ownership is judged at once, as `Get` does (entry 9):
a foreign hibernated VM is probed past, and ours is `ErrPermanent`
(resume or destroy it), like kvm2's paused domain. Container and
template entries at a VMID are foreign without a config read, and so
is a VM whose listed name is not `req.Name`, whatever its lock: only a
clone in progress is nameless, and waiting on another VM's `backup`
lock would hold Create for the whole clone budget on every retry
(review finding P2).
The wait in step 3 is what closes the duplicate-VM bug Phase 0 found: after
a crash mid-clone, the retry's by-name lookup misses the nameless VM, but
attempt 0 lands on the same VMID, waits for the clone to finish, and adopts
it instead of probing past it.
`Get` and `Delete` check the name and mark at the VMID in the ID. A VM there
with another name means ours is gone and the VMID was reused: `Get` answers
`ErrNotFound`, `Delete` succeeds without touching it.
**Evidence:** F4 (taken VMID fails synchronously); F8 (a mid-clone entry has
no name); contract §4.10 (allocate-then-verify, deterministic from the UID)
and §13 (adoption not verified: closed here).
## 3. Placement mapping
**Answer:** three fields are catalogue keys checked before any backend call,
one is ignored:
| Field | Meaning | Required | Unknown value |
|---|---|---|---|
| `Zone` | a node from `nodes` | only when more than one node is configured; with one node an empty zone selects it | `ErrPermanent`: `placement.zone "x" is not in proxmox.nodes` |
| `MachineType` | a key of `machineTypes` (`cores`, `memoryMiB`, optional `diskGiB`) | yes | `ErrPermanent`: `placement.machineType "x" is not in proxmox.machineTypes` |
| `Image` | a key of `templates` (a template VMID) | yes | `ErrPermanent`: `placement.image "x" is not in proxmox.templates` |
| `Region` | ignored | | still part of the spec hash: changing it replaces the VM |
The guide documents the ignored field and that catalogue edits affect only
new VMs.
**Evidence:** contract §2.1 (reject early, MAY ignore with documentation).
## 4. Image source
**Answer:** a template VM prepared by the admin, referenced by a catalogue
key: `templates.<key>.vmid`. Create makes a **full clone** (`full=1`), so a
VM owns copies of its disks and keeps nothing linked to the template: the
template may be edited, replaced or deleted without affecting running VMs.
Before cloning, Create reads the template's config and refuses it with
`ErrPermanent` when it is not a template (`template` is not 1) or when any
disk slot holds a Proxmox cloud-init drive (a value containing `cloudinit`),
which would compete with the provider's seed (entry 5). Because the clone
copies the template's config, what the config or resize step would refuse
on the clone is refused on the template too, before any VM exists: no
`net0` while `bridge` or `vlanTag` is set, and no boot disk, or one
without a readable `size=`, while `diskGiB` is set. A Permanent error
after the clone would leave a VM that fixing the template cannot help. The template must
boot from its first disk, run cloud-init with the NoCloud datasource, and
either ship `qemu-guest-agent` or be able to install it (entry 7).
Clone parameters: `newid`, `name`, `description` (the mark, entry 8),
`full=1`, `storage` when configured (empty clones onto the template's own
disk storage), and `target` when the chosen node differs from the template's
node. A cross-node clone needs the template on shared storage; this host has
one node, so that path is untested (entry 24).
Catalogue edit warning for the guide: changing a key's `vmid` changes only
VMs created afterwards. Add a new key rather than repointing one in use, so
the Proxies that name it keep a meaning you can reason about.
**Evidence:** F19 (template 9001: `scsi0`, `boot: order=scsi0`, `ide2`
free, `agent: 1`; template 9000 carries a cloud-init drive on `ide2`); F15
(a clone onto a storage without image support is refused up front); F16 (a
full clone of 9001 took about two seconds).
## 5. Cloud-init delivery
**Answer:** a NoCloud seed ISO built in-process by the shared
`internal/provider/nocloud` package (extracted from kvm2 in Phase 3):
volume label `cidata`, files `user-data` (the Proxy's `CloudInit`, byte for
byte, never parsed), `meta-data` (`instance-id: <name>` and
`local-hostname: <name>`) and `vendor-data` (the guest agent script, joined
with the configured `vendorData` as a MIME multipart when both are set;
identical to kvm2's rules). The builder stages under `os.TempDir()` (`/tmp`,
the manager's emptyDir) and writes into a `bytes.Buffer`; a seed is tens of
KiB, so buffering is within contract §1.4.
The ISO is uploaded with `POST /nodes/{node}/storage/{isoStorage}/upload`
as multipart (`content=iso`, file part `filename` named
`<name>-seed.iso`) and the returned `imgcopy` task is waited on. An upload
**overwrites** an existing file of the same name, so no delete precedes it;
the seed is rebuilt and re-uploaded on every Create attempt, since a
previous attempt may have been cut off. It is attached as
`ide2: <isoStorage>:iso/<name>-seed.iso,media=cdrom`; Proxmox then appends
`ide2` to the boot order on its own, which is harmless because the boot
disk stays first.
Only the Proxmox API is used: the storage upload endpoint accepts `iso`,
`vztmpl` or `import`, never `snippets`, so `cicustom` would need shell
access to the node. Size: the request timeout (entry 18) bounds the upload;
a seed with a multi-megabyte `vendorData` would need a larger
`requestTimeoutSeconds`.
**Evidence:** F12 (upload fields, UPID, overwrite, `.iso` suffix required);
F19 (`ide2` appended to the boot order); F21 (the seed on `ide2` installed
the agent on both the untagged network and VLAN 4); F23 (the upload is
staged in `/var/tmp` on the node, not the tmpfs `/tmp`); contract §2.1 and
§11 (byte for byte, sentinel test).
## 6. Machine type mapping
**Answer:** a catalogue: `machineTypes.<key>` is `{cores, memoryMiB,
diskGiB}`. `cores` and `memoryMiB` are written in the config step (entry
10). `diskGiB` is optional: when set and larger than the cloned boot disk,
the disk is grown with `PUT /nodes/{node}/qemu/{vmid}/resize` to the
absolute size `<diskGiB>G`, and the returned task is waited on; when unset
or not larger, the disk keeps the template's size and no call is made. The
provider never shrinks a disk. Cloud-init's growpart expands the root
filesystem at first boot.
The boot disk is the first entry of the config's `boot: order=…` that is a
disk key (`scsiN`, `virtioN`, `sataN`, `ideN`) and not a CD-ROM, else the
`bootdisk` key, else the first present of `scsi0`, `virtio0`, `sata0`,
`ide0`. Its current size comes from the `size=` option of that disk's value.
No boot disk, when `diskGiB` is set, is `ErrPermanent`.
**Evidence:** F20 (resize is a task; the same size is an `OK` no-op; a
shrink fails in the task with `shrinking disks is not supported`); F19
(`boot: order=scsi0`, becoming `order=scsi0;ide2` once the seed is
attached).
## 7. Network and IP discovery (`readiness`, `ip-source`)
**Answer:** `readiness`: `provisioning-window`. `ip-source`: `discovered`,
from the QEMU guest agent only (`GET /nodes/{node}/qemu/{vmid}/agent/network-get-interfaces`);
the API offers no ARP or DHCP-lease source. The address reported is the first
IPv4 of the first non-loopback interface, in the agent's order, that is not
loopback, link-local or unspecified.
The NIC is the template's `net0`, rewritten in the config step only when the
config sets `bridge` or `vlanTag`: the value's comma-separated options are
kept in order with the model and MAC untouched, `bridge=` and `tag=` are
replaced or appended. With neither set, `net0` is left as cloned.
The agent answers:
| Agent response | Meaning | Get reports |
|---|---|---|
| 200 with a routable IPv4 | booted | Running with that IP |
| 200 without one | network still coming up | Provisioning |
| 500 `QEMU guest agent is not running` | booting, or the agent install has not finished or failed | Provisioning |
| 500 `VM <vmid> is not running` | the VM stopped between the status read and this call | `ErrTransient` |
| 500 `No QEMU guest agent configured` | `agent: 1` was removed from the config; this VM can never report an address | Stopped, so the reconciler replaces it with a VM that has the agent on |
| anything else | cannot tell | `ErrTransient` |
The guest needs the agent. The vendor-data script installs it at first boot
unless `installGuestAgent: false`, and that install needs a package mirror:
**the proxy network must have a route to the internet**, which an egress
proxy needs anyway. Baking `qemu-guest-agent` into the template and setting
`installGuestAgent: false` removes the mirror from IP discovery. `Port` is
not used by the provider. Reachability is a prerequisite in the guide: the
operator's pod network and the clients must route to the proxy network (on
the user's host, VLAN 4, `192.168.4.0/24`).
**Evidence:** F5 (all three agent messages); F21 (agent response shape; an
address about 50 seconds after start on the untagged network and on VLAN 4);
the Phase 0 log's VLAN 4 section (no internet route meant no agent);
contract §1.3 and §2.2 (Running only with an IP).
## 8. GC marks and CreatedAt (`gc-mark`, `created-at`)
**Answer:** `gc-mark`: `written-metadata`. `created-at`: `written-at-create`.
The mark is three lines in the VM's `description`, passed in the clone call
itself, so there is no mark window:
```
proxy-operator-managed=true
proxy-operator-uid=<req.UID>
proxy-operator-created-at=<RFC3339 UTC>
```
The keys are `provider.LabelManaged`, `provider.LabelUID` and
`proxy-operator-created-at`; the managed value is `provider.LabelManagedYes`.
The parser reads `key=value` lines, tolerates CRLF, blank lines and any
other lines an admin adds, and treats the VM as managed only when the
managed value is exactly `true`. A missing `uid` line gives an empty UID. A
`created-at` that does not parse gives the zero time, which GC treats as
old; that is accepted. The clone's own `meta: …,ctime=…` is not usable:
Proxmox copies it from the template.
The config step also sets the tag `proxy_operator`, for filtering in the
Proxmox UI and as a ListByTag candidate hint (entry 12). It is not the mark:
tags cannot be passed to the clone call. Hyphens are accepted in tags on
9.2.20, but the underscore form is kept because it is valid under every
tag rule Proxmox has had. Uppercase is folded to lowercase by Proxmox.
The seed ISO carries no mark; its name `<name>-seed.iso` identifies it
(entry 11).
**Evidence:** F10 (the description survives the clone verbatim, newlines
included); F11 (the clone's `meta` is the template's); F9 (tag rules);
contract §2.1 (mark in the create call) and §4.5.
## 9. State mapping (`provisioning-state`)
**Answer:** `provisioning-state`: `create-blocks`. Create returns only once
`status/current` reports the VM running (entry 10), so a VM the provider
cloned but never started is only ever seen by a Create retry, which adopts
and starts it; the reconciler has no ID for it to `Get`.
`Get` checks in this order; the first match decides:
| Check | Source | Result |
|---|---|---|
| VMID absent | `/cluster/resources` | `ErrNotFound` |
| `lock` set, other than `suspended` | resources entry (a clone in progress also has no name) | `ErrTransient` |
| `lock: suspended` (hibernated to disk) | resources entry | Stopped |
| `status: unknown` (node not reachable, or a VM seconds old the listing has not reported yet: the first `Get` after Create met this on pv1-1) | resources entry | `ErrTransient` |
| config `does not exist` | `GET …/config` | `ErrNotFound` (deleted between the two calls) |
| config `lock` set | config | `ErrTransient` |
| `name` differs from the ID's name | config | `ErrNotFound`, logged: the VMID was reused |
| no mark in the description | config | `ErrNotFound`, logged: not ours |
| `status: stopped` | `status/current` | Stopped |
| `status: running`, `qmpstatus: running` | `status/current` | the agent decides (entry 7) |
| `status: running`, `qmpstatus: prelaunch` | `status/current` | Provisioning |
| `status: running`, any other `qmpstatus` (`paused`, `suspended`, `io-error`, …) | `status/current` | Stopped |
| any other `status` | `status/current` | Stopped |
A `suspended` lock decides the state but not ownership: the config is still
read, and a foreign name or a missing mark is still `ErrNotFound`, so a
hibernated VM that reused our VMID is never reported as ours. Only
`status/current` and the agent are skipped for it.
`status/current` is read, not only the resources entry, because a paused VM
shows as plain `running` in `/cluster/resources`. `UID` and `CreatedAt` come
from the description on every call. A running `Get` makes four calls:
resources, config, status and agent.
**Evidence:** F8 (resources fields; mid-clone entries lack a name, carry
`lock: clone` and `status: unknown`); F14 (stopped and paused as seen by
`status/current` and by resources); F2 (the `does not exist` message);
contract §2.2 and §4.11.
## 10. Create: blocking and task tracking (`create-return`, `async-ops`)
**Answer:** `create-return`: `blocking`. `async-ops`: `task-handle` and
`backend-lock`. Clone, upload, resize, start, stop, destroy and volume
delete each return a UPID; the provider polls
`GET /nodes/{node}/tasks/{upid}/status` every second until `status` is
`stopped`, then reads `exitstatus`: `OK` is success, and so is
`WARNINGS: <n>`, which Proxmox itself counts as a finished task (logged at
V(1)); anything else is `ErrTransient` carrying that text. The node comes
from the UPID (`UPID:<node>:…`). The handle is held within the call only: Create blocks
until the VM runs, so no handle outlives it, and a failure surfaces as
Create's own error with the task's message. Locks: a VM locked by a task is
waited on in Create (the ownership check) and reported as `ErrTransient` by
`Get` and `Delete`. A start issued while a clone holds the lock queues
behind it rather than failing; the provider never relies on that, because it
waits for the clone task first.
Steps, each idempotent, in order, under the per-name lock (entry 19):
1. Validate placement (entry 3). No backend call yet.
2. `GET /cluster/resources` once, then look the name up (entry 2).
3. When the name finds nothing: find the template's node (a missing
template is `ErrPermanent`), read the template's config and refuse it as
entry 4 says, then allocate the VMID (entry 2). The template is checked
only here, because a VM that already exists no longer depends on it
(entry 4): a template deleted after its clone must not fail that
clone's adoption. An adopted VM, found by name or by the probe, gets one
`status/current` read. Running, it returns its ID at once. Stopped, it
continues at step 5.
4. Clone and wait (clone budget).
5. Build the seed and upload it to the VM's node, overwriting; wait
(upload budget).
6. `PUT /nodes/{node}/qemu/{vmid}/config` (synchronous) with `cores`,
`memory`, `agent=1`, `tags=proxy_operator`,
`ide2=<isoStorage>:iso/<name>-seed.iso,media=cdrom` and, when configured,
the rewritten `net0` (entry 7).
7. Resize when `diskGiB` is larger than the boot disk (entry 6); wait
(resize budget).
8. `POST …/status/start`; wait (start budget).
9. Poll `status/current` every second until `status: running` and
`qmpstatus: running`, up to the running budget; expiry is
`ErrTransient`, and so is a `stopped` status after the start.
10. Return `vms/<vmid>/<name>`.
What a retry finds:
| Previous attempt stopped after | The retry |
|---|---|
| nothing | allocates, clones |
| submitting the clone | finds the VMID locked and nameless, waits, adopts, continues at 5 |
| the clone | adopts the stopped VM, continues at 5 |
| any of steps 5 to 7 | adopts, repeats 5 to 7 (upload overwrites, config is a replace, resize is skipped once large enough) |
| the start | adopts the running VM, returns its ID |
A Proxmox `does not exist` inside Create (a VM or volume vanishing between
steps) is `ErrTransient`, never `ErrNotFound`. The log line for a Create
records `cloudInitBytes`, never the content.
**Evidence:** F3 (a start queues behind the clone lock); F7 (a config write
under the clone lock is refused with `VM is locked (clone)`); F16 (task
status fields); F20 (resize is a task); the Phase 0 log's VLAN 4 rerun
(clone to started VM in about five seconds); contract §2.1, §4.2, §4.12 and
§5 ("do not return NotFound from Create").
## 11. Delete and secondary resources (`delete-return`, `secondary-resources`)
**Answer:** `delete-return`: `waits`. `secondary-resources`: `explicit` for
the seed ISO; the VM's disks are removed by the destroy call itself, with
`purge=1&destroy-unreferenced-disks=1`. Under the per-name lock:
1. Parse the ID (`ErrPermanent` if malformed).
2. `/cluster/resources`. VMID absent, or held by a container: skip to
step 6, trying every configured node because the VM's node is no longer
known.
3. `lock` set, other than `suspended`, or `status: unknown`: `ErrTransient`.
A clone or another task still running finishes first; the reconciler
calls Delete again after `DeletionPoll`.
4. Read the config. `does not exist`: skip to step 6 on every node. A
different name, or no mark: the VM is not ours; log it at V(1) and skip
to step 6 on every node without touching it. Our VM with a `lock` other
than `suspended` (taken since the listing): `ErrTransient`. Our VM with
`lock: suspended` (hibernated) is destroyed like a stopped one: the
listing shows it `stopped`, so step 5 is skipped, and the destroy
accepts it and removes its suspend state volume with the disks
(`destroy_hibernated_response.json`, `task_status_destroy_hibernated.json`,
captured in Phase 10). Until then Delete refused it as `ErrPermanent`
on the belief that the destroy would. A hibernated VM that is not ours
is left alone like any other.
5. Status (from the listing) other than `stopped`: `POST …/status/stop`,
wait (stop budget). Stop works on a paused VM too, which the listing
shows as `running`: the task ends `OK` and the VM is `stopped`
(`stop_paused_response.json`, `task_status_stop_paused.json`,
`status_current_after_stop_paused.json`, captured in Phase 10).
`does not exist` means the VM is gone.
6. `DELETE /nodes/{node}/storage/{isoStorage}/content/<isoStorage>:iso/<name>-seed.iso`,
wait (seed-delete budget), on the VM's node when step 5 was reached on
a VM of ours, and on every configured node in config order otherwise
(then step 7 is skipped). A missing volume is still a successful task,
so no message is matched. The seed goes **before** the destroy: once
the VM is gone `Get` is `ErrNotFound`, the reconciler releases the
finalizer and GC no longer lists it, so a seed left by a failed step
after the destroy would never be deleted (review finding CR1). On the
VM's node a failure fails the call and leaves the VM stopped for the
next Delete. On the every-node path a node whose request or task fails
is logged at V(1) and skipped, unless ctx ended or the failure is
`ErrPermanent` (a 403 for `Datastore.Allocate`, which every node
would give), which fail the call, so the missing privilege is visible
on both paths (review finding P1).
7. `DELETE /nodes/{node}/qemu/{vmid}?purge=1&destroy-unreferenced-disks=1`,
wait (destroy budget). `does not exist` is success. The destroy leaves
the seed alone (Phase 0 deleted it after the destroy), which is why
step 6 exists. That Proxmox deletes an ISO a stopped VM's `ide2` still
references was not in the Phase 0 captures; the Phase 10 real-host run
proved it (imgdel `OK` on the stopped VM's seed, then the destroy).
8. Return nil.
Everything is gone when Delete returns nil, so `Get` afterwards is
`ErrNotFound`. A concurrent `Get` during a Delete keeps seeing the VM (as
Stopped once it is stopped) until the destroy task removes its config,
because destroying sets no lock and leaves the resource entry in place until
the end. A second Delete waits on the lock, then finds nothing.
GC blind spot: a VM destroyed outside the operator leaves its seed ISO, which
ListByTag cannot see. The guide says seeds are named `proxy-*-seed.iso` on
`isoStorage` and any without a matching VM may be removed.
**Evidence:** F2 (a second destroy gives `does not exist`); F13 (volume
delete is an `imgdel` task, needs `Datastore.Allocate`, and succeeds on a
missing volume); the Phase 0 capture of resources and config mid-destroy
(`cluster_resources_during_destroy.json`, `config_during_destroy.json`);
contract §2.3.
## 12. ListByTag scope and partial failure (`list-partial`)
**Answer:** `fail-whole` for the one listing, with single unreadable VMs
skipped. Scope: every VM the cluster knows, from one
`GET /cluster/resources?type=vm`, which covers every node, including
unreachable ones, whose guests Proxmox lists with `status: unknown` (entry
13). Candidates are entries
with `type: qemu`, `template: 0`, and either the tag `proxy_operator` (tags
split on `;`, `,` or space) or a name starting with `proxy-`. A nameless
entry (a clone in progress) is not a candidate: its Create is live and it is
younger than `MinAge`. For each candidate the config is read and the
description parsed; managed VMs become an `Instance` with `ID`, `UID` and
`CreatedAt` only, which is all GC reads (contract §2.4); no status or agent
call is made. A candidate whose config cannot be read (a node down, a VM
destroyed mid-sweep) is skipped with a V(1) line, unless ctx ended, which
fails the call. The resources call failing fails the whole call. Filtering
by the mark is client-side; the tag narrows the candidates. Cost: one
resources call plus one config read per candidate per sweep.
**Evidence:** F8 (resources fields, tags separated by `;`, nameless mid-clone
entries); F9 (tags case-folded); contract §2.4 and §4.7.
## 13. Error classification (`not-found`, `quota`)
**Answer:** `not-found`: `error-code`, the 500 message
`Configuration file 'nodes/<node>/qemu-server/<vmid>.conf' does not exist`,
plus `list-and-miss` on `/cluster/resources`. The listing can miss a VM only
when it is gone: it is built from the cluster filesystem, which holds every
guest's config cluster-wide, and Proxmox lists the guests of an unreachable
node with `status: unknown` rather than dropping them. That last point is
Proxmox's documented design, not something the single-node host could show;
Phase 10 or a second node verifies it. `quota`: `absent`. The thin `local-lvm` storage
overprovisions, so no out-of-space response could be produced; capacity
failures are `ErrTransient`, and the Checker reports free space in
`Details` instead (entry 16).
The client carries each failure as `apiError{Status, Message, Errors}`:
`Message` is the body's `message` with the trailing newline trimmed, or the
reason phrase when the body has none (a 401); `Errors` is the 400 body's
per-parameter map. One `classify(op string, err error)`:
| Backend response | Class | Why |
|---|---|---|
| 401 `Authentication failed!`, 401 `No ticket` | Permanent | the token or its secret is wrong; fixed in the Secret |
| 403 `Permission check failed (<path>, <privilege>)` | Permanent | the token lacks a privilege; the message names it |
| 400 `Parameter verification failed.` with `errors` | Permanent | the request is wrong; the message includes the field errors |
| 400 as above with `storage: No such storage.` | Permanent from a call; the Checker's `StorageNotFound` (entry 16) | the storage does not exist in the cluster |
| other 4xx | Permanent | a request Proxmox will never accept |
| 500 `Configuration file '…' does not exist` | NotFound (Transient when `op == "create"`) | the VM is gone |
| 500 `VM is locked (<lock>)` | Transient | a task holds the VM |
| 500 `unable to create VM <vmid>: config file already exists` | handled by the allocator (entry 2); Transient if it escapes | the VMID is taken |
| 500 `QEMU guest agent is not running`, `No QEMU guest agent configured`, `VM <vmid> is not running` | handled by `Get` (entry 7); Transient if they escape | agent state |
| any other 5xx, including pveproxy's 595 and 596 | Transient | unknown means retry |
| task `exitstatus` other than `OK` or `WARNINGS: <n>` | Transient, with the text | the task failed; the next attempt retries |
| TLS verification failure (`x509.UnknownAuthorityError`, `x509.HostnameError`, `x509.CertificateInvalidError`) | Permanent | wrong URL or CA file; the admin must act |
| token file missing (`fs.ErrNotExist`) | Transient | the Secret has not landed yet |
| token file present but not `<user>@<realm>!<id>=<secret>` | Permanent | wrong content in the Secret |
| `context.Canceled`, `context.DeadlineExceeded`, `net.OpError`, `io.EOF`, `io.ErrUnexpectedEOF` | Transient | network or cancellation |
| an error already wrapped with a class | that class | validation and builders classify at the source |
| anything else | Transient | contract default |
Every matched string is backed by a captured response under `testdata/`;
the table test loads them rather than repeating the text. Matching is on
fixed strings anchored to the 500 status: `does not exist`,
`config file already exists`, and the whole messages
`QEMU guest agent is not running` and `No QEMU guest agent configured`,
plus, on a 400, the `storage` field error `No such storage.` (whole; a
clone onto a storage without image support has another `storage` error).
`VM is locked` and the stopped VM's `VM <vmid> is not running` are not
matched by the provider: both are Transient through the default branch.
The tests keep a matcher for each, with the whole stopped-VM message
against `^VM [0-9]+ is not running$` (its tail is a suffix of the
agent-not-running answer), only to hold the fake's wording to the
captures (review finding CR5).
**Evidence:** F1 (error text in the reason phrase and the body; 401 has no
body); F2, F4, F5, F7 (the 500 messages); F6 (no out-of-space response
could be produced on thin storage); F22 (both 401 texts); fixtures
`err_forbidden.json`, `err_bad_param.json`, `err_tag_invalid.json`,
`err_upload_bad_name.json`, and from Phase 10 `storage_status_missing.json`
(a missing storage is a 400, not the 500 the spec had assumed); the Phase 0 TLS finding (`192.168.2.8` not in
the node certificate); contract §5.
## 14. Credentials and TLS (`credentials`)
**Answer:** `mounted-files`. The config names `tokenFile`, one line
`<user>@<realm>!<tokenid>=<secret>`, sent as
`Authorization: PVEAPIToken=<that line>`. It is expected in the optional
Secret `proxmox-token`, key `token`, mounted at
`/etc/proxy-operator/proxmox`. The file is read on **every request**, so
rotating the Secret takes effect without a restart. `New` stats it and logs
a warning when it is missing, but never fails on it and never dials.
TLS: `ca-file`. `caFile` names a PEM bundle, typically the node's
`/etc/pve/pve-root-ca.pem`, carried in the same Secret as key `ca.pem`. It
is loaded on first use and cached, so rotating the CA needs a restart; a
missing CA file is `ErrTransient` until it lands, one that does not parse is
`ErrPermanent`. Without `caFile` the system roots are used. The URL's host
must be a name or address in the node certificate's SANs: on the user's
host that is `192.168.0.8` or `pv1-1.home.lab`, not `192.168.2.8`.
`insecureSkipTLSVerify: true` disables verification and is logged as a
warning by `New`.
Minimum privileges, as verified on 9.2.20, for `proxy-operator@pve` with a
token without privilege separation:
- a role on `/` with `VM.Allocate VM.Clone VM.Audit VM.Config.CDROM
VM.Config.CPU VM.Config.Memory VM.Config.Disk VM.Config.Network
VM.Config.Options VM.PowerMgmt VM.GuestAgent.Audit Datastore.Audit
Datastore.AllocateSpace Datastore.AllocateTemplate Sys.Audit SDN.Use`;
- a role on `/storage/<isoStorage>` with `Datastore.Allocate
Datastore.AllocateTemplate Datastore.AllocateSpace Datastore.Audit`,
because deleting a volume needs `Datastore.Allocate` and a grant on a more
specific path replaces the inherited one. A dedicated ISO-only storage
keeps that privilege away from backups.
`VM.Monitor` no longer exists in PVE 9 and is not needed.
`VM.Config.Cloudinit` is not needed. `SDN.Use` on `/` covers a VLAN tag on
`vmbr0`.
Hygiene: the token never appears in logs, span attributes or error strings.
`apiError.Error()` carries status and message only, never headers or URLs.
Tested with a sentinel secret that a test server echoes in a 401 body.
**Evidence:** F17 (privileges exercised and the one missing); the Phase 0
log's seed storage section; F22 (auth failure texts); contract §7.
## 15. Config block
**Answer:** a new block `proxmox:` with the Go type `provider.ProxmoxConfig`
in `internal/provider/config_proxmox.go`:
| Field | Required | Default | Validation | Ends up in |
|---|---|---|---|---|
| `url` | yes | | `https` URL with a host and no path other than `/` | request base, `…/api2/json` appended |
| `tokenFile` | yes | | non-empty | `Authorization` header |
| `caFile` | | system roots | non-empty when set | TLS roots |
| `insecureSkipTLSVerify` | | false | | TLS, warning at `New` |
| `nodes` | yes, at least one | | unique, `^[a-z0-9]([a-z0-9-]*[a-z0-9])?$` | request paths, clone `target` |
| `isoStorage` | yes | | Proxmox storage id `^[a-zA-Z][a-zA-Z0-9_.-]*$` | upload and delete paths, `ide2` |
| `storage` | | the template's disk storage | storage id regexp when set | clone `storage` |
| `bridge` | | as cloned | non-empty when set | `net0` `bridge=` |
| `vlanTag` | | as cloned | 1 to 4094 when set | `net0` `tag=` |
| `vmidRange.min`, `vmidRange.max` | | 100000, 199999 | min at least 100, max at most 999999999, min below max | `vmidFor` |
| `templates.<key>.vmid` | at least one template | | at least 100 | clone source |
| `machineTypes.<key>` | at least one | | `cores` and `memoryMiB` above zero, `diskGiB` not negative | config step, resize |
| `requestTimeoutSeconds` | | 30 | not negative | HTTP client timeout |
| `vendorData` | | | | seed `vendor-data` |
| `installGuestAgent` | | true | | seed `vendor-data` |
`isoStorage` has no default on purpose: the token must be allowed to delete
volumes there (entry 14), so the choice must be explicit. `nodes` holds node
names as Proxmox knows them (`pv1-1`), not addresses; the one URL reaches the
whole cluster.
Wiring: `ProviderConfig.Proxmox *ProxmoxConfig` with tag
`proxmox,omitempty`; `setBlocks` reports `proxmox`; the allow-list in
`Config.validate` and the constructor map in `cmd/main.go` both gain
`proxmox` in the same commit; `validateProxmox(prefix, c)` in
`config_proxmox.go` with `config_proxmox_test.go`.
**Evidence:** contract §10; the Phase 0 decisions (dedicated seed storage,
`vlanTag`, URL at `192.168.0.8`); F13 (why `isoStorage` needs
`Datastore.Allocate`).
## 16. Checker
**Answer:** one target per `nodes` entry, `Name` the node, `Address` the
URL, checked concurrently, returned in config order. The check shares the
provider's HTTP client; each request is independent, so a check that runs
out of time cancels only its own requests. Per node:
1. `GET /version` (`pveVersion`).
2. `GET /nodes/{node}/status` (the node answers; `kernel`).
3. `GET /nodes/{node}/storage/{isoStorage}/status`: the storage exists, is
active, and lists `iso` in its content (`isoStorageAvailGiB`).
4. `GET /cluster/resources?type=vm`: every configured template exists and
is a template (`templates` as `<found>/<configured>`).
Reasons, shared first: `Reachable`; `Timeout` (the round's ctx ended,
whichever request was running); `Unreachable` (TCP connect or TLS handshake
failed for a network reason, or pveproxy answered 595 or 596);
`Unauthorized` (401); `Forbidden` (403, the message names the privilege);
`APIError` (any other failure). Provider-specific, each fixed differently by
an admin:
- `CertificateInvalid`: TLS verification failed, or `caFile` is missing or
holds no certificate. Fix `url` or `caFile`, not the host.
- `CredentialsUnavailable`: the token file is missing or malformed. Fix the
Secret, not Proxmox.
- `StorageNotFound`: `isoStorage` is missing, inactive, or has no `iso`
content. Fix the storage on the node. A storage the cluster does not
know answers `400 Parameter verification failed.` with the field error
`storage: No such storage.` (`storage_status_missing.json`, captured in
Phase 10); that exact field error is matched, and the message carries
it. Any 500 from the storage status call counts as missing too,
whatever its wording: Proxmox knows the storage but cannot use it on
that node (not enabled there, or offline). That answer was not
captured, so no text is matched. Any other 4xx, and a 401, 403, 595 or
596, keeps its shared reason.
- `TemplateNotFound`: a configured template VMID does not exist or is not a
template. Fix the template or the catalogue.
Details: `pveVersion`, `kernel`, `tokenID` (the part before `=`, never the
secret), `isoStorage`, `isoStorageAvailGiB`, `templates`, filled as far as
the check got; `isoStorage` is always there. `templates` counts catalogue
entries. The Checker reads `tokenFile` itself, before the first request,
with the client's `readToken`: the seam stays one method per endpoint, and
a missing or malformed file answers `CredentialsUnavailable` without a
request. Only the part before `=` is kept.
`Timeout` also covers a request that runs out of its own timeout while the
round is still running. A dial or a connection cut mid-response is
`Unreachable`, even when the dial ran out of its own 10 s budget, because
the node did not answer. Cost: four requests per node per round. The `status.type`
comment in `api/v1alpha1/providerhealth_types.go` gains `proxmox` and the
CRD is regenerated.
**Evidence:** contract §8; fixtures `version.json`, `node_status.json`,
`storage_status.json`, `cluster_resources.json`.
## 17. Client and child spans (`client-kind`, `child-spans`)
**Answer:** `client-kind`: `thin-client`. `child-spans`: `transport-wrapper`.
The seam is the interface `api`, one method per endpoint, `ctx` first:
| Method | HTTP | Returns |
|---|---|---|
| `Version` | `GET /version` | version info |
| `NodeStatus` | `GET /nodes/{node}/status` | node status |
| `StorageStatus` | `GET /nodes/{node}/storage/{storage}/status` | storage status |
| `ClusterResources` | `GET /cluster/resources?type=vm` | entries |
| `VMConfig` | `GET /nodes/{node}/qemu/{vmid}/config` | config as `map[string]string` |
| `SetVMConfig` | `PUT /nodes/{node}/qemu/{vmid}/config` | nothing (synchronous) |
| `Clone` | `POST /nodes/{node}/qemu/{vmid}/clone` | UPID |
| `Resize` | `PUT /nodes/{node}/qemu/{vmid}/resize` | UPID |
| `Start` | `POST /nodes/{node}/qemu/{vmid}/status/start` | UPID |
| `Stop` | `POST /nodes/{node}/qemu/{vmid}/status/stop` | UPID |
| `VMStatus` | `GET /nodes/{node}/qemu/{vmid}/status/current` | status and qmpstatus |
| `Destroy` | `DELETE /nodes/{node}/qemu/{vmid}?purge=1&destroy-unreferenced-disks=1` | UPID |
| `TaskStatus` | `GET /nodes/{node}/tasks/{upid}/status` | status and exitstatus |
| `AgentNetworkInterfaces` | `GET /nodes/{node}/qemu/{vmid}/agent/network-get-interfaces` | interfaces |
| `UploadISO` | `POST /nodes/{node}/storage/{storage}/upload` (multipart) | UPID |
| `DeleteVolume` | `DELETE /nodes/{node}/storage/{storage}/content/{volid}` | UPID |
`httpAPI` is the production adapter; `fakeAPI` in `fake_test.go` keeps VMs,
tasks and ISOs in memory, simulates locks (a clone leaves a nameless locked
entry until its task completes), task polls, per-method errors and blocking,
and records a call log. Scalars in a config are rendered as strings by the
adapter (`cores` arrives as a JSON number, `memory` as a string).
The transport is `tracing.RestConfigWrapper(opts...)(base)`, a parent-gated
`otelhttp` transport, so each request is an `HTTP <METHOD>` span under
`provider.create`, `provider.get`, `provider.delete`, `provider.list` or the
monitor's `providerhealth.check`, and background traffic makes no root
spans. The `observed` decorator over `api` logs one line per call and starts
no span, so nothing is counted twice. Span attributes are otelhttp's
standard ones; the token is in a header, which otelhttp does not record.
Tests: a span recorder asserts the HTTP spans are children of the operation
span and carry no token; a reflection test asserts every `api` method logs
exactly one line.
**Evidence:** contract §4.15, §4.18, §9; `RestConfigWrapper` and
`parentGatedTransport` in `internal/tracing/transport.go`; F18 (HTTP/1.1
only).
## 18. Timeouts (`timeouts`)
**Answer:** `per-call`.
| Bound | Budget |
|---|---|
| TCP dial | 10 s |
| TLS handshake | 10 s |
| one request, response included (`http.Client.Timeout`) | `requestTimeoutSeconds`, default 30 s |
| clone task, and the wait for a VM's lock during the ownership check | 10 m |
| upload task | 2 m |
| resize task | 2 m |
| start task | 2 m |
| stop task | 2 m |
| destroy task | 5 m |
| seed delete task | 1 m |
| wait for `running` after the start | 60 s |
| task and status poll interval | 1 s |
The budgets live in one `budgets` struct that tests shrink. A task wait that
runs out is `ErrTransient` naming the UPID, and the task keeps running on
Proxmox; the next attempt finds its result through the ownership check or
the lock. A cancelled `ctx` cancels only that request: the transport keeps
its idle connections (`MaxIdleConnsPerHost` 4, `IdleConnTimeout` 90 s), and
no other call is disturbed. HTTP/2 is disabled explicitly
(`ForceAttemptHTTP2: false`, an empty `TLSNextProto`).
**Evidence:** contract §1.5 and §4.16; F16 (a full clone of the 3 GiB
template took about two seconds; tasks finish within a second or two); F21
(about 50 s from start to an address, which the reconciler's polling covers,
not the running wait).
## 19. Concurrency
**Answer:** shared state is the HTTP client (safe for concurrent use), the
lazily loaded CA pool (loaded once under a `sync.Once`-style guard), and a
lock per instance name from the shared `internal/provider/keyedlock` package
(extracted from kvm2 in Phase 3): a one-slot semaphore per key, waitable
with ctx, removed from the map once unused. Create and Delete for one name
run under that name's lock, so the reconciler's Delete and GC's Delete never
race on stop and destroy, and a Delete cannot interleave with a Create's
steps. Get and ListByTag take no lock. Two Creates for different Proxies
that land on the same VMID are safe: one clone wins, the other gets the
`config file already exists` answer, waits in the ownership check for the
winner's clone to finish, sees a different name, and probes on. Three
reconciles, a GC sweep and a check round at once are safe.
**Evidence:** contract §1.2 (concurrency); F4; entry 2.
## 20. Logging (`logging`)
**Answer:** `tiered`. Logger from `logf.FromContext(ctx)` on every call,
named `proxmox`, with `provider` on every line and `vmid`, `name` and
`node` on every line about an instance.
- V(1): one line per operation outcome (`proxmox vm ensured running`,
`proxmox vm fetched` with state, ip and uid, `proxmox vm deleted` with
`destroyed` (false when no VM of ours was left to destroy),
`proxmox vms listed` with counts, `proxmox node checked` with the reason),
one `proxmox api call` line per seam call from the decorator (`call`,
`node`, `vmid`, `storage`, `upid`, `duration`, plus `newid` and `name`
on a clone, `filename` on an upload and `volid` on a volume delete, and
on failure `httpStatus` and `error`), a `proxmox task failed` line with the UPID and
`exitstatus`, and the no-op lines of Delete and ListByTag (VM already
gone, foreign or unmarked VM left alone, seed delete skipped on a node,
unreadable candidate skipped).
- V(2): request parameters (without `description`'s content at V(1) and
never the upload body), config summaries, agent addresses, the resolved
Create parameters with `cloudInitBytes`.
- No V(5) wire tier.
- Never logged at any level: `CloudInit`, the token, the CA content, the
upload body.
**Evidence:** contract §4.17 and §9.
## 21. Tests (`backend-test`)
**Answer:** `backend-test`: `integration-tag`. Unit tests run over `fakeAPI`
behind the seam, the HTTP client over `httptest.NewTLSServer` serving the
captured fixtures, and `classify` over the fixtures directly. The
integration test is `proxmox_integration_test.go` behind
`//go:build integration`, skipped unless `PROXMOX_TEST_URL`,
`PROXMOX_TEST_TOKEN_FILE`, `PROXMOX_TEST_NODE`,
`PROXMOX_TEST_TEMPLATE_VMID` and `PROXMOX_TEST_ISO_STORAGE` are set;
optional `PROXMOX_TEST_CA_FILE`, `PROXMOX_TEST_INSECURE`,
`PROXMOX_TEST_STORAGE`, `PROXMOX_TEST_BRIDGE`, `PROXMOX_TEST_VLAN_TAG`,
`PROXMOX_TEST_VMID_MIN`, `PROXMOX_TEST_VMID_MAX`, `PROXMOX_TEST_TIMEOUT`
(default 20m), `PROXMOX_TEST_KEEP`, `PROXMOX_TEST_LOG_LEVEL`; run with
`make test-integration-proxmox`. It asserts, in order: `New` returns without
a request; `Check` reports the node Ready with details; `Create` returns
`vms/<vmid>/<name>` with the VMID in range; a second `Create` returns the
same ID and the VM count is unchanged; `Get` reaches Running with an IPv4
within the budget and its `UID` and `CreatedAt` equal what Create wrote;
`ListByTag` contains it; `Delete` returns nil; `Get` is `ErrNotFound`;
`Delete` again returns nil; no VM with the name and no `<name>-seed.iso`
remain; an HTTP span is a child of `provider.create` and no span attribute
carries the cloud-init text.
Contract §11 list, with the test for each item. Every name below exists
(Phase 8 checked each with a grep for `func <name>(`). The two integration
tests passed on pv1-1 on 2026-09-24 (timings in
[testing.md](testing.md#the-proxmox-integration-test-real-proxmox-ve-host)).
- [x] repeat `Create`: `TestCreate_repeatReturnsSameIDAndCreatesNothing`
- [x] `Get` after `Delete`: `TestGet_afterDeleteReturnsNotFound`
- [x] `Delete` twice: `TestDelete_twiceSucceeds`
- [x] malformed ID: `TestParseProviderID_malformed`, `TestGet_malformedIDIsPermanent`, `TestDelete_malformedIDIsPermanent`
- [x] unknown state maps to `StateStopped`: `TestMapState_table`, `TestGet_unknownStateIsStopped`
- [x] `StateRunning` requires an IP: `TestGet_runningWithoutIPIsProvisioning`, `TestGet_runningWithIPIsRunning`
- [x] `UID` and `CreatedAt` read back: `TestGet_readsUIDAndCreatedAtFromBackend`
- [x] `ListByTag` excludes unmarked instances: `TestListByTag_excludesUnmarkedVMs`, `TestListByTag_emptyUIDStillListed`
- [x] `Get` during an in-progress delete: `TestGet_duringDeleteStillReturnsInstance`
- [x] `Get` during an asynchronous operation or lock: `TestGet_lockedIsTransientNeverNotFound`, `TestGet_midCloneNamelessEntryIsTransient`
- [x] never started is never `StateStopped`: `TestCreate_returnsOnlyWhenRunning`
- [x] cancelled `ctx` returns promptly: `TestProvider_cancelledContextReturnsPromptly` (every method and `Check`, blocked on the first seam call, plus Create and Delete blocked mid-operation), `TestClient_cancelledContextReturnsPromptly`
- [x] `classify` table with captured responses: `TestClassify_table`
- [x] Checker reasons, expired `ctx`, stable order: `TestCheck_reasonsTable`, `TestCheck_expiredContextReturnsPromptly`, `TestCheck_oneEntryPerNodeInConfigOrder`
- [x] sentinel tests for `CloudInit` and the credential: `TestCreate_cloudInitSentinelNeverLoggedOrInErrors`, `TestClient_tokenNeverInErrorsOrLogs`, and `TestLogging_v0SilentV1OutcomeV2Detail` over a whole lifecycle at every level
- [x] child spans under the operation span: `TestTracing_httpSpansUnderOperationSpan`, `TestClient_childSpanUnderParent`
- [x] real-backend lifecycle test: `TestProxmox_lifecycle` and `TestProxmoxCheck_realHost` in `proxmox_integration_test.go`, passed on pv1-1
Tests that cover the Phase 0 corrections specifically, in addition to the
plan's per-phase lists:
- `TestCreate_resumesMidCloneVMAtDeterministicVMID`: a crash after the clone
was submitted; the retry waits for the lock and adopts, creating nothing.
- `TestCreate_takenVMIDInspectedBeforeProbing`: a foreign VM at attempt 0 is
skipped; ours at attempt 0 is adopted.
- `TestCreate_cloneAlreadyExistsRaceRunsOwnershipCheck`.
- `TestCreate_templateWithCloudInitDriveIsPermanent`.
- `TestCreate_seedUploadOverwritesWithoutDelete`.
- `TestCreate_resizeWaitsForTask`, `TestCreate_resizeSkippedWhenNotLarger`.
- `TestCreate_vlanTagAppendedKeepsMACAndOptions`.
- `TestBootDisk_table`, with a row for `order=scsi0;ide2`.
- `TestGet_pausedIsStoppedFromStatusCurrent`.
- `TestGet_agentMessagesTable` (the three messages and a 200 without IPv4).
- `TestDelete_seedDeleteIsTaskAndIdempotent`.
- `TestClient_x509FailureIsPermanent`.
**Evidence:** contract §11; docs/testing.md conventions.
## 22. Deployment
**Answer:** a new optional Secret and its mount, two Makefile targets, and
samples:
- `config/manager/manager.yaml`: an optional Secret `proxmox-token` mounted
read-only at `/etc/proxy-operator/proxmox`, keys `token` and `ca.pem`.
`/tmp` is already an emptyDir.
- Makefile: `proxmox-token-secret`, creating or updating `proxmox-token`
from `PROXMOX_TOKEN_FILE` (default `$(HOME)/.config/proxmox-token`) and
`PROXMOX_CA_FILE` (default `$(HOME)/.config/proxmox-ca.pem`), shaped like
`kvm-ssh-secret`; and `test-integration-proxmox` (`go test -race
-tags=integration -count=1 -v -timeout 30m -run TestProxmox
./internal/provider/proxmox/`).
- `config/manager/providers_config.yaml` and
`config/samples/providers-config.yaml`: a commented `proxmox-home` entry
for the user's host: `url: https://192.168.0.8:8006`, the two file paths,
`nodes: [pv1-1]`, `isoStorage: proxy-seeds`, `vlanTag: 4`,
`templates: {debian-13: {vmid: 9001}}`, a `small` machine type.
- `config/samples/proxy_proxmox.yaml`, listed in
`config/samples/kustomization.yaml`: provider `proxmox-home`, zone
`pv1-1`, machine type `small`, image `debian-13`, the Squid cloud-init
from `proxy_kvm2.yaml`.
- RBAC: none; the provider does not talk to the cluster's API.
- `hack/providers-dev.yaml`: no entry (needs credentials).
**Evidence:** contract §12.
## 23. Docs
**Answer:** `docs/proxmox-provider.md` with sections: Proxmox prerequisites
(the template recipe and what disqualifies a template, the dedicated seed
storage, the user, both roles and the token, the proxy network with DHCP and
internet, reserving the VMID range), Cluster-side setup (the Secret, the
config entry, the URL-must-match-the-certificate rule), Config reference
(entry 15), Placement (Region ignored, catalogue-edit warning), What lands on
Proxmox (the VMID scheme, the description mark, the tag, the seed ISO, the
config keys written, `qm unlock` for a stuck clone), Logging, Tracing,
Timeouts, Health check (entry 16), Error classes (entry 13), Known gaps
(entry 24), Testing. A fifth column in the three tables of
`docs/providers-existing.md` and a proxmox section. `docs/architecture.md`:
components row, a `proxmox` block in §5, the span note in §10, a §11 row and
the provider count, Decisions bullets (VMID from the UID hash with the
ownership check, the mark in the description, the seed by API upload, the
node resolved per call). A README row. `docs/providers.md` link.
`docs/testing.md` rows and a proxmox integration section. `CLAUDE.md`
Gotchas: `make test-integration-proxmox`. A `CHANGELOG.md` entry after the
user confirms.
**Evidence:** contract §12.
## 24. Known limits accepted at design time
**Answer:** ten limits, each with its consequence and workaround:
- **Create blocks** until the VM runs, holding one of three reconcile
workers. On the user's host that is about five seconds plus the clone;
large templates on slow storage take longer.
- **A clone lock that never clears** (a node crash mid-clone) makes every
Create for that Proxy wait out the clone budget and return
`ErrTransient`. Clear it with `qm unlock <vmid>` or destroy the VM.
- **Seed ISOs of VMs destroyed outside the operator** are invisible to GC.
Remove `proxy-*-seed.iso` files without a matching VM by hand.
- **The VMID range is shared with humans.** A VM created by hand inside it
costs a probe; a hand-made VM named like a proxy is refused as
`ErrPermanent`. Reserve the range.
- **Cross-node clones need shared storage** and are untested, because the
host has one node.
- **A VM whose agent was switched off is replaced**, not repaired (entry 7).
- **The proxy network needs internet** for the agent install, unless the
template ships the agent.
- **The CA is cached** after first use; rotating it needs a restart. The
token rotates freely.
- **A node that stops answering** makes `Get` and `Delete` for its VMs
`ErrTransient` until it returns, so deleting such a Proxy waits.
- Contract §13 limits that apply unchanged: config read once per process,
no installation identity in GC (two operators sharing a cluster and a
VMID range delete each other's VMs), an unconfigured provider latches the
Proxy.
Accepted after the Phase 10 review
([docs/reviews/2026-09-23-proxmox-provider-review/](reviews/2026-09-23-proxmox-provider-review/REPORT.md),
findings P3 and P4), and listed in the guide's Known gaps:
- **A Create that failed permanently after its clone keeps its VM**, and
the next Create adopts it by name even when `placement` has changed.
Destroy the VM before changing `placement` on such a Proxy.
- **A retry can clone a second VM** when a Create is cut off while
cloning onto a later probe VMID, the VM on the earlier one is removed,
and the retry comes before the clone ends. The next Create names both
VMIDs as `ErrPermanent`.
Changed by the Phase 10 captures of the three answers Phase 0 never saw:
- **A hibernated VM of ours** latches only Create's adoption
(`ErrPermanent`, resume or destroy it; entry 2). `Get` reports it
Stopped, so the reconciler replaces it, and Delete destroys it, since
the destroy takes a hibernated VM (entry 11). Before the capture, Delete
refused it as well.
- **A missing `isoStorage`** is a 400 with `storage: No such storage.`,
which the Checker reports as `StorageNotFound` (entry 16); a missing
storage at upload time is Permanent like any 4xx.
- **A paused VM** stops like a running one (entry 11, step 5), as assumed.
**Evidence:** contract §13; entries 2, 7, 10, 11, 14 above.
docs/proxmox-provider.md
+667
View File
@@ -0,0 +1,667 @@
# The proxmox provider
Runs Managed proxies as VMs on a Proxmox VE cluster. It is the third
VM-style provider and the second built from
[provider-contract.md](provider-contract.md), from the design in
[proxmox-provider-spec.md](proxmox-provider-spec.md). The operator talks to
the cluster's REST API only (`https://<host>:8006/api2/json`) with an API
token. It needs no SSH, no shell on the nodes, and no library beyond Go's
standard `net/http`.
Each VM is a **full clone of a template** you prepare once. The operator
builds the Proxy's cloud-init into a NoCloud seed ISO in-process, uploads
it to a storage on the VM's node, attaches it as a CD-ROM, sets the size,
starts the VM and learns its address from the QEMU guest agent. Everything
here was proven against **Proxmox VE 9.2.20** (QEMU 11.0) on the single
node `pv1-1`; every error text the provider matches was captured from that
host.
## Proxmox prerequisites
Once per cluster, as root on a node:
1. **Proxmox VE 9.x with the API reachable** from the operator's pods on
port 8006. The operator's pods and the crawler clients must also route to
the network the proxies land on (see step 5).
2. **A template without a Proxmox cloud-init drive.** The provider brings
its own seed on `ide2`; a Proxmox cloud-init drive would compete with it
and the Proxy's user-data would be ignored. Debian 13's generic cloud
image boots UEFI (under BIOS it hangs in GRUB on a headless VM), so the
template does too:
```sh
wget https://cloud.debian.org/images/cloud/trixie/latest/debian-13-generic-amd64.qcow2
qm create 9001 --name debian-13-template --memory 2048 --cores 2 --ostype l26 \
--machine q35 --bios ovmf --efidisk0 local-lvm:1,efitype=4m,pre-enrolled-keys=0 \
--scsihw virtio-scsi-pci --net0 virtio,bridge=vmbr0 \
--serial0 socket --vga serial0 --agent 1
qm set 9001 --scsi0 local-lvm:0,import-from=/root/debian-13-generic-amd64.qcow2
qm set 9001 --boot order=scsi0
qm template 9001
```
The template must boot from its first disk, run cloud-init with the
NoCloud datasource (the Debian and Ubuntu cloud images do), and either
ship `qemu-guest-agent` or be able to install it at first boot. Before
the first clone for a Proxy, `Create` reads the template's config and
refuses it with a permanent error when:
- it is not a template (`qm template` was not run);
- any disk slot holds a Proxmox cloud-init drive (a value containing
`cloudinit`), such as the one `qm set <vmid> --ide2 local-lvm:cloudinit`
adds for Proxmox's own cloud-init support;
- it has no `net0` while the config sets `bridge` or `vlanTag`;
- the machine type sets `diskGiB` and the template has no boot disk, or
one without a readable `size=`.
These are checked on the template because the clone copies its config:
a VM cloned from a bad template could not be fixed by fixing the
template. The template is checked only when a VM is about to be cloned;
a VM that already exists no longer depends on it. The health check sees
only whether each template exists and is a template.
3. **A storage for the seed ISOs**, holding `iso` content. A dedicated
directory storage is best, because the token needs `Datastore.Allocate`
on it to delete seeds, and on a dedicated storage that privilege cannot
touch backups or other ISOs:
```sh
pvesm add dir proxy-seeds --path /var/lib/proxy-seeds --content iso
```
A seed is tens of KiB. Proxmox stages uploads under `/var/tmp`, not the
node's tmpfs `/tmp`. On a cluster, a storage that is not shared keeps
each seed on the node the VM was created on (see *Known gaps*).
4. **A user, two roles and a token.** `VM.Monitor` no longer exists in PVE
9 and `pveum` rejects it; the agent read needs `VM.GuestAgent.Audit`
alone. `VM.Config.Cloudinit` is not needed for the seed on `ide2`.
`SDN.Use` on `/` is what lets the config write put a VLAN tag on
`vmbr0`:
```sh
pveum user add proxy-operator@pve --comment "egress-proxies-operator"
pveum role add ProxyOperator -privs "VM.Allocate VM.Clone VM.Audit VM.Config.CDROM VM.Config.CPU VM.Config.Memory VM.Config.Disk VM.Config.Network VM.Config.Options VM.PowerMgmt VM.GuestAgent.Audit Datastore.Audit Datastore.AllocateSpace Datastore.AllocateTemplate Sys.Audit SDN.Use"
pveum aclmod / -user proxy-operator@pve -role ProxyOperator
pveum role add ProxyOperatorSeeds -privs "Datastore.Allocate Datastore.AllocateTemplate Datastore.AllocateSpace Datastore.Audit"
pveum aclmod /storage/proxy-seeds -user proxy-operator@pve -role ProxyOperatorSeeds
pveum user token add proxy-operator@pve operator --privsep 0
```
The second role repeats three privileges of the first on purpose: an
ACL on a more specific path **replaces** what the user inherits from
`/` for that path, so a role with `Datastore.Allocate` alone would take
the upload's `Datastore.AllocateTemplate` away. The role on `/` covers
clone, config, resize, start, stop, the agent read, the upload and the
destroy; what it lacks is `Datastore.Allocate`, without which deleting
a seed is a 403 (`Permission check failed (/storage/<id>,
Datastore.Allocate)`). With `--privsep 0` the token carries the user's
privileges.
The token's secret is printed once. Save it on your workstation as one
line, `proxy-operator@pve!operator=<secret>`, in
`~/.config/proxmox-token` with mode 0600. Copy the node's CA,
`/etc/pve/pve-root-ca.pem`, to `~/.config/proxmox-ca.pem`.
5. **A proxy network with DHCP and a route to the internet.** Each VM
keeps the template's one NIC, optionally moved to another bridge or
VLAN by the config, and takes its address from that network's DHCP. The
network needs internet twice over: the vendor-data script installs
`qemu-guest-agent` from the distro mirrors at first boot, and the agent
is the only way the operator learns the VM's address; and an egress
proxy needs egress anyway. Baking `qemu-guest-agent` into the template
and setting `installGuestAgent: false` removes the mirror from IP
discovery. The operator's pods and the clients must route to that
network: the health engine probes `http://<ip>:<port>` through the
proxy, and clients get the same address from the discovery API. The
provider does not use `spec.port` and configures no Proxmox firewall.
6. **A reserved VMID range.** VMIDs are allocated from `vmidRange`
(default 100000 to 199999). Keep hand-made VMs out of it: one there
costs a probe, and one named like a proxy is refused (see *What lands
on Proxmox*).
Check from the operator's point of view before wiring it in. This sends
the token through a file descriptor, so it never reaches the command line:
```sh
pve() {
curl -s --cacert ~/.config/proxmox-ca.pem \
-H @<(printf 'Authorization: PVEAPIToken=%s\n' "$(tr -d '[:space:]' < ~/.config/proxmox-token)") \
"https://192.168.0.8:8006/api2/json/$1"; echo
}
pve version # the token works, TLS verifies
pve nodes/pv1-1/storage/proxy-seeds/status # active, content includes iso
pve 'cluster/resources?type=vm' # the template is listed with "template":1
```
## Cluster-side setup
1. Create the Secret. It holds two keys, `token` (the one-line token file)
and `ca.pem` (the node's CA):
```sh
make proxmox-token-secret
# or by hand:
kubectl -n egress-proxies-operator-system create secret generic proxmox-token \
--from-file=token=$HOME/.config/proxmox-token \
--from-file=ca.pem=$HOME/.config/proxmox-ca.pem
```
The make target runs that command as an apply, so it also updates an
existing Secret. Override the paths with
`PROXMOX_TOKEN_FILE=... PROXMOX_CA_FILE=...`. Before it applies, it
checks that both files exist, that the token file is exactly one line
of `<user>@<realm>!<tokenid>=<secret>` (without printing it), and that
the CA file holds a PEM certificate. With `insecureSkipTLSVerify`
instead of a CA, create the Secret by hand with the `token` key only.
`config/manager/manager.yaml` mounts the Secret (optional, read-only)
at `/etc/proxy-operator/proxmox`, so the files are
`/etc/proxy-operator/proxmox/token` and
`/etc/proxy-operator/proxmox/ca.pem`. The deployment starts fine
without it, even with a proxmox provider configured: `New` makes no
request, only checks that the token file exists, and logs a `WARNING:`
line when it does not. The provider's proxies then fail transiently and
recover once the Secret lands; the health check reports
`CredentialsUnavailable` meanwhile.
**Rotating the token needs no restart.** The token file is read on
every request, so the next request after the kubelet refreshes the
mount uses the new token. Keep the old token valid until then: a 401
is a permanent error, so a Proxy whose request lands in the gap
latches until its spec changes. Add the new token
(`pveum user token add`), update the Secret, wait a couple of minutes
for the mount to refresh, and only then remove the old token. A token
created with `--expire` needs the same rotation before it expires.
**Rotating the CA does** need a restart: it is loaded on first use and
cached for the life of the process. A missing CA file is
retried until it lands; one without a certificate is a permanent error.
2. Add a provider entry to the providers config
(`config/manager/providers_config.yaml` in-cluster; the full sample,
commented out, is in `config/samples/providers-config.yaml`):
```yaml
- name: proxmox-home
type: proxmox
proxmox:
url: https://192.168.0.8:8006
tokenFile: /etc/proxy-operator/proxmox/token
caFile: /etc/proxy-operator/proxmox/ca.pem
nodes: [pv1-1]
isoStorage: proxy-seeds
vlanTag: 4
templates:
debian-13: {vmid: 9001}
machineTypes:
small: {cores: 1, memoryMiB: 1024}
```
**The URL's host must be in the node certificate.** The node's
self-signed certificate lists its own addresses and names, and TLS
verification checks the URL's host against them. On `pv1-1` it lists
`192.168.0.8`, `pv1-1` and `pv1-1.home.lab`, so
`https://192.168.2.8:8006` reaches the same node and still fails
verification (`CertificateInvalid` in the health check, a permanent
error on every call). See what yours lists with:
```sh
openssl s_client -connect 192.168.0.8:8006 </dev/null 2>/dev/null | openssl x509 -noout -ext subjectAltName
```
If the address the operator must use is not there, either give the
node a certificate that names it (a custom certificate in the node's
Certificates panel) and put that certificate's CA in `caFile`, or set
`insecureSkipTLSVerify: true`, which `New` logs as a `WARNING:` line.
`nodes` holds node names as Proxmox knows them, not addresses: the one
URL reaches the whole cluster. The providers config is read once per
process, so adding a node, a template or a machine type needs a manager
restart.
3. Apply a Proxy (`config/samples/proxy_proxmox.yaml`) and watch it:
```sh
kubectl get px -w
# NAME MODE PROVIDER PHASE IP HEALTHY
# proxy-proxmox-sample Managed proxmox-home Provisioning
# proxy-proxmox-sample Managed proxmox-home Running 192.168.4.147
# proxy-proxmox-sample Managed proxmox-home Ready 192.168.4.147 True
```
(Illustrative.) `Create` returns once the VM runs, a few seconds after
the clone; the address follows when the guest agent is up, about 50
seconds after the start on `pv1-1` with the agent installed at first
boot.
## Config reference
The block is `proxmox:` and its Go type is `ProxmoxConfig` in
`internal/provider/config_proxmox.go`, validated at load by
`validateProxmox`; unknown keys are rejected. Defaults are applied by
`withDefaults` in `internal/provider/proxmox/proxmox.go`.
| Field | Required | Default | Meaning |
|---|---|---|---|
| `url` | yes | | `https://<host>:8006`, with no path (the client appends `/api2/json`). The host must be in the node certificate |
| `tokenFile` | yes | | file holding one line, `<user>@<realm>!<tokenid>=<secret>`, read on every request and sent as `Authorization: PVEAPIToken=<line>` |
| `caFile` | | system roots | PEM bundle trusted for the API's certificate, typically the node's `/etc/pve/pve-root-ca.pem`. Loaded on first use and cached |
| `insecureSkipTLSVerify` | | `false` | skip certificate verification; `New` logs a `WARNING:` line when it is on |
| `nodes` | yes, at least one | | node names `placement.zone` selects from; unique, lowercase letters, digits and dashes. They end up in request paths and as the clone's `target` |
| `isoStorage` | yes | | storage id the seeds are uploaded to and deleted from. Must hold `iso` content; the token needs `Datastore.Allocate` on it. No default on purpose |
| `storage` | | the template's disk storage | storage id the full clone's disks go to |
| `bridge` | | as cloned | replaces `bridge=` in the cloned `net0` |
| `vlanTag` | | as cloned | 1 to 4094; replaces or appends `tag=` in the cloned `net0` |
| `vmidRange.min`, `vmidRange.max` | | `100000`, `199999` | where VMIDs are allocated; at least 100, at most 999999999, min below max |
| `templates.<key>.vmid` | at least one template | | the key names the template in `placement.image`; the VMID is at least 100 |
| `machineTypes.<key>` | at least one | | `{cores, memoryMiB, diskGiB}` size profile. `cores` and `memoryMiB` above zero. `diskGiB` is optional: when larger than the cloned boot disk the disk is grown to it; it is never shrunk |
| `requestTimeoutSeconds` | | `30` | bound on one API request, response included. The seed upload is one request, so a multi-megabyte `vendorData` needs more |
| `vendorData` | | | extra cloud-init vendor-data placed on every seed next to the Proxy's user-data; a MIME multipart part beside the agent script when `installGuestAgent` is on. Cloud-init drops any top-level key of a vendor-data cloud-config that the user-data also defines |
| `installGuestAgent` | | `true` | add the vendor-data script that installs and starts `qemu-guest-agent`. Turn it off for templates that ship the agent |
## Placement
All three catalogue fields are checked before any request. An unknown
value is `ErrPermanent`, which the reconciler latches as `Failed` with the
message below until the spec changes.
| Proxy field | Meaning for proxmox | Error when unknown |
|---|---|---|
| `placement.zone` | a node from `nodes`. May be omitted when exactly one node is configured; otherwise required | `placement.zone "x" is not in proxmox.nodes`; with several nodes and no zone, `placement.zone is required when proxmox.nodes lists more than one node` |
| `placement.machineType` | a key in `machineTypes` | `placement.machineType "x" is not in proxmox.machineTypes` |
| `placement.image` | a key in `templates` | `placement.image "x" is not in proxmox.templates` |
| `placement.region` | ignored. It is still part of the spec hash, so changing it replaces the VM and does nothing else | |
The messages come from `placement` in `internal/provider/proxmox/create.go`
and `node` in `proxmox.go`. A key that exists but names a VMID that is
missing or not a template fails later, before any clone:
`placement.image "x": template VM 9001 does not exist` or `... is not a
template`.
**Catalogue edits affect only new VMs.** The spec hash covers the
placement values, not what the config maps them to, so repointing
`templates.debian-13.vmid` at another template changes the VMs created
afterwards and leaves the running ones alone. Add a new key rather than
repointing one in use, so the Proxies that name a key keep a meaning you
can reason about. A template may be edited, replaced or deleted without
affecting existing VMs: they are full clones and share nothing with it.
## What lands on Proxmox
For a Proxy whose deterministic name is `proxy-abc…`:
| Object | Name | Lifetime |
|---|---|---|
| VM | name `proxy-abc…`, a VMID from `vmidRange` | with the Proxy |
| its disks | the template's disks, fully copied onto `storage` (or the template's storage) | destroyed with the VM (`purge=1&destroy-unreferenced-disks=1`) |
| seed ISO | `<isoStorage>:iso/proxy-abc…-seed.iso` on the VM's node | with the Proxy |
**The VMID** is derived from the Proxy's UID, so a retried Create lands on
the same number: an FNV-1a hash of the UID placed in `vmidRange`, moving
forward one at a time on collision, up to 32 probes (`vmidFor` in
`id.go`). Before allocating, `Create` looks the name up in the cluster's
resources. A VMID that is taken, or a VM that already has the name, goes
through an ownership check (`inspect` in `create.go`): a VM with a lock
(a clone still running, which also has no name yet) is waited on; then
- another name: someone else's VM, so the next VMID is probed;
- our name and our mark with our UID or an empty one: **adopted**, and
Create carries on from where the previous attempt stopped;
- our name without the mark, or with another UID: **permanent error**
naming the VMID and both UIDs. The operator never takes over a VM it did
not create;
- our name, our mark, hibernated (`lock: suspended`): **permanent error**,
`resume or destroy it in Proxmox` (see *Known gaps*). This only happens
when Create has not recorded the VM yet; a recorded VM that gets
hibernated is reported Stopped and replaced.
Two VMs with the proxy's name are a permanent error naming every VMID;
remove all but one.
**The providerID** is `vms/<vmid>/<name>`. It names no node, because VMs
migrate: the node is looked up in `GET /cluster/resources` on every call.
`Get` and `Delete` check that the VM at that VMID still has that name and
the mark. Another name means ours is gone and the VMID was reused: `Get`
answers NotFound, `Delete` succeeds without touching it.
**The mark** is three lines in the VM's description (Notes in the UI),
passed in the clone call itself, so a VM is never unmarked:
```
proxy-operator-managed=true
proxy-operator-uid=0f3a…
proxy-operator-created-at=2026-09-23T17:42:11Z
```
That is how orphan GC recognises its own VMs and how a repeat `Create`
knows whether a VM of the same name belongs to this Proxy. Lines an admin
adds to the description are kept and ignored. The clone's own
`meta: ctime=` is useless for this: Proxmox copies it from the template.
**The config write.** After the clone and the seed upload, one
`PUT /nodes/{node}/qemu/{vmid}/config` sets:
| Key | Value |
|---|---|
| `cores`, `memory` | from the machine type |
| `agent` | `1` |
| `tags` | `proxy_operator` |
| `ide2` | `<isoStorage>:iso/<name>-seed.iso,media=cdrom` |
| `net0` | only when `bridge` or `vlanTag` is set: the cloned value with `bridge=` and `tag=` replaced or appended, the model, MAC and every other option kept |
The write **replaces the template's `agent` option and tags**: a
template's `agent: 1,fstrim_cloned_disks=1` becomes `agent: 1`, and its
tags give way to `proxy_operator`. The tag is for filtering in the
Proxmox UI and narrows GC's candidates; the mark decides. Proxmox appends
`ide2` to the boot order on its own (`order=scsi0;ide2`), which is
harmless because the boot disk stays first. When the machine type's
`diskGiB` is larger than the boot disk (the first disk in `boot: order=`
that is not a CD-ROM), a resize task grows it; cloud-init's growpart
expands the root filesystem at first boot.
**The seed** (`nocloud.BuildSeedISO`, shared with kvm2 in
`internal/provider/nocloud`) is an ISO 9660 image labelled `cidata`.
`user-data` is the Proxy's `cloudInit` **byte for byte**; the operator
never parses or re-renders it. `meta-data` holds `instance-id: <name>` and
`local-hostname: <name>`. `vendor-data` carries the guest-agent script
(a shell script, not a cloud-config, so a Proxy's own `packages:` cannot
shadow it), joined with `vendorData` as a MIME multipart when both are
set. It is staged under `/tmp` (the manager's emptyDir), built in memory
and uploaded with `POST /nodes/{node}/storage/{isoStorage}/upload`. An
upload overwrites a file of the same name, so it is rebuilt and
re-uploaded on every Create attempt with no delete first: a previous
attempt may have been cut off. Only the API is used; the storage upload
endpoint does not accept `snippets`, so Proxmox's own `cicustom` would
need shell access to the node.
**What a retried Create finds.** Every step is safe to repeat. After a
crash mid-clone, the retry lands on the same VMID, waits for the clone's
lock, and adopts the VM. A stopped VM of ours gets the seed, the config
and the resize again (the upload overwrites, the config is a replace, the
resize is skipped once large enough), then the start. A running one is
returned at once.
**A clone lock that never clears.** A node that crashes mid-clone can
leave the VM locked with `lock: clone` for good. Every Create for that
Proxy then waits out the 10 minute clone budget and returns a transient
error, and the reconciler keeps retrying rather than latching. Clear it
on the node:
```sh
qm unlock <vmid> # then let the next Create adopt and finish it
qm destroy <vmid> --purge # or remove it and let Create clone again
```
**Seeds without a VM.** A VM destroyed outside the operator leaves its
seed ISO, which GC cannot see (it lists VMs). Seeds are named
`proxy-*-seed.iso` on `isoStorage`; any without a matching VM may be
removed by hand:
```sh
pvesm list proxy-seeds --content iso
pvesm free proxy-seeds:iso/proxy-abc…-seed.iso
```
**Delete** runs under a per-name lock shared with Create: stop (a paused
VM too), delete the seed on the VM's node, then destroy with
`purge=1&destroy-unreferenced-disks=1`. Each is a task that is waited on.
The seed goes before the VM because once the VM is gone nothing calls
Delete again; a failed seed delete leaves the VM stopped, and the next
Delete finishes both. A VM that is already gone, not ours, or unmarked is
left alone, and the seed is then looked for on every configured node,
where a node that fails is logged and skipped. A refusal every node would
give fails the call instead: **without `Datastore.Allocate` on
`isoStorage` the 403 makes Delete permanent** on either path, and the
Proxy's deletion stays stuck with that message (a replaced Proxy stays
failed, its VM stopped) until the privilege is granted. When Delete
returns nil, nothing of the Proxy is left.
## Logging
The provider logs through controller-runtime's logger, named `proxmox`,
with `provider` (the configured name) on every line, and `vmid`, `name`
and `node` on every line about an instance. Every line inside a traced
operation also carries `traceID` and `spanID`.
- At startup (always on): `WARNING: token file is not readable; every
call fails until it is` with `path` and `reason`, and `WARNING:
insecureSkipTLSVerify is set`.
- `--zap-log-level=debug` (logr V(1)):
- one line per operation outcome: `proxmox vm ensured running`,
`proxmox vm adopted` with `status` and `qmpstatus`,
`proxmox vm fetched` with `state`, `ip`, `uid`, `status` and
`qmpstatus`, `proxmox vm not found` with `reason`,
`proxmox vm deleted` with `destroyed` (false when no VM of ours was
left to destroy), `proxmox vms listed` with `vms`, `candidates`,
`managed` and `skipped` counts, and `proxmox node checked` with
`ready`, `reason` and `latency`;
- **one `proxmox api call` line per API request**, written by the
decorator in `internal/provider/proxmox/observe.go`: `call` (the seam
method, for example `Clone`), the identifiers it was given (`node`,
`vmid`, `storage`), `upid` for a task, `newid` and `name` on a clone,
`filename` on an upload, `volid` on a volume delete, `duration`, and
on failure `httpStatus` and `error`. A running `Get` is four calls; a
fresh Create is a dozen or more, depending on how many polls each
task takes;
- `proxmox task failed` with `upid` and `exitstatus`, and
`proxmox task finished with warnings`;
- the no-op lines: `proxmox vm already gone`,
`proxmox vm at this vmid is not ours, left alone` with `vmName`,
`proxmox vm carries no proxy-operator mark, left alone`,
`proxmox seed delete failed on a node, skipped`,
`skipping unreadable proxmox vm`, and the probe's
`proxmox vmid taken by another vm, probing on`.
- `--zap-log-level=2`: `proxmox create resolved` with the resolved
parameters (node, machine type, cores, memory, disk, template VMID,
storages, bridge, VLAN tag) and `cloudInitBytes`; the `proxmox api
call` lines also carry the config write's parameters, a summary of each
config read, the clone's description, storage and target, the resize
size, VM and task status, the agent's addresses and the upload's byte
count; `proxmox guest agent answered`; `proxmox listed instance` per
listed VM.
- Never logged at any level: the cloud-init content, the token, the CA
content and the upload body. There is no raw wire tier.
## Tracing
With an OTLP exporter configured ([architecture.md
§10](architecture.md#10-tracing-internaltracing)), every API request is a
child span of the operation span the decorators open (`provider.create`,
`provider.get`, `provider.delete`, `provider.list`) or of the monitor's
`providerhealth.check`:
| Span | Around | Attributes |
|---|---|---|
| `HTTP GET`, `HTTP POST`, `HTTP PUT`, `HTTP DELETE` | one API request, from `otelhttp` on the client's transport | otelhttp's standard HTTP client attributes: method, URL, server address and port, response status |
The transport is `tracing.RestConfigWrapper()`, the same parent-gated
wrapper the kubernetes provider uses, so a request without a parent span
starts none. The logging decorator starts no span of its own, so nothing
is counted twice; its `proxmox api call` line carries the operation
span's `spanID`, not the HTTP span's. The token travels in a header,
which otelhttp does not record, and no attribute carries cloud-init.
## Timeouts
No caller puts a deadline on the context, so every request and every wait
carries its own budget (`defaultBudgets` in
`internal/provider/proxmox/proxmox.go`, and the transport in `client.go`):
| Bound | Budget |
|---|---|
| TCP connect | 10 s |
| TLS handshake | 10 s |
| one request, response included | `requestTimeoutSeconds`, default 30 s |
| clone task, and the wait for a VM's lock in the ownership check | 10 m |
| upload task, resize task, start task, stop task | 2 m each |
| destroy task | 5 m |
| seed delete task | 1 m |
| wait for `running` after the start | 60 s |
| task and status poll interval | 1 s |
A task wait that runs out is a transient error naming the UPID; the task
keeps running on Proxmox, and the next attempt finds its result through
the ownership check or the lock. A task that ends with any `exitstatus`
other than `OK` or `WARNINGS: <n>` is transient too, with Proxmox's text.
Requests share one HTTP client and its idle connections (4 per host,
closed after 90 s idle). A cancelled or overlong request ends only
itself: nothing else in flight is disturbed, and there is no connection
to kill as there is for kvm2. HTTP/2 is disabled, because pveproxy speaks
HTTP/1.1 only. Create and Delete for the same name are serialised on a
per-name lock, so the reconciler's Delete and GC's Delete never race on
stop and destroy; Get and ListByTag take no lock.
## Health check
The operator checks every configured node every
`--provider-check-interval` (default 1m) and publishes the result in the
`ProviderHealth` object named after the provider
(`kubectl get ph proxmox-home -o yaml`) and in the
`proxy_operator_provider_target_up{provider,target}` gauge. One target per
`nodes` entry, named after the node with the URL as its address, in
config order, all checked concurrently.
Each check (`checkNode` in `internal/provider/proxmox/check.go`) first
reads the token file for its id, then makes four requests on the shared
client: `GET /version`, `GET /nodes/{node}/status`,
`GET /nodes/{node}/storage/{isoStorage}/status` (the storage must exist,
be active and hold `iso` content), and `GET /cluster/resources?type=vm`
(every configured template must exist and be a template). A check that
runs out of time cancels only its own requests.
Every target's `details` carry `isoStorage`. As facts arrive they gain
`tokenID` (the part of the token before `=`, never the secret),
`pveVersion`, `kernel`, `isoStorageAvailGiB` and `templates`
(`<found>/<configured>`), so a target that fails late still says what the
node answered before. There is no quota signal; the storage's free space
is here instead.
| Reason | Meaning | Look at |
|---|---|---|
| `Reachable` | ready | |
| `CredentialsUnavailable` | the token file is missing or not `<user>@<realm>!<tokenid>=<secret>`; no request was made | the `proxmox-token` Secret; fixed in the cluster, not in Proxmox |
| `CertificateInvalid` | TLS verification failed, or `caFile` is missing or holds no certificate | `url` (its host must be in the node certificate) or `caFile`; not the node |
| `Unreachable` | the TCP connect or TLS handshake failed for a network reason, the connection was cut mid-response, or pveproxy answered 595 or 596 (it cannot reach the node behind it) | routing and firewalls between the cluster and `url`; `pveproxy` running; the node up |
| `Unauthorized` | 401: the token id or secret is wrong | the token file; `pveum user token list proxy-operator@pve` |
| `Forbidden` | 403; the message names the path and the privilege | the roles in *Proxmox prerequisites* |
| `Timeout` | the monitor's `--provider-check-timeout` (default 15s) ran out, whichever request was running, or a request ran out of `requestTimeoutSeconds` while the round still had time | a slow or hung pveproxy, a saturated link |
| `StorageNotFound` | `isoStorage` answered with a 500 on that node (it does not exist, is not enabled there, or is offline), is not active, or holds no `iso` content; the message carries Proxmox's text | `pvesm status`; the storage's nodes and content types |
| `TemplateNotFound` | a configured template VMID does not exist or is not a template | `qm list`; the `templates` catalogue |
| `APIError` | any other failure | the message carries Proxmox's own text |
The check does not look inside the templates: a cloud-init drive, a
missing `net0` or an unreadable disk size surface only when a Create is
about to clone.
## Error classes
Every error is classified once, in `classify` in
`internal/provider/proxmox/errors.go`, with errors that carry a class from
their source (validation, the token and CA files, the ownership check)
keeping it. The reconciler latches `ErrPermanent` as `Failed` until the
spec changes, recreates on `ErrNotFound` from `Get`, and retries
everything else with backoff. The provider never returns
`ErrQuotaExceeded`: the thin `local-lvm` storage overprovisions, so no
out-of-space answer could be produced to match, and capacity failures are
transient.
| Class | What lands there |
|---|---|
| `ErrPermanent` | placement values not in the catalogue; a template VMID that is missing or not a template, has a cloud-init drive, or lacks `net0` or a sized boot disk when the config needs them; a malformed providerID; any 4xx (401 `Authentication failed!` or `No ticket`: the token; 403 `Permission check failed (<path>, <privilege>)`: a missing privilege, including `Datastore.Allocate` for the seed delete; 400 `Parameter verification failed.` with the field errors); a TLS verification failure; a token file that exists but is not `<user>@<realm>!<tokenid>=<secret>`; a CA file with no certificate; a VM with the proxy's name that carries no mark or another UID; two VMs with the proxy's name; our VM hibernated (`lock: suspended`) found by Create's adoption |
| `ErrNotFound` | from `Get`: the VMID absent from the cluster's resources, the 500 `Configuration file 'nodes/<node>/qemu-server/<vmid>.conf' does not exist`, the VM at the VMID now named differently or unmarked. Inside `Create` the `does not exist` answer is `ErrTransient`, because the contract forbids NotFound from Create; `Delete` treats all of these as done |
| `ErrTransient` | everything else: a missing token or CA file (the Secret has not landed yet); network errors and cancellation; 500 `VM is locked (<lock>)` and any lock seen in the resources or config; a VM whose node does not answer, or one a few seconds old the listing has not reported yet (`status: unknown`); the agent's answers that `Get` cannot place, such as `VM <vmid> is not running`; any other 5xx, including pveproxy's 595 and 596; a task that fails or runs out of its budget; no free VMID after 32 probes; the wait for running expiring, or a VM that stops right after its start |
`Get` turns two agent answers into states rather than errors:
`QEMU guest agent is not running` is Provisioning (booting, or the agent
install has not finished), and `No QEMU guest agent configured` is
Stopped (someone removed `agent: 1`; the reconciler replaces the VM).
Error strings name VMIDs, nodes, storages, UPIDs and Proxmox's own text.
They never contain the token, the CA content or cloud-init.
## Known gaps
- **Create blocks** until the VM runs, holding one of the three reconcile
workers for its duration. On `pv1-1` that is about five seconds plus the
clone; large templates on slow storage take longer. Other Proxies keep
reconciling on the remaining workers.
- **A clone lock that never clears** (a node crash mid-clone) makes every
Create for that Proxy wait out the 10 minute clone budget and return a
transient error. Clear it with `qm unlock <vmid>` or destroy the VM (see
*What lands on Proxmox*).
- **Seed ISOs of VMs destroyed outside the operator are invisible to
GC.** Remove `proxy-*-seed.iso` files without a matching VM by hand.
On a storage that is not shared, a VM migrated to another node leaves
its seed on the original node; Delete then looks on the VM's current
node, finds nothing, and the seed stays. Shared ISO storage avoids it.
- **The VMID range is shared with humans.** A VM created by hand inside
it costs a probe; a hand-made VM named like a proxy is refused as
`ErrPermanent`. Reserve the range.
- **Cross-node clones need shared storage and are untested.** When the
zone's node differs from the template's, the clone is sent with
`target`, which Proxmox allows only when the template's disks are on
shared storage. The host this was built on has one node.
- **A VM whose agent was switched off is replaced**, not repaired: with
`agent: 1` removed it can never report an address, so `Get` reports it
Stopped.
- **The proxy network needs internet** for the agent install, unless the
template ships the agent and `installGuestAgent` is `false`. Without it
the VM boots, gets an address, and stays `Provisioning` forever,
because the agent is the only address source. Check
`systemctl status qemu-guest-agent` and
`/var/log/cloud-init-output.log` on the guest (`qm terminal <vmid>` on
the node, or the console in the UI).
- **The CA is cached** after first use; rotating it needs a manager
restart. The token rotates freely.
- **A node that stops answering** makes `Get` and `Delete` for its VMs
transient until it returns, so deleting such a Proxy waits. Proxmox
lists an unreachable node's guests with `status: unknown` rather than
dropping them, so they are never mistaken for gone. That is Proxmox's
documented behaviour; a single-node host could not show it.
- **Contract limits that apply unchanged** ([provider-contract.md
§13](provider-contract.md)): the providers config is read once per
process; GC has no installation identity, so two operators sharing a
Proxmox cluster and a VMID range delete each other's VMs; and a Proxy
naming an unconfigured provider latches until its spec changes.
Beyond the ten limits accepted in the design:
- **A hibernated VM is replaced, or latches a Create.** `Get` reports a
VM of ours with `lock: suspended` (hibernated to disk) as Stopped, so
the reconciler deletes it (the destroy removes its suspend state volume
too) and creates a new one. Only a Create that finds such a VM before it
was ever recorded, by name, fails permanently with
`resume or destroy it in Proxmox`: resume it by starting it
(`qm start <vmid>` restores the hibernated state) or destroy it on the
node, then touch the Proxy's spec.
- **A Create that failed permanently after its clone keeps its VM.**
The stopped VM is adopted by name on the next Create, even when the
Proxy's `placement` has changed since: the Proxy then runs the old
template on the old node. After changing `placement` on such a Proxy,
destroy the VM on the node (`qm destroy <vmid> --purge`) first, or
delete and recreate the Proxy.
- **A retry can clone a second VM** in one rare sequence: a Create is
cut off while cloning onto a later probe VMID (the first was taken by
someone else's VM), that other VM is removed, and the retry comes
while the clone is still running, so its VM is still nameless. The
retry clones onto the now free first VMID. The next Create then fails
permanently with `VMs … are all named …`; destroy either VM.
- **The seed upload is one request**, bounded by `requestTimeoutSeconds`.
A seed is tens of KiB, but a multi-megabyte `vendorData` would need a
larger timeout.
- **The sample cloud-init is an open proxy.**
`config/samples/proxy_proxmox.yaml` installs Squid with
`http_access allow all`; restrict it before exposing the proxy network
to anything but the cluster.
## Testing
Unit tests run over an in-memory fake of the API seam, and the HTTP
client over a TLS test server replaying the responses captured from
`pv1-1` (`internal/provider/proxmox/testdata/`). The optional integration
test against a real host is described in
[testing.md](testing.md#the-proxmox-integration-test-real-proxmox-ve-host),
which also lists the variables it needs; run it with
`make test-integration-proxmox`. `TestProxmoxCheck_realHost` runs the
health check against the same host and creates nothing:
`go test -tags=integration -count=1 -v -run TestProxmoxCheck ./internal/provider/proxmox/`.
Both passed on `pv1-1` (PVE 9.2.20) on 2026-09-24: Create took 6.2 s,
and the guest reported an address 50 s after Create returned.
docs/reviews/2026-09-17-kvm2-provider-review/REPORT.md
+183
View File
@@ -0,0 +1,183 @@
# kvm2 provider review: final report
Branch `feat/kvm2-provider` at `2510842`. Scope in `scope.md`; phase 1 detail in `sweep.md`;
phase 2 detail, measurements and unrun probes in `deep.md`.
**What this contains.** Confirmed findings only: the sweep findings (each reported by its
sweeper as verified in code) plus the escalations that came back CONFIRMED. Where a deep dive
refined a sweep finding, the refined version is given. **52 findings: 4 high, 20 medium, 28 low.**
**How far to trust it.** Nothing here was fixed or re-verified by a second reader except where
two or three reviewers reported the same thing independently (marked ×2/×3). Hypervisor access
was read-only, so every claim about what libvirt does on a *write* is from libvirt source or
memory, with the reviewer's stated confidence; the scripts that would settle them are in
`deep.md`. The kvm2 integration test has **not yet run on beelink** (Step 11 is unchecked in
the plan execution file; every `base-*` volume there predates kvm2), so no kvm2 behaviour in
this report has been observed against real libvirt.
Themes: **T1** unknown errors default to Transient and retry forever with no condition on the
Proxy · **T2** the ownership mark is enforced on Create/ListByTag but not Get/Delete ·
**T3** base image cache identity, lifetime and fencing · **T4** any caller can kill the shared
pooled link, and several callers misread the resulting error.
---
## High
### H1 — `markDead` blocks on a silent host and pins every reconcile worker (S1.1 + E1)
`internal/provider/kvm2/backend.go:138-148`, `ssh.go:109-114`
`markDead` runs `kill()` inside `once.Do` and only then closes `l.closed`. `kill` is `sshSocket.Close`, which closes the SSH channel first; that needs `ch.writeMu` and `t.mu`, both held by a writer stuck in a TCP write. Reproduced against x/crypto v0.54.0: with a 256 K send buffer the close was still blocked after 8 s, and a plain `client.Close()` released it instantly. Precondition: a bulk write in flight and a send buffer under the 2 MiB SSH window.
**Scenario.** Three Proxies placed on one host with an uncached image. Worker 1 uploads (`image.go:270`, 20 min budget); workers 2 and 3 park on the image lock (`kvm2.go:224-233`) with no deadline; `MaxConcurrentReconciles` is 3. Host is power-cut. The keepalive's `markDead` (`hosts.go:158`) blocks in the channel close; worker 1's `call` cannot see `done`, `l.closed`, or even a ctx cancel (it queues in `once.Do`). For ~15 min no Proxy on any provider reconciles. No `ReconciliationTimeout` is set, and one alone would not release worker 1.
**Fix.** Close the TCP connection first in `sshSocket.Close` (`s.client.Close()` before, or instead of, `s.Conn.Close()`); in `markDead`, `close(l.closed)` before `kill()` or run `kill` in a goroutine. Regression test with a frozen proxy and small `SO_SNDBUF`. Optionally set `ReconciliationTimeout`.
### H2 — An error during `StorageVolUpload` wedges go-libvirt's stream sender forever (S4.1)
`internal/provider/kvm2/image.go:273` (upstream `rpc.go:290-308`, `socket/socket.go:347-376`)
`abort` and `outErr` are unbuffered. On a reader error `SendStream` blocks forever on `outErr <-`; on a server-side error `requestStream` blocks forever on `abort <- true`. A socket kill does not unblock them (E6 found why: `listen` blocks in `callback`'s send under `cmux`), contrary to the comment at `backend.go:152-156`.
**Scenario.** A mirror truncates the body, or libvirtd hits ENOSPC mid-upload. The caller is rescued only after the 20 min transfer budget; each failed upload leaks 1-2 goroutines plus a 4 MiB buffer, accumulating under reconcile retries; `discardVolume` (`image.go:292`) deletes the base before the deferred `fresh.Close()` (`:265`), so the file is unlinked while the server stream holds the fd. `TestEnsureImage_shortReadRemovesVolumeAndIsTransient` cannot catch it: the fake returns the reader error immediately.
**Fix.** Have `countingReader` convert a read error to `io.EOF` and report it out of band; the existing `readErr` / `read != size` checks (`image.go:281-290`) still catch truncation. Close the dedicated connection before `discardVolume`. Consider an upstream patch for the unbuffered channels.
### H3 — `address()` swallows every error, demoting healthy Running proxies (S3.1)
`internal/provider/kvm2/get.go:104-119`
Every non-`context` error from both address sources becomes `("", nil)`, so `mapState(running, "")` = Provisioning.
**Scenario.** (a) On beelink the ARP source never answers (`kvm2-provider-spec.md:146`), so a `qemu-guest-agent` restart in the guest (`ErrAgentUnresponsive`) empties the address. (b) Another reconcile's cancel kills the shared pooled link between `DomainGetState` and the address read; the "connection is closed" error carries no ctx sentinel, so `Get` *succeeds* with `IP=""`. Then `proxy_controller.go:208-211` clears `status.IP` and sets `Provisioned=False`, and `internal/health/engine.go:200-207` prunes the proxy's health state. One RPC hiccup pulls a working proxy out of rotation. `TestGet_addressErrorIsProvisioning` pins this.
**Fix.** Return Transient from `Get` when the address read fails with a connection-class error (`isDead(be)` or the closed-link error) and return NotFound on `ErrNoDomain`; keep "no address yet" only for agent-unavailable/ARP-empty on a domain that has never reported an address. Consider having the controller keep the last known IP while an instance is Running.
### H4 — The base image cache is shared and unfenced across provider entries (S7.1 ×2, E3-3a/3b, E4)
`internal/provider/kvm2/volume.go:17-25`, `image.go:113-146`, `:235-245`, `kvm2.go:111`; `internal/provider/kvm/image.go:44-57`; `config/manager/providers_config.yaml`, `config/samples/providers-config.yaml:41,73`
The base is `base-<key>.<format>` with a `.sha256` marker, byte-identical in the kvm provider, in the same default pool, under **separate, per-entry locks**. The shipped config puts `kvm-home` and `kvm2-home` on `beelink`, key `debian-13`, pool `default`, and the docs say both types may share a host. On the host both live (kvm-made) overlays back onto that very file. Neither the name nor the marker records the URL.
**Scenarios.**
1. *Wrong image, silently.* The two entries' URLs drift and neither sets `sha256`: whoever fetched first owns the file. Likewise, editing `images.<key>.url` alone is a silent no-op.
2. *Digest mismatch (3a).* One entry pins `sha256: D2` while the cached base is D1: `cachedImage` (`image.go:129` → `:145`) deletes the base under the other provider's live overlays. libvirt refuses a delete only on `in_use`/`building` (`storage_driver.c:1844-1856`, read at v11.6.0). With two different pins, every Create on either side deletes and refetches.
3. *Concurrent cold fetch (3b, by source, unexecuted).* B finds A's base-without-marker and deletes it (`image.go:117-118`), recreates it and uploads. libvirt holds `in_use` only during stream setup (`storage_driver.c:2421-2427`). A's upload completes into the unlinked inode, A writes the marker, and `StorageVolGetPath` resolves to B's partial file: **A boots a guest from a half-written base.** If B is kvm2, its `writeMarker` then fails `ErrStorageVolExist` and `image.go:241-243` discards the complete base under A's running domain. If the process dies during B's upload, a truncated base stays under a valid marker forever, since nothing re-hashes the base.
The rollout/leader-handover variant of this was **refuted** for the shipped manifests (lease held until the old pod dies; SIGKILL at 10 s).
**Fix.** kvm2-specific prefix including the entry name, or one process-wide image lock table; reject two kvm/kvm2 entries with the same address+socket+pool at config load. Upload to `<base>.part-<random>`, verify, then publish the final name; never delete a name this call did not create, in particular drop `discardVolume(base)` on a marker `ErrStorageVolExist`. Record size and URL next to the digest and check the base's allocation before trusting a marker.
---
## Medium
### M1 — Paused domains are destroyed (S2.2 ×2)
`create.go:230-231`, `get.go:56`, `state.go:16-27`. `get.go:56` discards the `DomainGetState` reason; paused/pmsuspended map to Stopped; `reconcileManaged` (`proxy_controller.go:217-225`) then deletes and recreates. The "virsh resume it" guard in `ensureRunning` only fires before an ID is persisted. **Scenario:** the pool fills and QEMU pauses the guest with `PAUSED_IOERROR` (or `virsh suspend`/save/snapshot/migrate); at the next drift poll the VM and its disks are gone and the recreate refills the pool. **Fix:** keep the reason; map paused to a non-destructive state (Provisioning or a new Degraded) except for reasons that are terminal.
### M2 — `Get` and `Delete` do not enforce the kvm2 mark (S2.3 ×2, E3-2) [T2]
`get.go:52-73`, `delete.go:55-98` vs `create.go:80-82`. IDs are byte-identical to the kvm provider's (`kvm/id.go:11-13`). **Scenario:** an entry is re-typed `kvm` → `kvm2` under the same name without first deleting its Proxies (the documented migration, out of order): `status.ProviderID` survives, `Get` returns the v1 domain with `UID: ""`, and anything but Running leads to kvm2 destroying a domain the docs say it can never touch. A tampered `status.providerID` such as `hosts/beelink/domains/docker-29` destroys an unrelated VM. Nothing prevents a re-type (no webhook, `config.go:93-135`). The reverse direction (kvm provider against kvm2 domains) has no guard at all (`kvm/kvm.go:177-181`). **Fix:** `removeDomain` and `get` require `meta.managed()`; pass the expected UID to Delete or compare `inst.UID` with `p.UID` in the controller.
### M3 — Volumes created before `DomainDefineXML` are unreachable when Create returns no ID (S2.1)
`create.go:107-114`. **Scenario:** overlay and seed succeed, define fails Permanent (or the manager dies). No `ProviderID` is set; on Proxy deletion `reconcileDelete` takes the empty-ID shortcut (`proxy_controller.go:270-274`); `ListByTag` lists domains only. `proxy-<x>.qcow2` and `-seed.iso` leak permanently. Five such leftovers already sit in beelink's `default` pool (Appendix C). **Fix:** on a failed define, best-effort remove the two volumes before returning; or have `Delete`/GC derive volume names from the Proxy UID.
### M4 — The pool-space precheck reads a cached, reserve-inflated number (E8)
`create.go:92-100`, `image.go:450-458`, `check.go:106-112`. Measured on beelink: libvirt's `available` is cached per pool (three pools on one filesystem report two values, 1.68 GB apart; frozen over 80 s while `f_bfree` moved) and is `f_bfree`, **24.7 GB above** what unprivileged QEMU can use on the kvm2 filesystem. The only refresh is after an image upload. **Scenario:** with `f_bavail` between 0 and ~10.7 GB the precheck passes for a 10 GiB disk, `ErrQuotaExceeded` never fires, and Create, the upload and the guests hit ENOSPC (feeding H2 and M1). `poolAvailableGiB` in ProviderHealth over-reports the same way. **Fix:** `StoragePoolRefresh` right before `checkPoolSpace` (not in Check); add headroom to `neededBytes` from the discarded `capacity` or a `poolReserveGiB` setting; make the fake's pool info cached-until-refresh.
### M5 — An overlay smaller than the base's virtual size is accepted; the Proxy hangs in Provisioning (S5.6 + E10)
`volume.go:72`, `create.go:107,158`, `internal/provider/config_kvm.go:133`. Measured locally: `qemu-img create` with a capacity below the backing file exits 0; reads past the capacity fail; a GPT base under a short overlay shows only the protective MBR. Bases on the host are 3 GiB virtual. **Scenario:** `diskSizeGb: 2` validates; Create succeeds; the guest has no root device and never reports an address; `Get` says Provisioning forever (no provisioning deadline exists), nothing names `diskSizeGb`, and raising it later does not repair that Proxy. libvirt's pass-through is ~90% from source; the 200 KiB probe is in `deep.md`. **Fix:** add `StorageVolGetInfo` to the backend seam and fail Permanent when the base's capacity exceeds the overlay's; same check on a reused overlay.
### M6 — A peer-close right after ConnectOpen wedges a pooled link forever (E6a)
`backend.go:177-178`, `:229-236`, `hosts.go:75`, `errors.go:55` (upstream race `libvirt.go:297/305/821`, reported by `-race`). **Scenario:** libvirtd or the forwarded socket goes away within ~1 ms of a successful open (daemon restart). `Connect()` returns nil with a `Disconnected()` channel nothing will close; every later RPC fails `EINVAL`, classified Transient; no result marks the link dead; the SSH keepalive keeps succeeding. The host answers "invalid argument" until the operator restarts. Wedged in 546/1000 cycles over real SSH in the reviewer's harness. The hypothesised trigger (kvm2's own kill) is harmless. **Fix:** in `call`, `markDead` on `syscall.EINVAL` / `libvirt.ErrInterrupted`; or wrap the `net.Conn` so a failed `Read` marks the link dead.
### M7 — Permanent host/config rejections retry forever with nothing in status (S5.2 ×2, S3.5, E9b) [T1]
`errors.go:58-71`, `xml.go:26-27,194-206`, `proxy_controller.go:344-345`. `x86_64`/`q35`/`kvm`/host-passthrough/efi are unconditional with no capability gate; code 67 (`ErrConfigUnsupported`) and the bare `libvirt.ErrUnsupported` sentinel are unmapped; `providerFailure`'s Transient arm sets no condition. **Scenario:** an aarch64 or KVM-less host is added; Check says Ready; every Create retries an impossible define forever, rerunning `writeSeed` each time; a missing bridge (38, at start) or missing OVMF (9) behaves the same and Check looks at neither. "Unknown means retry" is a recorded spec decision (`kvm2-provider-spec.md:317`), but it never considered 67. **Fix:** 67 → Permanent on create; `ErrUnsupported` → Permanent; leave 9/38/1/84. Surface a non-latching condition for Transient create failures, or check domain capabilities and the bridge in the health probe.
### M8 — Seed ISO I/O failures latch as Permanent with no requeue (S5.1) [T1, inverted]
`create.go:199-201`, `seed.go:66-74`. `permanentf("building seed ISO …: %v")` covers a missing `/tmp`, ENOSPC, and even a `RemoveAll` failure on an otherwise successful build. **Scenario:** node disk pressure for a minute leaves the Proxy unprovisioned until a spec edit or the ~10 h resync. **Fix:** Permanent only for the empty-name guard; wrap the rest Transient with `%w`.
### M9 — `HEAD` 2xx without a usable Content-Length becomes a latching Permanent (S4.2) [T1]
`image.go:434-439`, `:391-396`. The GET fallback fires only on non-2xx. **Scenario:** a CDN (or Go's own `net/http`) answers HEAD with `Content-Length: 0`; the Proxy never provisions although GET works. **Fix:** fall back to GET whenever HEAD yields no positive length.
### M10 — A completed upload is discarded when the pooled link died meanwhile (S4.3) [T4]
`image.go:235-245`. Refresh and `writeMarker` run on the pooled `be` after a transfer on a dedicated link. **Scenario:** any other reconcile's cancel during a 20 min download kills the pooled link; the verified base is deleted and refetched. **Fix:** run the refresh and marker on the still-open dedicated connection.
### M11 — Precheck uses HEAD's length, the volume uses GET's, and there is no size cap (S4.4)
`image.go:419-443`, `:226-230`. **Scenario:** HEAD says 1 MB, GET says 4 TiB: a sparse 4 TiB volume is created and the stream fills the host pool. **Fix:** re-check space against the GET length; add `maxImageBytes`.
### M12 — `placement.image` is an unvalidated user-supplied URL fetched by the operator (S4.5)
`image.go:59-65`, `internal/provider/config_kvm.go:101-104`. No CRD validation, no host allowlist, `http.DefaultClient` follows redirects, and a URL image can never carry a digest. **Scenario:** anyone who can create a Proxy makes the operator issue requests to cluster-internal addresses and boots arbitrary bytes on the egress bridge. **Fix:** a config switch to allow URL images at all (default off) or a host allowlist; `CheckRedirect` policy; block link-local/private ranges.
### M13 — A `sha256` change replaces the base in place under live overlays (S4.6 ×3)
`image.go:118,129,145`. No in-use check, `V(1)` log; a mistyped digest deletes the base before anything is verified. The kvm2 sample omits the warning the kvm sample carries. Whether guests actually corrupt is Appendix A (E7). **Fix:** on a digest mismatch return Permanent "add the image under a new key"; or put the digest in the volume name.
### M14 — IP detection takes the first IPv4 from any guest interface (S3.2)
`state.go:35-49`. `Hwaddr` is available but unused. **Scenario:** `docker0`/`tailscale0` comes up before the DHCP lease; the Proxy latches `Provisioned=True` with 172.17.0.1 and is never corrected. **Fix:** match the interface by the domain NIC's MAC.
### M15 — A link that dies mid-list yields a silently partial `ListByTag` (S3.3) [T4]
`list.go:38-45`. The guard tests `ctx.Err()`, but a killed link has no ctx sentinel. GC only under-deletes, so nothing wrong is removed, but the sweep covers a fraction of the fleet and says so nowhere. **Fix:** fail the list on any connection-class error (`isDead(be)`).
### M16 — `KVMHost.Address` is not validated as `host:port` (S7.2) [T1]
`internal/provider/config_kvm.go:180-182`. **Scenario:** `address: beelink` loads; every dial fails "missing port in address", classified Transient, and Check reports `Unreachable`. **Fix:** `net.SplitHostPort` at load, or default `:22`.
### M17 — Plain `http://` images with no digest are accepted silently (S7.3)
`internal/provider/config_kvm.go:101-104,142-147`. The root disk of every VM can come from an unauthenticated channel and the marker then pins it. **Fix:** require `sha256` for `http://` catalogue entries, or warn loudly at load.
### M18 — The fake ignores the metadata namespace (S7.4) [T2]
`fake_test.go:362-376`, `:259-282`. No unit test can build a kvm-v1-marked domain and assert `ListByTag` skips it; if that filter regressed, kvm2 would hand kvm's VMs to orphan GC with the suite green. E3/E9 observed that real libvirt 11.6 does scope by namespace. **Fix:** key the fake's metadata on the element's namespace; add the test.
### M19 — The integration target's timeout can skip VM cleanup on the real host (S7.5)
`Makefile:75`. `-timeout 40m` is fixed while two tests each take up to `KVM2_TEST_TIMEOUT` (20 m) plus 5 m cleanup. **Scenario:** a cold download or a raised env timeout trips go test's own timeout before `t.Cleanup` (`kvm2_integration_test.go:373-382`); a domain and two volumes stay on beelink under a random name. **Fix:** derive `-timeout` from the env var with margin; print the name up front; add a `review`-style sweep-by-prefix cleanup target.
### M20 — `sshSocket.keepalive` has no test (S1.2)
`ssh.go:119-147`. It is the only detector of a silently dead pooled link between calls, and `testSSHServer` already services global requests. **Fix:** an end-to-end test that stops the server's request handling and asserts `markDead`.
---
## Low
| ID | Where | Defect → scenario | Fix sketch |
|---|---|---|---|
| L1 (S1.3) | `ssh.go:120` | `NewTicker` panics on interval ≤ 0 in a goroutine → a future `libvirtDialer{}` literal kills the operator on first dial | default the interval when ≤ 0 |
| L2 (S1.4) [T1] | `errors.go:45-51` | `*knownhosts.RevokedError` unrecognised → a revoked host key retries forever | classify it Permanent |
| L3 (S1.5) | `ssh.go:167-172` | `HostKeyAlgorithms` empty → a host pinned by RSA only but serving ed25519 reports a Permanent "mismatch" | on `KeyError` with `Want`, retry with those key types; or document `ssh-keyscan` without `-t` |
| L4 (S2.4) | `create.go:87` | Adoption path starts a defined domain without checking its disks → missing overlay/seed retries forever (E5's EIO residual lands here) | on a storage error in the adoption branch, undefine and fall through to a full create |
| L5 (S2.5) [T1] | `delete.go:15` | `undefineFlags` lacks `CheckpointsMetadata` → an admin/backup checkpoint wedges the finalizer | add the flag |
| L6 (S3.4) | `get.go:104-119` | `ErrNoDomain` swallowed → a vanished domain reads as Provisioning for one poll; comment about ctx-only errors is wrong (a budget expiry never falls back to ARP) | return NotFound; fix the comment |
| L7 (S4.7) | `image.go:62` | URL images assumed qcow2/uefi → a raw URL image fails to boot with a misleading error | sniff the qcow2 magic from the first bytes |
| L8 (S4.8) [T3] | `volume.go:22-25` | 64-bit URL key, namespace shared with catalogue keys → colliding URLs or a 16-hex catalogue key share a base | longer key and a distinct `url-` prefix |
| L9 (S5.3, narrowed by E2) | `metadata.go:44-50` | future-dated mark → a true orphan is never collected. (The zero-time half is benign: `gc.go:118` checks the UID first.) | clamp future timestamps to now |
| L10 (S5.4) | `seed.go:48-53` | `rc-update` fallback unguarded under `set -e` → agent installed but not started on non-systemd, non-OpenRC images | `elif command -v rc-update`, as the kvm provider does |
| L11 (S5.5) | `xml.go:178-193` | `VCPUs`/`MemoryMiB` not validated in the builder (latent) | reject ≤ 0 |
| L12 (S5.7) | `internal/provider/config_kvm.go:186-188` | `bridge`/`pool` only non-empty → opaque libvirt error at Create | apply `kvmIdentifier`-style validation |
| L13 (S5.8) | `seed.go:107-108` | multipart or headerless configured `vendorData` silently dropped with `installGuestAgent` | reject at load what cloud-init will not recognise |
| L14 (S6.1) | `check.go:90`, `backend.go:162,351` | `Close()` can send `ConnectClose` for up to 5 s after the round deadline | `markDead` when `call` refuses on a done ctx, or skip the RPC in Close then |
| L15 (S6.2) | `check.go:57-58` | Latency includes the teardown RPC | stop the clock before Close |
| L16 (S6.3) | `check.go:66` | round-wide `ctx.Err()` masks a per-host reason landing on the deadline | classify the error first; Timeout only if it carries the ctx error |
| L17 (S6.4) | `check.go:135-148`, `internal/providerhealth/monitor.go:113` | a 10 s round timeout collapses Unreachable into Timeout | validate round timeout > dial budget, or shrink the dial budget to fit |
| L18 (S6.5) | `check.go:101` | unbounded remote `hostname` in status `details` → status patch "too large" | truncate; add `maxLength` to the CRD |
| L19 (S6.6) | `check.go:41-45` | unbounded per-host fan-out, key re-read every dial | cap concurrency; cache credentials |
| L20 (S7.6) [T1] | `internal/provider/config_kvm.go:133`, `image.go:466` | no upper bound on `diskSizeGb`; `neededBytes` wraps at 2^34 → precheck passes; huge `memoryMiB` retries forever | bound both at load |
| L21 (S7.7) | `fake_test.go:190-208` | fake accepts calls on a done ctx → `discardVolume`'s `WithoutCancel` is untested | make `enter()` refuse like `call()` |
| L22 (S7.8) | `fake_test.go:579-584` | fake `Close()` leaves the backend live → use-after-close passes in tests | mark dead on Close |
| L23 (S7.9) | `Makefile:60-75`, `.gitea/workflows/build.yaml:27-32` | nothing compiles `-tags=integration` | add `go vet -tags=integration ./...` to the `check` job |
| L24 (S7.10) | `image.go:228,283-289,385-395` | full image URL, including any userinfo, in logs and wrapped errors | use `urlHost`/redacted URL everywhere; reject userinfo at load |
| L25 (S7.11) | `kvm2_integration_test.go:89,307` | `KVM2_TEST_KEEP=0` enables keep; 100 ms wall-clock assertion on `New()` | parse the bool; drop or loosen the timing check |
| L26 (S7.12) | `config/manager/manager.yaml:163-178`, `seed.go:56-74` | cloud-init payloads staged on node-local disk; comment names kvm only | `medium: Memory`; fix the comment |
| L27 (E6 side) | go-libvirt `socket/socket.go:233-234` | packet length < 28 panics the listen goroutine and the operator; a huge length allocates up to 4 GiB → `host.Socket` pointed at a non-libvirt socket (reproduced) | upstream bounds check; until then validate the socket in Check before pooling |
| L28 (E6 side) | go-libvirt `rpc.go:76-84` | duplicate reply deadlocks `callback` under `cmux`, leaking 3 goroutines that survive a socket kill (measured +420/200 cycles); same mechanism keeps H2's leak alive | upstream: send outside the lock |
---
## Appendix A — PLAUSIBLE, with what would settle it
- **E2 — a padded or case-changed `<uid>` in the GC mark makes GC delete a live Proxy's VM.** Reproduced on a verbatim copy of the parser and the `gc.go` switch; there is no second guard before `Delete`. No trigger libvirt produces on its own was found (it re-indents between children only), so it needs a hand edit or a reformatting tool. The kvm provider has the same field, and the two live VMs are kvm v1. *Settle:* the `virsh metadata --set` round trip in `deep.md` on a scratch libvirt. *The fix is three `TrimSpace` lines and worth making regardless.*
- **E7 — in-place base replacement corrupts existing overlays.** The delete is confirmed (M13). Corruption needs a new QEMU process on an old overlay; the controller never does that, but `libvirt-guests` is enabled on beelink and restarts guests after a host reboot. *Settle:* `grep -Ev '^\s*(#|$)' /etc/default/libvirt-guests` on beelink, and the `review-e7-*` inode probe in `deep.md`.
- **Unexecuted links inside confirmed findings.** H4-3b: libvirt allows `vol-delete` during an upload (read in 11.6.0 source; the `probe-race`/`e4probe` scripts). M5: libvirt passes a short capacity through (~90%; `review-e10-short`). H1: the real path's send buffer is under ~2.2 MB (`ss -tmi` during an upload). E9a: duplicate volume → code 90 (~95%; `review-e9-dup`).
## Appendix B — REFUTED
- **E5** overlay left corrupt by an abandoned `StorageVolCreateXML`: libvirtd finishes or unlinks the build regardless of the client (~85%, source memory); creation is 2-11 ms on the host; the residual path fails loudly.
- **E3-1** GC reconciling one provider's list against another's Proxies: the live set is global and the list is namespace-filtered (observed on the host).
- **E3-2** re-type destroying *another* Proxy's instance: the ID always names the Proxy's own UID-derived domain (the remaining hole is M2).
- **E4 rollout/leader handover** deleting a base mid-upload: the lease is held until the old pod dies; SIGKILL at 10 s; the residual window is theoretical and self-healing.
- **E6b** goroutine/fd accumulation from per-round check dials: +0/+0 over 1000 cycles on three transports.
- **E9a** idempotency codes differing from the fake: 42, 49, 50, 80 observed on libvirt 11.6; 90 and 55 sourced at high confidence.
- **E2 (half)** an unparsable `createdAt` deleting a live VM: `gc.go:118` checks the UID first.
- **E7 (half)** marker-missing trigger without an admin edit: marker errors are returned, not treated as unreadable.
- **No XML injection**: three sweepers independently found every value goes through `encoding/xml`.
## Appendix C — Out of scope but noticed
- **Orphan volumes on beelink, pool `default`:** `proxy-7hjzuf3d3w2mjnye.qcow2` + `-seed.iso`, `proxy-qxge3dbm2mnpc3ih.qcow2` + `-seed.iso`, `proxy-sbs36ruo73sxxtqc.qcow2`. 196 KiB each, root-owned, no domain. Reported independently by four deep reviewers; they are M3 in the wild (likely from kvm v1 integration runs). Nothing was deleted.
- **The kvm2 integration test has not run on beelink** (plan execution Step 11 unchecked).
- **The kvm provider shares most of these defects:** untrimmed mark UID (E2), no mark check on Get/Delete and adoption of any same-named domain (`kvm/kvm.go:177-181`), short-overlay hole (`kvm/volume.go:24-32`), same base names.
- **No provisioning deadline in the controller:** any VM that boots but never reports an address stays Provisioning forever with no signal (what makes M5 silent).
- **`providerFailure` sets no condition for Transient errors** (`proxy_controller.go:344-345`), which is what makes all of T1 invisible.
- **Manager wiring:** no `ReconciliationTimeout`, no `LeaderElectionReleaseOnCancel`, default RollingUpdate with one replica, 128Mi memory limit (an OOM mid-upload is one way to H4's truncated base).
- **Host key naming:** `~/.ssh/kvm_known_hosts` pins beelink under its Tailscale FQDN only, while the sample config says `address: beelink:22`; an operator using that file would fail host key verification (seen from this workstation; not verified from inside the operator).
- **Dropped escalations not investigated:** D1 delete-during-open-stream disk-space effect, D2 cloud-init `text/plain` vendor-data dispatch, D3 ProviderHealth status churn, D4 nested `kvm-ssh` volumeMount under the read-only config mount, D5 whether error strings with image URLs reach Proxy status.
docs/reviews/2026-09-17-kvm2-provider-review/deep.md
+265
View File
@@ -0,0 +1,265 @@
# kvm2 provider review: phase 2 deep dives
Ten `deep-reviewer` runs, one per escalation in `escalations.md`. Hypervisor access was
**read-only** (beelink, libvirt 11.6.0, QEMU 10.1.0); every probe that needs a write was left
unrun and is reproduced under "Probe to settle". Scratch programs live under the session
scratchpad, not in the repo. The orchestrator did not re-read code.
| ID | Verdict | One line |
|---|---|---|
| E1 | **CONFIRMED** (narrowed) | Blocking kill reproduced against x/crypto v0.54.0; pins all 3 reconcile workers ~15 min per incident, trigger is a bulk upload, not the name lock |
| E2 | **PLAUSIBLE** | Padded `<uid>` text does make GC delete a live VM (reproduced on a copy of the parser); no trigger libvirt produces on its own. `createdAt` half REFUTED |
| E3 | **mixed** | GC cross-provider: REFUTED. Re-type beyond S2.3: REFUTED. Shared `base-<key>.<format>` cache between kvm and kvm2: CONFIRMED |
| E4 | **CONFIRMED** (narrowed) | Rollout/leader-handover trigger REFUTED; the unfenced base is reachable in one process via the kvm provider or a second kvm2 entry on the same host+pool |
| E5 | **REFUTED** (~85%) | libvirtd finishes or unlinks a volume build regardless of client disconnect; overlay create is 2-11 ms on the host |
| E6 | (a) **CONFIRMED**, (b) **REFUTED** | Upstream race on `l.disconnected` is real; a peer-close right after ConnectOpen wedges a pooled link forever. No goroutine/fd growth over 1000 cycles |
| E7 | **PLAUSIBLE** | Unguarded in-place delete is real; corruption needs an admin `sha256` edit plus a host-side restart of an old overlay (`libvirt-guests` is enabled on beelink) |
| E8 | **CONFIRMED** | `available` is cached per pool in libvirtd (measured), and it is `f_bfree`, 24.7 GB above what QEMU can use on the kvm2 filesystem |
| E9 | (a) **REFUTED**, (b) **CONFIRMED** | Idempotency codes match libvirt 11.6 (observed 42/49/50/80). Codes 67 and the bare `ErrUnsupported` retry forever, per a recorded spec decision |
| E10 | **CONFIRMED** (libvirt half ~90%) | qemu-img accepts a short overlay (measured locally); guest sees a truncated disk, GPT rejected; Proxy stuck Provisioning forever |
Two facts the deep dives surfaced that correct earlier documents:
- **The kvm2 integration test has not run on beelink.** Step 11 is unchecked in `docs/plans-executions/2026-09-16-1333-kvm2-provider.md`; every `base-*` volume on the host is dated 2026-09-10 (kvm v1's test, same `test-image` key); both live `proxy-*` domains carry the kvm v1 metadata namespace. The "Checked" line of E2 in `escalations.md` (round trip asserted on libvirt 11.6) was a sweeper claim and is not yet true.
- **Sweep finding S5.3 is half wrong.** A zero `createdAt` does not endanger a live VM: `gc.go:118` tests `live[inst.UID]` before the age case, so it only affects true orphans. The future-dated-mark leak stands.
---
## E1 — wedged `markDead` starves the reconcile workers — CONFIRMED (narrowed)
**Premise reproduced** (`scratchpad/e1/main.go`, in-process SSH server behind a freezable TCP proxy, x/crypto v0.54.0). `sshSocket.Close` (`ssh.go:109-114`) calls `s.Conn.Close()` first: `channel.Close` (`channel.go:594`) → `sendMessage` → `writePacket` takes `ch.writeMu`, then `handshakeTransport.writePacket` (`handshake.go:579`) takes `t.mu`. A data writer stuck in a TCP write holds both. `ssh.Client.Close` is a bare TCP close (`connection.go:111`) and takes no locks.
| Scenario | Client send buffer | Writer blocked in | Channel `Close` |
|---|---|---|---|
| Bulk write, loopback default | 2.6 MB | `window.reserve` (no mutex) | 0 s |
| Bulk write, LAN-sized buffer | 256 K | `net.(*conn).Write` under `writeCipherPacket` | still blocked after 8 s |
| Small RPC-sized write only | 2.6 MB | not blocked | 0 s |
Precondition: the writer must be stuck in the TCP write, not SSH flow control (2 MiB window). That needs a send buffer below ~2 MiB, which the reviewer estimates (not measured) is the case on a LAN/Tailscale path during a bulk upload. In the blocked case a separate `client.Close()` released the channel close at once.
**Breaking path.**
1. Three Proxies placed on one kvm2 host with an uncached image. Worker 1 takes the name lock (`create.go:53`) and image lock (`image.go:87`), then `uploadImage` → `dialFresh` (`image.go:261`) → `StorageVolUpload` (`image.go:270`, 20 min budget). Workers 2 and 3 park in `imageLocks.acquire` (`kvm2.go:224-233`); the reconcile ctx has no deadline. `MaxConcurrentReconciles` is 3 (`proxy_controller.go:442`): all workers occupied.
2. Host is power-cut mid-upload. The upload goroutine blocks in the TCP write holding both mutexes.
3. The dedicated link's keepalive (`hosts.go:156-160`) cannot send (needs `t.mu`); after 15 s `hosts.go:158` calls `markDead`, whose `kill()` blocks in `s.Conn.Close()`. `close(l.closed)` (`backend.go:146`) is never reached.
4. Worker 1 sits in `call`'s select (`backend.go:176-187`): `done` cannot fire, `l.closed` is not closed, and a ctx cancel reaches `markDead` at `:180`, which blocks in `once.Do` behind the keepalive goroutine. Even cancellation is uninterruptible.
5. After ~15.5 min (`tcp_retries2=15`) or the 20 min budget everything unwinds Transient. Meanwhile no Proxy on any provider reconciles.
**Unknowns settled.** No `ReconciliationTimeout` is set (`cmd/main.go:288-295`; controller-runtime v0.24.1 applies one only when > 0), and one alone would not release worker 1. The name-lock part of the hypothesis is practically unreachable (controller-runtime never runs one key concurrently; Delete's RPCs are too small to fill the buffer). GC (`gc.go:104-130`, one goroutine, no deadline) normally moves on; it stalls only if the blocked kill is on the *pooled* link, whose only bulk write is the ~50 KiB seed upload (`create.go:210`): marginal. Of the five `markDead` sites, `hosts.go:153` runs under `e.sem` but only handshake bytes have been sent, so it does not block.
**Minimal fix.** (1) In `sshSocket.Close`, close the TCP connection first: `errors.Join(s.client.Close(), s.Conn.Close())`, filtering the expected "use of closed network connection", or drop `Conn.Close()`. (2) In `markDead` (`backend.go:138-148`), `close(l.closed)` before `l.kill()`, or run `kill` in a goroutine. (3) Regression test in `ssh_test.go`: frozen proxy, small client `SO_SNDBUF`, assert `Close` returns within 1 s while a writer is blocked. (4) Optional: set `ReconciliationTimeout` so image-lock waiters requeue.
**Probe to settle** (not run, needs a bulk upload in flight): `ss -tmi dst <beelink-ip>:22` on the operator host during an image upload; `tb` below ~2.2 MB means the blocking case applies.
## E2 — reformatted GC mark makes GC delete a live VM — PLAUSIBLE
Throwaway program (`scratchpad/e2/main.go`) with a verbatim copy of `instanceMetadata`, the same `time.Parse(time.RFC3339, …)` and the `gc.go:112-122` switch; clock 16 days after the mark, MinAge 10 m.
| Input | Parsed UID | zero time | GC |
|---|---|---|---|
| single line (as written) | clean | no | KEEP |
| libvirt's re-indent between children | clean | no | KEEP |
| `<uid> X </uid>` or newline-padded | padded | no | **DELETE** |
| uppercase UID | differs | no | **DELETE** |
| `<uid> </uid>` (whitespace only) | `" "` | no | **DELETE** |
| `<uid></uid>` | `""` | no | KEEP (guard `gc.go:113`) |
| bad/missing `createdAt`, UID clean | clean | yes | KEEP |
| padded `<managed>` | clean | no | KEEP (not listed) |
| future-dated, UID clean | clean | no | KEEP |
`encoding/xml` does not trim chardata. **Path:** padded or case-changed `<uid>` with `<managed>true</managed>` intact → `parseInstanceMetadata` (`metadata.go:66-78`) returns it untrimmed → `ListByTag` (`list.go:46-49`) emits it → `Sweeper.sweep` (`gc.go:113-121`): not empty, not live, older than MinAge → `prov.Delete` (`gc.go:125`) → `delete.go:21-53` destroys the domain and its disks. There is no second guard: no name-to-Proxy lookup, `Delete` does not re-read the mark, and provider scoping does not help because the owning provider lists it.
**Why only PLAUSIBLE.** On the real host libvirt puts newline + two spaces *between* children, strips the xmlns and leaves text bytes intact (`od -c` of the live `…/kvm` marks). Per libxml2, text-bearing elements are never re-indented, so a version change alone would not pad text. The reachable trigger is a hand edit or a tool that pads text nodes via `virsh edit`/`define`: rare, but the result is an unrecoverable delete. The old `internal/provider/kvm/metadata.go` has the same untrimmed field, and the two live VMs are kvm v1 domains.
**Refuted parts.** A bad `createdAt` never deletes a live VM (UID check comes first, consistent with `docs/provider-contract.md:238-241`). A future-dated mark is a leak only, and marks are stamped from the operator's clock (`create.go:127`), so host skew cannot cause it.
**Minimal fix.** At the end of `parseInstanceMetadata`: `TrimSpace` on `Managed`, `UID` (plus `ToLower`), `CreatedAt`. A whitespace-only UID then equals `""`, which `gc.go:113` keeps. Add table rows to `metadata_test.go`. Apply the same to `internal/provider/kvm/metadata.go`. Optional second guard in `gc.go`: skip and log when `provider.NameFromUID(inst.UID)` is not the name part of `inst.ID` (check the other providers' naming first).
**Probe to settle** (not run; use a scratch libvirt, not beelink):
```
virsh -c qemu:///system metadata <scratch-dom> https://gitea.home.hrajfrisbee.cz/kacerr/egress-proxies-operator/kvm2 --key kvm2 --set '<instance><managed>true</managed><uid> 3f1c2a9e-0000-4000-8000-000000000001 </uid><createdAt>2026-09-01T10:00:00Z</createdAt></instance>' --config
virsh -c qemu:///system metadata <scratch-dom> https://gitea.home.hrajfrisbee.cz/kacerr/egress-proxies-operator/kvm2 | od -c
```
## E3 — kvm and kvm2 on the same hypervisor — three paths, three verdicts
**1. GC reconciles provider A's list against provider B's Proxies — REFUTED.** `gc.go:91-101` builds one global `live` set from every Proxy; sharing it only makes GC more conservative. kvm2 `ListByTag` (`list.go:38-48`) is namespace-filtered, and that was checked on the host: both live domains return the mark under `…/kvm` and nothing under `…/kvm2`. What remains is leaks: after a re-type the old type's orphans are never collected, and the scoping still rests on libvirt behaviour no unit test covers (S7.4).
**2. Same UID under both types; reverse direction — REFUTED beyond S2.3.** `NameFromUID` depends only on the UID and `spec.provider` is CEL-immutable (`proxy_types.go:221`), so the same name under both types only comes from re-typing an entry. Nothing prevents that: `config.go:93-135` checks only name uniqueness and block/type agreement, there is no webhook (`cmd/main.go:262`), `status.providerID` records no type, and the controller never compares `inst.UID` with `p.UID`. What gets deleted is still that Proxy's own instance. Exceptions: a tampered `status.providerID` (kvm2 `Delete` checks no mark, so `hosts/beelink/domains/docker-29` would destroy an unrelated VM); the reverse direction has no guard at all (kvm `Create` adopts any same-named domain, `kvm/kvm.go:177-181`; its `Get`/`Delete` check no mark); and after a re-type kvm2 `ensureOverlay` boots a stale overlay the kvm provider left behind (`create.go:171-177`), one Proxy's own disk only.
**3. Shared `base-<key>.<format>` cache — CONFIRMED** (kvm2 code + libvirt 11.6.0 source, `storage_driver.c`). The marker protocols are compatible (both write `<base>.sha256` with 64 hex; both key URL images as `sha256(url)[:16]`), so kvm2 adopts a kvm-cached base. On the host every base and marker dates from 2026-09-10 (kvm-made), both live overlays back onto `/srv/vms/images/base-debian-13.qcow2`, and the sample `kvm2-home` entry's `debian-13` key would share exactly that volume. Both integration tests use key `test-image`.
- **3a Digest mismatch.** `kvm-home` cached `debian-13` from a `latest` URL (digest D1). The operator gives `kvm2-home` `sha256: D2`. First kvm2 Create: `cachedImage` `image.go:129` → `:145` deletes marker and base. libvirt's `storageVolDelete` refuses only on `in_use`/`building` (`storage_driver.c:1844-1856`), not on backing-store use. kvm2 uploads a different image to the same path under the kvm provider's live overlays. With both entries pinned to different digests, every Create on either side deletes and refetches.
- **3b Concurrent cold fetch** (by source, not executed). Image locks are per `Provider` instance (`kvm2.go:111`; `kvm/image.go:62-74`), so this applies kvm-vs-kvm2 and kvm2-vs-kvm2 alike. libvirt holds `in_use` only while the stream is set up (`storage_driver.c:2421-2427`), so the volume is unguarded for the whole transfer. Entry B finds base-without-marker and deletes it (`kvm2/image.go:117-118`; `kvm/image.go:103-108`), recreates it and starts its own upload. A's upload finishes cleanly into the unlinked inode, the digest matches, A writes the marker, and `StorageVolGetPath` resolves A's handle to B's partial file: A boots a guest from a half-written base. If B is kvm2, `writeMarker` then fails `ErrStorageVolExist` and `image.go:241-243` discards the now-complete base under A's running domain.
**Minimal fix.** Give kvm2 its own prefix (`volume.go:17` → `kvm2-base-…`, one extra ~415 MB download per host). Reject two kvm/kvm2 entries with the same address+socket+pool at load, or treat `ErrStorageVolExist` on the marker as success when its digest equals ours. Put the digest in the base name when `sha256` is set. Make `removeDomain` require `meta.managed()` and the expected UID; give `kvm/kvm.go:177` the same guard. Add a fake keyed on the metadata URI (S7.4).
**Probe to settle 3b** (not run):
```sh
truncate -s 4G /tmp/probe-race.bin
virsh -c qemu:///system vol-create-as default probe-race.raw 4G --format raw
virsh -c qemu:///system vol-upload --pool default probe-race.raw /tmp/probe-race.bin &
sleep 1; virsh -c qemu:///system vol-delete --pool default probe-race.raw; echo "delete exit=$?"
wait; echo "upload exit=$?"; virsh -c qemu:///system vol-list default | grep probe-race; rm /tmp/probe-race.bin
```
## E4 — no cross-process fencing on the base volume — CONFIRMED (narrowed)
**Rollout / leader handover: REFUTED for the shipped manifests.** `manager.yaml:64` passes `--leader-elect` (the `false` default at `cmd/main.go:107` applies only to a bare binary or `make run`). No strategy is set, so RollingUpdate with surge 1: old and new pods coexist, but the reconciler is a leader-election runnable, `LeaderElectionReleaseOnCancel` is not set (`cmd/main.go:288-295`), and the old pod holds the lease until it dies. Lease 15 s / renew 10 s / retry 2 s; the kubelet SIGKILLs at 10 s (`manager.yaml:180`). On cancel kvm2 stops the upload at once (`backend.go:179-181` → `markDead`); `discardVolume` runs uncancelled for up to 30 s (`image.go:331`) but is cut by SIGKILL. The residual two-process window needs A's delete RPC to stall ~7 s while B dials, HEADs and creates within ~5 s: theoretical, and self-healing (`image.go:100`, `:135`). Other two-process cases: `make run` without `--leader-elect` against the cluster's hypervisor; `make test-integration-kvm` and `-kvm2` at the same time (same `test-image` key).
**In one process: CONFIRMED.** The kvm provider is an always-present second writer: same names (`kvm/image.go:44-57` vs `kvm2/volume.go:17-25`), same default pool, same marker format, separate locks (`kvm/image.go:77-87` vs `Provider.imageLocks`; `newProvider` builds fresh `keyedLocks` per entry, `kvm2.go:111`, so two kvm2 entries do not share one either), concurrent via `MaxConcurrentReconciles: 3`, documented as supported (`docs/kvm2-provider.md:6-8`) and shipped that way (`config/manager/providers_config.yaml`: both entries on `beelink`, key `debian-13`, default pool). The interleaving is E3-3b. Extra consequence: after A's marker is written, if the process dies during B's upload (rollout SIGKILL, OOM at the 128Mi limit) the pool keeps a **truncated base under a valid marker**, and nothing re-hashes the base (`cachedImage` compares marker text only), so every later Create on that host backs onto it until a manual `vol-delete`.
Caveat: that `StorageVolDelete` succeeds mid-stream came from memory here; the E3 reviewer read it in the 11.6.0 source, which corroborates it. Still unexecuted.
**Minimal fix.** (1) Remove the shared name (kvm2-specific prefix; include the entry name or share one process-wide `keyedLocks`). (2) Make the base appear atomically: upload to `<base>.part-<random>`, verify, then create the final name; treat `ErrStorageVolExist` as "someone else won" and re-check; never delete a name this call did not create, in particular not `discardVolume(base)` on a marker `ErrStorageVolExist` (`image.go:241-243`). (3) Store the size next to the digest and check the base's allocation before trusting the marker. Set `strategy: Recreate` and `LeaderElectionReleaseOnCancel` deliberately.
**Probe to settle** (not run):
```
virsh -c qemu:///system vol-create-as default e4probe.raw 2G --format raw
mkfifo /tmp/e4.fifo; ( head -c 2G /dev/zero | pv -L 20m > /tmp/e4.fifo & )
virsh -c qemu:///system vol-upload --pool default e4probe.raw /tmp/e4.fifo &
sleep 5
virsh -c qemu:///system vol-delete --pool default e4probe.raw # succeeds => confirmed path; "still in use" => narrowed
virsh -c qemu:///system vol-create-as default e4probe.raw 2G --format raw
virsh -c qemu:///system pool-refresh default
wait; virsh -c qemu:///system vol-delete --pool default e4probe.raw; rm /tmp/e4.fifo
```
## E5 — overlay atomicity under an abandoned RPC — REFUTED (~85%, libvirtd side from source memory)
`storageVolCreateXML` rejects a duplicate name, adds the volume with `building = true`, bumps asyncjobs, drops the pool lock, runs `qemu-img create` via `virCommandRun`; on build failure the file is unlinked and the volume removed from the list. It runs on an RPC worker thread holding a client reference, so a disconnect does not cancel it; storage objects have no close callbacks. After a client drop a name is either still building or complete. Pool refresh refuses while asyncjobs > 0, and kvm2's XML has no `<allocation>` and passes `flags = 0`, so the preallocating path is never taken. The spec's "creation is atomic" (`kvm2-provider-spec.md:227-229`) holds.
**Cost on the host.** All six pools are `type='dir'` on local storage. Three leftover overlays show birth-to-mtime gaps of 2-11 ms and are each exactly 196768 bytes, complete. A create over 30 s needs a stalled filesystem.
**Residual path** (loud, never boots a corrupt overlay): Create #1 stalls > 30 s, Create #2 gets `ErrStorageVolExist` while `building` is set; if it reaches `DomainCreate`, QEMU cannot open the file (write lock, or "not in qcow2 format") → Transient → next retry starts cleanly. If `qemu-img` ends in EIO, libvirtd removes the volume and the defined domain points at a missing file; every retry fails until the Proxy is deleted (this is sweep finding S2.4's mechanism).
**Fix options.** None needed for the hypothesis. Minimal: correct the comment at `create.go:151-153` and the spec line to state the real reasons. Hardening: `deleteVolumeIfPresent` before `StorageVolCreateXML` in `ensureOverlay`, as the seed already does (no domain exists at that point under the name lock; cost 2-11 ms; a building volume returns `ErrOperationInvalid`, already Transient).
**Probe to settle** (not run, ~420 MiB in `default`):
```sh
C=qemu:///system; V=review-e5-clone.qcow2
virsh -c $C vol-clone --pool default base-debian-13.qcow2 $V & P=$!
sleep 0.3; kill -9 $P
for i in $(seq 1 40); do virsh -c $C vol-info --pool default $V 2>&1 | tr '\n' ' '; echo; stat -c '%s' /srv/vms/images/$V 2>&1; sleep 0.5; done
virsh -c $C vol-dumpxml --pool default $V
until virsh -c $C vol-delete --pool default $V 2>&1 | grep -qv 'still being allocated'; do sleep 1; done
virsh -c $C vol-list default | grep review-e5- || echo clean
```
## E6 — go-libvirt when the socket is killed under it — (a) CONFIRMED, (b) REFUTED
Scratch module `scratchpad/lvleak/` (same go-libvirt and x/crypto versions; copies kvm2's `link`, `call`, `connect`, `Close`, `sshSocket` in shape).
**(a) Data race on `Libvirt.disconnected`, reported by `-race`** over a real in-process SSH `direct-streamlocal` channel: read in `waitAndDisconnect` (`libvirt.go:821`, goroutine started at `:297`), write in `ConnectToURI` (`:305`), unsynchronised. Interleaving: ConnectOpen reply is routed → socket dies → `listen` closes `s.disconnected` (`socket.go:208`) → `waitAndDisconnect` reads the old channel and returns (`:828`) → the requester installs a fresh channel at `:305` that nothing will ever close, and `Connect()` returns nil.
- **Hypothesised trigger (kvm2's own `markDead`): harmless.** `call` has already taken the timer/ctx branch; the nil result lands in the buffered `done` and is dropped; `hosts.go:153` discards the backend. Measured 5, 1 and 33 times per 1000 cycles, no effect.
- **Peer-close trigger: real defect.** If libvirtd or the forwarded socket goes away within ~1 ms of a successful open (e.g. a daemon restart), `Connect` wins, the watcher (`backend.go:229-236`) holds a channel that never closes, every later RPC fails at `SendPacket` with `syscall.EINVAL` (`socket.go:321-325`), which `errors.go:55` classifies Transient. No call result marks the link dead, `hostTable.get` (`hosts.go:75`) keeps returning it, and the SSH keepalive keeps succeeding if sshd is alive. The host answers "invalid argument" until the operator restarts. Wedged in 953/1000 (unix), 824/1000 (`net.Pipe`), 546/1000 (SSH), 186/500 (SSH, `-race`) when the peer closes right after the open reply. On the Check path the same loser only costs a 5.001 s `Close()`.
**(b) No accumulation.** Six scenarios (peer never answers then kill; RPC hangs; ConnectClose hangs into the close budget; peer closes mid-handshake; clean probe; kill racing the end of the handshake) x three transports, 1000 cycles each (500 under `-race`): goroutines +0, fds +0. Peak in flight 26.
**Side findings (need a misbehaving peer).** (i) `callback` does an unbuffered `c <- res` holding `cmux` (`rpc.go:76-84`): a duplicate reply deadlocks `listen`, leaks three goroutines per hit (+420 over 200 cycles), and a socket kill does not help. On the stream path serials do repeat, which is why the S4.1 leak also survives a kill. (ii) `socket.go:233-234` computes `int(length) - 28` and `make`s it: a length field under 28 panics "makeslice: len out of range" in the listen goroutine and takes the operator down (reproduced); a large bogus length allocates up to 4 GiB. Reachable if `host.Socket` points at something that is not libvirt.
**Minimal fix.** In `call` (`backend.go:177-178`): on `done`, if the error is `syscall.EINVAL` or `libvirt.ErrInterrupted`, `markDead` before returning; add a `backend_test.go` case where the peer closes right after the open reply. Sturdier: give `libvirt.New` a `net.Conn` wrapper whose `Read` calls `link.markDead` on any non-temporary error, dropping the dependence on `Disconnected()` and the 5 s Close stall. Upstream: assign `l.disconnected` before `go l.waitAndDisconnect()`; send outside the lock in `callback`; reject `length < 28` and oversized lengths.
## E7 — in-place base replacement under live overlays — PLAUSIBLE
**Confirmed part.** Admin sets or changes `images.<key>.sha256` and restarts the manager (config is not hot-reloaded). Next Create: `image.go:129` sees the mismatch, `:145` deletes marker and base with no in-use check and a `V(1)` log; `fetchImage` creates a new file at the same path. `TestEnsureImage_sha256ChangeRedownloads` (`image_test.go:394`) asserts exactly this. A mistyped `sha256` is worse: the base is deleted first, then `image.go:289` returns Permanent, leaving every old overlay with a missing backing file (loud). libvirt does not refuse (~95% from the reviewer; the E3 reviewer read it in source).
**Why corruption is only plausible.** Running VMs keep the old inode, and `on_reboot=restart` (`xml.go:217`) keeps the same QEMU process. Corruption needs a *new* QEMU process on an old overlay:
- Controller: safe. Stopped → Delete (`proxy_controller.go:216-225`) removes the overlay (`delete.go:91`).
- **`libvirt-guests` on host reboot.** Enabled on beelink (`multi-user.target.wants/libvirt-guests.service`); `/etc/default/libvirt-guests` could not be read. Stock `ON_BOOT=start` restarts every guest that was running at shutdown regardless of autostart; the reviewer is ~70% sure the distro default is also `ON_SHUTDOWN=suspend`. While the host is down `Get` is Transient, not Stopped, so nothing is deleted; after boot the operator normally sees Running and keeps the VM.
- A manual `virsh start`.
- Create adoption path (`create.go:73-87` calls `ensureRunning` without `ensureImage`): only when `providerID` is empty and the overlay already holds guest writes. Narrow.
The missing/unreadable-marker trigger is **not** reachable within kvm2 without an edit (marker errors are returned, not treated as unreadable, `image.go:119-125`). The second deleter that does exist is the kvm provider (E3/E4). Host state: both live domains running, persistent, autostart disabled, no managed save; the hazard is documented (`kvm2-provider-spec.md:91-93`, `kvm2-provider.md:313-318`), but the code contradicts the "add a new key" rule instead of enforcing it, and the guide's warning names `url` although a `url` change alone never refetches.
**Minimal fix.** At `image.go:129-130` do not delete; return `permanentf("base %s is cached with sha256 %s but %s is configured; add the image under a new key, …")`. Keep deletion for the no-marker and non-hex branches. Alternative: digest in the volume name. Either way log at `V(0)`.
**Probe to settle** (not run):
```
P=default; V="virsh -c qemu:///system"
$V vol-create-as $P review-e7-base.qcow2 16M --format qcow2
$V vol-create-as $P review-e7-overlay.qcow2 16M --format qcow2 --backing-vol review-e7-base.qcow2 --backing-vol-format qcow2
stat -c '%i' /srv/vms/images/review-e7-base.qcow2
$V vol-delete --pool $P review-e7-base.qcow2 # expected: succeeds
$V vol-create-as $P review-e7-base.qcow2 16M --format qcow2
stat -c '%i' /srv/vms/images/review-e7-base.qcow2 # expected: different inode
$V vol-delete --pool $P review-e7-overlay.qcow2; $V vol-delete --pool $P review-e7-base.qcow2
```
Read-only, for the owner: `grep -Ev '^\s*(#|$)' /etc/default/libvirt-guests` (empty output = defaults apply).
## E8 — pool-space precheck freshness — CONFIRMED, and staleness is the smaller of two errors
Measured 2026-09-17 08:28-08:30 UTC, read-only. kvm2's pool is `default` at `/srv/vms/images`.
| pool | filesystem | libvirt Available | `f_bfree` | `f_bavail` | libvirt − bfree | libvirt − bavail | vols |
|---|---|---|---|---|---|---|---|
| default (kvm2) | ubuntu--vg-vms--lv | 266,038,706,176 | 266,038,378,496 | 241,304,276,992 | +327,680 | **+24,734,429,184** | 24 |
| pool | same | 267,718,541,312 | 266,038,378,496 | 241,304,276,992 | +1,680,162,816 | +26,414,264,320 | 15 |
| vms | same | 267,718,541,312 | 266,038,378,496 | 241,304,276,992 | +1,680,162,816 | +26,414,264,320 | 9 |
| boot / images | ubuntu--lv | 73,877,696,512 | 73,918,029,824 | 68,532,543,488 | −40,333,312 | +5,345,153,024 | 0 / 2 |
| vms-2 | nvme1n1p1 | 820,622,606,336 | 820,602,261,504 | 770,575,294,464 | +20,344,832 | +50,047,311,872 | 2 |
- **Cached per pool.** Three pools on one filesystem report two different values; a call-time value could not differ. Pool A's allocation is invisible to pool B. Over 80 s `f_bfree` moved on the `images` filesystem while libvirt stayed frozen, 40 MB stale in the pessimistic direction, so the cache errs both ways. `default` is nearly exact only because libvirt touched it ~1.6 h earlier.
- **The root reserve.** libvirt reports `f_bfree`, which includes ext4's 5% reserve. Overlays are owned by `libvirt-qemu:kvm`, so guest writes stop at `f_bavail = 0` while libvirt still reports ≥ 24.7 GB.
**Failing path.** `create.go:92-100`: pool lookup → `imageSizeIfUncached` → `checkPoolSpace`, no refresh; `image.go:450-458` compares raw `available`; the only refresh is `image.go:235` after an upload, which never runs again once the base is cached. Overlays start at ~196 KB and grow invisibly (live ones at 1.47 and 1.54 GB of 10 GiB), as does `kube-node-33.qcow2` (50.6 GB) in the same directory. Reserve scenario (deterministic here): with `f_bavail` between 0 and ~10.7 GB the precheck passes for a 10 GiB disk, `ErrQuotaExceeded` never fires, and Create, the upload and the guests hit ENOSPC, the condition behind S4.1's wedge and S2.2's paused guests. `check.go:106-112` has the same defect: `poolAvailableGiB` over-reports by ~23 GiB plus drift.
**Minimal fix.** (1) `StoragePoolRefresh` immediately before `checkPoolSpace`, after `imageSizeIfUncached`; failure is Transient; only on the domain-not-defined branch. Cost: readdir + stat per volume (24 today), already paid at `image.go:235`. (2) Do not refresh in Check (a mutation every round on a shared host); document the semantics. (3) Add headroom to `neededBytes`: per-host `poolReserveGiB` or a fraction of `capacity` (already returned and discarded at `image.go:450`); or document `tune2fs -m 0`. (4) Make the fake's `StoragePoolGetInfo` return a cached value that only `StoragePoolRefresh` updates; add `TestCreate_refreshesPoolBeforeSpaceCheck`.
**Probe to settle** (for the owner): `virsh pool-info --bytes vms`, `virsh pool-refresh vms`, `pool-info` again; expect Available to drop ~1.68 GB and still sit ~24.7 GB above `df -B1 --output=avail /srv/vms`.
## E9 — libvirt error-code assumptions — (a) REFUTED, (b) CONFIRMED
**(a) Idempotency codes match libvirt 11.6.** Codes identified from libvirt's fixed per-code message prefix with `LIBVIRT_DEBUG=1`.
| Operation | kvm2 expects | Real | Evidence |
|---|---|---|---|
| `DomainLookupByName`, missing (`create.go:88`, `delete.go:57`, `get.go:34`) | 42 | 42 | observed |
| `StorageVolLookupByName`, missing (`image.go:117,134,176,424`) | 50 | 50 | observed |
| `StoragePoolLookupByName`, missing (`delete.go:84`) | 49 | 49 | observed |
| `DomainGetMetadata`, no kvm2 element (`get.go:81`) | 80 | 80 | observed on `docker-29`, `postgres`, and a kvm-v1-marked proxy domain; namespace scoping holds |
| `StorageVolCreateXML`, duplicate (`create.go:171`) | 90 | 90 | sourced, ~95% |
| `DomainDestroy` inactive (`delete.go:67`) | 55 | 55 | sourced, high |
| `DomainCreate` active (`create.go:236`) | 55 | 55 | sourced, high; `waitRunning` re-checks anyway |
| Destroy/Undefine vanished domain | 42 | 42 | sourced, high |
| `StorageVolDelete` vanished (`image.go:182`) | 50 | 50 | sourced, high |
`TestKVM2_lifecycle` would exercise 42 and 50 but not 90 (the second Create returns from the adoption branch before `ensureOverlay`), not 55 from Destroy, and a wrong 80 would not fail it (`list.go:39-45` logs and continues). Two fake-vs-real differences, neither breaking convergence: an overlay file unknown to libvirt returns 55 not 90; an inactive pool returns 55 from the marker lookup in `imageSizeIfUncached` (`create.go:96`) before `checkPoolSpace` (`create.go:100`), so the Permanent arm at `image.go:454-456` is unreachable through Create (`image_test.go:821-828` tests it by direct call).
**(b) Classification.** `classifyLibvirt` (`errors.go:58-71`) sends every unlisted code to Transient; `providerFailure` (`proxy_controller.go:344-345`) returns the error and sets **no condition**, so the Proxy shows nothing and retries under workqueue backoff, rerunning `writeSeed` each time when the failure is at define.
| Rejection | Where | Code | Evidence |
|---|---|---|---|
| Host has no usable KVM | define | 67 | sourced, high |
| No OVMF firmware | define | 9 | sourced, ~85% |
| Bridge missing | `DomainCreate` | 38 | sourced, high; `check.go` never looks at the bridge |
| host-passthrough unsupported | `DomainCreate` | 67 | sourced |
| Non-x86 host | define | 67 or 1 | observed 67 on the `domcapabilities --arch aarch64` path |
| Unknown machine type | define | 8 (already Permanent) | observed |
beelink rejects none of these. The bare `libvirt.ErrUnsupported` sentinel (`rpc.go:392-394`) bypasses `libvirtCode`; small impact. This follows a recorded decision: spec entry 13 (`kvm2-provider-spec.md:317`) "unknown means retry", and contract §5 makes Permanent sticky until the generation changes, so a host-side fix would not unlatch. The spec never looked at 67, and it already latches `ErrNoStoragePool`, which is also host-fixed.
**Minimal fix.** `ErrConfigUnsupported` → Permanent when `op == "create"`; `errors.Is(err, libvirt.ErrUnsupported)` → Permanent before the `libvirtCode` check; leave 9, 38, 1, 84 alone. Give the Transient arm of `providerFailure` a non-latching `Provisioned=False` condition with the message, or have the health checker read `ConnectGetDomainCapabilities` and check the bridge. In the integration test: destroy+undefine over `dialFresh` then Create again (exercises 90), destroy before step 9's Delete (exercises 55), assert "skipping domain with unreadable metadata" never logs.
**Probe to settle** (not run):
```
virsh -c qemu:///system vol-create-as default review-e9-dup.qcow2 1M --format qcow2 && \
LIBVIRT_DEBUG=1 virsh -c qemu:///system vol-create-as default review-e9-dup.qcow2 1M --format qcow2 2>&1 | grep ' error : ' ; \
virsh -c qemu:///system vol-delete --pool default review-e9-dup.qcow2
```
## E10 — overlay capacity below the base's virtual size — CONFIRMED (libvirt pass-through ~90%, unexecuted)
**Host numbers.** `base-debian-13.qcow2` and `base-test-image.qcow2`: libvirt capacity 3221225472 (3 GiB virtual), allocation 435683328. Live overlays: 10 GiB over that base. So the 10 GiB default has ~7 GiB margin; only `diskSizeGb: 1` or `2` breaks Debian 13 (Ubuntu ~3.5 GiB and Fedora ~5 GiB from memory). libvirt reports the true virtual size because of the refresh at `image.go:235`.
**Local experiment** (qemu-img 10.2.1; host has 10.1.0): `qemu-img create -f qcow2 -b <64M base> -F qcow2 short.qcow2 16384K` exits 0, virtual size 16 MiB, `qemu-img check` clean; reads inside 16 MiB match the base, a read at 16M fails `Input/output error`. A 64M GPT base under a 16M overlay: `sfdisk` sees only the protective MBR, `sgdisk -v` reports the disk smaller than the header says and the backup header in error. The Linux kernel rejects such a GPT, so there is no root device; the guest would load GRUB and the kernel then drop to initramfs (inferred, no VM booted). libvirt's `virStorageBackendCreateQemuImgCmdFromVol` appends the capacity with no size comparison (from source memory).
**Failing path.** `diskSizeGb: 2` passes validation (`config_kvm.go:133` rejects only negatives; `kvm2.go:127-128` changes only 0) → `checkPoolSpace` with the small number (`create.go:100`) → base cached (`:103`) → `ensureOverlay` sends `diskSizeGB << 30` (`create.go:107,158`; `volume.go:72` checks only positive) → domain defined and started, `waitRunning` sees Running, Create succeeds (`:148,260`) → `Get`: active, no address → `StateProvisioning` (`state.go:20-21`) → requeue at `ProvisioningPoll` forever (`proxy_controller.go:208-211`; there is no provisioning deadline). Correction to the hypothesis: the Proxy is stuck Provisioning with "waiting for the instance to run", not reported healthy. Nothing points at `diskSizeGb`, and raising it later does not repair that Proxy (`create.go:75-87`, `:171-177`). The kvm provider has the same hole (`kvm/volume.go:24-32`).
**Minimal fix.** Add `StorageVolGetInfo` to `backend` (libvirtBackend, `observed`, fake already stores `fakeVol.capacity` at `fake_test.go:494`). Between `ensureImage` and `ensureOverlay`, if the base's capacity exceeds `DiskSizeGB << 30` return `permanentf("kvm2.diskSizeGb %d is smaller than image %s's virtual size …")`. One or two RPCs on the first Create only. Run the same check on a reused overlay and recreate it when short. `TestCreate_diskSmallerThanImage`. One sentence in the `diskSizeGb` doc rows. A config-time check is not possible.
**Probe to settle** (not run; one ~200 KiB volume, self-cleaning):
```bash
V=review-e10-short.qcow2; P=default
virsh -c qemu:///system vol-create-as $P $V 1G --format qcow2 \
--backing-vol /srv/vms/images/base-test-image.qcow2 --backing-vol-format qcow2; echo "create exit=$?"
virsh -c qemu:///system vol-dumpxml --pool $P $V | grep -E "capacity|backingStore|<path>"
virsh -c qemu:///system vol-delete --pool $P $V; echo "delete exit=$?"
virsh -c qemu:///system vol-list $P | grep review-e10- || echo "clean"
```
docs/reviews/2026-09-17-kvm2-provider-review/escalations.md
+102
View File
@@ -0,0 +1,102 @@
# kvm2 provider review: escalations for phase 2
18 raw Escalate items from the seven sweepers, merged to 15 root hypotheses, top 10 kept.
Ranked by blast radius x uncertainty. `deep-reviewer` is static (Read, Grep, Glob, Bash; it
can read the Go module cache and run local tests, but not touch the hypervisor), so each item
says whether it can be **settled statically** or is likely to end PLAUSIBLE without a probe on
beelink. Finding IDs (`S…`) refer to `sweep.md`.
| # | Hypothesis in one line | Blast | Uncertainty | Static? |
|---|---|---|---|---|
| E1 | One wedged `markDead` starves every reconcile worker, not just one host | operator-wide stall | medium | yes |
| E2 | A reformatted GC mark makes GC delete a live Proxy's VM | live VM destroyed | medium | mostly |
| E3 | kvm and kvm2 on one hypervisor: cross-provider destruction beyond the Get/Delete hole | other provider's VMs/disks | medium | yes |
| E4 | No cross-process fencing on the base volume: leader handover deletes a base mid-upload | corrupt base under new overlays | medium | yes |
| E5 | An abandoned `StorageVolCreateXML` leaves a corrupt overlay that the next Create adopts | silent guest disk corruption | high | partly |
| E6 | go-libvirt leaks or races when the socket is killed mid-connect; per-round check dials accumulate | slow leak in long-running leader | medium | yes |
| E7 | In-place base replacement corrupts existing overlays on their next boot | guest disk corruption, many VMs | low-medium | partly |
| E8 | Pool-space precheck reads a stale `available`; quota backoff never engages | host pool fills | medium | partly |
| E9 | libvirt error-code assumptions: idempotency codes and define-time rejection codes | retry loops that never converge | medium | partly |
| E10 | Overlay capacity below the base's virtual size is accepted and truncates the disk | silent guest corruption | high | no |
---
## E1 — A wedged `markDead` blocks name locks and starves the reconcile workers
Sources: chunk 1 Escalate 2; builds on finding S1.1.
**Entry points:** `internal/provider/kvm2/kvm2.go:186-243` (`lockName`, `keyedLocks`), `create.go`, `delete.go`, `backend.go:138-188`, `hosts.go:66-95`, `internal/gc/gc.go`, `internal/controller/proxy_controller.go` (+ `SetupWithManager`, `MaxConcurrentReconciles`), `cmd/main.go` manager wiring.
**Hypothesis:** a goroutine blocked inside `markDead` (S1.1) holds the per-name lock taken by Create/Delete, and reconcile and GC contexts carry no deadline, so `keyedLocks.acquire`'s ctx escape never fires. Delete/GC for that name block indefinitely instead of requeueing, and with a small `MaxConcurrentReconciles` a single silent host can pin every worker, stalling all Proxies on all providers.
**Checked:** lock acquisition is ctx-aware and refcounted correctly; only `internal/providerhealth/monitor.go:180` sets a deadline; the health checker and image upload use `dialFresh` and are insulated; `markDead` at `hosts.go:153` runs under `e.sem`.
**Unknown:** whether the controller-runtime work queue tolerates a permanently blocked Reconcile for one key (worker starvation across all Proxies); the actual `MaxConcurrentReconciles` and any reconcile timeout in the manager wiring; whether GC's single goroutine is blocked the same way; how many distinct call sites can enter `markDead` while holding `e.sem` or a name lock.
## E2 — GC mark parse fragility: a reformatted mark makes GC delete a live VM
Sources: chunk 3 Escalate 1; finding S5.3.
**Entry points:** `internal/provider/kvm2/metadata.go:44-78` (`createdAt`, `parseInstanceMetadata`), `list.go:49`, `get.go:79-88`, `internal/gc/gc.go:113-121`.
**Hypothesis:** if libvirt hands `<uid>` or `<createdAt>` back with surrounding whitespace or re-indented content (another libvirt version, or an admin's `virsh edit`/`virsh define` through a reformatting tool), `UID` stops matching `live[inst.UID]` (exact string compare, `gc.go:118`) and/or `createdAt()` returns the zero time, which makes the instance immediately past `MinAge`. GC then deletes a live Proxy's VM. No trim or tolerant parse exists as a backstop.
**Checked:** `newInstanceMetadata` writes a single-line element; `kvm2_integration_test.go:422-449` asserts the UID round-trips and `CreatedAt` is non-zero (sweeper's claim that this ran against real libvirt 11.6 was wrong: phase 2 found the kvm2 integration test has not yet run on beelink, see `deep.md`); libxml2 does not re-indent an element whose children have text content.
**Unknown:** behaviour on other libvirt versions and after `virsh edit`; whether `encoding/xml` unmarshal plus the current struct tags already tolerate whitespace; whether a UID mismatch alone (timestamp intact) is enough for deletion, and what GC does with `UID == ""`; whether GC has any second guard (name-to-Proxy lookup) before `Delete`.
## E3 — kvm and kvm2 on the same hypervisor
Sources: chunk 3 Escalate 2, chunk 7 Escalate 4; builds on findings S2.3, S7.1, S7.4.
**Entry points:** `internal/provider/kvm2/list.go:20-56`, `get.go:52-88`, `delete.go:55-98`, `volume.go:14-18`, `internal/provider/kvm/kvm.go:379`, `internal/provider/kvm/id.go:11-13`, `internal/provider/kvm/volume.go:12`, `internal/provider/kvm/image.go:46`, `internal/gc/gc.go:86-131`, `internal/provider/config.go:108-130`, `internal/controller/proxy_controller.go:197,237,287`, `docs/kvm2-provider.md:346-370`.
**Hypothesis:** the only thing keeping kvm2's `ListByTag` from handing kvm-made domains to GC is libvirt's namespace-scoped `DomainGetMetadata`; both providers also share `<name>.qcow2`, `<name>-seed.iso`, `base-<key>.<format>` names and the providerID format. Beyond the confirmed Get/Delete hole (S2.3), GC's per-provider scoping may let provider A's list be reconciled against provider B's Proxies, and a Proxy moved between a kvm and a kvm2 entry (same UID, same domain name) may have one side delete volumes the other side's domain uses.
**Checked:** domain names are UID-derived so one Proxy cannot exist under both at once; the two metadata URIs differ; `ListByTag` is mark-filtered; `spec.provider` is immutable but an entry's `name` is independent of its `type`; the fake cannot prove the namespace rule (S7.4).
**Unknown:** whether GC builds its live-UID set per provider or globally, and what it deletes when the same UID is visible through two entries; whether anything (config reload guard, admission check on ProviderID, deploy tooling) prevents a re-typed entry from inheriting live `status.ProviderID`s; the reverse direction (kvm v1 `Get`/`Delete`/`ListByTag` against kvm2-made domains).
## E4 — No cross-process fencing on the base volume
Source: chunk 4 Escalate A.
**Entry points:** `internal/provider/kvm2/image.go:85-146` (`ensureImage`, `lockImage`, `cachedImage`), `image.go:200-300` (`fetchImage`, `uploadImage`), `kvm2.go:199-243`, `cmd/main.go:107` (leader election flags), `config/manager/manager.yaml:24`.
**Hypothesis:** process A is inside `fetchImage` (base created, marker not yet written); process B's `cachedImage` sees base-without-marker and deletes it (`image.go:118`), then its `StorageVolCreateXML` either fails `ErrStorageVolExist` or succeeds against a half-uploaded file. A's stream writes into an unlinked inode and A writes a marker for a base that no longer exists. An overlay created in between backs onto a deleted or wrong-content path.
**Checked:** `keyedLocks` is process-local; `replicas: 1`; `--leader-elect` defaults to false; `Delete` never touches base volumes; nothing else in the repo names base or marker volumes.
**Unknown:** whether a leader-election handover or a rolling update (old pod still inside a 20 minute `fetchImage` while the new pod reconciles the same Proxy) reaches this with the shipped manifests; whether the manager cancels in-flight reconciles on lost lease and how long `terminationGracePeriodSeconds: 10` leaves the old process uploading; whether B's marker written over A's bytes can ever validate.
## E5 — Overlay atomicity under an abandoned RPC
Source: chunk 2 Escalate 1.
**Entry points:** `internal/provider/kvm2/create.go:151-181` (`ensureOverlay`), `backend.go:157-188` (`call`, budget, `markDead`), `backend.go:319-321` (`StorageVolCreateXML` under `budgets.rpc` = 30 s).
**Hypothesis:** if `StorageVolCreateXML` for the overlay exceeds the 30 s budget or the caller's ctx ends, `call` kills the SSH transport mid-RPC. If libvirtd still registers the volume, possibly partial, the next Create hits `ErrStorageVolExist`, logs "overlay volume already exists" and defines a domain over a corrupt overlay. The base image has a marker and the seed is rebuilt unconditionally for exactly this reason; the overlay has neither.
**Checked:** the reuse branch distinguishes only `ErrStorageVolExist`; no size, format or backing-path verification anywhere; `Delete` is the only thing that removes an overlay.
**Unknown:** libvirtd's storage-driver behaviour when the client drops mid `virStorageVolCreateXML` (unlink vs leave registered); whether overlay creation can exceed 30 s in normal operation (NFS, LVM, preallocating filesystems) rather than only on cancellation; whether a cheap verification (`StorageVolGetInfo` capacity, or delete-and-recreate when no domain references it) is available through the existing `backend` seam.
## E6 — go-libvirt internals when the socket is killed under it
Sources: chunk 1 Escalate 1, chunk 6 Escalate 1.
**Entry points:** `internal/provider/kvm2/backend.go:225-238` (`libvirtBackend.connect`, the `Disconnected()` watcher), `backend.go:351-367` (`Close`), `hosts.go:91-162` (`dialFresh`, `connect`), `check.go:85-119`; upstream `go-libvirt@v0.0.0-20260814190004-1a83157e1858`: `libvirt.go:286-305` (`ConnectToURI`), `:821/:834` (`waitAndDisconnect`), `socket/`.
**Hypothesis:** (a) when `call(…, "Connect", …)` expires and `markDead` closes the socket just as `ConnectToURI` reaches `l.disconnected = make(chan struct{})`, there is a data race on that field against `waitAndDisconnect`, and the loser leaves a `Libvirt` whose `Disconnected()` never fires; (b) `Check` opens a fresh connection per host per round (~1440/day/host), so any goroutine or fd go-libvirt leaves behind on the killed-socket path (abandoned Connect, or `Close` hitting its 5 s budget mid `ConnectClose`) accumulates linearly in a long-running leader.
**Checked:** kvm2's own goroutines are clean (watcher selects on `link.closed`, keepalive exits on `stop`, `sshSocket.Close` is once-guarded); `Disconnected()` is read only after a successful Connect and the backend is discarded on failure; `go test -race -count=2` of the unit tests is clean.
**Unknown:** whether `waitAndDisconnect`/`listenAndRespond` always terminate when the conn is closed mid-handshake; whether the socket layer leaves an fd; whether a looped dial-and-kill test (N=1000, sampling `runtime.NumGoroutine()`) against the in-process SSH test server shows growth. Note finding S4.1 already confirms one permanent go-libvirt goroutine leak on the stream path; this asks whether the connect/close path has another.
## E7 — In-place base-image replacement under overlays that pin its path
Source: chunk 2 Escalate 2; the code behaviour is finding S4.6 (three sweepers).
**Entry points:** `internal/provider/kvm2/create.go:103,154-165`, `image.go:113-146` (`cachedImage`), `image.go:235` (`StoragePoolRefresh`), `docs/provider-contract.md` ("Catalogue changes do not replace instances").
**Hypothesis:** when `images.<key>.sha256` changes (or the marker is missing or unreadable), the base is deleted and different content is downloaded to the same deterministic path while every earlier overlay records that path as its qcow2 backing file. Any such overlay that is later restarted (adoption path, host reboot, `ensureRunning` on a shutoff domain) reads mismatched backing data: guest filesystem corruption across every VM on that key.
**Checked:** `Delete` never removes the base; the path is stable per key+format; `BackingPath` is resolved once at define time and never re-verified; the contract explicitly allows catalogue edits not to replace instances.
**Unknown:** which restart paths actually exist for an existing overlay in this operator (does anything restart a shutoff domain rather than delete and recreate it, given Stopped → Delete in the controller?); whether the missing/unreadable-marker trigger can fire without any admin edit (e.g. after E4 or a failed `writeMarker`), which would make this reachable in normal operation; whether libvirt refuses `StorageVolDelete` on a volume in use as a backing store.
## E8 — Pool-space precheck freshness
Source: chunk 4 Escalate B.
**Entry points:** `internal/provider/kvm2/image.go:419-467` (`imageSizeIfUncached`, `checkPoolSpace`, `neededBytes`), `create.go:96-106`, `check.go:85-119` (also reads pool info).
**Hypothesis:** for a `dir` pool, `StoragePoolGetInfo.available` is what libvirtd computed at the last pool refresh, adjusted only for creates and deletes it performed; guest writes growing sparse overlays do not update it. `create.go` never issues `StoragePoolRefresh` before the check (the only refresh is after an upload, `image.go:235`), so on a host its VMs have filled, the precheck reports tens of GB free indefinitely, `ErrQuotaExceeded`/`QuotaRetry` never engages, and Create fails later with an ENOSPC-flavoured transient, which is also the case that wedges the upload stream (S4.1) and pauses guests (S2.2).
**Checked:** no refresh anywhere on the pre-create path; `available` is used raw; unit tests set the fake's value directly; the spec describes the precheck but says nothing about freshness.
**Unknown:** libvirt 11.x dir-pool semantics for `available` (statvfs at call time vs cached); whether `Check`'s `poolAvailableGiB` has the same staleness; the cost of a per-Create refresh on the shared pooled connection.
## E9 — libvirt error-code assumptions
Sources: chunk 5 Escalate 2, chunk 7 Escalate 2; findings S3.5, S5.2.
**Entry points:** `internal/provider/kvm2/errors.go:33-97` (`classify`, `classifyLibvirt`, `libvirtCode`, `hasCode`), `create.go:139-142,166-177`, `get.go:79-88`, `delete.go:55-98`, `image.go:174-186`, `fake_test.go:491-493`, `kvm2_integration_test.go`.
**Hypothesis:** two halves of one assumption. (a) Idempotency depends on exact codes: `ensureOverlay` needs `ErrStorageVolExist` for a duplicate volume on a dir pool, `readMetadata` needs `ErrNoDomainMetadata`, the delete paths need `ErrNoDomain`/`ErrNoStorageVol`. The fake returns exactly the codes the code expects, so unit tests cannot refute a mismatch; if real libvirt returns `ErrOperationFailed` or `ErrInternalError` for any of them, a retried Create after a partial failure never converges. (b) Every define-time rejection of the generated XML (arch, machine, firmware feature, host-passthrough) surfaces as `ErrConfigUnsupported` (67) or `ErrOperationUnsupported` (84), which default to Transient: an unbounded retry with no `ReasonPermanentError`.
**Checked:** only `ErrNoStoragePool`, `ErrXMLError`, `ErrNoSupport`, `ErrInvalidArg`, `ErrAuthFailed`, `ErrAccessDenied` map to Permanent; 67 and 84 exist in go-libvirt's const set and are unlisted; bare `libvirt.ErrUnsupported` bypasses `libvirtCode`; transient errors requeue forever.
**Unknown:** which idempotency codes the integration test actually exercises against libvirt 11.6 (a second Create on an existing overlay? Delete twice? Get on a foreign domain?), which settles (a) for those paths; for (b), which code libvirt really returns for each rejection, and whether keeping unknown codes Transient is a deliberate spec decision. Mapping too much to Permanent latches Proxies that would recover.
## E10 — Overlay capacity smaller than the backing image's virtual size
Source: chunk 5 Escalate 1; finding S5.6.
**Entry points:** `internal/provider/kvm2/volume.go:68-89` (`buildVolumeXML`), `create.go:107,154-181`, `image.go:449-467`, `internal/provider/config_kvm.go:133` (`diskSizeGb` validation).
**Hypothesis:** with `kvm2.diskSizeGb` below the base image's virtual size, libvirt runs `qemu-img create -b <base> … <capacity>` and succeeds, producing a truncated virtual disk whose partition table overruns the device. The VM fails to boot or boots with a corrupt root filesystem, and kvm2 reports a healthy running domain that never answers.
**Checked:** nothing in the package compares overlay capacity to the base's virtual size; the validator rejects only `diskSizeGb < 0`; the 10 GiB default exceeds current Debian/Ubuntu/Fedora cloud images.
**Unknown:** whether libvirt's storage backend or qemu-img errors on a short capacity (then this is only a poor error message). Needs a real qemu-img; the deep reviewer can at most establish whether kvm2 has any way to learn the base's virtual size through the existing seam (`StorageVolGetInfo` on the base) and what a guard would cost. **Expect PLAUSIBLE.**
---
## Dropped (not in the top 10)
- **D1** Deleting a volume while a libvirt upload stream is open on it (chunk 4 C): a consequence of confirmed finding S4.1, only changes whether its disk-space effect is real; needs a truncated-source probe on beelink.
- **D2** cloud-init dispatch of a `text/plain` part inside vendor-data (chunk 5 E3): invisible-failure risk but only a boot test on a real image settles it; nothing static to add beyond S5.8.
- **D3** ProviderHealth status churn every round waking watchers (chunk 6 E2): low blast radius, arguably intended per the code comment.
- **D4** Nested volumeMount, `kvm-ssh` Secret inside the read-only `providers-config` mount (chunk 7 E1): pre-existing and shared with kvm; refuted if kvm has ever run in-cluster with the Secret present, otherwise one `kubectl apply` settles it.
- **D5** Whether provider error strings (with the full image URL, S7.10) reach Proxy status conditions (chunk 7 E3): low blast radius; cheap and fully static if you want it added.
docs/reviews/2026-09-17-kvm2-provider-review/scope.md
+133
View File
@@ -0,0 +1,133 @@
# kvm2 provider review: scope
Branch `feat/kvm2-provider` at `2510842`. Seed: `grep -ril 'kvm2\|libvirt'` excluding vendor,
generated, testdata, docs. The minikube layout hints in the brief do not apply; this repo is
the egress-proxies operator and kvm2 is `internal/provider/kvm2/`, a pure-Go provider
(`digitalocean/go-libvirt` RPC over an SSH-tunnelled socket, no cgo, no driver plugin binary).
Checklist items with no counterpart here, so reviewers can answer them quickly rather than hunt:
- libvirt **network** lifecycle / DHCP / static IP: kvm2 creates no networks; it attaches to a
configured bridge and reads the address via `DomainInterfaceAddresses`. The live questions are
IP detection, timeouts and stale addresses.
- GPU/PCI/NUMA passthrough: none.
- driver plugin RPC boundary: none. Closest analogue is the libvirt RPC link over SSH
(`link`, `call`, budgets, dead-link detection, libvirt version differences).
- cgo/build tags: no cgo. Only tag is `//go:build integration`.
- Handle leaks: go-libvirt handles are plain values (no `Free`); the leakable resources are
SSH connections, the libvirt link, goroutines (keepalive, `await`, `call`), HTTP bodies,
temp dirs and half-written storage volumes.
## CORE (kvm2-owned)
`internal/provider/kvm2/` — 18 source files (~3.2k lines), 22 test files (~7.3k lines).
| File | Lines | What |
|---|---|---|
| kvm2.go | 243 | `Provider`, `New`, options, `withDefaults`, `host`, `wrapErr`, `lockName`, `keyedLocks` |
| ssh.go | 216 | `dialStage`/`dialError`, `loadCredentials`, `sshSocket` + keepalive, `dialSSHSocket`, `await` |
| hosts.go | 162 | `dialer`, `hostTable` (per-host cached backend, `get`/`dialFresh`), `libvirtDialer` |
| backend.go | 369 | `backend` interface, `budgets`, `link` (dead marking), `call`/`run`, `libvirtBackend` |
| observe.go | 263 | `observed` backend decorator: span + log per libvirt call |
| create.go | 280 | `Create`, `create`, `ensureOverlay`, `writeSeed`, `ensureRunning`, `waitRunning` |
| delete.go | 98 | `Delete`, `removeDomain`, `removeVolumes` |
| get.go | 119 | `Get`, `get`, `readMetadata`, `address` |
| list.go | 56 | `ListByTag` |
| state.go | 49 | `mapState`, `isActive`, `firstIPv4` |
| id.go | 28 | `formatProviderID`, `parseProviderID` |
| metadata.go | 78 | GC mark in domain `<metadata>`: marshal/parse, `managed`, `createdAt` |
| errors.go | 97 | `permanentf`/`transientf`, `classify`, `classifyLibvirt`, `libvirtCode`, `hasCode` |
| image.go | 467 | `resolveImage`, `lockImage`, `ensureImage`, `cachedImage`, marker read/write, `fetchImage`, `uploadImage`, `do`, `checkResponse`, pool space checks |
| volume.go | 95 | volume naming, `urlImageKey`, `buildVolumeXML` |
| xml.go | 256 | `domainParams`, `buildDomainXML` and the XML structs |
| seed.go | 124 | `buildSeedISO` (temp dir, iso9660), `buildVendorData` |
| check.go | 189 | `Check`, `checkHost`, `probeHost`, `checkReason`, formatting helpers |
Tests (each reviewed with its source file): `*_test.go` next to every file above, plus
`provider_test.go` (contract tests), `fake_test.go` (fake backend), `sshserver_test.go`
(in-process SSH server), `kvm2_integration_test.go` (`//go:build integration`, `TestKVM2*`).
Other kvm2-owned bits:
- `cmd/main.go:62,206` — import and `"kvm2": kvm2.New` registration
- `Makefile:73-75` — `test-integration-kvm2`
- `config/samples/proxy_kvm2.yaml`, `config/samples/kustomization.yaml:8`
- `config/samples/providers-config.yaml:58-`, `config/manager/providers_config.yaml:54-` (commented kvm2 entries)
Reference only, not under review: `docs/kvm2-provider-spec.md`, `docs/kvm2-provider.md`,
`docs/provider-contract.md`. Reviewers may read them to judge code against stated intent.
## SHARED (on the kvm2 call path; listed symbols only)
Called by kvm2:
- `internal/provider/config_kvm.go` — `KVMConfig`, `KVMImage`, `KVMMachineType`, `KVMHost`,
`KVMFormat*`/`KVMFirmware*` consts, `IsHTTPURL`, `validateKVM`, `validateKVMBlock`
(the validator is what stands between user YAML and kvm2's XML/URL/volume-name building)
- `internal/provider/config.go` — `ProviderConfig.KVM2`, `setBlocks`, the `kvm2` arms of `Config.validate`
- `internal/provider/provider.go` — `Provider` interface, `CreateRequest`, `Placement`, `Instance`,
`InstanceState` + `State*` consts, `LabelManagedYes`
- `internal/provider/errors.go` — `Error`, `Wrap`, `Class`, `ErrTransient`/`ErrPermanent`/`ErrNotFound`/`ErrQuotaExceeded`
- `internal/provider/check.go` — `Checker`, `TargetStatus`, `Reason*` consts
- `internal/provider/name.go` — `NameFromUID` (produces the domain name kvm2 puts in XML and volume names)
- `internal/tracing` — `StartSpan`, `Tracer`, `Option`
Calling into kvm2:
- `internal/controller/proxy_controller.go` — `reconcileManaged` (Create/Get/Delete calls at
156/197/218), `replaceInstance`, `reconcileDelete`, `providerFailure`, `deletionFailure`,
`resolveCloudInit`, `placementFrom`
- `internal/gc/gc.go` — `Sweeper.sweep` (`ListByTag` at 105, `Delete` at 125)
- `internal/providerhealth/monitor.go` — `Monitor.checkOnce`, `Monitor.run` (`Check` at 214; the timeout it passes)
- `internal/provider/registry/registry.go` — `Constructor`, `Build`
- `internal/provider/metrics.go`, `internal/provider/tracing.go` — `WithMetrics`/`WithTracing`
decorators (`instrumented`, `traced`): error passthrough, and that they hide `Checker`
- `cmd/main.go:196-225` — registry build, checker extraction from unwrapped providers, wrapping
- `config/manager/manager.yaml:150-178` — `kvm-ssh` Secret mount, `/tmp` emptyDir (64Mi) that `buildSeedISO` uses
## EXCLUDED but considered
- `internal/provider/kvm/` — the older kvm provider; kvm2 is a clean-room rewrite and imports nothing from it.
- `internal/provider/config_kvm_test.go`, `config_test.go` — reviewed only for kvm2 cases, inside chunk 7; rest is kvm.
- `internal/provider/gcp/`, `internal/provider/kubernetes/` — other providers, not on a kvm2 path.
- `internal/health`, `internal/discovery`, `internal/lease` — consume `Proxy` status, never call the provider.
- `internal/metrics`, `internal/tracing` internals — only the three entry points above are on the path.
- `internal/providerhealth/envtest_test.go` — mentions kvm2 only as a type string in a fixture.
- `api/v1alpha1/providerhealth_types.go:71`, `config/crd/bases/...providerhealths.yaml` — comment-only / generated.
- `go.mod`, `go.sum` — dependency pins (`go-libvirt` 20260814 pseudo-version, `iso9660` v0.4.0, `x/crypto` v0.54.0); noted, not reviewed. Third-party source may be read to understand a call, not reported on.
- `config/samples/proxy_kvm.yaml`, `Makefile:69-71,185-196` (`test-integration-kvm`, `kvm-ssh-secret`) — kvm-only or shared tooling with no kvm2 logic. `kvm-ssh-secret` is mentioned in chunk 7 for context only.
- `README.md`, `CHANGELOG.md`, `AGENTS.md`, `.claude/settings.json`, `docs/` — prose / tooling.
- `.gitea/workflows/build.yaml`, `Dockerfile` — no kvm2-specific content; integration test is not run in CI (worth stating in chunk 7, not a file to review).
## Proposed phase 1 chunking (7 chunks)
1. **Connection layer (SSH + libvirt link)**
Files: `ssh.go`, `hosts.go`, `backend.go`, + `ssh_test.go`, `sshserver_test.go`, `hosts_test.go`, `backend_test.go`.
SHARED in play: `KVMConfig` (SSHKeyFile, KnownHostsFile, InsecureIgnoreHostKey), `KVMHost`.
Focus: connection/goroutine leaks on error paths, dead-link detection and redial races in `hostTable`, budgets vs caller context, keepalive, host key handling.
2. **Domain lifecycle (Create / Delete)**
Files: `kvm2.go`, `create.go`, `delete.go`, `id.go`, + `kvm2_test.go`, `create_test.go`, `delete_test.go`, `id_test.go`, `provider_test.go`.
SHARED in play: `CreateRequest`, `Placement`, `NameFromUID`; `reconcileManaged`, `replaceInstance`, `reconcileDelete` (how Create/Delete are retried).
Focus: partial-create cleanup, idempotency, foreign-domain refusal, half-deleted domains, `keyedLocks` correctness, `waitRunning` timeout/cancellation.
3. **Observation, state and error classification (Get / ListByTag)**
Files: `get.go`, `list.go`, `state.go`, `errors.go`, + `get_test.go`, `list_test.go`, `state_test.go`, `errors_test.go`.
SHARED in play: `Instance`, `InstanceState`, `Error`/`Wrap`/`Class` and the `Err*` classes; `Sweeper.sweep`, `providerFailure`, `deletionFailure`.
Focus: libvirt state mapping (paused/crashed/pmsuspended/shutoff), IP detection and stale addresses, NotFound vs transient, one dead host failing the whole list and what GC does with a partial list.
4. **Base image cache and volumes**
Files: `image.go`, `volume.go`, + `image_test.go`, `volume_test.go`.
SHARED in play: `KVMImage`, `KVMFormat*`, `IsHTTPURL`, `validateKVMBlock` (image key charset).
Focus: download/upload streaming, sha256 verify + marker protocol, cross-replica and cross-Create races on the same base volume, cleanup of partial volumes, URL handling (SSRF, redirects, size caps), pool space check, volume-name injection via URL/key.
5. **XML and cloud-init seed templating**
Files: `xml.go`, `seed.go`, `metadata.go`, (`buildVolumeXML` in `volume.go`), + `xml_test.go`, `seed_test.go`, `metadata_test.go`.
SHARED in play: `KVMMachineType`, `KVMHost` (bridge, pool), `KVMFirmware*`, `CreateRequest.CloudInit`, `NameFromUID`; `resolveCloudInit`; `manager.yaml` `/tmp` emptyDir.
Focus: escaping of config- and user-supplied values, validity across libvirt versions, metadata parse robustness (GC trusts it), temp-dir cleanup and size limit, vendor-data merging.
6. **Health check and observability**
Files: `check.go`, `observe.go`, + `check_test.go`, `observe_test.go`.
SHARED in play: `Checker`, `TargetStatus`, `Reason*`; `Monitor.checkOnce`/`run`; `tracing.StartSpan`/`Tracer`; `WithMetrics`/`WithTracing` decorators; `cmd/main.go:213-225`.
Focus: dedicated check connections closed on every path, check timeout vs dial budget, reason mapping, secrets or cloud-init leaking into logs/spans, span end on every path.
7. **Config, registration, deployment and test gating**
Files: `internal/provider/config.go` + `config_kvm.go` (kvm2 arms), their tests' kvm2 cases, `internal/provider/registry/registry.go`, `cmd/main.go:196-225`, `config/manager/manager.yaml:150-178`, `config/manager/providers_config.yaml`, `config/samples/providers-config.yaml`, `config/samples/proxy_kvm2.yaml`, `Makefile:73-75`, `kvm2_integration_test.go`, `fake_test.go`.
SHARED in play: `validateKVMBlock`, `setBlocks`, `Config.validate`, `registry.Build`, `Constructor`.
Focus: validation gaps that let bad values reach chunks 4/5, `New` behaviour with a nil block or unreadable key, Secret mount perms, integration test gating/cleanup and whether the fake backend diverges from real libvirt in ways that hide bugs.
docs/reviews/2026-09-17-kvm2-provider-review/sweep.md
+216
View File
@@ -0,0 +1,216 @@
# kvm2 provider review: phase 1 sweep
Seven `sweep-reviewer` runs, one per chunk in `scope.md`, against `feat/kvm2-provider` at `2510842`.
Findings below are as the sweepers reported them (they state each was verified in code); the
orchestrator merged duplicates and ranked, and did not re-read code. 54 raw findings, 49 after
merging. IDs are `S<chunk>.<n>`; "also: chunk N" marks a finding reported independently by
another sweeper.
Severity counts after dedupe: **high 4, medium 17, low 28**.
Recurring themes across chunks:
- **T1 Unknown or unmapped errors default to Transient, so permanent misconfigurations retry forever**
(S1.4, S2.5, S3.7, S5.2, S7.2, S7.6b) while one genuinely transient case is made Permanent and latches (S5.1, S4.2).
- **T2 Ownership mark enforced on Create and ListByTag but not on Get/Delete** (S2.3), and the fake cannot test the namespace rule (S7.4).
- **T3 Base image cache identity and lifetime**: keyed by key+format only, deleted in place under live overlays, no cross-process fencing (S4.6, S7.1, S4.8, escalations E4/E7).
- **T4 The shared pooled link**: any caller's cancel or budget expiry kills it for everyone, and several callers misread the resulting "connection is closed" error (S1.1, S3.1, S3.6, S4.3).
## Top of the list (high)
| ID | Where | Defect |
|---|---|---|
| S1.1 | `backend.go:138-148`, `ssh.go:109-114` | `markDead` closes the transport before signalling death; the close can block ~15 min on a silent host, wedging callers and the host's dial semaphore |
| S4.1 | `image.go:273` | A reader or server-side error during `StorageVolUpload` wedges go-libvirt's stream sender forever: goroutine + 4 MiB leak per failed upload, 20 min stall |
| S3.1 | `get.go:104-119` | `address()` swallows every non-ctx error, so a dead link or a guest-agent restart demotes a healthy Running instance to Provisioning and wipes `status.IP` |
| S7.1 | `volume.go:17`, `image.go:113-146` | Base image cache is keyed by key+format, never URL: two provider entries on one host+pool (the shipped sample does this) silently share one image; a URL edit is a silent no-op |
---
## Chunk 1 — Connection layer (`ssh.go`, `hosts.go`, `backend.go`)
**S1.1 — high — `internal/provider/kvm2/backend.go:138-148` (kill at 143-145, `close(l.closed)` at 146), `ssh.go:109-114`**
`link.markDead` runs `l.kill()` synchronously inside `once.Do` and only then closes `l.closed`. `kill` is `sshSocket.Close`, which calls `s.Conn.Close()` (x/crypto `chanConn` → `channel.Close` → a raw TCP write under `writeMu`/`t.mu`) before `s.client.Close()`. That write blocks while the send buffer is full.
Scenario: hypervisor is power-cut or partitioned (no RST/FIN, normal over Tailscale) with data queued. A pooled call hits its 30 s/90 s budget, `call` (`backend.go:183`) invokes `markDead`, which blocks until the kernel gives up (`tcp_retries2`, ~15 min). During that window: the abandoning caller never returns (reconcile ctx has no deadline); `l.closed` is still open so `isDead` reports the backend live and `hostTable.get` keeps handing it out, each new call wedging behind the first in `once.Do`; `sshSocket.closeOnce` memoises the in-progress close so `libvirtBackend.Close`'s 5 s budget cannot bound it; when the blocked `markDead` is the one at `hosts.go:153` it runs while `hostTable.get` holds `e.sem`, blocking every caller for that host.
Fix direction: close `s.client` (plain `net.Conn.Close`) first and/or close `l.closed` before calling `kill`.
**S1.2 — medium — `ssh.go:119-147`**
`sshSocket.keepalive`, the only detector of a silently dead pooled connection between calls, has no test. `testSSHServer` already services global requests, so it is testable end to end. A regression is invisible to CI and shows up only as a stale "live" connection that burns a full 30 s budget. Violates the repo rule that all new Go code has tests.
**S1.3 — low — `ssh.go:120`**
`time.NewTicker(interval)` panics for `interval <= 0`, in a goroutine, which exits the operator. Only `newLibvirtDialer` sets the field today; a future struct literal without it compiles and panics on first successful dial. No guard at either end.
**S1.4 — low — `errors.go:45-51` (dial path from `ssh.go:87`)** [T1]
`classify` recognises only `*knownhosts.KeyError`. A key matched by an `@revoked` line yields `*knownhosts.RevokedError`, which falls through to Transient: a deliberately revoked hypervisor key is retried forever instead of failing Permanent.
**S1.5 — low — `ssh.go:167-172`**
`ClientConfig.HostKeyAlgorithms` is left empty. If `known_hosts` pins only the RSA key of a host that also serves ed25519, the server offers ed25519, the callback returns a `KeyError` with non-empty `Want`, and `classify` maps it to Permanent "host key mismatch" although the pinned entry is genuine.
Verified clean: all dial error paths close the TCP conn and SSH client; `call`/`await` orphan goroutines write into buffered channels; the never-closed pooled table is not a leak (Provider built once, no hot reload).
## Chunk 2 — Domain lifecycle (`kvm2.go`, `create.go`, `delete.go`, `id.go`)
**S2.1 — medium — `create.go:107-114`**
Per-VM volumes created before `DomainDefineXML` are unreachable by both `Delete` and orphan GC when `Create` never returns an ID.
Scenario: `ensureOverlay` and `writeSeed` succeed, `DomainDefineXML` (139) fails Permanent (or the manager dies between). `Create` returns `("", err)`; `reconcileManaged` never sets `status.ProviderID` and latches. On Proxy deletion, `reconcileDelete` takes the `ProviderID == ""` shortcut (`proxy_controller.go:270-274`) without calling `Delete`. `ListByTag` enumerates domains only, so `proxy-<x>.qcow2` (10 GiB capacity) and `proxy-<x>-seed.iso` stay in the pool permanently.
**S2.2 — medium — `create.go:230-231`, `state.go:16-27`, `get.go:56`** (also: chunk 3)
Paused domains are destroyed. `ensureRunning` refuses a paused/pmsuspended domain with a Permanent "virsh resume it" error, but that guard only fires before an ID is persisted. In steady state the reconciler calls `Get`; `get.go:56` discards the `DomainGetState` reason and `mapState` maps paused/pmsuspended to `StateStopped`, which sends `reconcileManaged` (`proxy_controller.go:217-225`) to `prov.Delete`: destroy, undefine, delete disks, recreate.
Scenario: QEMU pauses the guest on an I/O error because the pool is full (`VIR_DOMAIN_PAUSED_IOERROR`), or an admin runs `virsh suspend`/`save`/snapshot/live-migrate. At the next drift poll the VM is killed; in the ENOSPC case the recreate refills the pool or fails the space precheck.
**S2.3 — medium — `delete.go:56-63`, `get.go:52-73`** (also: chunk 3) [T2]
`Get` and `Delete` do not enforce the kvm2 mark that `Create` (`create.go:80-82`) enforces. `readMetadata` returns the zero value for an unmarked domain and `get` builds an Instance anyway (`UID: ""`); `removeDomain`/`removeVolumes` destroy whatever the name resolves to. `internal/provider/kvm/id.go:11-13` emits byte-identical `hosts/<host>/domains/<name>` IDs and both providers use `NameFromUID`.
Scenario (the documented migration path, `docs/kvm2-provider.md:346-370`, done out of order): an admin re-types an entry from `kvm` to `kvm2` keeping its name without first deleting every Proxy. `status.ProviderID` survives; kvm v1's mark is in another namespace, so `Get` returns the v1 domain as this Proxy's instance; if it is anything but Running the controller calls `Delete` and kvm2 destroys a kvm-v1 domain the docs promise it can never touch. Same exposure for a hand-edited `status.ProviderID`. `TestGet_unmanagedDomainHasEmptyUID` pins the current behaviour. `ListByTag` is correctly mark-filtered, so GC is not affected.
**S2.4 — low — `create.go:87`**
The adoption path starts an already-defined domain without checking its disks exist and cannot heal them. If the overlay or seed is gone and no ID was persisted, every retry skips the volume steps, `DomainCreate` fails with a storage error classified Transient, and the reconciler retries forever; the drift path that would heal it is unreachable without an ID.
**S2.5 — low — `delete.go:15`** [T1]
`undefineFlags` omits `DomainUndefineCheckpointsMetadata`. An admin- or backup-tool-created checkpoint makes `DomainUndefineFlags` fail Transient; `deletionFailure` retries forever and the Proxy cannot be deleted without manual `virsh checkpoint-delete`.
Verified clean: `keyedLocks` refcounting, cancel-while-waiting and unlock on every path; `waitRunning` cap and cancellation; providerID round trip.
## Chunk 3 — Observation, state, errors (`get.go`, `list.go`, `state.go`, `errors.go`)
**S3.1 — high — `get.go:104-119`** [T4]
`address()` treats every non-`context` error from both address sources as "skip" and returns `("", nil)`.
Triggers: (a) routine — on beelink the ARP source never answers (`docs/kvm2-provider-spec.md:146`), so a `qemu-guest-agent` restart in the guest returns `ErrAgentUnresponsive` → `""` → `mapState(running, "")` = Provisioning; (b) racy — any concurrent call that exceeds its budget kills the shared pooled link (`backend.go:180-184`), after which `call` returns `transientf("... connection is closed")` with no ctx sentinel, which `get.go:108` does not recognise; if the link dies between `DomainGetState` and the address read, `Get` succeeds with `State=Provisioning, IP=""`.
Downstream: `proxy_controller.go:208-211` clears `status.IP` and sets `Provisioned=False`; `internal/health/engine.go:200-207` prunes the proxy's health state and the Healthy counters restart from zero. One RPC hiccup pulls a working proxy out of rotation. `TestGet_addressErrorIsProvisioning` pins the behaviour.
**S3.2 — medium — `state.go:35-49`**
IP detection has no interface selection. The domain has exactly one bridge NIC and libvirt returns `Hwaddr` per interface, but `firstIPv4` filters only loopback/link-local/unspecified.
Scenario: the image or cloud-init brings up `docker0` (172.17.0.1), `tailscale0` or `wg0` before the DHCP lease lands and the agent reports it first. `Get` returns Running with a non-routable IP, the controller latches `Provisioned=True`, and the value is never corrected because any IPv4 keeps the state Running.
**S3.3 — medium — `list.go:38-45`** [T4]
A pooled link that dies mid-list makes `ListByTag` a silently successful partial list. The fail-whole guard is `ctx.Err() != nil`, but a killed link yields "connection is closed" with no ctx sentinel; the remaining domains each log "skipping domain with unreadable metadata" and the call returns nil error with "managed=5" as if authoritative. GC only under-deletes, so no wrong deletion, but nothing marks the result partial. A Permanent corrupt-mark parse error is likewise skipped forever with only a V(1) line.
**S3.4 — low — `get.go:104-119`**
`address()` also swallows `ErrNoDomain`: a domain that vanishes between `DomainGetState` and the address read is reported as live Provisioning for one poll. The comment "only ctx ending is an error" is also inaccurate: a 30 s budget expiry on the agent source wraps `DeadlineExceeded`, fails `Get`, and never falls back to ARP.
**S3.5 — low — `errors.go:58-71`** [T1] (also: chunks 5, 7)
Version and capability errors escape classification. go-libvirt turns an "unknown procedure" reply into the bare sentinel `libvirt.ErrUnsupported` (not a `libvirt.Error`), so `libvirtCode` misses it and it becomes Transient, while the same condition as `VIR_ERR_NO_SUPPORT` is Permanent. `ErrConfigUnsupported` (67) and `ErrOperationUnsupported` (84) are also unmapped. `Check` records `libvirtVersion` but gates nothing on a minimum. See S5.2 and escalation E9.
## Chunk 4 — Base image cache and volumes (`image.go`, `volume.go`)
**S4.1 — high — `image.go:273`**
`uploadImage` hands a `TeeReader` over `resp.Body` straight to `StorageVolUpload`. In go-libvirt (`rpc.go:290-308`, `socket/socket.go:347-376`) `abort` and `outErr` are unbuffered: on a reader error `SendStream` sends a `StatusError` packet then blocks forever on `outErr <-`, and `requestStream` blocks forever on `abort <- true`. `StorageVolUpload` never returns.
Scenario: mirror truncates the body (or libvirtd errors mid-upload, e.g. ENOSPC). kvm2's `call` rescues the caller only after `budgets.transfer` = 20 min, and `markDead`/socket close does not unblock either goroutine (contrary to the comment at `backend.go:152-156`). Consequences: 1-2 goroutines + 4 MiB buffer leaked per failed upload, accumulating under reconcile retries against a flaky mirror; a truncated download reported after 20 min instead of at once; `discardVolume` (`image.go:292`) deletes the base on the pooled connection before the deferred `fresh.Close()` (`image.go:265`), so the file is unlinked while the server-side stream holds the fd. `TestEnsureImage_shortReadRemovesVolumeAndIsTransient` gives false confidence: the fake returns the reader error immediately and never exercises the abort protocol.
Fix direction: have `countingReader` convert errors to `io.EOF` and report them out of band; the existing `readErr`/`read != size` checks still catch truncation.
**S4.2 — medium — `image.go:434-439`, `391-396`** [T1]
A `HEAD` that answers 2xx with `Content-Length: 0` or none (Go's own `net/http` does this; several CDNs too) becomes `ErrPermanent`, which latches `ReasonPermanentError` for the generation. The GET fallback fires only on non-2xx HEAD, although GET on the same URL would work. The tests never hit it because `serveBytes` sets Content-Length on HEAD.
**S4.3 — medium — `image.go:235-245`** [T4]
A completed upload is discarded when the pooled connection died during it. The base is uploaded on a dedicated connection, but `StoragePoolRefresh` and `writeMarker` then run on the pooled `be`, which any other reconcile's cancel can kill. On failure `discardVolume` deletes the fully downloaded, digest-verified base and the next Create re-downloads. The window is as wide as the download. The dedicated connection is still open at that point and could carry the refresh and marker.
**S4.4 — medium — `image.go:419-443`, `226-230`, `create.go:96-106`**
The pool-space precheck uses the **HEAD** Content-Length; `fetchImage` declares the volume capacity from the **GET** Content-Length and never re-checks. There is no `maxImageBytes` cap.
Scenario: a server answers HEAD with 1 MB and GET with 4 TiB; the precheck passes, a sparse 4 TiB qcow2 is created and the stream fills the host pool until ENOSPC, degrading every VM on the hypervisor. Benign variant: a mirror rotates a build between HEAD and GET.
**S4.5 — medium — `image.go:59-65`**
`placement.image` is an unvalidated user-supplied URL fetched by the operator. `PlacementSpec.Image` has no CRD validation; `IsHTTPURL` checks only scheme and non-empty host; `http.DefaultClient` follows redirects with no policy. Anyone who can create a Proxy in a watched namespace makes the operator issue HEAD/GET from its network position (cluster services, `169.254.169.254`) and makes the hypervisor boot the returned bytes on the egress bridge. URL images can never carry a `sha256`.
**S4.6 — medium — `image.go:118`, `image.go:145`** (also: chunks 2, 7) [T3]
Any Create can evict a base volume that live overlays back onto, with no in-use check and only a V(1) log. `cachedImage` runs on every Create and deletes the base when the marker is missing, unreadable, non-hex, or when `src.SHA256` differs, then `fetchImage` recreates the same path with different content.
Scenario: an admin adds or updates `images.<key>.sha256` while Proxies exist and the mirror has republished. Running qemu processes keep the old inode; every later boot of an existing overlay reads a different base. The spec records this as a doc warning ("add a new key instead"); the kvm sample carries the warning (`config/samples/providers-config.yaml:40`) but the kvm2 block (`:72`) omits it. See escalation E7.
**S4.7 — low — `image.go:62`**
A URL image is always assumed qcow2/uefi. A raw image given as `placement.image` is cached as `base-<key>.qcow2` with `BackingFormat: "qcow2"`; qemu rejects the chain or the VM fails to boot, and the diagnostic points at the domain.
**S4.8 — low — `volume.go:22-25`** [T3]
`urlImageKey` truncates sha256 to 64 bits and shares a namespace with catalogue keys. Two colliding URLs share one base (a ~2^32 birthday search for someone who can create Proxies), and URL images have no digest to catch it. A catalogue key that is 16 lowercase hex chars occupies the same `base-<key>.<format>` name as a URL image.
Verified clean: HTTP bodies and the dedicated upload connection are closed on every path; `cappedWriter` and `countingReader` are correct as written.
## Chunk 5 — XML and cloud-init seed (`xml.go`, `seed.go`, `metadata.go`, `buildVolumeXML`)
**No injection defect found.** Every config- and user-influenced value reaching domain, volume or metadata XML goes through `encoding/xml` struct marshalling; there is no string concatenation into markup; `Name` is `NameFromUID`'s base32 hash. Chunks 2 and 7 reached the same conclusion independently.
**S5.1 — medium — `create.go:199-201`, `seed.go:66-74`** [T1, inverted]
Every I/O failure inside `buildSeedISO` is turned into `ErrPermanent` by `permanentf("building seed ISO for %s: %v", ...)`: `os.MkdirTemp` when `/tmp` is missing, ENOSPC during `AddFile`/`WriteTo`, and via the deferred `errors.Join` even a `Cleanup()`/`RemoveAll` failure on an otherwise successful build. `providerFailure` then returns `ctrl.Result{}` with nil error: no requeue at all until a spec edit or the ~10 h resync. The `%v` also discards the wrapped error. The empty-name guard is correctly permanent.
**S5.2 — medium — `xml.go:26-27`, `194-206`** (also: chunk 7) [T1]
`x86_64`, `q35`, `type="kvm"`, `host-passthrough`, `<os firmware="efi">` (libvirt ≥ 5.2.0) and `<firmware><feature name="secure-boot">` (≥ 7.2.0) are unconditional with no capability or version gate, and no config field or validator mentions architecture.
Scenario: an aarch64 hypervisor or a libvirt 7.0 host is added to `kvm2.hosts`; `Check` reports Ready; every Create there fails at `DomainDefineXML` with code 67 or 84, which `classifyLibvirt` leaves Transient, so the reconciler retries an impossible define indefinitely with no actionable condition.
**S5.3 — low — `metadata.go:44-50`** (see escalation E2)
An unparsable `<createdAt>` returns `time.Time{}`; `gc.go:120` computes `now.Sub(CreatedAt) < MinAge`, so a mangled mark is immediately past the grace period, exactly the case MinAge protects. Mirror case: a mark written under a forward-skewed clock yields a negative age that is always `< MinAge`, so the domain is never collected. Only the first direction is mentioned in the doc comment.
**S5.4 — low — `seed.go:48-53`**
The guest-agent script's non-systemd fallback runs `rc-update` unguarded (the kvm provider guards it with `elif command -v rc-update`). With `set -e`, an image with neither exits 127 after the package install; the agent is installed but never started and discovery silently falls back to ARP only.
**S5.5 — low — `xml.go:178-193`**
`buildDomainXML` validates name, paths and disk format but not `VCPUs`/`MemoryMiB`; it will emit `<vcpu>0</vcpu>`. Unreachable today because `validateKVMBlock` checks them; latent for any future caller.
**S5.6 — low — `volume.go:68-89`** (see escalation E10)
No check that an overlay's capacity is at least the backing image's virtual size. `kvm2.diskSizeGb: 2` against a 3.5 GiB Ubuntu image produces `<capacity>` smaller than `<backingStore>`.
**S5.7 — low — `internal/provider/config_kvm.go:186-188`**
`bridge` and `pool` are checked non-empty only, unlike host names and image keys (`kvmIdentifier`). A value with whitespace or markup is faithfully escaped into the XML and surfaces as an opaque libvirt failure at Create time rather than a config-load error.
**S5.8 — low — `seed.go:107-108`**
A configured `vendorData` that is itself MIME multipart (or starts with any unrecognised line) is nested as `text/plain`, matches no cloud-init prefix, and is silently dropped when `installGuestAgent` is on. No guard in `buildVendorData`.
Verified clean: the iso9660 staging dir is cleaned on every return path; volume XML sets no `<permissions>`, so the seed ISO gets libvirt's 0600 default.
## Chunk 6 — Health check and observability (`check.go`, `observe.go`)
In good shape: the dedicated check connection is released on every path, every seam method is wrapped and forwards correctly (enforced by reflection in `observe_test.go:77`), spans always end, `liveness` survives the decorator, and no cloud-init or key material reaches a log or span at any verbosity. `go vet` and `go test -race` on the chunk were clean.
**S6.1 — low — `check.go:90`, `backend.go:162`, `:351`**
`call()` returns at its pre-flight `ctx.Err()` check without `markDead`, so if the round deadline falls between two probe RPCs the link is still live and the deferred `Close()` sends `ConnectClose` on a ctx-less 5 s budget. `Check` can work past the deadline the Monitor gave it, against `provider.Checker`'s "must return promptly once ctx ends". Bounded at 5 s, narrow window.
**S6.2 — low — `check.go:57-58`**
Reported `Latency` includes the teardown RPC because `defer b.Close()` runs inside `probeHost` before `time.Since(start)`. About one RTT slower than the spec's cost model.
**S6.3 — low — `check.go:66`**
The round-wide `ctx.Err()` test masks an actionable per-host reason. Host A blackholes and eats the round; host B fails at the deadline with "no route to host" and is reported `Timeout`. Same for `PoolNotActive`/`CredentialsUnavailable` landing on the deadline.
**S6.4 — low — `check.go:135-148` vs `internal/providerhealth/monitor.go:113`**
The Unreachable/Timeout distinction only holds while the round timeout exceeds the 10 s dial budget. `ValidateSchedule` accepts `--provider-check-interval=10s` or `--provider-check-timeout=10s`, under which a blackholed host reports `Timeout` instead of `Unreachable`.
**S6.5 — low — `check.go:101` → `api/v1alpha1/providerhealth_types.go:65`**
The remote-controlled `hostname` is copied unbounded into ProviderHealth `details`; `Message` is truncated to 1024 but `details` has no `maxLength`/`maxProperties`. A multi-MB hostname makes every status patch fail "too large". Requires a broken or hostile hypervisor.
**S6.6 — low — `check.go:41-45`**
Unbounded fan-out: one goroutine, key re-read and full SSH handshake per host per round, no cap. Harmless at 2-5 hosts.
## Chunk 7 — Config, registration, deployment, test gating
**S7.1 — high — `internal/provider/kvm2/volume.go:17`, `image.go:113-146`** (also: chunk 4) [T3]
The base image cache is keyed by image key + format only. `baseVolumeName` → `base-<key>.<format>`, and `internal/provider/kvm/image.go:46` produces the identical name. The shipped sample has `kvm-home` (`config/samples/providers-config.yaml:41`) and `kvm2-home` (`:73`) both with key `debian-13`, host `beelink`, pool `default`, and `:58-60` tells the admin to run both while porting.
Scenario: the two entries' URLs drift and neither sets `sha256`; `cachedImage` accepts any 64-hex marker when `src.SHA256 == ""`, so whichever provider downloaded first owns the file and the other boots the wrong image. Same for two `kvm2` entries naming one physical host. Related: changing `images.<key>.url` without changing the key or `sha256` is a silent no-op, since neither the volume name nor the marker records the URL.
**S7.2 — medium — `internal/provider/config_kvm.go:180-182`** [T1]
`KVMHost.Address` is validated only as non-empty although it must be `host:port`; nothing applies a default port. `address: beelink` passes `LoadConfig`; `dialSSHSocket` (`ssh.go:163`) gets "missing port in address", `classify` falls through to Transient, every operation retries forever, and `checkReason` reports `Unreachable`, pointing at the hypervisor rather than the config.
**S7.3 — medium — `internal/provider/config_kvm.go:101-104`, `142-147`**
`IsHTTPURL` accepts plain `http://` and `sha256` stays optional, so a VM root disk can come from an unauthenticated channel; `writeMarker` then pins whatever digest arrived. Nothing warns.
**S7.4 — medium — `internal/provider/kvm2/fake_test.go:362-376`, `:259-282`** [T2]
The fake's `DomainGetMetadata` ignores the namespace of the stored element, returning it whenever `uri[0] == metadataURI`; real libvirt selects strictly by namespace. So no unit test can build a domain carrying the kvm provider's `<instance>` and assert `list.go:38-48` skips it. If that filter regressed, kvm2's `ListByTag` would hand kvm's VMs to orphan GC and the suite would stay green.
**S7.5 — medium — `Makefile:75`**
`-timeout 40m` is not derived from `KVM2_TEST_TIMEOUT`. Two integration tests each take up to 20 m plus a 5 m cleanup budget; with `KVM2_TEST_TIMEOUT=40m` or a cold image download, go test's own timeout panics before `t.Cleanup` (`kvm2_integration_test.go:373-382`) runs, leaving a domain, overlay and seed on the real hypervisor under a random name printed only mid-run.
**S7.6 — low — `internal/provider/config_kvm.go:133`, `internal/provider/kvm2/image.go:466`** [T1]
No upper bound on `diskSizeGb`, and `neededBytes` wraps: `diskSizeGb: 17179869184` (2^34) shifts to 0, so `checkPoolSpace` passes and the failure surfaces later as "capacity must be positive". `machineTypes` vcpus/memoryMiB have no upper bound either (`config_kvm.go:158-162`); `memoryMiB: 1024000` yields an internal libvirt error that is Transient and retried forever.
**S7.7 — low — `internal/provider/kvm2/fake_test.go:190-208`**
The fake's `enter()` accepts calls on an expired ctx and never marks the connection dead, unlike `call()`. So `discardVolume`'s `context.WithoutCancel` (`image.go:331`) is not exercised: deleting it leaves every test green while production skips cleanup of a half-uploaded base on a cancelled Create.
**S7.8 — low — `internal/provider/kvm2/fake_test.go:579-584`**
The fake's `Close()` only bumps a counter; it stays usable and `isDead()` stays false. A use-after-close of a dedicated connection passes in tests and fails in production.
**S7.9 — low — `Makefile:60-75`, `.gitea/workflows/build.yaml:27-32`**
Nothing compiles the integration file: `go vet`/`go build`/`go test -short`/`make test` all run without `-tags=integration`. It compiles today (`go vet -tags=integration ./internal/provider/kvm2/` exits 0), but nothing keeps it that way.
**S7.10 — low — `internal/provider/kvm2/image.go:228`, `283-289`, `385-395`**
The full image URL reaches log lines and error messages while the span deliberately carries only the host (`urlHost`). `IsHTTPURL` accepts userinfo, so `http://user:pass@mirror/...` lands in the error that `provider.Wrap` propagates. Whether it reaches Proxy status is unverified (dropped escalation D5).
**S7.11 — low — `internal/provider/kvm2/kvm2_integration_test.go:89`, `:307`**
`KVM2_TEST_KEEP=0` or `false` enables keep, since the check is `!= ""`. And `if elapsed > 100*time.Millisecond` on `New()` is a wall-clock `t.Errorf` on a possibly loaded machine, a flaky failure at the head of a 20 minute test.
**S7.12 — low/informational — `config/manager/manager.yaml:175-178`, `internal/provider/kvm2/seed.go:56-74`**
The seed ISO staging dir is the `/tmp` emptyDir with no `medium: Memory`, so Proxy cloud-init payloads (which can carry proxy credentials) are written unencrypted to node-local disk for the duration of each Create. The mount comment (`manager.yaml:163-165`) attributes `/tmp` to the kvm provider only.
Verified clean: Secret mount permissions (x/crypto does no permission check); both `Close` defers on dedicated connections.
docs/reviews/2026-09-23-proxmox-provider-review/REPORT.md
+286
View File
@@ -0,0 +1,286 @@
# proxmox provider review: final report
Branch `feat/proxmox-provider` at `7f2073f` (Phases 0 to 9), diffed against
`main` (`2167819`): 123 files, 49 of them Go. The core is
`internal/provider/proxmox/`; also in scope were the `nocloud` and
`keyedlock` extraction from kvm2 (`0a6c862`), the config block in
`internal/provider/config*.go`, registration and the deploy files.
**Inputs.** [docs/proxmox-provider-spec.md](../../proxmox-provider-spec.md)
(the spec the code implements) and
[docs/provider-contract.md](../../provider-contract.md) (the MUST/SHOULD
rules). A disagreement with either counts as a finding.
**Reviewers.** Two independent passes, both read-only:
1. The `go-operator-reviewer` agent (`.claude/agents/go-operator-reviewer.md`),
briefed with the spec and the contract, findings **P1 to P6** and
simplifications **S1 to S3**.
2. `/code-review high` on the branch, findings **CR1 to CR10**.
Where both passes found the same thing, the finding is listed once with both
IDs. Every finding is **fixed** (with a regression test when it is code),
**accepted** (real, not fixed here, documented as a known limit), or
**rejected** (with the reason).
**Toolchain.** `make test` green before and after the fixes. `make lint`
panics inside staticcheck on the local Go 1.27 toolchain, as it did during
kvm2, so lint ran without it, over the proxmox package and
`internal/provider`, with and without the integration tag:
```bash
bin/golangci-lint run --disable staticcheck --max-same-issues 0 --max-issues-per-linter 0 ./internal/provider/proxmox/... ./internal/provider/
bin/golangci-lint run --build-tags integration --disable staticcheck --max-same-issues 0 --max-issues-per-linter 0 ./internal/provider/proxmox/... ./internal/provider/
```
Before the fixes both runs were at the Phase 8 and 9 baseline: logcheck 12,
modernize (mapsloop) 2, unparam 3 in `fake_test.go`, gocyclo 1
(`TestAPITypes_decodeCaptures`), goconst on repeated strings, and the three
goconst hits in `internal/provider`. After the fixes the counts are the same
except goconst, one lower (224, 231 with the tag). No hit is new. The first
version of the P2 test did add one: a third `"backup"` literal in the
package's tests, then `"migrate"`. It now uses `"snapshot-delete"`.
## Summary
| ID | Severity | Finding | Disposition |
|---|---|---|---|
| CR1 + P1 | medium-high | A seed delete that fails after the destroy leaks the seed for good, and the class of the error depends on the attempt | **fixed** |
| CR2 | medium | A 403 on the seed delete latches a replaced Proxy whose VM is already gone | **fixed** (same change as CR1) |
| CR3 + P2 | medium | The ownership check waits out any lock, even on a VM whose listed name is someone else's | **fixed** |
| P3 | low-medium | Create adopts a leftover stopped VM whatever the current placement says | **accepted**, documented |
| P4 | low | A crash mid-clone plus a freed earlier VMID can yield a second VM | **accepted**, documented |
| P5 | low | 401/403 are Permanent, which bites during a token rotation | **fixed** in the guide |
| P6 | low | Get and Delete trust the listing's `status`, which lags | **accepted**, confirmed on the host; error message fixed |
| CR5 | low | `isLocked` and `isVMNotRunning` are called only from tests | **fixed** |
| CR8 | low | `TestPermanentfAndTransientf` breaks the `Test<Function>_<scenario>` rule | **fixed** |
| CR9 | low | `get` repeats `mapState`'s "guest running" rule inline | **fixed** |
| CR4 + S2 | low | The Checker repeats the cluster-wide `/version` and `/cluster/resources` per node | **rejected** |
| CR6 | low | `lockName`, `permanentf` and `transientf` are copies of kvm2's | **rejected** |
| CR7 | low | `readToken`, the CA load and seed staging do file I/O without a ctx | **rejected** |
| CR10 | low | `New` runs `withDefaults` twice | **rejected** |
| S1 | simplification | The `observe.go` decorator could be one log line in `send` | **rejected** |
| S3 | simplification | `nocloud` exports `GuestAgentScript` and `SeedLabel` for tests only | **rejected** |
## Fixed
### CR1 + P1: the seed outlives a Delete that fails after the destroy
`internal/provider/proxmox/delete.go`. Delete stopped the VM, destroyed it,
then deleted the seed. If the seed delete failed (a 595, a task over its
1 minute budget, a 403), Delete returned an error, but the VM was already
gone. The reconciler's next `Get` was `ErrNotFound`, so `reconcileDelete`
released the finalizer without calling Delete again, and GC lists only VMs.
The seed stayed on `isoStorage` for good. That breaks contract §2.3 ("MUST
remove everything Create made ... GC sees only what ListByTag returns"). A
destroy task that outlived its budget but finished in the background had
the same effect.
P1 adds the second half. On the finalizer path the next Delete found the
VMID gone and took `deleteSeedEverywhere`, which logged the same 403 at V(1)
and returned nil. The seed leaked with nothing visible.
**Fix.** The seed now goes between the stop and the destroy, so the VM,
which is what `Get` looks at, is the last thing removed: a failed seed
delete leaves the VM stopped, `Get` keeps finding it, and the next Delete
finishes both. `deleteSeedEverywhere` now fails on an `ErrPermanent`
answer (a 401 or 403 every node would give) instead of skipping it. It
still skips a node that is down or lacks the storage. `runTask`'s "gone"
result lost its last reader and was dropped.
That Proxmox deletes an ISO a stopped VM's `ide2` still references is the
one assumption the fake cannot prove. From memory of the PVE source, not
checked against it: the content delete runs `vdisk_free` without looking
for references to `iso` volumes, and the destroy skips CD-ROM drives.
Phase 0 only ever deleted the seed after the destroy. **The Phase 10
real-host run proved it** on 2026-09-24: the imgdel task on the stopped
VM's seed ended `OK`, the destroy followed, and `TestProxmox_lifecycle`
step 10 found neither the VM nor the seed.
**Tests.** `TestDelete_seedDeleteFailureOnVMNodeFails` (403, 596 and a
failed task on the VM's node) now asserts that no Destroy was sent, that
`Get` still finds the VM, that a second Delete gives the same class, and
that a Delete after the fault clears removes both. It failed on the old
code with `Destroy calls = 1` and `Get = instance not found`.
`TestDelete_seedDeleteNodeFailureSkippedWhenNodeUnknown/a refused privilege
fails the call` covers the every-node 403. Updated for the new order:
`TestDelete_stoppedVMSkipsStop`, `TestDelete_runningVMStoppedFirst`,
`TestDelete_seedDeleteIsTaskAndIdempotent`,
`TestDelete_destroyTaskFailureIsTransient` (the seed is now gone and the VM
kept) and `TestDelete_goneMidwayIsSuccess` (a VM gone at the stop still
gets a Destroy call, which answers `does not exist`).
`TestLogging_noOpLinesAtV1`'s skip case now injects a node-level 500,
since a 403 no longer skips.
Docs: spec entry 11 (steps 5 to 7 reordered, the Permanent exception),
the guide's Delete paragraph, `providers-existing.md`'s Delete bullet and
its `delete-return` cell.
### CR2: a 403 on the seed delete latched a Proxy with no VM
Same code, same fix. On the replace path (`reconcileManaged` sees a Stopped
VM and calls Delete), the old order destroyed the VM, got a 403 on the
seed, and returned Permanent. `providerFailure` latched it, so the Proxy
sat Failed pointing at a VM that no longer existed. The next attempt would
have returned nil. Now the 403 comes before the destroy: the VM stays
stopped, and every attempt gives the same Permanent answer naming
`Datastore.Allocate` until the privilege is granted. The guide says so.
### CR3 + P2: the lock wait on a VM that is plainly not ours
`internal/provider/proxmox/create.go`, `allocate`. Any taken qemu VMID went
through `inspect`, which first waits for the VM's lock to clear (clone
budget, 10 minutes) and only then compares names. A VM on one of the
Proxy's probe VMIDs, listed with another name and holding a long lock (a
`vzdump` backup, or a stale lock a failed backup left behind), made every
Create wait 10 minutes and return Transient. `vmidFor` is deterministic,
so every retry landed on the same VMID. Only a clone in progress is
nameless (F8), so a listed name other than ours settles the question.
**Fix.** `allocate` sends a taken entry to `inspect` only when its listed
name is empty or ours; any other name probes on without a config read.
Spec entry 2's step 3 says so.
**Tests.** `TestCreate_takenVMIDInspectedBeforeProbing` gains "foreign VM
under a lock probes on without waiting" (a `snapshot-delete` lock with a
2 s clone budget; the old code spent the 2 s and failed Transient), and
"foreign VM probes on without a read" now expects one config read (the
template's). `TestCreate_vmidRangeExhaustedIsTransient` expects one
VMConfig call instead of five.
### P5: token rotation order
`docs/proxmox-provider.md`, cluster-side setup. A 401 is Permanent (the
same convention as GCP), so a request made with a revoked token before the
kubelet refreshes the Secret latches that Proxy. The guide now gives the
order: add the new token, update the Secret, wait for the mount, then
remove the old one, and the same before an `--expire` date.
### CR5: matchers used only by tests
`isLocked` and `isVMNotRunning` in `errors.go` had no production caller.
Both answers are Transient through `classify`'s default branch. They moved
to `fake_test.go`, where `TestFakeAPI_errorsMatchCaptures`,
`TestAPIError_matchers` and the fake's own checks still use them to hold the
fake's wording to the captures. Spec entry 13's matching sentence and
`providers-existing.md` now name only the matchers the provider uses.
### CR8: test name
`TestPermanentfAndTransientf` became `TestPermanentf_wrapsClassWithMessage`
and `TestTransientf_wrapsClassWithMessage`. kvm2 has the same name; it is
out of scope here.
### CR9: one rule, one place
`get` asked `status == "running" && qmpstatus in {running, ""}` inline to
decide whether to query the agent, the same rule `mapState` applies. Both
now call `guestRunning(status, qmpStatus)` in `state.go`, so a change to it
cannot leave the two out of step. `TestMapState_*` and the Get tests cover
it unchanged (`TestMapState_table`, `TestMapState_fromCaptures`).
## Accepted
### P3: adoption ignores a placement change
`create.go:94-118`. A Create that fails Permanent after the clone (a 403 on
the upload, say) leaves a stopped VM. If the user then changes
`placement.image` or `zone`, the next Create adopts that VM by name, and
the Proxy runs the old template on the old node. The fix the reviewer
proposed (record the template in the mark, re-clone a never-started VM
whose template differs) is a new mark field and a destroy path inside
Create. That is too much for this MR for a two-step, user-driven sequence.
Documented in the guide's Known gaps and spec entry 24: destroy the VM
before changing `placement` on such a Proxy.
### P4: a second VM after a crash mid-clone
The retry misses a nameless in-progress clone on a later probe VMID when
the VM on the earlier VMID was removed meanwhile, and clones again. It
takes a crash or cut-off during a clone of a few seconds, a foreign VM on
the first probe VMID, and that VM's removal in the same window. The
proposed fix (inspect every nameless `lock: clone` entry in the range
first) would make every Create wait on every other Proxy's clone. The
outcome is visible, not silent: the next Create fails Permanent naming
both VMIDs. Documented in the guide and spec entry 24.
### P6: the listing's `status` may lag
`get.go:51`, `delete.go:60`, `delete.go:90`. This is the reviewer's
inference, not observed: `/cluster/resources` comes from pvestatd, which
may lag a state change by seconds. Both cases heal on retry: `Get`
answering Transient for `unknown` is polled through, and a destroy of a VM
that started behind the listing's back fails its task as Transient and the
next Delete stops it first.
**Confirmed on pv1-1.** The first `Get` came 32 ms after the second Create,
about 6 s after the clone, and was Transient: the listing still showed the
VM as `status: unknown`. The next poll 5 s later saw it running. The
behaviour is right, but the message blamed the node ("its node pv1-1 is not
answering"), which was up. Get and Delete now say "its node … is not
answering, or the VM is too new to be reported yet".
`TestGet_unknownStatusIsTransient` asserts the wording, and spec entry 9
and the guide's error table name both causes. The Delete side (a stop
skipped because the listing lags) was not met: the run's Delete came a
minute after the start.
## Rejected
- **CR4 + S2, Checker repeats cluster-wide calls per node.** Spec entry 16
makes each node's target self-contained, so a failure is reported
against the node whose round saw it. The cost is two extra requests per
extra node per round, and the configured cluster
has one node. Worth revisiting with a multi-node config, not before.
- **CR6, helpers copied from kvm2.** `permanentf`/`transientf` are one
line each; `lockName` is five and wraps `keyedlock.Acquire`'s error in
`provider.ErrTransient`. Moving that into `keyedlock` would couple a
generic lock package to the provider sentinels, and putting the helpers
in `internal/provider` would touch every kvm2 call site in a proxmox MR.
- **CR7, file I/O without a ctx.** `readToken` and the CA load read a
mounted Secret with `os.ReadFile`, which cannot be cancelled; a ctx
parameter would be unused. kvm2's `os.ReadFile` of the SSH key is the
precedent. Seed staging is `nocloud`'s, shared with kvm2 and unchanged by
the extraction.
- **CR10, `withDefaults` twice in `New`.** Startup only, idempotent, a few
small maps. `newProvider` applies it because the tests build providers
through it; defaulting once would need a second constructor shape.
- **S1, replace the `observe.go` decorator with one log line in `send`.**
The decorator is spec entries 17 and 20's `span-decorator` answer: each
seam call gets a child span and a V(1)/V(2) line with node, VMID and
UPID, which `send` does not know. It mirrors kvm2's decorator, and the
reflection test guarantees every seam method is covered. Swapping the
design in the review phase is out of proportion.
- **S3, `nocloud` exports used only by tests.** `SeedLabel` is read by
both kvm2's and proxmox's Create tests to recognise a real seed, and
`GuestAgentScript` by proxmox's `TestBuildSeed_table`, which tests
proxmox's own mapping of its config fields onto the vendor-data, not
`nocloud`'s composition. The exports are the price of cross-package
tests.
## Checked and fine
From the reviewer agent's report, verified in code:
- The kvm2 extraction is a pure move. kvm2 still wraps every seed-build
error as Permanent, as before, and `keyedlock` is byte-identical to the
code it replaced.
- Delete stops or destroys only after the listing shows a qemu entry at
the VMID and the config carries the name and the mark; a reused VMID or
an unmarked VM is left alone.
- A missing VM inside Create is Transient, never NotFound; locks and
`status: unknown` are Transient, never NotFound.
- Create resumes at every step: the VMID comes from the UID, a nameless
locked entry is waited on and adopted, upload/config/resize are
repeatable, and the mark is written in the clone call.
- Secrets: `apiError` carries status and message only, the token never
reaches a `url.Error`, the Checker keeps only the token id, and the
cloud-init text appears only as `cloudInitBytes`.
- Every seam call takes the ctx; `pollUntil` honours ctx and budget;
`keyedlock.Acquire` gives up with ctx.
- Spec entries 1, 3, 4, 9, 11, 12, 13, 15 and 16 match the code (P2 was the
one disagreement, and the spec shared the flaw).
- Every §11 test name in spec entry 21 exists.
The agent also saw kvm2's `TestCheck_hostsCheckedConcurrently` fail once
under `-race` (199.9 ms against its 200 ms floor). That file is unchanged
on this branch; the flake is parked, as before.
docs/testing.md
+249 -6
View File
@@ -62,6 +62,18 @@ manual runs too.
| | `trace_test.go`, `conn_test.go`, `errors_test.go` | every `libvirtAPI` method traced exactly once (guarded by reflection on the interface), results and errors pass through, libvirt error fields, detail keys only at V2, upload content never logged; pool connect/drop lines, no wrapping below V1 | `fakeAPI` + `funcr` capture |
| | `check_test.go`, `conn_test.go` | health check: details, per-host verdicts in config order, hosts checked concurrently, a timeout that keeps the connection and refuses a second check while the first still waits, no waiting behind a reconcile's dial, the reason table; `probe` abandons without closing, but still drops on a transport error; waiting for a host's slot ends with the caller's deadline; an inactive or missing pool and daemon errors keep the connection; every dial stage tagged through the real SSH server, with classification unchanged | `fakeAPI`, fake dialers, in-process SSH server |
| | `kvm_integration_test.go` (`-tags=integration`) | one real VM: image fetch, Create, idempotent re-Create, Running with IP, ListByTag, Delete, pool state afterwards; `TestKVMCheck_realHost` runs the health check against the host and creates nothing | a **real hypervisor** — see below |
| `internal/provider/kvm2` | `xml_test.go`, `volume_test.go`, `seed_test.go`, `id_test.go`, `metadata_test.go`, `state_test.go` | pure builders: domain XML (file disks with resolved paths, UEFI with secure boot off, BIOS escape, the metadata element, the guest-agent channel), volume XML with and without a backing store, seed ISO reopened (files, label, user-data byte for byte, temp dir removed) and the vendor-data script vs. multipart rules, providerID round-trip and malformed forms, metadata in both namespace spellings, state mapping and the IPv4 filter | pure |
| | `errors_test.go`, `backend_test.go`, `hosts_test.go`, `ssh_test.go` | classification table (libvirt codes, SSH auth text, `knownhosts.KeyError`, missing files); the per-call budget adapter (deadline kills the connection and is Transient, cancel unblocks, killed by another caller, close idempotent) against a mocked libvirt socket; the host table (one dial per host under concurrency, redial after drop, `dialFresh` never cached, waiting ends with ctx); the real dialer end to end: socket forwarding, unknown and mismatched host keys, insecure mode, missing vs. unparsable key and known_hosts, wrong key, handshake timeout, cancel, every stage tagged, key material never in errors or logs | fake dialer; a **real in-process `x/crypto/ssh` server** forwarding a unix socket |
| | `image_test.go`, `create_test.go`, `get_test.go`, `delete_test.go`, `list_test.go`, `kvm2_test.go`, `provider_test.go` | image cache (fetch with the marker written last, cached, marker missing or without a digest, sha256 change or mismatch, 4xx/5xx, no Content-Length, short read, concurrent callers fetch once, cancel mid-stream, streamed without buffering, upload on a dedicated connection) and the pool-space quota precheck; Create resume after every partial step, returns only when running, adoption guard on UID, seed always rebuilt, cloud-init sentinel never logged; Get state and address mapping incl. agent-before-ARP; every Delete already-gone case, NVRAM flag, per-name lock; ListByTag across hosts, one down fails the whole call, unreadable domain skipped; `New` never dials and warns on missing files; every error wrapped with the configured name; logging tiers V0/V1/V2 | `httptest` image server + in-memory `fakeBackend` behind the `backend` seam |
| | `observe_test.go`, `check_test.go` | every seam method traced and logged exactly once (guarded by reflection), child spans under the parent, detail only at V2, upload bodies never logged, error fields; health check: one entry per host in config order, hosts checked concurrently, expired ctx returns promptly, never touches the pooled connection, the reason table, detail keys | `fakeBackend` + `funcr` capture + span recorder |
| | `kvm2_integration_test.go` (`-tags=integration`) | one real VM: `New` without a dial, `Check` Ready with details, Create, idempotent re-Create with an unchanged domain count, Running with IP and the metadata round-trip, ListByTag, Delete, NotFound, pool state afterwards, `kvm2.*` spans under `provider.create`; `TestKVM2Check_realHost` runs the health check against the host and creates nothing | a **real hypervisor** — see below |
| `internal/provider/nocloud` | `seed_test.go` | the NoCloud seed kvm2 and proxmox share: the ISO reopened (files, label, user-data byte for byte, temp dir removed, an empty vendor-data still a file, an empty name refused) and the vendor-data rules (a shell script, not cloud-config; multipart with configured extra vendor-data; the agent install switched off) | pure |
| `internal/provider/keyedlock` | `keyedlock_test.go` | the per-name lock kvm2 and proxmox share: the entry is removed once uncontended, waiters share one lock, a waiter gives up with its ctx | pure |
| `internal/provider/proxmox` | `id_test.go`, `mark_test.go`, `state_test.go`, `seed_test.go`, `errors_test.go`, `api_test.go` | pure pieces: providerID round-trip and malformed forms, the VMID derived from the UID (deterministic, in range, probing forward and wrapping), the description mark (keys, parsed from the captured clone, admin lines kept), state mapping and the IPv4 filter from captures, seed names and building (empty name Permanent without its payload, staging failure Transient); the `classify` table and the 500-message matchers against every captured error; the API types decoded from the captures | pure, over the responses Phase 0 captured from PVE 9.2.20 (`testdata/`) |
| | `client_test.go` | the thin HTTP client: the token read from its file on every request (missing Transient, unparsable Permanent), `apiError` from status, reason phrase and body, the data envelope, method, path and parameters of every seam call, the clone and destroy query shapes, the volid's slash escaped, HTTP/1.1 only, request timeout and cancel Transient and the client unaffected, a child span under a parent and none without, the token never in errors or logs, the CA file trusted and loaded lazily, an x509 failure Permanent, insecure mode, the upload streamed as multipart | `httptest.NewTLSServer` replaying the captures byte for byte |
| | `create_test.go`, `get_test.go`, `delete_test.go`, `list_test.go`, `task_test.go`, `proxmox_test.go`, `provider_test.go`, `fake_test.go` | Create: resume after every partial step and after a crash mid-clone, the VMID probe and ownership check, the template's shape refused before any clone, resize as a task, the seed overwritten without a delete, the net0 bridge and VLAN rewrite keeping the MAC, returns only when running, cloud-init sentinel never logged; Get: lock checked before name, the agent's messages, paused from `status/current`, hibernated, a reused VMID, four calls when running; Delete: every already-gone and not-ours case, the seed deleted before the destroy on the VM's node or on every node, a hibernated VM destroyed, the per-name lock; ListByTag: candidates by tag or name prefix, the mark decides, an unreadable VM skipped unless ctx ended; task and lock waits; `New` makes no request; the contract sweep: every method returns within a second of a cancel (blocked on its first seam call, and mid-operation), every error wrapped with the configured name, op and id, logging tiers V0/V1/V2 with the no-op lines | in-memory `fakeAPI` behind the `api` seam, its error texts checked against the captures |
| | `observe_test.go`, `check_test.go`, `provider_test.go` | the decorator: every seam method logs exactly one line (guarded by reflection), starts no span, identifiers at V1, detail at V2, upload body never logged, error fields; a Get through `provider.WithTracing` over the real client makes four `HTTP GET` spans, all children of `provider.get` and none carrying the token; health check: one entry per node in config order, nodes checked concurrently (a barrier, not a timer), expired ctx returns promptly, the reasons table at V2 with a sentinel secret, the transport reasons from real `net/http` and `crypto/tls` errors, detail keys | `fakeAPI` + `funcr` capture + span recorder; the real client over `httptest` for the tracing and transport-reason rows |
| | `proxmox_integration_test.go` (`-tags=integration`) | one real VM: `New` without a request, `Check` Ready with details, Create, idempotent re-Create with an unchanged VM count, Running with an IPv4 and the mark read back, ListByTag, Delete, NotFound, no VM and no seed left, HTTP spans under `provider.create`; `TestProxmoxCheck_realHost` runs the health check against the host and creates nothing. **Passed on pv1-1 (PVE 9.2.20), 2026-09-24**: Create 6.2 s, address 50 s later, whole run 61 s | a **real Proxmox VE host** — see below |
| `internal/health` | `probe_test.go` | probes through a **real CONNECT-capable proxy stub** to a real TLS server: tunnel success, refused CONNECT, wrong status, dead proxy, plain-http forward | `httptest` + hijacked bidirectional tunnel |
| | `engine_test.go` | thresholds, first-verdict, latency suppression matrix, dropped-event retry, stale-probe UID guard, tick scheduling/seeding/pruning, `Start` end-to-end | fake `client.Reader` + injected `probeFn` |
| `internal/lease` | `store_test.go` | capacity, `MaxLeases=0`, all three selection tie-breaks, target-scoped vs global cooldowns, expiry via fake clock, report-on-retained-lease, idempotent release, **40 concurrent acquires never exceeding capacity** | injectable clock; the concurrency case is why `-race` matters |
@@ -93,12 +105,33 @@ envtest suite the scaffold generated.
bug (below).
- **Live GCP**: no test talks to Google. The seam boundary
(`buildInsertRequest` + classification) is tested exhaustively instead.
- **Live libvirt, by default**: the KVM unit tests run over an in-memory
fake of the 20 libvirt calls the provider uses. Whether libvirt accepts
the domain XML, whether a raw qcow2 upload plus pool refresh yields a
usable backing image, and whether the guest actually boots and reports
an address are provable only on a real host — that is the opt-in
integration test below.
- **Live libvirt, by default**: the KVM and kvm2 unit tests run over an
in-memory fake of the 20 libvirt calls each provider uses. Whether
libvirt accepts the domain XML, whether a raw qcow2 upload plus pool
refresh yields a usable backing image, and whether the guest actually
boots and reports an address are provable only on a real host — that is
the opt-in integration test below (one per provider).
- **Live Proxmox, by default**: the proxmox unit tests run over an
in-memory fake of the 16 seam calls, and the HTTP client over
`httptest` replaying responses captured from PVE 9.2.20. Whether a
clone, config write, seed upload and start yield a guest that boots and
reports an address is provable only on a real host (the proxmox
integration test below, which has passed on pv1-1). Two kinds of path
have met only the fake:
- **the cross-node paths**: a clone that targets another node, the seed
deleted on every node when the VM's node is unknown, a Check round over
several nodes. The test host is a single node, so the integration test
cannot reach them either
([proxmox-provider-spec.md](proxmox-provider-spec.md) entry 24);
- **an answer never captured**: the storage status for a storage the
cluster knows but cannot use on a node (disabled there, or offline).
Every 500 there is taken as `StorageNotFound`, and what the fake
answers is its own guess. The three answers Phase 0 missed were
captured on pv1-1 on 2026-09-24: a missing storage is a 400 with
`storage: No such storage.`, a hibernated VM is destroyed without
complaint (its suspend state volume with it), and a stop on a paused
VM ends `OK` with the VM stopped.
## The KVM integration test (real hypervisor)
@@ -162,6 +195,216 @@ go test -tags=integration -count=1 -v -run TestKVMCheck ./internal/provider/kvm/
The test is not part of `make test` and not run in CI. Keep it compiling
with `go vet -tags=integration ./...` when the provider's internals move.
## The kvm2 integration test (real hypervisor)
`internal/provider/kvm2/kvm2_integration_test.go` compiles only under
`-tags=integration` and skips itself unless the environment names a
host. Like the kvm one it deviates from the "testcontainers for real
infra" convention on purpose: libvirt needs `/dev/kvm` and a bridge to a
DHCP network, and a container has neither. It targets whichever
hypervisor you point it at and leaves that host's pool with one more
cached base image (`base-test-image.qcow2` and its `.sha256` marker) than
it had. The kvm and kvm2 tests can share a host: kvm2 marks its domains
in its own metadata namespace and never lists or adopts the other's.
```sh
export KVM2_TEST_HOST=beelink:22 # SSH endpoint
export KVM2_TEST_USER=novakj # must be in the libvirt group
export KVM2_TEST_SSH_KEY=$HOME/.ssh/id_ed25519 # private key for that user
export KVM2_TEST_KNOWN_HOSTS=$HOME/.ssh/kvm_known_hosts # the hypervisors' host keys
export KVM2_TEST_BRIDGE=br0 # bridge to a network with DHCP
export KVM2_TEST_IMAGE_URL=https://cloud.debian.org/images/cloud/trixie/latest/debian-13-generic-amd64.qcow2
make test-integration-kvm2
```
`KVM2_TEST_KNOWN_HOSTS` is technically optional, but leaving it out turns
host-key verification **off** for the run (the log says so); keep a
`kvm_known_hosts` holding just the hypervisors rather than pointing the
test at your everyday `known_hosts`, so a re-keyed hypervisor is a
one-line fix in a file nothing else uses.
Also optional: `KVM2_TEST_IMAGE_SHA256` (verified after download and
compared with the cached marker), `KVM2_TEST_POOL` (default `default`),
`KVM2_TEST_SOCKET` (default `/var/run/libvirt/libvirt-sock`),
`KVM2_TEST_TIMEOUT` (default `20m`, the budget for the whole lifecycle;
it must cover the first-time image download), `KVM2_TEST_KEEP` to leave
the VM behind for `virsh console` when something goes wrong (the log then
prints the `virsh undefine --nvram` and `virsh vol-delete` lines that
clean it up), and `KVM2_TEST_LOG_LEVEL` (default `1`) for how much of the
provider's logging the `-v` output shows: at `1` every libvirt RPC is one
compact `libvirt call` line (`HH:MM:SS.mmm kvm2 "level"=1 "msg"="libvirt
call" "rpc"="DomainLookupByName" "domain"=... "duration"=...`, plus the
per-operation outcome lines and connection events), interleaved with the
test's own `t.Log` lines; `2` adds the domain and volume XML, metadata
and addresses; `0` leaves only `t.Log`. The per-run constants (provider,
host, domain name) are stated once in the first line and left off every
provider line. See the Logging section of
[kvm2-provider.md](kvm2-provider.md).
What one run asserts, in order, each step logged with its duration:
1. `New` returns within 100 ms and without dialling (no `libvirt host
connected` line before the first call).
2. `Check` reports the one host `Ready` with reason `Reachable` and the
`hostname`, `libvirtVersion`, `poolState` (`running`) and
`poolAvailableGiB` details, which are logged.
3. Every later call goes through `provider.WithTracing` with an in-memory
span recorder that the package's own spans also land in.
4. `Create` returns `hosts/kvm2-test/domains/<name>` (downloading the
image on a cold pool — the log prints how long it took).
5. A second `Create` returns the same ID and the host's domain count,
read over a dedicated connection, is unchanged.
6. `Get`, polled every 5 s, reaches `Running` with an IPv4 within the
budget; its `UID` is the one Create wrote and `CreatedAt` is non-zero
and within a few minutes of now. The time from Create's return to the
first address is logged.
7. `ListByTag` contains the ID with that UID.
8. A TCP connect to port 22 on the guest, logged only — the machine
running the test may not route to the bridge network.
9. `Delete` returns nil, `Get` is `ErrNotFound`, a second `Delete` is a
no-op.
10. The pool, inspected over a dedicated connection, holds the base image
and its marker but no `<name>.qcow2` or `<name>-seed.iso`.
11. Among the recorded spans, a `provider.create` span has at least one
`kvm2.DomainDefineXML` and one `kvm2.dial` child, and no span
attribute carries the cloud-init text.
With `KVM2_TEST_KEEP` steps 9 and 10 are skipped; otherwise a `t.Cleanup`
attempts the Delete even when an earlier assertion failed.
`TestKVM2Check_realHost` runs the provider health check
([kvm2-provider.md](kvm2-provider.md)) against the same host. It must
find the host reachable and the pool running, and it creates nothing. It
needs the same variables; to run only this test:
```sh
go test -tags=integration -count=1 -v -run TestKVM2Check ./internal/provider/kvm2/
```
The test is not part of `make test` and not run in CI. Keep it compiling
with `go vet -tags=integration ./...` when the provider's internals move.
## The proxmox integration test (real Proxmox VE host)
`internal/provider/proxmox/proxmox_integration_test.go` compiles only
under `-tags=integration` and skips itself unless the environment names a
host. It deviates from the testcontainers convention for the same reason
as the kvm tests: Proxmox VE is a hypervisor, not something a container
can stand in for. It needs a node with a cloud-init template without a
Proxmox cloud-init drive and with the guest agent enabled, and a token
with the privileges of [proxmox-provider-spec.md](proxmox-provider-spec.md)
entry 14: the role on `/` from the plan's "Before Phase 0" section
([2026-09-18-1502-proxmox-provider.md](plans/2026-09-18-1502-proxmox-provider.md)),
plus `Datastore.Allocate` on the ISO storage. A passing run leaves
nothing behind: its VM and its seed ISO are gone.
```sh
export PROXMOX_TEST_URL=https://192.168.0.8:8006 # host must be in the node certificate
export PROXMOX_TEST_TOKEN_FILE=$HOME/.config/proxmox-token # one line: user@realm!tokenid=secret
export PROXMOX_TEST_CA_FILE=$HOME/.config/proxmox-ca.pem # the node's pve-root-ca.pem
export PROXMOX_TEST_NODE=pv1-1
export PROXMOX_TEST_TEMPLATE_VMID=9001
export PROXMOX_TEST_ISO_STORAGE=proxy-seeds # holds iso content
export PROXMOX_TEST_VLAN_TAG=4 # template 9001's net0 is untagged; proxies live on VLAN 4
make test-integration-proxmox
```
Required: `PROXMOX_TEST_URL`, `PROXMOX_TEST_TOKEN_FILE`,
`PROXMOX_TEST_NODE`, `PROXMOX_TEST_TEMPLATE_VMID` and
`PROXMOX_TEST_ISO_STORAGE`; the test skips when any is missing. Optional:
`PROXMOX_TEST_CA_FILE` (empty trusts the system roots),
`PROXMOX_TEST_INSECURE` (`true` skips certificate verification; the log
says so), `PROXMOX_TEST_STORAGE` (the clone's target storage; empty keeps
the template's), `PROXMOX_TEST_BRIDGE` and `PROXMOX_TEST_VLAN_TAG` (the
cloned `net0`'s bridge and tag; empty keeps the template's),
`PROXMOX_TEST_VMID_MIN` and `PROXMOX_TEST_VMID_MAX` (default
`100000`..`199999`), `PROXMOX_TEST_TIMEOUT` (default `20m`, the budget for
the whole lifecycle), `PROXMOX_TEST_KEEP` to leave the VM behind, and
`PROXMOX_TEST_LOG_LEVEL` (default `1`): at `1` every request is one
`proxmox api call` line with the call, node, VMID, UPID and duration,
plus the per-operation outcome lines; `2` adds request parameters, config
summaries and agent addresses; `0` leaves only `t.Log`. Provider, node,
name and VMID are stated once and left off every provider line. Without
`Datastore.Allocate` on the ISO storage the seed delete at the end of
Delete is a 403, and Delete fails as Permanent.
What one run asserts, in order, each step logged with its duration:
1. `New` returns within 100 ms and makes no request (no `proxmox api
call` line before the first operation).
2. `Check` reports the node `Ready` with reason `Reachable` and the
`pveVersion`, `kernel`, `tokenID`, `isoStorageAvailGiB` and
`templates` (`1/1`) details, which are logged.
3. Every later call goes through `provider.WithTracing`, with an
in-memory span recorder on the transport as well.
4. `Create` returns `vms/<vmid>/<name>` with the VMID in range. The
machine type is 1 core, 1024 MiB and a 4 GiB disk, so a 3 GiB template
is grown by the resize task.
5. A second `Create` returns the same ID and the cluster's QEMU VM count
is unchanged.
6. `Get`, polled every 5 s, reaches `Running` with an IPv4 within the
budget (a Transient answer while the guest boots is logged and polled
through). Its `UID` and `CreatedAt` equal the mark in the VM's
description, read back from its config, and `CreatedAt` is within a
few minutes of now. The time from Create's return to the first address
is logged.
7. `ListByTag` contains the ID with that UID and CreatedAt.
8. A TCP connect to port 22 on the guest, logged only: the machine
running the test may not route to the VM's network.
9. `Delete` returns nil, `Get` is `ErrNotFound`, a second `Delete` is a
no-op.
10. No VM with the name is left in the cluster's resources, and no
`<isoStorage>:iso/<name>-seed.iso` in the storage's iso content. The
content is read with one extra read-only GET of
`/nodes/{node}/storage/{storage}/content?content=iso` made by the test
itself (`storage_content_iso.json` is its captured shape): the seam
has no method to list storage content, and does not get one for a
test.
11. Among the recorded spans, a `provider.create` span has at least one
`HTTP` child, and no span attribute carries the cloud-init text.
With `PROXMOX_TEST_KEEP` steps 9 and 10 are skipped, and at the end the
log prints what to run as root on the node to clean up:
`qm stop <vmid>`, `qm destroy <vmid> --purge` and
`pvesm free <isoStorage>:iso/<name>-seed.iso`. Otherwise a `t.Cleanup`
runs the Delete even when an earlier assertion failed. It finds the VM by
name, so a Create that failed halfway is cleaned up too.
Cost: one full clone of the template (about 2 s for the 3 GiB template on
pv1-1), one boot to an address (about 50 s), and a few seconds of
destroy; a run takes a couple of minutes. The Makefile target's 30 m
timeout covers the 20 m budget plus the cleanup.
`TestProxmoxCheck_realHost` runs the provider health check against the
same host. It must find the node reachable, the ISO storage usable and
the template present, and it creates nothing. It needs the same
variables; to run only this test:
```sh
go test -tags=integration -count=1 -v -run TestProxmoxCheck ./internal/provider/proxmox/
```
**Last run: 2026-09-24 on pv1-1** (Proxmox VE 9.2.20, kernel
7.0.14-17-pve, template 9001, `proxy-seeds`, VLAN tag 4). Both tests
passed; `go test` reported 60.9 s for the package.
| Step | Time |
|---|---|
| `Check` | 141 ms (125 ms in `TestProxmoxCheck_realHost`) |
| clone task, submit to done | about 3 s (seen done at the third 1 s poll) |
| `Create`, clone to started VM | 6.2 s |
| second `Create` (adoption, nothing created) | 32 ms |
| Create's return to the first address | 50.2 s (the agent install from the internet) |
| `Delete` (stop, seed delete, destroy), `Get`, second `Delete` | 3.1 s |
| `TestProxmox_lifecycle` | 59.7 s |
The first `Get`, 32 ms after the second Create, was Transient: the
listing still showed the new VM with `status: unknown`. The next poll 5 s
later saw it running. The Delete deleted the seed while the stopped VM's
`ide2` still referenced it, then destroyed the VM, and step 10 found
neither left. Like the kvm tests they are not part of `make test` and not
run in CI.
## The scaffolded `make test-e2e` suite
`test/e2e/` is the kubebuilder-generated smoke suite (build image → kind
go.mod
+1 -1
View File
@@ -23,6 +23,7 @@ require (
k8s.io/api v0.36.0
k8s.io/apimachinery v0.36.0
k8s.io/client-go v0.36.0
k8s.io/klog/v2 v2.140.0
sigs.k8s.io/controller-runtime v0.24.1
sigs.k8s.io/yaml v1.6.0
)
@@ -104,7 +105,6 @@ require (
k8s.io/apiextensions-apiserver v0.36.0 // indirect
k8s.io/apiserver v0.36.0 // indirect
k8s.io/component-base v0.36.0 // indirect
k8s.io/klog/v2 v2.140.0 // indirect
k8s.io/kube-openapi v0.0.0-20260317180543-43fb72c5454a // indirect
k8s.io/streaming v0.36.0 // indirect
k8s.io/utils v0.0.0-20260210185600-b8788abfbbc2 // indirect
internal/discovery/server.go
+6 -1
View File
@@ -9,6 +9,7 @@ import (
"context"
"crypto/subtle"
"encoding/json"
"log/slog"
"net"
"net/http"
"strings"
@@ -111,7 +112,11 @@ func (s *Server) Start(ctx context.Context) error {
s.mu.Unlock()
srv := &http.Server{
Handler: s.handler(),
Handler: s.handler(),
// Without this, net/http's own faults (a failed Accept, a panic that
// escapes a handler) go to the stdlib logger on stderr, outside the
// structured stream.
ErrorLog: slog.NewLogLogger(logr.ToSlogHandler(s.log), slog.LevelError),
ReadHeaderTimeout: readHeadTimeout,
ReadTimeout: 10 * time.Second,
WriteTimeout: 10 * time.Second,
internal/provider/config.go
+20 -1
View File
@@ -22,6 +22,11 @@ type ProviderConfig struct {
Kubernetes *KubernetesConfig `json:"kubernetes,omitempty"`
GCP *GCPConfig `json:"gcp,omitempty"`
KVM *KVMConfig `json:"kvm,omitempty"`
// KVM2 is the same block shape as KVM, for the kvm2 provider type: one
// or more libvirt/KVM hypervisors reached over SSH.
KVM2 *KVMConfig `json:"kvm2,omitempty"`
// Proxmox is a Proxmox VE cluster reached through its REST API.
Proxmox *ProxmoxConfig `json:"proxmox,omitempty"`
}
// setBlocks names the type-specific blocks that are non-nil, so validate can
@@ -37,6 +42,12 @@ func (p ProviderConfig) setBlocks() []string {
if p.KVM != nil {
out = append(out, "kvm")
}
if p.KVM2 != nil {
out = append(out, "kvm2")
}
if p.Proxmox != nil {
out = append(out, "proxmox")
}
return out
}
@@ -100,7 +111,7 @@ func (c *Config) validate() error {
prefix := fmt.Sprintf("providers[%d] %q", i, p.Name)
switch p.Type {
case "kubernetes", "gcp", "kvm":
case "kubernetes", "gcp", "kvm", "kvm2", "proxmox":
case "":
return fmt.Errorf("%s: type is required", prefix)
default:
@@ -120,6 +131,14 @@ func (c *Config) validate() error {
if err := validateKVM(prefix, p.KVM); err != nil {
return err
}
case "kvm2":
if err := validateKVMBlock(prefix, "kvm2", p.KVM2); err != nil {
return err
}
case "proxmox":
if err := validateProxmox(prefix, p.Proxmox); err != nil {
return err
}
}
}
return nil
internal/provider/config_kvm.go
+26 -13
View File
@@ -108,29 +108,42 @@ var sha256Hex = regexp.MustCompile(`^[0-9a-fA-F]{64}$`)
// kvmIdentifier constrains the config names that are pasted into
// identifiers elsewhere: a host name goes into the providerID
// hosts/<name>/domains/<vm>, which is split on "/" to parse, and an image
// key becomes the file name of the cached base volume. A slash in either
// produces an ID or a path that never resolves again, so the VM leaks with
// nothing able to address it.
// key becomes part of the file name of the cached base volume. A slash in
// either produces an ID or a path that never resolves again, so the VM
// leaks with nothing able to address it.
var kvmIdentifier = regexp.MustCompile(`^[A-Za-z0-9][A-Za-z0-9_.-]*$`)
// IsKVMIdentifier reports whether s is safe to paste into a providerID or a
// volume name. The provider entry's own name reaches a volume name too, and
// that name is validated for every provider type at once (only non-empty
// and unique), so the kvm2 provider checks its own at construction.
func IsKVMIdentifier(s string) bool { return kvmIdentifier.MatchString(s) }
func validateKVM(prefix string, c *KVMConfig) error {
return validateKVMBlock(prefix, "kvm", c)
}
// validateKVMBlock checks a KVMConfig under the given block name, so the
// kvm and kvm2 provider types share one validator and each reports its own
// block in messages.
func validateKVMBlock(prefix, block string, c *KVMConfig) error {
if c == nil {
return fmt.Errorf("%s: kvm block is required", prefix)
return fmt.Errorf("%s: %s block is required", prefix, block)
}
if c.SSHKeyFile == "" {
return fmt.Errorf("%s: kvm.sshKeyFile is required", prefix)
return fmt.Errorf("%s: %s.sshKeyFile is required", prefix, block)
}
if c.KnownHostsFile == "" && !c.InsecureIgnoreHostKey {
return fmt.Errorf("%s: kvm.knownHostsFile is required unless kvm.insecureIgnoreHostKey is true", prefix)
return fmt.Errorf("%s: %s.knownHostsFile is required unless %s.insecureIgnoreHostKey is true", prefix, block, block)
}
if c.DiskSizeGB < 0 {
return fmt.Errorf("%s: kvm.diskSizeGb must not be negative", prefix)
return fmt.Errorf("%s: %s.diskSizeGb must not be negative", prefix, block)
}
for name, img := range c.Images {
p := fmt.Sprintf("%s: kvm.images[%q]", prefix, name)
p := fmt.Sprintf("%s: %s.images[%q]", prefix, block, name)
if !kvmIdentifier.MatchString(name) {
return fmt.Errorf("%s: key must be letters, digits, dots, dashes or underscores"+
" — it becomes the cached volume name base-<key>.<format>", p)
" — it becomes part of the cached base volume's name", p)
}
if !IsHTTPURL(img.URL) {
return fmt.Errorf("%s.url must be an http(s) URL (got %q)", p, img.URL)
@@ -146,19 +159,19 @@ func validateKVM(prefix string, c *KVMConfig) error {
}
}
if len(c.MachineTypes) == 0 {
return fmt.Errorf("%s: kvm.machineTypes must define at least one profile", prefix)
return fmt.Errorf("%s: %s.machineTypes must define at least one profile", prefix, block)
}
for name, mt := range c.MachineTypes {
if mt.VCPUs <= 0 || mt.MemoryMiB <= 0 {
return fmt.Errorf("%s: kvm.machineTypes[%q] needs vcpus > 0 and memoryMiB > 0", prefix, name)
return fmt.Errorf("%s: %s.machineTypes[%q] needs vcpus > 0 and memoryMiB > 0", prefix, block, name)
}
}
if len(c.Hosts) == 0 {
return fmt.Errorf("%s: kvm.hosts must list at least one host", prefix)
return fmt.Errorf("%s: %s.hosts must list at least one host", prefix, block)
}
seen := make(map[string]bool, len(c.Hosts))
for i, h := range c.Hosts {
p := fmt.Sprintf("%s: kvm.hosts[%d]", prefix, i)
p := fmt.Sprintf("%s: %s.hosts[%d]", prefix, block, i)
if h.Name == "" {
return fmt.Errorf("%s: name is required", p)
}
internal/provider/config_kvm_test.go
+94
View File
@@ -214,6 +214,100 @@ providers:
}
}
func TestLoadConfig_kvm2Valid(t *testing.T) {
t.Parallel()
yaml := strings.Replace(validKVMYAML, "type: kvm\n kvm:\n", "type: kvm2\n kvm2:\n", 1)
cfg, err := LoadConfig([]byte(yaml))
if err != nil {
t.Fatalf("LoadConfig() error = %v, want nil", err)
}
if cfg.Providers[0].KVM != nil {
t.Errorf("providers[0].kvm = %+v, want nil", cfg.Providers[0].KVM)
}
k := cfg.Providers[0].KVM2
if k == nil {
t.Fatal("providers[0].kvm2 = nil")
}
if k.SSHKeyFile != "/etc/proxy-operator/kvm/id_ed25519" || k.KnownHostsFile != "/etc/proxy-operator/kvm/known_hosts" || k.DiskSizeGB != 20 {
t.Errorf("kvm2 = %+v, want sshKeyFile, knownHostsFile and diskSizeGb parsed", k)
}
if k.Images["ubuntu"].Format != "raw" {
t.Errorf("images[ubuntu].format = %q, want raw", k.Images["ubuntu"].Format)
}
if mt := k.MachineTypes["small"]; mt.VCPUs != 1 || mt.MemoryMiB != 1024 {
t.Errorf("machineTypes[small] = %+v, want {1 1024}", mt)
}
if len(k.Hosts) != 2 || k.Hosts[0].Name != "nas1" || k.Hosts[1].Pool != "fast" {
t.Errorf("hosts = %+v, want nas1 and lab2 with lab2's pool parsed", k.Hosts)
}
}
func TestLoadConfig_kvm2Invalid(t *testing.T) {
t.Parallel()
minimal := func(body string) string {
return "providers:\n - name: kvm2-home\n type: kvm2\n kvm2:\n" + indent(body, 6)
}
hostsAndTypes := `
machineTypes:
small: {vcpus: 1, memoryMiB: 1024}
hosts:
- {name: nas1, address: nas1:22, user: virt, bridge: br0}
`
tests := []struct {
name string
yaml string
wantErrSub string
}{
{
name: "missing kvm2 block",
yaml: "providers:\n - name: kvm2-home\n type: kvm2\n",
wantErrSub: "kvm2 block is required",
},
{
name: "missing sshKeyFile",
yaml: minimal("knownHostsFile: /kh\n" + hostsAndTypes),
wantErrSub: "kvm2.sshKeyFile is required",
},
{
name: "missing knownHostsFile without insecure flag",
yaml: minimal("sshKeyFile: /key\n" + hostsAndTypes),
wantErrSub: "kvm2.knownHostsFile is required unless kvm2.insecureIgnoreHostKey is true",
},
{
name: "no machine types",
yaml: minimal("sshKeyFile: /key\nknownHostsFile: /kh\nhosts:\n - {name: nas1, address: nas1:22, user: virt, bridge: br0}\n"),
wantErrSub: "kvm2.machineTypes must define at least one",
},
{
name: "no hosts",
yaml: minimal("sshKeyFile: /key\nknownHostsFile: /kh\nmachineTypes:\n small: {vcpus: 1, memoryMiB: 1024}\n"),
wantErrSub: "kvm2.hosts must list at least one host",
},
{
name: "duplicate host name",
yaml: minimal("sshKeyFile: /key\nknownHostsFile: /kh\nmachineTypes:\n small: {vcpus: 1, memoryMiB: 1024}\nhosts:\n - {name: nas1, address: a:22, user: virt, bridge: br0}\n - {name: nas1, address: b:22, user: virt, bridge: br0}\n"),
wantErrSub: `kvm2.hosts[1]: duplicate host name "nas1"`,
},
{
name: "image key with a slash",
yaml: minimal("sshKeyFile: /key\nknownHostsFile: /kh\nimages:\n ../etc/d: {url: https://x/y.qcow2}\n" + hostsAndTypes),
wantErrSub: `kvm2.images["../etc/d"]: key must be letters, digits, dots, dashes or underscores`,
},
}
for _, tc := range tests {
t.Run(tc.name, func(t *testing.T) {
t.Parallel()
_, err := LoadConfig([]byte(tc.yaml))
if err == nil {
t.Fatalf("LoadConfig() error = nil, want error containing %q", tc.wantErrSub)
}
if !strings.Contains(err.Error(), tc.wantErrSub) {
t.Errorf("LoadConfig() error = %q, want substring %q", err.Error(), tc.wantErrSub)
}
})
}
}
func indent(s string, n int) string {
pad := strings.Repeat(" ", n)
lines := strings.Split(strings.TrimLeft(s, "\n"), "\n")
internal/provider/config_proxmox.go
+179
View File
@@ -0,0 +1,179 @@
package provider
import (
"fmt"
"net/url"
"regexp"
)
// ProxmoxConfig configures a proxmox provider instance: one Proxmox VE
// cluster reached through its REST API with an API token. VMs are full
// clones of prepared templates, seeded with a NoCloud ISO uploaded to
// ISOStorage. The design is in docs/proxmox-provider-spec.md.
type ProxmoxConfig struct {
// URL is the API endpoint, https://<host>:8006. The host must be a name
// or address in the node certificate. Required.
URL string `json:"url"`
// TokenFile holds one line, <user>@<realm>!<tokenid>=<secret>, read on
// every request. Required.
TokenFile string `json:"tokenFile"`
// CAFile is a PEM bundle trusted for the API's certificate, typically the
// node's /etc/pve/pve-root-ca.pem. Empty means the system roots.
CAFile string `json:"caFile,omitempty"`
// InsecureSkipTLSVerify disables certificate verification.
InsecureSkipTLSVerify bool `json:"insecureSkipTLSVerify,omitempty"`
// Nodes are the Proxmox node names placement.zone selects from.
// Required, at least one, unique.
Nodes []string `json:"nodes"`
// ISOStorage is the storage seed ISOs are uploaded to and deleted from;
// the token needs Datastore.Allocate on it. Required.
ISOStorage string `json:"isoStorage"`
// Storage is the target storage of the full clone. Empty clones onto the
// template's own disk storage.
Storage string `json:"storage,omitempty"`
// Bridge replaces the bridge of the cloned net0. Empty keeps it.
Bridge string `json:"bridge,omitempty"`
// VLANTag sets the VLAN tag of the cloned net0, 1 to 4094. Zero keeps
// whatever the template has.
VLANTag int `json:"vlanTag,omitempty"`
// VMIDRange is where VMIDs are allocated. Default 100000 to 199999.
VMIDRange *ProxmoxVMIDRange `json:"vmidRange,omitempty"`
// Templates are the clone sources placement.image names. Required, at
// least one.
Templates map[string]ProxmoxTemplate `json:"templates"`
// MachineTypes are the size profiles placement.machineType names.
// Required, at least one.
MachineTypes map[string]ProxmoxMachineType `json:"machineTypes"`
// RequestTimeoutSeconds bounds one API request. Default 30.
RequestTimeoutSeconds int `json:"requestTimeoutSeconds,omitempty"`
// VendorData is optional cloud-init vendor-data placed on every seed next
// to the Proxy's own user-data, as for the kvm providers.
VendorData string `json:"vendorData,omitempty"`
// InstallGuestAgent adds a vendor-data script that installs and starts
// qemu-guest-agent, which is how the provider learns a VM's address.
// Default true. Turn it off for templates that ship the agent.
InstallGuestAgent *bool `json:"installGuestAgent,omitempty"`
}
// ProxmoxVMIDRange bounds the VMIDs the provider allocates, both inclusive.
type ProxmoxVMIDRange struct {
Min int `json:"min"`
Max int `json:"max"`
}
// ProxmoxTemplate is one clone source: a VM converted to a template.
type ProxmoxTemplate struct {
VMID int `json:"vmid"`
}
// ProxmoxMachineType is a VM size profile. DiskGiB grows the cloned boot
// disk when larger than the template's; zero keeps the template's size.
type ProxmoxMachineType struct {
Cores int32 `json:"cores"`
MemoryMiB int64 `json:"memoryMiB"`
DiskGiB int64 `json:"diskGiB,omitempty"`
}
// Proxmox's own limits on VMIDs and VLAN tags.
const (
ProxmoxMinVMID = 100
ProxmoxMaxVMID = 999999999
proxmoxMaxVLANTag = 4094
)
var (
// proxmoxNodeName is a Proxmox node name: a DNS label, as pvecm allows.
proxmoxNodeName = regexp.MustCompile(`^[a-z0-9]([a-z0-9-]*[a-z0-9])?$`)
// proxmoxStorageID is a Proxmox storage identifier.
proxmoxStorageID = regexp.MustCompile(`^[a-zA-Z][a-zA-Z0-9_.-]*$`)
)
func validateProxmox(prefix string, c *ProxmoxConfig) error {
if c == nil {
return fmt.Errorf("%s: proxmox block is required", prefix)
}
if err := validateProxmoxURL(c.URL); err != nil {
return fmt.Errorf("%s: proxmox.url %w", prefix, err)
}
if c.TokenFile == "" {
return fmt.Errorf("%s: proxmox.tokenFile is required", prefix)
}
if len(c.Nodes) == 0 {
return fmt.Errorf("%s: proxmox.nodes must list at least one node", prefix)
}
seen := make(map[string]bool, len(c.Nodes))
for i, n := range c.Nodes {
if !proxmoxNodeName.MatchString(n) {
return fmt.Errorf("%s: proxmox.nodes[%d] %q is not a Proxmox node name (lowercase letters, digits and dashes)", prefix, i, n)
}
if seen[n] {
return fmt.Errorf("%s: proxmox.nodes[%d]: duplicate node %q", prefix, i, n)
}
seen[n] = true
}
if c.ISOStorage == "" {
return fmt.Errorf("%s: proxmox.isoStorage is required: name a storage with iso content where the token holds Datastore.Allocate", prefix)
}
if !proxmoxStorageID.MatchString(c.ISOStorage) {
return fmt.Errorf("%s: proxmox.isoStorage %q is not a Proxmox storage id", prefix, c.ISOStorage)
}
if c.Storage != "" && !proxmoxStorageID.MatchString(c.Storage) {
return fmt.Errorf("%s: proxmox.storage %q is not a Proxmox storage id", prefix, c.Storage)
}
if c.VLANTag < 0 || c.VLANTag > proxmoxMaxVLANTag {
return fmt.Errorf("%s: proxmox.vlanTag must be between 1 and %d (got %d)", prefix, proxmoxMaxVLANTag, c.VLANTag)
}
if r := c.VMIDRange; r != nil {
if r.Min < ProxmoxMinVMID || r.Max > ProxmoxMaxVMID || r.Min >= r.Max {
return fmt.Errorf("%s: proxmox.vmidRange must satisfy %d <= min < max <= %d (got %d to %d)", prefix, ProxmoxMinVMID, ProxmoxMaxVMID, r.Min, r.Max)
}
}
if len(c.Templates) == 0 {
return fmt.Errorf("%s: proxmox.templates must define at least one template", prefix)
}
for key, t := range c.Templates {
if key == "" {
return fmt.Errorf("%s: proxmox.templates has an empty key", prefix)
}
if t.VMID < ProxmoxMinVMID {
return fmt.Errorf("%s: proxmox.templates[%q].vmid must be at least %d (got %d)", prefix, key, ProxmoxMinVMID, t.VMID)
}
}
if len(c.MachineTypes) == 0 {
return fmt.Errorf("%s: proxmox.machineTypes must define at least one profile", prefix)
}
for key, mt := range c.MachineTypes {
if key == "" {
return fmt.Errorf("%s: proxmox.machineTypes has an empty key", prefix)
}
if mt.Cores <= 0 || mt.MemoryMiB <= 0 {
return fmt.Errorf("%s: proxmox.machineTypes[%q] needs cores > 0 and memoryMiB > 0", prefix, key)
}
if mt.DiskGiB < 0 {
return fmt.Errorf("%s: proxmox.machineTypes[%q].diskGiB must not be negative", prefix, key)
}
}
if c.RequestTimeoutSeconds < 0 {
return fmt.Errorf("%s: proxmox.requestTimeoutSeconds must not be negative", prefix)
}
return nil
}
// validateProxmoxURL accepts only an https URL with a host and no path, so
// the client can append /api2/json itself.
func validateProxmoxURL(s string) error {
if s == "" {
return fmt.Errorf("is required")
}
u, err := url.Parse(s)
if err != nil {
return fmt.Errorf("does not parse: %w", err)
}
if u.Scheme != "https" || u.Host == "" {
return fmt.Errorf("must be an https URL with a host (got %q)", s)
}
if u.Path != "" && u.Path != "/" || u.RawQuery != "" || u.Fragment != "" {
return fmt.Errorf("must have no path, query or fragment; the client appends /api2/json (got %q)", s)
}
return nil
}
internal/provider/config_proxmox_test.go
+250
View File
@@ -0,0 +1,250 @@
package provider
import (
"os"
"slices"
"strings"
"testing"
"sigs.k8s.io/yaml"
)
const validProxmoxYAML = `
providers:
- name: proxmox-home
type: proxmox
proxmox:
url: https://192.168.0.8:8006
tokenFile: /etc/proxy-operator/proxmox/token
caFile: /etc/proxy-operator/proxmox/ca.pem
nodes: [pv1-1, pv1-2]
isoStorage: proxy-seeds
storage: local-lvm
bridge: vmbr0
vlanTag: 4
vmidRange: {min: 150000, max: 150999}
templates:
debian-13: {vmid: 9001}
machineTypes:
small: {cores: 1, memoryMiB: 1024}
large: {cores: 4, memoryMiB: 4096, diskGiB: 20}
requestTimeoutSeconds: 45
installGuestAgent: false
`
func TestLoadConfig_proxmoxValid(t *testing.T) {
t.Parallel()
cfg, err := LoadConfig([]byte(validProxmoxYAML))
if err != nil {
t.Fatalf("LoadConfig() error = %v, want nil", err)
}
c := cfg.Providers[0].Proxmox
if c == nil {
t.Fatal("providers[0].proxmox = nil")
}
if c.URL != "https://192.168.0.8:8006" || c.ISOStorage != "proxy-seeds" || c.VLANTag != 4 {
t.Errorf("proxmox = %+v, want url, isoStorage and vlanTag parsed", c)
}
if len(c.Nodes) != 2 || c.Nodes[1] != "pv1-2" {
t.Errorf("nodes = %v, want [pv1-1 pv1-2]", c.Nodes)
}
if c.VMIDRange == nil || c.VMIDRange.Min != 150000 || c.VMIDRange.Max != 150999 {
t.Errorf("vmidRange = %+v, want 150000 to 150999", c.VMIDRange)
}
if c.Templates["debian-13"].VMID != 9001 {
t.Errorf("templates[debian-13] = %+v, want vmid 9001", c.Templates["debian-13"])
}
if mt := c.MachineTypes["large"]; mt.Cores != 4 || mt.MemoryMiB != 4096 || mt.DiskGiB != 20 {
t.Errorf("machineTypes[large] = %+v, want {4 4096 20}", mt)
}
if c.InstallGuestAgent == nil || *c.InstallGuestAgent {
t.Errorf("installGuestAgent = %v, want explicit false", c.InstallGuestAgent)
}
}
func TestValidateProxmox_table(t *testing.T) {
t.Parallel()
block := func(body string) string {
return "providers:\n - name: proxmox-home\n type: proxmox\n proxmox:\n" + indent(body, 6)
}
const base = `
url: https://192.168.0.8:8006
tokenFile: /t
nodes: [pv1-1]
isoStorage: proxy-seeds
templates: {debian-13: {vmid: 9001}}
machineTypes: {small: {cores: 1, memoryMiB: 1024}}
`
with := func(old, repl string) string {
if !strings.Contains(base, old) {
t.Fatalf("base has no %q", old)
}
return block(strings.Replace(base, old, repl, 1))
}
tests := []struct {
name string
yaml string
wantErrSub string
}{
{name: "valid minimal", yaml: block(base)},
{name: "valid full", yaml: validProxmoxYAML},
{name: "valid with trailing slash", yaml: with("url: https://192.168.0.8:8006", "url: https://192.168.0.8:8006/")},
{name: "missing block", yaml: "providers:\n - name: proxmox-home\n type: proxmox\n", wantErrSub: "proxmox block is required"},
{name: "missing url", yaml: with("url: https://192.168.0.8:8006\n", ""), wantErrSub: "proxmox.url is required"},
{name: "http url", yaml: with("https://", "http://"), wantErrSub: "must be an https URL"},
{name: "url with api path", yaml: with("8006", "8006/api2/json"), wantErrSub: "must have no path"},
{name: "missing tokenFile", yaml: with("tokenFile: /t\n", ""), wantErrSub: "proxmox.tokenFile is required"},
{name: "no nodes", yaml: with("nodes: [pv1-1]", "nodes: []"), wantErrSub: "proxmox.nodes must list at least one node"},
{name: "duplicate node", yaml: with("nodes: [pv1-1]", "nodes: [pv1-1, pv1-1]"), wantErrSub: `duplicate node "pv1-1"`},
{name: "node with a slash", yaml: with("nodes: [pv1-1]", "nodes: [pv1/1]"), wantErrSub: "is not a Proxmox node name"},
{name: "node in uppercase", yaml: with("nodes: [pv1-1]", "nodes: [PV1]"), wantErrSub: "is not a Proxmox node name"},
{name: "missing isoStorage", yaml: with("isoStorage: proxy-seeds\n", ""), wantErrSub: "proxmox.isoStorage is required"},
{name: "bad isoStorage id", yaml: with("isoStorage: proxy-seeds", "isoStorage: 1seeds"), wantErrSub: "proxmox.isoStorage \"1seeds\" is not a Proxmox storage id"},
{name: "bad storage id", yaml: with("isoStorage: proxy-seeds", "isoStorage: proxy-seeds\nstorage: a/b"), wantErrSub: "proxmox.storage \"a/b\""},
{name: "vlanTag too large", yaml: with("isoStorage: proxy-seeds", "isoStorage: proxy-seeds\nvlanTag: 4095"), wantErrSub: "proxmox.vlanTag must be between 1 and 4094"},
{name: "vlanTag negative", yaml: with("isoStorage: proxy-seeds", "isoStorage: proxy-seeds\nvlanTag: -1"), wantErrSub: "proxmox.vlanTag"},
{name: "vmidRange min below 100", yaml: with("isoStorage: proxy-seeds", "isoStorage: proxy-seeds\nvmidRange: {min: 50, max: 500}"), wantErrSub: "proxmox.vmidRange must satisfy"},
{name: "vmidRange min above max", yaml: with("isoStorage: proxy-seeds", "isoStorage: proxy-seeds\nvmidRange: {min: 2000, max: 1000}"), wantErrSub: "proxmox.vmidRange must satisfy"},
{name: "vmidRange max too large", yaml: with("isoStorage: proxy-seeds", "isoStorage: proxy-seeds\nvmidRange: {min: 100, max: 1000000000}"), wantErrSub: "proxmox.vmidRange must satisfy"},
{name: "no templates", yaml: with("templates: {debian-13: {vmid: 9001}}", "templates: {}"), wantErrSub: "proxmox.templates must define at least one template"},
{name: "template vmid below 100", yaml: with("vmid: 9001", "vmid: 99"), wantErrSub: `proxmox.templates["debian-13"].vmid must be at least 100`},
{name: "no machineTypes", yaml: with("machineTypes: {small: {cores: 1, memoryMiB: 1024}}", "machineTypes: {}"), wantErrSub: "proxmox.machineTypes must define at least one profile"},
{name: "cores zero", yaml: with("cores: 1", "cores: 0"), wantErrSub: `proxmox.machineTypes["small"] needs cores > 0`},
{name: "memory zero", yaml: with("memoryMiB: 1024", "memoryMiB: 0"), wantErrSub: "memoryMiB > 0"},
{name: "negative diskGiB", yaml: with("memoryMiB: 1024", "memoryMiB: 1024, diskGiB: -5"), wantErrSub: "diskGiB must not be negative"},
{name: "negative timeout", yaml: with("isoStorage: proxy-seeds", "isoStorage: proxy-seeds\nrequestTimeoutSeconds: -1"), wantErrSub: "proxmox.requestTimeoutSeconds must not be negative"},
{name: "unknown key is rejected", yaml: with("isoStorage: proxy-seeds", "isoStorage: proxy-seeds\nsshKeyFile: /k"), wantErrSub: "parsing providers config"},
}
for _, tc := range tests {
t.Run(tc.name, func(t *testing.T) {
t.Parallel()
_, err := LoadConfig([]byte(tc.yaml))
if tc.wantErrSub == "" {
if err != nil {
t.Fatalf("LoadConfig() error = %v, want nil", err)
}
return
}
if err == nil {
t.Fatalf("LoadConfig() error = nil, want error containing %q", tc.wantErrSub)
}
if !strings.Contains(err.Error(), tc.wantErrSub) {
t.Errorf("LoadConfig() error = %q, want substring %q", err.Error(), tc.wantErrSub)
}
})
}
}
// commentedEntry cuts the commented-out providers entry named name out of a
// config file and strips its "# " at the entry's own indent, so a sample
// that is shipped commented out is still loaded by a test.
func commentedEntry(t *testing.T, path, name string) string {
t.Helper()
data, err := os.ReadFile(path)
if err != nil {
t.Fatalf("reading %s: %v", path, err)
}
var out []string
var prefix string
for line := range strings.SplitSeq(string(data), "\n") {
if prefix == "" {
if strings.TrimSpace(line) == "# - name: "+name {
prefix = line[:strings.Index(line, "#")]
out = append(out, prefix+"- name: "+name)
}
continue
}
if !strings.HasPrefix(line, prefix+"# ") {
break
}
out = append(out, prefix+strings.TrimPrefix(line, prefix+"# "))
}
if prefix == "" {
t.Fatalf("%s has no commented entry %q", path, name)
}
return strings.Join(out, "\n") + "\n"
}
func TestLoadConfig_proxmoxSampleEntry(t *testing.T) {
t.Parallel()
for name, path := range map[string]string{
"sample": "../../config/samples/providers-config.yaml",
"configmap": "../../config/manager/providers_config.yaml",
} {
t.Run(name, func(t *testing.T) {
t.Parallel()
cfg, err := LoadConfig([]byte("providers:\n" + commentedEntry(t, path, "proxmox-home")))
if err != nil {
t.Fatalf("LoadConfig(proxmox-home from %s) error = %v, want nil", path, err)
}
p := cfg.Providers[0]
// LoadConfig already refused a proxmox block under another type.
if p.Name != "proxmox-home" || p.Proxmox == nil {
t.Fatalf("provider %q has proxmox block %v, want proxmox-home with one", p.Name, p.Proxmox)
}
c := p.Proxmox
if c.URL != "https://192.168.0.8:8006" {
t.Errorf("url = %q, want https://192.168.0.8:8006 (the address in the node certificate)", c.URL)
}
if c.TokenFile != "/etc/proxy-operator/proxmox/token" || c.CAFile != "/etc/proxy-operator/proxmox/ca.pem" {
t.Errorf("tokenFile, caFile = %q, %q, want the proxmox-token Secret's mount paths", c.TokenFile, c.CAFile)
}
if c.InsecureSkipTLSVerify {
t.Error("insecureSkipTLSVerify = true, want it left commented out")
}
if len(c.Nodes) != 1 || c.Nodes[0] != "pv1-1" {
t.Errorf("nodes = %v, want [pv1-1]", c.Nodes)
}
if c.ISOStorage != "proxy-seeds" || c.Storage != "" || c.Bridge != "" || c.VLANTag != 4 {
t.Errorf("isoStorage, storage, bridge, vlanTag = %q, %q, %q, %d, want proxy-seeds, \"\", \"\", 4", c.ISOStorage, c.Storage, c.Bridge, c.VLANTag)
}
if len(c.Templates) != 1 || c.Templates["debian-13"].VMID != 9001 {
t.Errorf("templates = %v, want {debian-13: {vmid: 9001}}", c.Templates)
}
if mt, ok := c.MachineTypes["small"]; !ok || mt.Cores != 1 || mt.MemoryMiB != 1024 || mt.DiskGiB != 0 {
t.Errorf("machineTypes[small] = %+v (present %v), want {1 1024 0}", mt, ok)
}
if c.VMIDRange != nil || c.RequestTimeoutSeconds != 0 || c.InstallGuestAgent != nil || c.VendorData != "" {
t.Errorf("optional fields set: %+v, want vmidRange, requestTimeoutSeconds, installGuestAgent and vendorData left commented out", c)
}
})
}
}
func TestLoadConfig_proxmoxSampleProxyMatchesSampleEntry(t *testing.T) {
t.Parallel()
cfg, err := LoadConfig([]byte("providers:\n" + commentedEntry(t, "../../config/samples/providers-config.yaml", "proxmox-home")))
if err != nil {
t.Fatalf("LoadConfig() error = %v, want nil", err)
}
data, err := os.ReadFile("../../config/samples/proxy_proxmox.yaml")
if err != nil {
t.Fatalf("reading the sample Proxy: %v", err)
}
var px struct {
Spec struct {
Provider string `json:"provider"`
Placement struct {
Zone string `json:"zone"`
MachineType string `json:"machineType"`
Image string `json:"image"`
} `json:"placement"`
} `json:"spec"`
}
if err := yaml.Unmarshal(data, &px); err != nil {
t.Fatalf("parsing the sample Proxy: %v", err)
}
c := cfg.Providers[0].Proxmox
if px.Spec.Provider != cfg.Providers[0].Name {
t.Errorf("spec.provider = %q, want %q", px.Spec.Provider, cfg.Providers[0].Name)
}
if !slices.Contains(c.Nodes, px.Spec.Placement.Zone) {
t.Errorf("placement.zone = %q, not in nodes %v", px.Spec.Placement.Zone, c.Nodes)
}
if _, ok := c.MachineTypes[px.Spec.Placement.MachineType]; !ok {
t.Errorf("placement.machineType = %q, not in machineTypes", px.Spec.Placement.MachineType)
}
if _, ok := c.Templates[px.Spec.Placement.Image]; !ok {
t.Errorf("placement.image = %q, not in templates", px.Spec.Placement.Image)
}
}
internal/provider/config_test.go
+40
View File
@@ -118,6 +118,46 @@ providers:
image: custom-image`,
wantErrSub: "type is gcp but a kubernetes block is set",
},
{
name: "kvm type with kvm2 block",
yaml: `
providers:
- name: p1
type: kvm
kvm2:
sshKeyFile: /key`,
wantErrSub: "type is kvm but a kvm2 block is set",
},
{
name: "kvm2 type with kvm block",
yaml: `
providers:
- name: p1
type: kvm2
kvm:
sshKeyFile: /key`,
wantErrSub: "type is kvm2 but a kvm block is set",
},
{
name: "proxmox type with kvm block",
yaml: `
providers:
- name: p1
type: proxmox
kvm:
sshKeyFile: /key`,
wantErrSub: "type is proxmox but a kvm block is set",
},
{
name: "kvm type with proxmox block",
yaml: `
providers:
- name: p1
type: kvm
proxmox:
url: https://pve:8006`,
wantErrSub: "type is kvm but a proxmox block is set",
},
{
name: "strict mode rejects unknown top-level key",
yaml: `
internal/provider/keyedlock/keyedlock.go
+60
View File
@@ -0,0 +1,60 @@
// Package keyedlock provides one-slot semaphores by key, shared by the
// providers that serialise work per instance name or per cached image
// (kvm2, proxmox).
package keyedlock
import (
"context"
"sync"
)
// Locks is a set of one-slot semaphores by key: a semaphore rather than a
// mutex so a waiter can give up with its ctx. Each entry counts the callers
// holding or waiting for it and is dropped from the map at zero, so the map
// does not grow with every key ever seen while two waiters on one key still
// always share one semaphore. The zero value is not usable; call New.
type Locks struct {
mu sync.Mutex
locks map[string]*lock
}
type lock struct {
sem chan struct{}
refs int
}
func New() *Locks {
return &Locks{locks: make(map[string]*lock)}
}
// Acquire takes the lock for key, or gives up with ctx's error. The
// returned func releases it and must be called exactly once.
func (k *Locks) Acquire(ctx context.Context, key string) (release func(), err error) {
k.mu.Lock()
l, ok := k.locks[key]
if !ok {
l = &lock{sem: make(chan struct{}, 1)}
k.locks[key] = l
}
l.refs++
k.mu.Unlock()
select {
case l.sem <- struct{}{}:
return func() {
<-l.sem
k.unref(key, l)
}, nil
case <-ctx.Done():
k.unref(key, l)
return nil, ctx.Err()
}
}
func (k *Locks) unref(key string, l *lock) {
k.mu.Lock()
defer k.mu.Unlock()
l.refs--
if l.refs == 0 {
delete(k.locks, key)
}
}
internal/provider/keyedlock/keyedlock_test.go
+90
View File
@@ -0,0 +1,90 @@
package keyedlock
import (
"context"
"errors"
"sync"
"sync/atomic"
"testing"
"time"
)
func lockCount(k *Locks) int {
k.mu.Lock()
defer k.mu.Unlock()
return len(k.locks)
}
func TestKeyedLocks_entryRemovedWhenUncontended(t *testing.T) {
t.Parallel()
k := New()
release, err := k.Acquire(context.Background(), "a")
if err != nil {
t.Fatalf("acquire error = %v", err)
}
if lockCount(k) != 1 {
t.Errorf("entries while held = %d, want 1", lockCount(k))
}
release()
if lockCount(k) != 0 {
t.Errorf("entries after release = %d, want 0", lockCount(k))
}
if _, err := k.Acquire(context.Background(), "a"); err != nil {
t.Fatalf("acquire after release error = %v", err)
}
}
func TestKeyedLocks_waitersShareOneLock(t *testing.T) {
t.Parallel()
k := New()
const n = 20
var inside, overlaps atomic.Int32
var wg sync.WaitGroup
for range n {
wg.Go(func() {
release, err := k.Acquire(context.Background(), "shared")
if err != nil {
t.Errorf("acquire error = %v", err)
return
}
if inside.Add(1) > 1 {
overlaps.Add(1)
}
time.Sleep(time.Millisecond)
inside.Add(-1)
release()
})
}
wg.Wait()
if overlaps.Load() != 0 {
t.Errorf("%d holders overlapped on one key", overlaps.Load())
}
if lockCount(k) != 0 {
t.Errorf("entries after every holder released = %d, want 0", lockCount(k))
}
}
func TestKeyedLocks_waiterGivesUpWithContext(t *testing.T) {
t.Parallel()
k := New()
release, err := k.Acquire(context.Background(), "a")
if err != nil {
t.Fatalf("acquire error = %v", err)
}
ctx, cancel := context.WithTimeout(context.Background(), 20*time.Millisecond)
defer cancel()
start := time.Now()
if _, err := k.Acquire(ctx, "a"); !errors.Is(err, context.DeadlineExceeded) {
t.Fatalf("waiter error = %v, want DeadlineExceeded", err)
}
if time.Since(start) > time.Second {
t.Errorf("waiter took %v, want to give up with its ctx", time.Since(start))
}
if lockCount(k) != 1 {
t.Errorf("entries with the holder still in = %d, want 1: the waiter's exit must not drop the holder's entry", lockCount(k))
}
release()
if lockCount(k) != 0 {
t.Errorf("entries after the holder released = %d, want 0", lockCount(k))
}
}
internal/provider/kvm2/backend.go
+421
View File
@@ -0,0 +1,421 @@
package kvm2
import (
"context"
"errors"
"fmt"
"io"
"sync"
"time"
"github.com/digitalocean/go-libvirt"
"gitea.home.hrajfrisbee.cz/kacerr/egress-proxies-operator/internal/provider"
)
// backend is the seam over go-libvirt (spec entry 17): exactly the calls the
// provider uses, each with ctx first, since go-libvirt's own methods take
// none and block. The production adapter is libvirtBackend; tests use an
// in-memory fake. Nothing outside this package sees it.
type backend interface {
DomainLookupByName(ctx context.Context, name string) (libvirt.Domain, error)
DomainDefineXML(ctx context.Context, xml string) (libvirt.Domain, error)
DomainCreate(ctx context.Context, dom libvirt.Domain) error
DomainDestroy(ctx context.Context, dom libvirt.Domain) error
DomainUndefineFlags(ctx context.Context, dom libvirt.Domain, flags libvirt.DomainUndefineFlagsValues) error
DomainGetState(ctx context.Context, dom libvirt.Domain, flags uint32) (state int32, reason int32, err error)
DomainGetMetadata(ctx context.Context, dom libvirt.Domain, typ int32, uri libvirt.OptString, flags libvirt.DomainModificationImpact) (string, error)
DomainInterfaceAddresses(ctx context.Context, dom libvirt.Domain, source uint32, flags uint32) ([]libvirt.DomainInterface, error)
ConnectListAllDomains(ctx context.Context, needResults int32, flags libvirt.ConnectListAllDomainsFlags) ([]libvirt.Domain, uint32, error)
StoragePoolLookupByName(ctx context.Context, name string) (libvirt.StoragePool, error)
StoragePoolRefresh(ctx context.Context, pool libvirt.StoragePool, flags uint32) error
StoragePoolGetInfo(ctx context.Context, pool libvirt.StoragePool) (state uint8, capacity, allocation, available uint64, err error)
StorageVolLookupByName(ctx context.Context, pool libvirt.StoragePool, name string) (libvirt.StorageVol, error)
StorageVolGetPath(ctx context.Context, vol libvirt.StorageVol) (string, error)
StorageVolCreateXML(ctx context.Context, pool libvirt.StoragePool, xml string, flags libvirt.StorageVolCreateFlags) (libvirt.StorageVol, error)
StorageVolUpload(ctx context.Context, vol libvirt.StorageVol, r io.Reader, offset, length uint64, flags libvirt.StorageVolUploadFlags) error
StorageVolDownload(ctx context.Context, vol libvirt.StorageVol, w io.Writer, offset, length uint64, flags libvirt.StorageVolDownloadFlags) error
StorageVolDelete(ctx context.Context, vol libvirt.StorageVol, flags libvirt.StorageVolDeleteFlags) error
ConnectGetLibVersion(ctx context.Context) (uint64, error)
ConnectGetHostname(ctx context.Context) (string, error)
Close() error
}
// liveness is implemented by backends whose connection can die underneath
// them. The host table drops a backend whose dead channel is closed. It is
// deliberately not part of backend: the fake needs it, the seam does not.
type liveness interface {
dead() <-chan struct{}
}
func isDead(b backend) bool {
l, ok := b.(liveness)
if !ok {
return false
}
select {
case <-l.dead():
return true
default:
return false
}
}
// budgets are the per-call deadlines of spec entry 18, independent of the
// caller's ctx, which by contract carries none.
type budgets struct {
// dial bounds TCP connect, SSH handshake and libvirt connect together.
dial time.Duration
// rpc bounds every ordinary call.
rpc time.Duration
// create bounds DomainCreate, which waits for QEMU to start.
create time.Duration
// seedUpload bounds a StorageVolUpload of at most smallUploadBytes.
seedUpload time.Duration
// transfer caps a large upload or a download; ctx may end it earlier.
transfer time.Duration
}
var defaultBudgets = budgets{
dial: 10 * time.Second,
rpc: 30 * time.Second,
create: 90 * time.Second,
seedUpload: 60 * time.Second,
transfer: 20 * time.Minute,
}
// smallUploadBytes separates a seed ISO (tens of KiB) from a base image
// (hundreds of MiB) for the upload budget.
const smallUploadBytes = 16 << 20
// closeBudget bounds the courtesy ConnectClose RPC in Close.
const closeBudget = 5 * time.Second
// transferGrace keeps the image download's own deadline inside the transfer
// budget. A stalled mirror then ends by its body failing, which the upload
// seam turns into a clean end of stream, instead of by the budget killing
// the socket, which go-libvirt's stream sender does not survive.
const transferGrace = time.Minute
func (b budgets) upload(length uint64) time.Duration {
if length <= smallUploadBytes {
return b.seedUpload
}
return b.transfer
}
// link is the liveness of one connection. dead is closed exactly once, by
// the first of: an abandoned call, a keepalive failure, go-libvirt noticing
// the socket went away, or Close. kill closes the transport, which is the
// only way to unblock a go-libvirt call; go-libvirt then fails every other
// call in flight on the connection with ErrInterrupted.
type link struct {
label string
kill func()
closed chan struct{}
mu sync.Mutex
reason string
}
func newLink(label string, kill func()) *link {
return &link{label: label, kill: kill, closed: make(chan struct{})}
}
func (l *link) dead() <-chan struct{} { return l.closed }
func (l *link) isDead() bool {
select {
case <-l.closed:
return true
default:
return false
}
}
func (l *link) deadReason() string {
l.mu.Lock()
defer l.mu.Unlock()
return l.reason
}
// markDead records why the connection is unusable, signals dead and then
// closes the transport. Later calls are no-ops. Signalling before killing,
// and killing outside the lock, is what keeps a kill that takes its time
// from holding up the callers waiting on dead or the next markDead: a
// transport close can block for as long as the kernel retransmits.
func (l *link) markDead(reason string) {
l.mu.Lock()
if l.isDead() {
l.mu.Unlock()
return
}
l.reason = reason
close(l.closed)
l.mu.Unlock()
if l.kill != nil {
l.kill()
}
}
// call executes a blocking go-libvirt call with a budget and returns its
// result, or Transient when the link is already dead, ctx ends, the budget
// runs out, or another caller kills the link meanwhile. Abandoning a call
// kills the link (spec entry 18); the goroutine running fn then returns on
// its own once go-libvirt sees the closed socket, and its result is dropped.
// Results travel through the channel so an abandoned call never writes
// into its caller's frame.
func call[T any](ctx context.Context, l *link, name string, budget time.Duration, fn func() (T, error)) (T, error) {
var zero T
if l.isDead() {
return zero, transientf("libvirt call %s on %s: connection is closed: %s", name, l.label, l.deadReason())
}
if err := ctx.Err(); err != nil {
return zero, fmt.Errorf("%w: libvirt call %s on %s not started: %w", provider.ErrTransient, name, l.label, err)
}
type result struct {
v T
err error
}
done := make(chan result, 1)
go func() {
v, err := fn()
done <- result{v, err}
}()
timer := time.NewTimer(budget)
defer timer.Stop()
select {
case r := <-done:
return r.v, r.err
case <-ctx.Done():
l.markDead(fmt.Sprintf("call %s abandoned: %v", name, ctx.Err()))
return zero, fmt.Errorf("%w: libvirt call %s on %s abandoned: %w", provider.ErrTransient, name, l.label, ctx.Err())
case <-timer.C:
l.markDead(fmt.Sprintf("call %s exceeded its %s budget", name, budget))
return zero, fmt.Errorf("%w: libvirt call %s on %s exceeded its %s budget: %w", provider.ErrTransient, name, l.label, budget, context.DeadlineExceeded)
case <-l.closed:
return zero, transientf("libvirt call %s on %s: connection closed while waiting: %s", name, l.label, l.deadReason())
}
}
// run is call for go-libvirt methods that return only an error.
func (l *link) run(ctx context.Context, name string, budget time.Duration, fn func() error) error {
_, err := call(ctx, l, name, budget, func() (struct{}, error) { return struct{}{}, fn() })
return err
}
// libvirtBackend is the production adapter: a go-libvirt client over one
// forwarded socket, with the budgets applied around every call. It never
// logs; the observe decorator does.
type libvirtBackend struct {
lv *libvirt.Libvirt
link *link
budgets budgets
closeOnce sync.Once
closeErr error
}
var _ backend = (*libvirtBackend)(nil)
// newLibvirtBackend wraps lv, whose socket runs over transport. Closing the
// transport is how a hung call is abandoned, so it must close the socket
// conn given to libvirt.New as well as whatever carries it.
func newLibvirtBackend(lv *libvirt.Libvirt, label string, transport io.Closer, b budgets) *libvirtBackend {
return &libvirtBackend{
lv: lv,
link: newLink(label, func() { _ = transport.Close() }),
budgets: b,
}
}
func (b *libvirtBackend) dead() <-chan struct{} { return b.link.dead() }
// connect runs the libvirt handshake under the dial budget and, once it
// holds, drops the link as soon as go-libvirt reports the socket gone.
func (b *libvirtBackend) connect(ctx context.Context) error {
if err := b.link.run(ctx, "Connect", b.budgets.dial, b.lv.Connect); err != nil {
return err
}
disconnected := b.lv.Disconnected()
go func() {
select {
case <-disconnected:
b.link.markDead("connection lost")
case <-b.link.closed:
}
}()
return nil
}
func (b *libvirtBackend) DomainLookupByName(ctx context.Context, name string) (libvirt.Domain, error) {
return call(ctx, b.link, "DomainLookupByName", b.budgets.rpc, func() (libvirt.Domain, error) { return b.lv.DomainLookupByName(name) })
}
func (b *libvirtBackend) DomainDefineXML(ctx context.Context, xml string) (libvirt.Domain, error) {
return call(ctx, b.link, "DomainDefineXML", b.budgets.rpc, func() (libvirt.Domain, error) { return b.lv.DomainDefineXML(xml) })
}
func (b *libvirtBackend) DomainCreate(ctx context.Context, dom libvirt.Domain) error {
return b.link.run(ctx, "DomainCreate", b.budgets.create, func() error { return b.lv.DomainCreate(dom) })
}
func (b *libvirtBackend) DomainDestroy(ctx context.Context, dom libvirt.Domain) error {
return b.link.run(ctx, "DomainDestroy", b.budgets.rpc, func() error { return b.lv.DomainDestroy(dom) })
}
func (b *libvirtBackend) DomainUndefineFlags(ctx context.Context, dom libvirt.Domain, flags libvirt.DomainUndefineFlagsValues) error {
return b.link.run(ctx, "DomainUndefineFlags", b.budgets.rpc, func() error { return b.lv.DomainUndefineFlags(dom, flags) })
}
func (b *libvirtBackend) DomainGetState(ctx context.Context, dom libvirt.Domain, flags uint32) (int32, int32, error) {
type stateReason struct{ state, reason int32 }
r, err := call(ctx, b.link, "DomainGetState", b.budgets.rpc, func() (stateReason, error) {
state, reason, err := b.lv.DomainGetState(dom, flags)
return stateReason{state, reason}, err
})
return r.state, r.reason, err
}
func (b *libvirtBackend) DomainGetMetadata(ctx context.Context, dom libvirt.Domain, typ int32, uri libvirt.OptString, flags libvirt.DomainModificationImpact) (string, error) {
return call(ctx, b.link, "DomainGetMetadata", b.budgets.rpc, func() (string, error) { return b.lv.DomainGetMetadata(dom, typ, uri, flags) })
}
func (b *libvirtBackend) DomainInterfaceAddresses(ctx context.Context, dom libvirt.Domain, source uint32, flags uint32) ([]libvirt.DomainInterface, error) {
return call(ctx, b.link, "DomainInterfaceAddresses", b.budgets.rpc, func() ([]libvirt.DomainInterface, error) {
return b.lv.DomainInterfaceAddresses(dom, source, flags)
})
}
func (b *libvirtBackend) ConnectListAllDomains(ctx context.Context, needResults int32, flags libvirt.ConnectListAllDomainsFlags) ([]libvirt.Domain, uint32, error) {
type listed struct {
doms []libvirt.Domain
n uint32
}
r, err := call(ctx, b.link, "ConnectListAllDomains", b.budgets.rpc, func() (listed, error) {
doms, n, err := b.lv.ConnectListAllDomains(needResults, flags)
return listed{doms, n}, err
})
return r.doms, r.n, err
}
func (b *libvirtBackend) StoragePoolLookupByName(ctx context.Context, name string) (libvirt.StoragePool, error) {
return call(ctx, b.link, "StoragePoolLookupByName", b.budgets.rpc, func() (libvirt.StoragePool, error) { return b.lv.StoragePoolLookupByName(name) })
}
func (b *libvirtBackend) StoragePoolRefresh(ctx context.Context, pool libvirt.StoragePool, flags uint32) error {
return b.link.run(ctx, "StoragePoolRefresh", b.budgets.rpc, func() error { return b.lv.StoragePoolRefresh(pool, flags) })
}
func (b *libvirtBackend) StoragePoolGetInfo(ctx context.Context, pool libvirt.StoragePool) (uint8, uint64, uint64, uint64, error) {
type info struct {
state uint8
capacity, allocation, available uint64
}
r, err := call(ctx, b.link, "StoragePoolGetInfo", b.budgets.rpc, func() (info, error) {
state, capacity, allocation, available, err := b.lv.StoragePoolGetInfo(pool)
return info{state, capacity, allocation, available}, err
})
return r.state, r.capacity, r.allocation, r.available, err
}
func (b *libvirtBackend) StorageVolLookupByName(ctx context.Context, pool libvirt.StoragePool, name string) (libvirt.StorageVol, error) {
return call(ctx, b.link, "StorageVolLookupByName", b.budgets.rpc, func() (libvirt.StorageVol, error) { return b.lv.StorageVolLookupByName(pool, name) })
}
func (b *libvirtBackend) StorageVolGetPath(ctx context.Context, vol libvirt.StorageVol) (string, error) {
return call(ctx, b.link, "StorageVolGetPath", b.budgets.rpc, func() (string, error) { return b.lv.StorageVolGetPath(vol) })
}
func (b *libvirtBackend) StorageVolCreateXML(ctx context.Context, pool libvirt.StoragePool, xml string, flags libvirt.StorageVolCreateFlags) (libvirt.StorageVol, error) {
return call(ctx, b.link, "StorageVolCreateXML", b.budgets.rpc, func() (libvirt.StorageVol, error) { return b.lv.StorageVolCreateXML(pool, xml, flags) })
}
func (b *libvirtBackend) StorageVolUpload(ctx context.Context, vol libvirt.StorageVol, r io.Reader, offset, length uint64, flags libvirt.StorageVolUploadFlags) error {
src := &eofReader{r: r}
err := b.link.run(ctx, "StorageVolUpload", b.budgets.upload(length), func() error {
return b.lv.StorageVolUpload(vol, src, offset, length, flags)
})
if err == nil {
if srcErr := src.sourceErr(); srcErr != nil {
return fmt.Errorf("reading the upload source for %s: %w", vol.Name, srcErr)
}
}
return err
}
// eofReader hides a source error from go-libvirt's stream sender and keeps
// it for the caller. SendStream answers a non-EOF error with a StatusError
// packet and then blocks forever handing its result to an unbuffered
// channel that requestStream no longer reads (go-libvirt rpc.go), which
// wedges the call and leaks the goroutine along with its 4 MiB buffer;
// closing the socket does not help, because both ends are parked on
// channel sends rather than on I/O. io.EOF is the one outcome that ends
// the stream the normal way. The field is guarded because an abandoned
// call returns while go-libvirt is still reading.
type eofReader struct {
r io.Reader
mu sync.Mutex
err error
}
func (e *eofReader) Read(b []byte) (int, error) {
n, err := e.r.Read(b)
if err == nil || errors.Is(err, io.EOF) {
return n, err
}
e.mu.Lock()
if e.err == nil {
e.err = err
}
e.mu.Unlock()
return n, io.EOF
}
func (e *eofReader) sourceErr() error {
e.mu.Lock()
defer e.mu.Unlock()
return e.err
}
func (b *libvirtBackend) StorageVolDownload(ctx context.Context, vol libvirt.StorageVol, w io.Writer, offset, length uint64, flags libvirt.StorageVolDownloadFlags) error {
return b.link.run(ctx, "StorageVolDownload", b.budgets.transfer, func() error {
return b.lv.StorageVolDownload(vol, w, offset, length, flags)
})
}
func (b *libvirtBackend) StorageVolDelete(ctx context.Context, vol libvirt.StorageVol, flags libvirt.StorageVolDeleteFlags) error {
return b.link.run(ctx, "StorageVolDelete", b.budgets.rpc, func() error { return b.lv.StorageVolDelete(vol, flags) })
}
func (b *libvirtBackend) ConnectGetLibVersion(ctx context.Context) (uint64, error) {
return call(ctx, b.link, "ConnectGetLibVersion", b.budgets.rpc, b.lv.ConnectGetLibVersion)
}
func (b *libvirtBackend) ConnectGetHostname(ctx context.Context) (string, error) {
return call(ctx, b.link, "ConnectGetHostname", b.budgets.rpc, b.lv.ConnectGetHostname)
}
// Close sends libvirt's ConnectClose when the link is still live, bounded
// because a hung host would otherwise pin the caller, then kills the link,
// which closes the socket and the SSH client. Repeat calls return the first
// result.
func (b *libvirtBackend) Close() error {
b.closeOnce.Do(func() {
if b.link.isDead() {
return
}
done := make(chan error, 1)
go func() { done <- b.lv.Disconnect() }()
timer := time.NewTimer(closeBudget)
defer timer.Stop()
select {
case b.closeErr = <-done:
case <-timer.C:
b.closeErr = transientf("libvirt disconnect on %s exceeded its %s budget", b.link.label, closeBudget)
case <-b.link.closed:
}
b.link.markDead("closed")
})
return b.closeErr
}
internal/provider/kvm2/backend_test.go
+369
View File
@@ -0,0 +1,369 @@
package kvm2
import (
"context"
"errors"
"io"
"strings"
"sync/atomic"
"testing"
"time"
"github.com/digitalocean/go-libvirt"
"github.com/digitalocean/go-libvirt/libvirttest"
"gitea.home.hrajfrisbee.cz/kacerr/egress-proxies-operator/internal/provider"
)
// blockingFn returns a func that blocks until release is closed, plus a
// counter of how often it was started.
func blockingFn(t *testing.T) (fn func() error, started *atomic.Int32) {
t.Helper()
release := make(chan struct{})
t.Cleanup(func() { close(release) })
started = &atomic.Int32{}
return func() error {
started.Add(1)
<-release
return nil
}, started
}
func TestCall_deadlineClosesConnectionAndReturnsTransient(t *testing.T) {
t.Parallel()
var killed atomic.Int32
l := newLink("beelink", func() { killed.Add(1) })
fn, started := blockingFn(t)
start := time.Now()
err := l.run(context.Background(), "DomainCreate", 50*time.Millisecond, fn)
elapsed := time.Since(start)
if !errors.Is(err, provider.ErrTransient) {
t.Fatalf("run() error = %v, want ErrTransient", err)
}
if !errors.Is(err, context.DeadlineExceeded) {
t.Errorf("run() error = %v, want it to wrap context.DeadlineExceeded", err)
}
if !strings.Contains(err.Error(), "DomainCreate") || !strings.Contains(err.Error(), "beelink") {
t.Errorf("run() error = %q, want the call name and host in it", err)
}
if elapsed > time.Second {
t.Errorf("run() took %v, want about the 50ms budget", elapsed)
}
if killed.Load() != 1 {
t.Errorf("kill hook called %d times, want 1", killed.Load())
}
if !l.isDead() {
t.Error("link not dead after an abandoned call")
}
if !strings.Contains(l.deadReason(), "budget") {
t.Errorf("deadReason() = %q, want the budget mentioned", l.deadReason())
}
start = time.Now()
err = l.run(context.Background(), "DomainGetState", time.Minute, fn)
if !errors.Is(err, provider.ErrTransient) {
t.Fatalf("second run() error = %v, want ErrTransient", err)
}
if time.Since(start) > 100*time.Millisecond {
t.Errorf("second run() took %v, want immediate", time.Since(start))
}
if started.Load() != 1 {
t.Errorf("fn started %d times, want 1: a dead link must not touch the socket", started.Load())
}
if killed.Load() != 1 {
t.Errorf("kill hook called %d times, want still 1", killed.Load())
}
}
func TestCall_cancelUnblocksRPC(t *testing.T) {
t.Parallel()
var killed atomic.Int32
l := newLink("beelink", func() { killed.Add(1) })
fn, _ := blockingFn(t)
ctx, cancel := context.WithCancel(context.Background())
go func() {
time.Sleep(10 * time.Millisecond)
cancel()
}()
start := time.Now()
err := l.run(ctx, "ConnectListAllDomains", time.Minute, fn)
if time.Since(start) > 100*time.Millisecond {
t.Errorf("run() took %v after cancel, want under 100ms", time.Since(start))
}
if !errors.Is(err, provider.ErrTransient) || !errors.Is(err, context.Canceled) {
t.Errorf("run() error = %v, want ErrTransient wrapping context.Canceled", err)
}
if killed.Load() != 1 {
t.Errorf("kill hook called %d times, want 1", killed.Load())
}
}
func TestCall_alreadyCancelledDoesNotStart(t *testing.T) {
t.Parallel()
l := newLink("beelink", nil)
fn, started := blockingFn(t)
ctx, cancel := context.WithCancel(context.Background())
cancel()
err := l.run(ctx, "DomainGetState", time.Minute, fn)
if !errors.Is(err, provider.ErrTransient) || !errors.Is(err, context.Canceled) {
t.Errorf("run() error = %v, want ErrTransient wrapping context.Canceled", err)
}
if started.Load() != 0 {
t.Error("fn started although ctx was already cancelled")
}
if l.isDead() {
t.Error("link marked dead although nothing was abandoned")
}
}
func TestCall_returnsFnResult(t *testing.T) {
t.Parallel()
l := newLink("beelink", nil)
want := libvirt.Error{Code: uint32(libvirt.ErrNoDomain), Message: "gone"}
err := l.run(context.Background(), "DomainLookupByName", time.Second, func() error { return want })
if !hasCode(err, libvirt.ErrNoDomain) {
t.Errorf("run() error = %v, want the libvirt error through unchanged", err)
}
if l.isDead() {
t.Error("a backend error must not kill the link")
}
}
func TestCall_killedByAnotherCallerFailsTransient(t *testing.T) {
t.Parallel()
l := newLink("beelink", nil)
fn, _ := blockingFn(t)
done := make(chan error, 1)
go func() { done <- l.run(context.Background(), "DomainGetState", time.Minute, fn) }()
time.Sleep(10 * time.Millisecond)
l.markDead("keepalive failed")
select {
case err := <-done:
if !errors.Is(err, provider.ErrTransient) || !strings.Contains(err.Error(), "keepalive failed") {
t.Errorf("run() error = %v, want ErrTransient with the death reason", err)
}
case <-time.After(time.Second):
t.Fatal("run() did not return after the link was killed")
}
}
// TestMarkDead_slowKillDoesNotBlockCallers pins the guarantee the pooled
// connection rests on: a kill that takes its time — closing a transport can
// block for as long as the kernel retransmits to a host that went away —
// holds up neither the callers waiting on dead nor the next markDead. The
// call's own fn signals when it is in flight, so the parked caller is
// genuinely parked rather than refused by the dead check on the way in.
func TestMarkDead_slowKillDoesNotBlockCallers(t *testing.T) {
t.Parallel()
release := make(chan struct{})
t.Cleanup(func() { close(release) })
killing := make(chan struct{})
var killed atomic.Int32
l := newLink("beelink", func() {
killed.Add(1)
close(killing)
<-release
})
inflight := make(chan struct{})
waiting := make(chan error, 1)
go func() {
waiting <- l.run(context.Background(), "StorageVolUpload", time.Minute, func() error {
close(inflight)
<-release
return nil
})
}()
<-inflight
go l.markDead("keepalive failed")
<-killing
select {
case <-l.dead():
case <-time.After(time.Second):
t.Fatal("link not signalled dead while its kill is still running")
}
select {
case err := <-waiting:
if !errors.Is(err, provider.ErrTransient) || !strings.Contains(err.Error(), "keepalive failed") {
t.Errorf("parked call error = %v, want ErrTransient with the death reason", err)
}
case <-time.After(time.Second):
t.Fatal("parked call not released while the kill is still running")
}
second := make(chan struct{})
go func() {
l.markDead("closed")
close(second)
}()
select {
case <-second:
case <-time.After(time.Second):
t.Fatal("second markDead blocked behind the running kill")
}
if killed.Load() != 1 {
t.Errorf("kill hook called %d times, want 1", killed.Load())
}
if !strings.Contains(l.deadReason(), "keepalive failed") {
t.Errorf("deadReason() = %q, want the first reason", l.deadReason())
}
}
// TestEOFReader_sourceErrorNeverReachesTheStream pins the one property the
// upload seam needs: go-libvirt's stream sender must never see an error
// other than io.EOF, because it answers one by parking forever on a channel
// nothing reads. The real error has to survive for the caller.
func TestEOFReader_sourceErrorNeverReachesTheStream(t *testing.T) {
t.Parallel()
boom := errors.New("connection reset by peer")
tests := []struct {
name string
reader io.Reader
wantN int
wantErr error
}{
{name: "error after data", reader: io.MultiReader(strings.NewReader("abc"), errReader{boom}), wantN: 3, wantErr: boom},
{name: "error with no data", reader: errReader{boom}, wantErr: boom},
{name: "clean eof passes through", reader: strings.NewReader("abc"), wantN: 3},
}
for _, tc := range tests {
t.Run(tc.name, func(t *testing.T) {
t.Parallel()
e := &eofReader{r: tc.reader}
n, err := io.Copy(io.Discard, e)
if err != nil {
t.Fatalf("io.Copy through eofReader = %v, want nil: the stream sender must only ever see io.EOF", err)
}
if n != int64(tc.wantN) {
t.Errorf("copied %d bytes, want %d", n, tc.wantN)
}
if !errors.Is(e.sourceErr(), tc.wantErr) {
t.Errorf("sourceErr() = %v, want %v", e.sourceErr(), tc.wantErr)
}
})
}
}
func TestEOFReader_keepsTheFirstError(t *testing.T) {
t.Parallel()
first, second := errors.New("first"), errors.New("second")
e := &eofReader{r: io.MultiReader(errReader{first}, errReader{second})}
if _, err := io.Copy(io.Discard, e); err != nil {
t.Fatalf("io.Copy through eofReader = %v, want nil", err)
}
if !errors.Is(e.sourceErr(), first) {
t.Errorf("sourceErr() = %v, want the first error", e.sourceErr())
}
}
type errReader struct{ err error }
func (r errReader) Read([]byte) (int, error) { return 0, r.err }
func TestBudgets_upload(t *testing.T) {
t.Parallel()
tests := []struct {
name string
length uint64
want time.Duration
}{
{name: "seed sized", length: 400 << 10, want: defaultBudgets.seedUpload},
{name: "exactly the limit", length: smallUploadBytes, want: defaultBudgets.seedUpload},
{name: "one over the limit", length: smallUploadBytes + 1, want: defaultBudgets.transfer},
{name: "base image", length: 435 << 20, want: defaultBudgets.transfer},
}
for _, tc := range tests {
t.Run(tc.name, func(t *testing.T) {
t.Parallel()
if got := defaultBudgets.upload(tc.length); got != tc.want {
t.Errorf("upload(%d) = %v, want %v", tc.length, got, tc.want)
}
})
}
}
// mockAdapter wires a libvirtBackend to go-libvirt's own mock server over a
// net.Pipe, so the adapter is exercised with the real RPC client.
func mockAdapter(t *testing.T, b budgets) *libvirtBackend {
t.Helper()
m := libvirttest.New()
conn, err := m.Dial()
if err != nil {
t.Fatalf("mock dial: %v", err)
}
ad := newLibvirtBackend(libvirt.New(conn), "mock", conn, b)
if err := ad.connect(context.Background()); err != nil {
t.Fatalf("connect: %v", err)
}
t.Cleanup(func() { _ = ad.Close() })
return ad
}
func TestLibvirtBackend_roundTripThroughMock(t *testing.T) {
t.Parallel()
ad := mockAdapter(t, defaultBudgets)
v, err := ad.ConnectGetLibVersion(context.Background())
if err != nil {
t.Fatalf("ConnectGetLibVersion() error = %v", err)
}
if v == 0 {
t.Error("ConnectGetLibVersion() = 0, want the mock's version")
}
pool, err := ad.StoragePoolLookupByName(context.Background(), "default")
if err != nil {
t.Fatalf("StoragePoolLookupByName() error = %v", err)
}
if pool.Name == "" {
t.Error("StoragePoolLookupByName() returned an empty pool")
}
if isDead(ad) {
t.Error("adapter dead after successful calls")
}
}
func TestLibvirtBackend_unansweredCallIsAbandoned(t *testing.T) {
t.Parallel()
b := defaultBudgets
b.rpc = 100 * time.Millisecond
ad := mockAdapter(t, b)
// The mock does not implement ConnectGetHostname, so it never replies.
_, err := ad.ConnectGetHostname(context.Background())
if !errors.Is(err, provider.ErrTransient) || !errors.Is(err, context.DeadlineExceeded) {
t.Fatalf("ConnectGetHostname() error = %v, want ErrTransient wrapping DeadlineExceeded", err)
}
if !isDead(ad) {
t.Error("adapter not dead after an abandoned call")
}
select {
case <-ad.lv.Disconnected():
case <-time.After(time.Second):
t.Error("go-libvirt did not notice the closed socket")
}
if _, err := ad.ConnectGetLibVersion(context.Background()); !errors.Is(err, provider.ErrTransient) {
t.Errorf("call on dead adapter error = %v, want ErrTransient", err)
}
if err := ad.Close(); err != nil {
t.Errorf("Close() on dead adapter error = %v, want nil", err)
}
}
func TestLibvirtBackend_closeIsIdempotent(t *testing.T) {
t.Parallel()
ad := mockAdapter(t, defaultBudgets)
first := ad.Close()
second := ad.Close()
if first != nil || second != nil {
t.Errorf("Close() errors = %v, %v, want nil, nil", first, second)
}
if !isDead(ad) {
t.Error("adapter not dead after Close")
}
if _, err := ad.ConnectGetLibVersion(context.Background()); !errors.Is(err, provider.ErrTransient) {
t.Errorf("call after Close error = %v, want ErrTransient", err)
}
}
internal/provider/kvm2/check.go
+189
View File
@@ -0,0 +1,189 @@
package kvm2
import (
"context"
"errors"
"fmt"
"strings"
"sync"
"time"
"github.com/digitalocean/go-libvirt"
"golang.org/x/crypto/ssh/knownhosts"
"gitea.home.hrajfrisbee.cz/kacerr/egress-proxies-operator/internal/provider"
)
var _ provider.Checker = (*Provider)(nil)
// Reasons only this provider can tell apart (spec entry 16). Each names a
// failure an admin fixes somewhere other than where the shared reasons
// point.
const (
// reasonCredentialsUnavailable: the key or known_hosts file is missing
// or unparsable; fix the Secret in the cluster, not the host.
reasonCredentialsUnavailable = "CredentialsUnavailable"
// reasonPoolNotFound: the configured pool is not defined on the host;
// compare with `virsh pool-list --all`.
reasonPoolNotFound = "PoolNotFound"
// reasonPoolNotActive: the pool exists but is not running; `virsh
// pool-start <pool>` (and autostart) brings it back.
reasonPoolNotActive = "PoolNotActive"
)
// Check verifies every configured host concurrently, each on a fresh
// dedicated connection: dial, libvirt version, hostname, pool lookup, pool
// info, close. The pooled connection is never touched, so a check that runs
// out of time closes only its own connection and cannot fail a reconcile's
// call in flight. Every target gets a verdict, in config order.
func (p *Provider) Check(ctx context.Context) []provider.TargetStatus {
out := make([]provider.TargetStatus, len(p.cfg.Hosts))
var wg sync.WaitGroup
for i, host := range p.cfg.Hosts {
wg.Go(func() { out[i] = p.checkHost(ctx, host) })
}
wg.Wait()
return out
}
func (p *Provider) checkHost(ctx context.Context, host provider.KVMHost) provider.TargetStatus {
start := time.Now()
log := p.logger(ctx).WithValues("host", host.Name)
ts := provider.TargetStatus{
Name: host.Name,
Address: host.Address,
Details: map[string]string{"user": host.User, "pool": host.Pool},
}
err := p.probeHost(ctx, host, ts.Details)
ts.Latency = time.Since(start)
switch {
case err == nil:
ts.Ready = true
ts.Reason = provider.ReasonReachable
default:
var inactive *poolNotActiveError
switch {
case ctx.Err() != nil:
// The round ran out of time, whichever stage was running: the
// error's shape (a dial that saw the ctx end, an abandoned
// call) says nothing the admin should act on.
ts.Reason = provider.ReasonTimeout
case errors.As(err, &inactive):
ts.Reason = reasonPoolNotActive
default:
ts.Reason = checkReason(err)
}
ts.Message = err.Error()
}
log.V(1).Info("kvm2 host checked", "ready", ts.Ready, "reason", ts.Reason, "latency", ts.Latency)
return ts
}
// probeHost runs the check sequence on its own connection and fills details
// as facts arrive, so a failure late in the sequence still reports what the
// host said before it.
func (p *Provider) probeHost(ctx context.Context, host provider.KVMHost, details map[string]string) error {
b, err := p.hosts.dialFresh(ctx, host)
if err != nil {
return err
}
defer func() { _ = b.Close() }()
version, err := b.ConnectGetLibVersion(ctx)
if err != nil {
return err
}
details["libvirtVersion"] = formatLibVersion(version)
hostname, err := b.ConnectGetHostname(ctx)
if err != nil {
return err
}
details["hostname"] = hostname
pool, err := b.StoragePoolLookupByName(ctx, host.Pool)
if err != nil {
return err
}
state, capacity, _, available, err := b.StoragePoolGetInfo(ctx, pool)
if err != nil {
return err
}
details["poolState"] = poolStateName(state)
details["poolCapacityGiB"] = formatGiB(capacity)
details["poolAvailableGiB"] = formatGiB(available)
if libvirt.StoragePoolState(state) != libvirt.StoragePoolRunning {
return &poolNotActiveError{pool: host.Pool, state: poolStateName(state)}
}
return nil
}
// poolNotActiveError is the one verdict that is not an error from the
// backend: the pool answered, but is not usable.
type poolNotActiveError struct {
pool string
state string
}
func (e *poolNotActiveError) Error() string {
return fmt.Sprintf("storage pool %q is %s", e.pool, e.state)
}
// checkReason maps a failed probe to a reason once checkHost has ruled out
// the round's ctx ending. A dial failure is judged by its stage, so a TCP
// connect or an SSH handshake that ran out of the dial budget on its own is
// Unreachable, not Timeout: the host did not answer. Timeout is for a
// budget expiring once the connection was up.
func checkReason(err error) string {
var de *dialError
if errors.As(err, &de) {
switch de.stage {
case stageCredentials:
return reasonCredentialsUnavailable
case stageSSH:
var keyErr *knownhosts.KeyError
if errors.As(de.err, &keyErr) || strings.Contains(de.err.Error(), sshAuthFailed) {
return provider.ReasonUnauthorized
}
}
return provider.ReasonUnreachable
}
switch {
case errors.Is(err, context.DeadlineExceeded), errors.Is(err, context.Canceled):
return provider.ReasonTimeout
case hasCode(err, libvirt.ErrAccessDenied), hasCode(err, libvirt.ErrAuthFailed):
return provider.ReasonForbidden
case hasCode(err, libvirt.ErrNoStoragePool):
return reasonPoolNotFound
}
if _, ok := libvirtCode(err); ok {
return provider.ReasonAPIError
}
return provider.ReasonUnreachable
}
// formatLibVersion decodes ConnectGetLibVersion's major*1_000_000 +
// minor*1_000 + micro.
func formatLibVersion(v uint64) string {
return fmt.Sprintf("%d.%d.%d", v/1_000_000, v/1_000%1_000, v%1_000)
}
// poolStateName uses virsh's words for a pool state.
func poolStateName(state uint8) string {
switch libvirt.StoragePoolState(state) {
case libvirt.StoragePoolInactive:
return "inactive"
case libvirt.StoragePoolBuilding:
return "building"
case libvirt.StoragePoolRunning:
return "running"
case libvirt.StoragePoolDegraded:
return "degraded"
case libvirt.StoragePoolInaccessible:
return "inaccessible"
default:
return fmt.Sprintf("state %d", state)
}
}
func formatGiB(bytes uint64) string {
return fmt.Sprintf("%.1f", float64(bytes)/(1<<30))
}
internal/provider/kvm2/check_test.go
+579
View File
@@ -0,0 +1,579 @@
package kvm2
import (
"context"
"errors"
"fmt"
"os"
"path/filepath"
"slices"
"strings"
"sync/atomic"
"testing"
"time"
"github.com/digitalocean/go-libvirt"
"github.com/go-logr/logr/funcr"
"golang.org/x/crypto/ssh/knownhosts"
logf "sigs.k8s.io/controller-runtime/pkg/log"
"gitea.home.hrajfrisbee.cz/kacerr/egress-proxies-operator/internal/provider"
)
// checkContext is captureContext with a sink that tolerates the concurrent
// per-host log lines of Check.
func checkContext() (context.Context, *logCapture) {
logs := &logCapture{}
log := funcr.New(func(prefix, args string) {
logs.mu.Lock()
defer logs.mu.Unlock()
logs.lines = append(logs.lines, prefix+" "+args)
}, funcr.Options{Verbosity: 2})
return logf.IntoContext(context.Background(), log), logs
}
// checkDialer hands out a distinct fake per host name, fails the hosts in
// errs, counts every dial, and, with block set, waits for ctx to end and
// then fails the way dialSSHSocket does when the TCP connect sees the ctx
// end: a dialError at the tcp stage wrapping the ctx's error.
type checkDialer struct {
backends map[string]*fakeBackend
errs map[string]error
block bool
dials atomic.Int32
}
func (d *checkDialer) dial(ctx context.Context, host provider.KVMHost) (backend, error) {
d.dials.Add(1)
if d.block {
<-ctx.Done()
return nil, &dialError{stage: stageTCP, host: host.Name, err: fmt.Errorf("dial tcp %s: %w", host.Address, ctx.Err())}
}
if err := d.errs[host.Name]; err != nil {
return nil, err
}
b, ok := d.backends[host.Name]
if !ok {
return nil, fmt.Errorf("checkDialer: no backend for %s", host.Name)
}
return b, nil
}
func threeHosts() provider.KVMConfig {
cfg := twoHosts()
cfg.Hosts = append(cfg.Hosts, provider.KVMHost{Name: "lab3", Address: "lab3:22", User: "ops", Bridge: "br0", Pool: "ssd"})
return cfg
}
func fakesFor(cfg provider.KVMConfig) map[string]*fakeBackend {
out := make(map[string]*fakeBackend, len(cfg.Hosts))
for _, h := range cfg.Hosts {
f := newFakeBackend()
f.hostname = h.Name + ".home.lan"
if h.Pool != "" {
f.pools[h.Pool] = newFakePool()
}
out[h.Name] = f
}
return out
}
func detailKeys(d map[string]string) []string {
keys := make([]string, 0, len(d))
for k := range d {
keys = append(keys, k)
}
slices.Sort(keys)
return keys
}
func TestCheck_oneEntryPerHostInConfigOrder(t *testing.T) {
t.Parallel()
cfg := threeHosts()
d := &checkDialer{backends: fakesFor(cfg), errs: map[string]error{"lab2": errors.New("dial tcp 192.0.2.2:22: connect: no route to host")}}
p := newProvider(testProvider, cfg, d)
ctx, _ := checkContext()
got := p.Check(ctx)
if len(got) != len(cfg.Hosts) {
t.Fatalf("Check() returned %d targets, want %d", len(got), len(cfg.Hosts))
}
for i, h := range cfg.Hosts {
ts := got[i]
if ts.Name != h.Name || ts.Address != h.Address {
t.Errorf("target %d = %s (%s), want %s (%s)", i, ts.Name, ts.Address, h.Name, h.Address)
}
if ts.Name == "" || ts.Address == "" {
t.Errorf("target %d has an empty name or address: %+v", i, ts)
}
wantReady := h.Name != "lab2"
if ts.Ready != wantReady {
t.Errorf("target %s ready = %v, want %v (%s: %s)", ts.Name, ts.Ready, wantReady, ts.Reason, ts.Message)
}
}
if got[1].Reason != provider.ReasonUnreachable || got[0].Reason != provider.ReasonReachable {
t.Errorf("reasons = %s, %s, %s", got[0].Reason, got[1].Reason, got[2].Reason)
}
}
func TestCheck_hostsCheckedConcurrently(t *testing.T) {
t.Parallel()
cfg := twoHosts()
fakes := fakesFor(cfg)
for _, f := range fakes {
release := f.blockOn("ConnectGetLibVersion")
time.AfterFunc(200*time.Millisecond, func() { close(release) })
}
p := newProvider(testProvider, cfg, &checkDialer{backends: fakes})
ctx, _ := checkContext()
start := time.Now()
got := p.Check(ctx)
elapsed := time.Since(start)
if elapsed > 350*time.Millisecond {
t.Errorf("Check() took %v, want under 350ms for two 200ms hosts", elapsed)
}
for _, ts := range got {
if !ts.Ready {
t.Errorf("target %s not ready: %s: %s", ts.Name, ts.Reason, ts.Message)
}
if ts.Latency < 200*time.Millisecond {
t.Errorf("target %s latency = %v, want at least the 200ms block", ts.Name, ts.Latency)
}
}
}
// TestCheck_expiredContextReturnsPromptly drives the production shape of a
// round that ran out of time during the dial: the dialer fails at the tcp
// stage wrapping context.DeadlineExceeded, which checkReason alone would
// call Unreachable; the verdict must still be Timeout because the round's
// ctx ended.
func TestCheck_expiredContextReturnsPromptly(t *testing.T) {
t.Parallel()
cfg := twoHosts()
p := newProvider(testProvider, cfg, &checkDialer{block: true})
ctx, _ := checkContext()
ctx, cancel := context.WithTimeout(ctx, 0)
defer cancel()
start := time.Now()
got := p.Check(ctx)
if elapsed := time.Since(start); elapsed > time.Second {
t.Errorf("Check() took %v with an expired ctx, want under 1s", elapsed)
}
if len(got) != len(cfg.Hosts) {
t.Fatalf("Check() returned %d targets, want %d", len(got), len(cfg.Hosts))
}
for _, ts := range got {
if ts.Ready || ts.Reason != provider.ReasonTimeout {
t.Errorf("target %s = ready %v reason %s, want not ready, Timeout", ts.Name, ts.Ready, ts.Reason)
}
if !strings.Contains(ts.Message, context.DeadlineExceeded.Error()) || !strings.Contains(ts.Message, string(stageTCP)) {
t.Errorf("target %s message = %q, want the tcp stage and the deadline named", ts.Name, ts.Message)
}
}
}
// keyMaterial is the sentinel in the unusable key file of the credentials
// row; it must never reach a message.
const keyMaterial = "SENTINEL-PRIVATE-KEY-BYTES-9f3a"
func TestCheck_reasonsTable(t *testing.T) {
t.Parallel()
sshHandshake := func(cause error) error {
return &dialError{stage: stageSSH, host: testHost, err: fmt.Errorf("ssh: handshake failed: %w", cause)}
}
tests := []struct {
name string
// dialErr builds the dial failure; nil means the dial succeeds.
dialErr func(t *testing.T) error
prep func(f *fakeBackend)
timeout time.Duration
cancel bool
wantReady bool
wantReason string
wantMessage string
}{
{
name: "credentials unusable",
dialErr: func(t *testing.T) error {
path := filepath.Join(t.TempDir(), "id_ed25519")
if err := os.WriteFile(path, []byte(keyMaterial), 0o600); err != nil {
t.Fatal(err)
}
_, err := loadCredentials(provider.KVMConfig{SSHKeyFile: path, KnownHostsFile: "/nonexistent/known_hosts"})
if err == nil {
t.Fatal("loadCredentials accepted the sentinel file")
}
return &dialError{stage: stageCredentials, host: testHost, err: err}
},
wantReason: reasonCredentialsUnavailable,
},
{
name: "credentials missing",
dialErr: func(*testing.T) error {
_, err := loadCredentials(provider.KVMConfig{SSHKeyFile: "/nonexistent/kvm2/id_ed25519"})
return &dialError{stage: stageCredentials, host: testHost, err: err}
},
wantReason: reasonCredentialsUnavailable,
},
{
name: "tcp refused",
dialErr: func(*testing.T) error {
return &dialError{stage: stageTCP, host: testHost, err: errors.New("dial tcp 192.0.2.1:22: connect: connection refused")}
},
wantReason: provider.ReasonUnreachable,
},
{
name: "tcp timed out on the dial budget but ctx is fine",
dialErr: func(*testing.T) error {
return &dialError{stage: stageTCP, host: testHost, err: fmt.Errorf("dial tcp 192.0.2.1:22: %w", context.DeadlineExceeded)}
},
wantReason: provider.ReasonUnreachable,
},
{
name: "ssh host key unknown",
dialErr: func(*testing.T) error { return sshHandshake(&knownhosts.KeyError{}) },
wantReason: provider.ReasonUnauthorized,
},
{
name: "ssh host key mismatch",
dialErr: func(*testing.T) error {
return sshHandshake(&knownhosts.KeyError{Want: []knownhosts.KnownKey{{Filename: "known_hosts", Line: 3}}})
},
wantReason: provider.ReasonUnauthorized,
},
{
name: "ssh authentication refused",
dialErr: func(*testing.T) error {
return sshHandshake(errors.New("ssh: unable to authenticate, attempted methods [none publickey], no supported methods remain"))
},
wantReason: provider.ReasonUnauthorized,
},
{
name: "ssh handshake reset",
dialErr: func(*testing.T) error { return sshHandshake(errors.New("read tcp: connection reset by peer")) },
wantReason: provider.ReasonUnreachable,
},
{
name: "ssh handshake timed out on the dial budget but ctx is fine",
dialErr: func(*testing.T) error { return sshHandshake(context.DeadlineExceeded) },
wantReason: provider.ReasonUnreachable,
},
{
name: "socket refused",
dialErr: func(*testing.T) error {
return &dialError{stage: stageSocket, host: testHost, err: errors.New("opening /var/run/libvirt/libvirt-sock: ssh: rejected: connect failed (No such file or directory)")}
},
wantReason: provider.ReasonUnreachable,
wantMessage: "/var/run/libvirt/libvirt-sock",
},
{
name: "libvirt access denied",
prep: func(f *fakeBackend) {
f.failWith("ConnectGetLibVersion", fakeErr(libvirt.ErrAccessDenied, "access denied"))
},
wantReason: provider.ReasonForbidden,
},
{
name: "libvirt auth failed",
prep: func(f *fakeBackend) {
f.failWith("ConnectGetHostname", fakeErr(libvirt.ErrAuthFailed, "authentication failed"))
},
wantReason: provider.ReasonForbidden,
},
{
name: "pool not found",
prep: func(f *fakeBackend) {
f.mu.Lock()
delete(f.pools, fakePoolName)
f.mu.Unlock()
},
wantReason: reasonPoolNotFound,
wantMessage: fakePoolName,
},
{
name: "other libvirt error",
prep: func(f *fakeBackend) {
f.failWith("StoragePoolGetInfo", fakeErr(libvirt.ErrInternalError, "internal error: pool info"))
},
wantReason: provider.ReasonAPIError,
},
{
name: "rpc outlives the deadline",
prep: func(f *fakeBackend) { f.blockOn("ConnectGetLibVersion") },
timeout: 50 * time.Millisecond,
wantReason: provider.ReasonTimeout,
},
{
name: "ctx cancelled mid check",
prep: func(f *fakeBackend) { f.blockOn("StoragePoolLookupByName") },
cancel: true,
wantReason: provider.ReasonTimeout,
},
{
name: "link killed mid check",
prep: func(f *fakeBackend) {
f.failWith("StoragePoolLookupByName", transientf("libvirt call StoragePoolLookupByName on beelink: connection closed while waiting: keepalive: EOF"))
},
wantReason: provider.ReasonUnreachable,
},
{
name: "pool inactive",
prep: func(f *fakeBackend) { f.pools[fakePoolName].state = libvirt.StoragePoolInactive },
wantReason: reasonPoolNotActive,
wantMessage: `storage pool "default" is inactive`,
},
{
name: "ok",
wantReady: true,
wantReason: provider.ReasonReachable,
},
}
for _, tc := range tests {
t.Run(tc.name, func(t *testing.T) {
t.Parallel()
cfg := twoHosts()
cfg.Hosts = cfg.Hosts[:1]
f := newFakeBackend()
d := &checkDialer{backends: map[string]*fakeBackend{testHost: f}}
if tc.dialErr != nil {
d.errs = map[string]error{testHost: tc.dialErr(t)}
}
if tc.prep != nil {
tc.prep(f)
}
p := newProvider(testProvider, cfg, d)
ctx, _ := checkContext()
switch {
case tc.timeout > 0:
var cancel context.CancelFunc
ctx, cancel = context.WithTimeout(ctx, tc.timeout)
defer cancel()
case tc.cancel:
var cancel context.CancelFunc
ctx, cancel = context.WithCancel(ctx)
time.AfterFunc(50*time.Millisecond, cancel)
}
got := p.Check(ctx)
if len(got) != 1 {
t.Fatalf("Check() returned %d targets, want 1", len(got))
}
ts := got[0]
if ts.Ready != tc.wantReady || ts.Reason != tc.wantReason {
t.Errorf("verdict = ready %v reason %s (%s), want ready %v reason %s", ts.Ready, ts.Reason, ts.Message, tc.wantReady, tc.wantReason)
}
if !tc.wantReady && ts.Message == "" {
t.Error("failed check has no message")
}
if tc.wantReady && ts.Message != "" {
t.Errorf("successful check has a message: %q", ts.Message)
}
if strings.Contains(ts.Message, keyMaterial) {
t.Errorf("message leaks the key file content: %q", ts.Message)
}
if tc.wantMessage != "" && !strings.Contains(ts.Message, tc.wantMessage) {
t.Errorf("message = %q, want it to contain %q", ts.Message, tc.wantMessage)
}
if ts.Details["user"] != "virt" || ts.Details["pool"] != "default" {
t.Errorf("details = %v, want user and pool", ts.Details)
}
if tc.wantReady {
want := map[string]string{
"user": "virt", "pool": "default", "hostname": "fake-host", "libvirtVersion": "11.6.0",
"poolState": "running", "poolCapacityGiB": "100.0", "poolAvailableGiB": "50.0",
}
for k, v := range want {
if ts.Details[k] != v {
t.Errorf("details[%s] = %q, want %q", k, ts.Details[k], v)
}
}
}
})
}
}
func TestCheck_neverUsesPooledConnection(t *testing.T) {
t.Parallel()
cfg := threeHosts()
fakes := fakesFor(cfg)
d := &checkDialer{backends: fakes}
p := newProvider(testProvider, cfg, d)
ctx, _ := checkContext()
got := p.Check(ctx)
for _, ts := range got {
if !ts.Ready {
t.Errorf("target %s not ready: %s: %s", ts.Name, ts.Reason, ts.Message)
}
}
if n := d.dials.Load(); int(n) != len(cfg.Hosts) {
t.Errorf("dials = %d, want one fresh dial per host (%d)", n, len(cfg.Hosts))
}
p.hosts.mu.Lock()
pooled := len(p.hosts.hosts)
p.hosts.mu.Unlock()
if pooled != 0 {
t.Errorf("pooled table has %d entries after Check, want none", pooled)
}
for name, f := range fakes {
if f.closeCount() != 1 {
t.Errorf("fresh backend for %s closed %d times, want 1", name, f.closeCount())
}
}
}
func TestCheck_detailsKeys(t *testing.T) {
t.Parallel()
full := []string{"hostname", "libvirtVersion", "pool", "poolAvailableGiB", "poolCapacityGiB", "poolState", "user"}
tests := []struct {
name string
dialErr error
prep func(f *fakeBackend)
want []string
}{
{name: "success", want: full},
{name: "pool not active", prep: func(f *fakeBackend) { f.pools[fakePoolName].state = libvirt.StoragePoolDegraded }, want: full},
{
name: "dial failure",
dialErr: &dialError{stage: stageTCP, host: testHost, err: errors.New("dial tcp: connection refused")},
want: []string{"pool", "user"},
},
}
for _, tc := range tests {
t.Run(tc.name, func(t *testing.T) {
t.Parallel()
cfg := twoHosts()
cfg.Hosts = cfg.Hosts[:1]
f := newFakeBackend()
if tc.prep != nil {
tc.prep(f)
}
d := &checkDialer{backends: map[string]*fakeBackend{testHost: f}}
if tc.dialErr != nil {
d.errs = map[string]error{testHost: tc.dialErr}
}
p := newProvider(testProvider, cfg, d)
ctx, _ := checkContext()
got := p.Check(ctx)[0]
if keys := detailKeys(got.Details); !slices.Equal(keys, tc.want) {
t.Errorf("details keys = %v, want %v", keys, tc.want)
}
for k, v := range got.Details {
if v == "" {
t.Errorf("details[%s] is empty", k)
}
}
})
}
}
func TestCheck_logsVerdictPerHost(t *testing.T) {
t.Parallel()
cfg := twoHosts()
d := &checkDialer{backends: fakesFor(cfg), errs: map[string]error{"lab2": &dialError{stage: stageTCP, host: "lab2", err: errors.New("dial tcp: i/o timeout")}}}
p := newProvider(testProvider, cfg, d)
ctx, logs := checkContext()
p.Check(ctx)
checked := logs.matching("kvm2 host checked")
if len(checked) != 2 {
t.Fatalf("got %d verdict lines, want 2: %v", len(checked), checked)
}
for _, l := range checked {
switch {
case strings.Contains(l, `"host"="beelink"`):
if !strings.Contains(l, `"ready"=true`) || !strings.Contains(l, `"reason"="Reachable"`) || !strings.Contains(l, `"latency"=`) {
t.Errorf("beelink verdict line = %s", l)
}
case strings.Contains(l, `"host"="lab2"`):
if !strings.Contains(l, `"ready"=false`) || !strings.Contains(l, `"reason"="Unreachable"`) {
t.Errorf("lab2 verdict line = %s", l)
}
default:
t.Errorf("verdict line without a host: %s", l)
}
}
}
func TestCheckReason_table(t *testing.T) {
t.Parallel()
dial := func(stage dialStage, cause error) error {
return fmt.Errorf("wrapped: %w", &dialError{stage: stage, host: testHost, err: cause})
}
tests := []struct {
name string
err error
want string
}{
{name: "credentials stage", err: dial(stageCredentials, errors.New("reading ssh key file /k: no such file")), want: reasonCredentialsUnavailable},
{name: "credentials stage even when permanent", err: dial(stageCredentials, permanentf("parsing ssh key file /k: ssh: no key found")), want: reasonCredentialsUnavailable},
{name: "tcp stage", err: dial(stageTCP, errors.New("connect: connection refused")), want: provider.ReasonUnreachable},
{name: "tcp stage with a deadline", err: dial(stageTCP, context.DeadlineExceeded), want: provider.ReasonUnreachable},
{name: "ssh stage unknown host key", err: dial(stageSSH, &knownhosts.KeyError{}), want: provider.ReasonUnauthorized},
{name: "ssh stage wrapped key error", err: dial(stageSSH, fmt.Errorf("ssh: handshake failed: %w", &knownhosts.KeyError{Want: []knownhosts.KnownKey{{}}})), want: provider.ReasonUnauthorized},
{name: "ssh stage unable to authenticate", err: dial(stageSSH, errors.New("ssh: handshake failed: ssh: unable to authenticate, attempted methods [none publickey], no supported methods remain")), want: provider.ReasonUnauthorized},
{name: "ssh stage reset", err: dial(stageSSH, errors.New("read tcp: connection reset by peer")), want: provider.ReasonUnreachable},
{name: "ssh stage cancelled", err: dial(stageSSH, context.Canceled), want: provider.ReasonUnreachable},
{name: "socket stage", err: dial(stageSocket, errors.New("opening /run/libvirt/libvirt-sock: ssh: rejected")), want: provider.ReasonUnreachable},
{name: "deadline exceeded", err: fmt.Errorf("%w: libvirt call ConnectGetHostname on beelink exceeded its 30s budget: %w", provider.ErrTransient, context.DeadlineExceeded), want: provider.ReasonTimeout},
{name: "cancelled", err: fmt.Errorf("%w: libvirt call ConnectGetHostname on beelink abandoned: %w", provider.ErrTransient, context.Canceled), want: provider.ReasonTimeout},
{name: "access denied", err: fakeErr(libvirt.ErrAccessDenied, "access denied"), want: provider.ReasonForbidden},
{name: "auth failed wrapped", err: fmt.Errorf("connecting to libvirt on host beelink: %w", fakeErr(libvirt.ErrAuthFailed, "authentication failed")), want: provider.ReasonForbidden},
{name: "no storage pool", err: fakeErr(libvirt.ErrNoStoragePool, "Storage pool not found"), want: reasonPoolNotFound},
{name: "other libvirt error", err: fakeErr(libvirt.ErrInternalError, "internal error"), want: provider.ReasonAPIError},
{name: "link killed", err: transientf("libvirt call X on beelink: connection closed while waiting: keepalive: EOF"), want: provider.ReasonUnreachable},
{name: "plain error", err: errors.New("unexpected EOF"), want: provider.ReasonUnreachable},
{name: "nil", err: nil, want: provider.ReasonUnreachable},
}
for _, tc := range tests {
t.Run(tc.name, func(t *testing.T) {
t.Parallel()
if got := checkReason(tc.err); got != tc.want {
t.Errorf("checkReason(%v) = %s, want %s", tc.err, got, tc.want)
}
})
}
}
func TestFormatLibVersion(t *testing.T) {
t.Parallel()
tests := []struct {
in uint64
want string
}{
{0, "0.0.0"},
{11006000, "11.6.0"},
{10000003, "10.0.3"},
{9010012, "9.10.12"},
{100999999, "100.999.999"},
}
for _, tc := range tests {
t.Run(tc.want, func(t *testing.T) {
t.Parallel()
if got := formatLibVersion(tc.in); got != tc.want {
t.Errorf("formatLibVersion(%d) = %s, want %s", tc.in, got, tc.want)
}
})
}
}
func TestPoolStateName(t *testing.T) {
t.Parallel()
tests := []struct {
in uint8
want string
}{
{uint8(libvirt.StoragePoolInactive), "inactive"},
{uint8(libvirt.StoragePoolBuilding), "building"},
{uint8(libvirt.StoragePoolRunning), "running"},
{uint8(libvirt.StoragePoolDegraded), "degraded"},
{uint8(libvirt.StoragePoolInaccessible), "inaccessible"},
{42, "state 42"},
}
for _, tc := range tests {
t.Run(tc.want, func(t *testing.T) {
t.Parallel()
if got := poolStateName(tc.in); got != tc.want {
t.Errorf("poolStateName(%d) = %s, want %s", tc.in, got, tc.want)
}
})
}
}
internal/provider/kvm2/create.go
+281
View File
@@ -0,0 +1,281 @@
package kvm2
import (
"bytes"
"fmt"
"time"
"context"
"github.com/digitalocean/go-libvirt"
"github.com/go-logr/logr"
"gitea.home.hrajfrisbee.cz/kacerr/egress-proxies-operator/internal/provider"
"gitea.home.hrajfrisbee.cz/kacerr/egress-proxies-operator/internal/provider/nocloud"
)
// The wait after DomainCreate (spec entry 18): Create returns only once the
// domain is running, so Get never sees a domain kvm2 defined but did not
// start (spec entry 9). DomainCreate is synchronous, so the first read
// normally succeeds; the cap only covers state settling and must not pin a
// reconcile worker for long.
const (
defaultRunningWait = 15 * time.Second
defaultRunningPoll = 500 * time.Millisecond
)
// Create is spec entry 10: validate placement, then, under the per-name
// lock, find or build the domain step by step, every step idempotent, and
// return once it is running.
func (p *Provider) Create(ctx context.Context, req provider.CreateRequest) (string, error) {
host, err := p.host(req.Placement.Zone)
if err != nil {
if req.Placement.Zone != "" {
err = permanentf("placement.zone %q is not in kvm2.hosts", req.Placement.Zone)
}
return "", p.wrapErr("create", "", err)
}
mt, ok := p.cfg.MachineTypes[req.Placement.MachineType]
if !ok {
return "", p.wrapErr("create", "", permanentf("placement.machineType %q is not in kvm2.machineTypes", req.Placement.MachineType))
}
src, err := resolveImage(p.name, p.cfg, req.Placement.Image)
if err != nil {
return "", p.wrapErr("create", "", err)
}
id := formatProviderID(host.Name, req.Name)
log := p.logger(ctx).WithValues("host", host.Name, "name", req.Name)
log.V(2).Info("kvm2 create resolved",
"pool", host.Pool, "bridge", host.Bridge, "machineType", req.Placement.MachineType,
"vcpus", mt.VCPUs, "memoryMiB", mt.MemoryMiB, "diskSizeGb", p.cfg.DiskSizeGB,
"image", src.volumeName(), "firmware", src.Firmware, "cloudInitBytes", len(req.CloudInit))
unlock, err := p.lockName(ctx, req.Name)
if err != nil {
return "", p.wrapErr("create", id, err)
}
defer unlock()
be, err := p.hosts.get(ctx, host)
if err != nil {
logAPIError(log, "create", err)
return "", p.wrapErr("create", id, err)
}
if err := p.create(ctx, log, be, host, req, mt, src); err != nil {
logAPIError(log, "create", err)
return "", p.wrapErr("create", id, err)
}
log.V(1).Info("kvm2 domain ensured running", "id", id)
return id, nil
}
func (p *Provider) create(ctx context.Context, log logr.Logger, be backend, host provider.KVMHost, req provider.CreateRequest, mt provider.KVMMachineType, src imageSource) error {
dom, err := be.DomainLookupByName(ctx, req.Name)
switch {
case err == nil:
meta, err := readMetadata(ctx, be, dom)
if err != nil {
return err
}
if !meta.managed() {
return permanentf("domain %s exists but carries no kvm2 mark; it was not created by this provider", req.Name)
}
if meta.UID != "" && meta.UID != req.UID {
return permanentf("domain %s belongs to proxy uid %s, not %s", req.Name, meta.UID, req.UID)
}
log.V(1).Info("kvm2 domain already defined", "uid", meta.UID)
return p.ensureRunning(ctx, log, be, dom)
case !hasCode(err, libvirt.ErrNoDomain):
return fmt.Errorf("looking up domain %s: %w", req.Name, err)
}
pool, err := be.StoragePoolLookupByName(ctx, host.Pool)
if err != nil {
return fmt.Errorf("looking up storage pool %s: %w", host.Pool, err)
}
imageBytes, err := p.imageSizeIfUncached(ctx, be, pool, src)
if err != nil {
return err
}
if err := checkPoolSpace(ctx, be, pool, host.Pool, neededBytes(imageBytes, p.cfg.DiskSizeGB)); err != nil {
return err
}
basePath, err := p.ensureImage(ctx, host, pool, be, src)
if err != nil {
return err
}
overlay, err := ensureOverlay(ctx, log, be, pool, req.Name, basePath, src.Format, p.cfg.DiskSizeGB)
if err != nil {
return err
}
seed, err := p.writeSeed(ctx, log, be, pool, req)
if err != nil {
return err
}
diskPath, err := be.StorageVolGetPath(ctx, overlay)
if err != nil {
return fmt.Errorf("reading the path of overlay volume %s: %w", overlay.Name, err)
}
seedPath, err := be.StorageVolGetPath(ctx, seed)
if err != nil {
return fmt.Errorf("reading the path of seed volume %s: %w", seed.Name, err)
}
xmlDoc, err := buildDomainXML(domainParams{
Name: req.Name,
UID: req.UID,
CreatedAt: p.now(),
VCPUs: mt.VCPUs,
MemoryMiB: mt.MemoryMiB,
Firmware: src.Firmware,
Bridge: host.Bridge,
DiskPath: diskPath,
SeedPath: seedPath,
DiskFormat: provider.KVMFormatQcow2,
})
if err != nil {
return err
}
dom, err = be.DomainDefineXML(ctx, xmlDoc)
if err != nil {
return fmt.Errorf("defining domain %s: %w", req.Name, err)
}
kv := []any{"uid", req.UID, "firmware", src.Firmware}
if log.V(2).Enabled() {
kv = append(kv, "xml", xmlDoc)
}
log.V(1).Info("domain defined", kv...)
return p.ensureRunning(ctx, log, be, dom)
}
// ensureOverlay creates the per-VM qcow2 overlay backed by the base image,
// or reuses the one a previous attempt left: volume creation is atomic, so
// an existing overlay is complete.
func ensureOverlay(ctx context.Context, log logr.Logger, be backend, pool libvirt.StoragePool, name, basePath, baseFormat string, diskSizeGB int64) (libvirt.StorageVol, error) {
volName := overlayVolumeName(name)
xmlDoc, err := buildVolumeXML(volumeSpec{
Name: volName,
CapacityBytes: diskSizeGB << 30,
Format: provider.KVMFormatQcow2,
BackingPath: basePath,
BackingFormat: baseFormat,
})
if err != nil {
return libvirt.StorageVol{}, err
}
vol, err := be.StorageVolCreateXML(ctx, pool, xmlDoc, 0)
switch {
case err == nil:
log.V(1).Info("overlay volume created", "volume", volName)
return vol, nil
case hasCode(err, libvirt.ErrStorageVolExist):
vol, err = be.StorageVolLookupByName(ctx, pool, volName)
if err != nil {
return libvirt.StorageVol{}, fmt.Errorf("looking up overlay volume %s: %w", volName, err)
}
log.V(1).Info("overlay volume already exists", "volume", volName)
return vol, nil
default:
return libvirt.StorageVol{}, fmt.Errorf("creating overlay volume %s: %w", volName, err)
}
}
// writeSeed builds the NoCloud seed and uploads it as a raw volume,
// replacing whatever a previous attempt left (spec entry 5): a cut-off
// upload would boot a VM without cloud-init. The ISO is tens of KiB, so it
// is built in memory. Nothing about its content is logged or put in an
// error.
func (p *Provider) writeSeed(ctx context.Context, log logr.Logger, be backend, pool libvirt.StoragePool, req provider.CreateRequest) (libvirt.StorageVol, error) {
name := seedVolumeName(req.Name)
if err := deleteVolumeIfPresent(ctx, be, pool, name); err != nil {
return libvirt.StorageVol{}, err
}
installAgent := p.cfg.InstallGuestAgent == nil || *p.cfg.InstallGuestAgent
vendorData, err := nocloud.BuildVendorData(p.cfg.VendorData, installAgent)
if err != nil {
return libvirt.StorageVol{}, permanentf("building vendor-data for %s: %v", req.Name, err)
}
var iso bytes.Buffer
if err := nocloud.BuildSeedISO(req.Name, req.CloudInit, vendorData, &iso); err != nil {
return libvirt.StorageVol{}, permanentf("building seed ISO for %s: %v", req.Name, err)
}
xmlDoc, err := buildVolumeXML(volumeSpec{Name: name, CapacityBytes: int64(iso.Len()), Format: provider.KVMFormatRaw})
if err != nil {
return libvirt.StorageVol{}, err
}
vol, err := be.StorageVolCreateXML(ctx, pool, xmlDoc, 0)
if err != nil {
return libvirt.StorageVol{}, fmt.Errorf("creating seed volume %s: %w", name, err)
}
if err := be.StorageVolUpload(ctx, vol, bytes.NewReader(iso.Bytes()), 0, uint64(iso.Len()), 0); err != nil {
return libvirt.StorageVol{}, fmt.Errorf("uploading seed volume %s: %w", name, err)
}
log.V(1).Info("seed volume uploaded", "volume", name, "bytes", iso.Len())
return vol, nil
}
// ensureRunning starts the domain unless it is active already, then waits
// until libvirt reports it running. ErrOperationInvalid from DomainCreate
// means another caller started it between the state read and the call. A
// paused or pmsuspended domain is an admin's doing and DomainCreate would
// refuse it; the verdict is Permanent so the reconciler does not retry it.
func (p *Provider) ensureRunning(ctx context.Context, log logr.Logger, be backend, dom libvirt.Domain) error {
state, _, err := be.DomainGetState(ctx, dom, 0)
if err != nil {
return fmt.Errorf("reading state of domain %s: %w", dom.Name, err)
}
switch s := libvirt.DomainState(state); {
case isActive(s):
return nil
case s == libvirt.DomainPaused || s == libvirt.DomainPmsuspended:
return permanentf("domain %s is %s and cannot be started by the operator; resume or destroy it on the host", dom.Name, s)
}
switch err := be.DomainCreate(ctx, dom); {
case err == nil:
log.V(1).Info("domain started")
case hasCode(err, libvirt.ErrOperationInvalid):
log.V(1).Info("domain start reported it already running")
default:
return fmt.Errorf("starting domain %s: %w", dom.Name, err)
}
return p.waitRunning(ctx, be, dom)
}
// waitRunning polls DomainGetState until the domain is active, the cap
// passes (Transient: the next Create attempt finds the domain defined and
// tries again) or ctx ends. Only nostate is worth another poll; a domain
// that is shut off, crashed or paused right after its start is not going
// to come up on its own, so that is Transient at once.
func (p *Provider) waitRunning(ctx context.Context, be backend, dom libvirt.Domain) error {
deadline := time.NewTimer(p.runningWait)
defer deadline.Stop()
ticker := time.NewTicker(p.runningPoll)
defer ticker.Stop()
for {
state, _, err := be.DomainGetState(ctx, dom, 0)
if err != nil {
return fmt.Errorf("reading state of domain %s: %w", dom.Name, err)
}
switch s := libvirt.DomainState(state); {
case isActive(s):
return nil
case s != libvirt.DomainNostate:
return transientf("domain %s went %s right after start", dom.Name, s)
}
select {
case <-ticker.C:
case <-deadline.C:
return transientf("domain %s did not reach running within %s (state %s)", dom.Name, p.runningWait, libvirt.DomainState(state))
case <-ctx.Done():
return fmt.Errorf("%w: waiting for domain %s to run: %w", provider.ErrTransient, dom.Name, ctx.Err())
}
}
}
// logAPIError is the V(1) line per failed backend call of spec entry 20.
func logAPIError(log logr.Logger, op string, err error) {
kv := []any{"op", op, "error", err.Error()}
kv = append(kv, libvirtErrorFields(err)...)
log.V(1).Info("kvm2 backend call failed", kv...)
}
internal/provider/kvm2/create_test.go
+468
View File
@@ -0,0 +1,468 @@
package kvm2
import (
"bytes"
"errors"
"math"
"strconv"
"strings"
"testing"
"time"
"github.com/digitalocean/go-libvirt"
"gitea.home.hrajfrisbee.cz/kacerr/egress-proxies-operator/internal/provider"
"gitea.home.hrajfrisbee.cz/kacerr/egress-proxies-operator/internal/provider/nocloud"
)
func TestCreate_repeatReturnsSameIDAndCreatesNothing(t *testing.T) {
t.Parallel()
fx := newFixture(t, 1)
first := fx.create(testName)
before := len(fx.fake.callLog())
defines, vols, uploads := fx.fake.callCount("DomainDefineXML"), fx.fake.callCount("StorageVolCreateXML"), fx.fake.callCount("StorageVolUpload")
second := fx.create(testName)
if second != first {
t.Errorf("second Create = %q, want %q", second, first)
}
if got := fx.fake.callCount("DomainDefineXML"); got != defines {
t.Errorf("DomainDefineXML calls after repeat = %d, want %d", got, defines)
}
if got := fx.fake.callCount("StorageVolCreateXML"); got != vols {
t.Errorf("StorageVolCreateXML calls after repeat = %d, want %d", got, vols)
}
if got := fx.fake.callCount("StorageVolUpload"); got != uploads {
t.Errorf("StorageVolUpload calls after repeat = %d, want %d", got, uploads)
}
if got := fx.fake.callCount("DomainCreate"); got != 1 {
t.Errorf("DomainCreate calls = %d, want 1", got)
}
repeat := fx.fake.callLog()[before:]
for _, rpc := range repeat {
switch rpc {
case "DomainLookupByName", "DomainGetMetadata", "DomainGetState":
default:
t.Errorf("repeat Create made %s; want only lookup, metadata and state reads: %v", rpc, repeat)
}
}
if len(fx.logs.matching("kvm2 domain already defined")) != 1 {
t.Errorf("no 'already defined' line:\n%s", strings.Join(fx.logs.all(), "\n"))
}
}
func TestCreate_resumesAfterEachPartialStep(t *testing.T) {
t.Parallel()
overlay, seed := overlayVolumeName(testName), seedVolumeName(testName)
tests := []struct {
name string
setup func(fx *fixture)
check func(t *testing.T, fx *fixture)
}{
{
name: "base without marker is refetched",
setup: func(fx *fixture) {
fx.putVol(fx.baseName(), []byte("truncated"))
},
check: func(t *testing.T, fx *fixture) {
if got := fx.fake.vol(fx.baseName()); got == nil || !bytes.Equal(got.content, fx.image) {
t.Errorf("base volume not refetched")
}
if fx.fake.vol(fx.baseName()+".sha256") == nil {
t.Errorf("marker not written")
}
},
},
{
name: "overlay exists is reused",
setup: func(fx *fixture) {
fx.cacheImage()
fx.putVol(overlay, []byte("overlay"))
},
check: func(t *testing.T, fx *fixture) {
if got := fx.fake.vol(overlay); got == nil || string(got.content) != "overlay" {
t.Errorf("overlay volume was replaced")
}
if len(fx.logs.matching("overlay volume already exists")) != 1 {
t.Errorf("no reuse line:\n%s", strings.Join(fx.logs.all(), "\n"))
}
},
},
{
name: "truncated seed is replaced",
setup: func(fx *fixture) {
fx.cacheImage()
fx.putVol(seed, []byte("cut"))
},
check: func(t *testing.T, fx *fixture) {
got := fx.fake.vol(seed)
if got == nil || len(got.content) <= 3 || !bytes.Contains(got.content, []byte(nocloud.SeedLabel)) {
t.Errorf("seed volume not rebuilt")
}
},
},
{
name: "domain defined but shutoff is started",
setup: func(fx *fixture) {
fx.cacheImage()
fx.putVol(overlay, []byte("overlay"))
fx.putVol(seed, []byte("seed"))
fx.fake.addDomain(testName, libvirt.DomainShutoff, managedMetadata(testUID, time.Now()))
},
check: func(t *testing.T, fx *fixture) {
if got := fx.fake.callCount("DomainDefineXML"); got != 0 {
t.Errorf("DomainDefineXML calls = %d, want 0", got)
}
if got := fx.fake.callCount("DomainCreate"); got != 1 {
t.Errorf("DomainCreate calls = %d, want 1", got)
}
},
},
{
name: "domain running is returned as is",
setup: func(fx *fixture) {
fx.fake.addDomain(testName, libvirt.DomainRunning, managedMetadata(testUID, time.Now()))
},
check: func(t *testing.T, fx *fixture) {
for _, rpc := range []string{"DomainDefineXML", "DomainCreate", "StorageVolCreateXML"} {
if got := fx.fake.callCount(rpc); got != 0 {
t.Errorf("%s calls = %d, want 0", rpc, got)
}
}
},
},
}
for _, tc := range tests {
t.Run(tc.name, func(t *testing.T) {
t.Parallel()
fx := newFixture(t, 1)
tc.setup(fx)
fx.create(testName)
fx.withDomain(testName, func(d *fakeDomain) {
if d.state != libvirt.DomainRunning {
t.Errorf("domain state = %s, want running", d.state)
}
})
tc.check(t, fx)
})
}
}
func TestCreate_returnsOnlyWhenRunning(t *testing.T) {
t.Parallel()
t.Run("returns after the domain reports running", func(t *testing.T) {
t.Parallel()
fx := newFixture(t, 1)
fx.fake.startDelayPolls = 3
fx.create(testName)
fx.withDomain(testName, func(d *fakeDomain) {
if d.state != libvirt.DomainRunning {
t.Errorf("Create returned with domain state %s", d.state)
}
})
if got := fx.fake.callCount("DomainGetState"); got < 4 {
t.Errorf("DomainGetState calls = %d, want at least 4 (one before start, three polls)", got)
}
})
t.Run("never running is transient after the cap", func(t *testing.T) {
t.Parallel()
fx := newFixture(t, 1, withRunningWait(time.Millisecond, 30*time.Millisecond))
fx.fake.startDelayPolls = math.MaxInt32
start := time.Now()
id, err := fx.p.Create(fx.ctx, fx.req(testName))
if id != "" {
t.Errorf("Create = %q, want no ID", id)
}
assertProviderError(t, err, "create", provider.ErrTransient)
if !strings.Contains(err.Error(), "did not reach running") {
t.Errorf("error = %v, want the wait cap named", err)
}
if elapsed := time.Since(start); elapsed > 2*time.Second {
t.Errorf("Create took %v, want the shrunk cap to end it", elapsed)
}
})
}
func TestCreate_domainThatDiesAfterStartIsTransientAtOnce(t *testing.T) {
t.Parallel()
for _, state := range []libvirt.DomainState{libvirt.DomainCrashed, libvirt.DomainShutoff, libvirt.DomainPaused} {
t.Run(state.String(), func(t *testing.T) {
t.Parallel()
// The cap is well over the bound asserted below, so a poll that
// ran to the cap would fail the test.
fx := newFixture(t, 1, withRunningWait(time.Millisecond, 10*time.Second))
fx.fake.stateAfterStart = state
start := time.Now()
id, err := fx.p.Create(fx.ctx, fx.req(testName))
elapsed := time.Since(start)
if id != "" {
t.Errorf("Create = %q, want no ID", id)
}
assertProviderError(t, err, "create", provider.ErrTransient)
if !strings.Contains(err.Error(), "right after start") {
t.Errorf("error = %v, want the state after start named", err)
}
if elapsed > time.Second {
t.Errorf("Create took %v, want it to give up at once, not poll to the cap", elapsed)
}
if got := fx.fake.callCount("DomainCreate"); got != 1 {
t.Errorf("DomainCreate calls = %d, want 1", got)
}
})
}
}
func TestCreate_pausedDomainIsPermanent(t *testing.T) {
t.Parallel()
for _, state := range []libvirt.DomainState{libvirt.DomainPaused, libvirt.DomainPmsuspended} {
t.Run(state.String(), func(t *testing.T) {
t.Parallel()
fx := newFixture(t, 1)
fx.fake.addDomain(testName, state, managedMetadata(testUID, time.Now()))
id, err := fx.p.Create(fx.ctx, fx.req(testName))
if id != "" {
t.Errorf("Create = %q, want no ID", id)
}
assertProviderError(t, err, "create", provider.ErrPermanent)
if !strings.Contains(err.Error(), "cannot be started by the operator") {
t.Errorf("error = %v, want the admin action named", err)
}
for _, rpc := range []string{"DomainCreate", "DomainDefineXML", "DomainDestroy"} {
if got := fx.fake.callCount(rpc); got != 0 {
t.Errorf("%s calls = %d, want 0", rpc, got)
}
}
fx.withDomain(testName, func(d *fakeDomain) {
if d.state != state {
t.Errorf("domain state = %s, want it left %s", d.state, state)
}
})
})
}
}
func TestCreate_rejectsUnknownPlacement(t *testing.T) {
t.Parallel()
tests := []struct {
name string
mutate func(req *provider.CreateRequest)
wantMsg string
}{
{
name: "zone",
mutate: func(req *provider.CreateRequest) { req.Placement.Zone = "nas9" },
wantMsg: `placement.zone "nas9" is not in kvm2.hosts`,
},
{
name: "machineType",
mutate: func(req *provider.CreateRequest) { req.Placement.MachineType = "xlarge" },
wantMsg: `placement.machineType "xlarge" is not in kvm2.machineTypes`,
},
{
name: "image",
mutate: func(req *provider.CreateRequest) { req.Placement.Image = "ubuntu-24" },
wantMsg: `placement.image "ubuntu-24" is neither a key in kvm2.images nor an http(s) URL`,
},
}
for _, tc := range tests {
t.Run(tc.name, func(t *testing.T) {
t.Parallel()
fx := newFixture(t, 1)
req := fx.req(testName)
tc.mutate(&req)
id, err := fx.p.Create(fx.ctx, req)
if id != "" {
t.Errorf("Create = %q, want no ID", id)
}
assertProviderError(t, err, "create", provider.ErrPermanent)
if !strings.Contains(err.Error(), tc.wantMsg) {
t.Errorf("error = %q, want substring %q", err, tc.wantMsg)
}
if got := fx.fake.callLog(); len(got) != 0 {
t.Errorf("backend calls = %v, want none before validation", got)
}
if got := fx.dialer.dials.Load(); got != 0 {
t.Errorf("dials = %d, want 0", got)
}
})
}
}
func TestCreate_adoptionWithForeignUIDIsPermanent(t *testing.T) {
t.Parallel()
fx := newFixture(t, 1)
fx.fake.addDomain(testName, libvirt.DomainRunning, managedMetadata("other-uid", time.Now()))
id, err := fx.p.Create(fx.ctx, fx.req(testName))
if id != "" {
t.Errorf("Create = %q, want no ID", id)
}
assertProviderError(t, err, "create", provider.ErrPermanent)
if !strings.Contains(err.Error(), "other-uid") || !strings.Contains(err.Error(), testUID) {
t.Errorf("error = %q, want both UIDs named", err)
}
for _, rpc := range []string{"DomainCreate", "DomainDefineXML", "StorageVolCreateXML", "DomainDestroy"} {
if got := fx.fake.callCount(rpc); got != 0 {
t.Errorf("%s calls = %d, want 0", rpc, got)
}
}
}
func TestCreate_adoptsOnlyMarkedDomains(t *testing.T) {
t.Parallel()
tests := []struct {
name string
metadata string
wantAdopt bool
}{
{name: "matching uid", metadata: managedMetadata(testUID, time.Now()), wantAdopt: true},
{name: "managed, empty uid", metadata: managedMetadata("", time.Now()), wantAdopt: true},
{name: "no metadata element", metadata: ""},
{name: "element without the managed mark", metadata: `<instance><uid>` + testUID + `</uid></instance>`},
{name: "managed false", metadata: `<instance><managed>false</managed><uid>` + testUID + `</uid></instance>`},
}
for _, tc := range tests {
t.Run(tc.name, func(t *testing.T) {
t.Parallel()
fx := newFixture(t, 1)
fx.fake.addDomain(testName, libvirt.DomainRunning, tc.metadata)
id, err := fx.p.Create(fx.ctx, fx.req(testName))
if tc.wantAdopt {
if err != nil || id != formatProviderID(testHost, testName) {
t.Fatalf("Create = %q, %v; want the existing domain adopted", id, err)
}
} else {
if id != "" {
t.Errorf("Create = %q, want no ID", id)
}
assertProviderError(t, err, "create", provider.ErrPermanent)
if !strings.Contains(err.Error(), "carries no kvm2 mark") {
t.Errorf("error = %v, want the missing mark named", err)
}
}
for _, rpc := range []string{"DomainDefineXML", "DomainCreate", "StorageVolCreateXML", "DomainDestroy"} {
if got := fx.fake.callCount(rpc); got != 0 {
t.Errorf("%s calls = %d, want 0", rpc, got)
}
}
})
}
}
func TestCreate_poolOutOfSpaceIsQuotaExceeded(t *testing.T) {
t.Parallel()
fx := newFixture(t, 1)
fx.fake.mu.Lock()
fx.fake.pools[fakePoolName].available = 1 << 30
fx.fake.mu.Unlock()
_, err := fx.p.Create(fx.ctx, fx.req(testName))
assertProviderError(t, err, "create", provider.ErrQuotaExceeded)
if !strings.Contains(err.Error(), fakePoolName) || !strings.Contains(err.Error(), "needed") {
t.Errorf("error = %q, want the pool and the needed bytes named", err)
}
for _, rpc := range []string{"StorageVolCreateXML", "StorageVolUpload", "DomainDefineXML"} {
if got := fx.fake.callCount(rpc); got != 0 {
t.Errorf("%s calls = %d, want 0", rpc, got)
}
}
}
func TestCreate_libvirtNotFoundIsTransientNotNotFound(t *testing.T) {
t.Parallel()
tests := []struct {
name string
rpc string
err error
}{
{name: "volume vanished", rpc: "StorageVolGetPath", err: fakeErr(libvirt.ErrNoStorageVol, "Storage volume not found")},
{name: "domain vanished", rpc: "DomainGetState", err: fakeErr(libvirt.ErrNoDomain, "Domain not found")},
}
for _, tc := range tests {
t.Run(tc.name, func(t *testing.T) {
t.Parallel()
fx := newFixture(t, 1)
fx.fake.failWith(tc.rpc, tc.err)
_, err := fx.p.Create(fx.ctx, fx.req(testName))
assertProviderError(t, err, "create", provider.ErrTransient)
if errors.Is(err, provider.ErrNotFound) {
t.Errorf("Create error = %v, must never be NotFound", err)
}
})
}
}
func TestCreate_cloudInitSentinelNeverLoggedOrInErrors(t *testing.T) {
t.Parallel()
const sentinel = "SENTINEL-s3cr3t-passw0rd-7f3a"
fx := newFixture(t, 2)
fx.dialer.backend = observe(fx.fake, testProvider, testHost)
fx.fake.failWith("StorageVolUpload", fakeErr(libvirt.ErrInternalError, "internal error: upload failed"))
req := fx.req(testName)
req.CloudInit = "#cloud-config\nwrite_files:\n - content: " + sentinel + "\n"
_, err := fx.p.Create(fx.ctx, req)
assertProviderError(t, err, "create", provider.ErrTransient)
if strings.Contains(err.Error(), sentinel) {
t.Errorf("error carries the cloud-init: %v", err)
}
lines := fx.logs.all()
if len(lines) == 0 {
t.Fatal("no log lines captured at V(2)")
}
for _, l := range lines {
if strings.Contains(l, sentinel) {
t.Errorf("log line carries the cloud-init: %s", l)
}
}
if got := fx.logs.matching(`"cloudInitBytes"=` + strconv.Itoa(len(req.CloudInit))); len(got) != 1 {
t.Errorf("cloudInitBytes line = %d, want 1:\n%s", len(got), strings.Join(lines, "\n"))
}
}
func TestCreate_metadataWrittenInDefineCall(t *testing.T) {
t.Parallel()
fx := newFixture(t, 1)
fx.p.now = func() time.Time { return time.Date(2026, 9, 16, 12, 0, 0, 0, time.UTC) }
fx.create(testName)
fx.withDomain(testName, func(d *fakeDomain) {
for _, want := range []string{`xmlns="` + metadataURI + `"`, "<uid>" + testUID + "</uid>", "<managed>true</managed>", "<createdAt>2026-09-16T12:00:00Z</createdAt>"} {
if !strings.Contains(d.xml, want) {
t.Errorf("defined XML lacks %s:\n%s", want, d.xml)
}
}
meta, err := parseInstanceMetadata(d.metadata)
if err != nil {
t.Fatalf("metadata the define stored does not parse: %v", err)
}
if !meta.managed() || meta.UID != testUID || meta.createdAt().IsZero() {
t.Errorf("metadata = %+v, want managed, uid %s and a createdAt", meta, testUID)
}
})
}
func TestCreate_seedAlwaysRebuilt(t *testing.T) {
t.Parallel()
fx := newFixture(t, 1)
fx.create(testName)
seed := seedVolumeName(testName)
first := fx.fake.vol(seed)
if first == nil {
t.Fatal("no seed volume after Create")
}
uploads := fx.fake.callCount("StorageVolUpload")
// A crash between the seed upload and the define leaves the volumes
// without a domain; the next attempt must not trust the seed.
fx.fake.mu.Lock()
delete(fx.fake.domains, testName)
fx.fake.mu.Unlock()
fx.create(testName)
second := fx.fake.vol(seed)
if second == nil || second == first {
t.Errorf("seed volume reused across attempts")
}
if got := fx.fake.callCount("StorageVolUpload"); got != uploads+1 {
t.Errorf("StorageVolUpload calls = %d, want %d", got, uploads+1)
}
if len(second.content) != len(first.content) || !bytes.Contains(second.content, []byte(nocloud.SeedLabel)) {
t.Errorf("rebuilt seed is %d bytes, first was %d; want a complete seed of the same size", len(second.content), len(first.content))
}
}
internal/provider/kvm2/delete.go
+98
View File
@@ -0,0 +1,98 @@
package kvm2
import (
"context"
"fmt"
"github.com/digitalocean/go-libvirt"
"github.com/go-logr/logr"
"gitea.home.hrajfrisbee.cz/kacerr/egress-proxies-operator/internal/provider"
)
// undefineFlags removes everything libvirt keeps for a domain beside its
// definition; UEFI domains keep NVRAM and undefine refuses without the flag.
const undefineFlags = libvirt.DomainUndefineManagedSave | libvirt.DomainUndefineSnapshotsMetadata | libvirt.DomainUndefineNvram
// Delete is spec entry 11: under the per-name lock, destroy and undefine
// the domain, then remove its overlay and seed. Everything already gone is
// success, and nothing is left when nil is returned, so Get afterwards is
// NotFound. The base image stays: it is a shared cache.
func (p *Provider) Delete(ctx context.Context, providerID string) error {
hostName, name, err := parseProviderID(providerID)
if err != nil {
return provider.Wrap(provider.ErrPermanent, "delete", p.name, providerID, err)
}
log := p.logger(ctx).WithValues("host", hostName, "name", name)
host, err := p.host(hostName)
if err != nil {
log.V(1).Info("host is not configured; delete has nothing to clean up", "id", providerID)
return nil
}
unlock, err := p.lockName(ctx, name)
if err != nil {
return p.wrapErr("delete", providerID, err)
}
defer unlock()
be, err := p.hosts.get(ctx, host)
if err != nil {
logAPIError(log, "delete", err)
return p.wrapErr("delete", providerID, err)
}
if err := removeDomain(ctx, log, be, name); err != nil {
logAPIError(log, "delete", err)
return p.wrapErr("delete", providerID, err)
}
if err := removeVolumes(ctx, log, be, host.Pool, name); err != nil {
logAPIError(log, "delete", err)
return p.wrapErr("delete", providerID, err)
}
log.V(1).Info("kvm2 domain deleted")
return nil
}
func removeDomain(ctx context.Context, log logr.Logger, be backend, name string) error {
dom, err := be.DomainLookupByName(ctx, name)
if hasCode(err, libvirt.ErrNoDomain) {
log.V(1).Info("domain already gone")
return nil
}
if err != nil {
return fmt.Errorf("looking up domain %s: %w", name, err)
}
switch err := be.DomainDestroy(ctx, dom); {
case err == nil:
log.V(1).Info("domain destroyed")
case hasCode(err, libvirt.ErrOperationInvalid):
log.V(1).Info("domain not running, destroy skipped")
case hasCode(err, libvirt.ErrNoDomain):
log.V(1).Info("domain already gone")
return nil
default:
return fmt.Errorf("destroying domain %s: %w", name, err)
}
if err := be.DomainUndefineFlags(ctx, dom, undefineFlags); err != nil && !hasCode(err, libvirt.ErrNoDomain) {
return fmt.Errorf("undefining domain %s: %w", name, err)
}
log.V(1).Info("domain undefined")
return nil
}
func removeVolumes(ctx context.Context, log logr.Logger, be backend, poolName, name string) error {
pool, err := be.StoragePoolLookupByName(ctx, poolName)
if hasCode(err, libvirt.ErrNoStoragePool) {
log.V(1).Info("pool not found, no volumes to delete", "pool", poolName)
return nil
}
if err != nil {
return fmt.Errorf("looking up storage pool %s: %w", poolName, err)
}
for _, vol := range []string{overlayVolumeName(name), seedVolumeName(name)} {
if err := deleteVolumeIfPresent(ctx, be, pool, vol); err != nil {
return err
}
}
log.V(1).Info("per-VM volumes deleted")
return nil
}
internal/provider/kvm2/delete_test.go
+237
View File
@@ -0,0 +1,237 @@
package kvm2
import (
"errors"
"slices"
"strings"
"sync"
"testing"
"time"
"github.com/digitalocean/go-libvirt"
"gitea.home.hrajfrisbee.cz/kacerr/egress-proxies-operator/internal/provider"
)
func TestDelete_malformedIDIsPermanent(t *testing.T) {
t.Parallel()
for _, id := range []string{"", "proxy-x", "hosts/beelink/proxy-x", "hosts//domains/proxy-x", "hosts/beelink/domains/"} {
t.Run(id, func(t *testing.T) {
t.Parallel()
fx := newFixture(t, 1)
err := fx.p.Delete(fx.ctx, id)
assertProviderError(t, err, "delete", provider.ErrPermanent)
var pe *provider.Error
if errors.As(err, &pe) && pe.ID != id {
t.Errorf("error.ID = %q, want %q", pe.ID, id)
}
if got := fx.dialer.dials.Load(); got != 0 {
t.Errorf("dials = %d, want 0", got)
}
})
}
}
func TestDelete_twiceSucceeds(t *testing.T) {
t.Parallel()
fx := newFixture(t, 1)
id := fx.create(testName)
for i := range 2 {
if err := fx.p.Delete(fx.ctx, id); err != nil {
t.Fatalf("Delete #%d error = %v", i+1, err)
}
}
if fx.hasDomain(testName) {
t.Error("domain still defined after Delete")
}
if got := fx.logs.matching("domain already gone"); len(got) != 1 {
t.Errorf("'already gone' lines = %d, want 1 from the second Delete", len(got))
}
if got := fx.logs.matching("kvm2 domain deleted"); len(got) != 2 {
t.Errorf("'deleted' lines = %d, want 2", len(got))
}
}
func TestDelete_removesOverlayAndSeed(t *testing.T) {
t.Parallel()
fx := newFixture(t, 1)
id := fx.create(testName)
if err := fx.p.Delete(fx.ctx, id); err != nil {
t.Fatalf("Delete() error = %v", err)
}
for _, name := range []string{overlayVolumeName(testName), seedVolumeName(testName)} {
if fx.fake.vol(name) != nil {
t.Errorf("volume %s still present after Delete", name)
}
}
base := "base-" + testProvider + "-debian-13.qcow2"
for _, name := range []string{base, base + ".sha256"} {
if fx.fake.vol(name) == nil {
t.Errorf("shared base volume %s removed by Delete", name)
}
}
}
func TestDelete_runningDomainDestroyedFirst(t *testing.T) {
t.Parallel()
fx := newFixture(t, 1)
id := fx.create(testName)
before := len(fx.fake.callLog())
if err := fx.p.Delete(fx.ctx, id); err != nil {
t.Fatalf("Delete() error = %v", err)
}
calls := fx.fake.callLog()[before:]
destroy := slices.Index(calls, "DomainDestroy")
undefine := slices.Index(calls, "DomainUndefineFlags")
volDelete := slices.Index(calls, "StorageVolDelete")
if destroy < 0 || undefine < 0 || volDelete < 0 || !(destroy < undefine && undefine < volDelete) {
t.Errorf("call order = %v, want destroy before undefine before volume deletes", calls)
}
if n := strings.Count(strings.Join(calls, " "), "StorageVolDelete"); n != 2 {
t.Errorf("StorageVolDelete calls = %d, want 2 (overlay and seed)", n)
}
}
func TestDelete_stoppedDomainSkipsDestroy(t *testing.T) {
t.Parallel()
fx := newFixture(t, 1)
id := fx.create(testName)
fx.withDomain(testName, func(d *fakeDomain) { d.state = libvirt.DomainShutoff })
if err := fx.p.Delete(fx.ctx, id); err != nil {
t.Fatalf("Delete() error = %v", err)
}
if fx.hasDomain(testName) {
t.Error("domain still defined after Delete")
}
if got := fx.logs.matching("domain not running, destroy skipped"); len(got) != 1 {
t.Errorf("'destroy skipped' lines = %d, want 1:\n%s", len(got), strings.Join(fx.logs.all(), "\n"))
}
if got := fx.fake.callCount("DomainUndefineFlags"); got != 1 {
t.Errorf("DomainUndefineFlags calls = %d, want 1", got)
}
}
func TestDelete_toleratesMissingVolumesAndPool(t *testing.T) {
t.Parallel()
t.Run("domain without volumes", func(t *testing.T) {
t.Parallel()
fx := newFixture(t, 1)
fx.fake.addDomain(testName, libvirt.DomainRunning, managedMetadata(testUID, time.Now()))
if err := fx.p.Delete(fx.ctx, formatProviderID(testHost, testName)); err != nil {
t.Fatalf("Delete() error = %v", err)
}
if fx.hasDomain(testName) {
t.Error("domain still defined after Delete")
}
if got := fx.fake.callCount("StorageVolDelete"); got != 0 {
t.Errorf("StorageVolDelete calls = %d, want 0 for absent volumes", got)
}
})
t.Run("pool gone", func(t *testing.T) {
t.Parallel()
fx := newFixture(t, 1)
fx.p.cfg.Hosts[0].Pool = "retired"
fx.fake.addDomain(testName, libvirt.DomainShutoff, managedMetadata(testUID, time.Now()))
if err := fx.p.Delete(fx.ctx, formatProviderID(testHost, testName)); err != nil {
t.Fatalf("Delete() error = %v", err)
}
if fx.hasDomain(testName) {
t.Error("domain still defined after Delete")
}
if got := fx.logs.matching("pool not found, no volumes to delete"); len(got) != 1 || !strings.Contains(got[0], `"pool"="retired"`) {
t.Errorf("'pool not found' V(1) line = %v", got)
}
})
t.Run("nothing at all", func(t *testing.T) {
t.Parallel()
fx := newFixture(t, 1)
if err := fx.p.Delete(fx.ctx, formatProviderID(testHost, testName)); err != nil {
t.Fatalf("Delete() error = %v", err)
}
})
}
func TestDelete_unconfiguredHostIsNoOp(t *testing.T) {
t.Parallel()
fx := newFixture(t, 1)
id := formatProviderID("nas9", testName)
if err := fx.p.Delete(fx.ctx, id); err != nil {
t.Fatalf("Delete() error = %v, want nil", err)
}
if got := fx.dialer.dials.Load(); got != 0 {
t.Errorf("dials = %d, want 0", got)
}
if got := fx.fake.callLog(); len(got) != 0 {
t.Errorf("backend calls = %v, want none", got)
}
if got := fx.logs.matching("host is not configured; delete has nothing to clean up"); len(got) != 1 || !strings.Contains(got[0], `"host"="nas9"`) {
t.Errorf("V(1) line = %v, want one naming the host", got)
}
}
func TestDelete_noOpLinesAreSilentAtV0(t *testing.T) {
t.Parallel()
fx := newFixture(t, 0)
fx.p.cfg.Hosts[0].Pool = "retired"
for _, id := range []string{formatProviderID("nas9", testName), formatProviderID(testHost, testName)} {
if err := fx.p.Delete(fx.ctx, id); err != nil {
t.Fatalf("Delete(%s) error = %v, want nil", id, err)
}
}
if lines := fx.logs.all(); len(lines) != 0 {
t.Errorf("Delete logged %d lines at V(0), want none:\n%s", len(lines), strings.Join(lines, "\n"))
}
}
func TestDelete_concurrentSameNameSafe(t *testing.T) {
t.Parallel()
fx := newFixture(t, 1)
id := fx.create(testName)
const n = 10
errs := make([]error, n)
var wg sync.WaitGroup
for i := range n {
wg.Add(1)
go func() {
defer wg.Done()
errs[i] = fx.p.Delete(fx.ctx, id)
}()
}
wg.Wait()
for i, err := range errs {
if err != nil {
t.Errorf("Delete #%d error = %v, want nil", i, err)
}
}
if fx.hasDomain(testName) {
t.Error("domain still defined after concurrent Deletes")
}
for _, name := range []string{overlayVolumeName(testName), seedVolumeName(testName)} {
if fx.fake.vol(name) != nil {
t.Errorf("volume %s still present", name)
}
}
if got := fx.fake.callCount("DomainUndefineFlags"); got != 1 {
t.Errorf("DomainUndefineFlags calls = %d, want 1 (the lock serialises the rest)", got)
}
}
func TestDelete_undefineUsesNvramFlag(t *testing.T) {
t.Parallel()
fx := newFixture(t, 1)
id := fx.create(testName)
if err := fx.p.Delete(fx.ctx, id); err != nil {
t.Fatalf("Delete() error = %v", err)
}
fx.fake.mu.Lock()
flags, ok := fx.fake.undefineFlags[testName]
fx.fake.mu.Unlock()
if !ok {
t.Fatal("DomainUndefineFlags was not called")
}
for _, want := range []libvirt.DomainUndefineFlagsValues{libvirt.DomainUndefineNvram, libvirt.DomainUndefineManagedSave, libvirt.DomainUndefineSnapshotsMetadata} {
if flags&want == 0 {
t.Errorf("undefine flags = %d, lack %d", flags, want)
}
}
}
internal/provider/kvm2/errors.go
+97
View File
@@ -0,0 +1,97 @@
package kvm2
import (
"errors"
"fmt"
"strings"
"github.com/digitalocean/go-libvirt"
"golang.org/x/crypto/ssh/knownhosts"
"gitea.home.hrajfrisbee.cz/kacerr/egress-proxies-operator/internal/provider"
)
// permanentf and transientf classify an error at the source, so classify
// keeps that class no matter what wraps it later.
func permanentf(format string, args ...any) error {
return fmt.Errorf("%w: %s", provider.ErrPermanent, fmt.Sprintf(format, args...))
}
func transientf(format string, args ...any) error {
return fmt.Errorf("%w: %s", provider.ErrTransient, fmt.Sprintf(format, args...))
}
// sshAuthFailed is the substring x/crypto/ssh puts in every authentication
// failure ("ssh: handshake failed: ssh: unable to authenticate, attempted
// methods [...]").
const sshAuthFailed = "unable to authenticate"
// classify maps any error reaching the provider's boundary to one of the
// four contract sentinels, per spec entry 13. op is the contract operation:
// a vanished domain or volume inside Create is Transient, because the
// contract forbids NotFound from Create.
func classify(op string, err error) error {
if err == nil {
return provider.ErrTransient
}
for _, class := range []error{provider.ErrNotFound, provider.ErrQuotaExceeded, provider.ErrPermanent, provider.ErrTransient} {
if errors.Is(err, class) {
return class
}
}
if code, ok := libvirtCode(err); ok {
return classifyLibvirt(op, code)
}
var keyErr *knownhosts.KeyError
if errors.As(err, &keyErr) {
return provider.ErrPermanent
}
if strings.Contains(err.Error(), sshAuthFailed) {
return provider.ErrPermanent
}
// Everything else retries: a missing key or known_hosts file (the
// optional Secret has not landed yet), context cancellation and
// deadlines, net.OpError, io.EOF, and anything unrecognised.
return provider.ErrTransient
}
func classifyLibvirt(op string, code libvirt.ErrorNumber) error {
switch code {
case libvirt.ErrNoDomain, libvirt.ErrNoStorageVol:
if op == "create" {
return provider.ErrTransient
}
return provider.ErrNotFound
case libvirt.ErrNoStoragePool, libvirt.ErrXMLError, libvirt.ErrNoSupport, libvirt.ErrInvalidArg,
libvirt.ErrAuthFailed, libvirt.ErrAccessDenied:
return provider.ErrPermanent
default:
return provider.ErrTransient
}
}
// libvirtCode extracts the libvirt error number from err, if it is or wraps
// a libvirt.Error. That type has a value receiver, so errors.As needs a
// value target.
func libvirtCode(err error) (libvirt.ErrorNumber, bool) {
var le libvirt.Error
if !errors.As(err, &le) {
return 0, false
}
return libvirt.ErrorNumber(le.Code), true
}
func hasCode(err error, code libvirt.ErrorNumber) bool {
got, ok := libvirtCode(err)
return ok && got == code
}
// libvirtErrorFields returns the key/value pairs a log line carries for a
// libvirt error, or nil when err is not one.
func libvirtErrorFields(err error) []any {
code, ok := libvirtCode(err)
if !ok {
return nil
}
return []any{"libvirtCode", int(code), "libvirtError", code.String()}
}
internal/provider/kvm2/errors_test.go
+146
View File
@@ -0,0 +1,146 @@
package kvm2
import (
"context"
"errors"
"fmt"
"io"
"io/fs"
"net"
"os"
"testing"
"github.com/digitalocean/go-libvirt"
"golang.org/x/crypto/ssh"
"golang.org/x/crypto/ssh/knownhosts"
"gitea.home.hrajfrisbee.cz/kacerr/egress-proxies-operator/internal/provider"
)
func libvirtErr(code libvirt.ErrorNumber, msg string) error {
return libvirt.Error{Code: uint32(code), Message: msg}
}
// Captured from x/crypto/ssh when the offered key is not in authorized_keys.
const sshAuthFailedMsg = "ssh: handshake failed: ssh: unable to authenticate, attempted methods [none publickey], no supported methods remain"
func TestClassify_table(t *testing.T) {
t.Parallel()
tests := []struct {
name string
op string
err error
want error
}{
{name: "nil is transient", op: "get", err: nil, want: provider.ErrTransient},
{name: "ErrNoDomain on get", op: "get", err: libvirtErr(libvirt.ErrNoDomain, "Domain not found: no domain with matching name 'proxy-x'"), want: provider.ErrNotFound},
{name: "ErrNoDomain on delete", op: "delete", err: libvirtErr(libvirt.ErrNoDomain, "Domain not found"), want: provider.ErrNotFound},
{name: "ErrNoDomain on create is transient", op: "create", err: libvirtErr(libvirt.ErrNoDomain, "Domain not found"), want: provider.ErrTransient},
{name: "ErrNoStorageVol on get", op: "get", err: libvirtErr(libvirt.ErrNoStorageVol, "Storage volume not found: no storage vol with matching name 'proxy-x.qcow2'"), want: provider.ErrNotFound},
{name: "ErrNoStorageVol on create is transient", op: "create", err: libvirtErr(libvirt.ErrNoStorageVol, "Storage volume not found"), want: provider.ErrTransient},
{name: "ErrNoStoragePool", op: "create", err: libvirtErr(libvirt.ErrNoStoragePool, "Storage pool not found: no storage pool with matching name 'default'"), want: provider.ErrPermanent},
{name: "ErrXMLError", op: "create", err: libvirtErr(libvirt.ErrXMLError, "XML error: missing domain name"), want: provider.ErrPermanent},
{name: "ErrNoSupport", op: "create", err: libvirtErr(libvirt.ErrNoSupport, "this function is not supported by the connection driver"), want: provider.ErrPermanent},
{name: "ErrInvalidArg", op: "get", err: libvirtErr(libvirt.ErrInvalidArg, "invalid argument: unsupported flags"), want: provider.ErrPermanent},
{name: "ErrAuthFailed", op: "list", err: libvirtErr(libvirt.ErrAuthFailed, "authentication failed: access denied by policy"), want: provider.ErrPermanent},
{name: "ErrAccessDenied", op: "list", err: libvirtErr(libvirt.ErrAccessDenied, "access denied"), want: provider.ErrPermanent},
{name: "ErrOperationInvalid is transient", op: "create", err: libvirtErr(libvirt.ErrOperationInvalid, "Requested operation is not valid: domain is already running"), want: provider.ErrTransient},
{name: "ErrInternalError is transient", op: "get", err: libvirtErr(libvirt.ErrInternalError, "internal error: connection closed"), want: provider.ErrTransient},
{name: "unknown libvirt code is transient", op: "get", err: libvirtErr(libvirt.ErrorNumber(9999), "?"), want: provider.ErrTransient},
{name: "wrapped libvirt error keeps its code", op: "get", err: fmt.Errorf("looking up domain: %w", libvirtErr(libvirt.ErrNoDomain, "Domain not found")), want: provider.ErrNotFound},
{name: "knownhosts unknown host", op: "get", err: &knownhosts.KeyError{}, want: provider.ErrPermanent},
{name: "knownhosts mismatch", op: "get", err: &knownhosts.KeyError{Want: []knownhosts.KnownKey{{Filename: "/kh", Line: 1, Key: testPublicKey(t)}}}, want: provider.ErrPermanent},
{name: "wrapped knownhosts error", op: "create", err: fmt.Errorf("dialing beelink: %w", &knownhosts.KeyError{}), want: provider.ErrPermanent},
{name: "ssh unable to authenticate", op: "create", err: errors.New(sshAuthFailedMsg), want: provider.ErrPermanent},
{name: "wrapped ssh unable to authenticate", op: "list", err: fmt.Errorf("dialing beelink: %w", errors.New(sshAuthFailedMsg)), want: provider.ErrPermanent},
{name: "missing key file is transient", op: "create", err: fmt.Errorf("reading ssh key file /k: %w", fs.ErrNotExist), want: provider.ErrTransient},
{name: "os.PathError not exist is transient", op: "create", err: &os.PathError{Op: "open", Path: "/k", Err: fs.ErrNotExist}, want: provider.ErrTransient},
{name: "context canceled", op: "get", err: context.Canceled, want: provider.ErrTransient},
{name: "context deadline", op: "get", err: context.DeadlineExceeded, want: provider.ErrTransient},
{name: "net.OpError", op: "get", err: &net.OpError{Op: "dial", Net: "tcp", Err: errors.New("connection refused")}, want: provider.ErrTransient},
{name: "io.EOF", op: "list", err: io.EOF, want: provider.ErrTransient},
{name: "io.ErrUnexpectedEOF", op: "list", err: io.ErrUnexpectedEOF, want: provider.ErrTransient},
{name: "anything else is transient", op: "get", err: errors.New("anything"), want: provider.ErrTransient},
{name: "pre-classified NotFound stays", op: "create", err: fmt.Errorf("x: %w", provider.ErrNotFound), want: provider.ErrNotFound},
{name: "pre-classified Quota stays", op: "create", err: fmt.Errorf("x: %w", provider.ErrQuotaExceeded), want: provider.ErrQuotaExceeded},
{name: "pre-classified Permanent stays", op: "get", err: fmt.Errorf("x: %w", permanentf("bad %s", "thing")), want: provider.ErrPermanent},
{name: "pre-classified Transient stays", op: "get", err: fmt.Errorf("x: %w", transientf("blip %d", 1)), want: provider.ErrTransient},
{name: "pre-classified beats a libvirt code underneath", op: "get", err: fmt.Errorf("%w: %w", provider.ErrPermanent, libvirtErr(libvirt.ErrNoDomain, "x")), want: provider.ErrPermanent},
}
for _, tc := range tests {
t.Run(tc.name, func(t *testing.T) {
t.Parallel()
if got := classify(tc.op, tc.err); !errors.Is(got, tc.want) {
t.Errorf("classify(%q, %v) = %v, want %v", tc.op, tc.err, got, tc.want)
}
})
}
}
func TestHasCode(t *testing.T) {
t.Parallel()
tests := []struct {
name string
err error
code libvirt.ErrorNumber
want bool
}{
{name: "matching code", err: libvirtErr(libvirt.ErrNoDomain, "x"), code: libvirt.ErrNoDomain, want: true},
{name: "matching code wrapped", err: fmt.Errorf("w: %w", libvirtErr(libvirt.ErrStorageVolExist, "x")), code: libvirt.ErrStorageVolExist, want: true},
{name: "other code", err: libvirtErr(libvirt.ErrNoDomain, "x"), code: libvirt.ErrNoStorageVol, want: false},
{name: "not a libvirt error", err: errors.New("x"), code: libvirt.ErrNoDomain, want: false},
{name: "nil", err: nil, code: libvirt.ErrNoDomain, want: false},
}
for _, tc := range tests {
t.Run(tc.name, func(t *testing.T) {
t.Parallel()
if got := hasCode(tc.err, tc.code); got != tc.want {
t.Errorf("hasCode(%v, %v) = %v, want %v", tc.err, tc.code, got, tc.want)
}
})
}
}
func TestLibvirtErrorFields(t *testing.T) {
t.Parallel()
got := libvirtErrorFields(fmt.Errorf("w: %w", libvirtErr(libvirt.ErrNoDomain, "x")))
want := []any{"libvirtCode", 42, "libvirtError", "ErrNoDomain"}
if len(got) != len(want) {
t.Fatalf("libvirtErrorFields() = %v, want %v", got, want)
}
for i := range want {
if got[i] != want[i] {
t.Errorf("libvirtErrorFields()[%d] = %v (%T), want %v (%T)", i, got[i], got[i], want[i], want[i])
}
}
if fields := libvirtErrorFields(errors.New("x")); fields != nil {
t.Errorf("libvirtErrorFields(non-libvirt) = %v, want nil", fields)
}
if fields := libvirtErrorFields(libvirtErr(libvirt.ErrorNumber(9999), "x")); fields[3] != "ErrorNumber(9999)" {
t.Errorf("libvirtErrorFields(unknown code)[3] = %v, want ErrorNumber(9999)", fields[3])
}
}
func TestPermanentfAndTransientf(t *testing.T) {
t.Parallel()
perm := permanentf("host %q missing", "x")
if !errors.Is(perm, provider.ErrPermanent) || perm.Error() != `provider: permanent error: host "x" missing` {
t.Errorf("permanentf() = %q, want ErrPermanent with the message", perm)
}
trans := transientf("retry %d", 3)
if !errors.Is(trans, provider.ErrTransient) || trans.Error() != "provider: transient error: retry 3" {
t.Errorf("transientf() = %q, want ErrTransient with the message", trans)
}
}
// testPublicKey parses a fixed ed25519 public key so a knownhosts.KeyError
// with a Want entry can be built without generating keys.
func testPublicKey(t *testing.T) ssh.PublicKey {
t.Helper()
const line = "ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIGcZBhnMPTyDwFn9lBhwIutqLb2bSlhcZoqpuWv7VuKI test"
key, _, _, _, err := ssh.ParseAuthorizedKey([]byte(line))
if err != nil {
t.Fatalf("ParseAuthorizedKey: %v", err)
}
return key
}
internal/provider/kvm2/fake_test.go
+623
View File
@@ -0,0 +1,623 @@
package kvm2
import (
"context"
"crypto/sha256"
"encoding/hex"
"encoding/xml"
"fmt"
"io"
"slices"
"sync"
"sync/atomic"
"github.com/digitalocean/go-libvirt"
"gitea.home.hrajfrisbee.cz/kacerr/egress-proxies-operator/internal/provider"
)
// fakeBackend is the in-memory backend of spec entry 21: enough of a
// libvirt host for the provider's Create, Get, Delete and ListByTag, with
// per-method error hooks and blocking, call counters and an ordered log.
type fakeBackend struct {
// mu, domains and pools are the host's state; a twin shares them, so
// two connections to the same fake host see the same volumes.
mu *sync.Mutex
domains map[string]*fakeDomain
pools map[string]*fakePool
hostname string
version uint64
// hooks returns the error a method fails with before touching state.
hooks map[string]func() error
// block makes a method wait on its channel (or ctx) before running.
block map[string]chan struct{}
// discardUploads makes StorageVolUpload stream the reader through a
// sha256 in fixed chunks instead of holding the bytes, for tests that
// push more through the fake than they want in memory.
discardUploads bool
// startDelayPolls, when set, makes DomainCreate leave the domain in
// nostate; it turns running on the startDelayPolls-th DomainGetState
// after that, as a host whose QEMU is still settling would.
startDelayPolls int
// stateAfterStart, when not nostate, is what DomainCreate leaves the
// domain in instead of running, as a guest that dies at once would.
stateAfterStart libvirt.DomainState
// undefineFlags records the flags of the last DomainUndefineFlags per
// domain name.
undefineFlags map[string]libvirt.DomainUndefineFlagsValues
calls map[string]int
log []string
closed int
deadCh chan struct{}
deadMu sync.Once
}
type fakeDomain struct {
state libvirt.DomainState
xml string
metadata string
addrs map[uint32][]libvirt.DomainInterface
uuid libvirt.UUID
id int32
// pendingStart counts the DomainGetState calls left before a delayed
// start (see startDelayPolls) turns the domain running.
pendingStart int
}
type fakePool struct {
state libvirt.StoragePoolState
capacity uint64
available uint64
vols map[string]*fakeVol
}
type fakeVol struct {
content []byte
capacity uint64
// uploaded and sha256 record a discarded upload (see discardUploads).
uploaded int64
sha256 string
}
const fakePoolName = "default"
func newFakeBackend() *fakeBackend {
return &fakeBackend{
mu: &sync.Mutex{},
domains: make(map[string]*fakeDomain),
pools: map[string]*fakePool{fakePoolName: newFakePool()},
hostname: "fake-host",
version: 11006000,
hooks: make(map[string]func() error),
block: make(map[string]chan struct{}),
undefineFlags: make(map[string]libvirt.DomainUndefineFlagsValues),
calls: make(map[string]int),
deadCh: make(chan struct{}),
}
}
// twin returns a second connection to the same fake host: it shares the
// domains and pools (and their lock) but has its own hooks, counters, log,
// close count and liveness, so a test can tell which connection did what.
func (f *fakeBackend) twin() *fakeBackend {
return &fakeBackend{
mu: f.mu,
domains: f.domains,
pools: f.pools,
hostname: f.hostname,
version: f.version,
hooks: make(map[string]func() error),
block: make(map[string]chan struct{}),
discardUploads: f.discardUploads,
undefineFlags: make(map[string]libvirt.DomainUndefineFlagsValues),
calls: make(map[string]int),
deadCh: make(chan struct{}),
}
}
// vol returns the volume by name in the default pool, or nil.
func (f *fakeBackend) vol(name string) *fakeVol {
f.mu.Lock()
defer f.mu.Unlock()
return f.pools[fakePoolName].vols[name]
}
// closeCount is closed read under the lock, for twins closed by another
// goroutine.
func (f *fakeBackend) closeCount() int {
f.mu.Lock()
defer f.mu.Unlock()
return f.closed
}
func newFakePool() *fakePool {
return &fakePool{state: libvirt.StoragePoolRunning, capacity: 100 << 30, available: 50 << 30, vols: make(map[string]*fakeVol)}
}
func (f *fakeBackend) dead() <-chan struct{} { return f.deadCh }
func (f *fakeBackend) markDead() { f.deadMu.Do(func() { close(f.deadCh) }) }
// addDomain defines a domain directly, bypassing DomainDefineXML.
func (f *fakeBackend) addDomain(name string, state libvirt.DomainState, metadata string) *fakeDomain {
f.mu.Lock()
defer f.mu.Unlock()
d := &fakeDomain{state: state, metadata: metadata, addrs: make(map[uint32][]libvirt.DomainInterface), uuid: fakeUUID(name)}
f.domains[name] = d
return d
}
func (f *fakeBackend) failWith(method string, err error) {
f.mu.Lock()
defer f.mu.Unlock()
f.hooks[method] = func() error { return err }
}
func (f *fakeBackend) blockOn(method string) chan struct{} {
f.mu.Lock()
defer f.mu.Unlock()
ch := make(chan struct{})
f.block[method] = ch
return ch
}
func (f *fakeBackend) callCount(method string) int {
f.mu.Lock()
defer f.mu.Unlock()
return f.calls[method]
}
func (f *fakeBackend) callLog() []string {
f.mu.Lock()
defer f.mu.Unlock()
return slices.Clone(f.log)
}
func fakeUUID(name string) libvirt.UUID {
var u libvirt.UUID
copy(u[:], name)
return u
}
func fakeErr(code libvirt.ErrorNumber, format string, args ...any) error {
return libvirt.Error{Code: uint32(code), Message: fmt.Sprintf(format, args...)}
}
// enter records the call, honours a block and runs the hook. It is called
// without f.mu held; state access happens afterwards under the lock.
func (f *fakeBackend) enter(ctx context.Context, method string) error {
f.mu.Lock()
f.calls[method]++
f.log = append(f.log, method)
block := f.block[method]
hook := f.hooks[method]
f.mu.Unlock()
if block != nil {
select {
case <-block:
case <-ctx.Done():
return ctx.Err()
}
}
if hook != nil {
return hook()
}
return nil
}
func (f *fakeBackend) domain(name string) (*fakeDomain, error) {
d, ok := f.domains[name]
if !ok {
return nil, fakeErr(libvirt.ErrNoDomain, "Domain not found: no domain with matching name '%s'", name)
}
return d, nil
}
func (f *fakeBackend) pool(name string) (*fakePool, error) {
p, ok := f.pools[name]
if !ok {
return nil, fakeErr(libvirt.ErrNoStoragePool, "Storage pool not found: no storage pool with matching name '%s'", name)
}
return p, nil
}
func (f *fakeBackend) volume(vol libvirt.StorageVol) (*fakeVol, error) {
p, err := f.pool(vol.Pool)
if err != nil {
return nil, err
}
v, ok := p.vols[vol.Name]
if !ok {
return nil, fakeErr(libvirt.ErrNoStorageVol, "Storage volume not found: no storage vol with matching name '%s'", vol.Name)
}
return v, nil
}
func (f *fakeBackend) libvirtDomain(name string, d *fakeDomain) libvirt.Domain {
id := int32(-1)
if isActive(d.state) {
id = d.id
}
return libvirt.Domain{Name: name, UUID: d.uuid, ID: id}
}
func (f *fakeBackend) DomainLookupByName(ctx context.Context, name string) (libvirt.Domain, error) {
if err := f.enter(ctx, "DomainLookupByName"); err != nil {
return libvirt.Domain{}, err
}
f.mu.Lock()
defer f.mu.Unlock()
d, err := f.domain(name)
if err != nil {
return libvirt.Domain{}, err
}
return f.libvirtDomain(name, d), nil
}
func (f *fakeBackend) DomainDefineXML(ctx context.Context, doc string) (libvirt.Domain, error) {
if err := f.enter(ctx, "DomainDefineXML"); err != nil {
return libvirt.Domain{}, err
}
var parsed struct {
Name string `xml:"name"`
Metadata struct {
Inner string `xml:",innerxml"`
} `xml:"metadata"`
}
if err := xml.Unmarshal([]byte(doc), &parsed); err != nil || parsed.Name == "" {
return libvirt.Domain{}, fakeErr(libvirt.ErrXMLError, "XML error: %v", err)
}
f.mu.Lock()
defer f.mu.Unlock()
d, ok := f.domains[parsed.Name]
if !ok {
d = &fakeDomain{state: libvirt.DomainShutoff, addrs: make(map[uint32][]libvirt.DomainInterface), uuid: fakeUUID(parsed.Name)}
f.domains[parsed.Name] = d
}
d.xml = doc
d.metadata = parsed.Metadata.Inner
return f.libvirtDomain(parsed.Name, d), nil
}
func (f *fakeBackend) DomainCreate(ctx context.Context, dom libvirt.Domain) error {
if err := f.enter(ctx, "DomainCreate"); err != nil {
return err
}
f.mu.Lock()
defer f.mu.Unlock()
d, err := f.domain(dom.Name)
if err != nil {
return err
}
if isActive(d.state) {
return fakeErr(libvirt.ErrOperationInvalid, "Requested operation is not valid: domain is already running")
}
switch {
case f.startDelayPolls > 0:
d.state = libvirt.DomainNostate
d.pendingStart = f.startDelayPolls
return nil
case f.stateAfterStart != libvirt.DomainNostate:
d.state = f.stateAfterStart
return nil
}
d.state = libvirt.DomainRunning
d.id = int32(len(f.domains))
return nil
}
func (f *fakeBackend) DomainDestroy(ctx context.Context, dom libvirt.Domain) error {
if err := f.enter(ctx, "DomainDestroy"); err != nil {
return err
}
f.mu.Lock()
defer f.mu.Unlock()
d, err := f.domain(dom.Name)
if err != nil {
return err
}
if !isActive(d.state) {
return fakeErr(libvirt.ErrOperationInvalid, "Requested operation is not valid: domain is not running")
}
d.state = libvirt.DomainShutoff
return nil
}
func (f *fakeBackend) DomainUndefineFlags(ctx context.Context, dom libvirt.Domain, flags libvirt.DomainUndefineFlagsValues) error {
if err := f.enter(ctx, "DomainUndefineFlags"); err != nil {
return err
}
f.mu.Lock()
defer f.mu.Unlock()
if _, err := f.domain(dom.Name); err != nil {
return err
}
f.undefineFlags[dom.Name] = flags
delete(f.domains, dom.Name)
return nil
}
func (f *fakeBackend) DomainGetState(ctx context.Context, dom libvirt.Domain, _ uint32) (int32, int32, error) {
if err := f.enter(ctx, "DomainGetState"); err != nil {
return 0, 0, err
}
f.mu.Lock()
defer f.mu.Unlock()
d, err := f.domain(dom.Name)
if err != nil {
return 0, 0, err
}
if d.pendingStart > 0 {
d.pendingStart--
if d.pendingStart == 0 {
d.state = libvirt.DomainRunning
d.id = int32(len(f.domains))
}
}
return int32(d.state), 0, nil
}
func (f *fakeBackend) DomainGetMetadata(ctx context.Context, dom libvirt.Domain, typ int32, uri libvirt.OptString, _ libvirt.DomainModificationImpact) (string, error) {
if err := f.enter(ctx, "DomainGetMetadata"); err != nil {
return "", err
}
f.mu.Lock()
defer f.mu.Unlock()
d, err := f.domain(dom.Name)
if err != nil {
return "", err
}
if typ != int32(libvirt.DomainMetadataElement) || len(uri) != 1 || uri[0] != metadataURI || d.metadata == "" {
return "", fakeErr(libvirt.ErrNoDomainMetadata, "Requested metadata element is not present")
}
return d.metadata, nil
}
func (f *fakeBackend) DomainInterfaceAddresses(ctx context.Context, dom libvirt.Domain, source uint32, _ uint32) ([]libvirt.DomainInterface, error) {
if err := f.enter(ctx, "DomainInterfaceAddresses"); err != nil {
return nil, err
}
f.mu.Lock()
defer f.mu.Unlock()
d, err := f.domain(dom.Name)
if err != nil {
return nil, err
}
if !isActive(d.state) {
return nil, fakeErr(libvirt.ErrOperationInvalid, "Requested operation is not valid: domain is not running")
}
return slices.Clone(d.addrs[source]), nil
}
func (f *fakeBackend) ConnectListAllDomains(ctx context.Context, _ int32, _ libvirt.ConnectListAllDomainsFlags) ([]libvirt.Domain, uint32, error) {
if err := f.enter(ctx, "ConnectListAllDomains"); err != nil {
return nil, 0, err
}
f.mu.Lock()
defer f.mu.Unlock()
names := make([]string, 0, len(f.domains))
for name := range f.domains {
names = append(names, name)
}
slices.Sort(names)
out := make([]libvirt.Domain, 0, len(names))
for _, name := range names {
out = append(out, f.libvirtDomain(name, f.domains[name]))
}
return out, uint32(len(out)), nil
}
func (f *fakeBackend) StoragePoolLookupByName(ctx context.Context, name string) (libvirt.StoragePool, error) {
if err := f.enter(ctx, "StoragePoolLookupByName"); err != nil {
return libvirt.StoragePool{}, err
}
f.mu.Lock()
defer f.mu.Unlock()
if _, err := f.pool(name); err != nil {
return libvirt.StoragePool{}, err
}
return libvirt.StoragePool{Name: name, UUID: fakeUUID(name)}, nil
}
func (f *fakeBackend) StoragePoolRefresh(ctx context.Context, pool libvirt.StoragePool, _ uint32) error {
if err := f.enter(ctx, "StoragePoolRefresh"); err != nil {
return err
}
f.mu.Lock()
defer f.mu.Unlock()
_, err := f.pool(pool.Name)
return err
}
func (f *fakeBackend) StoragePoolGetInfo(ctx context.Context, pool libvirt.StoragePool) (uint8, uint64, uint64, uint64, error) {
if err := f.enter(ctx, "StoragePoolGetInfo"); err != nil {
return 0, 0, 0, 0, err
}
f.mu.Lock()
defer f.mu.Unlock()
p, err := f.pool(pool.Name)
if err != nil {
return 0, 0, 0, 0, err
}
return uint8(p.state), p.capacity, p.capacity - p.available, p.available, nil
}
func (f *fakeBackend) StorageVolLookupByName(ctx context.Context, pool libvirt.StoragePool, name string) (libvirt.StorageVol, error) {
if err := f.enter(ctx, "StorageVolLookupByName"); err != nil {
return libvirt.StorageVol{}, err
}
f.mu.Lock()
defer f.mu.Unlock()
vol := libvirt.StorageVol{Pool: pool.Name, Name: name, Key: fakeVolPath(pool.Name, name)}
if _, err := f.volume(vol); err != nil {
return libvirt.StorageVol{}, err
}
return vol, nil
}
func fakeVolPath(pool, name string) string { return "/pool/" + pool + "/" + name }
func (f *fakeBackend) StorageVolGetPath(ctx context.Context, vol libvirt.StorageVol) (string, error) {
if err := f.enter(ctx, "StorageVolGetPath"); err != nil {
return "", err
}
f.mu.Lock()
defer f.mu.Unlock()
if _, err := f.volume(vol); err != nil {
return "", err
}
return fakeVolPath(vol.Pool, vol.Name), nil
}
func (f *fakeBackend) StorageVolCreateXML(ctx context.Context, pool libvirt.StoragePool, doc string, _ libvirt.StorageVolCreateFlags) (libvirt.StorageVol, error) {
if err := f.enter(ctx, "StorageVolCreateXML"); err != nil {
return libvirt.StorageVol{}, err
}
var parsed struct {
Name string `xml:"name"`
Capacity uint64 `xml:"capacity"`
}
if err := xml.Unmarshal([]byte(doc), &parsed); err != nil || parsed.Name == "" {
return libvirt.StorageVol{}, fakeErr(libvirt.ErrXMLError, "XML error: %v", err)
}
f.mu.Lock()
defer f.mu.Unlock()
p, err := f.pool(pool.Name)
if err != nil {
return libvirt.StorageVol{}, err
}
if _, exists := p.vols[parsed.Name]; exists {
return libvirt.StorageVol{}, fakeErr(libvirt.ErrStorageVolExist, "storage volume '%s' exists already", parsed.Name)
}
p.vols[parsed.Name] = &fakeVol{capacity: parsed.Capacity}
return libvirt.StorageVol{Pool: pool.Name, Name: parsed.Name, Key: fakeVolPath(pool.Name, parsed.Name)}, nil
}
func (f *fakeBackend) StorageVolUpload(ctx context.Context, vol libvirt.StorageVol, r io.Reader, _, _ uint64, _ libvirt.StorageVolUploadFlags) error {
if err := f.enter(ctx, "StorageVolUpload"); err != nil {
return err
}
var (
content []byte
uploaded int64
digest string
)
if r != nil {
var err error
if f.discardUploads {
h := sha256.New()
if uploaded, err = io.CopyBuffer(h, r, make([]byte, 64<<10)); err != nil {
return err
}
digest = hex.EncodeToString(h.Sum(nil))
} else if content, err = io.ReadAll(r); err != nil {
return err
}
}
f.mu.Lock()
defer f.mu.Unlock()
v, err := f.volume(vol)
if err != nil {
return err
}
v.content = content
v.uploaded = uploaded
v.sha256 = digest
return nil
}
func (f *fakeBackend) StorageVolDownload(ctx context.Context, vol libvirt.StorageVol, w io.Writer, _, _ uint64, _ libvirt.StorageVolDownloadFlags) error {
if err := f.enter(ctx, "StorageVolDownload"); err != nil {
return err
}
f.mu.Lock()
v, err := f.volume(vol)
var content []byte
if v != nil {
content = slices.Clone(v.content)
}
f.mu.Unlock()
if err != nil {
return err
}
if w == nil {
return nil
}
_, err = w.Write(content)
return err
}
func (f *fakeBackend) StorageVolDelete(ctx context.Context, vol libvirt.StorageVol, _ libvirt.StorageVolDeleteFlags) error {
if err := f.enter(ctx, "StorageVolDelete"); err != nil {
return err
}
f.mu.Lock()
defer f.mu.Unlock()
if _, err := f.volume(vol); err != nil {
return err
}
delete(f.pools[vol.Pool].vols, vol.Name)
return nil
}
func (f *fakeBackend) ConnectGetLibVersion(ctx context.Context) (uint64, error) {
if err := f.enter(ctx, "ConnectGetLibVersion"); err != nil {
return 0, err
}
return f.version, nil
}
func (f *fakeBackend) ConnectGetHostname(ctx context.Context) (string, error) {
if err := f.enter(ctx, "ConnectGetHostname"); err != nil {
return "", err
}
return f.hostname, nil
}
func (f *fakeBackend) Close() error {
f.mu.Lock()
defer f.mu.Unlock()
f.closed++
return nil
}
// stubDialer covers every dialer a test needs: it counts dials, fails with
// err when set, waits on block (or ctx) when set, and hands out backend, or
// one from fresh per call, or a new fakeBackend when neither is given.
type stubDialer struct {
backend backend
fresh func() backend
err error
block chan struct{}
dials atomic.Int32
mu sync.Mutex
hosts []string
}
func (d *stubDialer) dial(ctx context.Context, host provider.KVMHost) (backend, error) {
d.dials.Add(1)
d.mu.Lock()
d.hosts = append(d.hosts, host.Name)
d.mu.Unlock()
if d.block != nil {
select {
case <-d.block:
case <-ctx.Done():
return nil, fmt.Errorf("%w: dialing %s: %w", provider.ErrTransient, host.Name, ctx.Err())
}
}
if d.err != nil {
return nil, d.err
}
switch {
case d.fresh != nil:
return d.fresh(), nil
case d.backend != nil:
return d.backend, nil
default:
return newFakeBackend(), nil
}
}
internal/provider/kvm2/get.go
+133
View File
@@ -0,0 +1,133 @@
package kvm2
import (
"context"
"errors"
"fmt"
"github.com/digitalocean/go-libvirt"
"github.com/go-logr/logr"
"gitea.home.hrajfrisbee.cz/kacerr/egress-proxies-operator/internal/provider"
)
// Get is spec entries 7 to 9: look the domain up by name, read the mark
// back, then state and address. Only ErrNoDomain becomes NotFound; every
// other failure is classified, never guessed.
func (p *Provider) Get(ctx context.Context, providerID string) (*provider.Instance, error) {
hostName, name, err := parseProviderID(providerID)
if err != nil {
return nil, provider.Wrap(provider.ErrPermanent, "get", p.name, providerID, err)
}
host, err := p.host(hostName)
if err != nil {
return nil, p.wrapErr("get", providerID, err)
}
log := p.logger(ctx).WithValues("host", host.Name, "name", name)
be, err := p.hosts.get(ctx, host)
if err != nil {
logAPIError(log, "get", err)
return nil, p.wrapErr("get", providerID, err)
}
inst, err := get(ctx, log, be, host.Name, name)
if err != nil {
if hasCode(err, libvirt.ErrNoDomain) {
log.V(1).Info("kvm2 domain not found")
} else {
logAPIError(log, "get", err)
}
return nil, p.wrapErr("get", providerID, err)
}
log.V(1).Info("kvm2 domain fetched", "state", inst.State, "ip", inst.IP, "uid", inst.UID)
return inst, nil
}
// get builds the Instance from the domain's metadata, its state and, for an
// active domain, its discovered address.
func get(ctx context.Context, log logr.Logger, be backend, hostName, name string) (*provider.Instance, error) {
dom, err := be.DomainLookupByName(ctx, name)
if err != nil {
return nil, fmt.Errorf("looking up domain %s: %w", name, err)
}
meta, err := readMetadata(ctx, be, dom)
if err != nil {
return nil, err
}
state, _, err := be.DomainGetState(ctx, dom, 0)
if err != nil {
return nil, fmt.Errorf("reading state of domain %s: %w", dom.Name, err)
}
s := libvirt.DomainState(state)
var ip string
if isActive(s) {
if ip, err = address(ctx, log, be, dom); err != nil {
return nil, err
}
}
return &provider.Instance{
ID: formatProviderID(hostName, dom.Name),
IP: ip,
State: mapState(s, ip),
UID: meta.UID,
CreatedAt: meta.createdAt(),
}, nil
}
// readMetadata returns the kvm2 mark of a domain, or the zero value (not
// managed, empty UID) when the domain carries none. A mark that does not
// parse is Permanent, from parseInstanceMetadata.
func readMetadata(ctx context.Context, be backend, dom libvirt.Domain) (instanceMetadata, error) {
raw, err := be.DomainGetMetadata(ctx, dom, int32(libvirt.DomainMetadataElement), libvirt.OptString{metadataURI}, libvirt.DomainAffectCurrent)
if hasCode(err, libvirt.ErrNoDomainMetadata) {
return instanceMetadata{}, nil
}
if err != nil {
return instanceMetadata{}, fmt.Errorf("reading metadata of domain %s: %w", dom.Name, err)
}
return parseInstanceMetadata(raw)
}
// addressSources is the discovery order of spec entry 7: the guest agent
// answers where the host's ARP table never sees guest traffic.
var addressSources = []struct {
name string
src libvirt.DomainInterfaceAddressesSource
}{
{"agent", libvirt.DomainInterfaceAddressesSrcAgent},
{"arp", libvirt.DomainInterfaceAddressesSrcArp},
}
// address returns the first usable IPv4 from the first source that yields
// one, or "" for "no address yet". One source failing is normal and is
// skipped: a booting VM has no agent yet, while the host's ARP table
// simply reports nothing, since it never sees traffic addressed
// elsewhere. A connection already gone, and every source failing, are
// errors instead — otherwise a link some other reconcile killed reads as
// a running VM that lost its address, and the controller clears its IP
// and the health engine drops its accumulated state.
func address(ctx context.Context, log logr.Logger, be backend, dom libvirt.Domain) (string, error) {
errs := make([]error, 0, len(addressSources))
for _, s := range addressSources {
ifaces, err := be.DomainInterfaceAddresses(ctx, dom, uint32(s.src), 0)
if err != nil {
if isDead(be) || errors.Is(err, context.Canceled) || errors.Is(err, context.DeadlineExceeded) {
return "", fmt.Errorf("reading addresses of domain %s from the %s source: %w", dom.Name, s.name, err)
}
log.V(2).Info("no addresses from source", "source", s.name, "error", err.Error())
errs = append(errs, fmt.Errorf("%s source: %w", s.name, err))
continue
}
if ip := firstIPv4(ifaces); ip != "" {
return ip, nil
}
}
// Transient explicitly: classify would otherwise pick a libvirt code
// out of the joined error, and ErrInvalidArg, ErrAccessDenied and
// ErrNoSupport are Permanent, which would latch a running proxy as
// Failed over an address read.
if len(errs) == len(addressSources) {
return "", fmt.Errorf("%w: reading addresses of domain %s: every source failed: %w",
provider.ErrTransient, dom.Name, errors.Join(errs...))
}
return "", nil
}
internal/provider/kvm2/get_test.go
+295
View File
@@ -0,0 +1,295 @@
package kvm2
import (
"context"
"errors"
"fmt"
"strings"
"testing"
"time"
"github.com/digitalocean/go-libvirt"
"gitea.home.hrajfrisbee.cz/kacerr/egress-proxies-operator/internal/provider"
)
func TestGet_afterDeleteReturnsNotFound(t *testing.T) {
t.Parallel()
fx := newFixture(t, 1)
id := fx.create(testName)
if err := fx.p.Delete(fx.ctx, id); err != nil {
t.Fatalf("Delete() error = %v", err)
}
inst, err := fx.p.Get(fx.ctx, id)
if inst != nil {
t.Errorf("Get after Delete = %+v, want nil", inst)
}
assertProviderError(t, err, "get", provider.ErrNotFound)
}
func TestGet_duringDeleteStillReturnsInstance(t *testing.T) {
t.Parallel()
t.Run("destroyed but still defined is stopped", func(t *testing.T) {
t.Parallel()
fx := newFixture(t, 1)
id := fx.create(testName)
fx.withDomain(testName, func(d *fakeDomain) { d.state = libvirt.DomainShutoff })
inst := fx.get(id)
if inst.State != provider.StateStopped || inst.UID != testUID {
t.Errorf("Get = %+v, want Stopped with uid %s", inst, testUID)
}
})
t.Run("undefine in flight still answers", func(t *testing.T) {
t.Parallel()
fx := newFixture(t, 1)
id := fx.create(testName)
release := fx.fake.blockOn("DomainUndefineFlags")
done := make(chan error, 1)
go func() { done <- fx.p.Delete(fx.ctx, id) }()
waitFor(t, "the undefine to block", func() bool { return fx.fake.callCount("DomainUndefineFlags") == 1 })
inst, err := fx.p.Get(fx.ctx, id)
if err != nil {
t.Fatalf("Get during Delete error = %v, want the instance", err)
}
if inst.State != provider.StateStopped || inst.ID != id {
t.Errorf("Get during Delete = %+v, want Stopped %s", inst, id)
}
close(release)
if err := <-done; err != nil {
t.Fatalf("Delete() error = %v", err)
}
_, err = fx.p.Get(fx.ctx, id)
assertProviderError(t, err, "get", provider.ErrNotFound)
})
}
func TestGet_backendBusyIsTransientNeverNotFound(t *testing.T) {
t.Parallel()
tests := []struct {
name string
err error
}{
{name: "operation invalid", err: fakeErr(libvirt.ErrOperationInvalid, "Requested operation is not valid: domain is being migrated")},
{name: "call timed out", err: fmt.Errorf("%w: libvirt call DomainGetState on beelink exceeded its 30s budget: %w", provider.ErrTransient, context.DeadlineExceeded)},
{name: "unknown libvirt code", err: fakeErr(libvirt.ErrInternalError, "internal error: qemu monitor unresponsive")},
{name: "not a libvirt error", err: errors.New("read tcp: connection reset by peer")},
}
for _, tc := range tests {
t.Run(tc.name, func(t *testing.T) {
t.Parallel()
fx := newFixture(t, 1)
id := fx.create(testName)
fx.fake.failWith("DomainGetState", tc.err)
inst, err := fx.p.Get(fx.ctx, id)
if inst != nil {
t.Errorf("Get = %+v, want nil", inst)
}
assertProviderError(t, err, "get", provider.ErrTransient)
if errors.Is(err, provider.ErrNotFound) {
t.Errorf("Get error = %v, must never be NotFound", err)
}
})
}
}
func TestGet_runningWithoutIPIsProvisioning(t *testing.T) {
t.Parallel()
fx := newFixture(t, 1)
inst := fx.get(fx.create(testName))
if inst.State != provider.StateProvisioning || inst.IP != "" {
t.Errorf("Get = %+v, want Provisioning with no IP", inst)
}
}
func TestGet_runningWithIPIsRunning(t *testing.T) {
t.Parallel()
fx := newFixture(t, 1)
id := fx.create(testName)
fx.setAddrs(testName, libvirt.DomainInterfaceAddressesSrcAgent, "127.0.0.1", "169.254.1.2", "10.0.0.5")
inst := fx.get(id)
if inst.State != provider.StateRunning || inst.IP != "10.0.0.5" {
t.Errorf("Get = %+v, want Running with 10.0.0.5", inst)
}
if got := fx.logs.matching("kvm2 domain fetched"); len(got) != 1 || !strings.Contains(got[0], `"ip"="10.0.0.5"`) || !strings.Contains(got[0], `"state"="Running"`) {
t.Errorf("fetched line = %v, want state and ip", got)
}
}
func TestGet_agentPreferredOverARP(t *testing.T) {
t.Parallel()
fx := newFixture(t, 1)
id := fx.create(testName)
fx.setAddrs(testName, libvirt.DomainInterfaceAddressesSrcAgent, "10.0.0.5")
fx.setAddrs(testName, libvirt.DomainInterfaceAddressesSrcArp, "10.0.0.6")
before := fx.fake.callCount("DomainInterfaceAddresses")
inst := fx.get(id)
if inst.IP != "10.0.0.5" {
t.Errorf("IP = %q, want the agent's 10.0.0.5", inst.IP)
}
if got := fx.fake.callCount("DomainInterfaceAddresses") - before; got != 1 {
t.Errorf("DomainInterfaceAddresses calls = %d, want 1 (ARP not consulted)", got)
}
}
func TestGet_arpFallbackWhenAgentFails(t *testing.T) {
t.Parallel()
fx := newFixture(t, 2)
id := fx.create(testName)
fx.setAddrs(testName, libvirt.DomainInterfaceAddressesSrcArp, "10.0.0.6")
var calls int
fx.fake.mu.Lock()
fx.fake.hooks["DomainInterfaceAddresses"] = func() error {
calls++
if calls%2 == 1 {
return fakeErr(libvirt.ErrAgentUnresponsive, "Guest agent is not responding")
}
return nil
}
fx.fake.mu.Unlock()
inst := fx.get(id)
if inst.State != provider.StateRunning || inst.IP != "10.0.0.6" {
t.Errorf("Get = %+v, want Running with ARP's 10.0.0.6", inst)
}
if got := fx.logs.matching("no addresses from source"); len(got) != 1 || !strings.Contains(got[0], `"source"="agent"`) {
t.Errorf("agent failure line = %v", got)
}
}
// TestGet_agentFailsAndARPEmptyIsProvisioning is a VM part way through
// booting on a plain bridge: the agent is not up yet, and the host's ARP
// table answers without error because it has nothing to report. One
// source failing must stay "no address yet", not an error.
func TestGet_agentFailsAndARPEmptyIsProvisioning(t *testing.T) {
t.Parallel()
fx := newFixture(t, 2)
id := fx.create(testName)
var calls int
fx.fake.mu.Lock()
fx.fake.hooks["DomainInterfaceAddresses"] = func() error {
calls++
if calls%2 == 1 {
return fakeErr(libvirt.ErrAgentUnresponsive, "Guest agent is not responding")
}
return nil
}
fx.fake.mu.Unlock()
inst := fx.get(id)
if inst.State != provider.StateProvisioning || inst.IP != "" {
t.Errorf("Get = %+v, want Provisioning with no IP", inst)
}
}
// TestGet_everyAddressSourceFailingIsTransient covers the other side of
// that line. A running VM whose address cannot be read is not a VM
// without an address: reporting Provisioning would make the controller
// clear its IP and the health engine drop its counters.
func TestGet_everyAddressSourceFailingIsTransient(t *testing.T) {
t.Parallel()
tests := []struct {
name string
setup func(fx *fixture)
}{
{
name: "both sources error",
setup: func(fx *fixture) {
fx.fake.failWith("DomainInterfaceAddresses", fakeErr(libvirt.ErrOperationInvalid, "domain is not running"))
},
},
{
name: "the pooled connection died under the read",
setup: func(fx *fixture) {
fx.fake.failWith("DomainInterfaceAddresses", transientf("libvirt call DomainInterfaceAddresses on beelink: connection is closed: keepalive failed"))
fx.fake.markDead()
},
},
}
for _, tc := range tests {
t.Run(tc.name, func(t *testing.T) {
t.Parallel()
fx := newFixture(t, 1)
id := fx.create(testName)
tc.setup(fx)
inst, err := fx.p.Get(fx.ctx, id)
if inst != nil {
t.Errorf("Get = %+v, want no instance", inst)
}
assertProviderError(t, err, "get", provider.ErrTransient)
})
}
}
func TestGet_readsUIDAndCreatedAtFromBackend(t *testing.T) {
t.Parallel()
fx := newFixture(t, 1)
created := time.Date(2026, 9, 16, 12, 0, 0, 0, time.UTC)
fx.p.now = func() time.Time { return created }
id := fx.create(testName)
inst := fx.get(id)
if inst.UID != testUID || !inst.CreatedAt.Equal(created) {
t.Errorf("Get = %+v, want uid %s created %s", inst, testUID, created)
}
later := created.Add(48 * time.Hour)
fx.withDomain(testName, func(d *fakeDomain) { d.metadata = managedMetadata("rewritten-uid", later) })
inst = fx.get(id)
if inst.UID != "rewritten-uid" || !inst.CreatedAt.Equal(later) {
t.Errorf("Get after metadata change = %+v, want uid rewritten-uid created %s (read back, not remembered)", inst, later)
}
}
func TestGet_unmanagedDomainHasEmptyUID(t *testing.T) {
t.Parallel()
fx := newFixture(t, 1)
fx.fake.addDomain("legacy-vm", libvirt.DomainRunning, "")
inst := fx.get(formatProviderID(testHost, "legacy-vm"))
if inst.UID != "" || !inst.CreatedAt.IsZero() || inst.State != provider.StateProvisioning {
t.Errorf("Get = %+v, want empty uid, zero createdAt, Provisioning", inst)
}
}
func TestGet_malformedIDIsPermanent(t *testing.T) {
t.Parallel()
ids := []string{"", "proxy-x", "beelink/proxy-x", "hosts/beelink/proxy-x", "hosts//domains/proxy-x", "hosts/beelink/domains/", "hosts/beelink/domains/a/b", "nodes/beelink/domains/proxy-x"}
for _, id := range ids {
t.Run(id, func(t *testing.T) {
t.Parallel()
fx := newFixture(t, 1)
inst, err := fx.p.Get(fx.ctx, id)
if inst != nil {
t.Errorf("Get(%q) = %+v, want nil", id, inst)
}
assertProviderError(t, err, "get", provider.ErrPermanent)
var pe *provider.Error
if errors.As(err, &pe) && pe.ID != id {
t.Errorf("error.ID = %q, want %q", pe.ID, id)
}
if got := fx.dialer.dials.Load(); got != 0 {
t.Errorf("dials = %d, want 0", got)
}
})
}
}
func TestGet_unconfiguredHostIsPermanent(t *testing.T) {
t.Parallel()
fx := newFixture(t, 1)
_, err := fx.p.Get(fx.ctx, formatProviderID("nas9", testName))
assertProviderError(t, err, "get", provider.ErrPermanent)
if !strings.Contains(err.Error(), `"nas9"`) {
t.Errorf("error = %v, want the host named", err)
}
if got := fx.dialer.dials.Load(); got != 0 {
t.Errorf("dials = %d, want 0", got)
}
}
func TestGet_unknownStateIsStopped(t *testing.T) {
t.Parallel()
fx := newFixture(t, 1)
fx.fake.addDomain(testName, libvirt.DomainState(42), managedMetadata(testUID, time.Now()))
inst := fx.get(formatProviderID(testHost, testName))
if inst.State != provider.StateStopped {
t.Errorf("State = %s, want Stopped for an unknown backend state", inst.State)
}
}
internal/provider/kvm2/hosts.go
+162
View File
@@ -0,0 +1,162 @@
package kvm2
import (
"context"
"fmt"
"sync"
"time"
"github.com/digitalocean/go-libvirt"
"go.opentelemetry.io/otel/attribute"
"go.opentelemetry.io/otel/codes"
"go.opentelemetry.io/otel/trace"
logf "sigs.k8s.io/controller-runtime/pkg/log"
"gitea.home.hrajfrisbee.cz/kacerr/egress-proxies-operator/internal/provider"
"gitea.home.hrajfrisbee.cz/kacerr/egress-proxies-operator/internal/tracing"
)
// dialer opens one connection to a host. The production dialer is
// libvirtDialer; tests inject one that hands out fakes.
type dialer interface {
dial(ctx context.Context, host provider.KVMHost) (backend, error)
}
// hostTable holds the pooled connection per host (spec entry 19). One dial
// per host at a time: concurrent callers wait for it and share the result.
// A dead backend (abandoned call, keepalive failure, socket lost) is dropped
// and redialled on the next get.
//
// What a cancelled or overlong call does to the shared connection (spec
// entry 18): the adapter kills it, so every other call in flight on that
// host fails once with Transient and its caller retries on a fresh
// connection. Callers that must not be exposed to that, the image transfer
// and the Checker, use dialFresh and get a connection of their own.
type hostTable struct {
dialer dialer
mu sync.Mutex
hosts map[string]*hostEntry
}
// hostEntry serialises dials for one host. The semaphore, not a mutex, so a
// waiter can give up when its ctx ends.
type hostEntry struct {
sem chan struct{}
backend backend
}
func newHostTable(d dialer) *hostTable {
return &hostTable{dialer: d, hosts: make(map[string]*hostEntry)}
}
func (t *hostTable) entry(name string) *hostEntry {
t.mu.Lock()
defer t.mu.Unlock()
e, ok := t.hosts[name]
if !ok {
e = &hostEntry{sem: make(chan struct{}, 1)}
t.hosts[name] = e
}
return e
}
// get returns the live pooled backend for host, dialling it when there is
// none. Waiting for another caller's dial ends with ctx.
func (t *hostTable) get(ctx context.Context, host provider.KVMHost) (backend, error) {
e := t.entry(host.Name)
select {
case e.sem <- struct{}{}:
case <-ctx.Done():
return nil, fmt.Errorf("%w: waiting for the connection to host %s: %w", provider.ErrTransient, host.Name, ctx.Err())
}
defer func() { <-e.sem }()
if e.backend != nil {
if !isDead(e.backend) {
return e.backend, nil
}
_ = e.backend.Close()
e.backend = nil
}
b, err := t.dialer.dial(ctx, host)
if err != nil {
return nil, err
}
e.backend = b
return b, nil
}
// dialFresh returns a dedicated connection that is never pooled. The
// caller owns it and must Close it.
func (t *hostTable) dialFresh(ctx context.Context, host provider.KVMHost) (backend, error) {
return t.dialer.dial(ctx, host)
}
// libvirtDialer is the production dialer: SSH transport, go-libvirt over
// the forwarded socket, the connect handshake, then the observe decorator,
// so provider code never holds an undecorated backend.
type libvirtDialer struct {
providerName string
cfg provider.KVMConfig
budgets budgets
keepalive time.Duration
tracing []tracing.Option
}
func newLibvirtDialer(providerName string, cfg provider.KVMConfig) *libvirtDialer {
return &libvirtDialer{
providerName: providerName,
cfg: cfg,
budgets: defaultBudgets,
keepalive: defaultKeepaliveInterval,
}
}
func (d *libvirtDialer) dial(ctx context.Context, host provider.KVMHost) (backend, error) {
ctx, span := tracing.StartSpan(ctx, tracing.Tracer(d.tracing...), "kvm2.dial",
trace.WithSpanKind(trace.SpanKindClient),
trace.WithAttributes(
attribute.String("kvm2.host", host.Name),
attribute.String("kvm2.address", host.Address),
attribute.String("kvm2.user", host.User),
))
defer span.End()
log := logf.FromContext(ctx).WithName("kvm2").WithValues("provider", d.providerName, "host", host.Name)
start := time.Now()
b, err := d.connect(ctx, host)
if err != nil {
span.SetStatus(codes.Error, err.Error())
kv := []any{"address", host.Address, "user", host.User, "duration", time.Since(start), "error", err.Error()}
if stage, ok := stageOf(err); ok {
kv = append(kv, "stage", string(stage))
}
kv = append(kv, libvirtErrorFields(err)...)
log.V(1).Info("libvirt host connect failed", kv...)
return nil, err
}
log.V(1).Info("libvirt host connected", "address", host.Address, "user", host.User, "duration", time.Since(start))
return observe(b, d.providerName, host.Name, d.tracing...), nil
}
// connect is the dial sequence under one budget: the SSH stages, then the
// libvirt handshake. On success the keepalive and the disconnect watcher
// run until the link dies.
func (d *libvirtDialer) connect(ctx context.Context, host provider.KVMHost) (*libvirtBackend, error) {
ctx, cancel := context.WithTimeout(ctx, d.budgets.dial)
defer cancel()
sock, err := dialSSHSocket(ctx, host, d.cfg, 0)
if err != nil {
return nil, err
}
b := newLibvirtBackend(libvirt.New(sock), host.Name, sock, d.budgets)
if err := b.connect(ctx); err != nil {
b.link.markDead("connect failed")
return nil, fmt.Errorf("connecting to libvirt on host %s: %w", host.Name, err)
}
go func() {
if err := sock.keepalive(d.keepalive, b.link.dead()); err != nil {
b.link.markDead(err.Error())
}
}()
return b, nil
}
internal/provider/kvm2/hosts_test.go
+195
View File
@@ -0,0 +1,195 @@
package kvm2
import (
"context"
"errors"
"sync"
"testing"
"time"
"gitea.home.hrajfrisbee.cz/kacerr/egress-proxies-operator/internal/provider"
)
var (
hostA = provider.KVMHost{Name: "beelink", Address: "beelink:22", User: "virt"}
hostB = provider.KVMHost{Name: "lab2", Address: "lab2:22", User: "virt"}
)
func TestHosts_oneDialPerHostUnderConcurrency(t *testing.T) {
t.Parallel()
shared := newFakeBackend()
d := &stubDialer{backend: shared}
table := newHostTable(d)
const n = 50
var wg sync.WaitGroup
got := make([]backend, n)
errs := make([]error, n)
for i := range n {
wg.Add(1)
go func() {
defer wg.Done()
got[i], errs[i] = table.get(context.Background(), hostA)
}()
}
wg.Wait()
for i := range n {
if errs[i] != nil {
t.Fatalf("get #%d error = %v", i, errs[i])
}
if got[i] != backend(shared) {
t.Fatalf("get #%d returned a different backend", i)
}
}
if d.dials.Load() != 1 {
t.Errorf("dials = %d, want 1", d.dials.Load())
}
}
func TestHosts_redialsAfterDrop(t *testing.T) {
t.Parallel()
first := newFakeBackend()
second := newFakeBackend()
backends := []backend{first, second}
d := &stubDialer{fresh: func() backend {
b := backends[0]
backends = backends[1:]
return b
}}
table := newHostTable(d)
b, err := table.get(context.Background(), hostA)
if err != nil || b != backend(first) {
t.Fatalf("first get = %v, %v, want the first backend", b, err)
}
if b, _ := table.get(context.Background(), hostA); b != backend(first) {
t.Fatal("second get did not reuse the live backend")
}
first.markDead()
b, err = table.get(context.Background(), hostA)
if err != nil || b != backend(second) {
t.Fatalf("get after drop = %v, %v, want the second backend", b, err)
}
if first.closed != 1 {
t.Errorf("dropped backend closed %d times, want 1", first.closed)
}
if d.dials.Load() != 2 {
t.Errorf("dials = %d, want 2", d.dials.Load())
}
}
func TestHosts_dialFreshIsNotCached(t *testing.T) {
t.Parallel()
d := &stubDialer{fresh: func() backend { return newFakeBackend() }}
table := newHostTable(d)
pooled, err := table.get(context.Background(), hostA)
if err != nil {
t.Fatalf("get error = %v", err)
}
fresh, err := table.dialFresh(context.Background(), hostA)
if err != nil {
t.Fatalf("dialFresh error = %v", err)
}
if fresh == pooled {
t.Fatal("dialFresh returned the pooled backend")
}
again, _ := table.get(context.Background(), hostA)
if again != pooled {
t.Error("get after dialFresh did not return the pooled backend")
}
if d.dials.Load() != 2 {
t.Errorf("dials = %d, want 2", d.dials.Load())
}
if fresh.(*fakeBackend).closed != 0 {
t.Error("dialFresh's backend was closed by the table; the caller owns it")
}
}
func TestHosts_waitingForDialEndsWithContext(t *testing.T) {
t.Parallel()
release := make(chan struct{})
d := &stubDialer{block: release}
table := newHostTable(d)
firstDone := make(chan error, 1)
go func() {
_, err := table.get(context.Background(), hostA)
firstDone <- err
}()
waitForDials(t, d, 1)
ctx, cancel := context.WithTimeout(context.Background(), 50*time.Millisecond)
defer cancel()
start := time.Now()
_, err := table.get(ctx, hostA)
if time.Since(start) > time.Second {
t.Errorf("second get took %v, want to give up with its ctx", time.Since(start))
}
if !errors.Is(err, provider.ErrTransient) || !errors.Is(err, context.DeadlineExceeded) {
t.Errorf("second get error = %v, want ErrTransient wrapping DeadlineExceeded", err)
}
close(release)
if err := <-firstDone; err != nil {
t.Errorf("first get error = %v, want nil once the dial completes", err)
}
if d.dials.Load() != 1 {
t.Errorf("dials = %d, want 1: the waiter must not dial itself", d.dials.Load())
}
}
func TestHosts_hostsDialIndependently(t *testing.T) {
t.Parallel()
release := make(chan struct{})
defer close(release)
dA := &stubDialer{block: release}
dB := &stubDialer{}
table := newHostTable(perHostDialer{hostA.Name: dA, hostB.Name: dB})
go func() { _, _ = table.get(context.Background(), hostA) }()
waitForDials(t, dA, 1)
ctx, cancel := context.WithTimeout(context.Background(), time.Second)
defer cancel()
start := time.Now()
if _, err := table.get(ctx, hostB); err != nil {
t.Fatalf("get(hostB) error = %v while hostA's dial is blocked", err)
}
if time.Since(start) > 500*time.Millisecond {
t.Errorf("get(hostB) took %v, want not to wait for hostA", time.Since(start))
}
}
func TestHosts_dialErrorIsNotCached(t *testing.T) {
t.Parallel()
d := &stubDialer{err: transientf("boom")}
table := newHostTable(d)
if _, err := table.get(context.Background(), hostA); !errors.Is(err, provider.ErrTransient) {
t.Fatalf("get error = %v, want the dial error", err)
}
d.err = nil
if _, err := table.get(context.Background(), hostA); err != nil {
t.Fatalf("get after the dialer recovered error = %v", err)
}
if d.dials.Load() != 2 {
t.Errorf("dials = %d, want 2", d.dials.Load())
}
}
// perHostDialer routes each host to its own stub.
type perHostDialer map[string]*stubDialer
func (p perHostDialer) dial(ctx context.Context, host provider.KVMHost) (backend, error) {
return p[host.Name].dial(ctx, host)
}
func waitForDials(t *testing.T, d *stubDialer, n int32) {
t.Helper()
deadline := time.Now().Add(2 * time.Second)
for d.dials.Load() < n {
if time.Now().After(deadline) {
t.Fatalf("dials = %d, want %d", d.dials.Load(), n)
}
time.Sleep(time.Millisecond)
}
}
internal/provider/kvm2/id.go
+28
View File
@@ -0,0 +1,28 @@
package kvm2
import (
"strings"
)
// The providerID is hosts/<host>/domains/<name>. Host names are validated at
// config load to contain no slash, and domain names come from NameFromUID,
// so a strict four-segment split is enough to parse it.
const (
idHostsSegment = "hosts"
idDomainsSegment = "domains"
)
func formatProviderID(host, name string) string {
return idHostsSegment + "/" + host + "/" + idDomainsSegment + "/" + name
}
// parseProviderID splits an ID back into host and domain name. Anything
// that is not exactly hosts/<host>/domains/<name> with both non-empty is
// ErrPermanent: no retry will make it resolve.
func parseProviderID(id string) (host, name string, err error) {
parts := strings.Split(id, "/")
if len(parts) != 4 || parts[0] != idHostsSegment || parts[2] != idDomainsSegment || parts[1] == "" || parts[3] == "" {
return "", "", permanentf("malformed providerID %q: want hosts/<host>/domains/<name>", id)
}
return parts[1], parts[3], nil
}
internal/provider/kvm2/id_test.go
+54
View File
@@ -0,0 +1,54 @@
package kvm2
import (
"errors"
"strings"
"testing"
"gitea.home.hrajfrisbee.cz/kacerr/egress-proxies-operator/internal/provider"
)
func TestFormatProviderID_roundTrip(t *testing.T) {
t.Parallel()
id := formatProviderID("beelink", "proxy-abc123def456ghij")
if id != "hosts/beelink/domains/proxy-abc123def456ghij" {
t.Fatalf("formatProviderID() = %q", id)
}
host, name, err := parseProviderID(id)
if err != nil || host != "beelink" || name != "proxy-abc123def456ghij" {
t.Errorf("parseProviderID(%q) = %q, %q, %v; want beelink, proxy-abc123def456ghij, nil", id, host, name, err)
}
}
func TestParseProviderID_malformed(t *testing.T) {
t.Parallel()
tests := []struct {
name string
id string
}{
{name: "empty", id: ""},
{name: "two segments", id: "hosts/h"},
{name: "three segments", id: "hosts/h/domains"},
{name: "empty host", id: "hosts//domains/n"},
{name: "empty name", id: "hosts/h/domains/"},
{name: "wrong first literal", id: "nodes/h/domains/n"},
{name: "wrong third literal", id: "hosts/h/vms/n"},
{name: "extra segment", id: "hosts/h/domains/n/extra"},
{name: "slash in name", id: "hosts/h/domains/a/b"},
}
for _, tc := range tests {
t.Run(tc.name, func(t *testing.T) {
t.Parallel()
host, name, err := parseProviderID(tc.id)
if err == nil {
t.Fatalf("parseProviderID(%q) = %q, %q, nil; want error", tc.id, host, name)
}
if !errors.Is(err, provider.ErrPermanent) {
t.Errorf("parseProviderID(%q) error = %v, want ErrPermanent", tc.id, err)
}
if !strings.Contains(err.Error(), tc.id) {
t.Errorf("parseProviderID(%q) error = %q, want it to name the ID", tc.id, err)
}
})
}
}
internal/provider/kvm2/image.go
+481
View File
@@ -0,0 +1,481 @@
package kvm2
import (
"context"
"crypto/sha256"
"encoding/hex"
"errors"
"fmt"
"io"
"net/http"
"net/url"
"regexp"
"strings"
"sync"
"time"
"github.com/digitalocean/go-libvirt"
"github.com/go-logr/logr"
"go.opentelemetry.io/otel/attribute"
"go.opentelemetry.io/otel/codes"
"go.opentelemetry.io/otel/trace"
"gitea.home.hrajfrisbee.cz/kacerr/egress-proxies-operator/internal/provider"
"gitea.home.hrajfrisbee.cz/kacerr/egress-proxies-operator/internal/tracing"
)
const (
// markerCapacityBytes is the marker volume's size: one lowercase hex
// sha256 digest, nothing else.
markerCapacityBytes = sha256.Size * 2
// markerReadCap bounds what is read back from a marker, so a wrong file
// under that name cannot grow memory.
markerReadCap = 1 << 10
// seedReserveBytes is what the quota precheck sets aside for the seed
// ISO next to the image and the overlay (spec entry 13).
seedReserveBytes = 16 << 20
)
var hexDigest = regexp.MustCompile(`^[0-9a-f]{64}$`)
// imageSource is a resolved placement.image (spec entries 3 and 4): the
// catalogue entry, or a URL given directly with the catalogue defaults, plus
// the key its base volume is named after.
type imageSource struct {
Entry string
Key string
provider.KVMImage
}
func (s imageSource) volumeName() string { return baseVolumeName(s.Entry, s.Key, s.Format) }
func (s imageSource) markerName() string { return markerVolumeName(s.volumeName()) }
// resolveImage maps placement.image to a source. cfg is already defaulted,
// so a catalogue entry carries its format and firmware; a URL gets qcow2
// and uefi. entry is the provider entry's name, which the base volume is
// named after so two entries never share one file.
func resolveImage(entry string, cfg provider.KVMConfig, image string) (imageSource, error) {
if img, ok := cfg.Images[image]; ok {
return imageSource{Entry: entry, Key: image, KVMImage: img}, nil
}
if provider.IsHTTPURL(image) {
return imageSource{
Entry: entry,
Key: urlImageKey(image),
KVMImage: provider.KVMImage{URL: image, Format: provider.KVMFormatQcow2, Firmware: provider.KVMFirmwareUEFI},
}, nil
}
return imageSource{}, permanentf("placement.image %q is neither a key in kvm2.images nor an http(s) URL", image)
}
// lockImage takes the per-(host, volume) lock of spec entry 19, or gives up
// with ctx: a caller whose ctx ends while another caller's download runs
// returns promptly (contract §1.5) instead of waiting out a 20-minute
// transfer.
func (p *Provider) lockImage(ctx context.Context, hostName, volume string) (func(), error) {
release, err := p.imageLocks.Acquire(ctx, hostName+"/"+volume)
if err != nil {
return nil, fmt.Errorf("%w: waiting for image %s on host %s: %w", provider.ErrTransient, volume, hostName, err)
}
return release, nil
}
// ensureImage makes the base image present and complete in the host's pool
// and returns its path (spec entry 4). be is the caller's pooled connection
// and carries the cheap checks; the upload itself goes over a connection of
// its own. Concurrent callers for the same (host, volume) queue on the
// lock: the first fetches, the rest find the marker.
func (p *Provider) ensureImage(ctx context.Context, host provider.KVMHost, pool libvirt.StoragePool, be backend, src imageSource) (string, error) {
unlock, err := p.lockImage(ctx, host.Name, src.volumeName())
if err != nil {
return "", err
}
defer unlock()
log := p.logger(ctx).WithValues("host", host.Name, "volume", src.volumeName())
path, err := cachedImage(ctx, log, pool, be, src)
if err != nil || path != "" {
return path, err
}
vol, err := p.fetchImage(ctx, host, pool, be, src)
if err != nil {
return "", err
}
path, err = be.StorageVolGetPath(ctx, vol)
if err != nil {
return "", fmt.Errorf("reading the path of base volume %s: %w", vol.Name, err)
}
return path, nil
}
// cachedImage returns the base volume's path when the marker says the base
// is complete and its digest is the configured one (or none is configured).
// Anything else is cleared so the fetch starts from nothing, and "" is
// returned: a base without a marker is an upload that was cut off; a marker
// without a base, or one whose content is not a digest, is debris; a marker
// with a different digest means the catalogue entry changed.
func cachedImage(ctx context.Context, log logr.Logger, pool libvirt.StoragePool, be backend, src imageSource) (string, error) {
name, markerName := src.volumeName(), src.markerName()
marker, err := be.StorageVolLookupByName(ctx, pool, markerName)
switch {
case hasCode(err, libvirt.ErrNoStorageVol):
return "", deleteVolumeIfPresent(ctx, be, pool, name)
case err != nil:
return "", fmt.Errorf("looking up marker %s: %w", markerName, err)
}
digest, err := readMarker(ctx, be, marker)
if err != nil {
return "", err
}
switch {
case !hexDigest.MatchString(digest):
log.V(1).Info("base image marker holds no digest; refetching")
case src.SHA256 != "" && !strings.EqualFold(digest, src.SHA256):
// Refusing rather than replacing: every overlay on this host names
// the base as its backing file, and libvirt will unlink it under
// running domains without a word.
return "", permanentf("base image %s is cached with sha256 %s but %s is configured; add the image under a new key and move Proxies to it, or delete the Proxies using it and evict the base by hand",
name, digest, strings.ToLower(src.SHA256))
default:
base, err := be.StorageVolLookupByName(ctx, pool, name)
switch {
case hasCode(err, libvirt.ErrNoStorageVol):
return "", deleteVolumeIfPresent(ctx, be, pool, markerName)
case err != nil:
return "", fmt.Errorf("looking up base volume %s: %w", name, err)
}
path, err := be.StorageVolGetPath(ctx, base)
if err != nil {
return "", fmt.Errorf("reading the path of base volume %s: %w", name, err)
}
return path, nil
}
return "", errors.Join(deleteVolumeIfPresent(ctx, be, pool, markerName), deleteVolumeIfPresent(ctx, be, pool, name))
}
// readMarker returns the digest a marker volume holds, lowercased and
// stripped of padding. The volume is 64 bytes; the read is capped anyway.
func readMarker(ctx context.Context, be backend, marker libvirt.StorageVol) (string, error) {
w := &cappedWriter{limit: markerReadCap}
if err := be.StorageVolDownload(ctx, marker, w, 0, 0, 0); err != nil {
return "", fmt.Errorf("reading marker %s: %w", marker.Name, err)
}
return strings.ToLower(strings.TrimRight(string(w.buf), "\x00 \r\n\t")), nil
}
// cappedWriter keeps the first limit bytes and accepts the rest without
// storing them, so a download into it is bounded whatever its length.
type cappedWriter struct {
limit int
buf []byte
}
func (w *cappedWriter) Write(b []byte) (int, error) {
if room := w.limit - len(w.buf); room > 0 {
w.buf = append(w.buf, b[:min(room, len(b))]...)
}
return len(b), nil
}
// deleteVolumeIfPresent removes the named volume; one that is already gone
// is success.
func deleteVolumeIfPresent(ctx context.Context, be backend, pool libvirt.StoragePool, name string) error {
vol, err := be.StorageVolLookupByName(ctx, pool, name)
if hasCode(err, libvirt.ErrNoStorageVol) {
return nil
}
if err != nil {
return fmt.Errorf("looking up volume %s: %w", name, err)
}
if err := be.StorageVolDelete(ctx, vol, 0); err != nil && !hasCode(err, libvirt.ErrNoStorageVol) {
return fmt.Errorf("deleting volume %s: %w", name, err)
}
return nil
}
// fetchImage streams the image from its URL into a new base volume and then
// writes the marker, under the kvm2.download span (spec entry 17). The
// upload goes over a connection of its own (spec entry 18): abandoning a
// call kills the connection it ran on, and a 20-minute upload must not die
// because a poll for another Proxy was cancelled on the pooled one. The
// marker is written last, on the pooled connection, after
// StoragePoolRefresh made libvirt read the file's real format; its presence
// is what marks the base complete.
//
// Every exit between the base volume's creation and the marker deletes the
// base volume (best effort): the next attempt would delete it anyway on
// finding no marker, and removing it now frees the space at once.
func (p *Provider) fetchImage(ctx context.Context, host provider.KVMHost, pool libvirt.StoragePool, be backend, src imageSource) (libvirt.StorageVol, error) {
name := src.volumeName()
ctx, cancel := context.WithTimeout(ctx, defaultBudgets.transfer)
defer cancel()
ctx, span := tracing.StartSpan(ctx, tracing.Tracer(p.tracing...), "kvm2.download",
trace.WithSpanKind(trace.SpanKindClient),
trace.WithAttributes(
attribute.String("kvm2.host", host.Name),
attribute.String("kvm2.volume", name),
attribute.String("kvm2.url_host", urlHost(src.URL)),
attribute.Bool("kvm2.sha256_configured", src.SHA256 != ""),
))
defer span.End()
// Taken after the span starts, so the lines carry its ID.
log := p.logger(ctx).WithValues("host", host.Name, "volume", name)
start := time.Now()
// The body runs out of time before the surrounding budget does, so a
// mirror that stalls mid-image ends the upload through its reader
// rather than through the killed socket the budget would leave.
httpCtx, cancelHTTP := context.WithTimeout(ctx, defaultBudgets.transfer-transferGrace)
defer cancelHTTP()
resp, err := p.do(httpCtx, http.MethodGet, src.URL)
if err == nil {
err = checkResponse(src.URL, resp)
}
if err != nil {
span.SetStatus(codes.Error, err.Error())
return libvirt.StorageVol{}, err
}
defer func() { _ = resp.Body.Close() }()
size := resp.ContentLength
span.SetAttributes(attribute.Int64("kvm2.bytes", size))
log.V(1).Info("fetching base image into host pool", "url", src.URL, "bytes", size)
vol, digest, err := p.uploadImage(ctx, host, pool, src, resp.Body, size)
if err != nil {
// uploadImage has returned, so its dedicated connection is closed
// and libvirtd has let go of the stream before the file goes.
p.discardVolume(ctx, log, be, pool, name)
span.SetStatus(codes.Error, err.Error())
return libvirt.StorageVol{}, err
}
if err := be.StoragePoolRefresh(ctx, pool, 0); err != nil {
err = fmt.Errorf("refreshing storage pool %s after the upload: %w", pool.Name, err)
p.discardVolume(ctx, log, be, pool, name)
span.SetStatus(codes.Error, err.Error())
return libvirt.StorageVol{}, err
}
if err := writeMarker(ctx, be, pool, src.markerName(), digest); err != nil {
p.discardVolume(ctx, log, be, pool, name)
span.SetStatus(codes.Error, err.Error())
return libvirt.StorageVol{}, err
}
log.V(1).Info("base image cached", "bytes", size, "duration", time.Since(start), "sha256", digest)
return vol, nil
}
// uploadImage creates the base volume and streams body into it over a
// dedicated connection with a running sha256, then checks length and
// digest. The error says why: a body that ended early is Transient, a
// digest that differs from the configured one is Permanent. Discarding
// the volume is the caller's, once this connection is closed.
func (p *Provider) uploadImage(ctx context.Context, host provider.KVMHost, pool libvirt.StoragePool, src imageSource, body io.Reader, size int64) (libvirt.StorageVol, string, error) {
name := src.volumeName()
xmlDoc, err := buildVolumeXML(volumeSpec{Name: name, CapacityBytes: size, Format: src.Format})
if err != nil {
return libvirt.StorageVol{}, "", err
}
fresh, err := p.hosts.dialFresh(ctx, host)
if err != nil {
return libvirt.StorageVol{}, "", fmt.Errorf("dialling host %s for the image upload: %w", host.Name, err)
}
defer func() { _ = fresh.Close() }()
vol, err := fresh.StorageVolCreateXML(ctx, pool, xmlDoc, 0)
if err != nil {
return libvirt.StorageVol{}, "", fmt.Errorf("creating base volume %s: %w", name, err)
}
hash := sha256.New()
counted := &countingReader{r: io.TeeReader(body, hash)}
uploadErr := fresh.StorageVolUpload(ctx, vol, counted, 0, uint64(size), 0)
read, readErr := counted.state()
var digest string
if uploadErr == nil {
// Only then has the upload stopped reading; an abandoned call's
// goroutine may still be writing into the hash.
digest = hex.EncodeToString(hash.Sum(nil))
}
switch {
case readErr != nil:
err = fmt.Errorf("%w: image %s: the response ended after %d of %d bytes: %w", provider.ErrTransient, src.URL, read, size, readErr)
case uploadErr != nil:
err = fmt.Errorf("uploading base volume %s: %w", name, uploadErr)
case read != size:
err = transientf("image %s: short read, %d of %d bytes", src.URL, read, size)
case src.SHA256 != "" && !strings.EqualFold(digest, src.SHA256):
err = permanentf("image %s: sha256 %s does not match the configured %s", src.URL, digest, strings.ToLower(src.SHA256))
}
if err != nil {
return libvirt.StorageVol{}, "", err
}
return vol, digest, nil
}
// countingReader counts what the upload consumed and keeps the first
// non-EOF error the source returned. The adapter can return from an
// abandoned upload while go-libvirt still reads, so the fields are guarded.
type countingReader struct {
r io.Reader
mu sync.Mutex
n int64
err error
}
func (c *countingReader) Read(b []byte) (int, error) {
n, err := c.r.Read(b)
c.mu.Lock()
c.n += int64(n)
if err != nil && err != io.EOF && c.err == nil {
c.err = err
}
c.mu.Unlock()
return n, err
}
func (c *countingReader) state() (int64, error) {
c.mu.Lock()
defer c.mu.Unlock()
return c.n, c.err
}
// discardVolume removes a base volume a failed fetch left behind. It runs
// with ctx's values but not its cancellation: the fetch may have failed
// because ctx ended, and the adapter refuses calls on a finished ctx. The
// budget bounds it on its own. Failure is logged, not returned: the next
// attempt finds the volume without a marker and deletes it then.
func (p *Provider) discardVolume(ctx context.Context, log logr.Logger, be backend, pool libvirt.StoragePool, name string) {
ctx, cancel := context.WithTimeout(context.WithoutCancel(ctx), defaultBudgets.rpc)
defer cancel()
if err := deleteVolumeIfPresent(ctx, be, pool, name); err != nil {
log.V(1).Info("incomplete base volume left behind; the next attempt removes it", "error", err.Error())
}
}
// writeMarker creates the raw marker volume and uploads the digest into it.
// A marker that could not be filled is removed again: an empty marker next
// to a complete base would read as debris and cost a refetch.
func writeMarker(ctx context.Context, be backend, pool libvirt.StoragePool, name, digest string) error {
xmlDoc, err := buildVolumeXML(volumeSpec{Name: name, CapacityBytes: markerCapacityBytes, Format: provider.KVMFormatRaw})
if err != nil {
return err
}
vol, err := be.StorageVolCreateXML(ctx, pool, xmlDoc, 0)
if err != nil {
return fmt.Errorf("creating marker %s: %w", name, err)
}
if err := be.StorageVolUpload(ctx, vol, strings.NewReader(digest), 0, uint64(len(digest)), 0); err != nil {
_ = deleteVolumeIfPresent(ctx, be, pool, name)
return fmt.Errorf("writing marker %s: %w", name, err)
}
return nil
}
// do issues one request bound to ctx. Accept-Encoding is fixed to identity
// because a transparently decompressed body reports no Content-Length. A
// transport error is Transient.
func (p *Provider) do(ctx context.Context, method, rawURL string) (*http.Response, error) {
req, err := http.NewRequestWithContext(ctx, method, rawURL, nil)
if err != nil {
return nil, permanentf("image %s: %v", rawURL, err)
}
req.Header.Set("Accept-Encoding", "identity")
resp, err := p.http.Do(req)
if err != nil {
return nil, fmt.Errorf("%w: fetching image %s: %w", provider.ErrTransient, rawURL, err)
}
return resp, nil
}
// checkResponse classifies the response at the source (spec entry 13) and
// closes its body when it is unusable. 404, 403 and 410, and every other
// 4xx, mean the catalogue entry is wrong; 5xx and 429 are worth a retry. A
// missing Content-Length is Permanent: the volume's capacity must be
// declared before the upload, and a mirror that only chunks is one the
// catalogue must not point at.
func checkResponse(rawURL string, resp *http.Response) error {
var err error
code := resp.StatusCode
switch {
case code >= 200 && code < 300:
case code == http.StatusTooManyRequests || code >= 500:
err = transientf("image %s: HTTP %s", rawURL, resp.Status)
case code >= 400:
err = permanentf("image %s: HTTP %s", rawURL, resp.Status)
default:
err = transientf("image %s: unexpected HTTP %s", rawURL, resp.Status)
}
if err == nil && resp.ContentLength < 0 {
err = permanentf("image %s: response has no Content-Length; the volume capacity must be known before upload", rawURL)
}
if err == nil && resp.ContentLength == 0 {
err = permanentf("image %s: response is empty (Content-Length 0)", rawURL)
}
if err != nil {
_ = resp.Body.Close()
}
return err
}
// urlHost is the only part of the URL a span carries (contract §9).
func urlHost(rawURL string) string {
u, err := url.Parse(rawURL)
if err != nil {
return ""
}
return u.Host
}
// imageSizeIfUncached returns the image's Content-Length when the base is
// not cached on the host, and 0 when the marker is present, for the quota
// precheck of spec entry 13. It never downloads: HEAD first, and a GET whose
// body is closed at once when HEAD answers anything but 2xx, since servers
// refuse HEAD with 405, 403 or 501 alike; the GET's status is what gets
// classified. A marker is taken at face value here; ensureImage decides
// whether it still matches.
func (p *Provider) imageSizeIfUncached(ctx context.Context, be backend, pool libvirt.StoragePool, src imageSource) (int64, error) {
_, err := be.StorageVolLookupByName(ctx, pool, src.markerName())
switch {
case err == nil:
return 0, nil
case !hasCode(err, libvirt.ErrNoStorageVol):
return 0, fmt.Errorf("looking up marker %s: %w", src.markerName(), err)
}
ctx, cancel := context.WithTimeout(ctx, defaultBudgets.rpc)
defer cancel()
resp, err := p.do(ctx, http.MethodHead, src.URL)
if err != nil {
return 0, err
}
_ = resp.Body.Close()
if resp.StatusCode < 200 || resp.StatusCode >= 300 {
if resp, err = p.do(ctx, http.MethodGet, src.URL); err != nil {
return 0, err
}
_ = resp.Body.Close()
}
if err := checkResponse(src.URL, resp); err != nil {
return 0, err
}
return resp.ContentLength, nil
}
// checkPoolSpace is the precheck of spec entry 13: the pool must be running
// and have needed bytes available, or Create returns ErrQuotaExceeded and
// the reconciler waits QuotaRetry instead of failing the volume creation.
func checkPoolSpace(ctx context.Context, be backend, pool libvirt.StoragePool, poolName string, needed uint64) error {
state, _, _, available, err := be.StoragePoolGetInfo(ctx, pool)
if err != nil {
return fmt.Errorf("reading storage pool %s: %w", poolName, err)
}
if s := libvirt.StoragePoolState(state); s != libvirt.StoragePoolRunning {
return permanentf("storage pool %s is not running (state %s)", poolName, s)
}
if available < needed {
return fmt.Errorf("%w: storage pool %s has %d bytes available, %d needed", provider.ErrQuotaExceeded, poolName, available, needed)
}
return nil
}
// neededBytes is what one Create commits to the pool: the image when it is
// not cached yet (0 otherwise), the overlay's capacity and the seed reserve.
func neededBytes(imageBytes int64, diskSizeGB int64) uint64 {
return uint64(max(imageBytes, 0)) + uint64(max(diskSizeGB, 0))<<30 + seedReserveBytes
}
internal/provider/kvm2/image_test.go
+1007
View File
File diff suppressed because it is too large Load Diff
internal/provider/kvm2/kvm2.go
+195
View File
@@ -0,0 +1,195 @@
// Package kvm2 implements the provider contract on libvirt/KVM hypervisors
// reached over SSH: the host's libvirt unix socket is forwarded through the
// SSH connection and driven with go-libvirt's RPC client. The design is in
// docs/kvm2-provider-spec.md.
package kvm2
import (
"context"
"fmt"
"net/http"
"os"
"time"
"github.com/go-logr/logr"
logf "sigs.k8s.io/controller-runtime/pkg/log"
"gitea.home.hrajfrisbee.cz/kacerr/egress-proxies-operator/internal/provider"
"gitea.home.hrajfrisbee.cz/kacerr/egress-proxies-operator/internal/provider/keyedlock"
"gitea.home.hrajfrisbee.cz/kacerr/egress-proxies-operator/internal/tracing"
)
const (
defaultSocket = "/var/run/libvirt/libvirt-sock"
defaultPool = "default"
defaultDiskSizeGB = 10
)
// Provider is one configured kvm2 entry: a set of hosts sharing one SSH key
// and one image and machine-type catalogue.
type Provider struct {
name string
cfg provider.KVMConfig
now func() time.Time
hosts *hostTable
http *http.Client
tracing []tracing.Option
// runningWait caps the wait for a started domain to report running,
// polled every runningPoll (spec entry 18).
runningWait time.Duration
runningPoll time.Duration
// imageLocks serialises image fetches per (host, volume), spec entry 19.
imageLocks *keyedlock.Locks
// nameLocks serialises Create and Delete per instance name, spec entry
// 19, so two Deletes never race on destroy and undefine and a Delete
// cannot interleave with a Create's steps.
nameLocks *keyedlock.Locks
}
// providerOption adjusts what newProvider builds; tests inject an HTTP
// client bound to a test server, a span recorder and a shorter running
// wait.
type providerOption func(*Provider)
func withHTTPClient(c *http.Client) providerOption {
return func(p *Provider) { p.http = c }
}
func withTracing(opts ...tracing.Option) providerOption {
return func(p *Provider) { p.tracing = opts }
}
// withRunningWait shrinks the poll interval and the cap of the wait for a
// domain to run.
func withRunningWait(poll, cap time.Duration) providerOption {
return func(p *Provider) { p.runningPoll, p.runningWait = poll, cap }
}
var _ provider.Provider = (*Provider)(nil)
// New builds a Provider from pc.KVM2. It never dials a host: the credential
// files are only checked for existence, and a missing one is a warning, not
// an error, because the Secret mount is optional and may land later.
func New(ctx context.Context, pc provider.ProviderConfig) (provider.Provider, error) {
if !provider.IsKVMIdentifier(pc.Name) {
return nil, permanentf("provider name %q: kvm2 names its cached base images after the entry, so the name must be letters, digits, dots, dashes or underscores", pc.Name)
}
cfg := provider.KVMConfig{}
if pc.KVM2 != nil {
cfg = *pc.KVM2
}
cfg = withDefaults(cfg)
p := newProvider(pc.Name, cfg, newLibvirtDialer(pc.Name, cfg))
log := p.logger(ctx)
if _, err := os.Stat(p.cfg.SSHKeyFile); err != nil {
log.Info("WARNING: ssh key file is not readable; every call fails until it is", "path", p.cfg.SSHKeyFile, "reason", err.Error())
}
switch {
case p.cfg.InsecureIgnoreHostKey:
log.Info("WARNING: insecureIgnoreHostKey is set; host keys are not verified")
default:
if _, err := os.Stat(p.cfg.KnownHostsFile); err != nil {
log.Info("WARNING: known_hosts file is not readable; every call fails until it is", "path", p.cfg.KnownHostsFile, "reason", err.Error())
}
}
return p, nil
}
// newProvider is the constructor tests use: d hands out the backends, the
// production one being newLibvirtDialer. Without options the image
// downloads use http.DefaultClient (system CA roots, no client timeout: the
// per-request ctx carries the transfer budget).
func newProvider(name string, cfg provider.KVMConfig, d dialer, opts ...providerOption) *Provider {
p := &Provider{
name: name,
cfg: withDefaults(cfg),
now: time.Now,
hosts: newHostTable(d),
http: http.DefaultClient,
runningWait: defaultRunningWait,
runningPoll: defaultRunningPoll,
imageLocks: keyedlock.New(),
nameLocks: keyedlock.New(),
}
for _, opt := range opts {
opt(p)
}
if p.http == nil {
p.http = http.DefaultClient
}
return p
}
// withDefaults returns cfg with every optional field filled in, so the rest
// of the package never re-checks for an empty value. The hosts and images
// are copied, not shared with the caller.
func withDefaults(cfg provider.KVMConfig) provider.KVMConfig {
if cfg.DiskSizeGB == 0 {
cfg.DiskSizeGB = defaultDiskSizeGB
}
hosts := make([]provider.KVMHost, len(cfg.Hosts))
for i, h := range cfg.Hosts {
if h.Socket == "" {
h.Socket = defaultSocket
}
if h.Pool == "" {
h.Pool = defaultPool
}
hosts[i] = h
}
cfg.Hosts = hosts
images := make(map[string]provider.KVMImage, len(cfg.Images))
for k, img := range cfg.Images {
if img.Format == "" {
img.Format = provider.KVMFormatQcow2
}
if img.Firmware == "" {
img.Firmware = provider.KVMFirmwareUEFI
}
images[k] = img
}
cfg.Images = images
return cfg
}
func (p *Provider) logger(ctx context.Context) logr.Logger {
return logf.FromContext(ctx).WithName("kvm2").WithValues("provider", p.name)
}
// host resolves a placement.zone or a providerID's host segment to a
// configured host. An empty zone is allowed only when exactly one host is
// configured.
func (p *Provider) host(zone string) (provider.KVMHost, error) {
if zone == "" {
if len(p.cfg.Hosts) == 1 {
return p.cfg.Hosts[0], nil
}
return provider.KVMHost{}, permanentf("placement.zone is required when kvm2.hosts lists more than one host")
}
for _, h := range p.cfg.Hosts {
if h.Name == zone {
return h, nil
}
}
return provider.KVMHost{}, permanentf("host %q is not in kvm2.hosts", zone)
}
// wrapErr is the single exit for errors: it classifies err and wraps it with
// the configured provider name, op and id.
func (p *Provider) wrapErr(op, id string, err error) error {
return provider.Wrap(classify(op, err), op, p.name, id, err)
}
// lockName takes the per-name lock of spec entry 19, or gives up with ctx,
// so a caller whose ctx ends while another caller's Create is downloading
// an image returns promptly.
func (p *Provider) lockName(ctx context.Context, name string) (func(), error) {
release, err := p.nameLocks.Acquire(ctx, name)
if err != nil {
return nil, fmt.Errorf("%w: waiting for the lock on %s: %w", provider.ErrTransient, name, err)
}
return release, nil
}
internal/provider/kvm2/kvm2_integration_test.go
+590
View File
@@ -0,0 +1,590 @@
//go:build integration
package kvm2
import (
"context"
"crypto/rand"
"encoding/hex"
"errors"
"fmt"
"net"
"os"
"strconv"
"strings"
"sync"
"testing"
"time"
"github.com/digitalocean/go-libvirt"
"github.com/go-logr/logr"
"github.com/go-logr/logr/funcr"
sdktrace "go.opentelemetry.io/otel/sdk/trace"
"go.opentelemetry.io/otel/sdk/trace/tracetest"
"k8s.io/apimachinery/pkg/types"
logf "sigs.k8s.io/controller-runtime/pkg/log"
"gitea.home.hrajfrisbee.cz/kacerr/egress-proxies-operator/internal/provider"
"gitea.home.hrajfrisbee.cz/kacerr/egress-proxies-operator/internal/tracing"
)
// The real-hypervisor test of spec entry 21. It runs only with
// -tags=integration and skips itself unless KVM2_TEST_* names a host; see
// docs/testing.md for the variables and what one run asserts.
const (
integrationProvider = "kvm2-integration"
integrationHost = "kvm2-test"
integrationImage = "test-image"
integrationMachine = "small"
integrationPort = 3128
// integrationCloudInit is the user-data the VM boots with. Nothing
// asserts on the guest running it; it exists so the sentinel check can
// prove that no span attribute carries the cloud-init text.
integrationCloudInit = "#cloud-config\nruncmd:\n - echo kvm2-integration\n"
defaultIntegrationTimeout = 20 * time.Minute
defaultIntegrationLogLvl = 1
getPollInterval = 5 * time.Second
cleanupBudget = 5 * time.Minute
)
type integrationEnv struct {
host, user, sshKey, knownHosts, bridge, imageURL string
imageSHA256, pool, socket string
timeout time.Duration
keep bool
logLevel int
}
// loadIntegrationEnv reads the KVM2_TEST_* variables, skipping the test
// under -short or when a required one is missing.
func loadIntegrationEnv(t *testing.T) integrationEnv {
t.Helper()
if testing.Short() {
t.Skip("real-hypervisor test skipped under -short")
}
env := integrationEnv{timeout: defaultIntegrationTimeout, logLevel: defaultIntegrationLogLvl}
required := []struct {
name string
dst *string
}{
{"KVM2_TEST_HOST", &env.host},
{"KVM2_TEST_USER", &env.user},
{"KVM2_TEST_SSH_KEY", &env.sshKey},
{"KVM2_TEST_BRIDGE", &env.bridge},
{"KVM2_TEST_IMAGE_URL", &env.imageURL},
}
for _, r := range required {
v := os.Getenv(r.name)
if v == "" {
t.Skipf("%s is not set; the kvm2 integration test needs a real hypervisor (see docs/testing.md)", r.name)
}
*r.dst = v
}
env.knownHosts = os.Getenv("KVM2_TEST_KNOWN_HOSTS")
env.imageSHA256 = os.Getenv("KVM2_TEST_IMAGE_SHA256")
env.pool = os.Getenv("KVM2_TEST_POOL")
env.socket = os.Getenv("KVM2_TEST_SOCKET")
env.keep = os.Getenv("KVM2_TEST_KEEP") != ""
if v := os.Getenv("KVM2_TEST_TIMEOUT"); v != "" {
d, err := time.ParseDuration(v)
if err != nil {
t.Fatalf("KVM2_TEST_TIMEOUT=%q: %v", v, err)
}
env.timeout = d
}
if v := os.Getenv("KVM2_TEST_LOG_LEVEL"); v != "" {
n, err := strconv.Atoi(v)
if err != nil || n < 0 {
t.Fatalf("KVM2_TEST_LOG_LEVEL=%q: want a non-negative integer", v)
}
env.logLevel = n
}
return env
}
// providerConfig is what the operator would load for this host: one host,
// one image, one machine type. Host keys are verified only when
// KVM2_TEST_KNOWN_HOSTS is set.
func (e integrationEnv) providerConfig() provider.ProviderConfig {
return provider.ProviderConfig{
Name: integrationProvider,
Type: "kvm2",
KVM2: &provider.KVMConfig{
SSHKeyFile: e.sshKey,
KnownHostsFile: e.knownHosts,
InsecureIgnoreHostKey: e.knownHosts == "",
Images: map[string]provider.KVMImage{
integrationImage: {URL: e.imageURL, SHA256: e.imageSHA256},
},
MachineTypes: map[string]provider.KVMMachineType{
integrationMachine: {VCPUs: 1, MemoryMiB: 1024},
},
Hosts: []provider.KVMHost{{
Name: integrationHost,
Address: e.host,
User: e.user,
Socket: e.socket,
Pool: e.pool,
Bridge: e.bridge,
}},
},
}
}
// integrationLog routes the provider's logr output into t.Log through funcr
// at the chosen verbosity, keeps every line for assertions, and strips the
// key/value pairs that are constant for the run (stated once up front) so
// each line stays short. Lines arriving after the test ended are dropped:
// t.Log panics then.
type integrationLog struct {
t *testing.T
mu sync.Mutex
drop []string
lines []string
done bool
}
func newIntegrationLog(t *testing.T, level int, drop ...string) (*integrationLog, logr.Logger) {
t.Helper()
sink := &integrationLog{t: t, drop: drop}
t.Cleanup(func() {
sink.mu.Lock()
defer sink.mu.Unlock()
sink.done = true
})
log := funcr.New(func(prefix, args string) { sink.write(prefix, args) }, funcr.Options{Verbosity: level})
return sink, log
}
// dropValue adds a rendered pair (`"key"="value"`) to strip from later lines.
func (l *integrationLog) dropValue(key, value string) {
l.mu.Lock()
defer l.mu.Unlock()
l.drop = append(l.drop, fmt.Sprintf("%q=%q", key, value))
}
func (l *integrationLog) write(prefix, args string) {
l.mu.Lock()
defer l.mu.Unlock()
l.lines = append(l.lines, args)
if l.done {
return
}
for _, d := range l.drop {
args = strings.ReplaceAll(args, " "+d, "")
}
l.t.Logf("%s %s %s", time.Now().Format("15:04:05.000"), prefix, args)
}
func (l *integrationLog) matching(substr string) []string {
l.mu.Lock()
defer l.mu.Unlock()
var out []string
for _, line := range l.lines {
if strings.Contains(line, substr) {
out = append(out, line)
}
}
return out
}
// newIntegrationProvider builds the provider the way New does, plus the span
// recorder's tracer on both places the package starts spans: the Provider
// (kvm2.download) and the dialer (kvm2.dial and, through observe, every
// kvm2.<Method>). newLibvirtDialer has no option for it, so the field is set
// directly; that is what makes the kvm2.* spans land in the same recorder
// as provider.WithTracing's provider.* spans.
func newIntegrationProvider(pc provider.ProviderConfig, tp *sdktrace.TracerProvider) *Provider {
cfg := withDefaults(*pc.KVM2)
opt := tracing.WithTracerProvider(tp)
d := newLibvirtDialer(pc.Name, cfg)
d.tracing = []tracing.Option{opt}
return newProvider(pc.Name, cfg, d, withTracing(opt))
}
// randomUID is a UUID-shaped string, so the run's domain never collides with
// a previous run's leftovers or with a real Proxy's.
func randomUID(t *testing.T) string {
t.Helper()
var b [16]byte
if _, err := rand.Read(b[:]); err != nil {
t.Fatalf("reading random bytes: %v", err)
}
h := hex.EncodeToString(b[:])
return h[0:8] + "-" + h[8:12] + "-" + h[12:16] + "-" + h[16:20] + "-" + h[20:32]
}
// stepDone logs one lifecycle step with its duration; the log is the record
// of how long a cold image download or a boot took on this host.
func stepDone(t *testing.T, step string, start time.Time) {
t.Helper()
t.Logf("step %s: %s", step, time.Since(start).Round(time.Millisecond))
}
// waitUntil polls cond every interval until it is true or ctx ends.
func waitUntil(ctx context.Context, t *testing.T, interval time.Duration, what string, cond func() bool) {
t.Helper()
ticker := time.NewTicker(interval)
defer ticker.Stop()
for !cond() {
select {
case <-ticker.C:
case <-ctx.Done():
t.Fatalf("gave up waiting for %s: %v", what, ctx.Err())
}
}
}
// domainCount lists every domain on the host over a dedicated connection.
func domainCount(ctx context.Context, t *testing.T, p *Provider, host provider.KVMHost) int {
t.Helper()
be, err := p.hosts.dialFresh(ctx, host)
if err != nil {
t.Fatalf("dialling %s to count domains: %v", host.Name, err)
}
defer func() { _ = be.Close() }()
doms, _, err := be.ConnectListAllDomains(ctx, 1, libvirt.ConnectListDomainsActive|libvirt.ConnectListDomainsInactive)
if err != nil {
t.Fatalf("listing domains on %s: %v", host.Name, err)
}
return len(doms)
}
// poolVolumes reports, for each name, whether the host's pool holds a volume
// of that name, over a dedicated connection. Any error other than
// ErrNoStorageVol fails the test.
func poolVolumes(ctx context.Context, t *testing.T, p *Provider, host provider.KVMHost, names ...string) map[string]bool {
t.Helper()
be, err := p.hosts.dialFresh(ctx, host)
if err != nil {
t.Fatalf("dialling %s to inspect the pool: %v", host.Name, err)
}
defer func() { _ = be.Close() }()
pool, err := be.StoragePoolLookupByName(ctx, host.Pool)
if err != nil {
t.Fatalf("looking up pool %s: %v", host.Pool, err)
}
out := make(map[string]bool, len(names))
for _, name := range names {
_, err := be.StorageVolLookupByName(ctx, pool, name)
switch {
case err == nil:
out[name] = true
case hasCode(err, libvirt.ErrNoStorageVol):
out[name] = false
default:
t.Fatalf("looking up volume %s in pool %s: %v", name, host.Pool, err)
}
}
return out
}
func TestKVM2_lifecycle(t *testing.T) {
env := loadIntegrationEnv(t)
pc := env.providerConfig()
sink, log := newIntegrationLog(t, env.logLevel,
fmt.Sprintf("%q=%q", "provider", integrationProvider),
fmt.Sprintf("%q=%q", "host", integrationHost))
ctx, cancel := context.WithTimeout(logf.IntoContext(context.Background(), log), env.timeout)
defer cancel()
t.Logf("provider=%s host=%s address=%s user=%s bridge=%s image=%s budget=%s log level %d; provider, host and name are left off every provider log line",
integrationProvider, integrationHost, env.host, env.user, env.bridge, env.imageURL, env.timeout, env.logLevel)
if env.knownHosts == "" {
t.Log("KVM2_TEST_KNOWN_HOSTS is not set: host keys are NOT verified for this run")
}
// 1. New returns without dialling.
start := time.Now()
built, err := New(ctx, pc)
elapsed := time.Since(start)
if err != nil {
t.Fatalf("New() error = %v", err)
}
if elapsed > 100*time.Millisecond {
t.Errorf("New() took %s, want under 100ms: it must not dial", elapsed)
}
if lines := sink.matching("libvirt host connected"); len(lines) > 0 {
t.Errorf("New() dialled the host before any call: %q", lines)
}
stepDone(t, "1 New", start)
// 2. Check reports the host Ready with details.
checker, ok := built.(provider.Checker)
if !ok {
t.Fatalf("New() returned %T, which is not a provider.Checker", built)
}
start = time.Now()
targets := checker.Check(ctx)
if len(targets) != 1 {
t.Fatalf("Check() returned %d targets, want 1: %+v", len(targets), targets)
}
ts := targets[0]
if !ts.Ready || ts.Reason != provider.ReasonReachable {
t.Fatalf("Check() = ready %v reason %q message %q, want Ready/%s (details %v)", ts.Ready, ts.Reason, ts.Message, provider.ReasonReachable, ts.Details)
}
for _, key := range []string{"hostname", "libvirtVersion", "poolState", "poolAvailableGiB"} {
if ts.Details[key] == "" {
t.Errorf("Check() details lack %q: %v", key, ts.Details)
}
}
if ts.Details["poolState"] != "running" {
t.Errorf("Check() poolState = %q, want running", ts.Details["poolState"])
}
t.Logf("host %s: hostname=%s libvirt=%s pool=%s (%s) available=%s GiB capacity=%s GiB latency=%s",
ts.Name, ts.Details["hostname"], ts.Details["libvirtVersion"], ts.Details["pool"], ts.Details["poolState"],
ts.Details["poolAvailableGiB"], ts.Details["poolCapacityGiB"], ts.Latency.Round(time.Millisecond))
stepDone(t, "2 Check", start)
// 3. Every later call goes through the tracing decorator, with the
// package's own spans recorded alongside.
sr := tracetest.NewSpanRecorder()
tp := sdktrace.NewTracerProvider(sdktrace.WithSpanProcessor(sr))
t.Cleanup(func() { _ = tp.Shutdown(context.Background()) })
p := newIntegrationProvider(pc, tp)
traced := provider.WithTracing(pc.Name, p, tracing.WithTracerProvider(tp))
host := p.cfg.Hosts[0]
// 4. Create.
uid := randomUID(t)
name := provider.NameFromUID(types.UID(uid))
sink.dropValue("name", name)
id := formatProviderID(integrationHost, name)
req := provider.CreateRequest{
Name: name,
UID: uid,
Namespace: "integration",
ProxyName: "kvm2-lifecycle",
Placement: provider.Placement{Zone: integrationHost, MachineType: integrationMachine, Image: integrationImage},
CloudInit: integrationCloudInit,
Port: integrationPort,
}
t.Logf("domain name %s, uid %s, id %s", name, uid, id)
deleted := false
if env.keep {
t.Logf("KVM2_TEST_KEEP is set: the VM stays behind. Clean up on the host with:\n"+
" virsh undefine --nvram %s\n virsh vol-delete --pool %s %s\n virsh vol-delete --pool %s %s",
name, host.Pool, overlayVolumeName(name), host.Pool, seedVolumeName(name))
} else {
t.Cleanup(func() {
if deleted {
return
}
cctx, ccancel := context.WithTimeout(logf.IntoContext(context.Background(), log), cleanupBudget)
defer ccancel()
if err := p.Delete(cctx, id); err != nil {
t.Errorf("cleanup Delete(%s) error = %v; remove the domain and its volumes by hand", id, err)
}
})
}
start = time.Now()
gotID, err := traced.Create(ctx, req)
if err != nil {
t.Fatalf("Create() error = %v", err)
}
if gotID != id {
t.Fatalf("Create() = %q, want %q", gotID, id)
}
createReturned := time.Now()
stepDone(t, "4 Create (includes the image download on a cold pool)", start)
// 5. Create again: same ID, nothing new on the host.
start = time.Now()
before := domainCount(ctx, t, p, host)
again, err := traced.Create(ctx, req)
if err != nil {
t.Fatalf("second Create() error = %v", err)
}
if again != id {
t.Errorf("second Create() = %q, want %q", again, id)
}
if after := domainCount(ctx, t, p, host); after != before {
t.Errorf("second Create() changed the host's domain count from %d to %d", before, after)
}
stepDone(t, "5 Create again", start)
// 6. Get reaches Running with an IPv4.
start = time.Now()
var inst *provider.Instance
waitUntil(ctx, t, getPollInterval, "the domain to be Running with an IPv4", func() bool {
var err error
inst, err = traced.Get(ctx, id)
if err != nil {
t.Fatalf("Get(%s) error = %v", id, err)
}
return inst.State == provider.StateRunning && inst.IP != ""
})
if inst.UID != uid {
t.Errorf("Get() UID = %q, want %q", inst.UID, uid)
}
if inst.CreatedAt.IsZero() {
t.Error("Get() CreatedAt is zero; the metadata mark did not round-trip")
} else if skew := time.Since(inst.CreatedAt).Abs(); skew > 5*time.Minute {
t.Errorf("Get() CreatedAt = %s, %s from now; want within a few minutes", inst.CreatedAt, skew)
}
t.Logf("domain running at %s, %s after Create returned (createdAt %s)", inst.IP, time.Since(createReturned).Round(time.Millisecond), inst.CreatedAt.Format(time.RFC3339))
stepDone(t, "6 Get until Running", start)
// 7. ListByTag contains it.
start = time.Now()
list, err := traced.ListByTag(ctx)
if err != nil {
t.Fatalf("ListByTag() error = %v", err)
}
found := false
for _, li := range list {
if li.ID == id {
found = true
if li.UID != uid {
t.Errorf("ListByTag() entry %s has UID %q, want %q", id, li.UID, uid)
}
}
}
if !found {
t.Errorf("ListByTag() = %d instances, none with ID %s", len(list), id)
}
stepDone(t, "7 ListByTag", start)
// 8. Reachability, logged only: this machine may not route to the bridge.
start = time.Now()
addr := net.JoinHostPort(inst.IP, "22")
if conn, err := net.DialTimeout("tcp", addr, 3*time.Second); err != nil {
t.Logf("TCP connect to %s failed (informational; the test host may not route to the bridge): %v", addr, err)
} else {
_ = conn.Close()
t.Logf("TCP connect to %s succeeded", addr)
}
stepDone(t, "8 TCP connect", start)
if env.keep {
t.Log("KVM2_TEST_KEEP is set: skipping Delete and the pool inspection")
} else {
// 9. Delete, Get is NotFound, Delete again.
start = time.Now()
if err := traced.Delete(ctx, id); err != nil {
t.Fatalf("Delete() error = %v", err)
}
deleted = true
if _, err := traced.Get(ctx, id); !errors.Is(err, provider.ErrNotFound) {
t.Errorf("Get() after Delete error = %v, want ErrNotFound", err)
}
if err := traced.Delete(ctx, id); err != nil {
t.Errorf("second Delete() error = %v, want nil", err)
}
stepDone(t, "9 Delete, Get, Delete", start)
// 10. The pool keeps the shared base image and its marker, and holds
// no per-VM volume.
start = time.Now()
base := baseVolumeName(integrationProvider, integrationImage, provider.KVMFormatQcow2)
marker := markerVolumeName(base)
overlay, seed := overlayVolumeName(name), seedVolumeName(name)
present := poolVolumes(ctx, t, p, host, base, marker, overlay, seed)
for _, want := range []string{base, marker} {
if !present[want] {
t.Errorf("pool %s lacks %s after Delete; the base image cache must stay", host.Pool, want)
}
}
for _, gone := range []string{overlay, seed} {
if present[gone] {
t.Errorf("pool %s still holds %s after Delete", host.Pool, gone)
}
}
stepDone(t, "10 pool inspection", start)
}
// 11. Spans: the package's spans are children of the decorator's.
assertLifecycleSpans(t, sr.Ended())
}
// assertLifecycleSpans checks that at least one provider.create span has a
// kvm2.DomainDefineXML and a kvm2.dial child (the first Create dials the
// pooled connection and defines the domain), and that no attribute of any
// span carries the cloud-init text.
func assertLifecycleSpans(t *testing.T, spans []sdktrace.ReadOnlySpan) {
t.Helper()
var creates []sdktrace.ReadOnlySpan
for _, s := range spans {
if s.Name() == "provider.create" {
creates = append(creates, s)
}
}
if len(creates) == 0 {
t.Fatalf("no provider.create span among %d spans: %v", len(spans), spanNames(spans))
}
satisfied := false
for _, create := range creates {
parent := create.SpanContext().SpanID()
var children, define, dial int
for _, s := range spans {
if s.Parent().SpanID() != parent || !strings.HasPrefix(s.Name(), "kvm2.") {
continue
}
children++
switch s.Name() {
case "kvm2.DomainDefineXML":
define++
case "kvm2.dial":
dial++
}
}
t.Logf("provider.create span %s: %d kvm2.* children (%d DomainDefineXML, %d dial)", parent, children, define, dial)
if define > 0 && dial > 0 {
satisfied = true
}
}
if !satisfied {
t.Errorf("no provider.create span has both a kvm2.DomainDefineXML and a kvm2.dial child; spans: %v", spanNames(spans))
}
for _, s := range spans {
for _, a := range s.Attributes() {
v := a.Value.Emit()
if strings.Contains(v, integrationCloudInit) || strings.Contains(v, "#cloud-config") {
t.Errorf("span %s attribute %s carries the cloud-init text", s.Name(), a.Key)
}
}
}
}
// TestKVM2Check_realHost runs the health check against the host and creates
// nothing.
func TestKVM2Check_realHost(t *testing.T) {
env := loadIntegrationEnv(t)
pc := env.providerConfig()
_, log := newIntegrationLog(t, env.logLevel,
fmt.Sprintf("%q=%q", "provider", integrationProvider),
fmt.Sprintf("%q=%q", "host", integrationHost))
ctx, cancel := context.WithTimeout(logf.IntoContext(context.Background(), log), env.timeout)
defer cancel()
if env.knownHosts == "" {
t.Log("KVM2_TEST_KNOWN_HOSTS is not set: host keys are NOT verified for this run")
}
built, err := New(ctx, pc)
if err != nil {
t.Fatalf("New() error = %v", err)
}
checker, ok := built.(provider.Checker)
if !ok {
t.Fatalf("New() returned %T, which is not a provider.Checker", built)
}
targets := checker.Check(ctx)
if len(targets) != 1 {
t.Fatalf("Check() returned %d targets, want 1: %+v", len(targets), targets)
}
ts := targets[0]
if !ts.Ready || ts.Reason != provider.ReasonReachable {
t.Fatalf("Check() = ready %v reason %q message %q, want Ready/%s (details %v)", ts.Ready, ts.Reason, ts.Message, provider.ReasonReachable, ts.Details)
}
if ts.Details["poolState"] != "running" {
t.Errorf("Check() poolState = %q, want running", ts.Details["poolState"])
}
t.Logf("host %s (%s): hostname=%s libvirt=%s pool=%s (%s) available=%s GiB latency=%s",
ts.Name, ts.Address, ts.Details["hostname"], ts.Details["libvirtVersion"], ts.Details["pool"], ts.Details["poolState"],
ts.Details["poolAvailableGiB"], ts.Latency.Round(time.Millisecond))
}
internal/provider/kvm2/kvm2_test.go
+280
View File
@@ -0,0 +1,280 @@
package kvm2
import (
"context"
"errors"
"strings"
"sync"
"testing"
"time"
"github.com/go-logr/logr/funcr"
logf "sigs.k8s.io/controller-runtime/pkg/log"
"gitea.home.hrajfrisbee.cz/kacerr/egress-proxies-operator/internal/provider"
)
// captureContext collects funcr output at V(2). The sink is locked because
// Check logs from one goroutine per host.
func captureContext() (context.Context, *[]string) {
lines := &[]string{}
var mu sync.Mutex
log := funcr.New(func(prefix, args string) {
mu.Lock()
defer mu.Unlock()
*lines = append(*lines, prefix+" "+args)
}, funcr.Options{Verbosity: 2})
return logf.IntoContext(context.Background(), log), lines
}
func twoHosts() provider.KVMConfig {
return provider.KVMConfig{
SSHKeyFile: "/nonexistent/kvm2/id_ed25519",
KnownHostsFile: "/nonexistent/kvm2/known_hosts",
MachineTypes: map[string]provider.KVMMachineType{"small": {VCPUs: 1, MemoryMiB: 1024}},
Hosts: []provider.KVMHost{
{Name: "beelink", Address: "beelink:22", User: "virt", Bridge: "br0"},
{Name: "lab2", Address: "lab2:22", User: "virt", Bridge: "br1", Socket: "/run/s", Pool: "fast"},
},
}
}
func TestNew_doesNotDialAndWarnsOnMissingFiles(t *testing.T) {
t.Parallel()
ctx, lines := captureContext()
cfg := twoHosts()
cfg.Hosts[0].Address = "192.0.2.1:22"
start := time.Now()
p, err := New(ctx, provider.ProviderConfig{Name: "kvm2-home", Type: "kvm2", KVM2: &cfg})
elapsed := time.Since(start)
if err != nil {
t.Fatalf("New() error = %v, want nil", err)
}
if p == nil {
t.Fatal("New() = nil provider")
}
if elapsed > 100*time.Millisecond {
t.Errorf("New() took %v, want under 100ms (it must not dial)", elapsed)
}
var warnings int
for _, l := range *lines {
if strings.Contains(l, "WARNING:") {
warnings++
}
if !strings.Contains(l, `"provider"="kvm2-home"`) {
t.Errorf("log line lacks the provider value: %s", l)
}
}
if warnings != 2 {
t.Errorf("got %d WARNING: lines, want 2 (key and known_hosts):\n%s", warnings, strings.Join(*lines, "\n"))
}
}
func TestNew_insecureIgnoreHostKeyWarns(t *testing.T) {
t.Parallel()
ctx, lines := captureContext()
cfg := twoHosts()
cfg.KnownHostsFile = ""
cfg.InsecureIgnoreHostKey = true
if _, err := New(ctx, provider.ProviderConfig{Name: "kvm2-home", Type: "kvm2", KVM2: &cfg}); err != nil {
t.Fatalf("New() error = %v, want nil", err)
}
found := false
for _, l := range *lines {
if strings.Contains(l, "WARNING:") && strings.Contains(l, "host keys are not verified") {
found = true
}
if strings.Contains(l, "known_hosts") {
t.Errorf("known_hosts warning emitted although host keys are ignored: %s", l)
}
}
if !found {
t.Errorf("no insecureIgnoreHostKey warning in:\n%s", strings.Join(*lines, "\n"))
}
}
func TestNew_nilBlockDoesNotPanic(t *testing.T) {
t.Parallel()
ctx, _ := captureContext()
p, err := New(ctx, provider.ProviderConfig{Name: "kvm2-empty", Type: "kvm2"})
if err != nil {
t.Fatalf("New() error = %v, want nil", err)
}
kp, ok := p.(*Provider)
if !ok {
t.Fatalf("New() = %T, want *Provider", p)
}
if kp.name != "kvm2-empty" || kp.cfg.DiskSizeGB != defaultDiskSizeGB || kp.now == nil {
t.Errorf("provider = %+v, want name kvm2-empty, default disk size and a clock", kp)
}
}
// TestNew_rejectsUnsafeEntryName guards the one config value that reaches a
// volume name without passing the kvm validator: the provider entry's own
// name, which is checked only for being non-empty and unique.
func TestNew_rejectsUnsafeEntryName(t *testing.T) {
t.Parallel()
ctx, _ := captureContext()
_, err := New(ctx, provider.ProviderConfig{Name: "kvm2/home", Type: "kvm2"})
if !errors.Is(err, provider.ErrPermanent) || !strings.Contains(err.Error(), "kvm2/home") {
t.Fatalf("New() error = %v, want ErrPermanent naming the entry", err)
}
}
func TestHost_table(t *testing.T) {
t.Parallel()
tests := []struct {
name string
hosts []provider.KVMHost
zone string
wantName string
wantErr string
}{
{
name: "one host and empty zone selects it",
hosts: twoHosts().Hosts[:1],
zone: "",
wantName: "beelink",
},
{
name: "several hosts and empty zone is permanent",
hosts: twoHosts().Hosts,
zone: "",
wantErr: "placement.zone is required when kvm2.hosts lists more than one host",
},
{
name: "known name",
hosts: twoHosts().Hosts,
zone: "lab2",
wantName: "lab2",
},
{
name: "unknown name is permanent",
hosts: twoHosts().Hosts,
zone: "nas9",
wantErr: `host "nas9" is not in kvm2.hosts`,
},
{
name: "no hosts at all is permanent",
hosts: nil,
zone: "",
wantErr: "placement.zone is required",
},
}
for _, tc := range tests {
t.Run(tc.name, func(t *testing.T) {
t.Parallel()
p := newProvider("kvm2-home", provider.KVMConfig{Hosts: tc.hosts}, &stubDialer{})
h, err := p.host(tc.zone)
if tc.wantErr != "" {
if err == nil {
t.Fatalf("host(%q) = %+v, want error containing %q", tc.zone, h, tc.wantErr)
}
if !errors.Is(err, provider.ErrPermanent) {
t.Errorf("host(%q) error = %v, want ErrPermanent", tc.zone, err)
}
if !strings.Contains(err.Error(), tc.wantErr) {
t.Errorf("host(%q) error = %q, want substring %q", tc.zone, err, tc.wantErr)
}
return
}
if err != nil {
t.Fatalf("host(%q) error = %v, want nil", tc.zone, err)
}
if h.Name != tc.wantName {
t.Errorf("host(%q).Name = %q, want %q", tc.zone, h.Name, tc.wantName)
}
})
}
}
func TestWithDefaults_table(t *testing.T) {
t.Parallel()
tests := []struct {
name string
in provider.KVMConfig
check func(t *testing.T, got provider.KVMConfig)
}{
{
name: "empty config gets disk size and empty maps",
in: provider.KVMConfig{},
check: func(t *testing.T, got provider.KVMConfig) {
if got.DiskSizeGB != 10 {
t.Errorf("DiskSizeGB = %d, want 10", got.DiskSizeGB)
}
if got.Hosts == nil || got.Images == nil {
t.Errorf("Hosts = %v, Images = %v, want non-nil", got.Hosts, got.Images)
}
},
},
{
name: "explicit disk size kept",
in: provider.KVMConfig{DiskSizeGB: 40},
check: func(t *testing.T, got provider.KVMConfig) {
if got.DiskSizeGB != 40 {
t.Errorf("DiskSizeGB = %d, want 40", got.DiskSizeGB)
}
},
},
{
name: "host socket and pool default",
in: provider.KVMConfig{Hosts: []provider.KVMHost{{Name: "a"}, {Name: "b", Socket: "/run/s", Pool: "fast"}}},
check: func(t *testing.T, got provider.KVMConfig) {
if got.Hosts[0].Socket != "/var/run/libvirt/libvirt-sock" || got.Hosts[0].Pool != "default" {
t.Errorf("hosts[0] = %+v, want default socket and pool", got.Hosts[0])
}
if got.Hosts[1].Socket != "/run/s" || got.Hosts[1].Pool != "fast" {
t.Errorf("hosts[1] = %+v, want explicit socket and pool kept", got.Hosts[1])
}
},
},
{
name: "image format and firmware default",
in: provider.KVMConfig{Images: map[string]provider.KVMImage{
"d": {URL: "https://x/d.qcow2"},
"u": {URL: "https://x/u.img", Format: provider.KVMFormatRaw, Firmware: provider.KVMFirmwareBIOS},
}},
check: func(t *testing.T, got provider.KVMConfig) {
if got.Images["d"].Format != provider.KVMFormatQcow2 || got.Images["d"].Firmware != provider.KVMFirmwareUEFI {
t.Errorf("images[d] = %+v, want qcow2 and uefi", got.Images["d"])
}
if got.Images["u"].Format != provider.KVMFormatRaw || got.Images["u"].Firmware != provider.KVMFirmwareBIOS {
t.Errorf("images[u] = %+v, want raw and bios kept", got.Images["u"])
}
},
},
}
for _, tc := range tests {
t.Run(tc.name, func(t *testing.T) {
t.Parallel()
tc.check(t, withDefaults(tc.in))
})
}
}
func TestWithDefaults_doesNotAliasInput(t *testing.T) {
t.Parallel()
in := provider.KVMConfig{
Hosts: []provider.KVMHost{{Name: "a"}},
Images: map[string]provider.KVMImage{"d": {URL: "https://x/d.qcow2"}},
}
got := withDefaults(in)
got.Hosts[0].Name = "changed"
got.Images["d"] = provider.KVMImage{URL: "changed"}
if in.Hosts[0].Name != "a" || in.Images["d"].URL != "https://x/d.qcow2" {
t.Errorf("input mutated through the defaulted copy: %+v", in)
}
}
func TestWrapErr_carriesProviderNameOpAndID(t *testing.T) {
t.Parallel()
p := newProvider("kvm2-home", twoHosts(), &stubDialer{})
err := p.wrapErr("get", "hosts/beelink/domains/proxy-x", permanentf("boom"))
var pe *provider.Error
if !errors.As(err, &pe) {
t.Fatalf("wrapErr() = %T, want *provider.Error", err)
}
if pe.Provider != "kvm2-home" || pe.Op != "get" || pe.ID != "hosts/beelink/domains/proxy-x" || pe.Class != provider.ErrPermanent {
t.Errorf("wrapErr() = %+v, want provider kvm2-home, op get, the id and ErrPermanent", pe)
}
}
internal/provider/kvm2/list.go
+56
View File
@@ -0,0 +1,56 @@
package kvm2
import (
"context"
"fmt"
"github.com/digitalocean/go-libvirt"
"gitea.home.hrajfrisbee.cz/kacerr/egress-proxies-operator/internal/provider"
)
// ListByTag is spec entry 12: every domain on every configured host, in
// config order, filtered client-side on the kvm2 mark. GC uses only ID, UID
// and CreatedAt (contract §2.4), all of which the name and the metadata
// element give, so no state or address RPC is made and State and IP stay
// empty. A host that cannot be reached or listed fails the whole call
// (fail-whole); a single domain whose metadata cannot be read is skipped,
// unless ctx ended, which fails the call rather than pass off an empty view
// as the truth.
func (p *Provider) ListByTag(ctx context.Context) ([]provider.Instance, error) {
log := p.logger(ctx)
var out []provider.Instance
for _, host := range p.cfg.Hosts {
hlog := log.WithValues("host", host.Name)
be, err := p.hosts.get(ctx, host)
if err != nil {
logAPIError(hlog, "list", err)
return nil, p.wrapErr("list", "", err)
}
doms, _, err := be.ConnectListAllDomains(ctx, 1, libvirt.ConnectListDomainsActive|libvirt.ConnectListDomainsInactive)
if err != nil {
err = fmt.Errorf("listing domains on host %s: %w", host.Name, err)
logAPIError(hlog, "list", err)
return nil, p.wrapErr("list", "", err)
}
for _, dom := range doms {
dlog := hlog.WithValues("name", dom.Name)
meta, err := readMetadata(ctx, be, dom)
if err != nil {
if ctx.Err() != nil {
return nil, p.wrapErr("list", "", err)
}
dlog.V(1).Info("skipping domain with unreadable metadata", "error", err.Error())
continue
}
if !meta.managed() {
continue
}
inst := provider.Instance{ID: formatProviderID(host.Name, dom.Name), UID: meta.UID, CreatedAt: meta.createdAt()}
dlog.V(2).Info("kvm2 listed instance", "id", inst.ID, "uid", inst.UID, "createdAt", inst.CreatedAt)
out = append(out, inst)
}
}
log.V(1).Info("kvm2 domains listed", "hosts", len(p.cfg.Hosts), "managed", len(out))
return out, nil
}
internal/provider/kvm2/list_test.go
+213
View File
@@ -0,0 +1,213 @@
package kvm2
import (
"errors"
"strings"
"testing"
"time"
"github.com/digitalocean/go-libvirt"
"gitea.home.hrajfrisbee.cz/kacerr/egress-proxies-operator/internal/provider"
)
func listIDs(list []provider.Instance) []string {
ids := make([]string, len(list))
for i, inst := range list {
ids[i] = inst.ID
}
return ids
}
func TestListByTag_excludesUnmarkedDomains(t *testing.T) {
t.Parallel()
fx := newFixture(t, 1)
id := fx.create(testName)
fx.fake.addDomain("legacy-vm", libvirt.DomainRunning, "")
fx.fake.addDomain("unmarked", libvirt.DomainRunning, `<instance><managed>false</managed><uid>x</uid></instance>`)
fx.fake.addDomain("unrelated", libvirt.DomainShutoff, `<instance><note>hand made</note></instance>`)
list, err := fx.p.ListByTag(fx.ctx)
if err != nil {
t.Fatalf("ListByTag() error = %v", err)
}
if got := listIDs(list); len(got) != 1 || got[0] != id {
t.Errorf("ListByTag = %v, want only %s", got, id)
}
if list[0].UID != testUID || list[0].CreatedAt.IsZero() {
t.Errorf("listed = %+v, want uid and createdAt", list[0])
}
if got := fx.logs.matching("kvm2 domains listed"); len(got) != 1 || !strings.Contains(got[0], `"hosts"=1`) || !strings.Contains(got[0], `"managed"=1`) {
t.Errorf("listed line = %v", got)
}
}
func TestListByTag_emptyUIDStillListed(t *testing.T) {
t.Parallel()
fx := newFixture(t, 1)
fx.fake.addDomain("proxy-nouid", libvirt.DomainRunning, managedMetadata("", time.Now()))
list, err := fx.p.ListByTag(fx.ctx)
if err != nil {
t.Fatalf("ListByTag() error = %v", err)
}
if len(list) != 1 || list[0].UID != "" || list[0].ID != formatProviderID(testHost, "proxy-nouid") {
t.Errorf("ListByTag = %+v, want the domain listed with an empty UID", list)
}
}
func TestListByTag_acrossHostsInConfigOrder(t *testing.T) {
t.Parallel()
cfg := twoHosts()
cfg.Hosts[1].Pool = ""
first, second := newFakeBackend(), newFakeBackend()
first.addDomain("proxy-z", libvirt.DomainRunning, managedMetadata("uid-z", time.Now()))
first.addDomain("proxy-a", libvirt.DomainShutoff, managedMetadata("uid-a", time.Now()))
second.addDomain("proxy-b", libvirt.DomainRunning, managedMetadata("uid-b", time.Now()))
p := newProvider(testProvider, cfg, &hostDialer{backends: map[string]backend{"beelink": first, "lab2": second}})
ctx, _ := captureContext()
list, err := p.ListByTag(ctx)
if err != nil {
t.Fatalf("ListByTag() error = %v", err)
}
want := []string{formatProviderID("beelink", "proxy-a"), formatProviderID("beelink", "proxy-z"), formatProviderID("lab2", "proxy-b")}
if got := listIDs(list); strings.Join(got, ",") != strings.Join(want, ",") {
t.Errorf("ListByTag = %v, want %v (config order, then name order)", got, want)
}
if list[0].UID != "uid-a" || list[1].UID != "uid-z" || list[2].UID != "uid-b" {
t.Errorf("uids = %s, %s, %s, want uid-a, uid-z, uid-b", list[0].UID, list[1].UID, list[2].UID)
}
}
func TestListByTag_doesNotReadStateOrAddresses(t *testing.T) {
t.Parallel()
fx := newFixture(t, 2)
id := fx.create(testName)
fx.setAddrs(testName, libvirt.DomainInterfaceAddressesSrcAgent, "10.0.0.5")
fx.fake.addDomain("proxy-off", libvirt.DomainShutoff, managedMetadata("uid-off", time.Now()))
before := len(fx.fake.callLog())
list, err := fx.p.ListByTag(fx.ctx)
if err != nil {
t.Fatalf("ListByTag() error = %v", err)
}
if len(list) != 2 || list[0].ID != id || list[1].ID != formatProviderID(testHost, "proxy-off") {
t.Fatalf("ListByTag = %v, want the two managed domains", listIDs(list))
}
for _, inst := range list {
if inst.State != "" || inst.IP != "" {
t.Errorf("listed %s = state %q ip %q, want both empty: GC does not use them", inst.ID, inst.State, inst.IP)
}
if inst.UID == "" || inst.CreatedAt.IsZero() {
t.Errorf("listed %s lacks uid or createdAt: %+v", inst.ID, inst)
}
}
for _, rpc := range fx.fake.callLog()[before:] {
switch rpc {
case "ConnectListAllDomains", "DomainGetMetadata":
default:
t.Errorf("ListByTag made %s; want only the listing and one metadata read per domain", rpc)
}
}
if got := fx.logs.matching("kvm2 listed instance"); len(got) != 2 || strings.Contains(got[0], `"state"=`) {
t.Errorf("listed instance lines = %v, want two without a state", got)
}
}
func TestListByTag_unreachableHostFailsWhole(t *testing.T) {
t.Parallel()
first := newFakeBackend()
first.addDomain("proxy-a", libvirt.DomainRunning, managedMetadata("uid-a", time.Now()))
d := &hostDialer{
backends: map[string]backend{"beelink": first},
errs: map[string]error{"lab2": errors.New("dial tcp 192.0.2.2:22: i/o timeout")},
}
p := newProvider(testProvider, twoHosts(), d)
ctx, _ := captureContext()
list, err := p.ListByTag(ctx)
if list != nil {
t.Errorf("ListByTag = %v, want nil with the error", list)
}
assertProviderError(t, err, "list", provider.ErrTransient)
}
func TestListByTag_unreadableDomainSkipped(t *testing.T) {
t.Parallel()
tests := []struct {
name string
setup func(fx *fixture)
line string
}{
{
name: "corrupt metadata on one domain",
setup: func(fx *fixture) {
fx.fake.addDomain("proxy-bad", libvirt.DomainRunning, "<bogus>not an instance</bogus>")
},
line: "skipping domain with unreadable metadata",
},
{
name: "metadata rpc fails for the first domain only",
setup: func(fx *fixture) {
fx.fake.addDomain("proxy-bad", libvirt.DomainRunning, managedMetadata("uid-bad", time.Now()))
var calls int
fx.fake.mu.Lock()
fx.fake.hooks["DomainGetMetadata"] = func() error {
calls++
if calls == 1 {
return fakeErr(libvirt.ErrInternalError, "internal error: metadata read failed")
}
return nil
}
fx.fake.mu.Unlock()
},
line: "skipping domain with unreadable metadata",
},
{
name: "domain gone between listing and metadata read",
setup: func(fx *fixture) {
fx.fake.addDomain("proxy-bad", libvirt.DomainRunning, managedMetadata("uid-bad", time.Now()))
var calls int
fx.fake.mu.Lock()
fx.fake.hooks["DomainGetMetadata"] = func() error {
calls++
if calls == 1 {
return fakeErr(libvirt.ErrNoDomain, "Domain not found")
}
return nil
}
fx.fake.mu.Unlock()
},
line: "skipping domain with unreadable metadata",
},
}
for _, tc := range tests {
t.Run(tc.name, func(t *testing.T) {
t.Parallel()
fx := newFixture(t, 1)
fx.fake.addDomain("proxy-good", libvirt.DomainRunning, managedMetadata("uid-good", time.Now()))
tc.setup(fx)
list, err := fx.p.ListByTag(fx.ctx)
if err != nil {
t.Fatalf("ListByTag() error = %v", err)
}
if got := listIDs(list); len(got) != 1 || got[0] != formatProviderID(testHost, "proxy-good") {
t.Errorf("ListByTag = %v, want only proxy-good", got)
}
if got := fx.logs.matching(tc.line); len(got) != 1 || !strings.Contains(got[0], `"name"="proxy-bad"`) {
t.Errorf("skip line = %v, want one naming proxy-bad", got)
}
})
}
}
func TestListByTag_includesInactiveDomains(t *testing.T) {
t.Parallel()
fx := newFixture(t, 1)
id := fx.create(testName)
fx.withDomain(testName, func(d *fakeDomain) { d.state = libvirt.DomainShutoff })
list, err := fx.p.ListByTag(fx.ctx)
if err != nil {
t.Fatalf("ListByTag() error = %v", err)
}
if len(list) != 1 || list[0].ID != id || list[0].UID != testUID {
t.Errorf("ListByTag = %+v, want the shut-off domain listed with its uid", list)
}
}
internal/provider/kvm2/metadata.go
+78
View File
@@ -0,0 +1,78 @@
package kvm2
import (
"encoding/xml"
"strings"
"time"
"gitea.home.hrajfrisbee.cz/kacerr/egress-proxies-operator/internal/provider"
)
// The GC mark lives in the domain's <metadata> as one element in kvm2's own
// namespace (spec entry 8). The namespace is what selects it in
// DomainGetMetadata; libvirt 11.6 hands the element back without the xmlns
// declaration, so parsing matches on the local name only.
const (
metadataURI = "https://gitea.home.hrajfrisbee.cz/kacerr/egress-proxies-operator/kvm2"
metadataLocalName = "instance"
)
// instanceMetadata is the mark as it is written and read back. The fields
// are the element's text as is; managed and createdAt interpret them.
type instanceMetadata struct {
XMLName xml.Name
Managed string `xml:"managed"`
UID string `xml:"uid"`
CreatedAt string `xml:"createdAt"`
}
func newInstanceMetadata(uid string, createdAt time.Time) instanceMetadata {
return instanceMetadata{
XMLName: xml.Name{Space: metadataURI, Local: metadataLocalName},
Managed: provider.LabelManagedYes,
UID: uid,
CreatedAt: createdAt.UTC().Format(time.RFC3339),
}
}
func (m instanceMetadata) managed() bool {
return m.Managed == provider.LabelManagedYes
}
// createdAt returns the zero time when the element carries no parsable
// timestamp; GC treats that as old, which the spec accepts.
func (m instanceMetadata) createdAt() time.Time {
t, err := time.Parse(time.RFC3339, m.CreatedAt)
if err != nil {
return time.Time{}
}
return t
}
// marshalInstanceMetadata renders the namespaced element that goes under
// the domain's <metadata>.
func marshalInstanceMetadata(m instanceMetadata) (string, error) {
m.XMLName = xml.Name{Space: metadataURI, Local: metadataLocalName}
out, err := xml.Marshal(m)
if err != nil {
return "", permanentf("marshalling instance metadata: %v", err)
}
return string(out), nil
}
// parseInstanceMetadata reads the element back, with or without its xmlns
// declaration. Anything that is not an <instance> element is ErrPermanent:
// the domain's metadata is corrupt and no retry fixes it.
func parseInstanceMetadata(s string) (instanceMetadata, error) {
var m instanceMetadata
if strings.TrimSpace(s) == "" {
return instanceMetadata{}, permanentf("parsing instance metadata: empty document")
}
if err := xml.Unmarshal([]byte(s), &m); err != nil {
return instanceMetadata{}, permanentf("parsing instance metadata: %v", err)
}
if m.XMLName.Local != metadataLocalName {
return instanceMetadata{}, permanentf("parsing instance metadata: root element is <%s>, want <%s>", m.XMLName.Local, metadataLocalName)
}
return m, nil
}
internal/provider/kvm2/metadata_test.go
+162
View File
@@ -0,0 +1,162 @@
package kvm2
import (
"errors"
"strings"
"testing"
"time"
"gitea.home.hrajfrisbee.cz/kacerr/egress-proxies-operator/internal/provider"
)
func TestMarshalInstanceMetadata_namespacedElement(t *testing.T) {
t.Parallel()
created := time.Date(2026, 9, 16, 10, 30, 0, 0, time.FixedZone("CEST", 2*3600))
got, err := marshalInstanceMetadata(newInstanceMetadata("uid-1", created))
if err != nil {
t.Fatal(err)
}
for _, want := range []string{
`<instance xmlns="` + metadataURI + `">`,
"<managed>true</managed>",
"<uid>uid-1</uid>",
"<createdAt>2026-09-16T08:30:00Z</createdAt>",
"</instance>",
} {
if !strings.Contains(got, want) {
t.Errorf("marshalled metadata %q lacks %q", got, want)
}
}
}
func TestParseMetadata_withNamespace(t *testing.T) {
t.Parallel()
in := `<instance xmlns="` + metadataURI + `"><managed>true</managed><uid>abc</uid><createdAt>2026-09-16T08:30:00Z</createdAt></instance>`
m, err := parseInstanceMetadata(in)
if err != nil {
t.Fatal(err)
}
if !m.managed() {
t.Error("managed() = false, want true")
}
if m.UID != "abc" {
t.Errorf("UID = %q, want abc", m.UID)
}
if want := time.Date(2026, 9, 16, 8, 30, 0, 0, time.UTC); !m.createdAt().Equal(want) {
t.Errorf("createdAt() = %v, want %v", m.createdAt(), want)
}
}
func TestParseMetadata_withoutNamespaceDeclaration(t *testing.T) {
t.Parallel()
in := "<instance>\n <managed>true</managed>\n <uid>abc</uid>\n <createdAt>2026-09-16T08:30:00Z</createdAt>\n</instance>"
m, err := parseInstanceMetadata(in)
if err != nil {
t.Fatal(err)
}
if !m.managed() || m.UID != "abc" {
t.Errorf("got managed=%v uid=%q, want true/abc", m.managed(), m.UID)
}
if m.createdAt().IsZero() {
t.Error("createdAt() is zero, want parsed timestamp")
}
}
func TestParseMetadata_absentGivesEmptyUID(t *testing.T) {
t.Parallel()
tests := []struct {
name string
in string
}{
{name: "empty element", in: "<instance/>"},
{name: "empty element with namespace", in: `<instance xmlns="` + metadataURI + `"></instance>`},
{name: "empty children", in: "<instance><managed></managed><uid></uid><createdAt></createdAt></instance>"},
}
for _, tc := range tests {
t.Run(tc.name, func(t *testing.T) {
t.Parallel()
m, err := parseInstanceMetadata(tc.in)
if err != nil {
t.Fatal(err)
}
if m.managed() {
t.Error("managed() = true, want false")
}
if m.UID != "" {
t.Errorf("UID = %q, want empty", m.UID)
}
if !m.createdAt().IsZero() {
t.Errorf("createdAt() = %v, want zero", m.createdAt())
}
})
}
}
func TestParseMetadata_badCreatedAtIsZeroTime(t *testing.T) {
t.Parallel()
tests := []struct {
name string
in string
}{
{name: "garbage", in: "<instance><managed>true</managed><uid>u</uid><createdAt>yesterday</createdAt></instance>"},
{name: "date only", in: "<instance><managed>true</managed><uid>u</uid><createdAt>2026-09-16</createdAt></instance>"},
{name: "missing", in: "<instance><managed>true</managed><uid>u</uid></instance>"},
}
for _, tc := range tests {
t.Run(tc.name, func(t *testing.T) {
t.Parallel()
m, err := parseInstanceMetadata(tc.in)
if err != nil {
t.Fatal(err)
}
if !m.createdAt().IsZero() {
t.Errorf("createdAt() = %v, want zero", m.createdAt())
}
if !m.managed() || m.UID != "u" {
t.Errorf("other fields lost: managed=%v uid=%q", m.managed(), m.UID)
}
})
}
}
func TestParseMetadata_notXMLIsPermanent(t *testing.T) {
t.Parallel()
tests := []struct {
name string
in string
}{
{name: "empty string", in: ""},
{name: "whitespace", in: " \n"},
{name: "plain text", in: "managed: true"},
{name: "unclosed element", in: "<instance><managed>true"},
}
for _, tc := range tests {
t.Run(tc.name, func(t *testing.T) {
t.Parallel()
_, err := parseInstanceMetadata(tc.in)
if !errors.Is(err, provider.ErrPermanent) {
t.Errorf("err = %v, want ErrPermanent", err)
}
})
}
}
func TestParseMetadata_wrongRootIsPermanent(t *testing.T) {
t.Parallel()
tests := []struct {
name string
in string
}{
{name: "other local name", in: "<proxy><managed>true</managed></proxy>"},
{name: "other local name with our namespace", in: `<vm xmlns="` + metadataURI + `"><uid>x</uid></vm>`},
}
for _, tc := range tests {
t.Run(tc.name, func(t *testing.T) {
t.Parallel()
_, err := parseInstanceMetadata(tc.in)
if !errors.Is(err, provider.ErrPermanent) {
t.Errorf("err = %v, want ErrPermanent", err)
}
})
}
}
internal/provider/kvm2/observe.go
+263
View File
@@ -0,0 +1,263 @@
package kvm2
import (
"context"
"io"
"time"
"github.com/digitalocean/go-libvirt"
"go.opentelemetry.io/otel/attribute"
"go.opentelemetry.io/otel/codes"
"go.opentelemetry.io/otel/trace"
logf "sigs.k8s.io/controller-runtime/pkg/log"
"gitea.home.hrajfrisbee.cz/kacerr/egress-proxies-operator/internal/tracing"
)
// observed is the span-decorator of spec entries 17 and 20: one child span
// kvm2.<Method> and one V(1) "libvirt call" line per call, with request and
// response detail at V(2). Applied once, in the dialer, over every backend
// that leaves it.
type observed struct {
inner backend
provider string
host string
tracer trace.Tracer
}
var (
_ backend = (*observed)(nil)
_ liveness = (*observed)(nil)
)
func observe(inner backend, providerName, hostName string, opts ...tracing.Option) backend {
return &observed{inner: inner, provider: providerName, host: hostName, tracer: tracing.Tracer(opts...)}
}
// dead forwards the inner backend's liveness; a backend without one is
// never dropped by the host table.
func (o *observed) dead() <-chan struct{} {
if l, ok := o.inner.(liveness); ok {
return l.dead()
}
return nil
}
// id is one identifier a call names: a domain, pool or volume. It becomes
// both the span attribute kvm2.<key> and the log key.
type id struct{ key, val string }
func domainID(name string) id { return id{"domain", name} }
func poolID(name string) id { return id{"pool", name} }
func volumeID(name string) id { return id{"volume", name} }
// detail returns V(2)-only key/value pairs. It runs only when V(2) is on,
// so building the list of a hundred domain names costs nothing otherwise.
type detail func() []any
// call wraps one backend call: span, timing, status and the log line. fn
// returns the V(2) detail for the line, or nil.
func (o *observed) call(ctx context.Context, rpc string, ids []id, fn func(ctx context.Context) (detail, error)) error {
attrs := make([]attribute.KeyValue, 0, len(ids)+1)
attrs = append(attrs, attribute.String("kvm2.host", o.host))
for _, i := range ids {
attrs = append(attrs, attribute.String("kvm2."+i.key, i.val))
}
ctx, span := tracing.StartSpan(ctx, o.tracer, "kvm2."+rpc, trace.WithSpanKind(trace.SpanKindClient), trace.WithAttributes(attrs...))
defer span.End()
log := logf.FromContext(ctx).WithName("kvm2").WithValues("provider", o.provider, "host", o.host)
start := time.Now()
more, err := fn(ctx)
kv := make([]any, 0, 8+2*len(ids))
kv = append(kv, "rpc", rpc)
for _, i := range ids {
kv = append(kv, i.key, i.val)
}
kv = append(kv, "duration", time.Since(start))
if err != nil {
span.SetStatus(codes.Error, err.Error())
if code, ok := libvirtCode(err); ok {
span.SetAttributes(attribute.Int("kvm2.libvirt_code", int(code)))
}
kv = append(kv, "error", err.Error())
kv = append(kv, libvirtErrorFields(err)...)
}
if more != nil && log.V(2).Enabled() {
kv = append(kv, more()...)
}
log.V(1).Info("libvirt call", kv...)
return err
}
func (o *observed) DomainLookupByName(ctx context.Context, name string) (dom libvirt.Domain, err error) {
err = o.call(ctx, "DomainLookupByName", []id{domainID(name)}, func(ctx context.Context) (detail, error) {
dom, err = o.inner.DomainLookupByName(ctx, name)
return nil, err
})
return dom, err
}
func (o *observed) DomainDefineXML(ctx context.Context, xml string) (dom libvirt.Domain, err error) {
err = o.call(ctx, "DomainDefineXML", nil, func(ctx context.Context) (detail, error) {
dom, err = o.inner.DomainDefineXML(ctx, xml)
return func() []any { return []any{"domain", dom.Name, "xml", xml} }, err
})
return dom, err
}
func (o *observed) DomainCreate(ctx context.Context, dom libvirt.Domain) error {
return o.call(ctx, "DomainCreate", []id{domainID(dom.Name)}, func(ctx context.Context) (detail, error) {
return nil, o.inner.DomainCreate(ctx, dom)
})
}
func (o *observed) DomainDestroy(ctx context.Context, dom libvirt.Domain) error {
return o.call(ctx, "DomainDestroy", []id{domainID(dom.Name)}, func(ctx context.Context) (detail, error) {
return nil, o.inner.DomainDestroy(ctx, dom)
})
}
func (o *observed) DomainUndefineFlags(ctx context.Context, dom libvirt.Domain, flags libvirt.DomainUndefineFlagsValues) error {
return o.call(ctx, "DomainUndefineFlags", []id{domainID(dom.Name)}, func(ctx context.Context) (detail, error) {
return nil, o.inner.DomainUndefineFlags(ctx, dom, flags)
})
}
func (o *observed) DomainGetState(ctx context.Context, dom libvirt.Domain, flags uint32) (state int32, reason int32, err error) {
err = o.call(ctx, "DomainGetState", []id{domainID(dom.Name)}, func(ctx context.Context) (detail, error) {
state, reason, err = o.inner.DomainGetState(ctx, dom, flags)
return func() []any { return []any{"state", libvirt.DomainState(state).String(), "reason", reason} }, err
})
return state, reason, err
}
func (o *observed) DomainGetMetadata(ctx context.Context, dom libvirt.Domain, typ int32, uri libvirt.OptString, flags libvirt.DomainModificationImpact) (md string, err error) {
err = o.call(ctx, "DomainGetMetadata", []id{domainID(dom.Name)}, func(ctx context.Context) (detail, error) {
md, err = o.inner.DomainGetMetadata(ctx, dom, typ, uri, flags)
return func() []any { return []any{"metadata", md} }, err
})
return md, err
}
func (o *observed) DomainInterfaceAddresses(ctx context.Context, dom libvirt.Domain, source uint32, flags uint32) (ifaces []libvirt.DomainInterface, err error) {
err = o.call(ctx, "DomainInterfaceAddresses", []id{domainID(dom.Name)}, func(ctx context.Context) (detail, error) {
ifaces, err = o.inner.DomainInterfaceAddresses(ctx, dom, source, flags)
return func() []any { return []any{"source", source, "addresses", interfaceAddresses(ifaces)} }, err
})
return ifaces, err
}
func (o *observed) ConnectListAllDomains(ctx context.Context, needResults int32, flags libvirt.ConnectListAllDomainsFlags) (doms []libvirt.Domain, n uint32, err error) {
err = o.call(ctx, "ConnectListAllDomains", nil, func(ctx context.Context) (detail, error) {
doms, n, err = o.inner.ConnectListAllDomains(ctx, needResults, flags)
return func() []any { return []any{"count", len(doms), "domains", domainNames(doms)} }, err
})
return doms, n, err
}
func (o *observed) StoragePoolLookupByName(ctx context.Context, name string) (pool libvirt.StoragePool, err error) {
err = o.call(ctx, "StoragePoolLookupByName", []id{poolID(name)}, func(ctx context.Context) (detail, error) {
pool, err = o.inner.StoragePoolLookupByName(ctx, name)
return nil, err
})
return pool, err
}
func (o *observed) StoragePoolRefresh(ctx context.Context, pool libvirt.StoragePool, flags uint32) error {
return o.call(ctx, "StoragePoolRefresh", []id{poolID(pool.Name)}, func(ctx context.Context) (detail, error) {
return nil, o.inner.StoragePoolRefresh(ctx, pool, flags)
})
}
func (o *observed) StoragePoolGetInfo(ctx context.Context, pool libvirt.StoragePool) (state uint8, capacity, allocation, available uint64, err error) {
err = o.call(ctx, "StoragePoolGetInfo", []id{poolID(pool.Name)}, func(ctx context.Context) (detail, error) {
state, capacity, allocation, available, err = o.inner.StoragePoolGetInfo(ctx, pool)
return func() []any {
return []any{"state", libvirt.StoragePoolState(state).String(), "capacity", capacity, "allocation", allocation, "available", available}
}, err
})
return state, capacity, allocation, available, err
}
func (o *observed) StorageVolLookupByName(ctx context.Context, pool libvirt.StoragePool, name string) (vol libvirt.StorageVol, err error) {
err = o.call(ctx, "StorageVolLookupByName", []id{poolID(pool.Name), volumeID(name)}, func(ctx context.Context) (detail, error) {
vol, err = o.inner.StorageVolLookupByName(ctx, pool, name)
return nil, err
})
return vol, err
}
func (o *observed) StorageVolGetPath(ctx context.Context, vol libvirt.StorageVol) (path string, err error) {
err = o.call(ctx, "StorageVolGetPath", []id{poolID(vol.Pool), volumeID(vol.Name)}, func(ctx context.Context) (detail, error) {
path, err = o.inner.StorageVolGetPath(ctx, vol)
return func() []any { return []any{"path", path} }, err
})
return path, err
}
func (o *observed) StorageVolCreateXML(ctx context.Context, pool libvirt.StoragePool, xml string, flags libvirt.StorageVolCreateFlags) (vol libvirt.StorageVol, err error) {
err = o.call(ctx, "StorageVolCreateXML", []id{poolID(pool.Name)}, func(ctx context.Context) (detail, error) {
vol, err = o.inner.StorageVolCreateXML(ctx, pool, xml, flags)
return func() []any { return []any{"volume", vol.Name, "xml", xml} }, err
})
return vol, err
}
// StorageVolUpload and StorageVolDownload log the length only; the bodies
// are a seed ISO carrying CloudInit or a base image.
func (o *observed) StorageVolUpload(ctx context.Context, vol libvirt.StorageVol, r io.Reader, offset, length uint64, flags libvirt.StorageVolUploadFlags) error {
return o.call(ctx, "StorageVolUpload", []id{poolID(vol.Pool), volumeID(vol.Name)}, func(ctx context.Context) (detail, error) {
err := o.inner.StorageVolUpload(ctx, vol, r, offset, length, flags)
return func() []any { return []any{"offset", offset, "length", length} }, err
})
}
func (o *observed) StorageVolDownload(ctx context.Context, vol libvirt.StorageVol, w io.Writer, offset, length uint64, flags libvirt.StorageVolDownloadFlags) error {
return o.call(ctx, "StorageVolDownload", []id{poolID(vol.Pool), volumeID(vol.Name)}, func(ctx context.Context) (detail, error) {
err := o.inner.StorageVolDownload(ctx, vol, w, offset, length, flags)
return func() []any { return []any{"offset", offset, "length", length} }, err
})
}
func (o *observed) StorageVolDelete(ctx context.Context, vol libvirt.StorageVol, flags libvirt.StorageVolDeleteFlags) error {
return o.call(ctx, "StorageVolDelete", []id{poolID(vol.Pool), volumeID(vol.Name)}, func(ctx context.Context) (detail, error) {
return nil, o.inner.StorageVolDelete(ctx, vol, flags)
})
}
func (o *observed) ConnectGetLibVersion(ctx context.Context) (v uint64, err error) {
err = o.call(ctx, "ConnectGetLibVersion", nil, func(ctx context.Context) (detail, error) {
v, err = o.inner.ConnectGetLibVersion(ctx)
return func() []any { return []any{"libvirtVersion", v} }, err
})
return v, err
}
func (o *observed) ConnectGetHostname(ctx context.Context) (name string, err error) {
err = o.call(ctx, "ConnectGetHostname", nil, func(ctx context.Context) (detail, error) {
name, err = o.inner.ConnectGetHostname(ctx)
return func() []any { return []any{"hostname", name} }, err
})
return name, err
}
// Close is passed through: it is not a libvirt call and gets no span.
func (o *observed) Close() error { return o.inner.Close() }
func domainNames(doms []libvirt.Domain) []string {
names := make([]string, len(doms))
for i, d := range doms {
names[i] = d.Name
}
return names
}
func interfaceAddresses(ifaces []libvirt.DomainInterface) []string {
var out []string
for _, iface := range ifaces {
for _, a := range iface.Addrs {
out = append(out, iface.Name+"="+a.Addr)
}
}
return out
}
internal/provider/kvm2/observe_test.go
+334
View File
@@ -0,0 +1,334 @@
package kvm2
import (
"context"
"errors"
"reflect"
"strconv"
"strings"
"sync"
"testing"
"github.com/digitalocean/go-libvirt"
"github.com/go-logr/logr/funcr"
"go.opentelemetry.io/otel/attribute"
"go.opentelemetry.io/otel/codes"
sdktrace "go.opentelemetry.io/otel/sdk/trace"
"go.opentelemetry.io/otel/sdk/trace/tracetest"
logf "sigs.k8s.io/controller-runtime/pkg/log"
"gitea.home.hrajfrisbee.cz/kacerr/egress-proxies-operator/internal/tracing"
)
// logCapture collects funcr output at a verbosity, safe for concurrent use.
type logCapture struct {
mu sync.Mutex
lines []string
}
func (c *logCapture) all() []string {
c.mu.Lock()
defer c.mu.Unlock()
return append([]string(nil), c.lines...)
}
func (c *logCapture) matching(substr string) []string {
var out []string
for _, l := range c.all() {
if strings.Contains(l, substr) {
out = append(out, l)
}
}
return out
}
// observeFixture is a decorated fake with a span recorder and a log capture
// wired into ctx.
type observeFixture struct {
fake *fakeBackend
b backend
sr *tracetest.SpanRecorder
tp *sdktrace.TracerProvider
logs *logCapture
}
func newObserveFixture(t *testing.T, verbosity int) (*observeFixture, context.Context) {
t.Helper()
sr := tracetest.NewSpanRecorder()
tp := sdktrace.NewTracerProvider(sdktrace.WithSpanProcessor(sr))
t.Cleanup(func() { _ = tp.Shutdown(context.Background()) })
logs := &logCapture{}
log := funcr.New(func(prefix, args string) {
logs.mu.Lock()
defer logs.mu.Unlock()
logs.lines = append(logs.lines, prefix+" "+args)
}, funcr.Options{Verbosity: verbosity})
fake := newFakeBackend()
fx := &observeFixture{
fake: fake,
b: observe(fake, "kvm2-home", "beelink", tracing.WithTracerProvider(tp)),
sr: sr,
tp: tp,
logs: logs,
}
return fx, logf.IntoContext(context.Background(), log)
}
func TestObserve_everySeamMethodInstrumented(t *testing.T) {
t.Parallel()
seam := reflect.TypeOf((*backend)(nil)).Elem()
for i := range seam.NumMethod() {
m := seam.Method(i)
if m.Name == "Close" {
continue
}
t.Run(m.Name, func(t *testing.T) {
t.Parallel()
fx, ctx := newObserveFixture(t, 1)
args := make([]reflect.Value, m.Type.NumIn())
args[0] = reflect.ValueOf(ctx)
for j := 1; j < m.Type.NumIn(); j++ {
args[j] = reflect.Zero(m.Type.In(j))
}
reflect.ValueOf(fx.b).MethodByName(m.Name).Call(args)
ended := fx.sr.Ended()
if len(ended) != 1 {
t.Fatalf("got %d spans, want 1", len(ended))
}
if got, want := ended[0].Name(), "kvm2."+m.Name; got != want {
t.Errorf("span name = %q, want %q", got, want)
}
if !hasAttr(ended[0].Attributes(), attribute.String("kvm2.host", "beelink")) {
t.Errorf("span attributes %v lack kvm2.host", ended[0].Attributes())
}
lines := fx.logs.matching(`"msg"="libvirt call"`)
if len(lines) != 1 {
t.Fatalf("got %d 'libvirt call' lines, want 1:\n%s", len(lines), strings.Join(fx.logs.all(), "\n"))
}
for _, want := range []string{`"rpc"="` + m.Name + `"`, `"provider"="kvm2-home"`, `"host"="beelink"`, `"duration"=`, `"traceID"=`} {
if !strings.Contains(lines[0], want) {
t.Errorf("log line lacks %s: %s", want, lines[0])
}
}
if fx.fake.callCount(m.Name) != 1 {
t.Errorf("inner %s called %d times, want 1", m.Name, fx.fake.callCount(m.Name))
}
})
}
}
func TestObserve_childSpansUnderParent(t *testing.T) {
t.Parallel()
fx, ctx := newObserveFixture(t, 1)
ctx, parent := tracing.StartSpan(ctx, fx.tp.Tracer("test"), "provider.get")
_, _ = fx.b.DomainLookupByName(ctx, "proxy-x")
parent.End()
var child sdktrace.ReadOnlySpan
for _, s := range fx.sr.Ended() {
if s.Name() == "kvm2.DomainLookupByName" {
child = s
}
}
if child == nil {
t.Fatal("no kvm2.DomainLookupByName span recorded")
}
if child.Parent().SpanID() != parent.SpanContext().SpanID() {
t.Errorf("child parent = %s, want %s", child.Parent().SpanID(), parent.SpanContext().SpanID())
}
if child.SpanContext().TraceID() != parent.SpanContext().TraceID() {
t.Error("child is in a different trace than its parent")
}
if !hasAttr(child.Attributes(), attribute.String("kvm2.domain", "proxy-x")) {
t.Errorf("child attributes %v lack kvm2.domain", child.Attributes())
}
lines := fx.logs.matching(`"msg"="libvirt call"`)
if len(lines) != 1 || !strings.Contains(lines[0], `"spanID"="`+child.SpanContext().SpanID().String()+`"`) {
t.Errorf("log line does not carry the child's spanID: %v", lines)
}
}
func TestObserve_logsOneLinePerCallAtV1(t *testing.T) {
t.Parallel()
fx, ctx := newObserveFixture(t, 1)
fx.fake.addDomain("proxy-x", libvirt.DomainRunning, "")
pool, _ := fx.b.StoragePoolLookupByName(ctx, fakePoolName)
_, _ = fx.b.DomainLookupByName(ctx, "proxy-x")
_, _ = fx.b.StorageVolLookupByName(ctx, pool, "proxy-x.qcow2")
lines := fx.logs.matching(`"msg"="libvirt call"`)
if len(lines) != 3 {
t.Fatalf("got %d lines, want 3:\n%s", len(lines), strings.Join(fx.logs.all(), "\n"))
}
if len(fx.logs.all()) != 3 {
t.Errorf("got %d lines in total, want only the 3 call lines:\n%s", len(fx.logs.all()), strings.Join(fx.logs.all(), "\n"))
}
wants := []string{
`"rpc"="StoragePoolLookupByName" "pool"="default"`,
`"rpc"="DomainLookupByName" "domain"="proxy-x"`,
`"rpc"="StorageVolLookupByName" "pool"="default" "volume"="proxy-x.qcow2"`,
}
for i, want := range wants {
if !strings.Contains(lines[i], want) {
t.Errorf("line %d = %s, want %s in it", i, lines[i], want)
}
if !strings.Contains(lines[i], `"level"=1`) {
t.Errorf("line %d is not V(1): %s", i, lines[i])
}
}
}
func TestObserve_detailOnlyAtV2(t *testing.T) {
t.Parallel()
const xmlDoc = "<domain><name>proxy-x</name><metadata><instance><uid>u1</uid></instance></metadata></domain>"
tests := []struct {
name string
verbosity int
wantXML bool
}{
{name: "V1 has no xml", verbosity: 1, wantXML: false},
{name: "V2 has xml", verbosity: 2, wantXML: true},
}
for _, tc := range tests {
t.Run(tc.name, func(t *testing.T) {
t.Parallel()
fx, ctx := newObserveFixture(t, tc.verbosity)
dom, err := fx.b.DomainDefineXML(ctx, xmlDoc)
if err != nil {
t.Fatalf("DomainDefineXML error = %v", err)
}
_, _ = fx.b.DomainGetMetadata(ctx, dom, int32(libvirt.DomainMetadataElement), libvirt.OptString{metadataURI}, 0)
_, _, _ = fx.b.ConnectListAllDomains(ctx, 1, 0)
all := strings.Join(fx.logs.all(), "\n")
if got := strings.Contains(all, "<domain>"); got != tc.wantXML {
t.Errorf("xml logged = %v, want %v:\n%s", got, tc.wantXML, all)
}
if got := strings.Contains(all, `"metadata"="<instance>`); got != tc.wantXML {
t.Errorf("metadata logged = %v, want %v:\n%s", got, tc.wantXML, all)
}
if got := strings.Contains(all, `"domains"=["proxy-x"]`); got != tc.wantXML {
t.Errorf("domain names logged = %v, want %v:\n%s", got, tc.wantXML, all)
}
if got := len(fx.logs.matching(`"msg"="libvirt call"`)); got != 3 {
t.Errorf("got %d call lines, want 3", got)
}
})
}
}
func TestObserve_uploadBodyNeverLogged(t *testing.T) {
t.Parallel()
const sentinel = "SENTINEL-CLOUDINIT-3c7a"
fx, ctx := newObserveFixture(t, 2)
pool, _ := fx.b.StoragePoolLookupByName(ctx, fakePoolName)
vol, err := fx.b.StorageVolCreateXML(ctx, pool, "<volume><name>proxy-x-seed.iso</name><capacity>1024</capacity></volume>", 0)
if err != nil {
t.Fatalf("StorageVolCreateXML error = %v", err)
}
body := "#cloud-config\npassword: " + sentinel + "\n"
if err := fx.b.StorageVolUpload(ctx, vol, strings.NewReader(body), 0, uint64(len(body)), 0); err != nil {
t.Fatalf("StorageVolUpload error = %v", err)
}
var sink strings.Builder
if err := fx.b.StorageVolDownload(ctx, vol, &sink, 0, 0, 0); err != nil {
t.Fatalf("StorageVolDownload error = %v", err)
}
if sink.String() != body {
t.Errorf("downloaded %q, want the uploaded body", sink.String())
}
all := strings.Join(fx.logs.all(), "\n")
if strings.Contains(all, sentinel) {
t.Errorf("upload body leaked into logs:\n%s", all)
}
uploads := fx.logs.matching(`"rpc"="StorageVolUpload"`)
if len(uploads) != 1 || !strings.Contains(uploads[0], `"length"=`+strconv.Itoa(len(body))) {
t.Errorf("upload line lacks length: %v", uploads)
}
for _, s := range fx.sr.Ended() {
for _, a := range s.Attributes() {
if strings.Contains(a.Value.Emit(), sentinel) {
t.Errorf("span %s attribute %s carries the body", s.Name(), a.Key)
}
}
}
}
func TestObserve_errorFieldsOnFailure(t *testing.T) {
t.Parallel()
fx, ctx := newObserveFixture(t, 1)
_, err := fx.b.DomainLookupByName(ctx, "missing")
if !hasCode(err, libvirt.ErrNoDomain) {
t.Fatalf("error = %v, want ErrNoDomain through the decorator", err)
}
lines := fx.logs.matching(`"msg"="libvirt call"`)
if len(lines) != 1 {
t.Fatalf("got %d lines, want 1", len(lines))
}
for _, want := range []string{`"error"="Domain not found`, `"libvirtCode"=42`, `"libvirtError"="ErrNoDomain"`} {
if !strings.Contains(lines[0], want) {
t.Errorf("line lacks %s: %s", want, lines[0])
}
}
ended := fx.sr.Ended()
if len(ended) != 1 {
t.Fatalf("got %d spans, want 1", len(ended))
}
if ended[0].Status().Code != codes.Error {
t.Errorf("span status = %v, want Error", ended[0].Status().Code)
}
if !hasAttr(ended[0].Attributes(), attribute.Int("kvm2.libvirt_code", 42)) {
t.Errorf("span attributes %v lack kvm2.libvirt_code=42", ended[0].Attributes())
}
fx.fake.failWith("ConnectGetHostname", errors.New("plain failure"))
_, _ = fx.b.ConnectGetHostname(ctx)
lines = fx.logs.matching(`"rpc"="ConnectGetHostname"`)
if len(lines) != 1 || strings.Contains(lines[0], "libvirtCode") || !strings.Contains(lines[0], `"error"="plain failure"`) {
t.Errorf("non-libvirt failure line = %v, want error without libvirt fields", lines)
}
}
func TestObserve_v0Silent(t *testing.T) {
t.Parallel()
fx, ctx := newObserveFixture(t, 0)
fx.fake.addDomain("proxy-x", libvirt.DomainRunning, "")
_, _ = fx.b.DomainLookupByName(ctx, "proxy-x")
_, _ = fx.b.DomainLookupByName(ctx, "missing")
if lines := fx.logs.all(); len(lines) != 0 {
t.Errorf("got %d lines at V(0), want none:\n%s", len(lines), strings.Join(lines, "\n"))
}
if got := len(fx.sr.Ended()); got != 2 {
t.Errorf("got %d spans, want 2: spans do not depend on verbosity", got)
}
}
func TestObserve_closeAndLivenessPassThrough(t *testing.T) {
t.Parallel()
fx, _ := newObserveFixture(t, 1)
if isDead(fx.b) {
t.Error("decorated live backend reported dead")
}
fx.fake.markDead()
if !isDead(fx.b) {
t.Error("decorated dead backend reported live")
}
if err := fx.b.Close(); err != nil || fx.fake.closed != 1 {
t.Errorf("Close() = %v, closed %d times, want nil and 1", err, fx.fake.closed)
}
if got := len(fx.sr.Ended()); got != 0 {
t.Errorf("Close produced %d spans, want none", got)
}
if lines := fx.logs.all(); len(lines) != 0 {
t.Errorf("Close produced log lines: %v", lines)
}
}
func hasAttr(attrs []attribute.KeyValue, want attribute.KeyValue) bool {
for _, a := range attrs {
if a.Key == want.Key && a.Value == want.Value {
return true
}
}
return false
}
internal/provider/kvm2/provider_test.go
+426
View File
@@ -0,0 +1,426 @@
package kvm2
import (
"context"
"errors"
"fmt"
"net/http/httptest"
"strings"
"testing"
"time"
"github.com/digitalocean/go-libvirt"
"github.com/go-logr/logr/funcr"
logf "sigs.k8s.io/controller-runtime/pkg/log"
"gitea.home.hrajfrisbee.cz/kacerr/egress-proxies-operator/internal/provider"
)
const (
testProvider = "kvm2-home"
testHost = "beelink"
testUID = "0f1e2d3c-uid-1"
testName = "proxy-abcdefghijklmnop"
)
// fixture is one configured host backed by a fake, an image server the
// catalogue points at, a log capture at a chosen verbosity and a provider
// with a fast running-wait, so a Create runs end to end in memory.
type fixture struct {
t *testing.T
p *Provider
fake *fakeBackend
dialer *stubDialer
logs *logCapture
ctx context.Context
image []byte
}
func newFixture(t *testing.T, verbosity int, opts ...providerOption) *fixture {
t.Helper()
fx := &fixture{t: t, fake: newFakeBackend(), logs: &logCapture{}, image: []byte("QFI\xfb fake base image bytes")}
srv := httptest.NewServer(serveBytes(fx.image))
t.Cleanup(srv.Close)
fx.dialer = &stubDialer{backend: fx.fake}
cfg := provider.KVMConfig{
Images: map[string]provider.KVMImage{imageKey: {URL: srv.URL + "/debian-13-generic-amd64.qcow2"}},
MachineTypes: map[string]provider.KVMMachineType{"small": {VCPUs: 1, MemoryMiB: 1024}},
Hosts: []provider.KVMHost{{Name: testHost, Address: "beelink:22", User: "virt", Bridge: "br0"}},
}
opts = append([]providerOption{withHTTPClient(srv.Client()), withRunningWait(time.Millisecond, 500*time.Millisecond)}, opts...)
fx.p = newProvider(testProvider, cfg, fx.dialer, opts...)
log := funcr.New(func(prefix, args string) {
fx.logs.mu.Lock()
defer fx.logs.mu.Unlock()
fx.logs.lines = append(fx.logs.lines, prefix+" "+args)
}, funcr.Options{Verbosity: verbosity})
fx.ctx = logf.IntoContext(context.Background(), log)
return fx
}
func (fx *fixture) req(name string) provider.CreateRequest {
return provider.CreateRequest{
Name: name,
UID: testUID,
Namespace: "default",
ProxyName: "egress",
Placement: provider.Placement{Zone: testHost, MachineType: "small", Image: imageKey},
CloudInit: "#cloud-config\npackages: [squid]\n",
Port: 3128,
}
}
func (fx *fixture) create(name string) string {
fx.t.Helper()
id, err := fx.p.Create(fx.ctx, fx.req(name))
if err != nil {
fx.t.Fatalf("Create(%s) error = %v", name, err)
}
if want := formatProviderID(testHost, name); id != want {
fx.t.Fatalf("Create(%s) = %q, want %q", name, id, want)
}
return id
}
func (fx *fixture) get(id string) *provider.Instance {
fx.t.Helper()
inst, err := fx.p.Get(fx.ctx, id)
if err != nil {
fx.t.Fatalf("Get(%s) error = %v", id, err)
}
return inst
}
// withDomain runs fn on the fake's domain under its lock.
func (fx *fixture) withDomain(name string, fn func(d *fakeDomain)) {
fx.t.Helper()
fx.fake.mu.Lock()
defer fx.fake.mu.Unlock()
d, ok := fx.fake.domains[name]
if !ok {
fx.t.Fatalf("fake has no domain %s", name)
}
fn(d)
}
func (fx *fixture) hasDomain(name string) bool {
fx.fake.mu.Lock()
defer fx.fake.mu.Unlock()
_, ok := fx.fake.domains[name]
return ok
}
func (fx *fixture) setAddrs(name string, src libvirt.DomainInterfaceAddressesSource, ips ...string) {
fx.withDomain(name, func(d *fakeDomain) {
var addrs []libvirt.DomainIPAddr
for _, ip := range ips {
addrs = append(addrs, libvirt.DomainIPAddr{Type: int32(libvirt.IPAddrTypeIpv4), Addr: ip, Prefix: 24})
}
d.addrs[uint32(src)] = []libvirt.DomainInterface{{Name: "enp1s0", Addrs: addrs}}
})
}
// putVol places a volume in the fake's pool directly.
func (fx *fixture) putVol(name string, content []byte) *fakeVol {
fx.fake.mu.Lock()
defer fx.fake.mu.Unlock()
v := &fakeVol{content: content, capacity: uint64(len(content))}
fx.fake.pools[fakePoolName].vols[name] = v
return v
}
// baseName is the cached base image's volume name for this fixture's
// provider entry and image key.
func (fx *fixture) baseName() string {
fx.t.Helper()
src, err := resolveImage(fx.p.name, fx.p.cfg, imageKey)
if err != nil {
fx.t.Fatal(err)
}
return src.volumeName()
}
// cacheImage places a complete base image and its marker, as a previous
// Create on the host would have.
func (fx *fixture) cacheImage() {
src, err := resolveImage(fx.p.name, fx.p.cfg, imageKey)
if err != nil {
fx.t.Fatal(err)
}
fx.putVol(src.volumeName(), fx.image)
fx.putVol(src.markerName(), []byte(digestOf(fx.image)))
}
func managedMetadata(uid string, createdAt time.Time) string {
s, err := marshalInstanceMetadata(newInstanceMetadata(uid, createdAt))
if err != nil {
panic(err)
}
return s
}
// assertProviderError checks err is a *provider.Error from the configured
// provider with the given op and class.
func assertProviderError(t *testing.T, err error, op string, class error) {
t.Helper()
if err == nil {
t.Fatalf("error = nil, want op %s class %v", op, class)
}
var pe *provider.Error
if !errors.As(err, &pe) {
t.Fatalf("error = %T (%v), want *provider.Error", err, err)
}
if pe.Provider != testProvider {
t.Errorf("error.Provider = %q, want %q", pe.Provider, testProvider)
}
if pe.Op != op {
t.Errorf("error.Op = %q, want %q", pe.Op, op)
}
if !errors.Is(err, class) {
t.Errorf("error = %v, want class %v", err, class)
}
for _, other := range []error{provider.ErrNotFound, provider.ErrQuotaExceeded, provider.ErrPermanent, provider.ErrTransient} {
if other != class && errors.Is(err, other) {
t.Errorf("error = %v, also matches %v", err, other)
}
}
}
// hostDialer hands out one backend per host name and fails hosts in errs.
type hostDialer struct {
backends map[string]backend
errs map[string]error
}
func (d *hostDialer) dial(_ context.Context, host provider.KVMHost) (backend, error) {
if err := d.errs[host.Name]; err != nil {
return nil, err
}
b, ok := d.backends[host.Name]
if !ok {
return nil, fmt.Errorf("hostDialer: no backend for %s", host.Name)
}
return b, nil
}
// waitFor polls cond until it holds or the deadline passes.
func waitFor(t *testing.T, what string, cond func() bool) {
t.Helper()
deadline := time.Now().Add(5 * time.Second)
for !cond() {
if time.Now().After(deadline) {
t.Fatalf("timed out waiting for %s", what)
}
time.Sleep(time.Millisecond)
}
}
func TestProvider_cancelledContextReturnsPromptly(t *testing.T) {
t.Parallel()
tests := []struct {
name string
firstRPC string
op string
call func(ctx context.Context, fx *fixture) error
}{
{
name: "create", firstRPC: "DomainLookupByName", op: "create",
call: func(ctx context.Context, fx *fixture) error { _, err := fx.p.Create(ctx, fx.req(testName)); return err },
},
{
name: "get", firstRPC: "DomainLookupByName", op: "get",
call: func(ctx context.Context, fx *fixture) error {
_, err := fx.p.Get(ctx, formatProviderID(testHost, testName))
return err
},
},
{
name: "delete", firstRPC: "DomainLookupByName", op: "delete",
call: func(ctx context.Context, fx *fixture) error {
return fx.p.Delete(ctx, formatProviderID(testHost, testName))
},
},
{
name: "list", firstRPC: "ConnectListAllDomains", op: "list",
call: func(ctx context.Context, fx *fixture) error { _, err := fx.p.ListByTag(ctx); return err },
},
}
for _, tc := range tests {
t.Run(tc.name, func(t *testing.T) {
t.Parallel()
fx := newFixture(t, 1)
release := fx.fake.blockOn(tc.firstRPC)
defer close(release)
ctx, cancel := context.WithCancel(fx.ctx)
done := make(chan error, 1)
go func() { done <- tc.call(ctx, fx) }()
waitFor(t, "the first RPC to block", func() bool { return fx.fake.callCount(tc.firstRPC) == 1 })
cancel()
select {
case err := <-done:
assertProviderError(t, err, tc.op, provider.ErrTransient)
case <-time.After(time.Second):
t.Fatalf("%s did not return within 1s of cancellation", tc.name)
}
})
}
}
func TestProvider_everyErrorIsWrappedWithConfiguredName(t *testing.T) {
t.Parallel()
dialErr := errors.New("dial tcp 192.0.2.1:22: connect: connection refused")
tests := []struct {
name string
setup func(fx *fixture)
call func(fx *fixture) error
op string
class error
}{
{
name: "create unknown zone",
call: func(fx *fixture) error {
req := fx.req(testName)
req.Placement.Zone = "nas9"
_, err := fx.p.Create(fx.ctx, req)
return err
},
op: "create", class: provider.ErrPermanent,
},
{
name: "create dial failure",
setup: func(fx *fixture) { fx.dialer.err = dialErr },
call: func(fx *fixture) error { _, err := fx.p.Create(fx.ctx, fx.req(testName)); return err },
op: "create", class: provider.ErrTransient,
},
{
name: "create pool out of space",
setup: func(fx *fixture) {
fx.fake.mu.Lock()
fx.fake.pools[fakePoolName].available = 1
fx.fake.mu.Unlock()
},
call: func(fx *fixture) error { _, err := fx.p.Create(fx.ctx, fx.req(testName)); return err },
op: "create", class: provider.ErrQuotaExceeded,
},
{
name: "create vanished volume",
setup: func(fx *fixture) { fx.fake.failWith("StorageVolGetPath", fakeErr(libvirt.ErrNoStorageVol, "gone")) },
call: func(fx *fixture) error { _, err := fx.p.Create(fx.ctx, fx.req(testName)); return err },
op: "create", class: provider.ErrTransient,
},
{
name: "get malformed id",
call: func(fx *fixture) error { _, err := fx.p.Get(fx.ctx, "beelink/"+testName); return err },
op: "get", class: provider.ErrPermanent,
},
{
name: "get missing domain",
call: func(fx *fixture) error { _, err := fx.p.Get(fx.ctx, formatProviderID(testHost, testName)); return err },
op: "get", class: provider.ErrNotFound,
},
{
name: "get busy backend",
setup: func(fx *fixture) {
fx.create(testName)
fx.fake.failWith("DomainGetState", fakeErr(libvirt.ErrOperationInvalid, "busy"))
},
call: func(fx *fixture) error { _, err := fx.p.Get(fx.ctx, formatProviderID(testHost, testName)); return err },
op: "get", class: provider.ErrTransient,
},
{
name: "delete malformed id",
call: func(fx *fixture) error { return fx.p.Delete(fx.ctx, "hosts//domains/"+testName) },
op: "delete", class: provider.ErrPermanent,
},
{
name: "delete dial failure",
setup: func(fx *fixture) { fx.dialer.err = dialErr },
call: func(fx *fixture) error { return fx.p.Delete(fx.ctx, formatProviderID(testHost, testName)) },
op: "delete", class: provider.ErrTransient,
},
{
name: "delete access denied",
setup: func(fx *fixture) { fx.fake.failWith("DomainLookupByName", fakeErr(libvirt.ErrAccessDenied, "denied")) },
call: func(fx *fixture) error { return fx.p.Delete(fx.ctx, formatProviderID(testHost, testName)) },
op: "delete", class: provider.ErrPermanent,
},
{
name: "list dial failure",
setup: func(fx *fixture) { fx.dialer.err = dialErr },
call: func(fx *fixture) error { _, err := fx.p.ListByTag(fx.ctx); return err },
op: "list", class: provider.ErrTransient,
},
{
name: "list rpc failure",
setup: func(fx *fixture) {
fx.fake.failWith("ConnectListAllDomains", fakeErr(libvirt.ErrInternalError, "boom"))
},
call: func(fx *fixture) error { _, err := fx.p.ListByTag(fx.ctx); return err },
op: "list", class: provider.ErrTransient,
},
}
for _, tc := range tests {
t.Run(tc.name, func(t *testing.T) {
t.Parallel()
fx := newFixture(t, 1)
if tc.setup != nil {
tc.setup(fx)
}
assertProviderError(t, tc.call(fx), tc.op, tc.class)
})
}
}
func TestLogging_v0SilentV1OutcomeV2Detail(t *testing.T) {
t.Parallel()
outcomes := []string{"overlay volume created", "seed volume uploaded", "domain defined", "kvm2 domain ensured running"}
t.Run("v0 is silent", func(t *testing.T) {
t.Parallel()
fx := newFixture(t, 0)
fx.create(testName)
if lines := fx.logs.all(); len(lines) != 0 {
t.Errorf("Create logged %d lines at V(0), want none:\n%s", len(lines), strings.Join(lines, "\n"))
}
})
t.Run("v1 has outcomes without xml", func(t *testing.T) {
t.Parallel()
fx := newFixture(t, 1)
fx.create(testName)
for _, want := range outcomes {
if len(fx.logs.matching(want)) != 1 {
t.Errorf("V(1) lines matching %q = %d, want 1:\n%s", want, len(fx.logs.matching(want)), strings.Join(fx.logs.all(), "\n"))
}
}
if got := fx.logs.matching("<domain"); len(got) != 0 {
t.Errorf("V(1) carries domain XML:\n%s", strings.Join(got, "\n"))
}
if got := fx.logs.matching("kvm2 create resolved"); len(got) != 0 {
t.Errorf("V(1) carries the V(2) detail line:\n%s", strings.Join(got, "\n"))
}
for _, l := range fx.logs.all() {
if !strings.Contains(l, `"provider"="kvm2-home"`) || !strings.Contains(l, `"host"="beelink"`) {
t.Errorf("line lacks provider or host: %s", l)
}
if !strings.Contains(l, `"name"="`+testName+`"`) && !strings.Contains(l, `"volume"="base-`) {
t.Errorf("instance line lacks the name: %s", l)
}
}
})
t.Run("v2 has detail", func(t *testing.T) {
t.Parallel()
fx := newFixture(t, 2)
fx.create(testName)
if got := fx.logs.matching("<domain"); len(got) == 0 {
t.Errorf("V(2) lacks domain XML:\n%s", strings.Join(fx.logs.all(), "\n"))
}
resolved := fx.logs.matching("kvm2 create resolved")
if len(resolved) != 1 {
t.Fatalf("V(2) lines matching resolved = %d, want 1", len(resolved))
}
for _, key := range []string{`"cloudInitBytes"=`, `"image"="base-` + testProvider + `-debian-13.qcow2"`, `"firmware"="uefi"`, `"vcpus"=1`, `"memoryMiB"=1024`, `"machineType"="small"`, `"bridge"="br0"`, `"pool"="default"`} {
if !strings.Contains(resolved[0], key) {
t.Errorf("resolved line lacks %s: %s", key, resolved[0])
}
}
})
}
internal/provider/kvm2/ssh.go
+218
View File
@@ -0,0 +1,218 @@
package kvm2
import (
"context"
"errors"
"fmt"
"io/fs"
"net"
"os"
"sync"
"time"
"golang.org/x/crypto/ssh"
"golang.org/x/crypto/ssh/knownhosts"
"gitea.home.hrajfrisbee.cz/kacerr/egress-proxies-operator/internal/provider"
)
// dialStage names the step of a dial that failed, so the Checker (spec
// entry 16) can pick a reason without parsing messages.
type dialStage string
const (
// stageCredentials: the key or known_hosts file is missing or unparsable.
stageCredentials dialStage = "credentials"
// stageTCP: the TCP connection to host.Address failed.
stageTCP dialStage = "tcp"
// stageSSH: the SSH handshake failed, including host key and auth.
stageSSH dialStage = "ssh"
// stageSocket: the forwarded channel to the libvirt socket was refused.
stageSocket dialStage = "socket"
)
// dialError tags a dial failure with its stage and host. Unwrap keeps the
// cause visible, so classify still sees *knownhosts.KeyError, fs.ErrNotExist,
// the "unable to authenticate" text and *net.OpError.
type dialError struct {
stage dialStage
host string
err error
}
func (e *dialError) Error() string {
return fmt.Sprintf("dialing host %s (%s): %v", e.host, e.stage, e.err)
}
func (e *dialError) Unwrap() error { return e.err }
// stageOf returns the failed stage of a dial error, if err is one.
func stageOf(err error) (dialStage, bool) {
var de *dialError
if !errors.As(err, &de) {
return "", false
}
return de.stage, true
}
const (
defaultKeepaliveInterval = 15 * time.Second
keepaliveRequest = "keepalive@openssh.com"
)
// credentials is what one dial needs from the mounted files. They are read
// on every dial and never cached (spec entry 14), so rotating the Secret
// takes effect on the next connection.
type credentials struct {
signer ssh.Signer
hostKey ssh.HostKeyCallback
}
// loadCredentials reads and parses the key and known_hosts files. A missing
// file keeps fs.ErrNotExist (Transient: the Secret has not landed); a file
// that does not parse is Permanent. Messages name the path and carry the
// parser's own error, never the file's content.
func loadCredentials(cfg provider.KVMConfig) (credentials, error) {
keyBytes, err := os.ReadFile(cfg.SSHKeyFile)
if err != nil {
return credentials{}, fmt.Errorf("reading ssh key file %s: %w", cfg.SSHKeyFile, err)
}
signer, err := ssh.ParsePrivateKey(keyBytes)
if err != nil {
return credentials{}, permanentf("parsing ssh key file %s: %v", cfg.SSHKeyFile, err)
}
if cfg.InsecureIgnoreHostKey {
return credentials{signer: signer, hostKey: ssh.InsecureIgnoreHostKey()}, nil
}
hostKey, err := knownhosts.New(cfg.KnownHostsFile)
if err != nil {
var pathErr *fs.PathError
if errors.As(err, &pathErr) {
return credentials{}, fmt.Errorf("reading known_hosts file %s: %w", cfg.KnownHostsFile, err)
}
return credentials{}, permanentf("parsing known_hosts file %s: %v", cfg.KnownHostsFile, err)
}
return credentials{signer: signer, hostKey: hostKey}, nil
}
// sshSocket is the libvirt socket forwarded through one SSH connection. The
// embedded Conn is the channel; Close closes the SSH client, whose TCP
// connection carries it, and that alone unblocks the channel's readers and
// writers. Closing the channel as well would be worse than useless: its
// Close writes a close message under the channel's write lock, so it would
// queue behind a transfer stuck in a TCP write, which is the case Close
// exists to survive.
type sshSocket struct {
net.Conn
client *ssh.Client
closeOnce sync.Once
closeErr error
}
func (s *sshSocket) Close() error {
s.closeOnce.Do(func() { s.closeErr = s.client.Close() })
return s.closeErr
}
// keepalive sends keepalive@openssh.com every interval until stop is closed
// or a request fails or takes longer than the interval, and returns why it
// stopped (nil for stop). The caller marks its link dead on an error.
func (s *sshSocket) keepalive(interval time.Duration, stop <-chan struct{}) error {
ticker := time.NewTicker(interval)
defer ticker.Stop()
for {
select {
case <-stop:
return nil
case <-ticker.C:
}
done := make(chan error, 1)
go func() {
_, _, err := s.client.SendRequest(keepaliveRequest, true, nil)
done <- err
}()
timer := time.NewTimer(interval)
select {
case err := <-done:
timer.Stop()
if err != nil {
return fmt.Errorf("keepalive: %w", err)
}
case <-timer.C:
return fmt.Errorf("keepalive: no reply within %s", interval)
case <-stop:
timer.Stop()
return nil
}
}
}
// dialSSHSocket opens host's libvirt socket through SSH: credentials, TCP,
// handshake, channel. timeout bounds the whole sequence on top of ctx. Each
// failure is a *dialError naming its stage. Nothing here logs.
func dialSSHSocket(ctx context.Context, host provider.KVMHost, cfg provider.KVMConfig, timeout time.Duration) (*sshSocket, error) {
if timeout > 0 {
var cancel context.CancelFunc
ctx, cancel = context.WithTimeout(ctx, timeout)
defer cancel()
}
creds, err := loadCredentials(cfg)
if err != nil {
return nil, &dialError{stage: stageCredentials, host: host.Name, err: err}
}
var nd net.Dialer
tcp, err := nd.DialContext(ctx, "tcp", host.Address)
if err != nil {
return nil, &dialError{stage: stageTCP, host: host.Name, err: err}
}
clientCfg := &ssh.ClientConfig{
User: host.User,
Auth: []ssh.AuthMethod{ssh.PublicKeys(creds.signer)},
HostKeyCallback: creds.hostKey,
Timeout: timeout,
}
client, err := await(ctx, tcp.Close, func() (*ssh.Client, error) {
c, chans, reqs, err := ssh.NewClientConn(tcp, host.Address, clientCfg)
if err != nil {
return nil, err
}
return ssh.NewClient(c, chans, reqs), nil
})
if err != nil {
_ = tcp.Close()
return nil, &dialError{stage: stageSSH, host: host.Name, err: err}
}
conn, err := await(ctx, func() error { return client.Close() }, func() (net.Conn, error) {
return client.Dial("unix", host.Socket)
})
if err != nil {
_ = client.Close()
return nil, &dialError{stage: stageSocket, host: host.Name, err: fmt.Errorf("opening %s: %w", host.Socket, err)}
}
return &sshSocket{Conn: conn, client: client}, nil
}
// await runs a blocking step in a goroutine. When ctx ends first it calls
// abort, which closes the transport the step is blocked on, waits for the
// step to give up, and returns ctx's error instead.
func await[T any](ctx context.Context, abort func() error, step func() (T, error)) (T, error) {
type result struct {
v T
err error
}
done := make(chan result, 1)
go func() {
v, err := step()
done <- result{v, err}
}()
select {
case r := <-done:
return r.v, r.err
case <-ctx.Done():
_ = abort()
<-done
var zero T
return zero, ctx.Err()
}
}
internal/provider/kvm2/ssh_test.go
+343
View File
@@ -0,0 +1,343 @@
package kvm2
import (
"context"
"errors"
"io/fs"
"net"
"os"
"strings"
"testing"
"time"
"golang.org/x/crypto/ssh/knownhosts"
"gitea.home.hrajfrisbee.cz/kacerr/egress-proxies-operator/internal/provider"
)
// sshFixture is one in-process server with a client key it accepts and a
// known_hosts file naming its host key.
type sshFixture struct {
server *testSSHServer
socket string
cfg provider.KVMConfig
host provider.KVMHost
}
func newSSHFixture(t *testing.T) *sshFixture {
t.Helper()
dir := t.TempDir()
socket := startEchoSocket(t)
keyPath, pub := writeKeyFile(t, dir, "id_ed25519")
server := startSSHServer(t, pub, socket)
return &sshFixture{
server: server,
socket: socket,
cfg: provider.KVMConfig{
SSHKeyFile: keyPath,
KnownHostsFile: writeKnownHosts(t, dir, "known_hosts", server.addr, server.hostKey),
},
host: provider.KVMHost{Name: "beelink", Address: server.addr, User: "virt", Socket: socket},
}
}
const dialTestTimeout = 5 * time.Second
func TestDial_forwardsUnixSocket(t *testing.T) {
t.Parallel()
fx := newSSHFixture(t)
for round := range 2 {
sock, err := dialSSHSocket(context.Background(), fx.host, fx.cfg, dialTestTimeout)
if err != nil {
t.Fatalf("round %d: dial error = %v", round, err)
}
msg := []byte("ping over libvirt-sock")
if _, err := sock.Write(msg); err != nil {
t.Fatalf("round %d: write: %v", round, err)
}
buf := make([]byte, len(msg))
_ = sock.SetReadDeadline(time.Now().Add(dialTestTimeout))
if _, err := readFull(sock, buf); err != nil {
t.Fatalf("round %d: read: %v", round, err)
}
if string(buf) != string(msg) {
t.Errorf("round %d: echoed %q, want %q", round, buf, msg)
}
if err := sock.Close(); err != nil {
t.Errorf("round %d: close error = %v", round, err)
}
if err := sock.Close(); err != nil {
t.Errorf("round %d: second close error = %v, want nil", round, err)
}
}
for _, p := range fx.server.requestedPaths() {
if p != fx.socket {
t.Errorf("server saw channel for %q, want %q", p, fx.socket)
}
}
}
func readFull(sock *sshSocket, buf []byte) (int, error) {
n := 0
for n < len(buf) {
m, err := sock.Read(buf[n:])
n += m
if err != nil {
return n, err
}
}
return n, nil
}
func TestDial_unknownHostKeyRejected(t *testing.T) {
t.Parallel()
fx := newSSHFixture(t)
fx.cfg.KnownHostsFile = writeKnownHosts(t, t.TempDir(), "known_hosts", "other.example:22", fx.server.hostKey)
_, err := dialSSHSocket(context.Background(), fx.host, fx.cfg, dialTestTimeout)
assertDialFailure(t, err, stageSSH, provider.ErrPermanent)
var keyErr *knownhosts.KeyError
if !errors.As(err, &keyErr) {
t.Fatalf("error = %v, want *knownhosts.KeyError", err)
}
if len(keyErr.Want) != 0 {
t.Errorf("KeyError.Want = %v, want empty (host unknown)", keyErr.Want)
}
}
func TestDial_mismatchedHostKeyRejected(t *testing.T) {
t.Parallel()
fx := newSSHFixture(t)
fx.cfg.KnownHostsFile = writeKnownHosts(t, t.TempDir(), "known_hosts", fx.server.addr, otherPublicKey(t))
_, err := dialSSHSocket(context.Background(), fx.host, fx.cfg, dialTestTimeout)
assertDialFailure(t, err, stageSSH, provider.ErrPermanent)
var keyErr *knownhosts.KeyError
if !errors.As(err, &keyErr) {
t.Fatalf("error = %v, want *knownhosts.KeyError", err)
}
if len(keyErr.Want) == 0 {
t.Error("KeyError.Want empty, want the mismatching known key")
}
}
func TestDial_insecureIgnoreHostKeyAccepts(t *testing.T) {
t.Parallel()
fx := newSSHFixture(t)
fx.cfg.KnownHostsFile = ""
fx.cfg.InsecureIgnoreHostKey = true
sock, err := dialSSHSocket(context.Background(), fx.host, fx.cfg, dialTestTimeout)
if err != nil {
t.Fatalf("dial error = %v, want nil with host keys ignored", err)
}
_ = sock.Close()
}
func TestDial_missingKeyFileIsTransient(t *testing.T) {
t.Parallel()
fx := newSSHFixture(t)
fx.cfg.SSHKeyFile = fx.cfg.SSHKeyFile + ".missing"
_, err := dialSSHSocket(context.Background(), fx.host, fx.cfg, dialTestTimeout)
assertDialFailure(t, err, stageCredentials, provider.ErrTransient)
if !errors.Is(err, fs.ErrNotExist) {
t.Errorf("error = %v, want fs.ErrNotExist", err)
}
if !strings.Contains(err.Error(), fx.cfg.SSHKeyFile) {
t.Errorf("error = %q, want the path in it", err)
}
}
func TestDial_missingKnownHostsIsTransient(t *testing.T) {
t.Parallel()
fx := newSSHFixture(t)
fx.cfg.KnownHostsFile = fx.cfg.KnownHostsFile + ".missing"
_, err := dialSSHSocket(context.Background(), fx.host, fx.cfg, dialTestTimeout)
assertDialFailure(t, err, stageCredentials, provider.ErrTransient)
if !errors.Is(err, fs.ErrNotExist) {
t.Errorf("error = %v, want fs.ErrNotExist", err)
}
}
func TestDial_unparsableKeyIsPermanent(t *testing.T) {
t.Parallel()
fx := newSSHFixture(t)
fx.cfg.SSHKeyFile = writeFile(t, t.TempDir(), "id_bad", "this is not a key\n")
_, err := dialSSHSocket(context.Background(), fx.host, fx.cfg, dialTestTimeout)
assertDialFailure(t, err, stageCredentials, provider.ErrPermanent)
if !strings.Contains(err.Error(), "parsing ssh key file "+fx.cfg.SSHKeyFile) {
t.Errorf("error = %q, want 'parsing ssh key file <path>'", err)
}
}
func TestDial_unparsableKnownHostsIsPermanent(t *testing.T) {
t.Parallel()
fx := newSSHFixture(t)
fx.cfg.KnownHostsFile = writeFile(t, t.TempDir(), "known_hosts", "host.example ssh-ed25519 not-base64!\n")
_, err := dialSSHSocket(context.Background(), fx.host, fx.cfg, dialTestTimeout)
assertDialFailure(t, err, stageCredentials, provider.ErrPermanent)
if !strings.Contains(err.Error(), "parsing known_hosts file "+fx.cfg.KnownHostsFile) {
t.Errorf("error = %q, want 'parsing known_hosts file <path>'", err)
}
}
func TestDial_wrongKeyIsUnauthorized(t *testing.T) {
t.Parallel()
fx := newSSHFixture(t)
fx.cfg.SSHKeyFile, _ = writeKeyFile(t, t.TempDir(), "id_other")
_, err := dialSSHSocket(context.Background(), fx.host, fx.cfg, dialTestTimeout)
assertDialFailure(t, err, stageSSH, provider.ErrPermanent)
if !strings.Contains(err.Error(), sshAuthFailed) {
t.Errorf("error = %q, want %q in it", err, sshAuthFailed)
}
}
func TestDial_handshakeTimeout(t *testing.T) {
t.Parallel()
fx := newSSHFixture(t)
fx.host.Address = silentListener(t)
const budget = 200 * time.Millisecond
start := time.Now()
_, err := dialSSHSocket(context.Background(), fx.host, fx.cfg, budget)
elapsed := time.Since(start)
assertDialFailure(t, err, stageSSH, provider.ErrTransient)
if !errors.Is(err, context.DeadlineExceeded) {
t.Errorf("error = %v, want context.DeadlineExceeded", err)
}
if elapsed > budget+time.Second {
t.Errorf("dial took %v, want about the %v budget", elapsed, budget)
}
}
func TestDial_cancelledContext(t *testing.T) {
t.Parallel()
fx := newSSHFixture(t)
fx.host.Address = silentListener(t)
ctx, cancel := context.WithCancel(context.Background())
go func() {
time.Sleep(20 * time.Millisecond)
cancel()
}()
start := time.Now()
_, err := dialSSHSocket(ctx, fx.host, fx.cfg, dialTestTimeout)
if time.Since(start) > time.Second {
t.Errorf("dial took %v after cancel", time.Since(start))
}
assertDialFailure(t, err, stageSSH, provider.ErrTransient)
if !errors.Is(err, context.Canceled) {
t.Errorf("error = %v, want context.Canceled", err)
}
}
func TestDial_refusedIsTransient(t *testing.T) {
t.Parallel()
fx := newSSHFixture(t)
fx.host.Address = closedPort(t)
_, err := dialSSHSocket(context.Background(), fx.host, fx.cfg, dialTestTimeout)
assertDialFailure(t, err, stageTCP, provider.ErrTransient)
var opErr *net.OpError
if !errors.As(err, &opErr) {
t.Errorf("error = %v, want *net.OpError", err)
}
}
func TestDial_socketRefusedIsSocketStage(t *testing.T) {
t.Parallel()
fx := newSSHFixture(t)
fx.host.Socket = "/run/nonexistent/libvirt-sock"
_, err := dialSSHSocket(context.Background(), fx.host, fx.cfg, dialTestTimeout)
assertDialFailure(t, err, stageSocket, provider.ErrTransient)
if !strings.Contains(err.Error(), fx.host.Socket) {
t.Errorf("error = %q, want the socket path in it", err)
}
}
func TestSSH_keyMaterialNeverInErrorsOrLogs(t *testing.T) {
t.Parallel()
const keySentinel = "SENTINEL-KEY-4f9c2b"
const hostsSentinel = "SENTINEL-HOSTS-8a1d7e"
fx := newSSHFixture(t)
dir := t.TempDir()
tests := []struct {
name string
cfg func(c *provider.KVMConfig)
}{
{
name: "bad key file",
cfg: func(c *provider.KVMConfig) {
c.SSHKeyFile = writeFile(t, dir, "id_bad", "-----BEGIN NOT A KEY-----\n"+keySentinel+"\n-----END NOT A KEY-----\n")
},
},
{
name: "bad known_hosts file",
cfg: func(c *provider.KVMConfig) {
c.KnownHostsFile = writeFile(t, dir, "known_hosts_bad", "beelink ssh-ed25519 "+hostsSentinel+"\n")
},
},
{
name: "good key, host unknown",
cfg: func(c *provider.KVMConfig) {
c.KnownHostsFile = writeFile(t, dir, "known_hosts_other", knownhosts.Line([]string{"other.example:22"}, fx.server.hostKey)+"\n")
},
},
}
keyBytes := readFileT(t, fx.cfg.SSHKeyFile)
for _, tc := range tests {
t.Run(tc.name, func(t *testing.T) {
t.Parallel()
cfg := fx.cfg
tc.cfg(&cfg)
_, err := dialSSHSocket(context.Background(), fx.host, cfg, dialTestTimeout)
if err == nil {
t.Fatal("dial succeeded, want an error")
}
msg := err.Error()
for _, s := range []string{keySentinel, hostsSentinel} {
if strings.Contains(msg, s) {
t.Errorf("error %q contains sentinel %q", msg, s)
}
}
if strings.Contains(msg, keyBytes) || strings.Contains(msg, "PRIVATE KEY") {
t.Errorf("error %q contains key material", msg)
}
})
}
}
func readFileT(t *testing.T, path string) string {
t.Helper()
b, err := os.ReadFile(path)
if err != nil {
t.Fatalf("reading %s: %v", path, err)
}
return strings.TrimSpace(string(b))
}
func assertDialFailure(t *testing.T, err error, wantStage dialStage, wantClass error) {
t.Helper()
if err == nil {
t.Fatal("dial succeeded, want an error")
}
stage, ok := stageOf(err)
if !ok {
t.Fatalf("error %v is not a *dialError", err)
}
if stage != wantStage {
t.Errorf("stage = %q, want %q (error: %v)", stage, wantStage, err)
}
if got := classify("get", err); !errors.Is(got, wantClass) {
t.Errorf("classify() = %v, want %v (error: %v)", got, wantClass, err)
}
}
func TestDialError_unwrapsToCause(t *testing.T) {
t.Parallel()
cause := &knownhosts.KeyError{}
err := &dialError{stage: stageSSH, host: "beelink", err: cause}
if !errors.Is(err, cause) {
t.Error("errors.Is(dialError, cause) = false")
}
if !strings.Contains(err.Error(), "beelink") || !strings.Contains(err.Error(), "(ssh)") {
t.Errorf("Error() = %q, want the host and the stage", err)
}
if _, ok := stageOf(errors.New("plain")); ok {
t.Error("stageOf(plain error) reported a stage")
}
}
internal/provider/kvm2/sshserver_test.go
+289
View File
@@ -0,0 +1,289 @@
package kvm2
import (
"bytes"
"crypto/ed25519"
"crypto/rand"
"encoding/pem"
"errors"
"io"
"net"
"os"
"path/filepath"
"sync"
"testing"
"golang.org/x/crypto/ssh"
"golang.org/x/crypto/ssh/knownhosts"
)
// testSSHServer is an in-process x/crypto/ssh server that accepts one
// authorised public key and forwards direct-streamlocal@openssh.com channels
// for socketPath to that unix socket, rejecting every other channel.
type testSSHServer struct {
addr string
hostKey ssh.PublicKey
socketPath string
listener net.Listener
mu sync.Mutex
conns []net.Conn
wg sync.WaitGroup
stopOnce sync.Once
reqMu sync.Mutex
requested []string
}
func startSSHServer(t *testing.T, authorized ssh.PublicKey, socketPath string) *testSSHServer {
t.Helper()
_, hostPriv, err := ed25519.GenerateKey(rand.Reader)
if err != nil {
t.Fatalf("generating host key: %v", err)
}
hostSigner, err := ssh.NewSignerFromKey(hostPriv)
if err != nil {
t.Fatalf("host signer: %v", err)
}
cfg := &ssh.ServerConfig{
PublicKeyCallback: func(_ ssh.ConnMetadata, key ssh.PublicKey) (*ssh.Permissions, error) {
if bytes.Equal(key.Marshal(), authorized.Marshal()) {
return &ssh.Permissions{}, nil
}
return nil, errors.New("key not authorised")
},
}
cfg.AddHostKey(hostSigner)
ln, err := net.Listen("tcp", "127.0.0.1:0")
if err != nil {
t.Fatalf("listen: %v", err)
}
s := &testSSHServer{addr: ln.Addr().String(), hostKey: hostSigner.PublicKey(), socketPath: socketPath, listener: ln}
s.wg.Add(1)
go s.serve(cfg)
t.Cleanup(s.stop)
return s
}
func (s *testSSHServer) serve(cfg *ssh.ServerConfig) {
defer s.wg.Done()
for {
conn, err := s.listener.Accept()
if err != nil {
return
}
s.mu.Lock()
s.conns = append(s.conns, conn)
s.mu.Unlock()
s.wg.Add(1)
go s.handle(conn, cfg)
}
}
func (s *testSSHServer) handle(conn net.Conn, cfg *ssh.ServerConfig) {
defer s.wg.Done()
sc, chans, reqs, err := ssh.NewServerConn(conn, cfg)
if err != nil {
_ = conn.Close()
return
}
defer sc.Close()
go ssh.DiscardRequests(reqs)
for nc := range chans {
if nc.ChannelType() != "direct-streamlocal@openssh.com" {
_ = nc.Reject(ssh.UnknownChannelType, "only direct-streamlocal@openssh.com is served")
continue
}
var open struct {
SocketPath string
Reserved0 string
Reserved1 uint32
}
if err := ssh.Unmarshal(nc.ExtraData(), &open); err != nil {
_ = nc.Reject(ssh.ConnectionFailed, "bad direct-streamlocal payload")
continue
}
s.reqMu.Lock()
s.requested = append(s.requested, open.SocketPath)
s.reqMu.Unlock()
if open.SocketPath != s.socketPath {
_ = nc.Reject(ssh.ConnectionFailed, "no such socket")
continue
}
target, err := net.Dial("unix", s.socketPath)
if err != nil {
_ = nc.Reject(ssh.ConnectionFailed, "socket unavailable")
continue
}
ch, chReqs, err := nc.Accept()
if err != nil {
_ = target.Close()
continue
}
go ssh.DiscardRequests(chReqs)
s.wg.Add(1)
go func() {
defer s.wg.Done()
proxy(ch, target)
}()
}
}
func proxy(ch ssh.Channel, target net.Conn) {
var wg sync.WaitGroup
wg.Add(2)
go func() {
defer wg.Done()
_, _ = io.Copy(target, ch)
_ = target.Close()
}()
go func() {
defer wg.Done()
_, _ = io.Copy(ch, target)
_ = ch.Close()
}()
wg.Wait()
}
func (s *testSSHServer) stop() {
s.stopOnce.Do(func() {
_ = s.listener.Close()
s.mu.Lock()
for _, c := range s.conns {
_ = c.Close()
}
s.mu.Unlock()
s.wg.Wait()
})
}
func (s *testSSHServer) requestedPaths() []string {
s.reqMu.Lock()
defer s.reqMu.Unlock()
return append([]string(nil), s.requested...)
}
// startEchoSocket listens on a unix socket that echoes what it reads. The
// path is kept short: unix socket paths are limited to about 100 bytes and
// t.TempDir() names are long.
func startEchoSocket(t *testing.T) string {
t.Helper()
dir, err := os.MkdirTemp("", "kvm2")
if err != nil {
t.Fatalf("temp dir: %v", err)
}
t.Cleanup(func() { _ = os.RemoveAll(dir) })
path := filepath.Join(dir, "libvirt-sock")
ln, err := net.Listen("unix", path)
if err != nil {
t.Fatalf("listen unix: %v", err)
}
t.Cleanup(func() { _ = ln.Close() })
go func() {
for {
c, err := ln.Accept()
if err != nil {
return
}
go func() {
_, _ = io.Copy(c, c)
_ = c.Close()
}()
}
}()
return path
}
// silentListener accepts TCP connections and never speaks, for handshake
// timeouts. It returns the address.
func silentListener(t *testing.T) string {
t.Helper()
ln, err := net.Listen("tcp", "127.0.0.1:0")
if err != nil {
t.Fatalf("listen: %v", err)
}
var conns []net.Conn
var mu sync.Mutex
go func() {
for {
c, err := ln.Accept()
if err != nil {
return
}
mu.Lock()
conns = append(conns, c)
mu.Unlock()
}
}()
t.Cleanup(func() {
_ = ln.Close()
mu.Lock()
defer mu.Unlock()
for _, c := range conns {
_ = c.Close()
}
})
return ln.Addr().String()
}
// closedPort returns an address nothing listens on.
func closedPort(t *testing.T) string {
t.Helper()
ln, err := net.Listen("tcp", "127.0.0.1:0")
if err != nil {
t.Fatalf("listen: %v", err)
}
addr := ln.Addr().String()
_ = ln.Close()
return addr
}
// writeKeyFile generates an ed25519 key, writes it in OpenSSH format and
// returns the path and the public half.
func writeKeyFile(t *testing.T, dir, name string) (string, ssh.PublicKey) {
t.Helper()
pub, priv, err := ed25519.GenerateKey(rand.Reader)
if err != nil {
t.Fatalf("generating key: %v", err)
}
block, err := ssh.MarshalPrivateKey(priv, "kvm2 test")
if err != nil {
t.Fatalf("marshalling key: %v", err)
}
path := writeFile(t, dir, name, string(pem.EncodeToMemory(block)))
sshPub, err := ssh.NewPublicKey(pub)
if err != nil {
t.Fatalf("public key: %v", err)
}
return path, sshPub
}
// writeKnownHosts writes one known_hosts line for addr and key.
func writeKnownHosts(t *testing.T, dir, name, addr string, key ssh.PublicKey) string {
t.Helper()
return writeFile(t, dir, name, knownhosts.Line([]string{addr}, key)+"\n")
}
func writeFile(t *testing.T, dir, name, content string) string {
t.Helper()
path := filepath.Join(dir, name)
if err := os.WriteFile(path, []byte(content), 0o600); err != nil {
t.Fatalf("writing %s: %v", path, err)
}
return path
}
// otherPublicKey returns a public key unrelated to any file, for mismatch
// cases.
func otherPublicKey(t *testing.T) ssh.PublicKey {
t.Helper()
pub, _, err := ed25519.GenerateKey(rand.Reader)
if err != nil {
t.Fatalf("generating key: %v", err)
}
sshPub, err := ssh.NewPublicKey(pub)
if err != nil {
t.Fatalf("public key: %v", err)
}
return sshPub
}
internal/provider/kvm2/state.go
+49
View File
@@ -0,0 +1,49 @@
package kvm2
import (
"net/netip"
"github.com/digitalocean/go-libvirt"
"gitea.home.hrajfrisbee.cz/kacerr/egress-proxies-operator/internal/provider"
)
// mapState turns a libvirt domain state plus the discovered address into the
// contract's state. nostate means libvirt could not tell, so the next poll
// decides; an active domain is Running only once it has an address; every
// inactive state, and anything unknown, is Stopped, which the reconciler
// answers by replacing the instance.
func mapState(s libvirt.DomainState, ip string) provider.InstanceState {
switch {
case s == libvirt.DomainNostate:
return provider.StateProvisioning
case isActive(s) && ip == "":
return provider.StateProvisioning
case isActive(s):
return provider.StateRunning
default:
return provider.StateStopped
}
}
func isActive(s libvirt.DomainState) bool {
return s == libvirt.DomainRunning || s == libvirt.DomainBlocked
}
// firstIPv4 picks the first usable IPv4 address from the guest's interfaces:
// not loopback, not link-local, not unspecified. Empty means no address yet.
func firstIPv4(ifaces []libvirt.DomainInterface) string {
for _, iface := range ifaces {
for _, a := range iface.Addrs {
if libvirt.IPAddrType(a.Type) != libvirt.IPAddrTypeIpv4 {
continue
}
addr, err := netip.ParseAddr(a.Addr)
if err != nil || !addr.Is4() || addr.IsLoopback() || addr.IsLinkLocalUnicast() || addr.IsUnspecified() {
continue
}
return addr.String()
}
}
return ""
}
internal/provider/kvm2/state_test.go
+92
View File
@@ -0,0 +1,92 @@
package kvm2
import (
"testing"
"github.com/digitalocean/go-libvirt"
"gitea.home.hrajfrisbee.cz/kacerr/egress-proxies-operator/internal/provider"
)
func TestMapState_table(t *testing.T) {
t.Parallel()
tests := []struct {
state libvirt.DomainState
noIP provider.InstanceState
withIP provider.InstanceState
display string
}{
{libvirt.DomainNostate, provider.StateProvisioning, provider.StateProvisioning, "nostate"},
{libvirt.DomainRunning, provider.StateProvisioning, provider.StateRunning, "running"},
{libvirt.DomainBlocked, provider.StateProvisioning, provider.StateRunning, "blocked"},
{libvirt.DomainPaused, provider.StateStopped, provider.StateStopped, "paused"},
{libvirt.DomainShutdown, provider.StateStopped, provider.StateStopped, "shutdown"},
{libvirt.DomainShutoff, provider.StateStopped, provider.StateStopped, "shutoff"},
{libvirt.DomainCrashed, provider.StateStopped, provider.StateStopped, "crashed"},
{libvirt.DomainPmsuspended, provider.StateStopped, provider.StateStopped, "pmsuspended"},
{libvirt.DomainState(99), provider.StateStopped, provider.StateStopped, "unknown code 99"},
}
for _, tc := range tests {
t.Run(tc.display, func(t *testing.T) {
t.Parallel()
if got := mapState(tc.state, ""); got != tc.noIP {
t.Errorf("mapState(%s, \"\") = %s, want %s", tc.display, got, tc.noIP)
}
if got := mapState(tc.state, "10.0.0.5"); got != tc.withIP {
t.Errorf("mapState(%s, ip) = %s, want %s", tc.display, got, tc.withIP)
}
})
}
}
func TestIsActive(t *testing.T) {
t.Parallel()
for s, want := range map[libvirt.DomainState]bool{
libvirt.DomainRunning: true, libvirt.DomainBlocked: true,
libvirt.DomainNostate: false, libvirt.DomainPaused: false, libvirt.DomainShutoff: false, libvirt.DomainState(99): false,
} {
if got := isActive(s); got != want {
t.Errorf("isActive(%d) = %v, want %v", s, got, want)
}
}
}
func addr(typ libvirt.IPAddrType, a string) libvirt.DomainIPAddr {
return libvirt.DomainIPAddr{Type: int32(typ), Addr: a}
}
func TestFirstIPv4_filters(t *testing.T) {
t.Parallel()
v4 := libvirt.IPAddrTypeIpv4
v6 := libvirt.IPAddrTypeIpv6
tests := []struct {
name string
ifaces []libvirt.DomainInterface
want string
}{
{name: "empty list", ifaces: nil, want: ""},
{name: "loopback only", ifaces: []libvirt.DomainInterface{{Name: "lo", Addrs: []libvirt.DomainIPAddr{addr(v4, "127.0.0.1")}}}, want: ""},
{name: "link-local only", ifaces: []libvirt.DomainInterface{{Name: "eth0", Addrs: []libvirt.DomainIPAddr{addr(v4, "169.254.10.1")}}}, want: ""},
{name: "ipv6 only", ifaces: []libvirt.DomainInterface{{Name: "eth0", Addrs: []libvirt.DomainIPAddr{addr(v6, "2001:db8::1"), addr(v6, "fe80::1")}}}, want: ""},
{name: "unspecified", ifaces: []libvirt.DomainInterface{{Name: "eth0", Addrs: []libvirt.DomainIPAddr{addr(v4, "0.0.0.0")}}}, want: ""},
{name: "unparsable", ifaces: []libvirt.DomainInterface{{Name: "eth0", Addrs: []libvirt.DomainIPAddr{addr(v4, "not-an-ip")}}}, want: ""},
{name: "ipv4 literal tagged as ipv6 is skipped", ifaces: []libvirt.DomainInterface{{Name: "eth0", Addrs: []libvirt.DomainIPAddr{addr(v6, "10.0.0.7")}}}, want: ""},
{name: "ipv6 literal tagged as ipv4 is skipped", ifaces: []libvirt.DomainInterface{{Name: "eth0", Addrs: []libvirt.DomainIPAddr{addr(v4, "2001:db8::1")}}}, want: ""},
{
name: "good one after bad ones across interfaces",
ifaces: []libvirt.DomainInterface{
{Name: "lo", Addrs: []libvirt.DomainIPAddr{addr(v4, "127.0.0.1")}},
{Name: "eth0", Addrs: []libvirt.DomainIPAddr{addr(v6, "fe80::5054:ff:fe12:3456"), addr(v4, "169.254.1.1"), addr(v4, "bogus"), addr(v4, "192.168.1.42"), addr(v4, "192.168.1.43")}},
},
want: "192.168.1.42",
},
}
for _, tc := range tests {
t.Run(tc.name, func(t *testing.T) {
t.Parallel()
if got := firstIPv4(tc.ifaces); got != tc.want {
t.Errorf("firstIPv4() = %q, want %q", got, tc.want)
}
})
}
}
internal/provider/kvm2/volume.go
+102
View File
@@ -0,0 +1,102 @@
package kvm2
import (
"crypto/sha256"
"encoding/hex"
"encoding/xml"
"gitea.home.hrajfrisbee.cz/kacerr/egress-proxies-operator/internal/provider"
)
// Volume names in the host's pool (spec entries 4 and 10). The per-VM
// volumes derive from the domain name; the base image from its catalogue
// key or, for a URL image, from a hash of the URL.
func overlayVolumeName(name string) string { return name + ".qcow2" }
func seedVolumeName(name string) string { return name + "-seed.iso" }
// baseVolumeName carries the provider entry because the base is shared by
// every Proxy on a host but must not be shared by two entries: the kvm
// provider builds base-<key>.<format> in the same pool from the same
// catalogue shape, and two kvm2 entries pointing one key at different URLs
// would otherwise agree on the name and disagree on the contents.
func baseVolumeName(entry, key, format string) string {
return "base-" + entry + "-" + key + "." + format
}
func markerVolumeName(baseName string) string { return baseName + ".sha256" }
// urlImageKey stands in for the catalogue key of an image given as a URL:
// the first 16 hex characters of sha256(url).
func urlImageKey(url string) string {
sum := sha256.Sum256([]byte(url))
return hex.EncodeToString(sum[:])[:16]
}
// volumeSpec is what StorageVolCreateXML needs. BackingPath and
// BackingFormat are set for an overlay only.
type volumeSpec struct {
Name string
CapacityBytes int64
Format string
BackingPath string
BackingFormat string
}
type volumeXML struct {
XMLName xml.Name `xml:"volume"`
Type string `xml:"type,attr"`
Name string `xml:"name"`
Capacity volumeCapacityXML `xml:"capacity"`
Target volumeTargetXML `xml:"target"`
BackingStore *backingStoreXML `xml:"backingStore"`
}
type volumeCapacityXML struct {
Unit string `xml:"unit,attr"`
Value int64 `xml:",chardata"`
}
type volumeTargetXML struct {
Format volumeFormatXML `xml:"format"`
}
type volumeFormatXML struct {
Type string `xml:"type,attr"`
}
type backingStoreXML struct {
Path string `xml:"path"`
Format volumeFormatXML `xml:"format"`
}
func validVolumeFormat(f string) bool {
return f == provider.KVMFormatQcow2 || f == provider.KVMFormatRaw
}
func buildVolumeXML(v volumeSpec) (string, error) {
if v.Name == "" {
return "", permanentf("volume XML: name is empty")
}
if v.CapacityBytes <= 0 {
return "", permanentf("volume XML for %q: capacity must be positive, got %d", v.Name, v.CapacityBytes)
}
if !validVolumeFormat(v.Format) {
return "", permanentf("volume XML for %q: unknown format %q (want %s or %s)", v.Name, v.Format, provider.KVMFormatQcow2, provider.KVMFormatRaw)
}
x := volumeXML{
Type: "file",
Name: v.Name,
Capacity: volumeCapacityXML{Unit: "bytes", Value: v.CapacityBytes},
Target: volumeTargetXML{Format: volumeFormatXML{Type: v.Format}},
}
if v.BackingPath != "" {
if !validVolumeFormat(v.BackingFormat) {
return "", permanentf("volume XML for %q: unknown backing format %q (want %s or %s)", v.Name, v.BackingFormat, provider.KVMFormatQcow2, provider.KVMFormatRaw)
}
x.BackingStore = &backingStoreXML{Path: v.BackingPath, Format: volumeFormatXML{Type: v.BackingFormat}}
}
out, err := xml.MarshalIndent(x, "", " ")
if err != nil {
return "", permanentf("volume XML for %q: %v", v.Name, err)
}
return string(out), nil
}
internal/provider/kvm2/volume_test.go
+156
View File
@@ -0,0 +1,156 @@
package kvm2
import (
"encoding/xml"
"errors"
"regexp"
"testing"
"gitea.home.hrajfrisbee.cz/kacerr/egress-proxies-operator/internal/provider"
)
type checkedVolume struct {
Type string `xml:"type,attr"`
Name string `xml:"name"`
Capacity struct {
Unit string `xml:"unit,attr"`
Value int64 `xml:",chardata"`
} `xml:"capacity"`
Target struct {
Format struct {
Type string `xml:"type,attr"`
} `xml:"format"`
} `xml:"target"`
BackingStores []struct {
Path string `xml:"path"`
Format struct {
Type string `xml:"type,attr"`
} `xml:"format"`
} `xml:"backingStore"`
}
func mustBuildVolume(t *testing.T, v volumeSpec) checkedVolume {
t.Helper()
out, err := buildVolumeXML(v)
if err != nil {
t.Fatalf("buildVolumeXML: %v", err)
}
var c checkedVolume
if err := xml.Unmarshal([]byte(out), &c); err != nil {
t.Fatalf("unmarshalling produced XML: %v\n%s", err, out)
}
return c
}
func TestVolumeXML_overlayBackingStore(t *testing.T) {
t.Parallel()
c := mustBuildVolume(t, volumeSpec{
Name: "proxy-abc.qcow2",
CapacityBytes: 10 << 30,
Format: provider.KVMFormatQcow2,
BackingPath: "/var/lib/libvirt/images/base-debian-13.qcow2",
BackingFormat: provider.KVMFormatQcow2,
})
if c.Type != "file" || c.Name != "proxy-abc.qcow2" {
t.Errorf("type=%q name=%q, want file/proxy-abc.qcow2", c.Type, c.Name)
}
if c.Capacity.Unit != "bytes" || c.Capacity.Value != 10<<30 {
t.Errorf("capacity = %d %s, want %d bytes", c.Capacity.Value, c.Capacity.Unit, 10<<30)
}
if c.Target.Format.Type != "qcow2" {
t.Errorf("target format = %q, want qcow2", c.Target.Format.Type)
}
if len(c.BackingStores) != 1 {
t.Fatalf("got %d backingStore elements, want 1", len(c.BackingStores))
}
bs := c.BackingStores[0]
if bs.Path != "/var/lib/libvirt/images/base-debian-13.qcow2" || bs.Format.Type != "qcow2" {
t.Errorf("backingStore = %+v, want the base path with qcow2 format", bs)
}
}
func TestVolumeXML_rawWithoutBacking(t *testing.T) {
t.Parallel()
c := mustBuildVolume(t, volumeSpec{Name: "proxy-abc-seed.iso", CapacityBytes: 385024, Format: provider.KVMFormatRaw})
if c.Target.Format.Type != "raw" {
t.Errorf("target format = %q, want raw", c.Target.Format.Type)
}
if c.Capacity.Value != 385024 || c.Capacity.Unit != "bytes" {
t.Errorf("capacity = %d %s, want 385024 bytes", c.Capacity.Value, c.Capacity.Unit)
}
if len(c.BackingStores) != 0 {
t.Errorf("got backingStore %+v, want none", c.BackingStores)
}
}
func TestVolumeXML_rejectsBadFormat(t *testing.T) {
t.Parallel()
tests := []struct {
name string
spec volumeSpec
}{
{name: "vmdk format", spec: volumeSpec{Name: "x", CapacityBytes: 1, Format: "vmdk"}},
{name: "empty format", spec: volumeSpec{Name: "x", CapacityBytes: 1}},
{name: "bad backing format", spec: volumeSpec{Name: "x", CapacityBytes: 1, Format: "qcow2", BackingPath: "/b", BackingFormat: "vdi"}},
{name: "backing path without format", spec: volumeSpec{Name: "x", CapacityBytes: 1, Format: "qcow2", BackingPath: "/b"}},
{name: "empty name", spec: volumeSpec{CapacityBytes: 1, Format: "qcow2"}},
{name: "zero capacity", spec: volumeSpec{Name: "x", Format: "qcow2"}},
}
for _, tc := range tests {
t.Run(tc.name, func(t *testing.T) {
t.Parallel()
out, err := buildVolumeXML(tc.spec)
if !errors.Is(err, provider.ErrPermanent) {
t.Errorf("err = %v, want ErrPermanent", err)
}
if out != "" {
t.Errorf("got XML %q alongside the error", out)
}
})
}
}
func TestVolumeNames(t *testing.T) {
t.Parallel()
tests := []struct {
name string
got string
want string
}{
{name: "overlay", got: overlayVolumeName("proxy-abc"), want: "proxy-abc.qcow2"},
{name: "seed", got: seedVolumeName("proxy-abc"), want: "proxy-abc-seed.iso"},
{name: "base qcow2", got: baseVolumeName("kvm2-home", "debian-13", provider.KVMFormatQcow2), want: "base-kvm2-home-debian-13.qcow2"},
{name: "base raw", got: baseVolumeName("kvm2-home", "alpine", provider.KVMFormatRaw), want: "base-kvm2-home-alpine.raw"},
{name: "marker", got: markerVolumeName("base-debian-13.qcow2"), want: "base-debian-13.qcow2.sha256"},
{name: "marker of url base", got: markerVolumeName(baseVolumeName("kvm2-home", urlImageKey("https://x/y.img"), "qcow2")), want: "base-kvm2-home-" + urlImageKey("https://x/y.img") + ".qcow2.sha256"},
// Two entries pointing one key at different images must not agree
// on a file name, and neither may collide with the kvm provider's
// base-<key>.<format> in the same pool.
{name: "another entry, same key", got: baseVolumeName("kvm2-lab", "debian-13", provider.KVMFormatQcow2), want: "base-kvm2-lab-debian-13.qcow2"},
}
for _, tc := range tests {
t.Run(tc.name, func(t *testing.T) {
t.Parallel()
if tc.got != tc.want {
t.Errorf("got %q, want %q", tc.got, tc.want)
}
})
}
t.Run("url key", func(t *testing.T) {
t.Parallel()
hex16 := regexp.MustCompile(`^[0-9a-f]{16}$`)
a := urlImageKey("https://cloud.debian.org/images/cloud/trixie/latest/debian-13-generic-amd64.qcow2")
b := urlImageKey("https://cloud.debian.org/images/cloud/trixie/latest/debian-13-generic-amd64.qcow2")
c := urlImageKey("https://cloud.debian.org/images/cloud/bookworm/latest/debian-12-generic-amd64.qcow2")
if !hex16.MatchString(a) {
t.Errorf("urlImageKey = %q, want 16 lowercase hex chars", a)
}
if a != b {
t.Errorf("urlImageKey not deterministic: %q vs %q", a, b)
}
if a == c {
t.Errorf("urlImageKey gives %q for two different URLs", a)
}
})
}
internal/provider/kvm2/xml.go
+256
View File
@@ -0,0 +1,256 @@
package kvm2
import (
"encoding/xml"
"time"
"gitea.home.hrajfrisbee.cz/kacerr/egress-proxies-operator/internal/provider"
)
// domainParams is everything buildDomainXML needs, already resolved: the
// paths are what StorageVolGetPath returned for the overlay and the seed.
type domainParams struct {
Name string
UID string
CreatedAt time.Time
VCPUs int32
MemoryMiB int64
Firmware string
Bridge string
DiskPath string
SeedPath string
DiskFormat string
}
const (
domainArch = "x86_64"
domainMachine = "q35"
guestAgentChannelName = "org.qemu.guest_agent.0"
)
type domainXML struct {
XMLName xml.Name `xml:"domain"`
Type string `xml:"type,attr"`
Name string `xml:"name"`
Memory domainMemoryXML `xml:"memory"`
VCPU int32 `xml:"vcpu"`
Metadata domainMetadataXML `xml:"metadata"`
OS domainOSXML `xml:"os"`
Features domainFeaturesXML `xml:"features"`
CPU domainCPUXML `xml:"cpu"`
OnPoweroff string `xml:"on_poweroff"`
OnReboot string `xml:"on_reboot"`
OnCrash string `xml:"on_crash"`
Devices domainDevicesXML `xml:"devices"`
}
type domainMemoryXML struct {
Unit string `xml:"unit,attr"`
Value int64 `xml:",chardata"`
}
type domainMetadataXML struct {
Instance instanceMetadata `xml:"instance"`
}
type domainOSXML struct {
// Firmware is the attribute firmware="efi"; FirmwareFeatures is the
// <firmware> child that turns secure boot off. Both are absent for BIOS.
Firmware string `xml:"firmware,attr,omitempty"`
Type domainOSTypeXML `xml:"type"`
FirmwareFeatures *domainFirmwareXML `xml:"firmware"`
}
type domainOSTypeXML struct {
Arch string `xml:"arch,attr"`
Machine string `xml:"machine,attr"`
Value string `xml:",chardata"`
}
type domainFirmwareXML struct {
Features []domainFirmwareFeatureXML `xml:"feature"`
}
type domainFirmwareFeatureXML struct {
Enabled string `xml:"enabled,attr"`
Name string `xml:"name,attr"`
}
type domainFeaturesXML struct {
ACPI *struct{} `xml:"acpi"`
APIC *struct{} `xml:"apic"`
}
type domainCPUXML struct {
Mode string `xml:"mode,attr"`
}
type domainDevicesXML struct {
Disks []domainDiskXML `xml:"disk"`
Interfaces []domainInterfaceXML `xml:"interface"`
Serials []domainSerialXML `xml:"serial"`
Consoles []domainConsoleXML `xml:"console"`
Channels []domainChannelXML `xml:"channel"`
RNGs []domainRNGXML `xml:"rng"`
}
type domainDiskXML struct {
Type string `xml:"type,attr"`
Device string `xml:"device,attr"`
Driver domainDiskDriverXML `xml:"driver"`
Source domainDiskSourceXML `xml:"source"`
Target domainDiskTargetXML `xml:"target"`
ReadOnly *struct{} `xml:"readonly"`
}
type domainDiskDriverXML struct {
Name string `xml:"name,attr"`
Type string `xml:"type,attr"`
}
type domainDiskSourceXML struct {
File string `xml:"file,attr"`
}
type domainDiskTargetXML struct {
Dev string `xml:"dev,attr"`
Bus string `xml:"bus,attr"`
}
type domainInterfaceXML struct {
Type string `xml:"type,attr"`
Source domainInterfaceSourceXML `xml:"source"`
Model domainModelXML `xml:"model"`
}
type domainInterfaceSourceXML struct {
Bridge string `xml:"bridge,attr"`
}
type domainModelXML struct {
Type string `xml:"type,attr"`
}
type domainSerialXML struct {
Type string `xml:"type,attr"`
Target domainPortXML `xml:"target"`
}
type domainPortXML struct {
Port int `xml:"port,attr"`
}
type domainConsoleXML struct {
Type string `xml:"type,attr"`
Target domainConsoleTargetXML `xml:"target"`
}
type domainConsoleTargetXML struct {
Type string `xml:"type,attr"`
Port int `xml:"port,attr"`
}
type domainChannelXML struct {
Type string `xml:"type,attr"`
Target domainChannelTargetXML `xml:"target"`
}
type domainChannelTargetXML struct {
Type string `xml:"type,attr"`
Name string `xml:"name,attr"`
}
type domainRNGXML struct {
Model string `xml:"model,attr"`
Backend domainRNGBackendXML `xml:"backend"`
}
type domainRNGBackendXML struct {
Model string `xml:"model,attr"`
Value string `xml:",chardata"`
}
// buildDomainXML renders the XML for DomainDefineXML. Both disks are
// type="file" with <source file>: AppArmor derives QEMU's profile from those
// paths, and type="volume" fails with permission denied on stock
// Debian/Ubuntu hosts. UEFI boots with secure boot off; libvirt picks the
// OVMF binary itself.
func buildDomainXML(p domainParams) (string, error) {
if p.Name == "" {
return "", permanentf("domain XML: name is empty")
}
if p.DiskPath == "" {
return "", permanentf("domain XML for %q: disk path is empty", p.Name)
}
if p.SeedPath == "" {
return "", permanentf("domain XML for %q: seed path is empty", p.Name)
}
if p.DiskFormat == "" {
p.DiskFormat = provider.KVMFormatQcow2
}
if p.DiskFormat != provider.KVMFormatQcow2 && p.DiskFormat != provider.KVMFormatRaw {
return "", permanentf("domain XML for %q: unknown disk format %q", p.Name, p.DiskFormat)
}
osx := domainOSXML{
Type: domainOSTypeXML{Arch: domainArch, Machine: domainMachine, Value: "hvm"},
}
switch p.Firmware {
case provider.KVMFirmwareUEFI:
osx.Firmware = "efi"
osx.FirmwareFeatures = &domainFirmwareXML{
Features: []domainFirmwareFeatureXML{{Enabled: "no", Name: "secure-boot"}},
}
case provider.KVMFirmwareBIOS:
default:
return "", permanentf("domain XML for %q: unknown firmware %q (want %s or %s)", p.Name, p.Firmware, provider.KVMFirmwareUEFI, provider.KVMFirmwareBIOS)
}
d := domainXML{
Type: "kvm",
Name: p.Name,
Memory: domainMemoryXML{Unit: "MiB", Value: p.MemoryMiB},
VCPU: p.VCPUs,
Metadata: domainMetadataXML{Instance: newInstanceMetadata(p.UID, p.CreatedAt)},
OS: osx,
Features: domainFeaturesXML{ACPI: &struct{}{}, APIC: &struct{}{}},
CPU: domainCPUXML{Mode: "host-passthrough"},
OnPoweroff: "destroy",
OnReboot: "restart",
OnCrash: "destroy",
Devices: domainDevicesXML{
Disks: []domainDiskXML{
{
Type: "file",
Device: "disk",
Driver: domainDiskDriverXML{Name: "qemu", Type: p.DiskFormat},
Source: domainDiskSourceXML{File: p.DiskPath},
Target: domainDiskTargetXML{Dev: "vda", Bus: "virtio"},
},
{
Type: "file",
Device: "cdrom",
Driver: domainDiskDriverXML{Name: "qemu", Type: provider.KVMFormatRaw},
Source: domainDiskSourceXML{File: p.SeedPath},
Target: domainDiskTargetXML{Dev: "sda", Bus: "sata"},
ReadOnly: &struct{}{},
},
},
Interfaces: []domainInterfaceXML{{
Type: "bridge",
Source: domainInterfaceSourceXML{Bridge: p.Bridge},
Model: domainModelXML{Type: "virtio"},
}},
Serials: []domainSerialXML{{Type: "pty", Target: domainPortXML{Port: 0}}},
Consoles: []domainConsoleXML{{Type: "pty", Target: domainConsoleTargetXML{Type: "serial", Port: 0}}},
Channels: []domainChannelXML{{
Type: "unix",
Target: domainChannelTargetXML{Type: "virtio", Name: guestAgentChannelName},
}},
RNGs: []domainRNGXML{{Model: "virtio", Backend: domainRNGBackendXML{Model: "random", Value: "/dev/urandom"}}},
},
}
out, err := xml.MarshalIndent(d, "", " ")
if err != nil {
return "", permanentf("domain XML for %q: %v", p.Name, err)
}
return string(out), nil
}
internal/provider/kvm2/xml_test.go
+309
View File
@@ -0,0 +1,309 @@
package kvm2
import (
"encoding/xml"
"errors"
"strings"
"testing"
"time"
"gitea.home.hrajfrisbee.cz/kacerr/egress-proxies-operator/internal/provider"
)
// checkedDomain is an independent view of the marshalled XML, so the tests
// assert on what libvirt would see rather than on the builder's own structs.
type checkedDomain struct {
Type string `xml:"type,attr"`
Name string `xml:"name"`
Memory struct {
Unit string `xml:"unit,attr"`
Value int64 `xml:",chardata"`
} `xml:"memory"`
VCPU int32 `xml:"vcpu"`
Metadata struct {
Inner string `xml:",innerxml"`
} `xml:"metadata"`
OS struct {
Firmware string `xml:"firmware,attr"`
Type struct {
Arch string `xml:"arch,attr"`
Machine string `xml:"machine,attr"`
Value string `xml:",chardata"`
} `xml:"type"`
FirmwareElems []struct {
Features []struct {
Enabled string `xml:"enabled,attr"`
Name string `xml:"name,attr"`
} `xml:"feature"`
} `xml:"firmware"`
} `xml:"os"`
Features struct {
ACPI []struct{} `xml:"acpi"`
APIC []struct{} `xml:"apic"`
} `xml:"features"`
CPU struct {
Mode string `xml:"mode,attr"`
} `xml:"cpu"`
OnPoweroff string `xml:"on_poweroff"`
OnReboot string `xml:"on_reboot"`
OnCrash string `xml:"on_crash"`
Devices struct {
Disks []struct {
Type string `xml:"type,attr"`
Device string `xml:"device,attr"`
Driver struct {
Name string `xml:"name,attr"`
Type string `xml:"type,attr"`
} `xml:"driver"`
Source struct {
File string `xml:"file,attr"`
Pool string `xml:"pool,attr"`
Vol string `xml:"volume,attr"`
} `xml:"source"`
Target struct {
Dev string `xml:"dev,attr"`
Bus string `xml:"bus,attr"`
} `xml:"target"`
ReadOnly []struct{} `xml:"readonly"`
} `xml:"disk"`
Interfaces []struct {
Type string `xml:"type,attr"`
Source struct {
Bridge string `xml:"bridge,attr"`
} `xml:"source"`
Model struct {
Type string `xml:"type,attr"`
} `xml:"model"`
} `xml:"interface"`
Serials []struct {
Type string `xml:"type,attr"`
} `xml:"serial"`
Consoles []struct {
Type string `xml:"type,attr"`
Target struct {
Type string `xml:"type,attr"`
} `xml:"target"`
} `xml:"console"`
Channels []struct {
Type string `xml:"type,attr"`
Target struct {
Type string `xml:"type,attr"`
Name string `xml:"name,attr"`
} `xml:"target"`
} `xml:"channel"`
RNGs []struct {
Model string `xml:"model,attr"`
Backend struct {
Model string `xml:"model,attr"`
Value string `xml:",chardata"`
} `xml:"backend"`
} `xml:"rng"`
Graphics []struct{} `xml:"graphics"`
} `xml:"devices"`
}
var testCreatedAt = time.Date(2026, 9, 16, 12, 0, 0, 0, time.UTC)
func testDomainParams() domainParams {
return domainParams{
Name: "proxy-abc",
UID: "0b1f9a2e-uid",
CreatedAt: testCreatedAt,
VCPUs: 2,
MemoryMiB: 2048,
Firmware: provider.KVMFirmwareUEFI,
Bridge: "br0",
DiskPath: "/var/lib/libvirt/images/proxy-abc.qcow2",
SeedPath: "/var/lib/libvirt/images/proxy-abc-seed.iso",
DiskFormat: provider.KVMFormatQcow2,
}
}
func mustBuildDomain(t *testing.T, p domainParams) (string, checkedDomain) {
t.Helper()
out, err := buildDomainXML(p)
if err != nil {
t.Fatalf("buildDomainXML: %v", err)
}
var d checkedDomain
if err := xml.Unmarshal([]byte(out), &d); err != nil {
t.Fatalf("unmarshalling produced XML: %v\n%s", err, out)
}
return out, d
}
func TestDomainXML_fileDisksWithResolvedPaths(t *testing.T) {
t.Parallel()
p := testDomainParams()
_, d := mustBuildDomain(t, p)
if d.Type != "kvm" || d.Name != p.Name {
t.Errorf("domain type=%q name=%q, want kvm/%s", d.Type, d.Name, p.Name)
}
if d.Memory.Unit != "MiB" || d.Memory.Value != p.MemoryMiB || d.VCPU != p.VCPUs {
t.Errorf("memory=%d %s vcpu=%d, want %d MiB / %d", d.Memory.Value, d.Memory.Unit, d.VCPU, p.MemoryMiB, p.VCPUs)
}
if len(d.Devices.Disks) != 2 {
t.Fatalf("got %d disks, want 2", len(d.Devices.Disks))
}
overlay, seed := d.Devices.Disks[0], d.Devices.Disks[1]
if overlay.Type != "file" || overlay.Device != "disk" {
t.Errorf("overlay type=%q device=%q, want file/disk", overlay.Type, overlay.Device)
}
if overlay.Source.File != p.DiskPath || overlay.Source.Pool != "" || overlay.Source.Vol != "" {
t.Errorf("overlay source = %+v, want file=%q only", overlay.Source, p.DiskPath)
}
if overlay.Driver.Name != "qemu" || overlay.Driver.Type != "qcow2" {
t.Errorf("overlay driver = %+v, want qemu/qcow2", overlay.Driver)
}
if overlay.Target.Dev != "vda" || overlay.Target.Bus != "virtio" {
t.Errorf("overlay target = %+v, want vda/virtio", overlay.Target)
}
if len(overlay.ReadOnly) != 0 {
t.Error("overlay is readonly")
}
if seed.Type != "file" || seed.Device != "cdrom" {
t.Errorf("seed type=%q device=%q, want file/cdrom", seed.Type, seed.Device)
}
if seed.Source.File != p.SeedPath || seed.Source.Pool != "" || seed.Source.Vol != "" {
t.Errorf("seed source = %+v, want file=%q only", seed.Source, p.SeedPath)
}
if seed.Driver.Name != "qemu" || seed.Driver.Type != "raw" {
t.Errorf("seed driver = %+v, want qemu/raw", seed.Driver)
}
if seed.Target.Dev != "sda" || seed.Target.Bus != "sata" {
t.Errorf("seed target = %+v, want sda/sata", seed.Target)
}
if len(seed.ReadOnly) != 1 {
t.Error("seed is not readonly")
}
if d.OnPoweroff != "destroy" || d.OnReboot != "restart" || d.OnCrash != "destroy" {
t.Errorf("lifecycle actions = %q/%q/%q, want destroy/restart/destroy", d.OnPoweroff, d.OnReboot, d.OnCrash)
}
}
func TestDomainXML_uefiDefaultSecureBootOff(t *testing.T) {
t.Parallel()
_, d := mustBuildDomain(t, testDomainParams())
if d.OS.Firmware != "efi" {
t.Errorf(`<os firmware> = %q, want "efi"`, d.OS.Firmware)
}
if len(d.OS.FirmwareElems) != 1 {
t.Fatalf("got %d <firmware> elements, want 1", len(d.OS.FirmwareElems))
}
feats := d.OS.FirmwareElems[0].Features
if len(feats) != 1 || feats[0].Name != "secure-boot" || feats[0].Enabled != "no" {
t.Errorf("firmware features = %+v, want one secure-boot enabled=no", feats)
}
if d.OS.Type.Value != "hvm" {
t.Errorf("os type = %q, want hvm", d.OS.Type.Value)
}
}
func TestDomainXML_biosEscape(t *testing.T) {
t.Parallel()
p := testDomainParams()
p.Firmware = provider.KVMFirmwareBIOS
out, d := mustBuildDomain(t, p)
if d.OS.Firmware != "" {
t.Errorf(`<os firmware> = %q, want absent`, d.OS.Firmware)
}
if len(d.OS.FirmwareElems) != 0 {
t.Errorf("got %d <firmware> elements, want none", len(d.OS.FirmwareElems))
}
if strings.Contains(out, "firmware") {
t.Errorf("BIOS domain XML still mentions firmware:\n%s", out)
}
if d.OS.Type.Value != "hvm" || d.OS.Type.Machine != "q35" {
t.Errorf("os type = %+v, want hvm on q35", d.OS.Type)
}
}
func TestDomainXML_metadataElement(t *testing.T) {
t.Parallel()
p := testDomainParams()
out, d := mustBuildDomain(t, p)
if !strings.Contains(out, `<instance xmlns="`+metadataURI+`">`) {
t.Errorf("domain XML lacks the namespaced instance element:\n%s", out)
}
m, err := parseInstanceMetadata(d.Metadata.Inner)
if err != nil {
t.Fatalf("parseInstanceMetadata(%q): %v", d.Metadata.Inner, err)
}
if !m.managed() {
t.Error("managed() = false, want true")
}
if m.UID != p.UID {
t.Errorf("UID = %q, want %q", m.UID, p.UID)
}
if m.CreatedAt != "2026-09-16T12:00:00Z" {
t.Errorf("CreatedAt = %q, want RFC3339 UTC", m.CreatedAt)
}
if !m.createdAt().Equal(p.CreatedAt) {
t.Errorf("createdAt() = %v, want %v", m.createdAt(), p.CreatedAt)
}
}
func TestDomainXML_guestAgentChannel(t *testing.T) {
t.Parallel()
p := testDomainParams()
_, d := mustBuildDomain(t, p)
if len(d.Devices.Channels) != 1 || d.Devices.Channels[0].Type != "unix" ||
d.Devices.Channels[0].Target.Type != "virtio" || d.Devices.Channels[0].Target.Name != "org.qemu.guest_agent.0" {
t.Errorf("channels = %+v, want one unix/virtio org.qemu.guest_agent.0", d.Devices.Channels)
}
if len(d.Devices.Interfaces) != 1 || d.Devices.Interfaces[0].Type != "bridge" ||
d.Devices.Interfaces[0].Source.Bridge != p.Bridge || d.Devices.Interfaces[0].Model.Type != "virtio" {
t.Errorf("interfaces = %+v, want one bridge %s with virtio model", d.Devices.Interfaces, p.Bridge)
}
if len(d.Devices.Serials) != 1 || d.Devices.Serials[0].Type != "pty" {
t.Errorf("serials = %+v, want one pty", d.Devices.Serials)
}
if len(d.Devices.Consoles) != 1 || d.Devices.Consoles[0].Type != "pty" || d.Devices.Consoles[0].Target.Type != "serial" {
t.Errorf("consoles = %+v, want one pty/serial", d.Devices.Consoles)
}
if len(d.Devices.RNGs) != 1 || d.Devices.RNGs[0].Model != "virtio" ||
d.Devices.RNGs[0].Backend.Model != "random" || d.Devices.RNGs[0].Backend.Value != "/dev/urandom" {
t.Errorf("rngs = %+v, want one virtio random /dev/urandom", d.Devices.RNGs)
}
if len(d.Devices.Graphics) != 0 {
t.Error("domain has <graphics>, want none")
}
if d.OS.Type.Arch != "x86_64" || d.OS.Type.Machine != "q35" {
t.Errorf("os type = %+v, want x86_64/q35", d.OS.Type)
}
if d.CPU.Mode != "host-passthrough" {
t.Errorf("cpu mode = %q, want host-passthrough", d.CPU.Mode)
}
if len(d.Features.ACPI) != 1 || len(d.Features.APIC) != 1 {
t.Errorf("features acpi=%d apic=%d, want one each", len(d.Features.ACPI), len(d.Features.APIC))
}
}
func TestDomainXML_rejectsBadInput(t *testing.T) {
t.Parallel()
tests := []struct {
name string
mutate func(*domainParams)
}{
{name: "unknown firmware", mutate: func(p *domainParams) { p.Firmware = "coreboot" }},
{name: "empty firmware", mutate: func(p *domainParams) { p.Firmware = "" }},
{name: "empty name", mutate: func(p *domainParams) { p.Name = "" }},
{name: "empty disk path", mutate: func(p *domainParams) { p.DiskPath = "" }},
{name: "empty seed path", mutate: func(p *domainParams) { p.SeedPath = "" }},
{name: "unknown disk format", mutate: func(p *domainParams) { p.DiskFormat = "vmdk" }},
}
for _, tc := range tests {
t.Run(tc.name, func(t *testing.T) {
t.Parallel()
p := testDomainParams()
tc.mutate(&p)
out, err := buildDomainXML(p)
if !errors.Is(err, provider.ErrPermanent) {
t.Errorf("err = %v, want ErrPermanent", err)
}
if out != "" {
t.Errorf("got XML %q alongside the error", out)
}
})
}
}
internal/provider/nocloud/seed.go
+130
View File
@@ -0,0 +1,130 @@
// Package nocloud builds cloud-init NoCloud seed ISOs in process, shared by
// the providers that attach a seed as a CD-ROM (kvm2, proxmox). It never
// parses or logs the payload.
package nocloud
import (
"bytes"
"errors"
"fmt"
"io"
"mime/multipart"
"net/textproto"
"strings"
"github.com/kdomanski/iso9660"
)
// SeedLabel is the volume label cloud-init's NoCloud datasource looks for.
const SeedLabel = "cidata"
// ErrEmptyName is returned by BuildSeedISO for an empty instance name.
var ErrEmptyName = errors.New("seed ISO: instance name is empty")
// GuestAgentScript is the vendor-data payload that gets qemu-guest-agent
// running so the provider can read the VM's address from the hypervisor. It is
// a shell script and not a cloud-config on purpose: cloud-init merges a
// vendor-data cloud-config under the user-data with mergers that keep the
// user-data's value for every top-level key and never append lists, so a
// Proxy with its own packages: would silently lose the agent. Scripts
// bypass the merge.
const GuestAgentScript = `#!/bin/sh
# Added by egress-proxies-operator: installs and starts qemu-guest-agent so
# the operator can learn this VM's IP address from the hypervisor.
# To leave this out, set installGuestAgent: false in the provider's config.
set -e
if command -v qemu-ga >/dev/null 2>&1 || [ -x /usr/sbin/qemu-ga ]; then
echo "egress-proxies-operator: qemu-guest-agent already installed"
elif command -v apt-get >/dev/null 2>&1; then
export DEBIAN_FRONTEND=noninteractive
apt-get update
apt-get install -y qemu-guest-agent
elif command -v dnf >/dev/null 2>&1; then
dnf install -y qemu-guest-agent
elif command -v yum >/dev/null 2>&1; then
yum install -y qemu-guest-agent
elif command -v zypper >/dev/null 2>&1; then
zypper --non-interactive install qemu-guest-agent
elif command -v apk >/dev/null 2>&1; then
apk add qemu-guest-agent
else
echo "egress-proxies-operator: no supported package manager found; install qemu-guest-agent by hand" >&2
exit 1
fi
if command -v systemctl >/dev/null 2>&1; then
systemctl enable --now qemu-guest-agent
else
rc-update add qemu-guest-agent default
rc-service qemu-guest-agent start
fi
`
// BuildSeedISO writes a NoCloud seed to w: user-data byte for byte,
// meta-data naming the instance, and vendor-data (an empty file when there
// is none, so cloud-init does not look elsewhere). iso9660.NewWriter stages
// the files in os.MkdirTemp("", ""), that is under os.TempDir(): $TMPDIR
// when set, else /tmp, the manager's 64Mi emptyDir. The staging dir is
// removed before returning. Nothing about the payload is logged here.
func BuildSeedISO(name, userData, vendorData string, w io.Writer) (err error) {
if name == "" {
return ErrEmptyName
}
iw, err := iso9660.NewWriter()
if err != nil {
return fmt.Errorf("creating seed ISO staging dir: %w", err)
}
defer func() {
if cerr := iw.Cleanup(); cerr != nil {
err = errors.Join(err, fmt.Errorf("removing seed ISO staging dir: %w", cerr))
}
}()
metaData := "instance-id: " + name + "\nlocal-hostname: " + name + "\n"
files := []struct{ name, content string }{
{"user-data", userData},
{"meta-data", metaData},
{"vendor-data", vendorData},
}
for _, f := range files {
if err := iw.AddFile(strings.NewReader(f.content), f.name); err != nil {
return fmt.Errorf("adding %s to seed ISO: %w", f.name, err)
}
}
if err := iw.WriteTo(w, SeedLabel); err != nil {
return fmt.Errorf("writing seed ISO: %w", err)
}
return nil
}
// BuildVendorData composes the seed's vendor-data from the configured
// document and the guest agent script. With both, they travel as separate
// parts of a MIME multipart, so cloud-init handles each by its own type; the
// MIME-Version header comes first because cloud-init detects multipart by
// finding "mime-version:" within the first 4096 bytes.
func BuildVendorData(configured string, installAgent bool) (string, error) {
switch {
case !installAgent:
return configured, nil
case strings.TrimSpace(configured) == "":
return GuestAgentScript, nil
}
var buf bytes.Buffer
mw := multipart.NewWriter(&buf)
fmt.Fprintf(&buf, "MIME-Version: 1.0\r\nContent-Type: multipart/mixed; boundary=%q\r\n\r\n", mw.Boundary())
parts := []struct{ contentType, body string }{
{"text/plain", configured},
{"text/x-shellscript", GuestAgentScript},
}
for _, p := range parts {
pw, err := mw.CreatePart(textproto.MIMEHeader{"Content-Type": {p.contentType}})
if err != nil {
return "", fmt.Errorf("creating vendor-data part %s: %w", p.contentType, err)
}
if _, err := io.WriteString(pw, p.body); err != nil {
return "", fmt.Errorf("writing vendor-data part %s: %w", p.contentType, err)
}
}
if err := mw.Close(); err != nil {
return "", fmt.Errorf("closing vendor-data multipart: %w", err)
}
return buf.String(), nil
}
internal/provider/nocloud/seed_test.go
+265
View File
@@ -0,0 +1,265 @@
package nocloud
import (
"bufio"
"bytes"
"errors"
"io"
"mime"
"mime/multipart"
"net/textproto"
"os"
"strings"
"testing"
"github.com/kdomanski/iso9660"
)
// readSeed builds a seed and returns its files by name plus the label.
func readSeed(t *testing.T, name, userData, vendorData string) (map[string]string, string) {
t.Helper()
var buf bytes.Buffer
if err := BuildSeedISO(name, userData, vendorData, &buf); err != nil {
t.Fatalf("BuildSeedISO: %v", err)
}
img, err := iso9660.OpenImage(bytes.NewReader(buf.Bytes()))
if err != nil {
t.Fatalf("OpenImage: %v", err)
}
root, err := img.RootDir()
if err != nil {
t.Fatalf("RootDir: %v", err)
}
children, err := root.GetChildren()
if err != nil {
t.Fatalf("GetChildren: %v", err)
}
files := make(map[string]string, len(children))
for _, c := range children {
if c.IsDir() {
t.Errorf("unexpected directory %q in seed", c.Name())
continue
}
data, err := io.ReadAll(c.Reader())
if err != nil {
t.Fatalf("reading %s: %v", c.Name(), err)
}
files[c.Name()] = string(data)
}
label, err := img.Label()
if err != nil {
t.Fatalf("Label: %v", err)
}
return files, label
}
func TestSeedISO_filesAndLabel(t *testing.T) {
t.Parallel()
userData := "#cloud-config\npackages: [squid]\n"
vendorData := "#!/bin/sh\necho hi\n"
files, label := readSeed(t, "proxy-abc", userData, vendorData)
if label != SeedLabel {
t.Errorf("label = %q, want %q", label, SeedLabel)
}
want := map[string]string{
"user-data": userData,
"meta-data": "instance-id: proxy-abc\nlocal-hostname: proxy-abc\n",
"vendor-data": vendorData,
}
if len(files) != len(want) {
t.Errorf("seed has %d files %v, want exactly %d", len(files), keys(files), len(want))
}
for name, content := range want {
if got, ok := files[name]; !ok {
t.Errorf("seed lacks %s", name)
} else if got != content {
t.Errorf("%s = %q, want %q", name, got, content)
}
}
}
func keys(m map[string]string) []string {
out := make([]string, 0, len(m))
for k := range m {
out = append(out, k)
}
return out
}
func TestSeedISO_userDataByteForByte(t *testing.T) {
t.Parallel()
tests := []struct {
name string
userData string
}{
{name: "mime multipart with CRLF", userData: "MIME-Version: 1.0\r\nContent-Type: multipart/mixed; boundary=\"b\"\r\n\r\n--b\r\nContent-Type: text/cloud-config\r\n\r\n#cloud-config\r\npackages: [squid]\r\n--b--\r\n"},
{name: "jinja header", userData: "## template: jinja\n#cloud-config\nhostname: {{ v1.local_hostname }}\n"},
{name: "no trailing newline", userData: "#cloud-config\nruncmd:\n - echo done"},
{name: "bytes above 0x7f", userData: "#cloud-config\nwrite_files:\n - content: \"caf\xc3\xa9 \xff\xfe\x80\"\n"},
{name: "empty", userData: ""},
{name: "trailing whitespace preserved", userData: "#!/bin/sh\necho x \n\n\n"},
}
for _, tc := range tests {
t.Run(tc.name, func(t *testing.T) {
t.Parallel()
files, _ := readSeed(t, "proxy-abc", tc.userData, "")
if got := files["user-data"]; got != tc.userData {
t.Errorf("user-data changed:\n got %q\nwant %q", got, tc.userData)
}
})
}
}
// Not parallel: it changes TMPDIR for the process.
func TestSeedISO_tempDirRemoved(t *testing.T) {
tmp := t.TempDir()
t.Setenv("TMPDIR", tmp)
if got := os.TempDir(); got != tmp {
t.Fatalf("os.TempDir() = %q, want %q", got, tmp)
}
var buf bytes.Buffer
if err := BuildSeedISO("proxy-abc", "#cloud-config\n", GuestAgentScript, &buf); err != nil {
t.Fatalf("BuildSeedISO: %v", err)
}
if buf.Len() == 0 {
t.Fatal("seed ISO is empty")
}
entries, err := os.ReadDir(tmp)
if err != nil {
t.Fatal(err)
}
if len(entries) != 0 {
names := make([]string, 0, len(entries))
for _, e := range entries {
names = append(names, e.Name())
}
t.Errorf("staging left behind in TMPDIR: %v", names)
}
}
func TestSeedISO_emptyVendorDataStillHasFile(t *testing.T) {
t.Parallel()
files, _ := readSeed(t, "proxy-abc", "#cloud-config\n", "")
got, ok := files["vendor-data"]
if !ok {
t.Fatalf("seed lacks vendor-data; files: %v", keys(files))
}
if got != "" {
t.Errorf("vendor-data = %q, want empty", got)
}
}
func TestSeedISO_rejectsEmptyName(t *testing.T) {
t.Parallel()
var buf bytes.Buffer
if err := BuildSeedISO("", "#cloud-config\n", "", &buf); !errors.Is(err, ErrEmptyName) {
t.Fatalf("BuildSeedISO with empty name error = %v, want ErrEmptyName", err)
}
if buf.Len() != 0 {
t.Errorf("wrote %d bytes alongside the error", buf.Len())
}
}
func TestVendorData_isShellScriptNotCloudConfig(t *testing.T) {
t.Parallel()
got, err := BuildVendorData("", true)
if err != nil {
t.Fatal(err)
}
if !strings.HasPrefix(got, "#!/bin/sh\n") {
t.Errorf("vendor-data starts with %q, want #!/bin/sh", firstLine(got))
}
if strings.HasPrefix(got, "#cloud-config") {
t.Error("vendor-data is a cloud-config; cloud-init would merge it away")
}
for _, want := range []string{"qemu-guest-agent", "set -e", "apt-get", "dnf", "yum", "zypper", "apk", "systemctl enable --now qemu-guest-agent", "rc-update", "installGuestAgent: false", "DEBIAN_FRONTEND=noninteractive"} {
if !strings.Contains(got, want) {
t.Errorf("script lacks %q", want)
}
}
if strings.Contains(got, "MIME-Version") {
t.Error("script alone should not be wrapped in a multipart")
}
}
func firstLine(s string) string {
line, _, _ := strings.Cut(s, "\n")
return line
}
func bufioReader(s string) *bufio.Reader {
return bufio.NewReader(strings.NewReader(s))
}
func TestVendorData_multipartWhenExtraProvided(t *testing.T) {
t.Parallel()
configured := "#cloud-config\nntp:\n enabled: true\n"
got, err := BuildVendorData(configured, true)
if err != nil {
t.Fatal(err)
}
if !strings.HasPrefix(got, "MIME-Version: 1.0\r\n") {
t.Errorf("vendor-data starts with %q, want MIME-Version: 1.0", firstLine(got))
}
r := textproto.NewReader(bufioReader(got))
hdr, err := r.ReadMIMEHeader()
if err != nil {
t.Fatalf("reading top-level headers: %v", err)
}
mediaType, params, err := mime.ParseMediaType(hdr.Get("Content-Type"))
if err != nil {
t.Fatalf("parsing Content-Type %q: %v", hdr.Get("Content-Type"), err)
}
if mediaType != "multipart/mixed" {
t.Errorf("media type = %q, want multipart/mixed", mediaType)
}
mr := multipart.NewReader(r.R, params["boundary"])
type part struct{ contentType, body string }
var parts []part
for {
p, err := mr.NextPart()
if err == io.EOF {
break
}
if err != nil {
t.Fatalf("NextPart: %v", err)
}
body, err := io.ReadAll(p)
if err != nil {
t.Fatalf("reading part: %v", err)
}
parts = append(parts, part{p.Header.Get("Content-Type"), string(body)})
}
want := []part{{"text/plain", configured}, {"text/x-shellscript", GuestAgentScript}}
if len(parts) != len(want) {
t.Fatalf("got %d parts, want %d", len(parts), len(want))
}
for i := range want {
if parts[i] != want[i] {
t.Errorf("part %d = {%q, %q}, want {%q, %q}", i, parts[i].contentType, firstLine(parts[i].body), want[i].contentType, firstLine(want[i].body))
}
}
}
func TestVendorData_offWithoutConfiguredIsEmpty(t *testing.T) {
t.Parallel()
got, err := BuildVendorData("", false)
if err != nil {
t.Fatal(err)
}
if got != "" {
t.Errorf("vendor-data = %q, want empty", got)
}
}
func TestVendorData_offWithConfiguredPassesThrough(t *testing.T) {
t.Parallel()
configured := "#cloud-config\nntp:\n enabled: true"
got, err := BuildVendorData(configured, false)
if err != nil {
t.Fatal(err)
}
if got != configured {
t.Errorf("vendor-data = %q, want the configured document unchanged", got)
}
}
internal/provider/proxmox/api.go
+119
View File
@@ -0,0 +1,119 @@
package proxmox
import (
"context"
"io"
)
// api is the seam over the Proxmox VE REST API: one method per endpoint the
// provider uses, ctx first (spec entry 17). Calls that start a Proxmox task
// return its UPID; the caller waits on it with TaskStatus.
type api interface {
Version(ctx context.Context) (versionInfo, error)
NodeStatus(ctx context.Context, node string) (nodeStatus, error)
StorageStatus(ctx context.Context, node, storage string) (storageStatus, error)
ClusterResources(ctx context.Context) ([]vmResource, error)
VMConfig(ctx context.Context, node string, vmid int) (vmConfig, error)
SetVMConfig(ctx context.Context, node string, vmid int, params map[string]string) error
Clone(ctx context.Context, node string, vmid int, p cloneParams) (upid string, err error)
Resize(ctx context.Context, node string, vmid int, disk, size string) (upid string, err error)
Start(ctx context.Context, node string, vmid int) (upid string, err error)
Stop(ctx context.Context, node string, vmid int) (upid string, err error)
VMStatus(ctx context.Context, node string, vmid int) (vmStatus, error)
Destroy(ctx context.Context, node string, vmid int) (upid string, err error)
TaskStatus(ctx context.Context, node, upid string) (taskStatus, error)
AgentNetworkInterfaces(ctx context.Context, node string, vmid int) ([]agentInterface, error)
UploadISO(ctx context.Context, node, storage, filename string, r io.Reader, size int64) (upid string, err error)
DeleteVolume(ctx context.Context, node, storage, volid string) (upid string, err error)
}
// versionInfo is GET /version.
type versionInfo struct {
Version string `json:"version"`
Release string `json:"release"`
RepoID string `json:"repoid"`
}
// nodeStatus is the part of GET /nodes/{node}/status the Checker reports.
type nodeStatus struct {
PVEVersion string `json:"pveversion"`
CurrentKernel struct {
Release string `json:"release"`
} `json:"current-kernel"`
}
// storageStatus is GET /nodes/{node}/storage/{storage}/status. Content is a
// comma-separated list such as "iso,import,backup,vztmpl".
type storageStatus struct {
Type string `json:"type"`
Content string `json:"content"`
Active int `json:"active"`
Enabled int `json:"enabled"`
Shared int `json:"shared"`
Total int64 `json:"total"`
Used int64 `json:"used"`
Avail int64 `json:"avail"`
}
// vmResource is one entry of GET /cluster/resources?type=vm. A VM being
// cloned is listed without a name, with lock "clone" and status "unknown"
// (spec entry 9). Tags are separated by ";".
type vmResource struct {
VMID int `json:"vmid"`
Name string `json:"name"`
Node string `json:"node"`
Type string `json:"type"`
Status string `json:"status"`
Lock string `json:"lock"`
Tags string `json:"tags"`
Template int `json:"template"`
}
// vmConfig is GET /nodes/{node}/qemu/{vmid}/config with every value
// rendered as a string: Proxmox sends some scalars as JSON numbers (cores,
// template) and others as strings (memory).
type vmConfig map[string]string
// vmStatus is GET /nodes/{node}/qemu/{vmid}/status/current. QMPStatus tells
// a paused VM from a running one; the resources listing cannot.
type vmStatus struct {
Status string `json:"status"`
QMPStatus string `json:"qmpstatus"`
Name string `json:"name"`
Lock string `json:"lock"`
Uptime int64 `json:"uptime"`
}
// taskStatus is GET /nodes/{node}/tasks/{upid}/status. ExitStatus is "OK"
// or the failure's text once Status is "stopped".
type taskStatus struct {
UPID string `json:"upid"`
Type string `json:"type"`
ID string `json:"id"`
Node string `json:"node"`
Status string `json:"status"`
ExitStatus string `json:"exitstatus"`
}
// agentInterface is one entry of the guest agent's network-get-interfaces
// result.
type agentInterface struct {
Name string `json:"name"`
IPAddresses []agentIP `json:"ip-addresses"`
}
type agentIP struct {
Address string `json:"ip-address"`
Type string `json:"ip-address-type"`
Prefix int `json:"prefix"`
}
// cloneParams are the parameters of POST .../clone the provider sets.
// Storage and Target are omitted when empty.
type cloneParams struct {
NewID int
Name string
Description string
Storage string
Target string
}
internal/provider/proxmox/api_test.go
+97
View File
@@ -0,0 +1,97 @@
package proxmox
import (
"strings"
"testing"
)
// TestAPITypes_decodeCaptures decodes the Phase 0 responses into the seam's
// types, so a misspelled JSON tag fails here and not in a later phase.
func TestAPITypes_decodeCaptures(t *testing.T) {
t.Parallel()
t.Run("version", func(t *testing.T) {
t.Parallel()
if v := decodeData[versionInfo](t, "version"); v.Version != "9.2.20" || v.Release != "9.2" || v.RepoID == "" {
t.Errorf("version = %+v", v)
}
})
t.Run("node status", func(t *testing.T) {
t.Parallel()
n := decodeData[nodeStatus](t, "node_status")
if !strings.HasPrefix(n.PVEVersion, "pve-manager/9.2.20") || n.CurrentKernel.Release != "7.0.14-17-pve" {
t.Errorf("node status = %+v", n)
}
})
t.Run("storage status", func(t *testing.T) {
t.Parallel()
s := decodeData[storageStatus](t, "storage_status")
if s.Type != "dir" || !strings.Contains(s.Content, "iso") || s.Active != 1 || s.Avail <= 0 || s.Total < s.Avail {
t.Errorf("storage status = %+v", s)
}
})
t.Run("cluster resources", func(t *testing.T) {
t.Parallel()
rs := decodeData[[]vmResource](t, "cluster_resources")
byID := map[int]vmResource{}
for _, r := range rs {
byID[r.VMID] = r
}
if r := byID[9001]; r.Name != "debian-13-template" || r.Template != 1 || r.Node != "pv1-1" || r.Type != "qemu" || r.Status != "stopped" {
t.Errorf("resources[9001] = %+v", r)
}
if r := byID[100]; r.Template != 0 || r.Name != "ubuntu-vm-1" {
t.Errorf("resources[100] = %+v", r)
}
})
t.Run("cluster resources during clone", func(t *testing.T) {
t.Parallel()
for _, r := range decodeData[[]vmResource](t, "cluster_resources_during_clone") {
if r.VMID != 199999 {
continue
}
if r.Name != "" || r.Lock != "clone" || r.Status != "unknown" {
t.Errorf("mid-clone entry = %+v, want no name, lock clone, status unknown", r)
}
return
}
t.Error("no entry for VMID 199999 in the mid-clone capture")
})
t.Run("cluster resources tags", func(t *testing.T) {
t.Parallel()
for _, r := range decodeData[[]vmResource](t, "cluster_resources_during_destroy") {
if r.VMID == 199999 && r.Tags != "proxy_operator" {
t.Errorf("tags = %q, want proxy_operator", r.Tags)
}
}
})
t.Run("status current", func(t *testing.T) {
t.Parallel()
if s := decodeData[vmStatus](t, "status_current_paused"); s.Status != "running" || s.QMPStatus != "paused" {
t.Errorf("paused status = %+v", s)
}
if s := decodeData[vmStatus](t, "status_current_stopped"); s.Status != "stopped" || s.QMPStatus != "stopped" {
t.Errorf("stopped status = %+v", s)
}
})
t.Run("task status", func(t *testing.T) {
t.Parallel()
if s := decodeData[taskStatus](t, "task_status_stopped_error"); s.Status != "stopped" || s.ExitStatus != "shrinking disks is not supported" || s.Type != "resize" {
t.Errorf("failed task = %+v", s)
}
if s := decodeData[taskStatus](t, "task_status_running"); s.Status != "running" || s.ExitStatus != "" || s.Node != "pv1-1" {
t.Errorf("running task = %+v", s)
}
if s := decodeData[taskStatus](t, "task_status_volume_missing"); s.ExitStatus != "OK" {
t.Errorf("deleting a missing volume = %+v, want exitstatus OK", s)
}
})
t.Run("UPID responses", func(t *testing.T) {
t.Parallel()
for _, c := range []string{"clone_response", "start_response", "stop_response", "destroy_response", "resize_response", "upload_response", "content_delete_response"} {
if upid := decodeData[string](t, c); !strings.HasPrefix(upid, "UPID:pv1-1:") {
t.Errorf("%s = %q, want a UPID of node pv1-1", c, upid)
}
}
})
}
internal/provider/proxmox/check.go
+224
View File
@@ -0,0 +1,224 @@
package proxmox
import (
"context"
"errors"
"fmt"
"io"
"maps"
"net"
"net/http"
"slices"
"strconv"
"strings"
"sync"
"time"
"gitea.home.hrajfrisbee.cz/kacerr/egress-proxies-operator/internal/provider"
)
var _ provider.Checker = (*Provider)(nil)
// Reasons only this provider can tell apart (spec entry 16). Each names a
// failure an admin fixes somewhere other than where the shared reasons
// point.
const (
// reasonCertificateInvalid: TLS verification failed or the CA file is
// unusable; fix url or caFile, not the host.
reasonCertificateInvalid = "CertificateInvalid"
// reasonCredentialsUnavailable: the token file is missing or malformed;
// fix the Secret, not Proxmox.
reasonCredentialsUnavailable = "CredentialsUnavailable"
// reasonStorageNotFound: isoStorage is missing, inactive, or holds no
// iso content on the node; fix the storage on the node.
reasonStorageNotFound = "StorageNotFound"
// reasonTemplateNotFound: a configured template VMID does not exist or
// is not a template; fix the template or the catalogue.
reasonTemplateNotFound = "TemplateNotFound"
)
// pveproxy's own answers when it cannot reach the node behind it.
const (
statusProxyConnect = 595
statusProxyBroken = 596
)
// Check verifies every configured node concurrently and returns one verdict
// per node in config order. The requests share the provider's HTTP client,
// but each is its own request, so a round that runs out of time cancels
// only its own requests and never a reconcile's.
func (p *Provider) Check(ctx context.Context) []provider.TargetStatus {
out := make([]provider.TargetStatus, len(p.cfg.Nodes))
var wg sync.WaitGroup
for i, node := range p.cfg.Nodes {
wg.Go(func() { out[i] = p.checkNode(ctx, node) })
}
wg.Wait()
return out
}
func (p *Provider) checkNode(ctx context.Context, node string) provider.TargetStatus {
start := time.Now()
log := p.logger(ctx).WithValues("node", node)
ts := provider.TargetStatus{
Name: node,
Address: p.cfg.URL,
Details: map[string]string{"isoStorage": p.cfg.ISOStorage},
}
err := p.probeNode(ctx, node, ts.Details)
ts.Latency = time.Since(start)
switch {
case err == nil:
ts.Ready = true
ts.Reason = provider.ReasonReachable
case ctx.Err() != nil:
// The round ran out of time, whichever request was running.
ts.Reason = provider.ReasonTimeout
ts.Message = err.Error()
default:
ts.Reason = checkReason(err)
ts.Message = err.Error()
}
log.V(1).Info("proxmox node checked", "ready", ts.Ready, "reason", ts.Reason, "latency", ts.Latency)
return ts
}
// probeNode runs spec entry 16's four requests and fills details as facts
// arrive, so a failure late in the sequence still reports what came before.
func (p *Provider) probeNode(ctx context.Context, node string, details map[string]string) error {
// The token file is read here only for its id: the secret after "="
// is dropped at once and never reaches details, logs or errors.
token, err := readToken(p.cfg.TokenFile)
if err != nil {
return err
}
tokenID, _, _ := strings.Cut(token, "=")
details["tokenID"] = tokenID
v, err := p.api.Version(ctx)
if err != nil {
return fmt.Errorf("reading the API version: %w", err)
}
details["pveVersion"] = v.Version
ns, err := p.api.NodeStatus(ctx, node)
if err != nil {
return fmt.Errorf("reading the status of node %s: %w", node, err)
}
details["kernel"] = ns.CurrentKernel.Release
if err := p.checkStorage(ctx, node, details); err != nil {
return err
}
return p.checkTemplates(ctx, details)
}
// checkStorage reads isoStorage on the node. A storage the cluster does
// not know is a 400 on the storage parameter (captured). A 500 is taken
// the same way, whatever its wording: Proxmox cannot use that storage on
// this node (not enabled here, or offline; not captured). The message
// carries Proxmox's text.
func (p *Provider) checkStorage(ctx context.Context, node string, details map[string]string) error {
storage := p.cfg.ISOStorage
st, err := p.api.StorageStatus(ctx, node, storage)
var ae *apiError
if errors.As(err, &ae) {
switch {
case isNoSuchStorage(ae):
return &checkFailure{reason: reasonStorageNotFound, msg: fmt.Sprintf("storage %s on node %s: %s", storage, node, ae.Errors["storage"])}
case ae.Status == http.StatusInternalServerError:
return &checkFailure{reason: reasonStorageNotFound, msg: fmt.Sprintf("storage %s on node %s: %s", storage, node, ae.Message)}
}
}
if err != nil {
return fmt.Errorf("reading storage %s on node %s: %w", storage, node, err)
}
details["isoStorageAvailGiB"] = formatGiB(st.Avail)
if st.Active == 0 {
return &checkFailure{reason: reasonStorageNotFound, msg: fmt.Sprintf("storage %s is not active on node %s", storage, node)}
}
if !slices.Contains(strings.Split(st.Content, ","), "iso") {
return &checkFailure{reason: reasonStorageNotFound, msg: fmt.Sprintf("storage %s on node %s does not hold iso content (content: %s)", storage, node, st.Content)}
}
return nil
}
// checkTemplates looks every catalogue entry up in the cluster's resources:
// a template may sit on any node, since a clone can target another.
func (p *Provider) checkTemplates(ctx context.Context, details map[string]string) error {
rs, err := p.api.ClusterResources(ctx)
if err != nil {
return fmt.Errorf("listing cluster resources: %w", err)
}
isTemplate := make(map[int]bool)
for _, r := range rs {
if r.Type == "qemu" {
isTemplate[r.VMID] = r.Template == 1
}
}
var problems []string
for _, key := range slices.Sorted(maps.Keys(p.cfg.Templates)) {
vmid := p.cfg.Templates[key].VMID
switch tmpl, found := isTemplate[vmid]; {
case !found:
problems = append(problems, fmt.Sprintf("template %q: VM %d does not exist", key, vmid))
case !tmpl:
problems = append(problems, fmt.Sprintf("template %q: VM %d is not a template", key, vmid))
}
}
details["templates"] = fmt.Sprintf("%d/%d", len(p.cfg.Templates)-len(problems), len(p.cfg.Templates))
if len(problems) > 0 {
return &checkFailure{reason: reasonTemplateNotFound, msg: strings.Join(problems, "; ")}
}
return nil
}
// checkFailure is a verdict the backend answered but the check refuses: the
// storage or a template is not usable.
type checkFailure struct {
reason string
msg string
}
func (e *checkFailure) Error() string { return e.msg }
// checkReason maps a failed probe to a reason once checkNode has ruled out
// the round's ctx ending. A failed connect or a cut connection is
// Unreachable; a request timeout with the round still running is Timeout.
func checkReason(err error) string {
var (
cf *checkFailure
ae *apiError
op *net.OpError
ne net.Error
)
switch {
case errors.As(err, &cf):
return cf.reason
case errors.Is(err, errCredentials):
return reasonCredentialsUnavailable
case errors.Is(err, errCAFile), isTLSVerification(err):
return reasonCertificateInvalid
case errors.As(err, &ae):
switch ae.Status {
case http.StatusUnauthorized:
return provider.ReasonUnauthorized
case http.StatusForbidden:
return provider.ReasonForbidden
case statusProxyConnect, statusProxyBroken:
return provider.ReasonUnreachable
}
return provider.ReasonAPIError
case errors.As(err, &op):
// Checked before the deadline: a dial that runs out of its own
// budget is a node that did not answer.
return provider.ReasonUnreachable
case errors.Is(err, context.DeadlineExceeded), errors.Is(err, context.Canceled):
return provider.ReasonTimeout
case errors.As(err, &ne), errors.Is(err, io.EOF), errors.Is(err, io.ErrUnexpectedEOF):
return provider.ReasonUnreachable
}
return provider.ReasonAPIError
}
func formatGiB(bytes int64) string {
return strconv.FormatFloat(float64(bytes)/(1<<30), 'f', 1, 64)
}
internal/provider/proxmox/check_test.go
+593
View File
@@ -0,0 +1,593 @@
package proxmox
import (
"context"
"crypto/x509"
"errors"
"fmt"
"io"
"maps"
"net"
"net/http"
"net/url"
"path/filepath"
"slices"
"strings"
"sync"
"syscall"
"testing"
"time"
"gitea.home.hrajfrisbee.cz/kacerr/egress-proxies-operator/internal/provider"
)
// checkSecret is the secret half of the Checker's token file; it must never
// reach a verdict or a log line.
const checkSecret = "SENTINEL-token-secret-4c1d"
// newCheckFixture is an observed create fixture at V(2) with a token file
// whose secret is checkSecret.
func newCheckFixture(t *testing.T) *fixture {
t.Helper()
fx := newCreateFixture(t, 2)
fx.p.api = observe(fx.fake, testProvider)
fx.p.cfg.TokenFile = filepath.Join(t.TempDir(), "token")
writeFile(t, fx.p.cfg.TokenFile, "proxy-operator@pve!operator="+checkSecret+"\n")
return fx
}
// assertNoSecret checks every verdict and every log line for checkSecret.
func assertNoSecret(t *testing.T, fx *fixture, got []provider.TargetStatus) {
t.Helper()
for _, ts := range got {
if strings.Contains(ts.Message, checkSecret) || slices.ContainsFunc(slices.Collect(maps.Values(ts.Details)), func(v string) bool { return strings.Contains(v, checkSecret) }) {
t.Errorf("verdict carries the token secret: %+v", ts)
}
}
for _, l := range fx.logs.all() {
if strings.Contains(l, checkSecret) {
t.Errorf("log line carries the token secret: %s", l)
}
}
}
func TestCheck_oneEntryPerNodeInConfigOrder(t *testing.T) {
t.Parallel()
fx := newCheckFixture(t)
fx.p.cfg.Nodes = []string{"pv1-3", fakeNode, "pv1-2"}
fx.fake.nodes["pv1-3"] = fx.fake.nodes[fakeNode]
got := fx.p.Check(fx.ctx)
if len(got) != len(fx.p.cfg.Nodes) {
t.Fatalf("Check() returned %d targets, want %d", len(got), len(fx.p.cfg.Nodes))
}
for i, node := range fx.p.cfg.Nodes {
ts := got[i]
if ts.Name != node || ts.Address != fx.p.cfg.URL {
t.Errorf("target %d = %s (%s), want %s (%s)", i, ts.Name, ts.Address, node, fx.p.cfg.URL)
}
if wantReady := node != "pv1-2"; ts.Ready != wantReady {
t.Errorf("target %s ready = %v, want %v (%s: %s)", node, ts.Ready, wantReady, ts.Reason, ts.Message)
}
}
if got[2].Reason != provider.ReasonAPIError || !strings.Contains(got[2].Message, "pv1-2") {
t.Errorf("pv1-2 verdict = %s: %s, want APIError naming the node", got[2].Reason, got[2].Message)
}
}
// TestCheck_nodesCheckedConcurrently holds every node's first request
// until all of them have arrived; checked one at a time, the first would
// give up and fail its verdict.
func TestCheck_nodesCheckedConcurrently(t *testing.T) {
t.Parallel()
fx := newCheckFixture(t)
fx.p.cfg.Nodes = []string{fakeNode, "pv1-2", "pv1-3"}
for _, n := range fx.p.cfg.Nodes[1:] {
fx.fake.nodes[n] = fx.fake.nodes[fakeNode]
}
var (
mu sync.Mutex
arrived int
)
all := make(chan struct{})
fx.fake.hookOn("Version", func() error {
mu.Lock()
if arrived++; arrived == len(fx.p.cfg.Nodes) {
close(all)
}
mu.Unlock()
select {
case <-all:
return nil
case <-time.After(2 * time.Second):
return errors.New("barrier: the nodes were not checked concurrently")
}
})
for _, ts := range fx.p.Check(fx.ctx) {
if !ts.Ready {
t.Errorf("target %s = %s: %s, want Ready", ts.Name, ts.Reason, ts.Message)
}
}
}
func TestCheck_expiredContextReturnsPromptly(t *testing.T) {
t.Parallel()
fx := newCheckFixture(t)
fx.p.cfg.Nodes = []string{fakeNode, "pv1-2"}
fx.fake.blockOn("Version")
ctx, cancel := context.WithTimeout(fx.ctx, 0)
defer cancel()
start := time.Now()
var got []provider.TargetStatus
within(t, 5*time.Second, func() { got = fx.p.Check(ctx) })
if elapsed := time.Since(start); elapsed > time.Second {
t.Errorf("Check() took %v with an expired ctx, want under 1s", elapsed)
}
if len(got) != 2 {
t.Fatalf("Check() returned %d targets, want 2", len(got))
}
for _, ts := range got {
if ts.Ready || ts.Reason != provider.ReasonTimeout || !strings.Contains(ts.Message, context.DeadlineExceeded.Error()) {
t.Errorf("target %s = ready %v reason %s (%s), want Timeout naming the deadline", ts.Name, ts.Ready, ts.Reason, ts.Message)
}
}
}
func TestCheck_reasonsTable(t *testing.T) {
t.Parallel()
urlErr := func(err error) error {
return &url.Error{Op: "Get", URL: "https://192.0.2.8:8006/api2/json/version", Err: err}
}
tests := []struct {
name string
prep func(fx *fixture)
timeout time.Duration
cancel bool
wantReady bool
wantReason string
wantMessage string
}{
{name: "ok", wantReady: true, wantReason: provider.ReasonReachable},
{
name: "token file missing",
prep: func(fx *fixture) { fx.p.cfg.TokenFile = filepath.Join(fx.t.TempDir(), "absent") },
wantReason: reasonCredentialsUnavailable,
wantMessage: "does not exist yet",
},
{
name: "token file malformed",
prep: func(fx *fixture) { writeFile(fx.t, fx.p.cfg.TokenFile, checkSecret+"\n") },
wantReason: reasonCredentialsUnavailable,
wantMessage: "is not <user>@<realm>!<tokenid>=<secret>",
},
{
name: "token file gone by the time the client reads it",
prep: func(fx *fixture) {
fx.fake.failWith("NodeStatus", fmt.Errorf("%w: %w: /etc/proxy-operator/proxmox/token does not exist yet", provider.ErrTransient, errCredentials))
},
wantReason: reasonCredentialsUnavailable,
},
{
name: "unknown certificate authority",
prep: func(fx *fixture) { fx.fake.failWith("Version", urlErr(x509.UnknownAuthorityError{})) },
wantReason: reasonCertificateInvalid,
},
{
name: "host not in the certificate",
prep: func(fx *fixture) { fx.fake.failWith("Version", urlErr(x509.HostnameError{Host: "192.168.2.8"})) },
wantReason: reasonCertificateInvalid,
},
{
name: "CA file unusable",
prep: func(fx *fixture) {
fx.fake.failWith("Version", fmt.Errorf("%w: %w: reading CA file: %w", provider.ErrTransient, errCAFile, syscall.ENOENT))
},
wantReason: reasonCertificateInvalid,
},
{
name: "wrong secret",
prep: func(fx *fixture) { fx.fake.failWith("Version", loadAPIError(fx.t, "err_unauthorized")) },
wantReason: provider.ReasonUnauthorized,
},
{
name: "no credentials sent",
prep: func(fx *fixture) { fx.fake.failWith("Version", loadAPIError(fx.t, "err_no_credentials")) },
wantReason: provider.ReasonUnauthorized,
},
{
name: "privilege missing",
prep: func(fx *fixture) { fx.fake.failWith("NodeStatus", loadAPIError(fx.t, "err_forbidden")) },
wantReason: provider.ReasonForbidden,
wantMessage: "Sys.Syslog",
},
{
name: "pveproxy cannot reach the node",
prep: func(fx *fixture) { fx.fake.failWith("NodeStatus", fakeErr(595, "Connection refused")) },
wantReason: provider.ReasonUnreachable,
},
{
name: "pveproxy broken pipe",
prep: func(fx *fixture) { fx.fake.failWith("NodeStatus", fakeErr(596, "Broken pipe")) },
wantReason: provider.ReasonUnreachable,
},
{
name: "connection refused",
prep: func(fx *fixture) {
fx.fake.failWith("Version", urlErr(&net.OpError{Op: "dial", Net: "tcp", Err: syscall.ECONNREFUSED}))
},
wantReason: provider.ReasonUnreachable,
},
{
name: "node status 500",
prep: func(fx *fixture) { fx.fake.failWith("NodeStatus", fakeErr(500, "fake: pve-cluster not running")) },
wantReason: provider.ReasonAPIError,
},
{
name: "iso storage missing",
prep: func(fx *fixture) {
fx.fake.mu.Lock()
delete(fx.fake.storages, fakeISOStorage)
fx.fake.mu.Unlock()
},
wantReason: reasonStorageNotFound,
wantMessage: "storage proxy-seeds on node pv1-1: No such storage.",
},
{
// Not captured: a storage Proxmox knows but cannot use here.
name: "iso storage unusable on the node",
prep: func(fx *fixture) {
fx.fake.failWith("StorageStatus", fakeErr(500, "fake: storage 'proxy-seeds' is not online"))
},
wantReason: reasonStorageNotFound,
wantMessage: "storage proxy-seeds on node pv1-1: fake: storage 'proxy-seeds' is not online",
},
{
name: "another bad parameter is not a missing storage",
prep: func(fx *fixture) { fx.fake.failWith("StorageStatus", loadAPIError(fx.t, "err_bad_param")) },
wantReason: provider.ReasonAPIError,
},
{
name: "iso storage inactive",
prep: func(fx *fixture) { setStorage(fx, func(s *storageStatus) { s.Active = 0 }) },
wantReason: reasonStorageNotFound,
wantMessage: "not active",
},
{
name: "iso storage without iso content",
prep: func(fx *fixture) { setStorage(fx, func(s *storageStatus) { s.Content = "images,rootdir,isoish" }) },
wantReason: reasonStorageNotFound,
wantMessage: "does not hold iso content (content: images,rootdir,isoish)",
},
{
name: "iso storage forbidden",
prep: func(fx *fixture) { fx.fake.failWith("StorageStatus", loadAPIError(fx.t, "err_forbidden")) },
wantReason: provider.ReasonForbidden,
},
{
name: "template missing",
prep: func(fx *fixture) {
fx.fake.mu.Lock()
delete(fx.fake.vms, testTemplateVMID)
fx.fake.mu.Unlock()
},
wantReason: reasonTemplateNotFound,
wantMessage: `template "debian-13": VM 9001 does not exist`,
},
{
name: "template VMID holds a plain VM",
prep: func(fx *fixture) { fx.fake.vms[testTemplateVMID].template = false },
wantReason: reasonTemplateNotFound,
wantMessage: `template "debian-13": VM 9001 is not a template`,
},
{
name: "template VMID holds a container",
prep: func(fx *fixture) { fx.fake.vms[testTemplateVMID].kind = "lxc" },
wantReason: reasonTemplateNotFound,
wantMessage: "does not exist",
},
{
name: "resources listing fails",
prep: func(fx *fixture) {
fx.fake.failWith("ClusterResources", fakeErr(500, "fake: cluster filesystem not ready"))
},
wantReason: provider.ReasonAPIError,
},
{
name: "request outlives the deadline",
prep: func(fx *fixture) { fx.fake.blockOn("StorageStatus") },
timeout: 50 * time.Millisecond,
wantReason: provider.ReasonTimeout,
},
{
// The real client's dial sees the round's ctx end and fails as
// a dial error, which checkReason alone would call Unreachable.
name: "dial cut short by the round's deadline",
prep: func(fx *fixture) {
fx.fake.hookOn("Version", func() error {
time.Sleep(200 * time.Millisecond)
return urlErr(&net.OpError{Op: "dial", Net: "tcp", Err: context.DeadlineExceeded})
})
},
timeout: 20 * time.Millisecond,
wantReason: provider.ReasonTimeout,
},
{
name: "ctx cancelled mid check",
prep: func(fx *fixture) { fx.fake.blockOn("ClusterResources") },
cancel: true,
wantReason: provider.ReasonTimeout,
},
}
for _, tc := range tests {
t.Run(tc.name, func(t *testing.T) {
t.Parallel()
fx := newCheckFixture(t)
if tc.prep != nil {
tc.prep(fx)
}
ctx := fx.ctx
switch {
case tc.timeout > 0:
var cancel context.CancelFunc
ctx, cancel = context.WithTimeout(ctx, tc.timeout)
defer cancel()
case tc.cancel:
var cancel context.CancelFunc
ctx, cancel = context.WithCancel(ctx)
time.AfterFunc(50*time.Millisecond, cancel)
}
var got []provider.TargetStatus
within(t, 5*time.Second, func() { got = fx.p.Check(ctx) })
if len(got) != 1 {
t.Fatalf("Check() returned %d targets, want 1", len(got))
}
ts := got[0]
if ts.Ready != tc.wantReady || ts.Reason != tc.wantReason {
t.Errorf("verdict = ready %v reason %s (%s), want ready %v reason %s", ts.Ready, ts.Reason, ts.Message, tc.wantReady, tc.wantReason)
}
if !tc.wantReady && ts.Message == "" {
t.Error("failed check has no message")
}
if tc.wantReady && ts.Message != "" {
t.Errorf("successful check has a message: %q", ts.Message)
}
if tc.wantMessage != "" && !strings.Contains(ts.Message, tc.wantMessage) {
t.Errorf("message = %q, want it to contain %q", ts.Message, tc.wantMessage)
}
if ts.Details["isoStorage"] != fakeISOStorage {
t.Errorf("details = %v, want isoStorage whatever failed", ts.Details)
}
assertNoSecret(t, fx, got)
})
}
}
func setStorage(fx *fixture, fn func(s *storageStatus)) {
fx.fake.mu.Lock()
defer fx.fake.mu.Unlock()
s := fx.fake.storages[fakeISOStorage]
fn(&s)
fx.fake.storages[fakeISOStorage] = s
}
// TestCheck_reasonsOverHTTPClient takes the transport-level reasons from
// the real client, so the errors checkReason sees are the ones net/http and
// crypto/tls produce, and the Ready row replays the Phase 0 captures.
func TestCheck_reasonsOverHTTPClient(t *testing.T) {
t.Parallel()
captures := func(t *testing.T) http.HandlerFunc {
return func(w http.ResponseWriter, r *http.Request) {
path := strings.TrimPrefix(r.URL.Path, "/api2/json/")
name := map[string]string{
"version": "version",
"nodes/pv1-1/status": "node_status",
"nodes/pv1-1/storage/local/status": "storage_status",
"cluster/resources": "cluster_resources",
}[path]
if name == "" {
t.Errorf("unexpected request %s", path)
http.NotFound(w, r)
return
}
serveJSON(w, readCapture(t, name).Body)
}
}
tests := []struct {
name string
handler func(t *testing.T) http.HandlerFunc
mutate func(t *testing.T, cfx *clientFixture)
wantReason string
}{
{name: "captures", handler: captures, wantReason: provider.ReasonReachable},
{name: "unknown authority", handler: captures, mutate: func(_ *testing.T, cfx *clientFixture) { cfx.cfg.CAFile = "" }, wantReason: reasonCertificateInvalid},
{
name: "host not in the certificate",
handler: captures,
mutate: func(_ *testing.T, cfx *clientFixture) {
cfx.cfg.URL = strings.Replace(cfx.srv.URL, "127.0.0.1", "localhost", 1)
},
wantReason: reasonCertificateInvalid,
},
{
name: "CA file missing",
handler: captures,
mutate: func(t *testing.T, cfx *clientFixture) { cfx.cfg.CAFile = filepath.Join(t.TempDir(), "ca.pem") },
wantReason: reasonCertificateInvalid,
},
{
name: "401 from pveproxy",
handler: func(t *testing.T) http.HandlerFunc {
return func(w http.ResponseWriter, _ *http.Request) { serveCapture(t, w, readCapture(t, "err_unauthorized")) }
},
wantReason: provider.ReasonUnauthorized,
},
{
name: "596 from pveproxy",
handler: func(t *testing.T) http.HandlerFunc {
return func(w http.ResponseWriter, _ *http.Request) {
serveCapture(t, w, capture{Status: 596, Reason: "Broken pipe"})
}
},
wantReason: provider.ReasonUnreachable,
},
{
name: "server gone",
handler: captures,
mutate: func(_ *testing.T, cfx *clientFixture) { cfx.srv.Close() },
wantReason: provider.ReasonUnreachable,
},
{
name: "connection cut mid response",
handler: func(t *testing.T) http.HandlerFunc {
return func(w http.ResponseWriter, _ *http.Request) {
conn, _, err := w.(http.Hijacker).Hijack()
if err != nil {
t.Errorf("hijack: %v", err)
return
}
_, _ = io.WriteString(conn, "HTTP/1.1 200 OK\r\nContent-Length: 100\r\n\r\n{\"data\":")
_ = conn.Close()
}
},
wantReason: provider.ReasonUnreachable,
},
}
for _, tc := range tests {
t.Run(tc.name, func(t *testing.T) {
t.Parallel()
cfx := newClientFixture(t, tc.handler(t))
if tc.mutate != nil {
tc.mutate(t, cfx)
}
cfg := testConfig()
cfg.ISOStorage = "local"
cfg.URL, cfg.TokenFile, cfg.CAFile = cfx.cfg.URL, cfx.cfg.TokenFile, cfx.cfg.CAFile
p := newProvider(testProvider, cfg, cfx.api(t))
got := p.Check(context.Background())
if len(got) != 1 {
t.Fatalf("Check() returned %d targets, want 1", len(got))
}
if ts := got[0]; ts.Reason != tc.wantReason || ts.Ready != (tc.wantReason == provider.ReasonReachable) {
t.Errorf("verdict = ready %v reason %s (%s), want %s", ts.Ready, ts.Reason, ts.Message, tc.wantReason)
}
})
}
}
func TestCheck_detailsKeys(t *testing.T) {
t.Parallel()
full := map[string]string{
"tokenID": "proxy-operator@pve!operator", "isoStorage": fakeISOStorage, "pveVersion": "9.2.20",
"kernel": "7.0.14-17-pve", "isoStorageAvailGiB": "90.0", "templates": "1/1",
}
tests := []struct {
name string
prep func(fx *fixture)
want []string
}{
{name: "success", want: slices.Sorted(maps.Keys(full))},
{name: "token unusable", prep: func(fx *fixture) { writeFile(fx.t, fx.p.cfg.TokenFile, "junk") }, want: []string{"isoStorage"}},
{name: "node status fails", prep: func(fx *fixture) { fx.fake.failWith("NodeStatus", fakeErr(596, "Broken pipe")) }, want: []string{"isoStorage", "pveVersion", "tokenID"}},
{name: "storage inactive", prep: func(fx *fixture) { setStorage(fx, func(s *storageStatus) { s.Active = 0 }) }, want: []string{"isoStorage", "isoStorageAvailGiB", "kernel", "pveVersion", "tokenID"}},
{name: "template missing", prep: func(fx *fixture) { fx.fake.vms[testTemplateVMID].template = false }, want: slices.Sorted(maps.Keys(full))},
}
for _, tc := range tests {
t.Run(tc.name, func(t *testing.T) {
t.Parallel()
fx := newCheckFixture(t)
if tc.prep != nil {
tc.prep(fx)
}
got := fx.p.Check(fx.ctx)[0]
if keys := slices.Sorted(maps.Keys(got.Details)); !slices.Equal(keys, tc.want) {
t.Errorf("details keys = %v, want %v", keys, tc.want)
}
for k, v := range got.Details {
if want, ok := full[k]; ok && tc.name == "success" && v != want {
t.Errorf("details[%s] = %q, want %q", k, v, want)
}
if v == "" {
t.Errorf("details[%s] is empty", k)
}
}
})
}
t.Run("templates counts found over configured", func(t *testing.T) {
t.Parallel()
fx := newCheckFixture(t)
fx.p.cfg.Templates["ubuntu"] = provider.ProxmoxTemplate{VMID: 9002}
fx.p.cfg.Templates["debian-again"] = provider.ProxmoxTemplate{VMID: testTemplateVMID}
got := fx.p.Check(fx.ctx)[0]
if got.Details["templates"] != "2/3" || got.Reason != reasonTemplateNotFound || !strings.Contains(got.Message, `template "ubuntu": VM 9002`) {
t.Errorf("verdict = %s %q, details %v; want TemplateNotFound naming ubuntu, templates 2/3", got.Reason, got.Message, got.Details)
}
})
}
func TestCheck_logsVerdictPerNode(t *testing.T) {
t.Parallel()
fx := newCheckFixture(t)
fx.p.cfg.Nodes = []string{fakeNode, "pv1-2"}
fx.p.Check(fx.ctx)
checked := fx.logs.matching(`"msg"="proxmox node checked"`)
if len(checked) != 2 {
t.Fatalf("got %d verdict lines, want 2: %v", len(checked), checked)
}
for _, l := range checked {
switch {
case strings.Contains(l, `"node"="pv1-1"`):
if !strings.Contains(l, `"ready"=true`) || !strings.Contains(l, `"reason"="Reachable"`) || !strings.Contains(l, `"latency"=`) {
t.Errorf("pv1-1 verdict line = %s", l)
}
case strings.Contains(l, `"node"="pv1-2"`):
if !strings.Contains(l, `"ready"=false`) || !strings.Contains(l, `"reason"="APIError"`) {
t.Errorf("pv1-2 verdict line = %s", l)
}
default:
t.Errorf("verdict line without a node: %s", l)
}
if !strings.Contains(l, `"provider"="`+testProvider+`"`) || !strings.Contains(l, `"level"=1`) {
t.Errorf("verdict line lacks the provider or is not V(1): %s", l)
}
}
}
func TestCheckReason_table(t *testing.T) {
t.Parallel()
dial := &url.Error{Op: "Get", URL: "https://192.0.2.8:8006/api2/json/version", Err: &net.OpError{Op: "dial", Net: "tcp", Err: context.DeadlineExceeded}}
tests := []struct {
name string
err error
want string
}{
{name: "storage verdict", err: &checkFailure{reason: reasonStorageNotFound, msg: "x"}, want: reasonStorageNotFound},
{name: "template verdict wrapped", err: fmt.Errorf("check: %w", &checkFailure{reason: reasonTemplateNotFound, msg: "x"}), want: reasonTemplateNotFound},
{name: "token missing", err: fmt.Errorf("%w: %w: /t does not exist yet", provider.ErrTransient, errCredentials), want: reasonCredentialsUnavailable},
{name: "token malformed", err: fmt.Errorf("%w: %w: /t is not a token", provider.ErrPermanent, errCredentials), want: reasonCredentialsUnavailable},
{name: "CA file", err: fmt.Errorf("%w: %w: /ca holds no PEM certificate", provider.ErrPermanent, errCAFile), want: reasonCertificateInvalid},
{name: "unknown authority", err: &url.Error{Err: x509.UnknownAuthorityError{}}, want: reasonCertificateInvalid},
{name: "certificate invalid", err: &url.Error{Err: x509.CertificateInvalidError{Reason: x509.Expired}}, want: reasonCertificateInvalid},
{name: "401", err: loadAPIError(t, "err_unauthorized"), want: provider.ReasonUnauthorized},
{name: "403 wrapped", err: fmt.Errorf("reading: %w", loadAPIError(t, "err_forbidden_volume_delete")), want: provider.ReasonForbidden},
{name: "595", err: fakeErr(595, "Connection refused"), want: provider.ReasonUnreachable},
{name: "596", err: fakeErr(596, "Broken pipe"), want: provider.ReasonUnreachable},
{name: "400", err: loadAPIError(t, "err_bad_param"), want: provider.ReasonAPIError},
{name: "500", err: loadAPIError(t, "err_vm_missing"), want: provider.ReasonAPIError},
{name: "dial ran out of its own budget", err: dial, want: provider.ReasonUnreachable},
{name: "connection reset", err: &url.Error{Err: &net.OpError{Op: "read", Net: "tcp", Err: syscall.ECONNRESET}}, want: provider.ReasonUnreachable},
{name: "request timeout", err: fmt.Errorf("reading: %w", &url.Error{Err: context.DeadlineExceeded}), want: provider.ReasonTimeout},
{name: "cancelled", err: context.Canceled, want: provider.ReasonTimeout},
{name: "server closed mid response", err: &url.Error{Err: io.ErrUnexpectedEOF}, want: provider.ReasonUnreachable},
{name: "eof", err: io.EOF, want: provider.ReasonUnreachable},
{name: "decode failure", err: errors.New("decoding GET version: invalid character '<'"), want: provider.ReasonAPIError},
{name: "nil", err: nil, want: provider.ReasonAPIError},
}
for _, tc := range tests {
t.Run(tc.name, func(t *testing.T) {
t.Parallel()
if got := checkReason(tc.err); got != tc.want {
t.Errorf("checkReason(%v) = %s, want %s", tc.err, got, tc.want)
}
})
}
}
internal/provider/proxmox/client.go
+382
View File
@@ -0,0 +1,382 @@
package proxmox
import (
"bytes"
"context"
"crypto/tls"
"crypto/x509"
"encoding/json"
"errors"
"fmt"
"io"
"io/fs"
"mime/multipart"
"net"
"net/http"
"net/url"
"os"
"regexp"
"strconv"
"strings"
"sync"
"time"
"gitea.home.hrajfrisbee.cz/kacerr/egress-proxies-operator/internal/provider"
"gitea.home.hrajfrisbee.cz/kacerr/egress-proxies-operator/internal/tracing"
)
const (
dialTimeout = 10 * time.Second
tlsHandshakeTimeout = 10 * time.Second
idleConnTimeout = 90 * time.Second
maxIdleConnsPerHost = 4
// maxErrorBody caps how much of a failed response is read for its
// message; Proxmox error bodies are a few hundred bytes.
maxErrorBody = 64 << 10
)
// errCredentials marks a token file that is missing or unusable, so the
// Checker can report CredentialsUnavailable instead of an API failure.
var errCredentials = errors.New("proxmox token file unusable")
// errCAFile marks a CA file that is missing or holds no certificate, so the
// Checker can report CertificateInvalid: the admin fixes caFile.
var errCAFile = errors.New("proxmox CA file unusable")
// tokenPattern is <user>@<realm>!<tokenid>=<secret> (spec entry 14).
var tokenPattern = regexp.MustCompile(`^[^@\s]+@[^!\s]+![^=\s]+=\S+$`)
// httpAPI is the production adapter behind the api seam: a thin net/http
// client for /api2/json (spec entry 17). It is safe for concurrent use.
type httpAPI struct {
base *url.URL
tokenFile string
caFile string
insecure bool
// timeout bounds one request, response included. Read when the client
// is built; tests shrink it before the first call.
timeout time.Duration
wrap func(http.RoundTripper) http.RoundTripper
// mu guards client, built on first use because the CA is loaded lazily
// and only cached once it loads (spec entry 14).
mu sync.Mutex
client *http.Client
}
var _ api = (*httpAPI)(nil)
// newHTTPAPI builds the adapter without touching the network or any file.
func newHTTPAPI(cfg provider.ProxmoxConfig, opts ...tracing.Option) (*httpAPI, error) {
u, err := url.Parse(cfg.URL)
if err != nil {
return nil, fmt.Errorf("proxmox.url: %w", err)
}
return &httpAPI{
base: u.JoinPath("api2", "json"),
tokenFile: cfg.TokenFile,
caFile: cfg.CAFile,
insecure: cfg.InsecureSkipTLSVerify,
timeout: time.Duration(cfg.RequestTimeoutSeconds) * time.Second,
wrap: tracing.RestConfigWrapper(opts...),
}, nil
}
// httpClient returns the shared client, building it once the CA loads. A
// missing CA file is Transient and retried on the next call; one without a
// certificate is Permanent.
func (a *httpAPI) httpClient() (*http.Client, error) {
a.mu.Lock()
defer a.mu.Unlock()
if a.client != nil {
return a.client, nil
}
tlsCfg := &tls.Config{MinVersion: tls.VersionTLS12, InsecureSkipVerify: a.insecure} //nolint:gosec // opt-in, warned about at startup
if a.caFile != "" && !a.insecure {
pem, err := os.ReadFile(a.caFile)
if err != nil {
return nil, fmt.Errorf("%w: %w: reading CA file: %w", provider.ErrTransient, errCAFile, err)
}
pool := x509.NewCertPool()
if !pool.AppendCertsFromPEM(pem) {
return nil, fmt.Errorf("%w: %w: %s holds no PEM certificate", provider.ErrPermanent, errCAFile, a.caFile)
}
tlsCfg.RootCAs = pool
}
transport := &http.Transport{
Proxy: http.ProxyFromEnvironment,
DialContext: (&net.Dialer{Timeout: dialTimeout, KeepAlive: 30 * time.Second}).DialContext,
TLSClientConfig: tlsCfg,
TLSHandshakeTimeout: tlsHandshakeTimeout,
ResponseHeaderTimeout: a.timeout,
IdleConnTimeout: idleConnTimeout,
MaxIdleConnsPerHost: maxIdleConnsPerHost,
// pveproxy speaks HTTP/1.1 only (F18); a non-nil empty map keeps
// the transport from offering h2.
ForceAttemptHTTP2: false,
TLSNextProto: map[string]func(string, *tls.Conn) http.RoundTripper{},
}
a.client = &http.Client{Transport: a.wrap(transport), Timeout: a.timeout}
return a.client, nil
}
// readToken reads the token file on every request, so a rotated Secret
// takes effect at once. Neither error names the file's content.
func readToken(path string) (string, error) {
raw, err := os.ReadFile(path)
if errors.Is(err, fs.ErrNotExist) {
return "", fmt.Errorf("%w: %w: %s does not exist yet", provider.ErrTransient, errCredentials, path)
}
if err != nil {
return "", fmt.Errorf("%w: reading %s: %w", errCredentials, path, err)
}
token := strings.TrimSpace(string(raw))
if !tokenPattern.MatchString(token) {
return "", fmt.Errorf("%w: %w: %s is not <user>@<realm>!<tokenid>=<secret>", provider.ErrPermanent, errCredentials, path)
}
return token, nil
}
// apiPath joins path segments, escaping each, so a volid's "/" travels as
// %2F as Phase 0 sent it.
func apiPath(segments ...string) string {
escaped := make([]string, len(segments))
for i, s := range segments {
escaped[i] = url.PathEscape(s)
}
return strings.Join(escaped, "/")
}
func vmPath(node string, vmid int, rest ...string) string {
return apiPath(append([]string{"nodes", node, "qemu", strconv.Itoa(vmid)}, rest...)...)
}
// do sends one request and decodes the response's data member into out,
// which may be nil. POST and PUT send form as the body; every method sends
// query in the URL. Any non-2xx answer is an *apiError.
func (a *httpAPI) do(ctx context.Context, method, path string, query, form url.Values, out any) error {
var body io.Reader
contentType := ""
if form != nil {
body = strings.NewReader(form.Encode())
contentType = "application/x-www-form-urlencoded"
}
return a.send(ctx, method, path, query, body, -1, contentType, out)
}
// send is do with a caller-built body. length is the body's size, or -1
// when do built it from a string the request can measure itself.
func (a *httpAPI) send(ctx context.Context, method, path string, query url.Values, body io.Reader, length int64, contentType string, out any) error {
token, err := readToken(a.tokenFile)
if err != nil {
return err
}
client, err := a.httpClient()
if err != nil {
return err
}
u := a.base.JoinPath(path)
u.RawQuery = query.Encode()
req, err := http.NewRequestWithContext(ctx, method, u.String(), body)
if err != nil {
return fmt.Errorf("building %s %s: %w", method, path, err)
}
if length >= 0 {
req.ContentLength = length
}
if contentType != "" {
req.Header.Set("Content-Type", contentType)
}
req.Header.Set("Accept", "application/json")
req.Header.Set("Authorization", "PVEAPIToken="+token)
resp, err := client.Do(req)
if err != nil {
return err
}
defer func() {
_, _ = io.Copy(io.Discard, io.LimitReader(resp.Body, maxErrorBody))
_ = resp.Body.Close()
}()
if resp.StatusCode < 200 || resp.StatusCode > 299 {
raw, _ := io.ReadAll(io.LimitReader(resp.Body, maxErrorBody))
reason := strings.TrimPrefix(resp.Status, strconv.Itoa(resp.StatusCode))
return redact(newAPIError(resp.StatusCode, reason, raw), token)
}
var env struct {
Data json.RawMessage `json:"data"`
}
if err := json.NewDecoder(resp.Body).Decode(&env); err != nil {
return fmt.Errorf("decoding %s %s: %w", method, path, err)
}
if out == nil || len(env.Data) == 0 {
return nil
}
if err := json.Unmarshal(env.Data, out); err != nil {
return fmt.Errorf("decoding %s %s data: %w", method, path, err)
}
return nil
}
// redact removes the token's secret from an error Proxmox echoed back, so
// it cannot reach a log line, a span or a Proxy's status (spec entry 14).
func redact(e *apiError, token string) *apiError {
_, secret, _ := strings.Cut(token, "=")
if secret == "" {
return e
}
e.Message = strings.ReplaceAll(e.Message, secret, "<redacted>")
for k, v := range e.Errors {
e.Errors[k] = strings.ReplaceAll(v, secret, "<redacted>")
}
return e
}
// decodeConfig renders every config value as a string: Proxmox sends some
// scalars as JSON numbers (cores) and others as strings (memory). A null
// value is dropped.
func decodeConfig(raw map[string]json.RawMessage) (vmConfig, error) {
cfg := make(vmConfig, len(raw))
for k, v := range raw {
switch {
case len(v) == 0 || string(v) == "null":
continue
case v[0] == '"':
var s string
if err := json.Unmarshal(v, &s); err != nil {
return nil, fmt.Errorf("config key %s: %w", k, err)
}
cfg[k] = s
default:
cfg[k] = string(v)
}
}
return cfg, nil
}
func (a *httpAPI) Version(ctx context.Context) (v versionInfo, err error) {
err = a.do(ctx, http.MethodGet, "version", nil, nil, &v)
return v, err
}
func (a *httpAPI) NodeStatus(ctx context.Context, node string) (s nodeStatus, err error) {
err = a.do(ctx, http.MethodGet, apiPath("nodes", node, "status"), nil, nil, &s)
return s, err
}
func (a *httpAPI) StorageStatus(ctx context.Context, node, storage string) (s storageStatus, err error) {
err = a.do(ctx, http.MethodGet, apiPath("nodes", node, "storage", storage, "status"), nil, nil, &s)
return s, err
}
func (a *httpAPI) ClusterResources(ctx context.Context) (rs []vmResource, err error) {
err = a.do(ctx, http.MethodGet, "cluster/resources", url.Values{"type": {"vm"}}, nil, &rs)
return rs, err
}
func (a *httpAPI) VMConfig(ctx context.Context, node string, vmid int) (vmConfig, error) {
var raw map[string]json.RawMessage
if err := a.do(ctx, http.MethodGet, vmPath(node, vmid, "config"), nil, nil, &raw); err != nil {
return nil, err
}
return decodeConfig(raw)
}
func (a *httpAPI) SetVMConfig(ctx context.Context, node string, vmid int, params map[string]string) error {
form := url.Values{}
for k, v := range params {
form.Set(k, v)
}
return a.do(ctx, http.MethodPut, vmPath(node, vmid, "config"), nil, form, nil)
}
func (a *httpAPI) Clone(ctx context.Context, node string, vmid int, p cloneParams) (upid string, err error) {
form := url.Values{
"newid": {strconv.Itoa(p.NewID)},
"name": {p.Name},
"description": {p.Description},
"full": {"1"},
}
if p.Storage != "" {
form.Set("storage", p.Storage)
}
if p.Target != "" {
form.Set("target", p.Target)
}
err = a.do(ctx, http.MethodPost, vmPath(node, vmid, "clone"), nil, form, &upid)
return upid, err
}
func (a *httpAPI) Resize(ctx context.Context, node string, vmid int, disk, size string) (upid string, err error) {
form := url.Values{"disk": {disk}, "size": {size}}
err = a.do(ctx, http.MethodPut, vmPath(node, vmid, "resize"), nil, form, &upid)
return upid, err
}
func (a *httpAPI) Start(ctx context.Context, node string, vmid int) (upid string, err error) {
err = a.do(ctx, http.MethodPost, vmPath(node, vmid, "status", "start"), nil, url.Values{}, &upid)
return upid, err
}
func (a *httpAPI) Stop(ctx context.Context, node string, vmid int) (upid string, err error) {
err = a.do(ctx, http.MethodPost, vmPath(node, vmid, "status", "stop"), nil, url.Values{}, &upid)
return upid, err
}
func (a *httpAPI) VMStatus(ctx context.Context, node string, vmid int) (s vmStatus, err error) {
err = a.do(ctx, http.MethodGet, vmPath(node, vmid, "status", "current"), nil, nil, &s)
return s, err
}
// Destroy purges the VM from backup jobs and HA and removes disks no config
// references, so nothing of it is left behind (spec entry 11).
func (a *httpAPI) Destroy(ctx context.Context, node string, vmid int) (upid string, err error) {
query := url.Values{"purge": {"1"}, "destroy-unreferenced-disks": {"1"}}
err = a.do(ctx, http.MethodDelete, vmPath(node, vmid), query, nil, &upid)
return upid, err
}
func (a *httpAPI) TaskStatus(ctx context.Context, node, upid string) (s taskStatus, err error) {
err = a.do(ctx, http.MethodGet, apiPath("nodes", node, "tasks", upid, "status"), nil, nil, &s)
return s, err
}
func (a *httpAPI) AgentNetworkInterfaces(ctx context.Context, node string, vmid int) ([]agentInterface, error) {
var out struct {
Result []agentInterface `json:"result"`
}
err := a.do(ctx, http.MethodGet, vmPath(node, vmid, "agent", "network-get-interfaces"), nil, nil, &out)
return out.Result, err
}
// UploadISO streams r as the file part of a multipart upload (F12). The
// multipart framing is rendered up front, so the request carries an exact
// Content-Length, as Phase 0's curl -F upload did, while r itself is read
// in the transport's chunks and never held in memory.
func (a *httpAPI) UploadISO(ctx context.Context, node, storage, filename string, r io.Reader, size int64) (upid string, err error) {
var frame bytes.Buffer
mw := multipart.NewWriter(&frame)
if err := mw.WriteField("content", "iso"); err != nil {
return "", fmt.Errorf("framing upload: %w", err)
}
if _, err := mw.CreateFormFile("filename", filename); err != nil {
return "", fmt.Errorf("framing upload: %w", err)
}
head := bytes.Clone(frame.Bytes())
frame.Reset()
if err := mw.Close(); err != nil {
return "", fmt.Errorf("framing upload: %w", err)
}
tail := frame.Bytes()
if r == nil {
r = bytes.NewReader(nil)
}
body := io.MultiReader(bytes.NewReader(head), io.LimitReader(r, size), bytes.NewReader(tail))
length := int64(len(head)) + size + int64(len(tail))
err = a.send(ctx, http.MethodPost, apiPath("nodes", node, "storage", storage, "upload"), nil, body, length, mw.FormDataContentType(), &upid)
return upid, err
}
func (a *httpAPI) DeleteVolume(ctx context.Context, node, storage, volid string) (upid string, err error) {
err = a.do(ctx, http.MethodDelete, apiPath("nodes", node, "storage", storage, "content", volid), nil, nil, &upid)
return upid, err
}
internal/provider/proxmox/client_test.go
+874
View File
@@ -0,0 +1,874 @@
package proxmox
import (
"bytes"
"context"
"encoding/json"
"encoding/pem"
"errors"
"fmt"
"io"
stdlog "log"
"net"
"net/http"
"net/http/httptest"
"net/url"
"os"
"path/filepath"
"reflect"
"strings"
"sync"
"sync/atomic"
"testing"
"time"
sdktrace "go.opentelemetry.io/otel/sdk/trace"
"go.opentelemetry.io/otel/sdk/trace/tracetest"
"gitea.home.hrajfrisbee.cz/kacerr/egress-proxies-operator/internal/provider"
"gitea.home.hrajfrisbee.cz/kacerr/egress-proxies-operator/internal/tracing"
)
const testToken = "proxy-operator@pve!operator=0b4e1c2a-5d6f-4a7b-8c9d-0e1f2a3b4c5d"
// clientFixture is a TLS test server, a CA file trusting it, a token file
// and the config pointing at them.
type clientFixture struct {
srv *httptest.Server
cfg provider.ProxmoxConfig
tokenFile string
caFile string
}
// newClientFixture starts h on a TLS server whose error log is discarded:
// the tests that fail a handshake on purpose would otherwise print it.
func newClientFixture(t *testing.T, h http.HandlerFunc) *clientFixture {
t.Helper()
srv := httptest.NewUnstartedServer(h)
srv.Config.ErrorLog = stdlog.New(io.Discard, "", 0)
srv.StartTLS()
t.Cleanup(srv.Close)
return fixtureFor(t, srv)
}
func fixtureFor(t *testing.T, srv *httptest.Server) *clientFixture {
t.Helper()
dir := t.TempDir()
fx := &clientFixture{srv: srv, tokenFile: filepath.Join(dir, "token"), caFile: filepath.Join(dir, "ca.pem")}
writeFile(t, fx.tokenFile, testToken+"\n")
writeFile(t, fx.caFile, string(pem.EncodeToMemory(&pem.Block{Type: "CERTIFICATE", Bytes: srv.Certificate().Raw})))
fx.cfg = provider.ProxmoxConfig{URL: srv.URL, TokenFile: fx.tokenFile, CAFile: fx.caFile, RequestTimeoutSeconds: 30}
return fx
}
func (fx *clientFixture) api(t *testing.T, opts ...tracing.Option) *httpAPI {
t.Helper()
a, err := newHTTPAPI(fx.cfg, opts...)
if err != nil {
t.Fatalf("newHTTPAPI error = %v", err)
}
return a
}
func writeFile(t *testing.T, path, content string) {
t.Helper()
if err := os.WriteFile(path, []byte(content), 0o600); err != nil {
t.Fatalf("writing %s: %v", path, err)
}
}
// serveCapture writes a Phase 0 capture back byte for byte, reason phrase
// included, which net/http's ResponseWriter cannot do.
func serveCapture(t *testing.T, w http.ResponseWriter, c capture) {
t.Helper()
conn, buf, err := w.(http.Hijacker).Hijack()
if err != nil {
t.Errorf("hijack: %v", err)
return
}
defer func() { _ = conn.Close() }()
_, _ = fmt.Fprintf(buf, "HTTP/1.1 %d %s\r\nContent-Type: application/json;charset=UTF-8\r\nContent-Length: %d\r\nConnection: close\r\n\r\n%s",
c.Status, c.Reason, len(c.Body), c.Body)
_ = buf.Flush()
}
func serveJSON(w http.ResponseWriter, body string) {
w.Header().Set("Content-Type", "application/json;charset=UTF-8")
_, _ = io.WriteString(w, body)
}
func assertClass(t *testing.T, err, class error) {
t.Helper()
if got := classify("get", err); got != class {
t.Errorf("classify(%v) = %v, want %v", err, got, class)
}
}
func TestClient_sendsTokenHeaderFromFileOnEveryRequest(t *testing.T) {
t.Parallel()
var (
mu sync.Mutex
headers []string
)
fx := newClientFixture(t, func(w http.ResponseWriter, r *http.Request) {
mu.Lock()
headers = append(headers, r.Header.Get("Authorization"))
mu.Unlock()
serveJSON(w, readCapture(t, "version").Body)
})
a := fx.api(t)
const rotated = "proxy-operator@pve!operator=11111111-2222-3333-4444-555555555555"
for i, token := range []string{testToken, rotated} {
writeFile(t, fx.tokenFile, token+"\n")
v, err := a.Version(context.Background())
if err != nil || v.Version != "9.2.20" {
t.Fatalf("call %d: Version() = %+v, %v", i, v, err)
}
}
want := []string{"PVEAPIToken=" + testToken, "PVEAPIToken=" + rotated}
mu.Lock()
defer mu.Unlock()
if !reflect.DeepEqual(headers, want) {
t.Errorf("Authorization headers = %q, want %q", headers, want)
}
}
func TestClient_missingTokenFileIsTransient(t *testing.T) {
t.Parallel()
var requests atomic.Int32
fx := newClientFixture(t, func(http.ResponseWriter, *http.Request) { requests.Add(1) })
fx.cfg.TokenFile = filepath.Join(t.TempDir(), "absent")
_, err := fx.api(t).Version(context.Background())
if !errors.Is(err, provider.ErrTransient) || !errors.Is(err, errCredentials) {
t.Errorf("error = %v, want ErrTransient and errCredentials", err)
}
if n := requests.Load(); n != 0 {
t.Errorf("server saw %d requests, want none without a token", n)
}
}
func TestClient_unparsableTokenFileIsPermanent(t *testing.T) {
t.Parallel()
const sentinel = "SENTINEL-7f3e"
tests := []struct{ name, content string }{
{name: "empty", content: ""},
{name: "secret only", content: sentinel},
{name: "no secret", content: "proxy-operator@pve!operator"},
{name: "no token id", content: "proxy-operator@pve=" + sentinel},
{name: "no realm", content: "proxy-operator!operator=" + sentinel},
{name: "two lines", content: "proxy-operator@pve!operator=" + sentinel + "\nsecond=line"},
}
for _, tc := range tests {
t.Run(tc.name, func(t *testing.T) {
t.Parallel()
var requests atomic.Int32
fx := newClientFixture(t, func(http.ResponseWriter, *http.Request) { requests.Add(1) })
writeFile(t, fx.tokenFile, tc.content)
_, err := fx.api(t).Version(context.Background())
if !errors.Is(err, provider.ErrPermanent) || !errors.Is(err, errCredentials) {
t.Errorf("error = %v, want ErrPermanent and errCredentials", err)
}
if err != nil && strings.Contains(err.Error(), sentinel) {
t.Errorf("error carries the file's content: %v", err)
}
if n := requests.Load(); n != 0 {
t.Errorf("server saw %d requests, want none", n)
}
})
}
}
// TestClient_apiErrorFromStatusReasonAndBody serves every captured error
// verbatim and expects the apiError the captures describe (F1): the body's
// message, or the reason phrase when the body is empty.
func TestClient_apiErrorFromStatusReasonAndBody(t *testing.T) {
t.Parallel()
paths, err := filepath.Glob(filepath.Join("testdata", "err_*.json"))
if err != nil || len(paths) == 0 {
t.Fatalf("no error captures: %v", err)
}
for _, path := range paths {
name := strings.TrimSuffix(filepath.Base(path), ".json")
if readCapture(t, name).Status < 400 {
continue // err_tag_hyphen: hyphens turned out to be accepted (F9)
}
t.Run(name, func(t *testing.T) {
t.Parallel()
c := readCapture(t, name)
fx := newClientFixture(t, func(w http.ResponseWriter, _ *http.Request) { serveCapture(t, w, c) })
_, err := fx.api(t).Version(context.Background())
var ae *apiError
if !errors.As(err, &ae) {
t.Fatalf("error = %v (%T), want *apiError", err, err)
}
want := loadAPIError(t, name)
if ae.Status != want.Status || ae.Message != want.Message || !reflect.DeepEqual(ae.Errors, want.Errors) {
t.Errorf("apiError = %+v, want %+v", ae, want)
}
if ae.Message == "" || strings.HasSuffix(ae.Message, "\n") {
t.Errorf("message %q is empty or untrimmed", ae.Message)
}
})
}
}
func TestClient_decodesDataEnvelope(t *testing.T) {
t.Parallel()
ctx := context.Background()
tests := []struct {
capture string
call func(a *httpAPI) (any, error)
check func(t *testing.T, got any)
}{
{capture: "version", call: func(a *httpAPI) (any, error) { return a.Version(ctx) }, check: func(t *testing.T, got any) {
if v := got.(versionInfo); v.Version != "9.2.20" {
t.Errorf("version = %+v", v)
}
}},
{capture: "node_status", call: func(a *httpAPI) (any, error) { return a.NodeStatus(ctx, "pv1-1") }, check: func(t *testing.T, got any) {
if s := got.(nodeStatus); s.CurrentKernel.Release != "7.0.14-17-pve" {
t.Errorf("node status = %+v", s)
}
}},
{capture: "storage_status", call: func(a *httpAPI) (any, error) { return a.StorageStatus(ctx, "pv1-1", "local") }, check: func(t *testing.T, got any) {
if s := got.(storageStatus); s.Avail <= 0 || !strings.Contains(s.Content, "iso") {
t.Errorf("storage status = %+v", s)
}
}},
{capture: "cluster_resources_during_clone", call: func(a *httpAPI) (any, error) { return a.ClusterResources(ctx) }, check: func(t *testing.T, got any) {
rs := got.([]vmResource)
for _, r := range rs {
if r.VMID == 199999 && r.Lock == "clone" && r.Name == "" {
return
}
}
t.Errorf("resources = %+v, want the nameless locked clone", rs)
}},
{capture: "vm_config_clone", call: func(a *httpAPI) (any, error) { return a.VMConfig(ctx, "pv1-1", 199999) }, check: func(t *testing.T, got any) {
cfg := got.(vmConfig)
if cfg["cores"] != "2" || cfg["memory"] != "2048" || cfg["agent"] != "1" || cfg["boot"] != "order=scsi0" {
t.Errorf("config scalars = cores %q memory %q agent %q boot %q", cfg["cores"], cfg["memory"], cfg["agent"], cfg["boot"])
}
if m := parseDescription(cfg["description"]); !m.Managed || m.UID == "" {
t.Errorf("description %q did not survive decoding", cfg["description"])
}
}},
{capture: "status_current_paused", call: func(a *httpAPI) (any, error) { return a.VMStatus(ctx, "pv1-1", 199999) }, check: func(t *testing.T, got any) {
if s := got.(vmStatus); s.Status != "running" || s.QMPStatus != "paused" {
t.Errorf("status = %+v", s)
}
}},
{capture: "task_status_stopped_error", call: func(a *httpAPI) (any, error) { return a.TaskStatus(ctx, "pv1-1", "UPID:pv1-1:x") }, check: func(t *testing.T, got any) {
if s := got.(taskStatus); s.Status != "stopped" || s.ExitStatus != "shrinking disks is not supported" {
t.Errorf("task = %+v", s)
}
}},
{capture: "agent_network_get_interfaces", call: func(a *httpAPI) (any, error) { return a.AgentNetworkInterfaces(ctx, "pv1-1", 199999) }, check: func(t *testing.T, got any) {
if ip := firstIPv4(got.([]agentInterface)); ip != "192.168.0.150" {
t.Errorf("first IPv4 = %q", ip)
}
}},
{capture: "set_config_response", call: func(a *httpAPI) (any, error) {
return nil, a.SetVMConfig(ctx, "pv1-1", 199999, map[string]string{"cores": "1"})
}, check: func(*testing.T, any) {}},
{capture: "clone_response", call: func(a *httpAPI) (any, error) { return a.Clone(ctx, "pv1-1", 9001, cloneParams{NewID: 199999}) }, check: wantUPID("qmclone")},
{capture: "resize_response", call: func(a *httpAPI) (any, error) { return a.Resize(ctx, "pv1-1", 199999, "scsi0", "4G") }, check: wantUPID("resize")},
{capture: "start_response", call: func(a *httpAPI) (any, error) { return a.Start(ctx, "pv1-1", 199999) }, check: wantUPID("qmstart")},
{capture: "stop_response", call: func(a *httpAPI) (any, error) { return a.Stop(ctx, "pv1-1", 199999) }, check: wantUPID("qmstop")},
{capture: "destroy_response", call: func(a *httpAPI) (any, error) { return a.Destroy(ctx, "pv1-1", 199999) }, check: wantUPID("qmdestroy")},
{capture: "upload_response", call: func(a *httpAPI) (any, error) {
return a.UploadISO(ctx, "pv1-1", "local", "x.iso", strings.NewReader("iso"), 3)
}, check: wantUPID("imgcopy")},
{capture: "content_delete_response", call: func(a *httpAPI) (any, error) {
return a.DeleteVolume(ctx, "pv1-1", "proxy-seeds", "proxy-seeds:iso/x.iso")
}, check: wantUPID("imgdel")},
}
for _, tc := range tests {
t.Run(tc.capture, func(t *testing.T) {
t.Parallel()
c := readCapture(t, tc.capture)
fx := newClientFixture(t, func(w http.ResponseWriter, r *http.Request) {
_, _ = io.Copy(io.Discard, r.Body)
serveCapture(t, w, c)
})
got, err := tc.call(fx.api(t))
if err != nil {
t.Fatalf("call error = %v", err)
}
tc.check(t, got)
})
}
}
func wantUPID(kind string) func(t *testing.T, got any) {
return func(t *testing.T, got any) {
t.Helper()
if upid := got.(string); !strings.HasPrefix(upid, "UPID:pv1-1:") || !strings.Contains(upid, ":"+kind+":") {
t.Errorf("upid = %q, want a %s UPID of pv1-1", upid, kind)
}
}
}
// seenRequest is what a test server records of one request.
type seenRequest struct {
method, path, escapedPath string
query, form url.Values
contentType string
}
func recordingServer(t *testing.T, seen chan<- seenRequest) *clientFixture {
t.Helper()
return newClientFixture(t, func(w http.ResponseWriter, r *http.Request) {
req := seenRequest{method: r.Method, path: r.URL.Path, escapedPath: r.URL.EscapedPath(), query: r.URL.Query(), contentType: r.Header.Get("Content-Type")}
if r.Method == http.MethodPost || r.Method == http.MethodPut {
if err := r.ParseForm(); err != nil {
t.Errorf("parsing form: %v", err)
}
req.form = r.PostForm
}
seen <- req
serveJSON(w, `{"data":null}`)
})
}
// TestClient_methodPathAndParams pins the HTTP method, path and parameters
// of every seam method to the endpoint map of spec entry 17.
func TestClient_methodPathAndParams(t *testing.T) {
t.Parallel()
ctx := context.Background()
const upid = "UPID:pv1-1:0010FBF1:02804D0D:6AB378EA:qmstart:199999:proxy-operator@pve!operator:"
tests := map[string]struct {
call func(a *httpAPI) error
method string
path string
query url.Values
form url.Values
}{
"Version": {call: func(a *httpAPI) error { _, err := a.Version(ctx); return err }, method: "GET", path: "/api2/json/version"},
"NodeStatus": {call: func(a *httpAPI) error { _, err := a.NodeStatus(ctx, "pv1-1"); return err },
method: "GET", path: "/api2/json/nodes/pv1-1/status"},
"StorageStatus": {call: func(a *httpAPI) error { _, err := a.StorageStatus(ctx, "pv1-1", "proxy-seeds"); return err },
method: "GET", path: "/api2/json/nodes/pv1-1/storage/proxy-seeds/status"},
"ClusterResources": {call: func(a *httpAPI) error { _, err := a.ClusterResources(ctx); return err },
method: "GET", path: "/api2/json/cluster/resources", query: url.Values{"type": {"vm"}}},
"VMConfig": {call: func(a *httpAPI) error { _, err := a.VMConfig(ctx, "pv1-1", 150042); return err },
method: "GET", path: "/api2/json/nodes/pv1-1/qemu/150042/config"},
"SetVMConfig": {call: func(a *httpAPI) error {
return a.SetVMConfig(ctx, "pv1-1", 150042, map[string]string{"cores": "1", "ide2": "proxy-seeds:iso/proxy-x-seed.iso,media=cdrom"})
}, method: "PUT", path: "/api2/json/nodes/pv1-1/qemu/150042/config",
form: url.Values{"cores": {"1"}, "ide2": {"proxy-seeds:iso/proxy-x-seed.iso,media=cdrom"}}},
"Clone": {call: func(a *httpAPI) error {
_, err := a.Clone(ctx, "pv1-1", 9001, cloneParams{NewID: 150042, Name: "proxy-x", Description: "a=1\nb=2", Storage: "local-lvm", Target: "pv1-2"})
return err
}, method: "POST", path: "/api2/json/nodes/pv1-1/qemu/9001/clone",
form: url.Values{"newid": {"150042"}, "name": {"proxy-x"}, "description": {"a=1\nb=2"}, "full": {"1"}, "storage": {"local-lvm"}, "target": {"pv1-2"}}},
"Resize": {call: func(a *httpAPI) error { _, err := a.Resize(ctx, "pv1-1", 150042, "scsi0", "8G"); return err },
method: "PUT", path: "/api2/json/nodes/pv1-1/qemu/150042/resize", form: url.Values{"disk": {"scsi0"}, "size": {"8G"}}},
"Start": {call: func(a *httpAPI) error { _, err := a.Start(ctx, "pv1-1", 150042); return err },
method: "POST", path: "/api2/json/nodes/pv1-1/qemu/150042/status/start", form: url.Values{}},
"Stop": {call: func(a *httpAPI) error { _, err := a.Stop(ctx, "pv1-1", 150042); return err },
method: "POST", path: "/api2/json/nodes/pv1-1/qemu/150042/status/stop", form: url.Values{}},
"VMStatus": {call: func(a *httpAPI) error { _, err := a.VMStatus(ctx, "pv1-1", 150042); return err },
method: "GET", path: "/api2/json/nodes/pv1-1/qemu/150042/status/current"},
"Destroy": {call: func(a *httpAPI) error { _, err := a.Destroy(ctx, "pv1-1", 150042); return err },
method: "DELETE", path: "/api2/json/nodes/pv1-1/qemu/150042", query: url.Values{"purge": {"1"}, "destroy-unreferenced-disks": {"1"}}},
"TaskStatus": {call: func(a *httpAPI) error { _, err := a.TaskStatus(ctx, "pv1-1", upid); return err },
method: "GET", path: "/api2/json/nodes/pv1-1/tasks/" + upid + "/status"},
"AgentNetworkInterfaces": {call: func(a *httpAPI) error { _, err := a.AgentNetworkInterfaces(ctx, "pv1-1", 150042); return err },
method: "GET", path: "/api2/json/nodes/pv1-1/qemu/150042/agent/network-get-interfaces"},
"UploadISO": {call: func(a *httpAPI) error {
_, err := a.UploadISO(ctx, "pv1-1", "proxy-seeds", "proxy-x-seed.iso", strings.NewReader("iso"), 3)
return err
}, method: "POST", path: "/api2/json/nodes/pv1-1/storage/proxy-seeds/upload"}, // multipart, see TestClient_uploadStreamsMultipart
"DeleteVolume": {call: func(a *httpAPI) error {
_, err := a.DeleteVolume(ctx, "pv1-1", "proxy-seeds", "proxy-seeds:iso/proxy-x-seed.iso")
return err
}, method: "DELETE", path: "/api2/json/nodes/pv1-1/storage/proxy-seeds/content/proxy-seeds:iso/proxy-x-seed.iso"},
}
for m := range reflect.TypeFor[api]().Methods() {
if _, ok := tests[m.Name]; !ok {
t.Errorf("no row for seam method %s", m.Name)
}
}
for name, tc := range tests {
t.Run(name, func(t *testing.T) {
t.Parallel()
seen := make(chan seenRequest, 1)
fx := recordingServer(t, seen)
if err := tc.call(fx.api(t)); err != nil {
t.Fatalf("call error = %v", err)
}
got := <-seen
if got.method != tc.method || got.path != tc.path {
t.Errorf("request = %s %s, want %s %s", got.method, got.path, tc.method, tc.path)
}
if len(got.query) != 0 || len(tc.query) != 0 {
if !reflect.DeepEqual(got.query, tc.query) {
t.Errorf("query = %v, want %v", got.query, tc.query)
}
}
if tc.form == nil {
if got.form != nil && len(got.form) != 0 {
t.Errorf("form = %v, want none", got.form)
}
return
}
if got.contentType != "application/x-www-form-urlencoded" {
t.Errorf("content type = %q, want a urlencoded form", got.contentType)
}
for k, want := range tc.form {
if !reflect.DeepEqual(got.form[k], want) {
t.Errorf("form %s = %q, want %q", k, got.form[k], want)
}
}
if len(got.form) != len(tc.form) {
t.Errorf("form = %v, want exactly %v", got.form, tc.form)
}
})
}
}
func TestClient_cloneOmitsEmptyStorageAndTarget(t *testing.T) {
t.Parallel()
seen := make(chan seenRequest, 1)
fx := recordingServer(t, seen)
if _, err := fx.api(t).Clone(context.Background(), "pv1-1", 9001, cloneParams{NewID: 150042, Name: "proxy-x"}); err != nil {
t.Fatalf("Clone error = %v", err)
}
got := <-seen
if _, ok := got.form["storage"]; ok {
t.Errorf("form = %v, want no storage", got.form)
}
if _, ok := got.form["target"]; ok {
t.Errorf("form = %v, want no target", got.form)
}
}
// TestClient_destroyQueryFlags checks that destroy leaves nothing behind:
// the purge and unreferenced-disk flags travel in the query of the DELETE
// (spec entry 11).
func TestClient_destroyQueryFlags(t *testing.T) {
t.Parallel()
seen := make(chan seenRequest, 1)
fx := recordingServer(t, seen)
if _, err := fx.api(t).Destroy(context.Background(), "pv1-1", 150042); err != nil {
t.Fatalf("Destroy error = %v", err)
}
got := <-seen
if got.method != http.MethodDelete || got.query.Get("purge") != "1" || got.query.Get("destroy-unreferenced-disks") != "1" {
t.Errorf("request = %s %s?%s, want DELETE with purge=1 and destroy-unreferenced-disks=1", got.method, got.path, got.query.Encode())
}
}
// TestClient_volIDSlashIsEscaped sends the volid's "/" as %2F, the way
// Phase 0 deleted volumes, so the router sees one path segment.
func TestClient_volIDSlashIsEscaped(t *testing.T) {
t.Parallel()
seen := make(chan seenRequest, 1)
fx := recordingServer(t, seen)
if _, err := fx.api(t).DeleteVolume(context.Background(), "pv1-1", "proxy-seeds", "proxy-seeds:iso/proxy-x-seed.iso"); err != nil {
t.Fatalf("DeleteVolume error = %v", err)
}
if got := <-seen; !strings.HasSuffix(got.escapedPath, "/content/proxy-seeds:iso%2Fproxy-x-seed.iso") {
t.Errorf("escaped path = %q, want the volid as one segment with %%2F", got.escapedPath)
}
}
func TestClient_http1Only(t *testing.T) {
t.Parallel()
protos := make(chan int, 2)
srv := httptest.NewUnstartedServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
protos <- r.ProtoMajor
serveJSON(w, readCapture(t, "version").Body)
}))
srv.EnableHTTP2 = true
srv.StartTLS()
t.Cleanup(srv.Close)
resp, err := srv.Client().Get(srv.URL)
if err != nil {
t.Fatalf("sanity request error = %v", err)
}
_ = resp.Body.Close()
if p := <-protos; p != 2 {
t.Fatalf("sanity: server spoke HTTP/%d to an h2 client, want 2", p)
}
if _, err := fixtureFor(t, srv).api(t).Version(context.Background()); err != nil {
t.Fatalf("Version error = %v", err)
}
if p := <-protos; p != 1 {
t.Errorf("client spoke HTTP/%d, want HTTP/1.1 (F18)", p)
}
}
func TestClient_requestTimeoutIsTransient(t *testing.T) {
t.Parallel()
fx := newClientFixture(t, func(_ http.ResponseWriter, r *http.Request) {
select {
case <-r.Context().Done():
case <-time.After(5 * time.Second):
}
})
a := fx.api(t)
a.timeout = 100 * time.Millisecond
start := time.Now()
_, err := a.Version(context.Background())
if elapsed := time.Since(start); elapsed > 2*time.Second {
t.Errorf("Version took %v, want the 100ms request timeout to cut it", elapsed)
}
var ne net.Error
if !errors.As(err, &ne) || !ne.Timeout() {
t.Errorf("error = %v, want a timeout", err)
}
assertClass(t, err, provider.ErrTransient)
}
// TestClient_cancelledContextReturnsPromptly also checks that the
// cancellation touches only that request: the next call on the same client
// succeeds.
func TestClient_cancelledContextReturnsPromptly(t *testing.T) {
t.Parallel()
fx := newClientFixture(t, func(w http.ResponseWriter, r *http.Request) {
if strings.HasSuffix(r.URL.Path, "/status") {
select {
case <-r.Context().Done():
case <-time.After(5 * time.Second):
}
return
}
serveJSON(w, readCapture(t, "version").Body)
})
a := fx.api(t)
ctx, cancel := context.WithCancel(context.Background())
time.AfterFunc(50*time.Millisecond, cancel)
start := time.Now()
_, err := a.NodeStatus(ctx, "pv1-1")
if elapsed := time.Since(start); elapsed > time.Second {
t.Errorf("NodeStatus took %v after cancel, want prompt", elapsed)
}
if !errors.Is(err, context.Canceled) {
t.Errorf("error = %v, want context.Canceled", err)
}
assertClass(t, err, provider.ErrTransient)
if _, err := a.Version(context.Background()); err != nil {
t.Errorf("next call error = %v, want the client unaffected", err)
}
}
func tracedFixture(t *testing.T) (*clientFixture, *tracetest.SpanRecorder, *sdktrace.TracerProvider, func() []string) {
t.Helper()
var (
mu sync.Mutex
traceparent []string
)
fx := newClientFixture(t, func(w http.ResponseWriter, r *http.Request) {
mu.Lock()
traceparent = append(traceparent, r.Header.Get("Traceparent"))
mu.Unlock()
serveJSON(w, readCapture(t, "version").Body)
})
sr := tracetest.NewSpanRecorder()
tp := sdktrace.NewTracerProvider(sdktrace.WithSpanProcessor(sr))
t.Cleanup(func() { _ = tp.Shutdown(context.Background()) })
sent := func() []string {
mu.Lock()
defer mu.Unlock()
return append([]string(nil), traceparent...)
}
return fx, sr, tp, sent
}
func TestClient_childSpanUnderParent(t *testing.T) {
t.Parallel()
fx, sr, tp, traceparent := tracedFixture(t)
a := fx.api(t, tracing.WithTracerProvider(tp))
ctx, parent := tp.Tracer("test").Start(context.Background(), "provider.get")
if _, err := a.Version(ctx); err != nil {
t.Fatalf("Version error = %v", err)
}
parent.End()
var child sdktrace.ReadOnlySpan
for _, s := range sr.Ended() {
if s.Name() != "provider.get" {
child = s
}
}
if child == nil {
t.Fatalf("no HTTP span recorded; spans: %d", len(sr.Ended()))
}
if child.Parent().SpanID() != parent.SpanContext().SpanID() || child.SpanContext().TraceID() != parent.SpanContext().TraceID() {
t.Errorf("HTTP span %q is not a child of provider.get", child.Name())
}
secret := testToken[strings.Index(testToken, "=")+1:]
for _, kv := range child.Attributes() {
if strings.Contains(kv.Value.Emit(), secret) {
t.Errorf("span attribute %s carries the token", kv.Key)
}
}
if sent := traceparent(); len(sent) != 1 || !strings.Contains(sent[0], child.SpanContext().TraceID().String()) {
t.Errorf("traceparent sent = %q, want the trace id", sent)
}
}
func TestClient_noSpanWithoutParent(t *testing.T) {
t.Parallel()
fx, sr, tp, traceparent := tracedFixture(t)
if _, err := fx.api(t, tracing.WithTracerProvider(tp)).Version(context.Background()); err != nil {
t.Fatalf("Version error = %v", err)
}
if n := len(sr.Ended()); n != 0 {
t.Errorf("got %d spans without a parent, want none", n)
}
if sent := traceparent(); len(sent) != 1 || sent[0] != "" {
t.Errorf("traceparent sent = %q, want none", sent)
}
}
// TestClient_tokenNeverInErrorsOrLogs has the server echo the secret back
// in every place an error's text comes from; neither the error nor the
// decorator's V(2) lines may carry it.
func TestClient_tokenNeverInErrorsOrLogs(t *testing.T) {
t.Parallel()
const secret = "SENTINEL-SECRET-9d41"
token := "proxy-operator@pve!operator=" + secret
tests := []struct {
name string
resp func(auth string) capture
}{
{name: "401 body echoes the header", resp: func(auth string) capture {
return capture{Status: 401, Reason: "Authentication failed!", Body: `{"data":null,"message":"bad credentials: ` + auth + `\n"}`}
}},
{name: "401 reason echoes the header", resp: func(auth string) capture {
return capture{Status: 401, Reason: "bad credentials " + auth}
}},
{name: "400 errors echo the header", resp: func(auth string) capture {
return capture{Status: 400, Reason: "Parameter verification failed.", Body: `{"data":null,"message":"Parameter verification failed.\n","errors":{"token":"` + auth + `"}}`}
}},
}
for _, tc := range tests {
t.Run(tc.name, func(t *testing.T) {
t.Parallel()
fx := newClientFixture(t, func(w http.ResponseWriter, r *http.Request) {
serveCapture(t, w, tc.resp(r.Header.Get("Authorization")))
})
writeFile(t, fx.tokenFile, token)
logs := &logCapture{}
ctx := logs.context(2)
_, err := observe(fx.api(t), "proxmox-home").Version(ctx)
var ae *apiError
if !errors.As(err, &ae) {
t.Fatalf("error = %v, want an apiError", err)
}
if !strings.Contains(err.Error(), "<redacted>") {
t.Errorf("error = %v, want the echoed secret redacted", err)
}
for _, s := range append([]string{err.Error(), fmt.Sprintf("%+v", err), fmt.Sprintf("%#v", ae)}, logs.all()...) {
if strings.Contains(s, secret) {
t.Errorf("secret leaked: %s", s)
}
}
if len(logs.all()) != 1 {
t.Errorf("got %d log lines, want the one call line", len(logs.all()))
}
})
}
}
func TestClient_caFileTrustsPrivateCA(t *testing.T) {
t.Parallel()
fx := newClientFixture(t, func(w http.ResponseWriter, _ *http.Request) { serveJSON(w, readCapture(t, "version").Body) })
if _, err := fx.api(t).Version(context.Background()); err != nil {
t.Errorf("with the CA file: error = %v, want nil", err)
}
fx.cfg.CAFile = ""
_, err := fx.api(t).Version(context.Background())
if !isTLSVerification(err) {
t.Errorf("with system roots: error = %v, want a verification failure", err)
}
}
// TestClient_caFileLoadedLazily covers spec entry 14: a missing CA file is
// Transient and retried until it lands; one without a certificate is
// Permanent.
func TestClient_caFileLoadedLazily(t *testing.T) {
t.Parallel()
fx := newClientFixture(t, func(w http.ResponseWriter, _ *http.Request) { serveJSON(w, readCapture(t, "version").Body) })
pemBytes, err := os.ReadFile(fx.caFile)
if err != nil {
t.Fatal(err)
}
fx.cfg.CAFile = filepath.Join(t.TempDir(), "ca.pem")
a := fx.api(t)
if _, err := a.Version(context.Background()); !errors.Is(err, provider.ErrTransient) {
t.Errorf("missing CA file: error = %v, want ErrTransient", err)
}
writeFile(t, fx.cfg.CAFile, "not a certificate")
if _, err := a.Version(context.Background()); !errors.Is(err, provider.ErrPermanent) {
t.Errorf("CA file without a certificate: error = %v, want ErrPermanent", err)
}
writeFile(t, fx.cfg.CAFile, string(pemBytes))
if _, err := a.Version(context.Background()); err != nil {
t.Errorf("CA file landed: error = %v, want nil", err)
}
}
func TestClient_x509FailureIsPermanent(t *testing.T) {
t.Parallel()
tests := []struct {
name string
mutate func(fx *clientFixture)
}{
{name: "unknown authority", mutate: func(fx *clientFixture) { fx.cfg.CAFile = "" }},
{name: "host not in the certificate", mutate: func(fx *clientFixture) {
fx.cfg.URL = strings.Replace(fx.srv.URL, "127.0.0.1", "localhost", 1)
}},
}
for _, tc := range tests {
t.Run(tc.name, func(t *testing.T) {
t.Parallel()
var requests atomic.Int32
fx := newClientFixture(t, func(http.ResponseWriter, *http.Request) { requests.Add(1) })
tc.mutate(fx)
_, err := fx.api(t).Version(context.Background())
if !isTLSVerification(err) {
t.Errorf("error = %v, want a verification failure", err)
}
assertClass(t, err, provider.ErrPermanent)
if n := requests.Load(); n != 0 {
t.Errorf("server saw %d requests, want none past the handshake", n)
}
})
}
}
func TestClient_insecureSkipsVerification(t *testing.T) {
t.Parallel()
fx := newClientFixture(t, func(w http.ResponseWriter, _ *http.Request) { serveJSON(w, readCapture(t, "version").Body) })
fx.cfg.CAFile = ""
fx.cfg.InsecureSkipTLSVerify = true
if _, err := fx.api(t).Version(context.Background()); err != nil {
t.Errorf("error = %v, want nil with verification off", err)
}
}
// gatedReader hands out first, then waits for release before handing out
// rest. A client that read the whole body before sending would never get
// past first.
type gatedReader struct {
first, rest []byte
release <-chan struct{}
waited bool
}
func (g *gatedReader) Read(p []byte) (int, error) {
if len(g.first) > 0 {
n := copy(p, g.first)
g.first = g.first[n:]
return n, nil
}
if !g.waited {
select {
case <-g.release:
case <-time.After(5 * time.Second):
return 0, errors.New("the server saw nothing of the file before the reader was drained: the body is not streamed")
}
g.waited = true
}
if len(g.rest) == 0 {
return 0, io.EOF
}
n := copy(p, g.rest)
g.rest = g.rest[n:]
return n, nil
}
func TestClient_uploadStreamsMultipart(t *testing.T) {
t.Parallel()
first := bytes.Repeat([]byte("a"), 256<<10)
rest := bytes.Repeat([]byte("b"), 256<<10)
sawFirst := make(chan struct{})
type upload struct {
contentLength int64
chunked bool
content, name string
fileName, bytes string
}
got := make(chan upload, 1)
fx := newClientFixture(t, func(w http.ResponseWriter, r *http.Request) {
u := upload{contentLength: r.ContentLength, chunked: len(r.TransferEncoding) > 0}
mr, err := r.MultipartReader()
if err != nil {
t.Errorf("multipart reader: %v", err)
return
}
for {
part, err := mr.NextPart()
if err == io.EOF {
break
}
if err != nil {
t.Errorf("next part: %v", err)
return
}
switch part.FormName() {
case "content":
b, _ := io.ReadAll(part)
u.content = string(b)
case "filename":
u.name, u.fileName = part.FormName(), part.FileName()
var buf bytes.Buffer
if _, err := io.CopyN(&buf, part, 1); err != nil {
t.Errorf("reading the file part: %v", err)
return
}
close(sawFirst)
_, _ = io.Copy(&buf, part)
u.bytes = buf.String()
}
}
got <- u
serveJSON(w, readCapture(t, "upload_response").Body)
})
body := &gatedReader{first: first, rest: rest, release: sawFirst}
size := int64(len(first) + len(rest))
upid, err := fx.api(t).UploadISO(context.Background(), "pv1-1", "proxy-seeds", "proxy-x-seed.iso", body, size)
if err != nil {
t.Fatalf("UploadISO error = %v", err)
}
if !strings.Contains(upid, ":imgcopy:") {
t.Errorf("upid = %q, want the imgcopy task", upid)
}
u := <-got
if u.chunked || u.contentLength <= size {
t.Errorf("Content-Length %d (chunked %v), want an exact length above the file's %d bytes", u.contentLength, u.chunked, size)
}
if u.content != "iso" || u.name != "filename" || u.fileName != "proxy-x-seed.iso" {
t.Errorf("fields content=%q part=%q file=%q, want iso, filename, proxy-x-seed.iso (F12)", u.content, u.name, u.fileName)
}
if u.bytes != string(first)+string(rest) {
t.Errorf("server got %d file bytes, want the %d sent", len(u.bytes), size)
}
}
func TestDecodeConfig_scalarsAsStrings(t *testing.T) {
t.Parallel()
in := map[string]json.RawMessage{"cores": json.RawMessage(`2`), "memory": json.RawMessage(`"2048"`), "template": json.RawMessage(`1`), "balloon": json.RawMessage(`null`), "description": json.RawMessage(`"a\nb"`)}
got, err := decodeConfig(in)
if err != nil {
t.Fatalf("decodeConfig error = %v", err)
}
want := vmConfig{"cores": "2", "memory": "2048", "template": "1", "description": "a\nb"}
if !reflect.DeepEqual(got, want) {
t.Errorf("decodeConfig = %v, want %v", got, want)
}
}
func TestReadToken_trimsTrailingNewline(t *testing.T) {
t.Parallel()
path := filepath.Join(t.TempDir(), "token")
writeFile(t, path, " "+testToken+"\r\n")
if got, err := readToken(path); err != nil || got != testToken {
t.Errorf("readToken = %q, %v; want the token without whitespace", got, err)
}
}
internal/provider/proxmox/create.go
+428
View File
@@ -0,0 +1,428 @@
package proxmox
import (
"context"
"fmt"
"maps"
"regexp"
"slices"
"strconv"
"strings"
"github.com/go-logr/logr"
"gitea.home.hrajfrisbee.cz/kacerr/egress-proxies-operator/internal/provider"
)
// maxProbes bounds the VMID probe of spec entry 2.
const maxProbes = 32
// diskKey is a Proxmox disk slot: the keys a boot disk, a CD-ROM or a
// cloud-init drive can sit on.
var diskKey = regexp.MustCompile(`^(ide|sata|scsi|virtio)[0-9]+$`)
// ownedVM is a VM Create has cloned or adopted: where it lives and its
// config as last read, which the config and resize steps start from.
type ownedVM struct {
node string
vmid int
cfg vmConfig
}
// Create is spec entry 10: validate placement, then, under the per-name
// lock, adopt the VM a previous attempt left or clone a new one, and carry
// it through seed, config, resize and start until it runs.
func (p *Provider) Create(ctx context.Context, req provider.CreateRequest) (string, error) {
node, mt, tmpl, err := p.placement(req)
if err != nil {
return "", p.wrapErr("create", "", err)
}
log := p.logger(ctx).WithValues("name", req.Name)
log.V(2).Info("proxmox create resolved",
"zone", node, "machineType", req.Placement.MachineType, "cores", mt.Cores, "memoryMiB", mt.MemoryMiB,
"diskGiB", mt.DiskGiB, "image", req.Placement.Image, "templateVMID", tmpl.VMID, "storage", p.cfg.Storage,
"isoStorage", p.cfg.ISOStorage, "bridge", p.cfg.Bridge, "vlanTag", p.cfg.VLANTag, "cloudInitBytes", len(req.CloudInit))
unlock, err := p.lockName(ctx, req.Name)
if err != nil {
return "", p.wrapErr("create", "", err)
}
defer unlock()
vm, err := p.create(ctx, log, node, mt, tmpl, req)
if err != nil {
return "", p.wrapErr("create", "", err)
}
log.V(1).Info("proxmox vm ensured running", "vmid", vm.vmid, "node", vm.node)
return formatProviderID(vm.vmid, req.Name), nil
}
// placement is step 1: every catalogue key is checked before any backend
// call, and an unknown one is Permanent naming the field and the value
// (spec entry 3). Region is ignored.
func (p *Provider) placement(req provider.CreateRequest) (string, provider.ProxmoxMachineType, provider.ProxmoxTemplate, error) {
var (
mt provider.ProxmoxMachineType
tmpl provider.ProxmoxTemplate
)
if req.Name == "" {
return "", mt, tmpl, permanentf("the request has no instance name")
}
node, err := p.node(req.Placement.Zone)
if err != nil {
return "", mt, tmpl, err
}
mt, ok := p.cfg.MachineTypes[req.Placement.MachineType]
if !ok {
return "", mt, tmpl, permanentf("placement.machineType %q is not in proxmox.machineTypes", req.Placement.MachineType)
}
tmpl, ok = p.cfg.Templates[req.Placement.Image]
if !ok {
return "", mt, tmpl, permanentf("placement.image %q is not in proxmox.templates", req.Placement.Image)
}
return node, mt, tmpl, nil
}
// create is steps 2 to 9. The template is checked only when a clone is
// needed: a VM a previous attempt created no longer depends on it (spec
// entry 4), so a template removed since must not fail its adoption.
func (p *Provider) create(ctx context.Context, log logr.Logger, node string, mt provider.ProxmoxMachineType, tmpl provider.ProxmoxTemplate, req provider.CreateRequest) (ownedVM, error) {
rs, err := p.api.ClusterResources(ctx)
if err != nil {
return ownedVM{}, fmt.Errorf("listing cluster resources: %w", err)
}
vm, adopted, err := p.adoptByName(ctx, rs, req)
if err != nil {
return ownedVM{}, err
}
if !adopted {
src, err := p.template(ctx, rs, req.Placement.Image, tmpl.VMID, mt)
if err != nil {
return ownedVM{}, err
}
if vm, adopted, err = p.allocate(ctx, log, rs, node, src, req); err != nil {
return ownedVM{}, err
}
}
log = log.WithValues("vmid", vm.vmid, "node", vm.node)
if adopted {
st, err := p.api.VMStatus(ctx, vm.node, vm.vmid)
if err != nil {
return ownedVM{}, fmt.Errorf("reading status of VM %d: %w", vm.vmid, err)
}
log.V(1).Info("proxmox vm adopted", "status", st.Status, "qmpstatus", st.QMPStatus)
if st.Status == "running" {
return vm, nil
}
}
return vm, p.provision(ctx, log, vm, mt, req)
}
// adoptByName is the by-name half of spec entry 2: two VMs with the name
// are for an admin to sort out, one goes through the ownership check.
func (p *Provider) adoptByName(ctx context.Context, rs []vmResource, req provider.CreateRequest) (ownedVM, bool, error) {
var found []vmResource
for _, r := range rs {
if r.Type == "qemu" && r.Template == 0 && r.Name == req.Name {
found = append(found, r)
}
}
switch len(found) {
case 0:
return ownedVM{}, false, nil
case 1:
return p.inspect(ctx, found[0], req)
}
vmids := make([]string, len(found))
for i, r := range found {
vmids[i] = strconv.Itoa(r.VMID)
}
return ownedVM{}, false, permanentf("VMs %s are all named %s; remove all but one", strings.Join(vmids, ", "), req.Name)
}
// inspect is the ownership check of spec entry 2 on a VMID that is taken.
// It waits out a task's lock (a clone in progress is also nameless), then
// reads the name and the mark: another name is someone else's VM, and our
// name without our mark is never taken over.
func (p *Provider) inspect(ctx context.Context, r vmResource, req provider.CreateRequest) (ownedVM, bool, error) {
cfg, err := p.waitUnlocked(ctx, r.Node, r.VMID, p.budgets.clone)
if err != nil {
return ownedVM{}, false, err
}
if cfg["name"] != req.Name {
return ownedVM{}, false, nil
}
mark := parseDescription(cfg["description"])
switch {
case !mark.Managed:
return ownedVM{}, false, permanentf("VM %d is named %s but carries no proxy-operator mark; it was not created by this operator", r.VMID, req.Name)
case mark.UID != "" && mark.UID != req.UID:
return ownedVM{}, false, permanentf("VM %d named %s belongs to proxy uid %s, not %s", r.VMID, req.Name, mark.UID, req.UID)
case cfg["lock"] == lockSuspended:
return ownedVM{}, false, permanentf("VM %d named %s is hibernated (lock suspended); resume or destroy it in Proxmox", r.VMID, req.Name)
}
return ownedVM{node: r.Node, vmid: r.VMID, cfg: cfg}, true, nil
}
// template finds the clone source and refuses one that is not a template
// or brings a Proxmox cloud-init drive, which would compete with the seed
// (spec entry 4). The clone copies the template's config, so what the
// config or resize step would refuse on the clone is refused here, while no
// VM exists that a fixed template could no longer help.
func (p *Provider) template(ctx context.Context, rs []vmResource, image string, vmid int, mt provider.ProxmoxMachineType) (vmResource, error) {
i := slices.IndexFunc(rs, func(r vmResource) bool { return r.VMID == vmid && r.Type == "qemu" })
if i < 0 {
return vmResource{}, permanentf("placement.image %q: template VM %d does not exist", image, vmid)
}
src := rs[i]
cfg, err := p.api.VMConfig(ctx, src.Node, vmid)
if err != nil {
return vmResource{}, fmt.Errorf("reading config of template %d: %w", vmid, err)
}
if cfg["template"] != "1" {
return vmResource{}, permanentf("placement.image %q: VM %d is not a template", image, vmid)
}
for _, k := range slices.Sorted(maps.Keys(cfg)) {
if diskKey.MatchString(k) && strings.Contains(cfg[k], "cloudinit") {
return vmResource{}, permanentf("placement.image %q: template %d has a Proxmox cloud-init drive on %s; remove it, the provider attaches its own seed", image, vmid, k)
}
}
shape := ownedVM{node: src.Node, vmid: vmid, cfg: cfg}
if _, err := p.configParams(shape, mt, ""); err != nil {
return vmResource{}, err
}
if _, _, _, err := resizeTarget(shape, mt.DiskGiB); err != nil {
return vmResource{}, err
}
return src, nil
}
// allocate is the probe of spec entry 2: the attempt-th VMID for the UID
// is cloned onto when the listing shows it free, and put through the
// ownership check when it is taken, or when the clone finds it taken after
// all (F4), by a VM that is nameless or carries our name. The boolean
// reports an adoption rather than a fresh clone.
func (p *Provider) allocate(ctx context.Context, log logr.Logger, rs []vmResource, node string, src vmResource, req provider.CreateRequest) (ownedVM, bool, error) {
taken := make(map[int]vmResource, len(rs))
for _, r := range rs {
taken[r.VMID] = r
}
rng := p.cfg.VMIDRange
probes := min(maxProbes, rng.Max-rng.Min+1)
for attempt := range probes {
vmid := vmidFor(req.UID, rng.Min, rng.Max, attempt)
r, isTaken := taken[vmid]
if !isTaken {
vm, err := p.clone(ctx, log, node, src, vmid, req)
if !isAlreadyExists(err) {
return vm, false, err
}
var found bool
if r, found, err = p.resolve(ctx, vmid); err != nil {
return ownedVM{}, false, err
}
if !found {
log.V(1).Info("proxmox vmid taken by something other than a VM, probing on", "vmid", vmid, "attempt", attempt)
continue
}
}
// A listed name other than ours is someone else's VM whatever its
// lock: only a clone in progress is nameless.
if r.Type == "qemu" && r.Template == 0 && (r.Name == "" || r.Name == req.Name) {
vm, ours, err := p.inspect(ctx, r, req)
if err != nil || ours {
return vm, ours, err
}
}
log.V(1).Info("proxmox vmid taken by another vm, probing on", "vmid", vmid, "attempt", attempt)
}
return ownedVM{}, false, transientf("no free VMID for %s after %d probes in %d..%d", req.Name, probes, rng.Min, rng.Max)
}
// clone is step 4: a full clone carrying the mark, so the VM is never
// unmarked (spec entry 8), then its unlocked config. A taken VMID comes
// back as the API's error for allocate to judge.
func (p *Provider) clone(ctx context.Context, log logr.Logger, node string, src vmResource, vmid int, req provider.CreateRequest) (ownedVM, error) {
params := cloneParams{
NewID: vmid,
Name: req.Name,
Description: formatDescription(req.UID, p.now()),
Storage: p.cfg.Storage,
}
if node != src.Node {
params.Target = node
}
upid, err := p.api.Clone(ctx, src.Node, src.VMID, params)
if err != nil {
return ownedVM{}, fmt.Errorf("cloning template %d to VM %d: %w", src.VMID, vmid, err)
}
if err := p.waitTask(ctx, log, upid, p.budgets.clone); err != nil {
return ownedVM{}, fmt.Errorf("cloning template %d to VM %d: %w", src.VMID, vmid, err)
}
cfg, err := p.waitUnlocked(ctx, node, vmid, p.budgets.clone)
if err != nil {
return ownedVM{}, err
}
return ownedVM{node: node, vmid: vmid, cfg: cfg}, nil
}
// provision is steps 5 to 9, each safe to repeat on a VM a previous attempt
// left stopped: the upload overwrites, the config write replaces, and the
// resize is skipped once the disk is large enough. Everything that can be
// refused from the config alone is checked before the first write.
func (p *Provider) provision(ctx context.Context, log logr.Logger, vm ownedVM, mt provider.ProxmoxMachineType, req provider.CreateRequest) error {
params, err := p.configParams(vm, mt, req.Name)
if err != nil {
return err
}
disk, size, grow, err := resizeTarget(vm, mt.DiskGiB)
if err != nil {
return err
}
if err := p.uploadSeed(ctx, log, vm.node, req); err != nil {
return err
}
if err := p.api.SetVMConfig(ctx, vm.node, vm.vmid, params); err != nil {
return fmt.Errorf("configuring VM %d: %w", vm.vmid, err)
}
if grow {
upid, err := p.api.Resize(ctx, vm.node, vm.vmid, disk, size)
if err != nil {
return fmt.Errorf("resizing %s of VM %d to %s: %w", disk, vm.vmid, size, err)
}
if err := p.waitTask(ctx, log, upid, p.budgets.resize); err != nil {
return fmt.Errorf("resizing %s of VM %d to %s: %w", disk, vm.vmid, size, err)
}
}
upid, err := p.api.Start(ctx, vm.node, vm.vmid)
if err != nil {
return fmt.Errorf("starting VM %d: %w", vm.vmid, err)
}
if err := p.waitTask(ctx, log, upid, p.budgets.start); err != nil {
return fmt.Errorf("starting VM %d: %w", vm.vmid, err)
}
return p.waitRunning(ctx, vm.node, vm.vmid)
}
// configParams is step 6's write: size, agent, tag and seed, and net0 only
// when the config sets a bridge or a VLAN tag (spec entry 7).
func (p *Provider) configParams(vm ownedVM, mt provider.ProxmoxMachineType, name string) (map[string]string, error) {
params := map[string]string{
"cores": strconv.Itoa(int(mt.Cores)),
"memory": strconv.FormatInt(mt.MemoryMiB, 10),
"agent": "1",
"tags": proxyTag,
"ide2": seedVolID(p.cfg.ISOStorage, name) + ",media=cdrom",
}
if p.cfg.Bridge == "" && p.cfg.VLANTag == 0 {
return params, nil
}
net0 := vm.cfg["net0"]
if net0 == "" {
return nil, permanentf("VM %d has no net0 to apply proxmox.bridge or proxmox.vlanTag to; the template needs a network device", vm.vmid)
}
params["net0"] = rewriteNet0(net0, p.cfg.Bridge, p.cfg.VLANTag)
return params, nil
}
// rewriteNet0 replaces or appends bridge= and tag= in a net0 value, keeping
// every other option, the model and the MAC in place. An empty bridge or a
// zero tag leaves that option alone.
func rewriteNet0(net0, bridge string, tag int) string {
want := map[string]string{}
if bridge != "" {
want["bridge"] = bridge
}
if tag != 0 {
want["tag"] = strconv.Itoa(tag)
}
opts := strings.Split(net0, ",")
for i, opt := range opts {
key, _, _ := strings.Cut(opt, "=")
if v, ok := want[key]; ok {
opts[i] = key + "=" + v
delete(want, key)
}
}
for _, key := range []string{"bridge", "tag"} {
if v, ok := want[key]; ok {
opts = append(opts, key+"="+v)
}
}
return strings.Join(opts, ",")
}
// resizeTarget decides step 7: grow the boot disk to diskGiB when that is
// larger than its size= (spec entry 6). A disk is never shrunk.
func resizeTarget(vm ownedVM, diskGiB int64) (disk, size string, grow bool, err error) {
if diskGiB == 0 {
return "", "", false, nil
}
disk, ok := bootDisk(vm.cfg)
if !ok {
return "", "", false, permanentf("VM %d has no boot disk to grow to %dG", vm.vmid, diskGiB)
}
cur, ok := diskBytes(vm.cfg[disk])
if !ok {
return "", "", false, permanentf("boot disk %s of VM %d has no readable size in %q", disk, vm.vmid, vm.cfg[disk])
}
if cur >= diskGiB<<30 {
return "", "", false, nil
}
return disk, strconv.FormatInt(diskGiB, 10) + "G", true, nil
}
// bootDisk is the first disk of boot: order=… that is not a CD-ROM, else
// the legacy bootdisk key, else the first present of scsi0, virtio0,
// sata0 and ide0. Proxmox appends the seed's ide2 to the order (F19), so
// CD-ROMs are skipped.
func bootDisk(cfg vmConfig) (string, bool) {
isDisk := func(k string) bool {
v, ok := cfg[k]
return ok && diskKey.MatchString(k) && diskOption(v, "media") != "cdrom"
}
for dev := range strings.SplitSeq(diskOption(cfg["boot"], "order"), ";") {
if isDisk(dev) {
return dev, true
}
}
if k := cfg["bootdisk"]; isDisk(k) {
return k, true
}
for _, k := range []string{"scsi0", "virtio0", "sata0", "ide0"} {
if isDisk(k) {
return k, true
}
}
return "", false
}
// diskOption returns the value of key= among a config value's
// comma-separated options.
func diskOption(value, key string) string {
for opt := range strings.SplitSeq(value, ",") {
if v, ok := strings.CutPrefix(opt, key+"="); ok {
return v
}
}
return ""
}
// diskBytes reads a disk value's size=, in Proxmox's <n>[KMGT] notation
// (bytes without a suffix).
func diskBytes(disk string) (int64, bool) {
s := diskOption(disk, "size")
if s == "" {
return 0, false
}
shift := map[byte]uint{'K': 10, 'M': 20, 'G': 30, 'T': 40}
unit, ok := shift[s[len(s)-1]]
if ok {
s = s[:len(s)-1]
}
n, err := strconv.ParseFloat(s, 64)
if err != nil || n < 0 {
return 0, false
}
return int64(n * float64(int64(1)<<unit)), true
}
internal/provider/proxmox/create_test.go
+1287
View File
File diff suppressed because it is too large Load Diff
internal/provider/proxmox/delete.go
+152
View File
@@ -0,0 +1,152 @@
package proxmox
import (
"context"
"errors"
"fmt"
"time"
"github.com/go-logr/logr"
"gitea.home.hrajfrisbee.cz/kacerr/egress-proxies-operator/internal/provider"
)
// Delete is spec entry 11: under the per-name lock, stop the VM when it is
// ours, delete its seed, then destroy it. Everything already gone is
// success, and nothing of ours is left when nil is returned, so Get
// afterwards is NotFound.
func (p *Provider) Delete(ctx context.Context, id string) error {
vmid, name, err := parseProviderID(id)
if err != nil {
return p.wrapErr("delete", id, err)
}
log := p.logger(ctx).WithValues("vmid", vmid, "name", name)
unlock, err := p.lockName(ctx, name)
if err != nil {
return p.wrapErr("delete", id, err)
}
defer unlock()
node, err := p.destroy(ctx, log, vmid, name)
if err != nil {
return p.wrapErr("delete", id, err)
}
if node == "" {
if err := p.deleteSeedEverywhere(ctx, log, name); err != nil {
return p.wrapErr("delete", id, err)
}
} else {
log = log.WithValues("node", node)
}
log.V(1).Info("proxmox vm deleted", "destroyed", node != "")
return nil
}
// destroy is steps 2 to 7. It returns the node of the VM it destroyed, or
// "" when there was no VM of ours to destroy: then the seed's node is not
// known, and the seed is looked for on every node.
func (p *Provider) destroy(ctx context.Context, log logr.Logger, vmid int, name string) (string, error) {
r, ok, err := p.resolve(ctx, vmid)
if err != nil {
return "", err
}
if !ok {
log.V(1).Info("proxmox vm already gone")
return "", nil
}
log = log.WithValues("node", r.Node)
if r.Lock != "" && r.Lock != lockSuspended {
return "", transientf("VM %d is locked (%s)", vmid, r.Lock)
}
if r.Status == "unknown" {
return "", transientf("VM %d has status unknown: its node %s is not answering, or the VM is too new to be reported yet", vmid, r.Node)
}
cfg, err := p.api.VMConfig(ctx, r.Node, vmid)
if isNotExist(err) {
log.V(1).Info("proxmox vm already gone")
return "", nil
}
if err != nil {
return "", fmt.Errorf("reading config of VM %d: %w", vmid, err)
}
if cfg["name"] != name {
log.V(1).Info("proxmox vm at this vmid is not ours, left alone", "vmName", cfg["name"])
return "", nil
}
if !parseDescription(cfg["description"]).Managed {
log.V(1).Info("proxmox vm carries no proxy-operator mark, left alone")
return "", nil
}
// A hibernated VM is stopped with lock suspended, and the destroy
// takes it, state volume included (captured on pv1-1).
if lock := cfg["lock"]; lock != "" && lock != lockSuspended {
return "", transientf("VM %d is locked (%s)", vmid, lock)
}
// The listing shows a paused VM as running, and Stop ends it too.
if r.Status != "stopped" {
if err := p.runTask(ctx, log, "stopping", vmid, p.budgets.stop, func() (string, error) { return p.api.Stop(ctx, r.Node, vmid) }); err != nil {
return "", err
}
}
// The seed goes while the VM is still there: once it is gone, Get is
// NotFound and nothing calls Delete again to finish the job.
if err := p.deleteSeed(ctx, log, r.Node, name); err != nil {
return "", err
}
if err := p.runTask(ctx, log, "destroying", vmid, p.budgets.destroy, func() (string, error) { return p.api.Destroy(ctx, r.Node, vmid) }); err != nil {
return "", err
}
return r.Node, nil
}
// runTask submits a task on a VM and waits for it. A VM that is gone by the
// time the task is submitted is success: it is what the task was for.
func (p *Provider) runTask(ctx context.Context, log logr.Logger, what string, vmid int, budget time.Duration, submit func() (string, error)) error {
upid, err := submit()
if isNotExist(err) {
log.V(1).Info("proxmox vm already gone")
return nil
}
if err != nil {
return fmt.Errorf("%s VM %d: %w", what, vmid, err)
}
if err := p.waitTask(ctx, log, upid, budget); err != nil {
return fmt.Errorf("%s VM %d: %w", what, vmid, err)
}
return nil
}
// deleteSeed is step 6 on one node. A missing volume still ends in a
// successful task (F13), so nothing is matched.
func (p *Provider) deleteSeed(ctx context.Context, log logr.Logger, node, name string) error {
volid := seedVolID(p.cfg.ISOStorage, name)
upid, err := p.api.DeleteVolume(ctx, node, p.cfg.ISOStorage, volid)
if err != nil {
return fmt.Errorf("deleting seed %s on %s: %w", volid, node, err)
}
if err := p.waitTask(ctx, log, upid, p.budgets.seedDelete); err != nil {
return fmt.Errorf("deleting seed %s on %s: %w", volid, node, err)
}
return nil
}
// deleteSeedEverywhere is step 6 when the VM's node is not known: every
// configured node is tried, and one that fails is logged and skipped, so a
// node that is down or lacks the storage cannot hold the Delete up. An
// ended ctx fails the call instead, and so does a Permanent refusal such as
// a 403 for Datastore.Allocate, which every node would give and which
// would otherwise leave the seed behind unseen.
func (p *Provider) deleteSeedEverywhere(ctx context.Context, log logr.Logger, name string) error {
for _, node := range p.cfg.Nodes {
err := p.deleteSeed(ctx, log, node, name)
if err == nil {
continue
}
if ctx.Err() != nil || errors.Is(classify("delete", err), provider.ErrPermanent) {
return err
}
log.V(1).Info("proxmox seed delete failed on a node, skipped", "node", node, "reason", err.Error())
}
return nil
}
internal/provider/proxmox/delete_test.go
+760
View File
@@ -0,0 +1,760 @@
package proxmox
import (
"context"
"errors"
"net/http"
"slices"
"strconv"
"strings"
"sync"
"testing"
"time"
"gitea.home.hrajfrisbee.cz/kacerr/egress-proxies-operator/internal/provider"
)
// putSeed uploads testName's seed to the fake's ISO storage, as Create
// leaves it.
func (fx *fixture) putSeed() {
fx.t.Helper()
upid, err := fx.fake.UploadISO(context.Background(), fakeNode, fakeISOStorage, seedISOName(testName), strings.NewReader("seed"), 4)
if err != nil {
fx.t.Fatalf("UploadISO error = %v", err)
}
pollTask(fx.t, fx.fake, upid)
}
func (fx *fixture) seedPresent() bool {
_, ok := fx.fake.iso(fakeISOStorage, seedISOName(testName))
return ok
}
// deleteFails runs Delete on fx.id(), checks the error's class and returns
// its text.
func (fx *fixture) deleteFails(class error) string {
fx.t.Helper()
err := fx.p.Delete(fx.ctx, fx.id())
assertProviderError(fx.t, err, "delete", class)
return err.Error()
}
func (fx *fixture) delete() {
fx.t.Helper()
if err := fx.p.Delete(fx.ctx, fx.id()); err != nil {
fx.t.Fatalf("Delete error = %v", err)
}
}
// seedDeleteNodes returns the node of every DeleteVolume call, read from
// the decorator's lines; the fixture's api must be observed.
func (fx *fixture) seedDeleteNodes() []string {
lines := fx.logs.matching(`"call"="DeleteVolume"`)
nodes := make([]string, 0, len(lines))
for _, l := range lines {
_, rest, _ := strings.Cut(l, `"node"="`)
node, _, _ := strings.Cut(rest, `"`)
nodes = append(nodes, node)
}
return nodes
}
// twoNodeFixture is an observed fixture at V(1) with nodes pv1-1 and pv1-2.
func twoNodeFixture(t *testing.T) *fixture {
t.Helper()
fx := newFixture(t, 1)
fx.p.api = observe(fx.fake, testProvider)
fx.p.cfg.Nodes = []string{fakeNode, "pv1-2"}
return fx
}
func TestDelete_malformedIDIsPermanent(t *testing.T) {
t.Parallel()
for _, id := range []string{"", "vms/150042", "vms/0150042/" + testName, "vm/150042/" + testName, "vms/abc/" + testName, "vms/150042/a/b", "vms/99/" + testName} {
t.Run(id, func(t *testing.T) {
t.Parallel()
fx := newFixture(t, 1)
assertProviderError(t, fx.p.Delete(fx.ctx, id), "delete", provider.ErrPermanent)
if calls := fx.fake.callLog(); len(calls) != 0 {
t.Errorf("calls = %v, want none", calls)
}
})
}
}
func TestDelete_twiceSucceeds(t *testing.T) {
t.Parallel()
fx := newFixture(t, 1)
fx.addOurVM(testVMID, testName, "running")
fx.putSeed()
fx.delete()
fx.delete()
if _, ok := fx.fake.vmSnapshot(testVMID); ok {
t.Error("VM still exists")
}
if fx.seedPresent() {
t.Error("seed still exists")
}
for method, want := range map[string]int{"Stop": 1, "Destroy": 1, "DeleteVolume": 2} {
if got := fx.fake.callCount(method); got != want {
t.Errorf("%s calls = %d, want %d", method, got, want)
}
}
}
// TestDelete_absentStillDeletesSeedOnEveryNode replaces the plan's
// TestDelete_absentIsNoOp: spec entry 11 deletes the seed even when the VM
// is gone, on every node, because the VM's node is no longer known.
func TestDelete_absentStillDeletesSeedOnEveryNode(t *testing.T) {
t.Parallel()
fx := twoNodeFixture(t)
fx.putSeed()
fx.delete()
if got := fx.seedDeleteNodes(); !slices.Equal(got, []string{fakeNode, "pv1-2"}) {
t.Errorf("seed deleted on %v, want every node in config order", got)
}
if fx.seedPresent() {
t.Error("seed still exists")
}
for _, method := range []string{"VMConfig", "Stop", "Destroy"} {
if n := fx.fake.callCount(method); n != 0 {
t.Errorf("%s calls = %d, want 0 for an absent VM", method, n)
}
}
if lines := fx.logs.matching(`"msg"="proxmox vm already gone"`); len(lines) != 1 {
t.Errorf("already-gone lines = %v, want one", lines)
}
}
func TestDelete_containerAtVMIDIsLeftAlone(t *testing.T) {
t.Parallel()
fx := twoNodeFixture(t)
fx.addOurVM(testVMID, testName, "running").kind = "lxc"
fx.delete()
if vm, ok := fx.fake.vmSnapshot(testVMID); !ok || vm.status != "running" {
t.Errorf("container = %+v, %v; want it untouched", vm, ok)
}
if got := fx.seedDeleteNodes(); len(got) != 2 {
t.Errorf("seed deleted on %v, want every node", got)
}
}
// TestDelete_foreignNameLeavesVMAlone replaces the plan's
// TestDelete_foreignNameIsNoOp: the VM is not touched, but our seed is
// still deleted, on every node since ours is gone.
func TestDelete_foreignNameLeavesVMAlone(t *testing.T) {
t.Parallel()
fx := twoNodeFixture(t)
fx.addOurVM(testVMID, "proxy-someoneelse00000", "running")
fx.putSeed()
fx.delete()
if vm, ok := fx.fake.vmSnapshot(testVMID); !ok || vm.status != "running" {
t.Errorf("foreign VM = %+v, %v; want it running and untouched", vm, ok)
}
for _, method := range []string{"Stop", "Destroy", "SetVMConfig"} {
if n := fx.fake.callCount(method); n != 0 {
t.Errorf("%s calls = %d, want 0", method, n)
}
}
if got := fx.seedDeleteNodes(); !slices.Equal(got, []string{fakeNode, "pv1-2"}) {
t.Errorf("seed deleted on %v, want every node", got)
}
lines := fx.logs.matching(`"msg"="proxmox vm at this vmid is not ours, left alone"`)
if len(lines) != 1 || !strings.Contains(lines[0], `"vmName"="proxy-someoneelse00000"`) {
t.Errorf("no-op lines = %v, want one naming the other VM", lines)
}
}
// TestDelete_unmarkedLeavesVMAlone replaces the plan's
// TestDelete_unmarkedIsNoOp, for the same reason.
func TestDelete_unmarkedLeavesVMAlone(t *testing.T) {
t.Parallel()
for name, desc := range map[string]string{
"no description": "",
"managed false": "proxy-operator-managed=false\nproxy-operator-uid=" + testUID,
} {
t.Run(name, func(t *testing.T) {
t.Parallel()
fx := twoNodeFixture(t)
vm := fx.fake.addVM(testVMID, testName, "stopped")
if desc != "" {
vm.config["description"] = desc
}
fx.delete()
if _, ok := fx.fake.vmSnapshot(testVMID); !ok {
t.Error("unmarked VM was destroyed")
}
if n := fx.fake.callCount("Destroy"); n != 0 {
t.Errorf("Destroy calls = %d, want 0", n)
}
if got := fx.seedDeleteNodes(); len(got) != 2 {
t.Errorf("seed deleted on %v, want every node", got)
}
if lines := fx.logs.matching(`"msg"="proxmox vm carries no proxy-operator mark, left alone"`); len(lines) != 1 {
t.Errorf("no-op lines = %v, want one", lines)
}
})
}
}
// TestDelete_runningVMStoppedFirst covers a paused VM too: the listing
// shows it as running (F14), and Stop ends it. The fake refuses to destroy
// a running VM, and each task takes two polls, so a Destroy that does not
// wait for the stop fails.
func TestDelete_runningVMStoppedFirst(t *testing.T) {
t.Parallel()
// pv1-1 took a stop on a paused VM as a task that ended OK with the VM
// stopped (Phase 10 captures), which is what the fake does for all three.
accepted := readCapture(t, "stop_paused_response")
done := decodeData[taskStatus](t, "task_status_stop_paused")
after := decodeData[vmStatus](t, "status_current_after_stop_paused")
if accepted.Status != http.StatusOK || done.Type != "qmstop" || done.ExitStatus != "OK" || after.Status != "stopped" || after.QMPStatus != "stopped" {
t.Fatalf("captures = %d, %+v, %+v; want the stop accepted, OK, and the VM stopped", accepted.Status, done, after)
}
for _, qmp := range []string{"running", "paused", "prelaunch"} {
t.Run(qmp, func(t *testing.T) {
t.Parallel()
fx := newFixture(t, 1)
fx.fake.taskPolls = 2
fx.addOurVM(testVMID, testName, "running").qmpstatus = qmp
fx.delete()
log := fx.fake.callLog()
stop, destroy := slices.Index(log, "Stop"), slices.Index(log, "Destroy")
if stop < 0 || destroy < stop {
t.Fatalf("calls = %v, want Stop before Destroy", log)
}
if n := callsBefore(log[stop:], "TaskStatus", "DeleteVolume"); n != fx.fake.taskPolls+1 {
t.Errorf("stop task polled %d times before the seed delete, want %d", n, fx.fake.taskPolls+1)
}
if _, ok := fx.fake.vmSnapshot(testVMID); ok {
t.Error("VM still exists")
}
})
}
}
func TestDelete_stoppedVMSkipsStop(t *testing.T) {
t.Parallel()
fx := newFixture(t, 1)
fx.addOurVM(testVMID, testName, "stopped")
fx.delete()
want := []string{"ClusterResources", "VMConfig", "DeleteVolume", "TaskStatus", "Destroy", "TaskStatus"}
if got := fx.fake.callLog(); !slices.Equal(got, want) {
t.Errorf("calls = %v, want %v", got, want)
}
}
func TestDelete_lockedIsTransient(t *testing.T) {
t.Parallel()
for _, lock := range []string{"backup", "migrate", "snapshot", "rollback", "create"} {
t.Run("resources lock "+lock, func(t *testing.T) {
t.Parallel()
fx := newFixture(t, 1)
fx.addOurVM(testVMID, testName, "running").lock = lock
fx.putSeed()
if msg := fx.deleteFails(provider.ErrTransient); !strings.Contains(msg, "locked ("+lock+")") {
t.Errorf("error = %s, want it to name the lock", msg)
}
if got := fx.fake.callLog(); !slices.Equal(got, []string{"UploadISO", "TaskStatus", "ClusterResources"}) {
t.Errorf("calls = %v, want only the listing after the seed setup", got)
}
if !fx.seedPresent() {
t.Error("seed deleted while the VM is locked")
}
})
}
t.Run("mid-clone nameless entry", func(t *testing.T) {
t.Parallel()
fx := newFixture(t, 1)
fx.fake.addTemplate(9001, "debian-13-template")
if _, err := fx.fake.Clone(fx.ctx, fakeNode, 9001, cloneParams{NewID: testVMID, Name: testName, Description: formatDescription(testUID, testCreatedAt)}); err != nil {
t.Fatalf("Clone error = %v", err)
}
fx.deleteFails(provider.ErrTransient)
if _, ok := fx.fake.vmSnapshot(testVMID); !ok {
t.Error("mid-clone VM is gone")
}
})
t.Run("lock taken between the resources and config reads", func(t *testing.T) {
t.Parallel()
fx := newFixture(t, 1)
vm := fx.addOurVM(testVMID, testName, "running")
mutateOn(fx.fake, "VMConfig", func() { vm.lock = "backup" })
fx.deleteFails(provider.ErrTransient)
if n := fx.fake.callCount("Stop"); n != 0 {
t.Errorf("Stop calls = %d, want 0", n)
}
})
}
func TestDelete_unknownStatusIsTransient(t *testing.T) {
t.Parallel()
fx := newFixture(t, 1)
fx.addOurVM(testVMID, testName, "running").status = "unknown"
if msg := fx.deleteFails(provider.ErrTransient); !strings.Contains(msg, fakeNode) {
t.Errorf("error = %s, want it to name the node", msg)
}
if n := fx.fake.callCount("VMConfig"); n != 0 {
t.Errorf("VMConfig calls = %d, want 0 while the node is unknown", n)
}
}
// TestDelete_hibernatedVMIsDestroyedIfOurs: on pv1-1 the destroy took a
// hibernated VM, suspend state volume included, and its task ended OK.
// Delete used to refuse it as Permanent on the belief that the destroy
// would. A hibernated VM that is not ours is still left alone.
func TestDelete_hibernatedVMIsDestroyedIfOurs(t *testing.T) {
t.Parallel()
t.Run("ours", func(t *testing.T) {
t.Parallel()
accepted := readCapture(t, "destroy_hibernated_response")
done := decodeData[taskStatus](t, "task_status_destroy_hibernated")
if accepted.Status != http.StatusOK || done.Type != "qmdestroy" || done.ExitStatus != "OK" {
t.Fatalf("captures = %d, %+v; want the destroy accepted and its task OK", accepted.Status, done)
}
fx := newFixture(t, 1)
fx.addOurVM(testVMID, testName, "stopped").lock = lockSuspended
fx.putSeed()
fx.delete()
want := []string{"ClusterResources", "VMConfig", "DeleteVolume", "TaskStatus", "Destroy", "TaskStatus"}
if got := fx.fake.callLog()[2:]; !slices.Equal(got, want) {
t.Errorf("calls = %v, want %v: no stop, since the VM is not running", got, want)
}
if _, ok := fx.fake.vmSnapshot(testVMID); ok {
t.Error("hibernated VM still exists")
}
if fx.seedPresent() {
t.Error("seed still exists")
}
})
t.Run("someone else's", func(t *testing.T) {
t.Parallel()
fx := newFixture(t, 1)
fx.addOurVM(testVMID, "proxy-someoneelse00000", "stopped").lock = lockSuspended
fx.delete()
if vm, ok := fx.fake.vmSnapshot(testVMID); !ok || vm.lock != lockSuspended {
t.Errorf("foreign hibernated VM = %+v, %v; want it untouched", vm, ok)
}
})
}
func TestDelete_removesSeedISO(t *testing.T) {
t.Parallel()
fx := twoNodeFixture(t)
fx.addOurVM(testVMID, testName, "stopped").node = "pv1-2"
fx.putSeed()
fx.delete()
if fx.seedPresent() {
t.Error("seed still exists")
}
if got := fx.seedDeleteNodes(); !slices.Equal(got, []string{"pv1-2"}) {
t.Errorf("seed deleted on %v, want only the VM's node pv1-2", got)
}
lines := fx.logs.matching(`"call"="DeleteVolume"`)
if len(lines) != 1 || !strings.Contains(lines[0], `"volid"="`+seedVolID(fakeISOStorage, testName)+`"`) || !strings.Contains(lines[0], `"storage"="`+fakeISOStorage+`"`) {
t.Errorf("DeleteVolume lines = %v, want the seed's volid on the ISO storage", lines)
}
}
// TestDelete_seedDeleteIsTaskAndIdempotent replaces the plan's
// TestDelete_toleratesMissingISO: Proxmox ends a delete of a missing volume
// with a successful imgdel task (F13), so nothing is matched, but the task
// is still waited on.
func TestDelete_seedDeleteIsTaskAndIdempotent(t *testing.T) {
t.Parallel()
missing := decodeData[taskStatus](t, "task_status_volume_missing")
if missing.Type != "imgdel" || missing.ExitStatus != "OK" {
t.Fatalf("capture task_status_volume_missing = %+v, want an imgdel task ending OK", missing)
}
for _, withSeed := range []bool{true, false} {
t.Run("seed present "+strconv.FormatBool(withSeed), func(t *testing.T) {
t.Parallel()
fx := newFixture(t, 1)
fx.addOurVM(testVMID, testName, "stopped")
if withSeed {
fx.putSeed()
}
fx.fake.taskPolls = 2
fx.delete()
log := fx.fake.callLog()
last := len(log) - 1
for log[last] != "DeleteVolume" {
last--
}
if n := callsBefore(log[last:], "TaskStatus", "Destroy"); n != fx.fake.taskPolls+1 {
t.Errorf("seed delete task polled %d times, want %d: %v", n, fx.fake.taskPolls+1, log)
}
if fx.seedPresent() {
t.Error("seed still exists")
}
})
}
}
// TestDelete_seedDeleteFailureOnVMNodeFails: the VM outlives a failed seed
// delete, so Get keeps finding it and the next Delete finishes the job with
// the same class of answer (review findings CR1, CR2, P1).
func TestDelete_seedDeleteFailureOnVMNodeFails(t *testing.T) {
t.Parallel()
tests := []struct {
name string
setup func(fx *fixture)
class error
}{
{name: "token lacks Datastore.Allocate", setup: func(fx *fixture) {
fx.fake.failWith("DeleteVolume", loadAPIError(t, "err_forbidden_volume_delete"))
}, class: provider.ErrPermanent},
{name: "pveproxy 596", setup: func(fx *fixture) { fx.fake.failWith("DeleteVolume", fakeErr(596, "Broken pipe")) }, class: provider.ErrTransient},
{name: "task failed", setup: func(fx *fixture) {
fx.fake.hookOn("TaskStatus", func() error {
if fx.fake.callCount("DeleteVolume") > 0 {
return fakeErr(500, "fake: task log unreadable")
}
return nil
})
}, class: provider.ErrTransient},
}
for _, tc := range tests {
t.Run(tc.name, func(t *testing.T) {
t.Parallel()
fx := newFixture(t, 1)
fx.addOurVM(testVMID, testName, "stopped")
tc.setup(fx)
fx.putSeed()
fx.deleteFails(tc.class)
if n := fx.fake.callCount("Destroy"); n != 0 {
t.Errorf("Destroy calls = %d, want 0 while the seed is still there", n)
}
if _, err := fx.p.Get(fx.ctx, fx.id()); errors.Is(err, provider.ErrNotFound) {
t.Fatalf("Get = %v, want the VM still found so Delete runs again", err)
}
fx.deleteFails(tc.class)
fx.fake.clearHooks()
fx.delete()
if _, ok := fx.fake.vmSnapshot(testVMID); ok {
t.Error("VM still exists after the retry")
}
if fx.seedPresent() {
t.Error("seed still exists after the retry")
}
})
}
}
func TestDelete_seedDeleteNodeFailureSkippedWhenNodeUnknown(t *testing.T) {
t.Parallel()
t.Run("one node refuses", func(t *testing.T) {
t.Parallel()
fx := twoNodeFixture(t)
fx.putSeed()
var calls int
fx.fake.hookOn("DeleteVolume", func() error {
if calls++; calls == 1 {
return fakeErr(500, "fake: storage 'proxy-seeds' is not available on node 'pv1-1'")
}
return nil
})
fx.delete()
if got := fx.seedDeleteNodes(); !slices.Equal(got, []string{fakeNode, "pv1-2"}) {
t.Errorf("seed deleted on %v, want both nodes tried", got)
}
lines := fx.logs.matching(`"msg"="proxmox seed delete failed on a node, skipped"`)
if len(lines) != 1 || !strings.Contains(lines[0], `"node"="pv1-1"`) || !strings.Contains(lines[0], "not available") {
t.Errorf("skip lines = %v, want one naming pv1-1 and the reason", lines)
}
if fx.seedPresent() {
t.Error("seed still exists after pv1-2's delete")
}
})
t.Run("a refused privilege fails the call", func(t *testing.T) {
t.Parallel()
fx := twoNodeFixture(t)
fx.putSeed()
fx.fake.failWith("DeleteVolume", loadAPIError(t, "err_forbidden_volume_delete"))
if msg := fx.deleteFails(provider.ErrPermanent); !strings.Contains(msg, "Datastore.Allocate") {
t.Errorf("error = %s, want the missing privilege named", msg)
}
if n := fx.fake.callCount("DeleteVolume"); n != 1 {
t.Errorf("DeleteVolume calls = %d, want 1: every node would refuse the same", n)
}
})
t.Run("ended ctx fails the call", func(t *testing.T) {
t.Parallel()
fx := twoNodeFixture(t)
ctx, cancel := context.WithCancel(fx.ctx)
defer cancel()
fx.fake.hookOn("DeleteVolume", func() error {
cancel()
return context.Canceled
})
err := fx.p.Delete(ctx, fx.id())
assertProviderError(t, err, "delete", provider.ErrTransient)
if n := fx.fake.callCount("DeleteVolume"); n != 1 {
t.Errorf("DeleteVolume calls = %d, want 1: the ended ctx stops the loop", n)
}
})
}
func TestDelete_destroyTaskFailureIsTransient(t *testing.T) {
t.Parallel()
fx := newFixture(t, 1)
fx.addOurVM(testVMID, testName, "stopped")
fx.putSeed()
fx.fake.destroyFailsWith = "fake: unable to remove disk"
if msg := fx.deleteFails(provider.ErrTransient); !strings.Contains(msg, "unable to remove disk") {
t.Errorf("error = %s, want the task's exitstatus", msg)
}
if _, ok := fx.fake.vmSnapshot(testVMID); !ok {
t.Error("VM gone after a failed destroy task")
}
if fx.seedPresent() {
t.Error("seed still exists: it goes before the destroy")
}
if lines := fx.logs.matching(`"msg"="proxmox task failed"`); len(lines) != 1 {
t.Errorf("task failed lines = %v, want one", lines)
}
}
// TestDelete_goneMidwayIsSuccess: the VM vanishing at any step counts as
// done, and F2's does-not-exist answer is what each step then sees.
func TestDelete_goneMidwayIsSuccess(t *testing.T) {
t.Parallel()
tests := []struct {
name string
status string
method string
wantDestroy int
wantNodes []string
}{
{name: "before the config read", status: "stopped", method: "VMConfig", wantNodes: []string{fakeNode, "pv1-2"}},
{name: "before the stop", status: "running", method: "Stop", wantDestroy: 1, wantNodes: []string{fakeNode}},
{name: "before the destroy", status: "stopped", method: "Destroy", wantDestroy: 1, wantNodes: []string{fakeNode}},
}
for _, tc := range tests {
t.Run(tc.name, func(t *testing.T) {
t.Parallel()
fx := twoNodeFixture(t)
fx.addOurVM(testVMID, testName, tc.status)
mutateOn(fx.fake, tc.method, func() { delete(fx.fake.vms, testVMID) })
fx.delete()
if n := fx.fake.callCount("Destroy"); n != tc.wantDestroy {
t.Errorf("Destroy calls = %d, want %d", n, tc.wantDestroy)
}
if got := fx.seedDeleteNodes(); !slices.Equal(got, tc.wantNodes) {
t.Errorf("seed deleted on %v, want %v", got, tc.wantNodes)
}
})
}
}
func TestDelete_resourcesFailureIsTransient(t *testing.T) {
t.Parallel()
fx := newFixture(t, 1)
fx.addOurVM(testVMID, testName, "running")
fx.fake.failWith("ClusterResources", fakeErr(595, "Connection refused"))
fx.deleteFails(provider.ErrTransient)
if n := fx.fake.callCount("DeleteVolume"); n != 0 {
t.Errorf("DeleteVolume calls = %d, want 0 when the VM's state is unknown", n)
}
}
// TestDelete_usesPurgeAndDestroyUnreferencedDisks runs Delete over the real
// client against Phase 0's destroy captures: the DELETE carries both flags
// and the seed delete sends the volid as one escaped segment.
func TestDelete_usesPurgeAndDestroyUnreferencedDisks(t *testing.T) {
t.Parallel()
const (
vmid = 199999
name = "proxy-reconaaaaaaaaaaa"
)
var (
mu sync.Mutex
seen []string
)
cfx := newClientFixture(t, func(w http.ResponseWriter, r *http.Request) {
path := strings.TrimPrefix(r.URL.EscapedPath(), "/api2/json/")
mu.Lock()
seen = append(seen, r.Method+" "+path+"?"+r.URL.RawQuery)
mu.Unlock()
capture := ""
switch {
case r.Method == http.MethodGet && path == "cluster/resources":
capture = "cluster_resources_during_destroy"
case r.Method == http.MethodGet && path == "nodes/pv1-1/qemu/199999/config":
capture = "config_during_destroy"
case r.Method == http.MethodDelete && path == "nodes/pv1-1/qemu/199999":
capture = "destroy_response"
case r.Method == http.MethodDelete && path == "nodes/pv1-1/storage/proxy-seeds/content/proxy-seeds:iso%2F"+name+"-seed.iso":
capture = "content_delete_response"
case r.Method == http.MethodGet && strings.Contains(path, ":imgdel:"):
capture = "task_status_volume_missing"
case r.Method == http.MethodGet && strings.Contains(path, ":qmdestroy:"):
capture = "task_status_stopped_ok"
default:
t.Errorf("unexpected request %s %s", r.Method, path)
http.Error(w, "unexpected", http.StatusNotImplemented)
return
}
serveJSON(w, readCapture(t, capture).Body)
})
cfg := testConfig()
cfg.URL, cfg.TokenFile, cfg.CAFile = cfx.cfg.URL, cfx.cfg.TokenFile, cfx.cfg.CAFile
p := newProvider(testProvider, cfg, cfx.api(t))
p.budgets = testBudgets()
if err := p.Delete(context.Background(), formatProviderID(vmid, name)); err != nil {
t.Fatalf("Delete error = %v", err)
}
mu.Lock()
defer mu.Unlock()
if !slices.Contains(seen, "DELETE nodes/pv1-1/qemu/199999?destroy-unreferenced-disks=1&purge=1") {
t.Errorf("requests = %v, want the destroy with purge=1 and destroy-unreferenced-disks=1", seen)
}
if !slices.ContainsFunc(seen, func(s string) bool { return strings.HasPrefix(s, "DELETE nodes/pv1-1/storage/proxy-seeds/content/") }) {
t.Errorf("requests = %v, want the seed delete", seen)
}
if slices.ContainsFunc(seen, func(s string) bool { return strings.Contains(s, "status/stop") }) {
t.Errorf("requests = %v, want no stop for the captured stopped VM", seen)
}
}
func TestDelete_concurrentSameNameSafe(t *testing.T) {
t.Parallel()
fx := newFixture(t, 1)
fx.addOurVM(testVMID, testName, "running")
release := fx.fake.blockOn("Stop")
errs := make([]error, 2)
first := make(chan struct{})
var wg sync.WaitGroup
wg.Go(func() {
defer close(first)
errs[0] = fx.p.Delete(fx.ctx, fx.id())
})
for fx.fake.callCount("Stop") == 0 {
select {
case <-first:
t.Fatalf("first Delete returned %v without stopping the VM", errs[0])
case <-time.After(time.Millisecond):
}
}
wg.Go(func() { errs[1] = fx.p.Delete(fx.ctx, fx.id()) })
close(release)
wg.Wait()
for i, err := range errs {
if err != nil {
t.Errorf("Delete %d error = %v", i, err)
}
}
for method, want := range map[string]int{"Stop": 1, "Destroy": 1} {
if got := fx.fake.callCount(method); got != want {
t.Errorf("%s calls = %d, want %d", method, got, want)
}
}
}
// TestDelete_waitsOnTheNameLock: Delete takes the per-name lock before any
// call, so it cannot interleave with a Create or another Delete of that
// name (spec entry 19).
func TestDelete_waitsOnTheNameLock(t *testing.T) {
t.Parallel()
fx := newFixture(t, 1)
fx.addOurVM(testVMID, testName, "running")
release, err := fx.p.lockName(fx.ctx, testName)
if err != nil {
t.Fatalf("lockName error = %v", err)
}
defer release()
ctx, cancel := context.WithTimeout(fx.ctx, 20*time.Millisecond)
defer cancel()
within(t, 5*time.Second, func() { err = fx.p.Delete(ctx, fx.id()) })
assertProviderError(t, err, "delete", provider.ErrTransient)
if !errors.Is(err, context.DeadlineExceeded) {
t.Errorf("error = %v, want it to wrap DeadlineExceeded", err)
}
if calls := fx.fake.callLog(); len(calls) != 0 {
t.Errorf("calls = %v, want none while the name is locked", calls)
}
}
func TestDelete_cancelledMidWaitReturnsPromptly(t *testing.T) {
t.Parallel()
fx := newFixture(t, 1)
fx.p.budgets = defaultBudgets()
fx.p.budgets.poll = time.Millisecond
fx.fake.taskPolls = 1 << 30
fx.addOurVM(testVMID, testName, "running")
ctx, cancel := context.WithCancel(fx.ctx)
defer cancel()
fx.fake.hookOn("TaskStatus", func() error {
cancel()
return nil
})
start := time.Now()
var err error
within(t, 10*time.Second, func() { err = fx.p.Delete(ctx, fx.id()) })
if elapsed := time.Since(start); elapsed > 5*time.Second {
t.Errorf("Delete took %v after cancellation", elapsed)
}
assertProviderError(t, err, "delete", provider.ErrTransient)
if !errors.Is(err, context.Canceled) {
t.Errorf("error = %v, want it to wrap context.Canceled", err)
}
}
func TestDelete_outcomeLine(t *testing.T) {
t.Parallel()
tests := []struct {
name string
setup func(fx *fixture)
want []string
never string
}{
{
name: "destroyed",
setup: func(fx *fixture) { fx.addOurVM(testVMID, testName, "stopped") },
want: []string{`"node"="` + fakeNode + `"`, `"destroyed"=true`},
},
{name: "already gone", setup: func(*fixture) {}, want: []string{`"destroyed"=false`}, never: `"node"=`},
}
for _, tc := range tests {
t.Run(tc.name, func(t *testing.T) {
t.Parallel()
fx := newFixture(t, 1)
tc.setup(fx)
fx.delete()
lines := fx.logs.matching(`"msg"="proxmox vm deleted"`)
if len(lines) != 1 {
t.Fatalf("outcome lines = %d, want 1:\n%s", len(lines), strings.Join(fx.logs.all(), "\n"))
}
for _, want := range append(tc.want, `"vmid"=150042`, `"name"="`+testName+`"`, `"provider"="`+testProvider+`"`) {
if !strings.Contains(lines[0], want) {
t.Errorf("outcome line lacks %s: %s", want, lines[0])
}
}
if tc.never != "" && strings.Contains(lines[0], tc.never) {
t.Errorf("outcome line has %s: %s", tc.never, lines[0])
}
})
}
}
func TestDelete_noOpLinesAreSilentAtV0(t *testing.T) {
t.Parallel()
tests := map[string]func(fx *fixture){
"absent": func(*fixture) {},
"foreign": func(fx *fixture) { fx.addOurVM(testVMID, "proxy-someoneelse00000", "running") },
"unmarked": func(fx *fixture) { fx.fake.addVM(testVMID, testName, "running") },
"ours": func(fx *fixture) { fx.addOurVM(testVMID, testName, "running") },
}
for name, setup := range tests {
t.Run(name, func(t *testing.T) {
t.Parallel()
fx := newFixture(t, 0)
fx.p.api = observe(fx.fake, testProvider)
setup(fx)
fx.delete()
if lines := fx.logs.all(); len(lines) != 0 {
t.Errorf("V(0) lines = %v, want none", lines)
}
})
}
}

Some files were not shown because too many files have changed in this diff Show More