Author SHA1 Message Date
kacerrandClaude Opus 5.5 3d72507894 proxmox: review fixes, real-host validation record, changelog
Phase 10. Two review passes (go-operator-reviewer, /code-review high)
over the branch, recorded in docs/reviews/2026-09-23-proxmox-provider-review/:
9 findings fixed, 3 accepted as documented limits, 7 rejected with a
reason.

- Delete removes the seed between the stop and the destroy, so a failed
  seed delete leaves the VM for the next Delete instead of leaking the
  ISO; a Permanent refusal on the every-node path fails the call.
- Create skips a probe VMID whose listed name is someone else's without
  waiting on its lock.
- The three answers Phase 0 never captured are fixtures now: a missing
  storage is a 400 (the Checker reports StorageNotFound), a hibernated
  VM is destroyable (Delete destroys ours), a paused VM stops.
- TestProxmox_lifecycle and TestProxmoxCheck_realHost passed on pv1-1
  (PVE 9.2.20); the docs record the run and drop "not yet run".

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-09-24 10:28:28 +00:00
kacerrandClaude Opus 5.5 7f2073f94b proxmox: deployment files, sample, user guide and doc rows
The optional proxmox-token Secret (token, ca.pem) mounted at
/etc/proxy-operator/proxmox, make proxmox-token-secret with content
checks that never print the token, a commented proxmox-home entry in
both providers configs, and config/samples/proxy_proxmox.yaml.
TestLoadConfig_proxmoxSampleEntry loads the commented entry from both
files, and a second test ties the sample Proxy's placement to it.

docs/proxmox-provider.md is the user guide: prerequisites and privileges,
cluster setup, config reference, placement, what lands on Proxmox,
logging, tracing, timeouts, health check, error classes, known gaps and
testing. providers-existing.md gains the proxmox column and section;
architecture.md, the README, providers.md and CLAUDE.md gain their rows.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-09-23 21:14:18 +00:00
kacerrandClaude Opus 5.5 6e9ef26dcb proxmox: contract and observability tests, integration test, testing docs
The contract sweep in provider_test.go: every method and Check returns
within a second of a cancel, every wrapErr path carries the configured
name, op and id, the V0/V1/V2 logging tiers and the no-op lines, and a
Get over the real client whose HTTP spans are children of provider.get.
proxmox_integration_test.go (-tags=integration) holds TestProxmox_lifecycle
and TestProxmoxCheck_realHost; both compile and skip without
PROXMOX_TEST_*, and have not run against a host yet. make
test-integration-proxmox runs them. docs/testing.md gains the proxmox,
nocloud and keyedlock rows, the Live Proxmox gaps and the integration
section; spec entry 21's section 11 list is ticked.

Waits that could hang the package under a mutation are bounded: the
concurrent-Create loop and eight direct blocking calls in older tests.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-09-23 20:43:00 +00:00
kacerrandClaude Opus 5.5 2f89c1fffb proxmox: Delete with seed cleanup, health Checker, CRD type comment
Delete stops and destroys a VM of ours under the per-name lock, then
deletes its seed on the VM's node, or on every node when no VM of ours
was left. A hibernated VM of ours is Permanent, as in Create. The
Checker verifies each node concurrently (version, node status, ISO
storage, templates) and adds CertificateInvalid, CredentialsUnavailable,
StorageNotFound and TemplateNotFound to the shared reasons. The
ProviderHealth type comment gains proxmox.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-09-23 20:14:14 +00:00
kacerrandClaude Opus 5.5 34f132d421 proxmox: Create by cloning a template, seed upload, config, start
Create validates placement, then under the per-name lock adopts the VM a
previous attempt left (by name, or at a taken VMID after the ownership
check of spec entry 2) or full-clones the template onto the UID's
deterministic VMID with the mark in the clone call. It then uploads the
seed over any existing file, writes cores, memory, agent, tag, ide2 and
net0 when configured, grows the boot disk when diskGiB is larger, starts,
and waits for running/running.

waitUnlocked no longer waits on a suspended lock: a hibernated VM is
judged at once, so ours is Permanent and a foreign one is probed past.
The template is checked only on the allocation path, and its shape
(net0, boot disk size) is refused before any clone exists. Spec entries
2, 4 and 10 updated to match.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-09-23 15:32:31 +00:00
kacerrandClaude Opus 5.5 803716459a proxmox: Get, ListByTag and task polling
Get follows spec entry 9's check order (lock and unknown status before
the name, suspended is Stopped after the name and mark checks), reads
status/current so paused is Stopped, and maps the three agent messages
of entry 7. ListByTag reads one config per candidate and fails whole
only on the resources call or an ended ctx. waitTask polls the UPID's
node every budgets.poll, logs failed tasks at V(1) and counts
"WARNINGS: <n>" as success; waitUnlocked serves Create's ownership check.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-09-23 15:11:20 +00:00
kacerrandClaude Opus 5.5 e05d994974 proxmox: thin HTTP client, seam, log decorator and in-memory fake
httpAPI implements the sixteen seam methods over net/http: the token is
read per request, non-2xx answers become apiError from the body message or
the reason phrase, echoed secrets are redacted, the CA loads lazily, HTTP/2
is off, and uploads stream with an exact Content-Length. The observed
decorator logs one V(1) line per call and starts no span; New wires both.
fakeAPI simulates clone locks, tasks and ISOs with the captured error texts.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-09-23 14:47:50 +00:00
kacerrandClaude Opus 5.5 0a6c8624f3 provider: extract nocloud seed builder and keyedlock from kvm2
kvm2's NoCloud seed builder moves to internal/provider/nocloud (exported,
with an ErrEmptyName sentinel instead of permanentf) and its keyed locks
to internal/provider/keyedlock, tests moved under their existing names.
kvm2 behaviour is unchanged. proxmox gains the per-name lock and the pure
seed helpers (seedISOName, seedVolID, buildSeed); unlike kvm2 (review
finding S5.1), a seed staging failure there is Transient, not Permanent.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-09-23 14:01:27 +00:00
kacerrandClaude Opus 5.5 6e996ae2c7 proxmox: config block, IDs, marks, state mapping, classify, registration
Phase 2 of the proxmox provider plan. Adds the proxmox: config block with
its validation, registers the type in the config allow-list and the
constructor map, and starts internal/provider/proxmox: the api seam and
its data types, providerIDs (vms/<vmid>/<name>) with the UID-derived VMID,
the description mark, state mapping, and error classification.

Tests read the responses captured from pv1-1 in Phase 0 rather than
repeating Proxmox's wording, and every matcher is run against every
captured error so none fires on the wrong message. The contract methods
are Transient stubs until Phases 5 to 7.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-09-23 09:42:46 +00:00
kacerrandClaude Opus 5.5 4e974f30f6 docs: proxmox provider spec
Phase 1 of the proxmox provider plan: docs/proxmox-provider-spec.md
answers all 24 entries of the provider spec template against the
contract, citing the Phase 0 facts from pv1-1 as evidence.

It folds in the ten corrections recon found. The two that matter most:
Create inspects a taken VMID, waiting out a clone lock, before probing
past it, so a crash-retry adopts its own half-cloned VM instead of cloning
a duplicate; and Get checks the lock before the name, because a VM being
cloned has no name yet.

The plan now opens with a note that the spec wins where they disagree.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-09-23 09:18:29 +00:00
kacerrandClaude Opus 5.5 ff94fe70bb proxmox: captured API fixtures and host facts from pv1-1 (PVE 9.2.20)
Phase 0 of the proxmox provider plan. 55 fixtures under
internal/provider/proxmox/testdata, each the status, reason phrase and
raw body of a real response from the user's host, and the execution log
answering the plan's facts F1 to F23.

Recon ran a clone of the Debian 13 template through the whole path the
provider will take, twice: untagged and on VLAN 4. On both, the NoCloud
seed on ide2 installed the guest agent and an address came back about 50
seconds after start. It also found two bugs in the plan: a VM being
cloned has no name in /cluster/resources, so name-first lookups would read
it as gone, and a crash-retry would clone a duplicate. Those and eight
smaller corrections are listed for the spec.

Decisions made with the user: seeds go to a dedicated proxy-seeds
storage where the token can delete them; proxies land on a VLAN set in
config (vlanTag); the API URL is https://192.168.0.8:8006, the address in
the node certificate.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-09-23 08:22:41 +00:00
kacerrandClaude Opus 5 2167819157 docs: retarget the proxmox plan at Proxmox VE 9.2
The host is 9.2.20, not 8.x. Three corrections that would have failed
at runtime:

- VM.Monitor was removed in PVE 9.0, so it is dropped from the role and
  the agent read rests on VM.GuestAgent.Audit alone.
- The GC-assist tag becomes proxy_operator: current tag validation takes
  only lowercase letters, digits and underscore, so proxy-operator would
  have been rejected on every Create. The description mark keeps its
  hyphenated keys; it is free text, not a tag.
- HTTP/2 is documented as absent from pveproxy on every release, so the
  client disabling it is a fact rather than a guess.

Two new Phase 0 facts: F22 for the 9.2 unified authentication-failure
message, which invalidates any 401 text remembered from 8.x, and F23 for
the node /tmp tmpfs move. F12 records that the upload endpoint takes
iso, vztmpl or import on 9.x and still refuses snippets, which is what
keeps the seed ISO route the only API-only path for user-data.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-09-18 15:49:14 +00:00
kacerrandClaude Fable 5.1 708989c40f docs: plan the proxmox provider
Eleven self-contained phases (host recon and fixtures, spec, config and
skeleton, nocloud/keyedlock extraction from kvm2, HTTP client and fake,
Get/List, Create, Delete/Checker, test sweep and integration test,
deployment and docs, review and MR), with the Proxmox host setup
checklist and the backend facts Phase 0 must verify on 192.168.2.8.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
2026-09-18 15:02:53 +00:00
kacerrandClaude Opus 5 8fa9e8aad9 Housekeeping: review docs into docs/, refreshed agent guide and settings
Four unrelated working-tree changes, none of them touching operator
behaviour:

- review/*.md -> docs/reviews/2026-09-17-kvm2-provider-review/, byte for
  byte, so the kvm2 review sits with the rest of the docs under a name
  that says which review it was.
- AGENTS.md: replaced the generic kubebuilder scaffold guide with what is
  actually true of this repo — the real component layout, the lint/test
  commands that work here, and the behavioural gotchas (replace-on-change
  hashing, the cloud-init Secret label, in-memory leases).
- .claude/settings.json: the allow list still carried /Users/jan.novak
  paths from the macOS checkout; re-pointed at /home/novakj and collapsed
  the one-off entries into prefixes.
- config/samples/proxy_kvm2_crawl_only_one.yaml: a one-VM cut of the crawl
  pool, the rest of the members left commented out.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-09-18 08:12:27 +00:00
kacerrandClaude Opus 5 1fa3e55b9c docs: start docs/operating.md with the libvirt-RPC log query
A place for helpers against a running operator, as opposed to the design
docs. First entry is the jq filter for timestamp + rpc: both libvirt
providers log the same "libvirt call" msg, so one select covers kvm and
kvm2, and a fromjson? variant survives a stream with non-JSON lines in it.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-09-18 07:53:12 +00:00
kacerrandClaude Opus 5 c3a581af95 samples: a kvm2 crawl pool, not just the single-proxy sample
proxy_kvm2.yaml shows one VM; nothing showed the shape a crawler actually
leases from. Add a five-Proxy pool on the kvm2-home entry: four share one
labelled cloud-init Secret, and the fifth carries its own inline block
because it runs squid on 8080 and spec.port has to match the http_port a
shared Secret would pin.

It stays out of the samples kustomization so `apply -k config/samples`
doesn't boot a pool along with the one-per-provider samples. All six
objects validate against the installed CRD under --dry-run=server.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-09-18 07:45:16 +00:00
kacerr 5928c3fe75 Merge pull request 'kvm2 provider: clean-room libvirt/KVM provider, code review, and the four high findings' (#8) from feat/kvm2-provider into main
Reviewed-on: #8
2026-09-17 23:12:46 +02:00
141 changed files with 16933 additions and 671 deletions
+174 -102
View File
@@ -1,113 +1,185 @@
{
"permissions": {
"allow": [
"Bash(go version *)",
"Bash(go mod *)",
"Read(//Users/jan.novak/.claude/skills/go-project-bootstrap/references/**)",
"Bash(ls -la /Users/jan.novak/.claude/skills/go-project-bootstrap/assets/ /Users/jan.novak/.claude/skills/go-project-bootstrap/assets/commands/ /Users/jan.novak/.claude/skills/go-project-bootstrap/assets/agents/ 2>&1)",
"Read(//Users/jan.novak/.claude/skills/go-project-bootstrap/assets/**)",
"Read(//Users/jan.novak/.claude/skills/go-project-bootstrap/assets/agents/**)",
"Read(//Users/jan.novak/.claude/skills/go-project-bootstrap/assets/commands/**)",
"Bash(mkdir -p /Users/jan.novak/srv/go/egress-proxies-operator/docs/plans && touch /Users/jan.novak/srv/go/egress-proxies-operator/docs/plans/.gitkeep && mkdir -p /Users/jan.novak/srv/go/egress-proxies-operator/.claude/commands /Users/jan.novak/srv/go/egress-proxies-operator/.claude/agents && ls -la /Users/jan.novak/srv/go/egress-proxies-operator)",
"Bash(git status)",
"Bash(git status:*)",
"Bash(git diff:*)",
"Bash(git log:*)",
"Bash(git branch:*)",
"Bash(git add:*)",
"Bash(git checkout:*)",
"Bash(git pull:*)",
"Bash(git fetch:*)",
"Bash(date:*)",
"Read(//tmp/**)",
"Read(//private/tmp/**)",
"Bash(go build:*)",
"Bash(go test:*)",
"Bash(go vet:*)",
"Bash(go mod:*)",
"Bash(go run:*)",
"Bash(go version:*)",
"Bash(go get:*)",
"Bash(golangci-lint run:*)",
"Bash(golangci-lint --version)",
"Bash(gofumpt:*)",
"Bash(tea pr create:*)",
"Bash(curl -sL https://go.dev/dl/?mode=json)",
"Bash(find / -iname \"docs-conventions.md\" -maxdepth 6)",
"Read(//Users/jan.novak/.claude/**)",
"Read(//Users/jan.novak/.claude/skills/**)",
"Bash(mkdir -p docs/plans && date \"+%Y-%m-%d-%H%M\")",
"Bash(go install *)",
"Bash(go env *)",
"Bash(git rm *)",
"Bash(mkdir -p /Users/jan.novak/srv/go/egress-proxies-operator/docs/plans-executions)",
"Bash(cd /Users/jan.novak/srv/go/egress-proxies-operator *)",
"Bash(echo \"build: $?\")",
"Bash(echo \"vet: $?\")",
"Bash(perl -i -pe 's{^\\\\t\\\\t\\\\t\\\\t\\\\t// TODO\\\\\\(user\\\\\\): Specify other spec details if needed\\\\.\\\\n}{\\\\t\\\\t\\\\t\\\\t\\\\t// A minimal, schema-valid spec so this placeholder test survives the\\\\n\\\\t\\\\t\\\\t\\\\t\\\\t// CEL/CRD validation added in Step 1. Rewritten wholesale in Step 4\\\\n\\\\t\\\\t\\\\t\\\\t\\\\t// alongside the real reconciler and envtest suite.\\\\n\\\\t\\\\t\\\\t\\\\t\\\\tSpec: crawlv1alpha1.ProxySpec{\\\\n\\\\t\\\\t\\\\t\\\\t\\\\t\\\\tMode: crawlv1alpha1.ModeExternal,\\\\n\\\\t\\\\t\\\\t\\\\t\\\\t\\\\tEndpoint: &crawlv1alpha1.EndpointSpec{Host: \"10.0.0.1\"},\\\\n\\\\t\\\\t\\\\t\\\\t\\\\t},\\\\n}' internal/controller/proxy_controller_test.go)",
"Bash(go tool *)",
"Bash(docker version *)",
"Bash(curl -sI --max-time 5 https://hub.docker.com)",
"Bash(kind get *)",
"Bash(curl -s --max-time 5 \"https://hub.docker.com/v2/repositories/ubuntu/squid/tags?page_size=10\")",
"Bash(python3 -c ' *)",
"Bash(curl -s --max-time 5 \"https://hub.docker.com/v2/repositories/ubuntu/squid/tags?page_size=100\")",
"Bash(go doc *)",
"Bash(go list *)",
"Bash(gofmt -w internal/provider/config.go)",
"Bash(gofmt -l .)",
"Bash(git restore *)",
"Bash(make manifests *)",
"Bash(make test *)",
"Bash(KUBEBUILDER_ASSETS=\"/Users/jan.novak/srv/go/egress-proxies-operator/bin/k8s/1.36.2-darwin-arm64\" go test -race ./internal/controller/)",
"Bash(grep -n 'func Channel' -A8 __CMDSUB_OUTPUT__/sigs.k8s.io/controller-runtime@v0.24.1/pkg/source/source.go)",
"Bash(grep -n 'type GenericEvent' __CMDSUB_OUTPUT__/sigs.k8s.io/controller-runtime@v0.24.1/pkg/event/event.go)",
"Bash(KUBEBUILDER_ASSETS=__TRACKED_VAR__/bin/k8s/1.36.2-darwin-arm64 go test -race ./...)",
"Bash(cat >> *)",
"Bash(make run-dev *)",
"Bash(kubectl get *)",
"Bash(kubectl delete *)",
"Bash(make docker-build *)",
"Bash(kind load *)",
"Bash(make deploy *)",
"Bash(kubectl -n egress-proxies-operator-system rollout status deploy/egress-proxies-operator-controller-manager --timeout=120s)",
"Bash(kubectl -n egress-proxies-operator-system rollout restart deploy/egress-proxies-operator-controller-manager)",
"Bash(git -C /Users/jan.novak/srv/go/egress-proxies-operator add docs/plans/2026-08-11-1742-gcp-provider-verbose-logging.md)",
"Bash(git -C /Users/jan.novak/srv/go/egress-proxies-operator commit -m 'Add plan: verbose V-level logging in the GCP provider *)",
"Bash(echo \"exit: $?\")",
"Bash(echo \"tests exit: $?\")",
"Bash(./bin/manager --version)",
"Bash(./bin/manager-stamped --version)",
"Bash(./bin/manager-pkg --version)",
"Bash(docker run *)",
"Bash(kubectl -n egress-proxies-operator-system get pods -o wide)",
"Bash(kubectl -n egress-proxies-operator-system get deploy egress-proxies-operator-controller-manager -o jsonpath='{.spec.template.spec.containers[0].args}')",
"Bash(kubectl -n egress-proxies-operator-system logs deploy/egress-proxies-operator-controller-manager)",
"Bash(python3 -c \"import json; d=json.load\\(open\\('docs/deploy/sa_key.json'\\)\\); print\\(d.get\\('type'\\), d.get\\('client_email'\\)\\)\")",
"Bash(tea pr *)",
"Bash(git worktree *)",
"Bash(python3 -c \"import yaml; yaml.safe_load\\(open\\('.gitea/workflows/build.yaml'\\)\\); print\\('YAML OK'\\)\")",
"Bash(ruby -ryaml -e \"YAML.load_file\\('.gitea/workflows/build.yaml'\\); puts 'YAML OK'\")",
"Bash(git -C /Users/jan.novak/srv/go/egress-proxies-operator status --short --branch)",
"Bash(git -C /Users/jan.novak/srv/go/egress-proxies-operator log --oneline -1)",
"Bash(git -C /Users/jan.novak/srv/go/egress-proxies-operator tag 0.01)",
"Bash(git -C /Users/jan.novak/srv/go/egress-proxies-operator push origin 0.01)",
"Bash(chmod +x docs/demo/run-demo.sh)",
"Bash(bash -n docs/demo/run-demo.sh)",
"Bash(git add *)",
"Bash(git commit -q -m 'Add plan: KVM/libvirt provider *)",
"Bash(go get *)",
"Bash(go build *)",
"Bash(go test *)",
"Bash(git commit *)"
"Bash(go vet *)",
"Bash(go run *)",
"Bash(go mod *)",
"Bash(go get *)",
"Bash(go install *)",
"Bash(go env *)",
"Bash(go tool *)",
"Bash(go doc *)",
"Bash(go list *)",
"Bash(go version *)",
"Bash(gofmt *)",
"Bash(gofumpt *)",
"Bash(golangci-lint run *)",
"Bash(golangci-lint --version)",
"Bash(./bin/manager* --version)",
"Bash(make -n *)",
"Bash(make manifests *)",
"Bash(make test *)",
"Bash(make test-integration-kvm *)",
"Bash(make run-dev *)",
"Bash(make docker-build *)",
"Bash(make deploy *)",
"Bash(make kvm-ssh-secret *)",
"Bash(git status *)",
"Bash(git diff *)",
"Bash(git log *)",
"Bash(git branch)",
"Bash(git branch -a)",
"Bash(git branch --show-current)",
"Bash(git add *)",
"Bash(git commit *)",
"Bash(git fetch *)",
"Bash(git pull *)",
"Bash(git worktree list *)",
"Bash(git config --get *)",
"Bash(git config --list *)",
"Bash(git merge-tree *)",
"Bash(git ls-remote *)",
"Bash(git -c credential.helper='!f\\(\\) { echo username=kacerr; echo \"password=$\\(tr -d \"[:space:]\" < ~/.config/gitea-token\\)\"; }; f' push origin feat/kvm-provider)",
"Bash(git -c credential.helper='!f\\(\\) { echo username=kacerr; echo \"password=$\\(tr -d \"[:space:]\" < ~/.config/gitea-token\\)\"; }; f' push -q origin feat/kvm-provider)",
"Bash(git -c credential.helper='!f\\(\\) { echo username=kacerr; echo \"password=$\\(tr -d \"[:space:]\" < ~/.config/gitea-token\\)\"; }; f' push -u origin feat/kvm-provider)",
"Bash(tea pr create *)",
"Bash(tea pr list *)",
"Bash(curl -sL https://go.dev/dl/?mode=json)",
"Bash(curl -s --max-time 10 https://gitea.home.hrajfrisbee.cz/api/v1/repos/kacerr/egress-proxies-operator/*)",
"Bash(curl -s --max-time 10 'https://gitea.home.hrajfrisbee.cz/api/v1/repos/kacerr/egress-proxies-operator/*')",
"Bash(curl -sI --max-time 10 https://gitea.home.hrajfrisbee.cz/v2/kacerr/egress-proxies-operator/*)",
"Bash(curl -s -H \"Authorization: token $\\(tr -d '[:space:]' < ~/.config/gitea-token\\)\" https://gitea.home.hrajfrisbee.cz/api/v1/repos/kacerr/egress-proxies-operator/pulls/5)",
"Bash(kubectl get *)",
"Bash(kubectl wait *)",
"Bash(kubectl kustomize *)",
"Bash(kubectl apply *)",
"Bash(kubectl run *)",
"Bash(kubectl config current-context)",
"Bash(kubectl config get-contexts *)",
"Bash(kubectl config view *)",
"Bash(kubectl -n egress-proxies-operator-system get *)",
"Bash(kubectl -n egress-proxies-operator-system logs *)",
"Bash(kubectl -n egress-proxies-operator-system rollout status *)",
"Bash(kubectl -n egress-proxies-operator-system rollout restart *)",
"Bash(kind get *)",
"Bash(kind load *)",
"Bash(kind version *)",
"Bash(docker version *)",
"Bash(docker info *)",
"Bash(docker image ls *)",
"Bash(docker image inspect *)",
"Bash(docker manifest inspect *)",
"Bash(docker history *)",
"Bash(docker save *)",
"Bash(docker run *)",
"Bash(docker exec *)",
"Bash(virsh -c qemu:///system list *)",
"Bash(virsh -c qemu:///system net-list *)",
"Bash(virsh -c qemu:///system net-dumpxml *)",
"Bash(virsh -c qemu:///system pool-list *)",
"Bash(virsh -c qemu:///system pool-dumpxml *)",
"Bash(virsh -c qemu:///system vol-list *)",
"Bash(ssh -i ~/.ssh/proxy-operator_ed25519 * novakj@beelink.tail8c69c.ts.net *)",
"Bash(ip -br *)",
"Bash(getent hosts *)",
"Bash(tailscale status *)",
"Bash(python3 *)",
"Bash(grep *)",
"Bash(echo \"*$?\")",
"Bash(date *)",
"Bash(env)",
"Bash(command -v *)",
"Bash(cat >> *)",
"Bash(tar -tf *)",
"Bash(bash -n *)",
"Bash(mkdir -p docs/*)",
"Read(//home/novakj/**)",
"Read(//Users/jan.novak/.claude/**)",
"Read(//tmp/**)",
"Read(//private/tmp/**)",
"Read(//etc/**)",
"Read(//dev/**)",
"Read(//run/libvirt/**)",
"WebSearch",
"WebFetch(domain:gitlab.com)",
"WebFetch(domain:bugs.debian.org)",
"WebFetch(domain:tracker.debian.org)",
"WebFetch(domain:cloud.debian.org)",
"WebFetch(domain:alioth-lists.debian.net)",
"WebFetch(domain:www.mail-archive.com)",
"WebFetch(domain:forum.proxmox.com)",
"Bash(curl -s --max-time 10 'https://gitea.home.hrajfrisbee.cz/api/v1/repos/kacerr/egress-proxies-operator/pulls?state=open')",
"Bash(python3 -c 'import json,sys; [print\\(p[\"number\"], p[\"head\"][\"ref\"], \"->\", p[\"base\"][\"ref\"], \"|\", p[\"title\"]\\) for p in json.load\\(sys.stdin\\)]')",
"Bash(curl -s --max-time 10 'https://gitea.home.hrajfrisbee.cz/api/v1/repos/kacerr/egress-proxies-operator/pulls/5')",
"Bash(python3 -c 'import json,sys; p=json.load\\(sys.stdin\\); print\\(p[\"state\"], \"merged=\", p[\"merged\"], p.get\\(\"merge_commit_sha\"\\)\\)')",
"Read(//home/novakj/srv/egress-proxies-operator-simplify/origin/**)",
"Bash(echo \"exit=$?\")",
"Read(//home/novakj/srv/egress-proxies-operator-simplify/**)",
"Bash(git -C /home/novakj/srv/egress-proxies-operator-simplify diff 82eeeef -- internal/controller/proxy_controller.go internal/provider/kvm/errors.go)",
"Bash(git switch *)",
"Bash(git -C /home/novakj/srv/egress-proxies-operator-simplify add -A)",
"Bash(git -C /home/novakj/srv/egress-proxies-operator-simplify commit -q -m 'WIP: uncommitted simplify-pass edits preserved before worktree removal *)",
"Bash(git -C /home/novakj/srv/egress-proxies-operator-simplify log --oneline -1)",
"Bash(git worktree *)",
"Bash(git branch *)",
"Bash(git stash *)",
"Bash(make generate *)",
"Bash(bin/kustomize build *)",
"Bash(grep -Ev \"^\\\\s*$\")",
"Bash(make lint *)",
"Bash(command -v kind docker)",
"Bash(bin/golangci-lint version *)",
"Bash(make test-e2e *)",
"Bash(echo \"make test-e2e exit=$?\")",
"Bash(ls \"$\\(go env GOMODCACHE\\)/golang.org/\")",
"Bash(ls \"$\\(go env GOMODCACHE\\)/golang.org/toolchain@\"* -d)",
"Bash(ps -eo pid,ppid,etime,args)",
"Bash(kubectl config *)",
"Bash(git commit -q -m 'Plan the provider blueprint document *)",
"Bash(grep -nE '^#{1,3} ' docs/architecture.md; echo ---; sed -n '/Implemented providers/,/^## /p' README.md | head -30; echo ---; sed -n 140,180p config/manager/manager.yaml; echo ---; cat Dockerfile; echo ---; ls -a | grep -i golangci; echo ---; grep -nE 'Placement|placement|Region|Zone|MachineType|Image ' api/v1alpha1/proxy_types.go | head -40; echo ---; grep -nE 'Type|type' api/v1alpha1/providerhealth_types.go | head -20; echo ---; grep -n -i 'placement' docs/api.md | head; echo ---; grep -nE 'WithTimeout|Deadline|timeout' internal/controller/proxy_controller.go internal/gc/gc.go internal/providerhealth/monitor.go | head -20; echo ---; cat .gitea/workflows/build.yaml | sed -n 1,60p)",
"WebFetch(domain:pve.proxmox.com)",
"Bash(git -c credential.helper='!f\\(\\) { echo username=kacerr; echo \"password=$\\(tr -d \"[:space:]\" < ~/.config/gitea-token\\)\"; }; f' push -u origin feat/provider-blueprint)",
"Bash(curl -s -H \"Authorization: token $\\(tr -d '[:space:]' < ~/.config/gitea-token\\)\" \"https://gitea.home.hrajfrisbee.cz/api/v1/repos/kacerr/egress-proxies-operator/pulls?state=open\")",
"Bash(curl -s -H \"Authorization: token $\\(tr -d '[:space:]' < ~/.config/gitea-token\\)\" \"https://gitea.home.hrajfrisbee.cz/api/v1/repos/kacerr/egress-proxies-operator/pulls/7\")",
"Bash(curl -s -X PATCH -H \"Authorization: token $\\(tr -d '[:space:]' < ~/.config/gitea-token\\)\" -H 'Content-Type: application/json' -d @/tmp/claude-1000/-home-novakj-srv-egress-proxies-operator/7662ab9d-4540-4676-a943-0874ac6b081d/scratchpad/pr7.json \"https://gitea.home.hrajfrisbee.cz/api/v1/repos/kacerr/egress-proxies-operator/pulls/7\")",
"Bash(bin/golangci-lint run *)",
"Bash(awk '/^## The image cache/,/^## [^T]/' docs/kvm-provider.md)",
"Bash(awk '/^#+ .*Step 11/,0' docs/plans/2026-09-16-1333-kvm2-provider.md)",
"Bash(sort -t: -k2 -u)",
"Bash(awk '/kvm2/&&/^#/{f=1} f' docs/testing.md)",
"Bash(awk -F'\"' '{print $2}')",
"Bash(sed -n '/targets:/,/type: object/p' config/crd/bases/*providerhealths.yaml | sed -n 1,45p)",
"Bash(ssh -i ~/.ssh/proxy-operator_ed25519 -o IdentitiesOnly=yes -o BatchMode=yes -o ConnectTimeout=10 -o UserKnownHostsFile=~/.ssh/kvm_known_hosts -o StrictHostKeyChecking=yes novakj@beelink 'hostname; id -un; virsh -c qemu:///system version; virsh -c qemu:///system pool-list --all; virsh -c qemu:///system list --all')",
"Bash(awk '{print $1, $2}' ~/.ssh/kvm_known_hosts)",
"Bash(ssh-keygen -F beelink -f ~/.ssh/kvm_known_hosts)",
"Bash(ssh -i ~/.ssh/proxy-operator_ed25519 -o IdentitiesOnly=yes -o BatchMode=yes -o ConnectTimeout=10 -o UserKnownHostsFile=~/.ssh/kvm_known_hosts -o StrictHostKeyChecking=yes novakj@beelink.tail8c69c.ts.net 'hostname; id -un; id -Gn; virsh -c qemu:///system version; virsh -c qemu:///system pool-list --all --details; virsh -c qemu:///system list --all; command -v qemu-img')",
"Bash(qemu-img --version)",
"Bash(./probe)",
"Bash(./probe -proxyrcv 8192)",
"Bash(./probe -small)",
"Bash(sysctl net.ipv4.tcp_wmem net.ipv4.tcp_retries2)",
"Bash(GOFLAGS=-mod=mod GOPROXY=off go build -o probe .)",
"Bash(./probe -clientsnd 131072 -proxyrcv 8192)",
"Bash(truncate -s 64M gpt.raw)",
"Bash(sfdisk -l gptshort.raw)",
"Bash(sgdisk -v gptshort.raw)",
"Bash(mkdir -p /tmp/claude-1000/-home-novakj-srv-egress-proxies-operator/8db56daf-938c-48b8-8a17-084f3e1dea44/scratchpad)",
"Bash(curl -sS -m 30 -o storage_driver.c https://raw.githubusercontent.com/libvirt/libvirt/v11.6.0/src/storage/storage_driver.c)",
"Bash(GOFLAGS=-mod=mod GOPROXY=off go vet ./...)",
"Bash(GOFLAGS=-mod=mod GOPROXY=off go test -count=1 -run TestLeak -v . -args -n 1000 -transport unix)",
"Bash(curl -sS -m 30 -o storage_util.c https://raw.githubusercontent.com/libvirt/libvirt/v11.6.0/src/storage/storage_util.c)",
"Bash(awk '/^virStorageBackendVolUploadLocal/,/^}/' storage_util.c)",
"Bash(export GOFLAGS=-mod=mod GOPROXY=off)",
"Bash(awk '/WARNING: DATA RACE/{c++} c==1' race_ssh.log)",
"Bash(kubectl version *)",
"Bash(timeout 5 kubectl cluster-info)"
],
"additionalDirectories": [
"/Users/jan.novak/srv/go/egress-proxies-operator/.claude",
"/Users/jan.novak/srv/go/egress-proxies-operator/docs/plans",
"/Users/jan.novak/srv/go/egress-proxies-operator/docs",
"/Users/jan.novak/srv/go/egress-proxies-operator/docs/prompts",
"/tmp",
"/Users/jan.novak/srv/go/egress-proxies-operator/docs/plans-executions"
"/home/novakj/srv/egress-proxies-operator",
"/home/novakj/srv/egress-proxies-operator/config/crd/bases"
]
}
}
+39 -317
View File
@@ -1,320 +1,42 @@
# egress-proxies-operator - AI Agent Guide
# egress-proxies-operator
## Project Structure
## Sources and architecture
**Single-group layout (default):**
- Trust `go.mod`, `Makefile`, and current code over version tables or bootstrap prose. `CLAUDE.md`'s opening architecture/commands sections are stale; this is a working operator, not an empty scaffold.
- `cmd/main.go` assembles the provider registry, reconciler, proxy health engine, in-memory lease store, discovery HTTP API, orphan GC, and provider-health monitor on one controller-runtime manager.
- `api/v1alpha1` contains namespaced `Proxy` and cluster-scoped `ProviderHealth` resources (`crawl.example.com`). Validation/defaulting uses CRD markers/CEL; there are no admission webhooks.
- `internal/provider` defines the backend contract independently of CRD types. Constructors are registered in `cmd/main.go`; `spec.provider` selects a configured **instance name**, not a provider type. `kvm` and `kvm2` are separate implementations and do not adopt each other's VMs.
- Before changing a provider, read `docs/provider-contract.md`; use `docs/provider-spec-template.md` for a new one. Callers rely on self-contained provider IDs, truthful `ErrNotFound`, idempotent create/delete, and persisted UID/creation time for orphan GC. See `docs/providers-existing.md` for implementation-specific exceptions.
- `docs/architecture.md` explains component ownership and state transitions; `docs/api.md` documents discovery/leases; `docs/testing.md` covers test boundaries and real-infrastructure setup.
## Commands and verification
- Go version comes from `go.mod` (currently 1.26.0). Make installs development tools into `bin/`.
- After Go changes: `make lint-fix`, then `make test`. The latter regenerates manifests/DeepCopy, formats, vets, sets up envtest, then runs non-e2e tests with `-race` and writes `cover.out`.
- Fast checks: `go test -short -race ./...` skips envtest. Focused package/test: `go test -race ./internal/lease/` or `go test -race -run TestAcquire ./internal/lease/`.
- Envtest needs local API-server/etcd binaries; use `make setup-envtest` before targeted runs. For the controller suite with the current Kubernetes minor:
```sh
KUBEBUILDER_ASSETS="$(./bin/setup-envtest use 1.36 --bin-dir "$PWD/bin" -p path)" go test -race ./internal/controller/ -run TestControllers -ginkgo.focus='<spec regex>'
```
cmd/main.go Manager entry (registers controllers/webhooks)
api/<version>/*_types.go CRD schemas (+kubebuilder markers)
api/<version>/zz_generated.* Auto-generated (DO NOT EDIT)
internal/controller/* Reconciliation logic
internal/webhook/* Validation/defaulting (if present)
config/crd/bases/* Generated CRDs (DO NOT EDIT)
config/rbac/role.yaml Generated RBAC (DO NOT EDIT)
config/samples/* Example CRs (edit these)
Makefile Build/test/deploy commands
PROJECT Kubebuilder metadata Auto-generated (DO NOT EDIT)
```
**Multi-group layout** (for projects with multiple API groups):
```
api/<group>/<version>/*_types.go CRD schemas by group
internal/controller/<group>/* Controllers by group
internal/webhook/<group>/<version>/* Webhooks by group and version (if present)
```
Multi-group layout organizes APIs by group name (e.g., `batch`, `apps`). Check the `PROJECT` file for `multigroup: true`.
**To convert to multi-group layout:**
1. Run: `kubebuilder edit --multigroup=true`
2. Move APIs: `mkdir -p api/<group> && mv api/<version> api/<group>/`
3. Move controllers: `mkdir -p internal/controller/<group> && mv internal/controller/*.go internal/controller/<group>/`
4. Move webhooks (if present): `mkdir -p internal/webhook/<group> && mv internal/webhook/<version> internal/webhook/<group>/`
5. Update import paths in all files
6. Fix `path` in `PROJECT` file for each resource
7. Update test suite CRD paths (add one more `..` to relative paths)
## Critical Rules
### Never Edit These (Auto-Generated)
- `config/crd/bases/*.yaml` - from `make manifests`
- `config/rbac/role.yaml` - from `make manifests`
- `config/webhook/manifests.yaml` - from `make manifests`
- `**/zz_generated.*.go` - from `make generate`
- `PROJECT` - from `kubebuilder [OPTIONS]`
### Never Remove Scaffold Markers
Do NOT delete `// +kubebuilder:scaffold:*` comments. CLI injects code at these markers.
### Keep Project Structure
Do not move files around. The CLI expects files in specific locations.
### Always Use CLI Commands
Always use `kubebuilder create api` and `kubebuilder create webhook` to scaffold. Do NOT create files manually.
### E2E Tests Require an Isolated Kind Cluster
The e2e tests are designed to validate the solution in an isolated environment (similar to GitHub Actions CI).
Ensure you run them against a dedicated [Kind](https://kind.sigs.k8s.io/) cluster (not your “real” dev/prod cluster).
## After Making Changes
**After editing `*_types.go` or markers:**
```
make manifests # Regenerate CRDs/RBAC from markers
make generate # Regenerate DeepCopy methods
```
**After editing `*.go` files:**
```
make lint-fix # Auto-fix code style
make test # Run unit tests
```
## CLI Commands Cheat Sheet
### Create API (your own types)
```bash
kubebuilder create api --group <group> --version <version> --kind <Kind>
```
### Deploy Image Plugin (scaffold to deploy/manage ANY container image)
Generate a controller that deploys and manages a container image (nginx, redis, memcached, your app, etc.):
```bash
# Example: deploying memcached
kubebuilder create api --group example.com --version v1alpha1 --kind Memcached \
--image=memcached:alpine \
--plugins=deploy-image.go.kubebuilder.io/v1-alpha
```
Scaffolds good-practice code: reconciliation logic, status conditions, finalizers, RBAC. Use as a reference implementation.
### Create Webhooks
```bash
# Validation + defaulting
kubebuilder create webhook --group <group> --version <version> --kind <Kind> \
--defaulting --programmatic-validation
# Conversion webhook (for multi-version APIs)
kubebuilder create webhook --group <group> --version v1 --kind <Kind> \
--conversion --spoke v2
```
### Controller for Core Kubernetes Types
```bash
# Watch Pods
kubebuilder create api --group core --version v1 --kind Pod \
--controller=true --resource=false
# Watch Deployments
kubebuilder create api --group apps --version v1 --kind Deployment \
--controller=true --resource=false
```
### Controller for External Types (e.g., from other operators)
Watch resources from external APIs (cert-manager, Argo CD, Istio, etc.):
```bash
# Example: watching cert-manager Certificate resources
kubebuilder create api \
--group cert-manager --version v1 --kind Certificate \
--controller=true --resource=false \
--external-api-path=github.com/cert-manager/cert-manager/pkg/apis/certmanager/v1 \
--external-api-domain=io \
--external-api-module=github.com/cert-manager/cert-manager
```
**Note:** Use `--external-api-module=<module>@<version>` only if you need a specific version. Otherwise, omit `@<version>` to use what's in go.mod.
### Webhook for External Types
```bash
# Example: validating external resources
kubebuilder create webhook \
--group cert-manager --version v1 --kind Issuer \
--defaulting \
--external-api-path=github.com/cert-manager/cert-manager/pkg/apis/certmanager/v1 \
--external-api-domain=io \
--external-api-module=github.com/cert-manager/cert-manager
```
## Testing & Development
```bash
make test # Run unit tests (uses envtest: real K8s API + etcd)
make run # Run locally (uses current kubeconfig context)
```
Tests use **Ginkgo + Gomega** (BDD style). Check `suite_test.go` for setup.
## Deployment Workflow
```bash
# 1. Regenerate manifests
make manifests generate
# 2. Build & deploy
export IMG=<registry>/<project>:tag
make docker-build docker-push IMG=$IMG # Or: kind load docker-image $IMG --name <cluster>
make deploy IMG=$IMG
# 3. Test
kubectl apply -k config/samples/
# 4. Debug
kubectl logs -n <project>-system deployment/<project>-controller-manager -c manager -f
```
### API Design
**Key markers for** `api/<version>/*_types.go`:
```go
// +kubebuilder:object:root=true
// +kubebuilder:subresource:status
// +kubebuilder:resource:scope=Namespaced
// +kubebuilder:printcolumn:name="Status",type=string,JSONPath=".status.conditions[?(@.type=='Ready')].status"
// On fields:
// +kubebuilder:validation:Required
// +kubebuilder:validation:Minimum=1
// +kubebuilder:validation:MaxLength=100
// +kubebuilder:validation:Pattern="^[a-z]+$"
// +kubebuilder:default="value"
```
- **Use** `metav1.Condition` for status (not custom string fields)
- **Use predefined types**: `metav1.Time` instead of `string` for dates
- **Follow K8s API conventions**: Standard field names (`spec`, `status`, `metadata`)
### Controller Design
**RBAC markers in** `internal/controller/*_controller.go`:
```go
// +kubebuilder:rbac:groups=mygroup.example.com,resources=mykinds,verbs=get;list;watch;create;update;patch;delete
// +kubebuilder:rbac:groups=mygroup.example.com,resources=mykinds/status,verbs=get;update;patch
// +kubebuilder:rbac:groups=mygroup.example.com,resources=mykinds/finalizers,verbs=update
// +kubebuilder:rbac:groups=events.k8s.io,resources=events,verbs=create;patch
// +kubebuilder:rbac:groups=apps,resources=deployments,verbs=get;list;watch;create;update;patch;delete
```
**Implementation rules:**
- **Idempotent reconciliation**: Safe to run multiple times
- **Re-fetch before updates**: `r.Get(ctx, req.NamespacedName, obj)` before `r.Update` to avoid conflicts
- **Structured logging**: `log := log.FromContext(ctx); log.Info("msg", "key", val)`
- **Owner references**: Enable automatic garbage collection (`SetControllerReference`)
- **Watch secondary resources**: Use `.Owns()` or `.Watches()`, not just `RequeueAfter`
- **Finalizers**: Clean up external resources (buckets, VMs, DNS entries)
### Logging
**Follow Kubernetes logging message style guidelines:**
- Start from a capital letter
- Do not end the message with a period
- Active voice: subject present (`"Deployment could not create Pod"`) or omitted (`"Could not create Pod"`)
- Past tense: `"Could not delete Pod"` not `"Cannot delete Pod"`
- Specify object type: `"Deleted Pod"` not `"Deleted"`
- Balanced key-value pairs
```go
log.Info("Starting reconciliation")
log.Info("Created Deployment", "name", deploy.Name)
log.Error(err, "Failed to create Pod", "name", name)
```
**Reference:** https://github.com/kubernetes/community/blob/master/contributors/devel/sig-instrumentation/logging.md#message-style-guidelines
### Webhooks
- **Create all types together**: `--defaulting --programmatic-validation --conversion`
- **When`--force`is used**: Backup custom logic first, then restore after scaffolding
- **For multi-version APIs**: Use hub-and-spoke pattern (`--conversion --spoke v2`)
- Hub version: Usually oldest stable version (v1)
- Spoke versions: Newer versions that convert to/from hub (v2, v3)
- Example: `--group crew --version v1 --kind Captain --conversion --spoke v2` (v1 is hub, v2 is spoke)
### Learning from Examples
The **deploy-image plugin** scaffolds a complete controller following good practices. Use it as a reference implementation:
```bash
kubebuilder create api --group example --version v1alpha1 --kind MyApp \
--image=<your-image> --plugins=deploy-image.go.kubebuilder.io/v1-alpha
```
Generated code includes: status conditions (`metav1.Condition`), finalizers, owner references, events, idempotent reconciliation.
## Distribution Options
### Option 1: YAML Bundle (Kustomize)
```bash
# Generate dist/install.yaml from Kustomize manifests
make build-installer IMG=<registry>/<project>:tag
```
**Key points:**
- The `dist/install.yaml` is generated from Kustomize manifests (CRDs, RBAC, Deployment)
- Commit this file to your repository for easy distribution
- Users only need `kubectl` to install (no additional tools required)
**Example:** Users install with a single command:
```bash
kubectl apply -f https://raw.githubusercontent.com/<org>/<repo>/<tag>/dist/install.yaml
```
### Option 2: Helm Chart
```bash
kubebuilder edit --plugins=helm/v2-alpha # Generates dist/chart/ (default)
kubebuilder edit --plugins=helm/v2-alpha --output-dir=charts # Generates charts/chart/
```
**For development:**
```bash
make helm-deploy IMG=<registry>/<project>:<tag> # Deploy manager via Helm
make helm-deploy IMG=$IMG HELM_EXTRA_ARGS="--set ..." # Deploy with custom values
make helm-status # Show release status
make helm-uninstall # Remove release
make helm-history # View release history
make helm-rollback # Rollback to previous version
```
**For end users/production:**
```bash
helm install my-release ./<output-dir>/chart/ --namespace <ns> --create-namespace
```
**Important:** If you add webhooks or modify manifests after initial chart generation:
1. Backup any customizations in `<output-dir>/chart/values.yaml` and `<output-dir>/chart/manager/manager.yaml`
2. Re-run: `kubebuilder edit --plugins=helm/v2-alpha --force` (use same `--output-dir` if customized)
3. Manually restore your custom values from the backup
### Publish Container Image
```bash
export IMG=<registry>/<project>:<version>
make docker-build docker-push IMG=$IMG
```
## References
### Essential Reading
- **Kubebuilder Book**: https://book.kubebuilder.io (comprehensive guide)
- **controller-runtime FAQ**: https://github.com/kubernetes-sigs/controller-runtime/blob/main/FAQ.md (common patterns and questions)
- **Good Practices**: https://book.kubebuilder.io/reference/good-practices.html (why reconciliation is idempotent, status conditions, etc.)
- **Logging Conventions**: https://github.com/kubernetes/community/blob/master/contributors/devel/sig-instrumentation/logging.md#message-style-guidelines (message style, verbosity levels)
### API Design & Implementation
- **API Conventions**: https://github.com/kubernetes/community/blob/master/contributors/devel/sig-architecture/api-conventions.md
- **Operator Pattern**: https://kubernetes.io/docs/concepts/extend-kubernetes/operator/
- **Markers Reference**: https://book.kubebuilder.io/reference/markers.html
### Tools & Libraries
- **controller-runtime**: https://github.com/kubernetes-sigs/controller-runtime
- **controller-tools**: https://github.com/kubernetes-sigs/controller-tools
- **Kubebuilder Repo**: https://github.com/kubernetes-sigs/kubebuilder
The Makefile derives the envtest **minor** from `go.mod`; do not replace it with a dependency patch version that may have no published binary bundle.
- Tests are mostly stdlib `testing`; controller envtest and e2e use Ginkgo/Gomega. Fake clients do not execute CEL or schema defaults: test those with envtest, and use the API's `*OrDefault` helpers in unit tests. Envtest has no kubelet, so it cannot prove Squid starts or serves traffic.
- `make lint` / `make lint-fix` builds a custom golangci-lint with the `logcheck` plugin from `.custom-gcl.yml`; a stock global binary is insufficient. `.golangci.yml` forbids `sort` (use `slices`). Logging uses controller-runtime/logr, with Kubernetes-style messages.
- Real-host tests are opt-in: `make test-integration-kvm` / `make test-integration-kvm2` require the corresponding `KVM_TEST_*` / `KVM2_TEST_*` variables from `docs/testing.md`. After provider refactors, compile-check these with `go vet -tags=integration ./...`.
- E2E requires Docker, kind, kubectl, and an isolated cluster. `make test-e2e` uses `KIND_CLUSTER=egress-proxies-operator-test-e2e`; ensure kubectl targets that cluster, especially if it already exists. On failure, run `make cleanup-test-e2e` explicitly. Compile-check with `go vet -tags=e2e ./...`. Tracing specs additionally require both `TEMPO_URL` and `OTLP_ENDPOINT`.
- `.gitea/workflows/build.yaml` runs only on tags/manual dispatch; its gate is `go vet ./...`, `go build ./...`, `go test -short ./...`. It does not replace local lint, race, or envtest verification.
## Code generation
- After API types or Kubebuilder markers change, run `make manifests generate`. Edit source markers, never generated `config/crd/bases/*.yaml`, `config/rbac/role.yaml`, or `**/zz_generated.*.go`.
- Keep `+kubebuilder:rbac` markers in standalone package-level comment blocks, outside function/type documentation; otherwise controller-gen can silently ignore them. Inspect `git diff config/rbac/role.yaml` after generation.
- Preserve `+kubebuilder:scaffold:*` markers and the single-group layout. Use `kubebuilder create api` / `kubebuilder create webhook` for new scaffolding; `PROJECT` is CLI-managed.
## Behaviour and operational gotchas
- Managed proxies are replace-on-change: `internal/controller/spechash.go` hashes placement, resolved cloud-init content, and effective port. Do not hash all of `ProxySpec`: attributes, lease limits, and health settings must not churn VMs.
- Cloud-init Secrets must carry `crawl.example.com/cloud-init: "true"`; the manager's Secret cache excludes unlabelled Secrets. Rotating labelled Secret content triggers VM replacement.
- Proxy health checks fetch **through** the proxy. The health engine supplies snapshots/events and the reconciler writes status; status-affecting transitions drive writes, so `lastHealthCheckTime` is not a last-probe heartbeat.
- Leases/cooldowns are in-memory and disappear on restart. Discovery is not leader-elected; the deployment uses one replica. Adding replicas alone does not provide shared lease capacity.
- Use `make run-dev` locally: plain `make run` omits the required `--providers-config`. Host-side probes generally cannot route to kind pod IPs; deploy in-cluster using the README quickstart to verify kubernetes-provider health.
- `make deploy IMG=...` and `make build-installer IMG=...` modify `config/manager/kustomization.yaml` in place. Check that diff before including it in a change.
- Namespace-restricted mode disables orphan GC by default. `--gc-allow-namespaced` bypasses that guard and can classify instances owned outside the visible namespace as orphans.
+40
View File
@@ -1,5 +1,45 @@
# Changelog
## 2026-09-24 10:28 UTC — proxmox provider: reviewed, fixed, proven on a real Proxmox VE 9.2 host
The `proxmox` provider (VMs cloned from a template on Proxmox VE through its REST API)
finished its last phase: a two-pass review, a real-host run on pv1-1, and captures of the
three API answers the design had only guessed.
- Delete now removes the seed ISO **before** destroying the VM. Before, a seed delete that
failed after the destroy was never retried (the next Get is NotFound), so the ISO leaked.
A 403 for `Datastore.Allocate` now fails Delete on every path instead of being skipped.
- Create skips a probe VMID held by someone else's VM at once, instead of waiting up to 10
minutes on that VM's lock on every retry.
- The captures corrected two assumptions: a missing ISO storage is a 400 (the health check
now says `StorageNotFound`, not `APIError`), and a hibernated VM can be destroyed (Delete
destroys ours instead of latching the Proxy).
- `TestProxmox_lifecycle` and `TestProxmoxCheck_realHost` passed on PVE 9.2.20: Create 6.2 s,
an address 50 s later. A kind pass took a sample Proxy to Ready and back.
- Review record: `docs/reviews/2026-09-23-proxmox-provider-review/REPORT.md`. Timings:
`docs/testing.md`. Code: `internal/provider/proxmox/{delete,create,check,errors}.go`.
## 2026-09-18 07:52 UTC — docs/operating.md, starting with the libvirt-RPC log query
New home for day-to-day helpers against a running operator, seeded with the `jq` query
that pulls timestamp + rpc out of the manager's JSON logs (both libvirt providers log the
same `"libvirt call"` msg, so one filter covers `kvm` and `kvm2`), plus the `fromjson?`
variant for streams carrying non-JSON lines. Linked from the README's doc index.
## 2026-09-18 07:44 UTC — a kvm2 crawl-pool sample
Until now the only kvm2 manifest was the one-VM `config/samples/proxy_kvm2.yaml`, which
left the multi-proxy shape — the one a crawler actually leases from — undocumented.
- Added `config/samples/proxy_kvm2_crawl_pool.yaml`: five Managed Proxies on the `kvm2-home`
entry, four sharing one labelled cloud-init Secret and one inline (it runs squid on 8080,
so it cannot share a Secret that pins `http_port 3128`). Covers the `tier` attribute the
discovery API selects on, a `medium`/`ubuntu-24.04` member with a raised `maxLeases`, and
a member with a tightened `healthCheck`.
- Kept out of `config/samples/kustomization.yaml` on purpose so `apply -k` doesn't boot a
whole pool; the exclusion is commented there next to the existing `providers-config.yaml`
one. Validated with `kubectl apply --dry-run=server`.
## 2026-09-17 20:36 UTC — kvm2: the four high findings from the code review, and its first run on a real hypervisor
A scoped review of the kvm2 provider (`review/REPORT.md`: 52 findings, 4 high, 20 medium,
+3
View File
@@ -226,3 +226,6 @@ A lightweight `check` job (`go vet` / `go build` / `go test -short`) gates the b
- `+kubebuilder:rbac` markers are package-level. Keep them in their own comment block,
never inside a type's or function's doc comment: controller-gen silently ignores
them there. After `make manifests`, check `git diff config/rbac/role.yaml`.
- `make test-integration-proxmox` runs the proxmox provider's integration tests against
a real Proxmox VE host. They skip unless `PROXMOX_TEST_URL`, `_TOKEN_FILE`, `_NODE`,
`_TEMPLATE_VMID` and `_ISO_STORAGE` are set; see docs/testing.md for the full env block.
+21
View File
@@ -74,6 +74,10 @@ test-integration-kvm: ## Run the KVM provider test against a real hypervisor. Ne
test-integration-kvm2: ## Run the kvm2 provider test against a real hypervisor. Needs KVM2_TEST_HOST/USER/SSH_KEY/BRIDGE/IMAGE_URL; see docs/testing.md.
go test -race -tags=integration -count=1 -v -timeout 40m -run TestKVM2 ./internal/provider/kvm2/
.PHONY: test-integration-proxmox
test-integration-proxmox: ## Run the proxmox provider test against a real Proxmox VE host. Needs PROXMOX_TEST_URL/TOKEN_FILE/NODE/TEMPLATE_VMID/ISO_STORAGE; see docs/testing.md.
go test -race -tags=integration -count=1 -v -timeout 30m -run TestProxmox ./internal/provider/proxmox/
# TODO(user): To use a different vendor for e2e tests, modify the setup under 'tests/e2e'.
# The default setup assumes Kind is pre-installed and builds/loads the Manager Docker image locally.
# kubectl kuberc is disabled by default for test isolation; enable with:
@@ -196,6 +200,23 @@ kvm-ssh-secret: ## Create or update the kvm-ssh Secret from KVM_SSH_KEY (private
--from-file=known_hosts="$(KVM_KNOWN_HOSTS)" \
--dry-run=client -o yaml | "$(KUBECTL)" apply -f -
PROXMOX_TOKEN_FILE ?= $(HOME)/.config/proxmox-token
PROXMOX_CA_FILE ?= $(HOME)/.config/proxmox-ca.pem
PROXMOX_NAMESPACE ?= egress-proxies-operator-system
.PHONY: proxmox-token-secret
proxmox-token-secret: ## Create or update the proxmox-token Secret from PROXMOX_TOKEN_FILE (user@realm!tokenid=secret) and PROXMOX_CA_FILE.
@test -f "$(PROXMOX_TOKEN_FILE)" || { echo "PROXMOX_TOKEN_FILE=$(PROXMOX_TOKEN_FILE) not found"; exit 1; }
@test -f "$(PROXMOX_CA_FILE)" || { echo "PROXMOX_CA_FILE=$(PROXMOX_CA_FILE) not found"; exit 1; }
@test "$$(grep -cv '^[[:space:]]*$$' "$(PROXMOX_TOKEN_FILE)")" = 1 && \
grep -qxE '[[:space:]]*[^@[:space:]]+@[^![:space:]]+![^=[:space:]]+=[^[:space:]]+[[:space:]]*' "$(PROXMOX_TOKEN_FILE)" || \
{ echo "PROXMOX_TOKEN_FILE=$(PROXMOX_TOKEN_FILE) is not one line of user@realm!tokenid=secret"; exit 1; }
@grep -q 'BEGIN CERTIFICATE' "$(PROXMOX_CA_FILE)" || { echo "PROXMOX_CA_FILE=$(PROXMOX_CA_FILE) holds no PEM certificate"; exit 1; }
"$(KUBECTL)" -n $(PROXMOX_NAMESPACE) create secret generic proxmox-token \
--from-file=token="$(PROXMOX_TOKEN_FILE)" \
--from-file=ca.pem="$(PROXMOX_CA_FILE)" \
--dry-run=client -o yaml | "$(KUBECTL)" apply -f -
.PHONY: deploy
deploy: manifests kustomize ## Deploy controller to the K8s cluster specified in ~/.kube/config.
cd config/manager && "$(KUSTOMIZE)" edit set image controller=${IMG}
+4 -1
View File
@@ -192,6 +192,7 @@ can be two differently-configured instances of the same `gcp` type (see
| `gcp` | A Compute Engine VM with an ephemeral external IP | `project` (required), `network`, `networkTag`, `diskSizeGb` | The real egress fleet. Deliberately uses only four API calls (Insert / Get / Delete / AggregatedList), all fire-and-forget: `Create` returns as soon as the operation is submitted and the reconciler discovers progress by polling `Get`. Auth is Application Default Credentials — workload identity in-cluster, `gcloud` ADC locally; no key-file plumbing. |
| `kvm` | A libvirt VM on one of your own hypervisors, reached over SSH | `sshKeyFile`, `knownHostsFile` (unless `insecureIgnoreHostKey`), `machineTypes`, `hosts` (required); `images`, `diskSizeGb`, `vendorData`, `installGuestAgent` | Home-lab or on-prem egress. The base image is downloaded into each host's pool on first use; `Create` waits for the VM to start. See [docs/kvm-provider.md](docs/kvm-provider.md). |
| `kvm2` | The same: a libvirt VM on one of your own hypervisors, reached over SSH | the same block shape as `kvm`, under `kvm2:` | The second-generation libvirt provider, built from the provider contract. Adds a child span per libvirt RPC, a pool-space precheck reported as quota (the reconciler waits instead of failing), SSH credentials re-read on every dial, and image downloads and health checks on connections of their own. It does not adopt `kvm`'s VMs. See [docs/kvm2-provider.md](docs/kvm2-provider.md). |
| `proxmox` | A VM on a Proxmox VE cluster: a full clone of a template you prepare, reached through the Proxmox REST API with an API token | `url`, `tokenFile`, `nodes`, `isoStorage`, `templates`, `machineTypes` (required); `caFile` or `insecureSkipTLSVerify`, `storage`, `bridge`, `vlanTag`, `vmidRange`, `requestTimeoutSeconds`, `vendorData`, `installGuestAgent` | Home-lab or on-prem egress without SSH. Cloud-init goes on a seed ISO uploaded through the API; the address comes from the QEMU guest agent, so the proxy network needs a route to a package mirror unless the template ships the agent. `Create` waits for the VM to run. The token is read on every request, from the optional `proxmox-token` Secret (`make proxmox-token-secret`). Built from the provider contract and run end to end against Proxmox VE 9.2.20. See [docs/proxmox-provider.md](docs/proxmox-provider.md). |
### Adding a provider
@@ -310,4 +311,6 @@ and the manual kind verification procedure — is in
Project layout, reconcile-loop diagrams, and the decision log are in
[docs/architecture.md](docs/architecture.md); the build history is in
[docs/plans-executions/](docs/plans-executions/).
[docs/plans-executions/](docs/plans-executions/). Day-to-day helpers for a
running operator — log queries and one-off inspections — are collected in
[docs/operating.md](docs/operating.md).
+1 -1
View File
@@ -68,7 +68,7 @@ type ProviderTarget struct {
// ProviderHealthStatus is the result of the latest check of one provider.
type ProviderHealthStatus struct {
// type is the provider's type from the providers config: kubernetes,
// gcp, kvm or kvm2.
// gcp, kvm, kvm2 or proxmox.
// +optional
Type string `json:"type,omitempty"`
+2
View File
@@ -61,6 +61,7 @@ import (
"gitea.home.hrajfrisbee.cz/kacerr/egress-proxies-operator/internal/provider/kubernetes"
"gitea.home.hrajfrisbee.cz/kacerr/egress-proxies-operator/internal/provider/kvm"
"gitea.home.hrajfrisbee.cz/kacerr/egress-proxies-operator/internal/provider/kvm2"
"gitea.home.hrajfrisbee.cz/kacerr/egress-proxies-operator/internal/provider/proxmox"
"gitea.home.hrajfrisbee.cz/kacerr/egress-proxies-operator/internal/provider/registry"
"gitea.home.hrajfrisbee.cz/kacerr/egress-proxies-operator/internal/providerhealth"
"gitea.home.hrajfrisbee.cz/kacerr/egress-proxies-operator/internal/tracing"
@@ -213,6 +214,7 @@ func main() {
},
"kvm": kvm.New,
"kvm2": kvm2.New,
"proxmox": proxmox.New,
})
if err != nil {
setupLog.Error(err, "Failed to build providers")
@@ -183,7 +183,7 @@ spec:
type:
description: |-
type is the provider's type from the providers config: kubernetes,
gcp, kvm or kvm2.
gcp, kvm, kvm2 or proxmox.
type: string
type: object
type: object
+17
View File
@@ -166,6 +166,18 @@ spec:
- name: kvm-ssh
mountPath: /etc/proxy-operator/kvm
readOnly: true
# API token and CA for the proxmox provider (tokenFile / caFile in
# the providers config point here). Optional: the mount is empty when
# the Secret does not exist, and only proxmox entries need it. The
# token file is read on every request; the CA once, on first use.
# Create it with:
# kubectl -n egress-proxies-operator-system create secret generic proxmox-token \
# --from-file=token=<file holding user@realm!tokenid=secret> \
# --from-file=ca.pem=<the node's /etc/pve/pve-root-ca.pem>
# or `make proxmox-token-secret`.
- name: proxmox-token
mountPath: /etc/proxy-operator/proxmox
readOnly: true
# Scratch space: the root filesystem is read-only, and the kvm
# provider builds each VM's cloud-init seed ISO through a temp
# directory (a few hundred KB, deleted right after upload).
@@ -180,6 +192,11 @@ spec:
secret:
secretName: kvm-ssh
optional: true
# The proxmox provider's token and CA; see the mount above.
- name: proxmox-token
secret:
secretName: proxmox-token
optional: true
- name: tmp
emptyDir:
sizeLimit: 64Mi
+28
View File
@@ -84,4 +84,32 @@ data:
bridge: br0
# socket: /var/run/libvirt/libvirt-sock
# pool: default
# proxmox clones a prepared template on a Proxmox VE cluster through its
# REST API and seeds it with a NoCloud ISO uploaded to isoStorage; it needs
# no SSH. Both files come from the optional proxmox-token Secret mounted by
# config/manager/manager.yaml. The url's host must be a name or address in
# the node certificate. See docs/proxmox-provider.md.
# - name: proxmox-home
# type: proxmox
# proxmox:
# url: https://192.168.0.8:8006
# tokenFile: /etc/proxy-operator/proxmox/token
# caFile: /etc/proxy-operator/proxmox/ca.pem
# # insecureSkipTLSVerify: true # instead of caFile; logged as a warning
# nodes: [pv1-1] # placement.zone names a node; optional with one node
# isoStorage: proxy-seeds # iso content; the token needs Datastore.Allocate here
# # storage: local-lvm # clone target; default is the template's disk storage
# # bridge: vmbr0 # replaces the cloned net0's bridge
# vlanTag: 4 # set on the cloned net0, MAC kept
# # vmidRange: {min: 100000, max: 199999}
# # requestTimeoutSeconds: 30
# # installGuestAgent: false
# # vendorData: |
# # #cloud-config
# # ...
# templates: # placement.image names a key; add keys, never repoint one
# debian-13: {vmid: 9001}
# machineTypes: # placement.machineType names a key here
# small: {cores: 1, memoryMiB: 1024}
# medium: {cores: 2, memoryMiB: 2048, diskGiB: 10}
+4
View File
@@ -1,10 +1,14 @@
## Append samples of your project ##
# providers-config.yaml is deliberately absent: it is a sample
# --providers-config file, not a Kubernetes manifest.
# proxy_kvm2_crawl_pool.yaml is deliberately absent too: it is a whole pool
# of VMs, so it stays an explicit `kubectl apply -f` rather than something
# `apply -k` boots along with the one-per-provider samples.
resources:
- proxy_kubernetes.yaml
- proxy_gcp.yaml
- proxy_kvm.yaml
- proxy_kvm2.yaml
- proxy_proxmox.yaml
- proxy_external.yaml
# +kubebuilder:scaffold:manifestskustomizesamples
+28
View File
@@ -88,3 +88,31 @@ providers:
# bridge: br0
# # socket: /var/run/libvirt/libvirt-sock
# # pool: default
# proxmox clones a prepared template on a Proxmox VE cluster through its
# REST API and seeds it with a NoCloud ISO uploaded to isoStorage; it needs
# no SSH. Both files come from the optional proxmox-token Secret mounted by
# config/manager/manager.yaml. The url's host must be a name or address in
# the node certificate. See docs/proxmox-provider.md.
# - name: proxmox-home
# type: proxmox
# proxmox:
# url: https://192.168.0.8:8006
# tokenFile: /etc/proxy-operator/proxmox/token
# caFile: /etc/proxy-operator/proxmox/ca.pem
# # insecureSkipTLSVerify: true # instead of caFile; logged as a warning
# nodes: [pv1-1] # placement.zone names a node; optional with one node
# isoStorage: proxy-seeds # iso content; the token needs Datastore.Allocate here
# # storage: local-lvm # clone target; default is the template's disk storage
# # bridge: vmbr0 # replaces the cloned net0's bridge
# vlanTag: 4 # set on the cloned net0, MAC kept
# # vmidRange: {min: 100000, max: 199999}
# # requestTimeoutSeconds: 30
# # installGuestAgent: false
# # vendorData: |
# # #cloud-config
# # ...
# templates: # placement.image names a key; add keys, never repoint one
# debian-13: {vmid: 9001}
# machineTypes: # placement.machineType names a key here
# small: {cores: 1, memoryMiB: 1024}
# medium: {cores: 2, memoryMiB: 2048, diskGiB: 10}
@@ -0,0 +1,103 @@
# A pool of Managed crawl proxies on the kvm2 provider — the multi-proxy
# counterpart to the single proxy_kvm2.yaml sample. Everything here boots on
# the one host the kvm2 entry configures (beelink); the proxies differ in the
# attributes crawlers select on, in size, and in image.
#
# Requires the same prerequisites as proxy_kvm2.yaml: a providers-config
# entry named "kvm2-home" and the kvm-ssh Secret that the kvm and kvm2
# providers share (docs/kvm2-provider.md).
#
# Deliberately NOT in kustomization.yaml: `kubectl apply -k config/samples`
# would boot every VM in here at once. Apply it on its own when you want the
# pool:
#
# kubectl apply -f config/samples/proxy_kvm2_crawl_pool.yaml
#
# Crawlers reach these through the discovery API by attribute, e.g.
# `GET /v1/proxies?attr.purpose=crawl&healthy=true`, or by leasing one with
# the same selector. Attribute matching is exact equality, so keep the keys
# consistent across the pool and vary only the values you want to select on.
#
# The first Proxy for an image on the host waits for that image to download;
# the rest reuse the cached copy. debian-13 and ubuntu-24.04 are separate
# downloads, so the ubuntu member below is the slow one on a cold host.
---
# One cloud-init Secret shared by the whole pool: the proxies below point at
# it with secretRef instead of each carrying its own copy of the squid
# config. The crawl.example.com/cloud-init label is mandatory — the manager's
# cache only holds Secrets that carry it, so an unlabelled Secret is
# invisible to the operator and the Proxy never resolves.
#
# Editing this Secret changes the resolved cloud-init of every Proxy that
# references it, and that replaces each of their VMs. That is the trade for
# sharing one definition: convenient for a config rollout, disruptive if you
# only meant to touch one member. Give a member its own Secret (or an inline
# block) when it needs to diverge.
apiVersion: v1
kind: Secret
metadata:
name: crawl-squid-cloud-init
labels:
crawl.example.com/cloud-init: "true"
type: Opaque
stringData:
# The default key name; SecretKeySelector.key defaults to user-data, so
# the proxies below omit it.
user-data: |
#cloud-config
packages:
- squid
write_files:
- path: /etc/squid/conf.d/proxy-operator.conf
content: |
http_port 3128
http_access allow all
via off
forwarded_for off
runcmd:
- systemctl restart squid
# Optional: log in with `ssh debian@<ip>` (or `ubuntu@<ip>` on the
# ubuntu-24.04 member) to look around; `virsh console <name>` on the
# host works too.
# ssh_authorized_keys:
# - ssh-ed25519 AAAA... you@laptop
---
apiVersion: crawl.example.com/v1alpha1
kind: Proxy
metadata:
name: crawl-home-1
spec:
mode: Managed
provider: kvm2-home
placement:
zone: beelink
machineType: small
image: debian-13
port: 3128
cloudInit:
secretRef:
name: crawl-squid-cloud-init
attributes:
geo: home
purpose: crawl
tier: general
# ---
# apiVersion: crawl.example.com/v1alpha1
# kind: Proxy
# metadata:
# name: crawl-home-2
# spec:
# mode: Managed
# provider: kvm2-home
# placement:
# zone: beelink
# machineType: small
# image: debian-13
# port: 3128
# cloudInit:
# secretRef:
# name: crawl-squid-cloud-init
# attributes:
# geo: home
# purpose: crawl
# tier: general
+190
View File
@@ -0,0 +1,190 @@
# A pool of Managed crawl proxies on the kvm2 provider — the multi-proxy
# counterpart to the single proxy_kvm2.yaml sample. Everything here boots on
# the one host the kvm2 entry configures (beelink); the proxies differ in the
# attributes crawlers select on, in size, and in image.
#
# Requires the same prerequisites as proxy_kvm2.yaml: a providers-config
# entry named "kvm2-home" and the kvm-ssh Secret that the kvm and kvm2
# providers share (docs/kvm2-provider.md).
#
# Deliberately NOT in kustomization.yaml: `kubectl apply -k config/samples`
# would boot every VM in here at once. Apply it on its own when you want the
# pool:
#
# kubectl apply -f config/samples/proxy_kvm2_crawl_pool.yaml
#
# Crawlers reach these through the discovery API by attribute, e.g.
# `GET /v1/proxies?attr.purpose=crawl&healthy=true`, or by leasing one with
# the same selector. Attribute matching is exact equality, so keep the keys
# consistent across the pool and vary only the values you want to select on.
#
# The first Proxy for an image on the host waits for that image to download;
# the rest reuse the cached copy. debian-13 and ubuntu-24.04 are separate
# downloads, so the ubuntu member below is the slow one on a cold host.
---
# One cloud-init Secret shared by the whole pool: the proxies below point at
# it with secretRef instead of each carrying its own copy of the squid
# config. The crawl.example.com/cloud-init label is mandatory — the manager's
# cache only holds Secrets that carry it, so an unlabelled Secret is
# invisible to the operator and the Proxy never resolves.
#
# Editing this Secret changes the resolved cloud-init of every Proxy that
# references it, and that replaces each of their VMs. That is the trade for
# sharing one definition: convenient for a config rollout, disruptive if you
# only meant to touch one member. Give a member its own Secret (or an inline
# block) when it needs to diverge.
apiVersion: v1
kind: Secret
metadata:
name: crawl-squid-cloud-init
labels:
crawl.example.com/cloud-init: "true"
type: Opaque
stringData:
# The default key name; SecretKeySelector.key defaults to user-data, so
# the proxies below omit it.
user-data: |
#cloud-config
packages:
- squid
write_files:
- path: /etc/squid/conf.d/proxy-operator.conf
content: |
http_port 3128
http_access allow all
via off
forwarded_for off
runcmd:
- systemctl restart squid
# Optional: log in with `ssh debian@<ip>` (or `ubuntu@<ip>` on the
# ubuntu-24.04 member) to look around; `virsh console <name>` on the
# host works too.
# ssh_authorized_keys:
# - ssh-ed25519 AAAA... you@laptop
---
apiVersion: crawl.example.com/v1alpha1
kind: Proxy
metadata:
name: crawl-home-1
spec:
mode: Managed
provider: kvm2-home
placement:
zone: beelink
machineType: small
image: debian-13
port: 3128
cloudInit:
secretRef:
name: crawl-squid-cloud-init
attributes:
geo: home
purpose: crawl
tier: general
---
apiVersion: crawl.example.com/v1alpha1
kind: Proxy
metadata:
name: crawl-home-2
spec:
mode: Managed
provider: kvm2-home
placement:
zone: beelink
machineType: small
image: debian-13
port: 3128
cloudInit:
secretRef:
name: crawl-squid-cloud-init
attributes:
geo: home
purpose: crawl
tier: general
---
apiVersion: crawl.example.com/v1alpha1
kind: Proxy
metadata:
name: crawl-home-3
spec:
mode: Managed
provider: kvm2-home
placement:
zone: beelink
machineType: small
image: debian-13
port: 3128
cloudInit:
secretRef:
name: crawl-squid-cloud-init
attributes:
geo: home
purpose: crawl
tier: general
---
# A heavier member: two vCPUs and 2 GiB from the medium machine type, a
# different base image, and room for more concurrent leases. tier=heavy is
# what a crawler selects on to land here rather than on the general members.
apiVersion: crawl.example.com/v1alpha1
kind: Proxy
metadata:
name: crawl-home-heavy-1
spec:
mode: Managed
provider: kvm2-home
placement:
zone: beelink
machineType: medium
image: ubuntu-24.04
port: 3128
cloudInit:
secretRef:
name: crawl-squid-cloud-init
attributes:
geo: home
purpose: crawl
tier: heavy
maxLeases: 10
---
# A member that probes harder than the pool default (30s interval, 3 failures
# to go unhealthy): a tighter loop takes a flapping proxy out of rotation
# sooner, at the cost of more probe traffic. Everything under healthCheck is
# defaulted, so set only what you want to change.
#
# This one also runs squid on a non-default port, which is why it carries its
# own inline cloud-init: spec.port and the http_port in the config have to
# agree, and the shared Secret above pins 3128.
apiVersion: crawl.example.com/v1alpha1
kind: Proxy
metadata:
name: crawl-home-probe-fast-1
spec:
mode: Managed
provider: kvm2-home
placement:
zone: beelink
machineType: small
image: debian-13
port: 8080
cloudInit:
inline: |
#cloud-config
packages:
- squid
write_files:
- path: /etc/squid/conf.d/proxy-operator.conf
content: |
http_port 8080
http_access allow all
via off
forwarded_for off
runcmd:
- systemctl restart squid
healthCheck:
intervalSeconds: 10
timeoutSeconds: 3
failureThreshold: 2
attributes:
geo: home
purpose: crawl
tier: general
+46
View File
@@ -0,0 +1,46 @@
# A Managed proxy backed by a Proxmox VE cluster through the proxmox
# provider: the operator full-clones a prepared template on the named node,
# attaches a NoCloud seed ISO carrying this cloud-init, and starts the VM on
# the configured network (here VLAN 4 on vmbr0, IP from that network's
# DHCP), where cloud-init installs Squid. Requires a providers-config entry
# named "proxmox-home" (see providers-config.yaml) and the proxmox-token
# Secret (docs/proxmox-provider.md).
#
# placement.zone is a node name from proxmox.nodes (optional when only one
# node is configured); machineType and image are keys in the provider's
# machineTypes and templates maps. The address comes from the guest agent,
# which a vendor-data script installs at first boot, so the VM's network
# needs a route to a package mirror.
apiVersion: crawl.example.com/v1alpha1
kind: Proxy
metadata:
name: proxy-proxmox-sample
spec:
mode: Managed
provider: proxmox-home
placement:
zone: pv1-1
machineType: small
image: debian-13
port: 3128
cloudInit:
inline: |
#cloud-config
packages:
- squid
write_files:
- path: /etc/squid/conf.d/proxy-operator.conf
content: |
http_port 3128
http_access allow all
via off
forwarded_for off
runcmd:
- systemctl restart squid
# Optional: log in with `ssh debian@<ip>` (the Debian image's default
# user) to look around; `qm terminal <vmid>` on the node works too.
# ssh_authorized_keys:
# - ssh-ed25519 AAAA... you@laptop
attributes:
geo: home
purpose: crawl
+68 -1
View File
@@ -11,6 +11,8 @@
| gcp provider | `internal/provider/gcp` | library | — | Compute Engine VMs, four API calls, fire-and-forget ops |
| kvm provider | `internal/provider/kvm` | library | — | VMs on libvirt hosts over SSH-forwarded sockets; image cache, overlay disks, NoCloud seeds |
| kvm2 provider | `internal/provider/kvm2` | library | — | the same backend, built clean-room from the provider contract as a second type: per-RPC child spans, pool-space quota, per-call budgets, its own metadata namespace, credentials re-read on every dial |
| proxmox provider | `internal/provider/proxmox` | library | — | VMs on a Proxmox VE cluster through its REST API: full clones of a template, a NoCloud seed uploaded through the API, the VMID derived from the Proxy UID, every task waited on, HTTP child spans |
| Shared provider helpers | `internal/provider/nocloud`, `internal/provider/keyedlock` | library | — | the NoCloud seed ISO and vendor-data builder, and one-slot semaphores by key; used by kvm2 and proxmox |
| Health engine | `internal/health` | Runnable | yes | through-the-proxy probes, thresholds, transition events |
| Lease store | `internal/lease` | Runnable (expiry sweep) | no | in-memory leases + cooldowns, single mutex |
| Discovery API | `internal/discovery` | Runnable | no | HTTP list/lease/release/report on `:8090` |
@@ -203,6 +205,38 @@ outlives its ctx or budget kills the pooled connection (the only way to unblock
go-libvirt), so other calls in flight on that host fail once with Transient and
retry; the next call redials. Each RPC is a child span kvm2.<Method> under
provider.*, plus kvm2.dial and kvm2.download.
proxmox provider (internal/provider/proxmox/) - the Proxmox VE REST API, one URL per cluster
prov.Create ──► placement checked against the catalogue (Permanent before any request)
GET /cluster/resources: a VM with the name → ownership check ─┐ every step
else: template config checked (not a template, cloud-init drive, │ idempotent,
no net0 or boot disk it needs = Permanent) │ under a
VMID = hash(UID) in vmidRange, probing on; a taken VMID → │ per-name lock
ownership check: wait out a lock, adopt our name + mark + │ shared with
UID, probe past another name, Permanent otherwise │ Delete; each
POST .../clone (full, description = the mark) → wait task │ task's UPID
upload seed ISO to isoStorage (overwrites) → wait task │ polled every
PUT .../config: cores, memory, agent=1, tag, ide2 = seed, │ 1s to its end
net0 bridge/tag when configured │
resize if diskGiB is larger → wait task │
start → wait task → poll status/current until running (60s) ─┘
prov.Get ──► /cluster/resources (absent = NotFound; lock or node unknown = Transient)
+ config (name and mark must match, else NotFound: VMID reused)
+ status/current (paused and hibernated = Stopped)
+ agent network-get-interfaces (first routable IPv4; agent not
up yet = Provisioning)
prov.Delete ──► resources + config: gone, not ours or unmarked = leave it;
stop (unless stopped or hibernated) → delete the seed on the
VM's node → destroy (purge); with no VM of ours left, the seed
is deleted on every node
prov.ListByTag ─► one /cluster/resources for the whole cluster, one config read per
candidate (tag or proxy- name); the description's mark decides
Check ──► per node: /version, node status, isoStorage status, templates in
/cluster/resources
providerID = vms/<vmid>/<name>: no node, because VMs migrate; the node is looked
up in /cluster/resources on every call. Token read from its file on every
request. Each request is an HTTP <METHOD> child span from the parent-gated
transport wrapper; a cancelled request ends only itself.
```
The reconciler never watches provider-side resources (Pods or GCP VMs).
@@ -387,6 +421,12 @@ incoming trace context):
from a span decorator over the seam, applied once in the dialer, so no
RPC is unspanned. The `libvirt call` log line of an RPC is written
inside its span. The `kvm` provider has no child spans.
- proxmox child spans: one `HTTP <METHOD>` client span per API request,
from `tracing.RestConfigWrapper()` on the provider's own HTTP transport,
under `provider.*` or `providerhealth.check`. Parent-gated like the
Kubernetes API spans, so a request without a parent starts none. The
provider's logging decorator starts no span; its `proxmox api call` line
carries the operation span's IDs.
- Kubernetes API calls become client child spans via a wrapped
`rest.Config` transport — gated on an existing parent span, so informer
list/watch long-polls and leader-election renewals never create root
@@ -412,12 +452,13 @@ the slog handler's ctx.
Answers "can this cluster reach each provider's backend right now?"
without waiting for a Proxy to fail. A provider opts in by implementing
`provider.Checker`: `Check(ctx) []TargetStatus` returns one verdict per
target and never fails as a whole. All four providers do:
target and never fails as a whole. All five providers do:
| Provider | Target | What one check does |
|---|---|---|
| kvm | each `kvm.hosts` entry | over the pooled SSH+libvirt connection: libvirt version and hostname, then the configured pool must exist and be running; reports pool capacity and free space ([details](kvm-provider.md#health-check)) |
| kvm2 | each `kvm2.hosts` entry | on a fresh SSH+libvirt connection dialled for the check and closed after it: libvirt version and hostname, then the configured pool must exist and be running; `details` carry `user`, `pool`, `hostname`, `libvirtVersion`, `poolState`, `poolCapacityGiB` and `poolAvailableGiB` ([details](kvm2-provider.md#health-check)) |
| proxmox | each `proxmox.nodes` entry | four requests on the shared HTTP client: API version, node status, the `isoStorage` status (it must be active and hold `iso` content), and the cluster's resources (every configured template must exist and be a template); `details` carry `isoStorage`, `tokenID`, `pveVersion`, `kernel`, `isoStorageAvailGiB` and `templates` ([details](proxmox-provider.md#health-check)) |
| kubernetes | `apiserver` | lists at most one managed Pod cluster-wide: the verb and scope `ListByTag` and GC rely on |
| gcp | the project | one page of an aggregated instance list with `maxResults=1`: credentials, Compute API enabled, `compute.instances.list` |
@@ -703,3 +744,29 @@ they read as choices, not accidents. Chronological by build step.
the Secret needs no restart; nothing is cached. The constructor only
checks that the files exist, to warn early without failing the provider
set.
- **proxmox derives the VMID from the Proxy UID and checks ownership
before using a taken one.** Proxmox addresses VMs by VMID, and names are
not unique, so the name cannot be the key. A hash of the UID placed in
`vmidRange` makes a retry land on the same VMID; a taken one is waited
on while locked, then adopted when it has our name, mark and UID,
probed past when it has another name, and refused otherwise. The wait
is what stops a retry after a crash mid-clone from cloning a duplicate:
a clone in progress has no name yet, so the by-name lookup misses it.
- **proxmox writes its mark into the VM's description**, in the clone
call itself, so no VM of ours exists unmarked. Tags cannot be passed to
the clone, and the clone's own creation time is copied from the
template. The `proxy_operator` tag is set afterwards for the Proxmox UI
and to narrow GC's candidates; the mark decides.
- **proxmox delivers cloud-init as a seed ISO uploaded through the API**,
not as Proxmox's own cloud-init drive or a `cicustom` snippet. The
upload endpoint does not accept snippets, and the operator has no shell
on the nodes; the seed code is kvm2's, extracted into
`internal/provider/nocloud`. Templates must therefore carry no Proxmox
cloud-init drive, and seeds of VMs destroyed outside the operator are a
documented GC blind spot.
- **proxmox resolves a VM's node on every call** from
`/cluster/resources`, and its providerID names none, because VMs migrate
between nodes. The cost is one extra request per operation; the gain is
that a migrated VM stays addressable and the listing, which includes the
guests of an unreachable node as `status: unknown`, never mistakes a
down node for a deleted VM.
+1 -1
View File
@@ -511,7 +511,7 @@ every 15 s) marks the connection dead on failure.
per host, dialled lazily, under a mutex with a per-host dial mutex so
concurrent callers dial once and share the result), the per-`(host, volume)`
download locks, and a lock per instance name. Both lock sets are one instance
each of `keyedLocks`: a one-slot semaphore per key, so a waiter gives up when
each of `keyedlock.Locks` (`internal/provider/keyedlock`): a one-slot semaphore per key, so a waiter gives up when
its ctx ends, reference-counted so an entry is removed from the map once no
caller holds or waits for it and the map does not grow with every name ever
seen, while two waiters on one key always share one semaphore. Create and
+3 -2
View File
@@ -252,7 +252,8 @@ UIDs, and a marked domain with a matching or empty `uid` is adopted. The
namespace is kvm2's own, so the `kvm` provider's domains are invisible to
it (see *Migrating from kvm*).
The seed (`buildSeedISO` in `seed.go`) is an ISO 9660 image labelled
The seed (`nocloud.BuildSeedISO`, shared with the proxmox provider in
`internal/provider/nocloud`) is an ISO 9660 image labelled
`cidata` with three files. `user-data` is the Proxy's `cloudInit`, **byte
for byte**: any format cloud-init accepts works, including a Jinja
template, a MIME archive, a shell script or a gzipped payload; the
@@ -267,7 +268,7 @@ previous attempt may have been cut off mid-upload and a truncated seed
would boot a VM without cloud-init.
The guest agent rides along in `vendor-data` as a **shell script**
(`guestAgentScript` in `seed.go`), not a cloud-config, on purpose:
(`nocloud.GuestAgentScript`), not a cloud-config, on purpose:
cloud-init merges a vendor-data cloud-config under the user-data with
mergers that keep the user-data's value for every top-level key both
define and never append lists, so a Proxy carrying its own `packages:`
+58
View File
@@ -0,0 +1,58 @@
# Operating the operator
Helpers for running the operator day to day: log queries, one-off
inspections, and the small recipes that are easier to look up than to
reconstruct. Not a runbook — it assumes you already know what you are
looking for.
## Log queries
The manager logs one JSON object per line, so `jq` is the tool. The
`.msg` field is the discriminator: it is a fixed string per event, and
the variable parts live in their own fields rather than being
interpolated into the message.
### Libvirt RPCs, with timestamps
Both libvirt providers log one `"libvirt call"` per RPC, so this catches
`kvm` and `kvm2` alike:
```sh
jq -r 'select(.msg == "libvirt call") | "\(.ts) \(.rpc)"'
```
```text
2026-09-18T07:48:29Z virStoragePoolGetInfo
2026-09-18T07:48:29Z ConnectGetLibVersion
2026-09-18T07:48:29Z ConnectGetHostname
2026-09-18T07:48:29Z StoragePoolLookupByName
2026-09-18T07:48:29Z StoragePoolGetInfo
```
These lines are debug level, which is the binary's default
(`zap.Options{Development: true}` in `cmd/main.go`) — nothing to turn on.
They disappear if the Deployment has been given `--zap-log-level=info`,
which is the switch for outcomes only.
Piping from a live manager:
```sh
kubectl logs -n egress-proxies-operator-system \
deploy/egress-proxies-operator-controller-manager -f \
| jq -r 'select(.msg == "libvirt call") | "\(.ts) \(.rpc)"'
```
`jq` aborts on the first line that is not JSON, which happens when
something bypasses the structured logger — a panic trace, or output from
a tool wrapping the manager. Reading line-by-line with `fromjson?` skips
those instead:
```sh
jq -rR 'fromjson? | select(.msg == "libvirt call") | [.ts, .rpc] | @tsv'
```
Note that the connection-level events use their own `.msg` values
(`"libvirt host connected"`, `"kvm2 host checked"`), so they drop out of
the filters above. Widen to `select(.msg | startswith("libvirt"))` to
keep the connect lines — usually what you want when chasing a latency
spike rather than a call count.
File diff suppressed because it is too large Load Diff
@@ -0,0 +1,884 @@
# Plan: proxmox, a Proxmox VE provider built from the provider contract
**Created:** 2026-09-18 15:02
## Context
> **Superseded in places by the spec.** Phase 0 recon found ten corrections
> to the design below (listed in the execution log), and Phase 1 folded them
> into [docs/proxmox-provider-spec.md](../proxmox-provider-spec.md). Where a
> phase step here disagrees with the spec (the Create allocation loop, Get's
> check order, resize and seed delete as tasks, no delete before upload,
> `isoStorage` required, `vlanTag`, the agent messages), **the spec wins**.
> Test names in the phases below remain the minimum; spec entry 21 adds the
> ones for the corrections.
The provider contract (`docs/provider-contract.md`), the spec template
(`docs/provider-spec-template.md`) and the kvm2 build proved the workflow:
spec first, then implementation against the contract, one commit per step.
This plan adds a third VM-style backend, `proxmox`, on the Proxmox VE 9.x
REST API at `https://<host>:8006/api2/json`. The user has a real host at
192.168.2.8 running **9.2.20** (Debian 13 Trixie, QEMU 11) and prepares the
API token and a cloud-init-capable template before Phase 0 (checklist
below). Everything below is written against 9.2; where 8.x differed the
difference is called out, because the appendix and the guide must name the
version they were proven on.
The implementation runs in sessions whose context gets compacted, so every
phase is self-contained: it names what to read first, what it creates or
edits, the test names it must add, a `Verify:` clause, a "Done when" line
and a commit subject. A fresh session reading only this plan, the spec
(`docs/proxmox-provider-spec.md`, from Phase 1 on) and the execution log's
checklist (`docs/plans-executions/2026-09-18-1502-proxmox-provider.md`) must be able to
pick up where the last one stopped.
**Resumption protocol, run at the start of every phase.** Read this plan's
Context and the phase entry; read the spec; read the execution log's
checklist; run `git branch --show-current` (must be `feat/proxmox-provider`
from Phase 0 on), `git status --short` and `git log --oneline -8`. The first
unchecked box is the phase to do. If its files already exist partially, its
"Done when" line says whether it counts as done; finish it, never redo it. A
phase is finished only when its "Done when" holds and its execution-log entry
(a few sentences, exact commands, "worth noting", the box ticked) is
committed together with the code. Follow `CLAUDE.md` for the log format.
## Before Phase 0: what the user prepares on 192.168.2.8
Run as root on the Proxmox node. **`VM.Monitor` was removed in PVE 9.0**, so
the agent read rests on `VM.GuestAgent.Audit` alone and the role below must
not mention `VM.Monitor`. Drop any privilege `pveum` still rejects and note
it for F17.
```bash
pveum user add proxy-operator@pve --comment "egress-proxies-operator"
pveum role add ProxyOperator -privs "VM.Allocate VM.Clone VM.Audit VM.Config.CDROM VM.Config.CPU VM.Config.Memory VM.Config.Disk VM.Config.Network VM.Config.Options VM.PowerMgmt VM.GuestAgent.Audit Datastore.Audit Datastore.AllocateSpace Datastore.AllocateTemplate Sys.Audit SDN.Use"
pveum aclmod / -user proxy-operator@pve -role ProxyOperator
pveum user token add proxy-operator@pve operator --privsep 0
# The secret is printed once. Save it on the workstation as one line:
# proxy-operator@pve!operator=<secret>
# in ~/.config/proxmox-token (mode 0600). The same file becomes the
# Kubernetes Secret later.
cp /etc/pve/pve-root-ca.pem /tmp/ # copy to the workstation as ~/.config/proxmox-ca.pem, or use insecureSkipTLSVerify
```
A cloud-init template without a Proxmox cloud-init drive (the provider puts
its own NoCloud ISO on `ide2`). Debian 13's BIOS boot path did not boot
under kvm2, so the template boots UEFI:
```bash
wget https://cloud.debian.org/images/cloud/trixie/latest/debian-13-generic-amd64.qcow2
qm create 9000 --name debian-13-template --memory 2048 --cores 2 --ostype l26 \
--machine q35 --bios ovmf --efidisk0 local-lvm:1,efitype=4m,pre-enrolled-keys=0 \
--scsihw virtio-scsi-pci --net0 virtio,bridge=vmbr0 \
--serial0 socket --vga serial0 --agent 1
qm set 9000 --scsi0 local-lvm:0,import-from=/root/debian-13-generic-amd64.qcow2
qm set 9000 --boot order=scsi0
qm template 9000
```
Facts the user hands over at Phase 0: the token file path, the CA file path
(or "insecure"), the node name, the template VMID, the ISO-content storage
id (usually `local`), the VM disk storage (`local-lvm`), the bridge, and
whether the operator's cluster can route to that bridge's LAN. Everything
else Phase 0 reads from the API.
## Design decisions made with the user (not re-opened)
- **Backend and client.** Proxmox VE 9.x REST API, JSON, HTTP/1.1 only
(pveproxy offers no HTTP/2 on any release; only Proxmox Backup Server
does).
Thin client on `net/http` (`client-kind: thin-client`), no SDK, no new
dependency. Auth header
`Authorization: PVEAPIToken=<user>@<realm>!<tokenid>=<secret>`, read from a
mounted token file on every request (rotation free). TLS: `caFile` or
`insecureSkipTLSVerify` (warned at startup). Child spans from a parent-gated
otelhttp transport: `tracing.RestConfigWrapper()(base)` returns a plain
`http.RoundTripper` and works as is (`child-spans: transport-wrapper`).
- **Image source.** Clone a prepared template. Config catalogue
`templates: {<key>: {vmid: 9000}}`; `Placement.Image` is a key.
`Placement.Zone` is a node name from `nodes: [..]` (optional with one node).
`Placement.MachineType` is a key of `machineTypes: {<key>: {cores, memoryMiB,
diskGiB}}`; `diskGiB` (optional) grows the boot disk via `PUT .../resize`.
`Placement.Region` is ignored (documented).
- **Identity.** `primary-key: allocated-id`. VMID derived deterministically
from the Proxy UID (FNV-1a 64 into `vmidRange: {min, max}`, default
100000 to 199999) with forward probing on collision; a clone whose `newid`
is taken fails with a 500, which is the allocate-then-verify step. Create
first looks the name up in `GET /cluster/resources?type=vm`; an existing VM
with that name must carry our mark and a matching or empty UID, else
Permanent. providerID is location-free: `vms/<vmid>/<name>` (VMs migrate;
the node is resolved from `/cluster/resources` on every call). Get and
Delete verify the VM at that VMID has that name and our mark; a different
name means our VM is gone and the VMID was reused: Get returns NotFound,
Delete returns nil without touching it.
- **Cloud-init.** NoCloud seed ISO built in-process by reusing kvm2's seed
code, extracted into a shared package `internal/provider/nocloud` (its own
phase). Uploaded to `isoStorage` (default `local`) on the target node via
`POST /nodes/{node}/storage/{storage}/upload` (multipart, `content=iso`,
`filename=<name>-seed.iso`, streamed through `io.Pipe`, returns a UPID),
attached as `ide2: <storage>:iso/<name>-seed.iso,media=cdrom`. The template
must have no Proxmox cloudinit drive on ide2 (documented prerequisite).
Vendor-data installs qemu-guest-agent like kvm2 (`installGuestAgent`,
`vendorData`).
- **GC mark.** `gc-mark: written-metadata` in the VM `description`, which the
clone call accepts (`tags` is not a clone parameter, so tags would be a
mark window). Content, one per line: `proxy-operator-managed=true`,
`proxy-operator-uid=<uid>`, `proxy-operator-created-at=<RFC3339>`.
`created-at: written-at-create`. The config step after the clone also sets
tag `proxy_operator` for UI filtering only. **Underscore, not hyphen:**
current PVE tag validation accepts only lowercase letters, digits and
underscore, and a hyphen is rejected; the underscore form is valid under
both the strict and the permissive reading, so it is used regardless of
what F9 finds. ListByTag: one
`GET /cluster/resources?type=vm`; candidates are entries with tag
`proxy_operator` or name prefix `proxy-`; a `GET /nodes/{node}/qemu/{vmid}/config`
per candidate for the description; unmarked skipped; one unreadable VM
skipped with a V(1) line unless ctx ended; the resources call failing fails
the whole list (`list-partial: fail-whole`).
- **Create** (`create-return: blocking`, `provisioning-state: create-blocks`,
`async-ops: task-handle` and `backend-lock`), every step idempotent, in
this order: validate placement (Permanent before any call); resources
lookup by name (adopt or allocate); `POST /nodes/{tmplnode}/qemu/{tmplvmid}/clone`
`{newid, name, description, full=1, target?, storage?}`; wait the task
(`exitstatus != OK` is Transient with the message; a VM with `lock=clone`
on retry is waited on); upload the seed (always rebuilt, delete then
upload; before the config step because `ide2` may refuse a volume that
does not exist yet); config (`cores`, `memory`, `net0` bridge override if
configured, `agent=1`, `tags`, `ide2` seed); optional resize;
`POST .../status/start` and wait; poll `status/current` until `running`
(capped); return `vms/<vmid>/<name>`. Retry: name found running returns
the ID; found stopped with our mark continues from the seed step; found
locked waits.
- **Get.** Resources lookup by vmid: absent is NotFound; name mismatch or no
mark is NotFound (logged); `lock` set is Transient; `running` reads
`GET /nodes/{node}/qemu/{vmid}/agent/network-get-interfaces`, first
routable IPv4 excluding loopback and link-local gives Running, else
Provisioning (the agent not running is a 500 "QEMU guest agent is not
running", which is Provisioning, not an error); `stopped` is Stopped;
`unknown` is Transient; `qmpstatus=paused` is Stopped; anything else is
Stopped. UID and CreatedAt come from the description on every call, so a
Get is three calls: resources, config, agent.
- **Delete** (`delete-return: waits`, `secondary-resources: explicit`):
resolve by vmid via resources; absent or foreign is nil (logged); running
gets `POST status/stop` and a task wait; then
`DELETE /nodes/{node}/qemu/{vmid}?purge=1&destroy-unreferenced-disks=1` and
a task wait; then delete the seed ISO
`DELETE /nodes/{node}/storage/{storage}/content/{storage}:iso/<name>-seed.iso`
tolerating absence; a locked VM is Transient. Called again mid-delete it
repeats the same tolerant steps. GC blind spot: seed ISOs without a VM
(documented; the ISO name carries the instance name).
- **Errors**, one `classify(op, err)` over `*apiError{Status, Message, Errors}`:
401, 403, 400 and 404 are Permanent (404 means the path never exists;
NotFound comes only from the list-and-miss on resources and from the 500
"does not exist" message in Get and Delete, matched against captured
fixtures); 500 "does not exist" is NotFound (Transient inside Create); 500
"VM is locked" is Transient; 500 "already exists" is handled by the
allocator, otherwise Transient; 500 "not enough free space" or "no space
left" is QuotaExceeded once a real message is captured, else `quota:
absent`; other 5xx, 595, 596, network and ctx errors are Transient;
anything not an apiError is Transient. Error text: Proxmox puts most of it
in the HTTP status reason phrase (`HTTP/1.1 500 Configuration file '...'
does not exist`) and newer releases also in the body's `message`; the
client reads both, body first.
- **Checker.** One target per configured node, in config order, concurrently,
on the same http client: `GET /version` (Details `pveVersion`),
`GET /nodes/{node}/status` (Details `kernel`),
`GET /nodes/{node}/storage/{isoStorage}/status` (Details `isoStorage`,
`isoStorageAvailGiB`), template existence from `GET /cluster/resources?type=vm`
(the `template` flag, any node; Details `templates` as `<found>/<configured>`).
Reasons: the shared ones plus `TemplateNotFound` and `StorageNotFound`.
The `status.type` comment in `api/v1alpha1/providerhealth_types.go` gains
`proxmox`; `make manifests`.
- **Timeouts** (`timeouts: per-call`): request timeout 30s
(`requestTimeoutSeconds`); task-wait caps clone 10m, upload 2m, start and
stop 2m, destroy 5m, running wait 60s; all in a `budgets` struct
overridable in tests. A cancelled ctx cancels that request only; the
connection pool is unaffected. Per-name keyed lock for Create and Delete
(`internal/provider/keyedlock`, extracted from kvm2); Get and ListByTag
lock-free.
- **Logging** (`logging: tiered`): `logf.FromContext(ctx).WithName("proxmox").WithValues("provider", name)`,
`vmid` and `name` on instance lines; V(1) outcomes and one
`proxmox api call` line per seam call from the `observed` decorator; V(2)
request params and response summaries; never the token, never cloud-init,
never the multipart body, never `description` at V(1); no V(5). The
decorator logs, the transport traces: no `proxmox.<Method>` spans, so
nothing is double-counted.
- **Config writes** use `PUT .../config` (synchronous; the VM is stopped at
that point). `resize` is skipped when the boot disk's current `size=` is
at or above the target. Boot disk: first disk key in `boot: order=...`
that is not `net*` and not the ide2 cdrom, else `bootdisk`, else the first
of `scsi0`, `virtio0`, `sata0`, `ide0`; none is Permanent.
- **A UPID names its node** (`UPID:<node>:...`); task polling uses the node
parsed from the UPID.
- **Two VMs with our name** (Proxmox does not enforce unique names) make
Create and Get return Permanent naming both VMIDs; an admin resolves it.
- **Seam.** `api` interface with one method per endpoint (table below),
`httpAPI` production adapter, `fakeAPI` in `fake_test.go` keeping VMs, tasks
and ISOs in memory with lock simulation, task polls, per-method errors and
block channels, and a call log.
- **Integration test** `proxmox_integration_test.go` behind
`//go:build integration`, `PROXMOX_TEST_URL`, `PROXMOX_TEST_TOKEN_FILE`,
`PROXMOX_TEST_NODE`, `PROXMOX_TEST_TEMPLATE_VMID`, `PROXMOX_TEST_ISO_STORAGE`,
optional `_CA_FILE`, `_INSECURE`, `_BRIDGE`, `_STORAGE`, `_VMID_MIN`,
`_VMID_MAX`, `_TIMEOUT`, `_KEEP`, `_LOG_LEVEL`; `make test-integration-proxmox`.
- **Deployment.** Secret `proxmox-token` mounted optional at
`/etc/proxy-operator/proxmox/token`, `make proxmox-token-secret`, commented
entries in `config/manager/providers_config.yaml` and
`config/samples/providers-config.yaml`, `config/samples/proxy_proxmox.yaml`,
`docs/proxmox-provider.md`, `providers-existing.md` column and section,
`architecture.md` rows, README row, `testing.md` section, CHANGELOG after
confirmation. Branch `feat/proxmox-provider`, MR via the Gitea API (no `tea`).
## Backend facts Phase 0 must verify on 192.168.2.8 (numbered for citation)
- F1. Where error text lives: status reason phrase vs. JSON `message`; the
`errors` map shape on a 400.
- F2. Exact 500 message for a missing VM (`GET /nodes/{node}/qemu/999999/config`).
- F3. Exact 500 message for a locked VM (`POST status/start` on a VM with
`lock` set), and whether `PUT config` with `lock=backup` sets the lock and
`delete=lock` clears it through the API, or `qm unlock` is needed.
- F4. Exact 500 message when `newid` is in use on clone.
- F5. Exact 500 message for the agent not running, and for the agent call
on a stopped VM.
- F6. Exact message for no space, if it can be produced cheaply; otherwise
`quota: absent`.
- F7. Whether `PUT .../config` is refused while `lock=clone`, and with which
message.
- F8. `/cluster/resources?type=vm` fields present: `vmid`, `name`, `node`,
`status` values (`running`, `stopped`, `unknown`), `type` (`qemu`, `lxc`),
`lock`, `tags` (separator), `template`; whether `lock` appears there during
a clone.
- F9. Tag rules on 9.2: confirm a hyphen is rejected and `proxy_operator`
accepted, plus case folding, length, and the `tags` separator on
`PUT config` and in `/cluster/resources`. The underscore form stays either
way; this fact only decides what the guide tells users about their own
tags.
- F10. `description` on `POST .../clone`: accepted, survives verbatim
(newlines), max length; whether `name` accepts the 22-character
`proxy-<base32>` form.
- F11. `meta` in the cloned config: is `ctime` the clone time (evidence only;
`written-at-create` stays the profile).
- F12. Upload endpoint: field names (`content`, `filename`, file part name
`filename`), the filename rule (`.iso` suffix, no slashes), the response
(UPID), the privilege it needs, and whether re-uploading an existing name
is refused (then delete-then-upload) or overwrites. On 9.x `content`
accepts `iso`, `vztmpl` or `import` and still refuses `snippets`, which is
why the seed travels as an ISO; confirm `iso` is what an `.iso` needs.
- F13. `DELETE /nodes/{node}/storage/{storage}/content/{volid}`: response
shape (UPID or null) and the message for a missing volume.
- F14. `status/current` on a stopped VM: is `qmpstatus` present; on a paused
VM: `status=running`, `qmpstatus=paused`.
- F15. `POST .../clone`: whether a failed clone destroys the partial VM by
itself, and what a VM interrupted mid-clone looks like (config present,
`lock=clone`).
- F16. Task status shape: `status` (`running`, `stopped`), `exitstatus`
(`OK` or the error string), and how long a full clone of the template
takes.
- F17. Privileges: which of the prepared list the token really needs, in
particular the exact ACL path that satisfies `SDN.Use` for the `net0`
write (the 9.x docs describe it as "access SDN vnets and local network
bridges" without spelling out the path for a plain bridge), and
`VM.GuestAgent.Audit` for the agent read. Also whether the config step
needs `VM.Config.Cloudinit` even though only `ide2` is written.
- F18. HTTP/2 is not offered by pveproxy on any release (documented), so the
client disables it; confirm with `curl --http2 -I` that the connection
falls back to HTTP/1.1.
- F19. The template's disk key (`scsi0` or `virtio0`), its `boot` line, that
ide2 is free and no `cloudinit` drive exists.
- F20. `resize` on a stopped VM: request shape, the message when the target
is smaller or equal, and that it is synchronous.
- F21. Agent response shape: `result[].name`, `result[]["ip-addresses"][]`
with `ip-address`, `ip-address-type`.
- F22. The 401 body and reason phrase on 9.2. PVE 9.2 unified the
authentication-failure message so a failed login no longer leaks whether
the user exists, so any 401 text remembered from 8.x is wrong. Capture it
fresh into `err_unauthorized.json`.
- F23. Where the upload endpoint stages the request body, given that PVE 9
moved the node's `/tmp` to a tmpfs capped at half of RAM and cleans `/tmp`
and `/var/tmp` periodically. A seed ISO is tens of KiB so this cannot
bite in practice, but the observation belongs in the guide next to the
size limit.
## Proxmox endpoint map for the seam
| `api` method | HTTP |
|---|---|
| `Version(ctx)` | `GET /version` |
| `NodeStatus(ctx, node)` | `GET /nodes/{node}/status` |
| `StorageStatus(ctx, node, storage)` | `GET /nodes/{node}/storage/{storage}/status` |
| `ClusterResources(ctx)` | `GET /cluster/resources?type=vm` |
| `VMConfig(ctx, node, vmid)` | `GET /nodes/{node}/qemu/{vmid}/config` |
| `SetVMConfig(ctx, node, vmid, params)` | `PUT /nodes/{node}/qemu/{vmid}/config` |
| `Clone(ctx, node, vmid, cloneParams)` | `POST /nodes/{node}/qemu/{vmid}/clone` |
| `Resize(ctx, node, vmid, disk, size)` | `PUT /nodes/{node}/qemu/{vmid}/resize` |
| `Start(ctx, node, vmid)` | `POST /nodes/{node}/qemu/{vmid}/status/start` |
| `Stop(ctx, node, vmid)` | `POST /nodes/{node}/qemu/{vmid}/status/stop` |
| `VMStatus(ctx, node, vmid)` | `GET /nodes/{node}/qemu/{vmid}/status/current` |
| `Destroy(ctx, node, vmid)` | `DELETE /nodes/{node}/qemu/{vmid}?purge=1&destroy-unreferenced-disks=1` |
| `TaskStatus(ctx, node, upid)` | `GET /nodes/{node}/tasks/{upid}/status` |
| `AgentNetworkInterfaces(ctx, node, vmid)` | `GET /nodes/{node}/qemu/{vmid}/agent/network-get-interfaces` |
| `UploadISO(ctx, node, storage, filename, r, size)` | `POST /nodes/{node}/storage/{storage}/upload` (multipart) |
| `DeleteVolume(ctx, node, storage, volid)` | `DELETE /nodes/{node}/storage/{storage}/content/{volid}` |
## Package layout (`internal/provider/proxmox/`)
`proxmox.go` (`Provider`, `New`, `newProvider`, `withDefaults`, `logger`,
`node`, `wrapErr`, `lockName`, `providerOption`, `budgets`), `api.go` (the
`api` interface and DTOs `vmResource`, `vmConfig`, `vmStatus`, `taskStatus`,
`agentInterface`, `storageStatus`, `nodeStatus`, `versionInfo`,
`cloneParams`), `client.go` (`httpAPI`, `newHTTPAPI`, `do`, `readToken`,
`decodeConfig`), `errors.go` (`apiError`, `classify`, `permanentf`,
`transientf`, `isNotExist`, `isLocked`, `isAlreadyExists`,
`isAgentNotRunning`, `isNoSpace`), `id.go` (`formatProviderID`,
`parseProviderID`, `vmidFor`), `mark.go` (`instanceMark`,
`formatDescription`, `parseDescription`), `state.go` (`mapState`,
`firstIPv4`), `observe.go` (`observed`, `observe`), `task.go` (`parseUPID`,
`waitTask`, `waitUnlocked`, `waitRunning`), `seed.go` (`seedISOName`,
`seedVolID`, `uploadSeed`), `create.go`, `get.go`, `delete.go`, `list.go`,
`check.go`, plus `fake_test.go`, `provider_test.go`,
`proxmox_integration_test.go`, `testdata/*.json`. Shared:
`internal/provider/config_proxmox.go`, `internal/provider/nocloud/`,
`internal/provider/keyedlock/`.
Reused as-is: `provider.NameFromUID`, `provider.Wrap`, the `Err*` sentinels,
`provider.Label*` constants, `provider.Reason*` constants,
`tracing.RestConfigWrapper`, `tracing.WithTracerProvider`, kvm2's test
idioms (`fixture`, `assertProviderError`, `logCapture`).
## Steps (one commit per phase, on `feat/proxmox-provider` off `main`)
- **Phase 0 (orchestrator with the user): plan file, branch, host recon,
fixtures.** Read first: contract §5, §4.9, §7;
`docs/plans-executions/2026-09-16-1333-kvm2-provider.md` Step 0 entry.
Do, in this order: `git switch main && git pull --ff-only`; copy this plan
to `docs/plans/2026-09-18-1502-proxmox-provider.md` with the real timestamp
(`date "+%Y-%m-%d-%H%M"`), commit it alone on `main`
(`docs: plan the proxmox provider`); create `feat/proxmox-provider`. Ask
the user for the hand-over facts listed above. Then, with the user's
go-ahead (each `curl` is a permission prompt; read-only ones first, one at
a time), capture F1 to F21 with
`curl -s --cacert "$CA" -D /tmp/h.txt -o /tmp/b.json -w '%{http_code}\n' -H "Authorization: PVEAPIToken=$(cat "$TOKEN_FILE")" https://192.168.2.8:8006/api2/json/<path>`,
storing each response as `internal/provider/proxmox/testdata/<name>.json`
in the shape `{"status": <code>, "reason": "<status line after the
code>", "body": "<raw body>"}`: `version.json`, `node_status.json`,
`storage_status.json`, `cluster_resources.json`,
`vm_config_template.json`, `vm_config_clone.json`,
`status_current_running.json`, `status_current_stopped.json`,
`agent_network_get_interfaces.json`, `task_status_running.json`,
`task_status_stopped_ok.json`, `task_status_stopped_error.json`,
`upload_response.json`, `content_delete_response.json`,
`err_vm_missing.json`, `err_locked.json`, `err_newid_in_use.json`,
`err_agent_not_running.json`, `err_agent_vm_stopped.json`,
`err_unauthorized.json` (a wrong secret), `err_forbidden.json` (a path
the token lacks), `err_bad_param.json` (`newid=abc`), `err_no_space.json`
(only if F6 is cheap), `err_volume_missing.json` (F13). The write
experiments: one clone of the template into the top of the VMID range
(`newid=199999`, a multi-line `description`, a `name` of the `proxy-`
form), a config read of the clone (F10, F11, F19), a `PUT config` while
it is locked if timing allows (F7), an upload of a 1 KiB
`recon-seed.iso` (F12) and its delete (F13), a resize attempt (F20), a
start, an agent read (F5, F21), a stop, then destroy with
`purge=1&destroy-unreferenced-disks=1`; a final resources read shows
nothing remains. Redact nothing but the token. Write the answers to F1 to
F21 and the privilege outcome into the execution log's Phase 0 entry, one
line per fact; Phase 1 cites them as Evidence. Create the execution log
`docs/plans-executions/2026-09-18-1502-proxmox-provider.md` with the checklist (Phase
0 to Phase 10) and this entry.
Verify: `ls internal/provider/proxmox/testdata | wc -l` is at least 22;
`grep -rl 'PVEAPIToken' internal/provider/proxmox/testdata` is empty;
every fixture parses as JSON; the host shows no VM in the range and no
`recon-seed.iso`.
Done when: the plan commit is on `main`, the branch exists, the fixtures
and the execution log with F1 to F21 are committed. Commit: `proxmox:
captured API fixtures and host facts from 192.168.2.8`.
- **Phase 1 (orchestrator): the spec.** Read first: contract §2, §4, §5,
§6, §11, §12; `docs/provider-spec-template.md` end to end;
`docs/kvm2-provider-spec.md` for the level of detail; this plan's
decisions; the Phase 0 log entry. Create `docs/proxmox-provider-spec.md`
answering all 24 entries: 1 `vms/<vmid>/<name>`, strict parsing, the
node-name and storage-id regexps; 2 `allocated-id`, `vmidFor(uid, rng,
attempt)`, the two-VMs-with-our-name rule, the adoption guard; 3 the
placement table (Region ignored); 4 the template catalogue and the "add a
key, never edit" warning; 5 the seed ISO via `nocloud` and the ide2
prerequisite; 6 `machineTypes` with `diskGiB`; 7 `provisioning-window`,
`discovered` via the guest agent, the IPv4 filter, `Port` unused; 8
`written-metadata` in `description` plus the tag, `written-at-create`, F9
to F11 as Evidence; 9 the state table, `create-blocks`; 10 the ordered
Create steps with what a retry finds at each, `task-handle` plus
`backend-lock`, the budgets; 11 the Delete order, the ISO blind spot,
node-local ISO storage after a migration; 12 `fail-whole`, the candidate
rule, one config read per candidate; 13 the classify table with the F1 to
F6 messages, `not-found: error-code` plus list-and-miss on resources
(allowed because offline nodes' guests appear as `unknown`, not absent);
14 `mounted-files` (Secret `proxmox-token`, key `token`, path
`/etc/proxy-operator/proxmox/token`), `ca-file` or
`insecureSkipTLSVerify`, the privilege list from F17, rotation free; 15
the config table (Phase 2's struct); 16 the Checker; 17 `thin-client`,
`transport-wrapper`, the seam's method list; 18 the budgets table; 19 the
keyed lock per name; 20 the log keys; 21 the §11 list with the test names
from Phases 2 to 8 (copy from this plan); 22 the deployment list; 23 the
docs list; 24 the accepted limits (blocking Create up to the clone budget,
ISO blind spot, VMID range shared with humans, node-local ISO storage, no
adoption of foreign VMs, two-installs GC caveat, `unknown` status while a
node is down).
Verify: `grep -c '^\*\*Answer:\*\*$' docs/proxmox-provider-spec.md` is 0;
every axis code used appears in contract §4.20; Evidence lines cite F
numbers; `grep -c '—' docs/proxmox-provider-spec.md` is 0.
Done when: the spec is committed with all 24 entries and the Phase 1 box
ticked. Commit: `docs: proxmox provider spec`.
- **Phase 2 (implementer): config block, registration, package skeleton.**
Read first: spec entries 1, 2, 3, 6, 8, 9, 13, 15; contract §10, §5, §6;
`internal/provider/config.go`, `config_kvm.go`, `config_test.go`,
`config_kvm_test.go`; `internal/provider/kvm2/{kvm2.go,id.go,metadata.go,state.go,errors.go}`
and their tests as the pattern; `cmd/main.go`; `testdata/err_*.json`.
Create `internal/provider/config_proxmox.go`: `ProxmoxConfig{URL,
TokenFile, CAFile string; InsecureSkipTLSVerify bool; Nodes []string;
ISOStorage, Storage, Bridge string; VMIDRange *ProxmoxVMIDRange; Templates
map[string]ProxmoxTemplate; MachineTypes map[string]ProxmoxMachineType;
RequestTimeoutSeconds int; VendorData string; InstallGuestAgent *bool}`,
`ProxmoxVMIDRange{Min, Max int}`, `ProxmoxTemplate{VMID int}`,
`ProxmoxMachineType{Cores int32; MemoryMiB int64; DiskGiB int64}`,
`validateProxmox(prefix string, c *ProxmoxConfig) error` (URL http(s)
with a host; `tokenFile` non-empty; at least one node, unique, matching
`proxmoxNodeName = ^[a-z0-9][a-z0-9-]*$`; `isoStorage` and `storage`
matching `proxmoxStorageID = ^[A-Za-z][A-Za-z0-9_.-]*$`; `vmidRange` min
at least 100, max at most 999999999, min below max; at least one template
with `vmid` at least 100; at least one machine type with cores and
memoryMiB above zero and diskGiB not negative; `requestTimeoutSeconds`
not negative). Edit `config.go`: `Proxmox *ProxmoxConfig` with tag
`proxmox,omitempty`, `setBlocks`, the allow-list, the dispatch. Edit
`cmd/main.go`: `"proxmox": proxmox.New`. Create in
`internal/provider/proxmox/`: `proxmox.go` (`Provider{name, cfg, api,
now, budgets, nameLocks, tracing}`; until Phase 3 a local `keyedLocks`
copy in `locks.go` that Phase 3 deletes; `New` never dials: builds the
client, `os.Stat`s the token file and warns, warns on
`insecureSkipTLSVerify`; `withDefaults`: `isoStorage` `local`, `vmidRange`
100000 to 199999, `requestTimeoutSeconds` 30, `installGuestAgent` true;
`node(zone)` mirrors kvm2's `host`; the four methods stubbed with
`transientf("not implemented")`), `api.go` (the interface and DTOs;
`vmConfig` is `map[string]string` with scalars rendered as strings by
`decodeConfig`), `id.go`, `mark.go` (keys from `provider.LabelManaged`,
`provider.LabelManagedYes`, `provider.LabelUID` and
`proxy-operator-created-at`; `parseDescription` tolerates extra lines
and CRLF), `state.go`, `errors.go` (`apiError{Status int; Message
string; Errors map[string]string}` whose `Error()` never includes headers
or URLs; `classify` per spec entry 13; matcher helpers keyed on the exact
substrings captured in F2 to F6). Tests: `config_proxmox_test.go`
`TestValidateProxmox_table` (missing url, non-http scheme, missing
tokenFile, no nodes, duplicate node, node with a slash, bad isoStorage
id, vmidRange min below 100, min above max, no templates, template vmid
below 100, no machineTypes, cores zero, negative diskGiB, negative
timeout, valid full, valid minimal) with a `validProxmoxYAML` const;
`config_test.go` rows `proxmox type with kvm block`, `kvm type with
proxmox block`, `proxmox minimal entry loads`. `id_test.go`:
`TestFormatProviderID_roundTrip`, `TestParseProviderID_malformed`,
`TestVMIDFor_deterministicAndInRange`, `TestVMIDFor_probesForwardAndWraps`.
`mark_test.go`: `TestFormatDescription_keys`, `TestParseDescription_table`.
`state_test.go`: `TestMapState_table`, `TestFirstIPv4_filters`.
`errors_test.go`: `TestClassify_table` (every class, every matched
message loaded from `testdata/err_*.json` through `loadFixture(t, name)
*apiError`, `op == "create"` turning NotFound into Transient,
`context.Canceled`, a `*net.OpError`, `errors.New("x")`,
already-classified errors keep their class), `TestAPIError_matchers`,
`TestPermanentfAndTransientf`. `proxmox_test.go`:
`TestNew_doesNotDialAndWarnsOnMissingToken`,
`TestNew_insecureSkipTLSVerifyWarns`, `TestNew_nilBlockDoesNotPanic`,
`TestWithDefaults_table`, `TestNode_table`,
`TestWrapErr_carriesProviderNameOpAndID`.
Verify: `grep -c '"proxmox"' internal/provider/config.go` is 3;
`go build ./... && go vet ./... && go test -race -count=1 ./internal/provider/... ./cmd/... && go test -short ./...`;
the fixture loader fails loudly on a missing file.
Done when: `go test -race ./internal/provider/proxmox/` passes with the
stubs in place and the constructor is registered. Commit: `proxmox:
config block, IDs, marks, state mapping, classify, registration`.
- **Phase 3 (implementer): extract `nocloud` and `keyedlock` from kvm2.**
Read first: `internal/provider/kvm2/{seed.go,seed_test.go,kvm2.go,kvm2_test.go,create.go}`;
contract §1.4, §2.1; spec entries 5 and 19;
`grep -n 'seed.go\|keyedLocks\|buildSeedISO' docs/*.md`. Create
`internal/provider/nocloud/seed.go` exporting `BuildSeedISO(name,
userData, vendorData string, w io.Writer) error` (plain errors, an
`ErrEmptyName` sentinel instead of kvm2's `permanentf`),
`BuildVendorData(configured string, installAgent bool) (string, error)`,
`GuestAgentScript`, `SeedLabel`; move the seed tests to
`nocloud/seed_test.go` under their existing names. Create
`internal/provider/keyedlock/keyedlock.go` exporting `Locks`, `New()
*Locks`, `(*Locks).Acquire(ctx, key) (release func(), err error)`; move
the `TestKeyedLocks_*` tests. Edit kvm2: delete `seed.go`, `seed_test.go`,
the `keyedLocks` type and its tests; `create.go` calls
`nocloud.BuildVendorData` and `nocloud.BuildSeedISO` and wraps their
errors with `permanentf`; `kvm2.go` uses `*keyedlock.Locks`. Edit
`internal/provider/proxmox/proxmox.go` to use `keyedlock.Locks` and delete
`locks.go`. Fix the doc mentions so they name the new packages.
Verify: `go test -race -count=1 ./internal/provider/...`; kvm2's test
count drops by exactly the moved tests and the two new packages carry
them; `grep -rn 'iso9660' internal/provider/kvm2/` is empty;
`go vet -tags=integration ./...` still compiles.
Done when: kvm2 and proxmox import both packages, all tests pass, the docs
name the new packages. Commit: `provider: extract nocloud seed builder
and keyedlock from kvm2`.
- **Phase 4 (implementer): seam, HTTP client, decorator, fake.** Read first:
spec entries 14, 17, 18, 20; contract §4.15, §7, §9;
`internal/tracing/transport.go`; `internal/provider/kvm2/{observe.go,observe_test.go,fake_test.go}`;
`internal/provider/proxmox/{api.go,errors.go}`; `testdata/*.json`. Create
`client.go`: `httpAPI{base *url.URL; http *http.Client; tokenFile
string}`; `newHTTPAPI(cfg provider.ProxmoxConfig, opts ...tracing.Option)
(*httpAPI, error)` builds a transport with `DialContext` timeout 10s,
`TLSHandshakeTimeout` 10s, `ResponseHeaderTimeout` from the request
timeout, `IdleConnTimeout` 90s, `MaxIdleConnsPerHost` 4,
`ForceAttemptHTTP2: false`, an empty `TLSNextProto` map (F18), `RootCAs`
from `caFile` or `InsecureSkipVerify` from the flag, wrapped with
`tracing.RestConfigWrapper(opts...)(transport)`; `http.Client.Timeout` is
the request timeout; `do(ctx, method, path string, query, form
url.Values, out any) error` reads the token file on every call (missing
file `transientf`, a file without `!` and `=` `permanentf`), sets the
header, decodes `{"data": ...}` into `out`, turns any non-2xx into
`*apiError` per F1 with the error body read capped at 64 KiB;
`UploadISO` streams a multipart body through `io.Pipe` with a goroutine
writing `content=iso`, `filename` and the file part. Create `observe.go`:
`observed{inner api; provider string}` writing one V(1) `proxmox api
call` line per seam call with `call`, `node`, `vmid`, `storage`, `upid`,
`duration`, `httpStatus` and `error` on failure, V(2) adding request
params and a response summary; `observe(inner api, providerName string)
api`; `New` wraps `httpAPI` in it. Create `fake_test.go`: `fakeAPI{mu; vms
map[int]*fakeVM; tasks map[string]*fakeTask; isos
map[string]map[string][]byte; hooks; block; calls; log; taskPolls int;
cloneFailsWith, startFailsWith string; agentNotRunning, cloneLeavesLock
bool}`, `fakeVM{node, name, status, qmpstatus, lock string; template
bool; config map[string]string; ifaces []agentInterface}`,
`fakeTask{node string; polls int; exit string; onDone func()}`; `Clone`
creates the VM with `lock=clone` and a task that clears the lock or, per
`cloneFailsWith`, removes the VM (F15); `Start` and `Stop` flip status
through a task; `Destroy` removes the VM through a task; `UploadISO`
records bytes; `DeleteVolume` errors with the F13 message when absent;
`enter(ctx, method)` for blocks and hooks like kvm2; helpers `addVM`,
`addTemplate`, `failWith`, `blockOn`, `callCount`, `callLog`,
`fakeErr(status int, msg string) *apiError`. Tests: `client_test.go` over
`httptest.NewTLSServer`: `TestClient_sendsTokenHeaderFromFileOnEveryRequest`
(rotate the file between two calls), `TestClient_missingTokenFileIsTransient`,
`TestClient_unparsableTokenFileIsPermanent`,
`TestClient_apiErrorFromStatusReasonAndBody` (serve each
`testdata/err_*.json` verbatim), `TestClient_decodesDataEnvelope`,
`TestClient_http1Only`, `TestClient_requestTimeoutIsTransient`,
`TestClient_cancelledContextReturnsPromptly`,
`TestClient_childSpanUnderParent`, `TestClient_noSpanWithoutParent`,
`TestClient_tokenNeverInErrorsOrLogs` (sentinel secret echoed by the
server in a 401 body), `TestClient_caFileTrustsPrivateCA`,
`TestClient_insecureSkipsVerification`, `TestClient_uploadStreamsMultipart`
(the server parses the multipart; the client never `io.ReadAll`s the
reader), `TestClient_destroyQueryFlags`. `observe_test.go`:
`TestObserve_everySeamMethodLogsOneLine` (reflection over `api`),
`TestObserve_detailOnlyAtV2`, `TestObserve_uploadBodyNeverLogged`,
`TestObserve_errorFieldsOnFailure`, `TestObserve_v0Silent`.
Verify: `go test -race -count=2 ./internal/provider/proxmox/`;
`grep -n 'ReadAll' internal/provider/proxmox/client.go` shows only the
bounded error-body read; `grep -n 'context.Background()' internal/provider/proxmox/*.go`
is empty outside tests.
Done when: the client passes its tests against the fixtures and the fake
compiles against the full `api` interface. Commit: `proxmox: thin HTTP
client, seam, log decorator and in-memory fake`.
- **Phase 5 (implementer): Get, ListByTag, task helpers.** Read first: spec
entries 7, 8, 9, 12, 13; contract §2.2, §2.4, §4.7, §4.12;
`internal/provider/kvm2/{get.go,get_test.go,list.go,list_test.go}`; the
resources, config, status, agent and task fixtures. Create `task.go`:
`parseUPID(upid) (node string, err error)` (Permanent when malformed),
`waitTask(ctx, log, api, upid, budget) error` polling `TaskStatus` every
`budgets.taskPoll` (2s; tests shrink it) until `status == "stopped"`,
`exitstatus != "OK"` giving `transientf` with the message, the budget
giving `transientf`, ctx giving Transient; `waitUnlocked(ctx, api, node,
vmid, budget)` polling `VMConfig` until no `lock`. Create `get.go`:
`resolve(ctx, api, vmid) (vmResource, bool, error)` (one resources call,
`Type == "qemu"` and the vmid), the name check, `VMConfig` for the mark,
`lock` Transient, `unknown` Transient, `running` reads the agent
(`isAgentNotRunning` gives no IP, other errors Transient), `mapState`.
Create `list.go` per spec entry 12; `Instance` carries ID, UID, CreatedAt
only. Replace the two stubs. Tests: `task_test.go` `TestParseUPID_table`,
`TestWaitTask_table` (ok, error exitstatus, budget exceeded, ctx
cancelled, poll count), `TestWaitUnlocked_table`. `get_test.go`:
`TestGet_missingVMIsNotFound`, `TestGet_nameMismatchIsNotFound`,
`TestGet_unmarkedIsNotFound`, `TestGet_runningWithoutIPIsProvisioning`,
`TestGet_runningWithIPIsRunning`, `TestGet_agentNotRunningIsProvisioning`,
`TestGet_agentOtherErrorIsTransient`, `TestGet_ipFilterSkipsLoopbackAndLinkLocal`,
`TestGet_stoppedIsStopped`, `TestGet_pausedIsStopped`,
`TestGet_unknownStatusIsTransient`, `TestGet_lockedIsTransientNeverNotFound`,
`TestGet_unknownStateIsStopped`, `TestGet_readsUIDAndCreatedAtFromBackend`,
`TestGet_unmanagedDescriptionHasEmptyUID`, `TestGet_malformedIDIsPermanent`,
`TestGet_resourcesFailureIsTransient`, `TestGet_twoVMsWithOurNameIsPermanent`.
`list_test.go`: `TestListByTag_excludesUnmarkedVMs`,
`TestListByTag_emptyUIDStillListed`, `TestListByTag_candidatesByTagOrNamePrefix`,
`TestListByTag_skipsLXCAndTemplates`, `TestListByTag_resourcesFailureFailsWhole`,
`TestListByTag_unreadableVMSkipped`, `TestListByTag_unreadableVMWithEndedContextFails`,
`TestListByTag_unknownStatusStillAttempted`, `TestListByTag_doesNotReadStatusOrAgent`,
`TestListByTag_idsAcceptedByParse`.
Verify: `go test -race -count=2 ./internal/provider/proxmox/`;
`grep -n 'not implemented' internal/provider/proxmox/*.go` lists only
Create and Delete.
Done when: Get and ListByTag are real and every listed test passes.
Commit: `proxmox: Get, ListByTag and task polling`.
- **Phase 6 (implementer): Create.** Read first: spec entries 2, 3, 4, 5, 6,
10, 19; contract §2.1, §4.2, §4.10, §4.11, §5;
`internal/provider/kvm2/{create.go,create_test.go}`;
`internal/provider/nocloud`; F4, F7, F10, F12, F15, F16, F19, F20 in the
Phase 0 log. Create `seed.go`: `seedISOName(name)` is `<name>-seed.iso`,
`seedVolID(storage, name)` is `<storage>:iso/<name>-seed.iso`,
`uploadSeed(ctx, log, api, node, storage, req, cfg)` builds vendor-data
and the ISO into a `bytes.Buffer` (tens of KiB), deletes an existing
volume tolerating absence, uploads with the buffer's length, waits the
task with `budgets.upload`, logs `cloudInitBytes` only. Create `create.go`
implementing the ordered steps: placement validation (`node`,
`MachineTypes`, `Templates`, all Permanent naming field and value), the
per-name lock, `ClusterResources` once for both the name lookup (adoption
guard, then `waitUnlocked` if locked, then continue from the seed step if
stopped, return if running) and the template's node (absent template is
Permanent); allocation loop `for attempt := 0; attempt < maxProbes (32)`
calling `Clone` with `vmidFor(req.UID, rng, attempt)`, `isAlreadyExists`
moving to the next attempt, exhaustion Transient; `waitTask` with
`budgets.clone`; `uploadSeed`; `SetVMConfig` with `cores`, `memory`,
`agent=1`, `tags=proxy_operator`, `ide2=<volid>,media=cdrom`, and `net0`
rewritten from the cloned config with `bridge=<cfg.Bridge>` only when
configured (MAC kept); `bootDisk(cfg)` and `Resize` when `diskGiB` is
above the current `size=`; `Start` and `waitTask` with `budgets.start`;
`waitRunning` on `VMStatus` with `budgets.running`; return the ID.
`isNotExist` inside Create is Transient (classify with `op == "create"`).
Tests `create_test.go`: `TestCreate_repeatReturnsSameIDAndCreatesNothing`,
`TestCreate_resumesAfterEachPartialStep` (after clone task, after seed
upload, after config, after resize, after start, after running),
`TestCreate_returnsOnlyWhenRunning`, `TestCreate_runningWaitCapIsTransient`,
`TestCreate_rejectsUnknownPlacement`, `TestCreate_zoneOptionalWithOneNode`,
`TestCreate_regionIgnored`, `TestCreate_adoptionWithForeignUIDIsPermanent`,
`TestCreate_adoptsOnlyMarkedVMs`, `TestCreate_duplicateNameIsPermanent`,
`TestCreate_vmidCollisionProbesForward`, `TestCreate_vmidRangeExhaustedIsTransient`,
`TestCreate_waitsForCloneLockOnRetry`, `TestCreate_cloneTaskFailureIsTransient`,
`TestCreate_cloneTaskTimeoutIsTransient`, `TestCreate_markWrittenInCloneCall`,
`TestCreate_tagSetInConfigStep`, `TestCreate_seedAlwaysRebuilt`,
`TestCreate_seedUploadedBeforeConfig`, `TestCreate_configSetsCoresMemoryAgentAndSeed`,
`TestCreate_bridgeOverrideKeepsMAC`, `TestCreate_noBridgeOverrideLeavesNet0`,
`TestCreate_resizeOnlyWhenLarger`, `TestBootDisk_table`,
`TestCreate_startTaskFailureIsTransient`, `TestCreate_templateMissingIsPermanent`,
`TestCreate_notExistInsideCreateIsTransientNotNotFound`,
`TestCreate_cloudInitSentinelNeverLoggedOrInErrors` (V(2) capture, every
error path), `TestCreate_concurrentSameNameSerialised`,
`TestCreate_cancelledMidWaitReturnsPromptly`.
Verify: `grep -n 'CloudInit' internal/provider/proxmox/create.go internal/provider/proxmox/seed.go`
shows only `len(req.CloudInit)` and the pass-through into
`nocloud.BuildSeedISO`; `go test -race -count=2 ./internal/provider/proxmox/`;
`grep -n 'not implemented'` lists only Delete.
Done when: Create is real and every listed test passes. Commit:
`proxmox: Create by cloning a template, seed upload, config, start`.
- **Phase 7 (implementer): Delete, Checker, CRD comment.** Read first: spec
entries 11, 16; contract §2.3, §4.3, §4.13, §8;
`internal/provider/kvm2/{delete.go,delete_test.go,check.go,check_test.go}`;
`internal/provider/check.go`; `api/v1alpha1/providerhealth_types.go`; F13,
F17 in the Phase 0 log. Create `delete.go`: parse the ID (Permanent), the
per-name lock, `resolve`; absent is a logged no-op; name mismatch or
unmarked is a logged no-op; `lock` Transient; `unknown` Transient;
`running` gets `Stop` and `waitTask` (`budgets.stop`); `Destroy` and
`waitTask` (`budgets.destroy`), `isNotExist` from `Destroy` being
success; `DeleteVolume` of the seed on the VM's node tolerating the F13
absent message; nil. Create `check.go`: `var _ provider.Checker =
(*Provider)(nil)`; `reasonTemplateNotFound = "TemplateNotFound"`,
`reasonStorageNotFound = "StorageNotFound"`; `Check` fans out one
goroutine per node in config order (`wg.Go`), each `checkNode` running
`Version`, `NodeStatus`, `StorageStatus` (a 500 "does not exist" is
`StorageNotFound`; a storage whose `content` lacks `iso` is
`StorageNotFound` with a message saying so), `ClusterResources` for the
templates; `checkReason(err)`: ctx ended `Timeout`, 401 `Unauthorized`,
403 `Forbidden`, other `*apiError` `APIError`, network `Unreachable`;
Details `pveVersion`, `kernel`, `isoStorage`, `isoStorageAvailGiB`,
`templates`; `Name` is the node, `Address` the base URL. Edit
`api/v1alpha1/providerhealth_types.go` (`kubernetes, gcp, kvm, kvm2 or
proxmox`) and run `make manifests`. Tests `delete_test.go`:
`TestDelete_malformedIDIsPermanent`, `TestDelete_twiceSucceeds`,
`TestDelete_absentIsNoOp`, `TestDelete_foreignNameIsNoOp`,
`TestDelete_unmarkedIsNoOp`, `TestDelete_runningVMStoppedFirst`,
`TestDelete_stoppedVMSkipsStop`, `TestDelete_lockedIsTransient`,
`TestDelete_unknownStatusIsTransient`, `TestDelete_removesSeedISO`,
`TestDelete_toleratesMissingISO`, `TestDelete_destroyTaskFailureIsTransient`,
`TestDelete_usesPurgeAndDestroyUnreferencedDisks`,
`TestDelete_concurrentSameNameSafe`, `TestDelete_noOpLinesAreSilentAtV0`;
in `get_test.go`: `TestGet_afterDeleteReturnsNotFound`,
`TestGet_duringDeleteStillReturnsInstance` (block the fake's `Destroy`).
`check_test.go`: `TestCheck_oneEntryPerNodeInConfigOrder`,
`TestCheck_nodesCheckedConcurrently`, `TestCheck_expiredContextReturnsPromptly`,
`TestCheck_reasonsTable`, `TestCheck_detailsKeys`, `TestCheck_logsVerdictPerNode`,
`TestCheckReason_table`.
Verify: `go test -race -count=2 ./internal/provider/proxmox/ ./internal/providerhealth/`;
`git diff --stat config/` shows only the CRD description line;
`grep -rn 'not implemented' internal/provider/proxmox/` is empty.
Done when: all four contract methods and `Check` are real. Commit:
`proxmox: Delete with seed cleanup, health Checker, CRD type comment`.
- **Phase 8 (implementer): contract sweep, observability sweep, integration
test, Makefile, testing.md.** Read first: contract §11 and §9; spec
entries 17, 18, 20, 21; `internal/provider/kvm2/{provider_test.go,kvm2_integration_test.go}`;
`internal/provider/tracing_test.go`; `Makefile` integration targets;
`docs/testing.md` kvm2 rows and section. Create `provider_test.go`: the
shared `fixture` (fake, log capture, budgets shrunk), `assertProviderError`,
`TestProvider_cancelledContextReturnsPromptly` (each method blocked on
its first seam call), `TestProvider_everyErrorIsWrappedWithConfiguredName`,
`TestLogging_v0SilentV1OutcomeV2Detail` (V(1) lines `proxmox vm ensured
running`, `proxmox vm fetched`, `proxmox vm deleted`, `proxmox vms
listed`, `proxmox node checked`, every line with `provider`, instance
lines with `vmid` and `name`; V(2) has `cloudInitBytes` and never the
text), `TestTracing_httpSpansUnderOperationSpan` (a real `httpAPI`
against an `httptest` server playing the fixtures for a Get, wrapped in
`provider.WithTracing`, with a span recorder: every HTTP span is a child
of `provider.get` and no attribute carries the token). Create
`proxmox_integration_test.go` (`//go:build integration`):
`loadIntegrationEnv` with the `PROXMOX_TEST_*` variables, `providerConfig()`,
the `integrationLog` pattern from kvm2; `TestProxmox_lifecycle` asserting
in order: `New` returns within 100 ms without a request; `Check` Ready
with `pveVersion`, `isoStorageAvailGiB` and `templates`; `Create` returns
`vms/<vmid>/<name>` with the vmid in range; a second `Create` returns the
same ID and the resources count is unchanged; `Get` reaches Running with
an IPv4 within the budget, UID and CreatedAt as written; `ListByTag`
contains it; a TCP connect to the guest's port is logged only; `Delete`
nil; `Get` NotFound; `Delete` again nil; the ISO storage has no
`<name>-seed.iso` and resources has no VM with the name; an HTTP span is
a child of `provider.create` and no attribute carries the cloud-init
text; `PROXMOX_TEST_KEEP` skips the teardown and prints the `qm destroy`
and ISO delete commands. `TestProxmoxCheck_realHost` creates nothing. Edit
`Makefile`: `test-integration-proxmox` (`go test -race -tags=integration
-count=1 -v -timeout 30m -run TestProxmox ./internal/provider/proxmox/`).
Edit `docs/testing.md`: rows for the package in the suites table and a
"The proxmox integration test" section. Update spec entry 21's tick list
with the final test names.
Verify: `go test -race -count=2 ./internal/provider/proxmox/`;
`go vet -tags=integration ./... && go vet -tags=e2e ./...`;
`make -n test-integration-proxmox`; every §11 item has a test name in
spec entry 21 that exists.
Done when: the package covers every §11 item by name, the integration
test compiles, the target exists. Commit: `proxmox: contract and
observability tests, integration test, testing docs`.
- **Phase 9 (implementer): deployment and documentation.** Read first:
contract §12 items 11 to 13; spec entries 22, 23, 24;
`config/manager/manager.yaml`, `config/manager/providers_config.yaml`,
`config/samples/{providers-config.yaml,proxy_kvm2.yaml,kustomization.yaml}`;
`Makefile` `kvm-ssh-secret`; `docs/kvm2-provider.md` headings;
`docs/providers-existing.md`; `docs/architecture.md` lines mentioning
kvm2; `README.md`; `docs/providers.md`; `CLAUDE.md` Gotchas. Edit
`config/manager/manager.yaml`: an optional Secret mount `proxmox-token` at
`/etc/proxy-operator/proxmox` (key `token`). Edit `Makefile`:
`PROXMOX_TOKEN_FILE ?= $(HOME)/.config/proxmox-token`,
`proxmox-token-secret` creating `proxmox-token` from `--from-file=token=`.
Edit both providers-config files: a commented `proxmox-home` entry (`url:
https://192.168.2.8:8006`, `tokenFile`, `caFile` or
`insecureSkipTLSVerify`, `nodes`, `isoStorage`, `storage`, `bridge`,
`vmidRange`, `templates`, `machineTypes`). Create
`config/samples/proxy_proxmox.yaml` (provider `proxmox-home`, zone the
node, machine type `small`, image the template key, the Squid cloud-init
from `proxy_kvm2.yaml`) and list it in `config/samples/kustomization.yaml`.
Create `docs/proxmox-provider.md` with sections: Proxmox prerequisites
(the template commands from this plan, no cloudinit drive on ide2, the
guest agent or outbound access, the bridge, the privilege list from F17,
reserving the VMID range), Cluster-side setup (the Secret, the config
entry), Config reference (spec entry 15's table), Placement (Region
ignored, catalogue-edit warning), What lands on Proxmox (VMID scheme,
description mark, tag, seed ISO, config keys, `qm unlock` for a stuck
clone), Logging, Tracing, Timeouts, Health check, Error classes, Known
gaps (spec entry 24), Testing. Edit `docs/providers-existing.md`: a fifth
column in the three tables and a proxmox section mirroring kvm2's. Edit
`docs/architecture.md`: components row, a proxmox block in §5, the span
note in §10, a §11 row (and the "All four providers" count), Decisions
bullets. Edit `README.md`, `docs/providers.md`, `CLAUDE.md` Gotchas
(`make test-integration-proxmox`).
Verify: `bin/kustomize build config/default >/dev/null && bin/kustomize build config/samples >/dev/null`;
a test `TestLoadConfig_proxmoxSampleEntry` loads the uncommented block
from `config/samples/providers-config.yaml` through `provider.LoadConfig`;
a link sweep over the edited docs finds no missing target;
`grep -c '—' docs/proxmox-provider.md` is 0.
Done when: manifests build, the sample loads, docs are linked and
link-clean. Commit: `proxmox: deployment files, sample, user guide and
doc rows`.
- **Phase 10 (orchestrator with the user): review, real-host validation,
changelog, MR.** Read first: `docs/plans-executions/2026-09-17-1515-kvm2-review-fixes.md`;
`.claude/agents/go-operator-reviewer.md`; the latest report under
`docs/reviews/` for the shape; the spec; this plan's Risks. Do: run
`go-operator-reviewer` on the branch with the spec and the contract as
inputs, then `/code-review high`, `make lint` (if the local toolchain
panics in staticcheck as it did during kvm2, record it and run
`bin/golangci-lint run ./internal/provider/proxmox/...` with staticcheck
disabled), `make test`. Fix accepted findings with regression tests in
one commit; record rejected ones under `docs/reviews/<date>-proxmox-provider-review/`.
Hand the user the host checklist (template ready, no leftover VMs in the
range, ISO storage has space, the token's privileges per F17) and the
`PROXMOX_TEST_*` export lines; with the go-ahead run
`make test-integration-proxmox` and record the clone duration and time to
IP. Optionally the in-cluster pass on kind: build and load the image,
`make proxmox-token-secret`, a scratch ConfigMap with the `proxmox-home`
entry, apply `proxy_proxmox.yaml`, wait for `Ready`, `kubectl get
providerhealth proxmox-home -o yaml`, a `curl -x` through the proxy,
delete and confirm the VM and ISO are gone; scratch overrides are not
committed. After the user confirms: CHANGELOG entry
(`date "+%Y-%m-%d %H:%M %Z"`), push with `-u` using the token helper from
memory (`~/.config/gitea-token`), open the MR with the Gitea API
(`POST /api/v1/repos/kacerr/egress-proxies-operator/pulls` with `head:
feat/proxmox-provider`, `base: main`, the body ending with the
attribution line; `pr.json` in the scratchpad, never committed). Never
merge.
Verify: the integration run's ordered assertions pass on 192.168.2.8;
`make test` green; the review report lists every finding as accepted or
rejected.
Done when: the MR URL is printed and the execution log's last box is
ticked. Commit: `proxmox: review fixes, real-host validation record,
changelog`.
## Verification
- Per phase: the `Verify:` clause, plus `go build ./... && go test -short ./...`
before every commit and `git diff --stat` reviewed for files outside the
phase's list.
- Contract mapping: after Phase 8 every rule of contract §2 and §8 has a
named test in the package, and spec entry 21 maps the §11 list 1:1 to
existing test names.
- Fixture discipline: every message `classify` or a matcher helper compares
against is loaded from `testdata/` in `TestClassify_table` or
`TestAPIError_matchers`; a string literal in `errors.go` without a fixture
is a review finding.
- Secrets: `TestClient_tokenNeverInErrorsOrLogs` and
`TestCreate_cloudInitSentinelNeverLoggedOrInErrors` run at V(2), and the
integration test asserts no span attribute carries the cloud-init text.
- End to end: the integration run (Phase 8's assertions) on 192.168.2.8 and
the optional kind pass in Phase 10.
## Risks
- **Error text location and exact wording (F1 to F6)** decide the whole
NotFound path; if messages vary by storage type or PVE minor version, the
matchers get one fixture per variant and the guide names the tested
version. This is not hypothetical: 9.2 rewrote the authentication-failure
message (F22), so fixtures captured from an 8.x host would already be
wrong. Every matched string comes from 192.168.2.8 at 9.2.20, and the
guide says so.
- **Clone cleanup (F15).** A failed or interrupted clone may leave a VM with
`lock=clone` that never clears; Create then waits out `budgets.clone` and
returns Transient every time. The guide documents `qm unlock`; the
reconciler keeps retrying rather than latching.
- **`SDN.Use` (F17).** PVE 9 refuses `net0` writes to non-root users without
it; a missing privilege is a 403 Permanent that latches the Proxy until
its spec changes. The guide's privilege list is the mitigation.
- **The ide2 seed and a template with a cloudinit drive** both claim
cloud-init; the Proxy's user-data would be ignored. Documented
prerequisite; the Checker cannot see it.
- **VMID range shared with humans.** A human VM inside the range costs a
probe; a human VM carrying our name is Permanent. The guide says to
reserve the range.
- **Node-local ISO storage after a migration** leaves the seed on the
original node; Delete tolerates its absence and the ISO is a documented
blind spot. Shared ISO storage avoids it.
- **Blocking Create** holds a reconcile worker for up to the clone budget;
the guide sets expectations.
- **The 30s request timeout on the seed upload** is fine for tens of KiB; a
multi-megabyte `vendorData` would trip it. Documented.
- **Two operator installs** on one Proxmox cluster delete each other's VMs
(contract §13); documented in entry 24.
- **kvm2 edits in Phase 3** touch a merged provider; the moved tests are the
guard, and `go vet -tags=integration` keeps its integration test compiling.
+260 -66
View File
@@ -1,17 +1,20 @@
# The existing providers
This is the appendix to [provider-contract.md](provider-contract.md): how the
four providers that exist today (kubernetes, gcp, kvm, kvm2) meet the
contract, which profile each one chose on every axis, and what is still
five providers that exist today (kubernetes, gcp, kvm, kvm2, proxmox) meet
the contract, which profile each one chose on every axis, and what is still
missing. It is reference material for whoever writes the next provider, not
a user guide; the user guides are [kvm-provider.md](kvm-provider.md),
[kvm2-provider.md](kvm2-provider.md),
[proxmox-provider.md](proxmox-provider.md),
[gcp-vm-validation.md](gcp-vm-validation.md) and
[gcp-in-specific-project.md](gcp-in-specific-project.md). The README's
"Implemented providers" table stays the short public summary. kvm2 is the
one provider built from the contract itself (clean-room, from
[kvm2-provider-spec.md](kvm2-provider-spec.md), without reading kvm's code),
so its column is also a record of what the contract produced.
"Implemented providers" table stays the short public summary. kvm2 and
proxmox are the two providers built from the contract itself (kvm2
clean-room from [kvm2-provider-spec.md](kvm2-provider-spec.md) without
reading kvm's code, proxmox from
[proxmox-provider-spec.md](proxmox-provider-spec.md)), so their columns are
also a record of what the contract produced.
The per-provider call flows are drawn in
[architecture.md §5](architecture.md#5-what-provider-calls-do-in-the-outside-world),
@@ -27,59 +30,59 @@ Code references are a file path plus a function or symbol name.
One row per axis of contract §4, in the same order, with the profile name
and the symbol that shows it.
| Axis | kubernetes | gcp | kvm | kvm2 |
|---|---|---|---|---|
| `readiness` | immediate: a Pod is scheduled within seconds (`stateFromPod`) | provisioning-window: minutes of boot (`mapState`) | provisioning-window: boot plus, on first use, an image download (`imageCache.ensure`) | provisioning-window: boot plus, on first use, an image download (`ensureImage`) |
| `create-return` | submit: returns after `client.Create` | submit: `Insert` is never awaited, `Operation.Wait` is not called | **blocking**: waits through `DomainCreate` (`ensureRunning`), and the first image fetch runs inside (`imageCache.fetch`) | **blocking**: waits through `DomainCreate` and a 15-second poll for running (`ensureRunning`, `waitRunning`), and the first image fetch runs inside (`fetchImage`) |
| `delete-return` | returns-early: `client.Delete`, Pod terminates on its own | returns-early: `Delete` submits an operation each time | waits: destroy, undefine and volume removal complete before return (`removeDomain`, `removeVolumes`) | waits: destroy, undefine and volume removal complete before return (`removeDomain`, `removeVolumes`) |
| `ip-source` | assigned-at-create: `PodIP` (`instanceFromPod`) | assigned-at-create: the external NAT IP of the first NIC (`toInstance`) | discovered: ARP table first, then the guest agent, first routable IPv4 (`address`, `firstIPv4`) | discovered: the guest agent first, then the ARP table, first routable IPv4 (`addressSources`, `address`, `firstIPv4`) |
| `gc-mark` | native-labels: Pod labels (`buildPod`) | native-labels: instance labels (`buildInsertRequest`) | written-metadata: a namespaced `<metadata>` element written in `DomainDefineXML` (`metadata.go`) | written-metadata: an element in kvm2's own namespace written in `DomainDefineXML` (`metadataURI`, `newInstanceMetadata`) |
| `created-at` | backend-timestamp: `CreationTimestamp` | backend-timestamp: `CreationTimestamp`, parse error gives zero | written-at-create: `createdAt` in the metadata element (`newInstanceMetadata`) | written-at-create: `createdAt` in the metadata element, parse error gives zero (`newInstanceMetadata`, `createdAt`) |
| `list-partial` | n/a: one call, all namespaces | omit-unreachable: `ReturnPartialSuccess` on `AggregatedList` | fail-whole per host, one unreadable domain skipped (`ListByTag`, `managedInstance`) | fail-whole per host, one unreadable domain skipped unless ctx ended; `ID`, `UID` and `CreatedAt` from the name and the mark only, no state or address RPC (`ListByTag`) |
| `quota` | absent: a quota 403 is Permanent (`classify`) | detected: 429, and 403 with reason `quotaExceeded` or `rateLimitExceeded` (`classify`, `hasReason`) | absent (`classify`) | detected: a pool-space precheck before any volume is created (`checkPoolSpace`, `neededBytes`) |
| `not-found` | status-404: `apierrors.IsNotFound` | status-404: `isNotFound` | error-code: `ErrNoDomain`, `ErrNoStorageVol` (`hasCode`) | error-code: `ErrNoDomain`, `ErrNoStorageVol`, Transient inside Create (`classifyLibvirt`) |
| `primary-key` | name-is-key: `namespace/name` | name-is-key: `zones/<zone>/instances/<name>` | name-is-key: `hosts/<host>/domains/<name>` | name-is-key: `hosts/<host>/domains/<name>`, with a UID guard on adoption (`create`) |
| `provisioning-state` | native: `PodPending` | native: `PROVISIONING`, `STAGING`, `REPAIRING` | create-blocks: a domain is defined and started in one `Create`; `nostate` maps to Provisioning | create-blocks: defined, started and polled to running in one `Create`; `nostate` maps to Provisioning (`mapState`) |
| `async-ops` | none | task-handle, but the handle is discarded: an insert that fails after acceptance is never read | none | none |
| `secondary-resources` | none | cascade: `AutoDelete` on the boot disk (`buildInsertRequest`) | explicit: overlay and seed removed by name (`removeVolumes`); base image kept | explicit: overlay and seed removed by name (`removeVolumes`); base image kept |
| `credentials` | workload-identity: in-cluster config or kubeconfig (`NewWithTransportWrapper`) | environment: Application Default Credentials | mounted-files: SSH key and known_hosts from the optional `kvm-ssh` Secret, read lazily (`newLazySSH`) | mounted-files: the same `kvm-ssh` Secret, read on every dial and never cached (`loadCredentials`) |
| `client-kind` | vendor-sdk: controller-runtime `client.Client`, no seam interface | vendor-sdk: `cloud.google.com/go/compute` behind `instancesAPI` | rpc-over-ssh: `go-libvirt` over an SSH-forwarded unix socket, behind `libvirtAPI` (`conn.go`) | rpc-over-ssh: `go-libvirt` over an SSH-forwarded unix socket, behind the ctx-first `backend` seam (`backend.go`) |
| `timeouts` | ctx-only: no `rest.Config.Timeout` is set | ctx-only: SDK defaults, with SDK retries on Get and List | per-call for the dial only (`defaultHandshakeTimeout` in `dialSSH`); RPCs are ctx-only and a cancelled `call` closes the connection | per-call: a budget around the dial, every RPC, the uploads and the download (`defaultBudgets`, `call`); a cancelled or overlong call kills the pooled connection |
| `logging` | none | tiered: V(1) outcomes and `logAPIError`, V(2) detail, V(5) wire log (`wirelog.go`) | tiered: V(1) outcomes and one `libvirt call` line per RPC (`trace.go`), V(2) XML and addresses; no wire tier | tiered: V(1) outcomes and one `libvirt call` line per RPC (`observe.go`), V(2) XML and addresses; no wire tier |
| `child-spans` | transport-wrapper: `tracing.RestConfigWrapper()` on its own `rest.Config`, parent-gated | sdk-instrumentation: the SDK adds `otelhttp` itself; not parent-gated, but every call site has a parent | **none**: `trace.go` is a log decorator, not spans | span-decorator: `kvm2.<Method>` per RPC (`observe`), plus `kvm2.dial` (`libvirtDialer`) and `kvm2.download` (`fetchImage`) |
| `backend-test` | cluster-e2e: kind (`test/e2e`) | manual-runbook: [gcp-vm-validation.md](gcp-vm-validation.md) | integration-tag: `kvm_integration_test.go`, `make test-integration-kvm` | integration-tag: `kvm2_integration_test.go`, `make test-integration-kvm2` |
| Axis | kubernetes | gcp | kvm | kvm2 | proxmox |
|---|---|---|---|---|---|
| `readiness` | immediate: a Pod is scheduled within seconds (`stateFromPod`) | provisioning-window: minutes of boot (`mapState`) | provisioning-window: boot plus, on first use, an image download (`imageCache.ensure`) | provisioning-window: boot plus, on first use, an image download (`ensureImage`) | provisioning-window: boot plus the guest agent's install at first boot, about 50 s to an address on the test host (`agentAddress`) |
| `create-return` | submit: returns after `client.Create` | submit: `Insert` is never awaited, `Operation.Wait` is not called | **blocking**: waits through `DomainCreate` (`ensureRunning`), and the first image fetch runs inside (`imageCache.fetch`) | **blocking**: waits through `DomainCreate` and a 15-second poll for running (`ensureRunning`, `waitRunning`), and the first image fetch runs inside (`fetchImage`) | **blocking**: waits through the clone, upload, resize and start tasks and a 60-second poll for running (`provision`, `waitRunning`) |
| `delete-return` | returns-early: `client.Delete`, Pod terminates on its own | returns-early: `Delete` submits an operation each time | waits: destroy, undefine and volume removal complete before return (`removeDomain`, `removeVolumes`) | waits: destroy, undefine and volume removal complete before return (`removeDomain`, `removeVolumes`) | waits: the stop, seed delete and destroy tasks complete before return (`destroy`, `deleteSeed`) |
| `ip-source` | assigned-at-create: `PodIP` (`instanceFromPod`) | assigned-at-create: the external NAT IP of the first NIC (`toInstance`) | discovered: ARP table first, then the guest agent, first routable IPv4 (`address`, `firstIPv4`) | discovered: the guest agent first, then the ARP table, first routable IPv4 (`addressSources`, `address`, `firstIPv4`) | discovered: the guest agent only, first routable IPv4 in the agent's order (`agentAddress`, `firstIPv4`) |
| `gc-mark` | native-labels: Pod labels (`buildPod`) | native-labels: instance labels (`buildInsertRequest`) | written-metadata: a namespaced `<metadata>` element written in `DomainDefineXML` (`metadata.go`) | written-metadata: an element in kvm2's own namespace written in `DomainDefineXML` (`metadataURI`, `newInstanceMetadata`) | written-metadata: three `key=value` lines in the VM's description, passed in the clone call itself (`formatDescription`, `parseDescription`) |
| `created-at` | backend-timestamp: `CreationTimestamp` | backend-timestamp: `CreationTimestamp`, parse error gives zero | written-at-create: `createdAt` in the metadata element (`newInstanceMetadata`) | written-at-create: `createdAt` in the metadata element, parse error gives zero (`newInstanceMetadata`, `createdAt`) | written-at-create: `proxy-operator-created-at` in the description, parse error gives zero (`parseDescription`) |
| `list-partial` | n/a: one call, all namespaces | omit-unreachable: `ReturnPartialSuccess` on `AggregatedList` | fail-whole per host, one unreadable domain skipped (`ListByTag`, `managedInstance`) | fail-whole per host, one unreadable domain skipped unless ctx ended; `ID`, `UID` and `CreatedAt` from the name and the mark only, no state or address RPC (`ListByTag`) | fail-whole for the one cluster-wide `/cluster/resources` call, one unreadable VM skipped unless ctx ended; `ID`, `UID` and `CreatedAt` from the description only, no status or agent call (`ListByTag`) |
| `quota` | absent: a quota 403 is Permanent (`classify`) | detected: 429, and 403 with reason `quotaExceeded` or `rateLimitExceeded` (`classify`, `hasReason`) | absent (`classify`) | detected: a pool-space precheck before any volume is created (`checkPoolSpace`, `neededBytes`) | absent: thin storage overprovisions, so no out-of-space answer exists to match; free space goes to the Checker's details (`classify`) |
| `not-found` | status-404: `apierrors.IsNotFound` | status-404: `isNotFound` | error-code: `ErrNoDomain`, `ErrNoStorageVol` (`hasCode`) | error-code: `ErrNoDomain`, `ErrNoStorageVol`, Transient inside Create (`classifyLibvirt`) | error-code: the 500 `Configuration file '…' does not exist`, plus list-and-miss on `/cluster/resources`; Transient inside Create (`isNotExist`, `classifyAPI`) |
| `primary-key` | name-is-key: `namespace/name` | name-is-key: `zones/<zone>/instances/<name>` | name-is-key: `hosts/<host>/domains/<name>` | name-is-key: `hosts/<host>/domains/<name>`, with a UID guard on adoption (`create`) | **allocated-id**: `vms/<vmid>/<name>`, the VMID from an FNV-1a hash of the UID with forward probing, each taken VMID put through an ownership check with a UID guard (`vmidFor`, `allocate`, `inspect`) |
| `provisioning-state` | native: `PodPending` | native: `PROVISIONING`, `STAGING`, `REPAIRING` | create-blocks: a domain is defined and started in one `Create`; `nostate` maps to Provisioning | create-blocks: defined, started and polled to running in one `Create`; `nostate` maps to Provisioning (`mapState`) | create-blocks: cloned, configured, started and polled to running in one `Create`; `qmpstatus: prelaunch` maps to Provisioning (`mapState`) |
| `async-ops` | none | task-handle, but the handle is discarded: an insert that fails after acceptance is never read | none | none | task-handle and backend-lock: every clone, upload, resize, start, stop, destroy and volume delete returns a UPID polled to its end within the call (`waitTask`); a VM's lock is waited on in Create and Transient in Get and Delete (`waitUnlocked`) |
| `secondary-resources` | none | cascade: `AutoDelete` on the boot disk (`buildInsertRequest`) | explicit: overlay and seed removed by name (`removeVolumes`); base image kept | explicit: overlay and seed removed by name (`removeVolumes`); base image kept | explicit: the seed ISO deleted by name (`deleteSeed`, `deleteSeedEverywhere`); the disks go with the destroy's `purge=1&destroy-unreferenced-disks=1` |
| `credentials` | workload-identity: in-cluster config or kubeconfig (`NewWithTransportWrapper`) | environment: Application Default Credentials | mounted-files: SSH key and known_hosts from the optional `kvm-ssh` Secret, read lazily (`newLazySSH`) | mounted-files: the same `kvm-ssh` Secret, read on every dial and never cached (`loadCredentials`) | mounted-files: the token from the optional `proxmox-token` Secret, read on every request (`readToken`); the CA from the same Secret, loaded once and cached (`httpClient`) |
| `client-kind` | vendor-sdk: controller-runtime `client.Client`, no seam interface | vendor-sdk: `cloud.google.com/go/compute` behind `instancesAPI` | rpc-over-ssh: `go-libvirt` over an SSH-forwarded unix socket, behind `libvirtAPI` (`conn.go`) | rpc-over-ssh: `go-libvirt` over an SSH-forwarded unix socket, behind the ctx-first `backend` seam (`backend.go`) | thin-client: `net/http` and `encoding/json` behind the ctx-first `api` seam, one method per endpoint (`api.go`, `httpAPI`) |
| `timeouts` | ctx-only: no `rest.Config.Timeout` is set | ctx-only: SDK defaults, with SDK retries on Get and List | per-call for the dial only (`defaultHandshakeTimeout` in `dialSSH`); RPCs are ctx-only and a cancelled `call` closes the connection | per-call: a budget around the dial, every RPC, the uploads and the download (`defaultBudgets`, `call`); a cancelled or overlong call kills the pooled connection | per-call: dial and TLS handshake 10 s, each request `requestTimeoutSeconds`, a budget per task wait (`defaultBudgets`); a cancelled request ends only itself |
| `logging` | none | tiered: V(1) outcomes and `logAPIError`, V(2) detail, V(5) wire log (`wirelog.go`) | tiered: V(1) outcomes and one `libvirt call` line per RPC (`trace.go`), V(2) XML and addresses; no wire tier | tiered: V(1) outcomes and one `libvirt call` line per RPC (`observe.go`), V(2) XML and addresses; no wire tier | tiered: V(1) outcomes and one `proxmox api call` line per request (`observe.go`), V(2) parameters, config summaries and addresses; no wire tier |
| `child-spans` | transport-wrapper: `tracing.RestConfigWrapper()` on its own `rest.Config`, parent-gated | sdk-instrumentation: the SDK adds `otelhttp` itself; not parent-gated, but every call site has a parent | **none**: `trace.go` is a log decorator, not spans | span-decorator: `kvm2.<Method>` per RPC (`observe`), plus `kvm2.dial` (`libvirtDialer`) and `kvm2.download` (`fetchImage`) | transport-wrapper: `tracing.RestConfigWrapper()` on the client's transport, parent-gated, one `HTTP <METHOD>` span per request (`newHTTPAPI`) |
| `backend-test` | cluster-e2e: kind (`test/e2e`) | manual-runbook: [gcp-vm-validation.md](gcp-vm-validation.md) | integration-tag: `kvm_integration_test.go`, `make test-integration-kvm` | integration-tag: `kvm2_integration_test.go`, `make test-integration-kvm2` | integration-tag: `proxmox_integration_test.go`, `make test-integration-proxmox`; passed on a PVE 9.2.20 host |
## 2. Other differences
Rows that are not profile axes but that a new provider author compares
against.
| Feature | kubernetes | gcp | kvm | kvm2 |
|---|---|---|---|---|
| Create verifies the adopted instance's UID | no | no | no | **yes**: a domain without the kvm2 mark or with a different non-empty UID is Permanent; a marked one with a matching or empty UID is adopted (`create`) |
| Placement used | none | Zone, MachineType, Image | Zone (host), MachineType, Image | Zone (host), MachineType, Image |
| CloudInit delivered | **no**: ignored | yes: `user-data` metadata | yes: NoCloud seed ISO built in-process (`buildSeedISO`) | yes: NoCloud seed ISO built in-process (`buildSeedISO`) |
| Port used | yes: Squid config and containerPort (`squidConf`) | no: firewall is manual | no | no |
| States produced | Provisioning, Running, Terminated | all four | Provisioning, Running, Stopped | Provisioning, Running, Stopped |
| Deleting instance still reports its state | yes: a Pod being deleted reports its phase | n/a | n/a | n/a: Delete waits, and a concurrent Get sees the domain until it is undefined |
| Error.Provider is the configured name | **no**: hardcoded `"kubernetes"` | yes | yes | yes (`wrapErr`) |
| Checker target | `apiserver` | the project | each host, concurrently | each host, concurrently, on a fresh connection |
| Checker Details | none | none | `user`, `pool`, `hostname`, `libvirtVersion`, `poolState`, `poolCapacity`, `poolAvailable` | `user`, `pool`, `hostname`, `libvirtVersion`, `poolState`, `poolCapacityGiB`, `poolAvailableGiB`, filled in as far as the check got |
| Credential rotation without restart | n/a | via ADC | no: cached after first successful load | yes: both files read on every dial (`loadCredentials`) |
| Connection handling | client-go | SDK client, never closed | one pooled SSH plus libvirt connection per host, 15s keepalive (`hostPool`) | one pooled SSH plus libvirt connection per host, 15s keepalive (`hostTable`); dedicated connections for the image upload and the check (`dialFresh`) |
| Unit-test double | controller-runtime fake client | `fakeAPI` behind `instancesAPI`, plus `httptest` for the real SDK client in `check_test.go` | `fakeAPI` behind `libvirtAPI`, plus an in-process `x/crypto/ssh` server in `conn_test.go` | `fakeBackend` behind `backend` (`fake_test.go`), plus an in-process `x/crypto/ssh` server (`sshserver_test.go`) and go-libvirt's own mock for the adapter |
| Proxy software | generated Squid config (`pod.go`) | from `cloudInit`; the image must run cloud-init | from `cloudInit`, plus a vendor-data script that installs the guest agent (`seed.go`) | from `cloudInit`, plus a vendor-data script that installs the guest agent (`seed.go`) |
| IP reachable from the operator | Pod network, so in-cluster only (not from `make run-dev` on a laptop) | external IP, once a firewall rule allows the cluster's egress IP | LAN address on the host's bridge; the operator needs a route to that LAN | LAN address on the host's bridge; the operator needs a route to that LAN |
| Distinct egress IP per proxy | no: the cluster's egress | yes: ephemeral external IP | whatever the host's LAN routes through; not per VM by itself | whatever the host's LAN routes through; not per VM by itself |
| Feature | kubernetes | gcp | kvm | kvm2 | proxmox |
|---|---|---|---|---|---|
| Create verifies the adopted instance's UID | no | no | no | **yes**: a domain without the kvm2 mark or with a different non-empty UID is Permanent; a marked one with a matching or empty UID is adopted (`create`) | **yes**: a VM with the proxy's name and no mark, or with a different non-empty UID, is Permanent; a marked one with a matching or empty UID is adopted (`inspect`) |
| Placement used | none | Zone, MachineType, Image | Zone (host), MachineType, Image | Zone (host), MachineType, Image | Zone (node), MachineType, Image (a template key) |
| CloudInit delivered | **no**: ignored | yes: `user-data` metadata | yes: NoCloud seed ISO built in-process (`buildSeedISO`) | yes: NoCloud seed ISO built in-process (`nocloud.BuildSeedISO`) | yes: NoCloud seed ISO built in-process (`nocloud.BuildSeedISO`) and uploaded through the API (`uploadSeed`) |
| Port used | yes: Squid config and containerPort (`squidConf`) | no: firewall is manual | no | no | no |
| States produced | Provisioning, Running, Terminated | all four | Provisioning, Running, Stopped | Provisioning, Running, Stopped | Provisioning, Running, Stopped |
| Deleting instance still reports its state | yes: a Pod being deleted reports its phase | n/a | n/a | n/a: Delete waits, and a concurrent Get sees the domain until it is undefined | n/a: Delete waits, and a concurrent Get sees the VM (Stopped once stopped) until the destroy removes its config |
| Error.Provider is the configured name | **no**: hardcoded `"kubernetes"` | yes | yes | yes (`wrapErr`) | yes (`wrapErr`) |
| Checker target | `apiserver` | the project | each host, concurrently | each host, concurrently, on a fresh connection | each node, concurrently, on the shared HTTP client |
| Checker Details | none | none | `user`, `pool`, `hostname`, `libvirtVersion`, `poolState`, `poolCapacity`, `poolAvailable` | `user`, `pool`, `hostname`, `libvirtVersion`, `poolState`, `poolCapacityGiB`, `poolAvailableGiB`, filled in as far as the check got | `isoStorage`, `tokenID`, `pveVersion`, `kernel`, `isoStorageAvailGiB`, `templates`, filled in as far as the check got |
| Credential rotation without restart | n/a | via ADC | no: cached after first successful load | yes: both files read on every dial (`loadCredentials`) | the token yes, read on every request (`readToken`); the CA no, cached after first use |
| Connection handling | client-go | SDK client, never closed | one pooled SSH plus libvirt connection per host, 15s keepalive (`hostPool`) | one pooled SSH plus libvirt connection per host, 15s keepalive (`hostTable`); dedicated connections for the image upload and the check (`dialFresh`) | one `net/http` client, 4 idle connections per host, HTTP/1.1 only; a cancelled request ends only itself |
| Unit-test double | controller-runtime fake client | `fakeAPI` behind `instancesAPI`, plus `httptest` for the real SDK client in `check_test.go` | `fakeAPI` behind `libvirtAPI`, plus an in-process `x/crypto/ssh` server in `conn_test.go` | `fakeBackend` behind `backend` (`fake_test.go`), plus an in-process `x/crypto/ssh` server (`sshserver_test.go`) and go-libvirt's own mock for the adapter | `fakeAPI` behind `api` (`fake_test.go`), plus `httptest.NewTLSServer` replaying the Phase 0 captures for the real client (`client_test.go`) |
| Proxy software | generated Squid config (`pod.go`) | from `cloudInit`; the image must run cloud-init | from `cloudInit`, plus a vendor-data script that installs the guest agent (`seed.go`) | from `cloudInit`, plus a vendor-data script that installs the guest agent (`nocloud.GuestAgentScript`) | from `cloudInit`, plus a vendor-data script that installs the guest agent (`nocloud.GuestAgentScript`) |
| IP reachable from the operator | Pod network, so in-cluster only (not from `make run-dev` on a laptop) | external IP, once a firewall rule allows the cluster's egress IP | LAN address on the host's bridge; the operator needs a route to that LAN | LAN address on the host's bridge; the operator needs a route to that LAN | address on the proxy network (bridge or VLAN from the config); the operator needs a route to it |
| Distinct egress IP per proxy | no: the cluster's egress | yes: ephemeral external IP | whatever the host's LAN routes through; not per VM by itself | whatever the host's LAN routes through; not per VM by itself | whatever the proxy network routes through; not per VM by itself |
## 3. Observability status
| Layer | kubernetes | gcp | kvm | kvm2 |
|---|---|---|---|---|
| 1. Operation spans and metrics | yes: decorators | yes: decorators | yes: decorators | yes: decorators |
| 2. Child spans per backend call | yes: `tracing.RestConfigWrapper()`, parent-gated (`cmd/main.go`, `NewWithTransportWrapper`) | yes: the SDK's own `otelhttp`; not parent-gated | **no** | yes: `kvm2.<Method>` from the `observe` decorator, `kvm2.dial`, `kvm2.download`; always under the caller's span, since every call carries the caller's ctx |
| 3. Logs carry `traceID` and `spanID` | n/a: the package does not log | yes, including the V(5) wire log | yes: every `libvirt call` line at V(1) | yes: every `libvirt call` line at V(1), written inside its RPC's span |
| Tested against a real collector | yes: the kind e2e checks `provider.create` and `provider.delete` in Tempo (`test/e2e/tracing_test.go`), operation spans only | no | no | no; the integration test asserts the parent-child topology in an in-memory recorder (`assertLifecycleSpans`) |
| Layer | kubernetes | gcp | kvm | kvm2 | proxmox |
|---|---|---|---|---|---|
| 1. Operation spans and metrics | yes: decorators | yes: decorators | yes: decorators | yes: decorators | yes: decorators |
| 2. Child spans per backend call | yes: `tracing.RestConfigWrapper()`, parent-gated (`cmd/main.go`, `NewWithTransportWrapper`) | yes: the SDK's own `otelhttp`; not parent-gated | **no** | yes: `kvm2.<Method>` from the `observe` decorator, `kvm2.dial`, `kvm2.download`; always under the caller's span, since every call carries the caller's ctx | yes: `tracing.RestConfigWrapper()` on the transport, parent-gated, one `HTTP <METHOD>` per request |
| 3. Logs carry `traceID` and `spanID` | n/a: the package does not log | yes, including the V(5) wire log | yes: every `libvirt call` line at V(1) | yes: every `libvirt call` line at V(1), written inside its RPC's span | yes: every `proxmox api call` line at V(1), with the operation span's `spanID` |
| Tested against a real collector | yes: the kind e2e checks `provider.create` and `provider.delete` in Tempo (`test/e2e/tracing_test.go`), operation spans only | no | no | no; the integration test asserts the parent-child topology in an in-memory recorder (`assertLifecycleSpans`) | no; `TestTracing_httpSpansUnderOperationSpan` asserts the parent-child topology in an in-memory recorder |
## 4. kubernetes (`internal/provider/kubernetes`)
@@ -347,8 +350,10 @@ same libvirt.
production adapter and the budgets), `ssh.go` (credentials, the forwarded
socket, keepalive), `hosts.go` (the pooled table and the production
dialer), `observe.go` (the span and log decorator), `image.go` (cache,
download, quota precheck), `seed.go`, `xml.go`, `volume.go`,
`metadata.go`, `id.go`, `state.go`, `errors.go`, `check.go`. The seam is
download, quota precheck), `xml.go`, `volume.go`, `metadata.go`,
`id.go`, `state.go`, `errors.go`, `check.go`; the seed ISO comes from
the shared `internal/provider/nocloud` and both lock sets are
`internal/provider/keyedlock`. The seam is
`backend`: the twenty go-libvirt calls the provider uses, each with `ctx`
first (go-libvirt's own methods take none and block), plus `Close`.
`libvirtBackend` adapts `*libvirt.Libvirt` and runs every call through
@@ -526,20 +531,209 @@ same libvirt.
its backing image or size.
- The sample cloud-init installs Squid with `http_access allow all`.
## 8. Gaps shared by the existing providers
## 8. proxmox (`internal/provider/proxmox`)
- **Adoption is not verified in three of the four.** kubernetes, gcp and
The full user guide is [proxmox-provider.md](proxmox-provider.md) and the
design it was built from is
[proxmox-provider-spec.md](proxmox-provider-spec.md); this section is the
author's view. It is the second provider built from the contract, and the
first on a REST API of the hypervisor's own: no SSH, no libvirt. Every
error text it matches was captured from a PVE 9.2.20 host before the code
was written.
- **Package and seam.** `proxmox.go` (`Provider`, `New`, `withDefaults`,
`node`, `wrapErr`, the per-name lock, the budgets), `create.go`,
`get.go`, `delete.go`, `list.go` (the four methods), `api.go` (the
seam), `client.go` (the production adapter `httpAPI`, the token and CA
loading), `observe.go` (the logging decorator), `task.go` (task, lock
and running waits), `seed.go`, `mark.go`, `id.go`, `state.go`,
`errors.go`, `check.go`; the seed ISO comes from the shared
`internal/provider/nocloud` and the name lock from
`internal/provider/keyedlock`. The seam is `api`: one method per
endpoint, sixteen in all, each with `ctx` first. `New` wraps `httpAPI` in
`observe` once, so provider code never holds an undecorated seam;
`newProvider` is the test constructor.
- **Config.** `ProviderConfig.Proxmox`, the `ProxmoxConfig` type in
`internal/provider/config_proxmox.go`, validated by `validateProxmox`
(the URL shape, node names and storage ids by regexp because they end
up in request paths, `vlanTag`, `vmidRange`, the two catalogues).
`isoStorage` has no default on purpose: the token needs
`Datastore.Allocate` there. Defaults (`vmidRange` 100000 to 199999,
`requestTimeoutSeconds` 30, `installGuestAgent` true) are filled by
`withDefaults`. `New` never makes a request; it warns when the token
file does not exist, or when `insecureSkipTLSVerify` is on.
- **ID.** `formatProviderID(vmid, name)` gives `vms/<vmid>/<name>`, with no
node, because VMs migrate; the node is resolved from
`/cluster/resources` on every call (`resolve`). `parseProviderID`
accepts exactly three segments, the literal `vms`, a decimal VMID
without leading zeros between 100 and 999999999 and a non-empty name,
else `ErrPermanent`. The name is there so `Get` and `Delete` can tell
our VM from another that later got the same VMID.
- **Create.** `placement` checks `Zone` (`node`; optional with one node),
`MachineType` and `Image` (both catalogue keys) before any request, and
logs the resolved parameters with `cloudInitBytes` at V(2). Then, under
the per-name lock, `create`: one `ClusterResources` read;
`adoptByName` (two VMs with the name are Permanent, one goes to
`inspect`); with none, `template` reads the template's config and
refuses a non-template, a cloud-init drive, a missing `net0` when the
config rewrites it, and an unsized boot disk when `diskGiB` is set, all
before any clone; `allocate` probes `vmidFor(uid, attempt)` up to 32
times, cloning onto a free VMID and running `inspect` on a taken one or
on the clone's `config file already exists` answer. `inspect` waits out
a lock (`waitUnlocked`; a clone in progress is also nameless), then
adopts our name with our mark and a matching or empty UID, probes past
another name, and is Permanent for our name without the mark, with
another UID, or hibernated. An adopted VM gets one `VMStatus`: running
returns at once. Otherwise `provision`: `uploadSeed` (rebuilt and
overwritten every attempt), `SetVMConfig` (`configParams`: cores,
memory, `agent=1`, the tag, `ide2`, and `net0` through `rewriteNet0`
when a bridge or VLAN is configured), the resize task when
`resizeTarget` says the boot disk is smaller than `diskGiB`, the start
task, and `waitRunning` (every second up to 60 s, Transient on expiry
or on a stop). `Create` returns only when the VM runs.
- **Get and state.** `get` checks in order: absent from the resources is
NotFound; a lock other than `suspended` is Transient (checked before the
name, because a mid-clone entry has none); `status: unknown` is
Transient; the config's `does not exist` is NotFound, a config lock
Transient; another name or no mark is NotFound; hibernated is Stopped
without further calls. Then `VMStatus` and `mapState(status,
qmpstatus, ip)`:
| `status` | `qmpstatus` | agent | Provider state |
|---|---|---|---|
| running | running | an IPv4 | Running |
| running | running | 200 without an IPv4, or `QEMU guest agent is not running` | Provisioning |
| running | running | `No QEMU guest agent configured` | Stopped |
| running | prelaunch | not asked | Provisioning |
| running | paused, suspended, io-error, anything else | not asked | Stopped |
| stopped, anything else | any | not asked | Stopped |
Any other agent answer, including `VM <vmid> is not running`, is
Transient. `UID` and `CreatedAt` are read from the description on every
call. A running `Get` is four calls: resources, config, status, agent.
- **Delete.** Under the per-name lock, `destroy`: `resolve`; a lock other
than `suspended` or `status: unknown` is Transient; the config's `does
not exist`, another name or no mark leave the VM alone; our VM
hibernated is destroyed like a stopped one, since the destroy takes it.
Otherwise stop (unless stopped), `deleteSeed` on the
VM's node, where a failure fails the call and leaves the VM for the
next Delete (a 403 for `Datastore.Allocate` is Permanent), then destroy
with `purge=1&destroy-unreferenced-disks=1`. Stop and destroy are tasks
waited on through `runTask`, where `does not exist` means gone. The
seed goes first because a VM already destroyed is NotFound to `Get`,
and nothing would call Delete again. With no VM of ours,
`deleteSeedEverywhere` tries every configured node, where a node that
fails is logged and skipped unless ctx ended or the failure is
Permanent. A missing volume still ends in a successful task, so no
message is matched. Nothing is left when nil is returned.
- **ListByTag.** One `ClusterResources` covers every node, unreachable
ones included (their guests are listed with `status: unknown`).
`isCandidate` keeps QEMU VMs that are not templates and carry the tag
`proxy_operator` or a name starting `proxy-`; a nameless entry (a clone
in progress) is not a candidate. One `VMConfig` per candidate; the mark
in the description decides, and the `Instance` is `ID`, `UID` and
`CreatedAt` only. A candidate whose config cannot be read is skipped at
V(1), unless ctx ended, which fails the call; the resources call failing
fails the whole call.
- **Errors.** `classify(op, err)`: an error already carrying a class keeps
it; then `apiError`s by `classifyAPI` (every 4xx Permanent; the 500
`does not exist` NotFound, Transient when `op == "create"`; every other
status Transient, including the lock, the taken VMID and pveproxy's 595
and 596); TLS verification failures (`tls.CertificateVerificationError`
and the three `x509` types) Permanent; everything else Transient. The
500 matchers (`isNotExist`, `isAlreadyExists`, the two agent ones)
are anchored to the 500 status and each is backed by a captured
response. `permanentf` and `transientf` classify at the
source: placement, the template checks, the ownership check, a
malformed token file or CA file (Permanent); a missing token or CA file,
an expired wait, a failed task (Transient). Nothing is Quota.
- **Check.** `Check` runs `checkNode` per node concurrently on the shared
client. `probeNode` reads the token file for `tokenID` (no request when
it is unusable), then `Version`, `NodeStatus`, `checkStorage` (a 500
whatever its text, inactive, or no `iso` content is `StorageNotFound`)
and `checkTemplates` (each catalogue VMID exists and is a template,
else `TemplateNotFound`). `checkNode` answers `Timeout` whenever the
round's ctx ended; otherwise `checkReason`: the two failures above,
`errCredentials` `CredentialsUnavailable`, `errCAFile` or a TLS
verification failure `CertificateInvalid`, 401 `Unauthorized`, 403
`Forbidden`, 595 and 596 `Unreachable`, any other status `APIError`, a
`net.OpError` `Unreachable` (before the deadline, so a dial out of its
own budget is a node that did not answer), a deadline or cancellation
`Timeout`, a cut connection `Unreachable`.
- **Connections.** One `http.Client`, built on first use once the CA
loads (`httpClient`), shared by every call and the Checker: dial and
TLS handshake 10 s each, `Client.Timeout` `requestTimeoutSeconds`, 4
idle connections per host kept 90 s, HTTP/2 disabled. A cancelled
request ends only itself. The token file is read by `readToken` on
every request.
- **Logging.** Logger from the context, named `proxmox`, with `provider`,
and `vmid`, `name` and `node` on lines about an instance. V(1): one line
per operation outcome (`proxmox vm ensured running`, `proxmox vm
fetched`, `proxmox vm deleted` with `destroyed`, `proxmox vms listed`,
`proxmox node checked`), one `proxmox api call` line per seam call from
`observe` (call, identifiers, UPID, duration, `httpStatus` and `error`),
`proxmox task failed`, and the no-op lines. V(2): request parameters,
config summaries, agent addresses, the resolved Create parameters.
Cloud-init, the token, the CA and the upload body are never logged.
- **Tracing.** No span of its own: `newHTTPAPI` wraps the transport in
`tracing.RestConfigWrapper(opts...)`, so each request is an
`HTTP <METHOD>` client span under the caller's span, and none without
one. `observe` only logs, so nothing is counted twice.
- **Tests.** Pure pieces (`id_test.go`, `mark_test.go`, `state_test.go`,
`seed_test.go`, `errors_test.go` with the classify table over the
captures, `api_test.go`); `client_test.go` (the real client over
`httptest.NewTLSServer`: the token never in errors or logs, x509
failures Permanent, cancellation, the child span); `create_test.go`,
`get_test.go`, `delete_test.go`, `list_test.go`, `task_test.go`,
`proxmox_test.go`, `check_test.go`, `observe_test.go` (every seam
method logs exactly one line, by reflection) and `provider_test.go`
(the contract §11 sweep, named test by test in the spec's entry 21).
The double is `fakeAPI` in `fake_test.go`: VMs, tasks and ISOs in
memory, a clone that leaves a nameless locked entry until its task
ends, per-method errors and blocking, a call log; its error texts are
checked against the captures. Real backend:
`proxmox_integration_test.go` behind `//go:build integration`, driven
by `PROXMOX_TEST_*` variables and `make test-integration-proxmox`,
described in
[testing.md](testing.md#the-proxmox-integration-test-real-proxmox-ve-host);
it passed against PVE 9.2.20 on 2026-09-24.
- **Deployment.** The optional `proxmox-token` Secret (keys `token` and
`ca.pem`) mounted at `/etc/proxy-operator/proxmox`
(`config/manager/manager.yaml`), `make proxmox-token-secret`, a
commented `proxmox-home` entry in both providers configs (loaded by
`TestLoadConfig_proxmoxSampleEntry`), `config/samples/proxy_proxmox.yaml`,
the `proxmox` type in the `status.type` comment of
`api/v1alpha1/providerhealth_types.go`. No RBAC, and no
`hack/providers-dev.yaml` entry, since it needs credentials.
- **Known gaps** (see also [proxmox-provider.md § Known gaps](proxmox-provider.md#known-gaps)).
- `Create` blocks until the VM runs.
- A clone lock that never clears makes every Create for that Proxy wait
out the 10-minute clone budget; `qm unlock` by hand.
- GC blind spot: a seed ISO whose VM was destroyed outside the operator
is invisible to GC, and a seed on node-local storage stays behind
when its VM migrated.
- The VMID range is shared with humans; a hand-made VM with a proxy's
name is Permanent.
- Cross-node clones (`target`) need shared storage and are untested.
- A hibernated VM of ours latches its Proxy when Create's adoption
finds it; a recorded one is replaced.
- The CA is cached after first use; the token is not.
## 9. Gaps shared by the existing providers
- **Adoption is not verified in three of the five.** kubernetes, gcp and
kvm adopt any existing instance with the right name without checking
`LabelUID`; kvm2 returns `ErrPermanent` when the UID differs.
- **The Create comment does not match kvm and kvm2.** The interface comment
in `internal/provider/provider.go` says `Create` returns as soon as the
request is submitted; both libvirt providers block until the domain has
started.
`LabelUID`; kvm2 and proxmox return `ErrPermanent` when the UID differs.
- **The Create comment does not match kvm, kvm2 and proxmox.** The
interface comment in `internal/provider/provider.go` says `Create`
returns as soon as the request is submitted; the three VM providers on
your own hypervisors block until the VM has started.
- **`Stopped` and `Terminated` are produced inconsistently.** kubernetes
never produces Stopped, kvm and kvm2 never produce Terminated, and the
reconciler treats them the same.
never produces Stopped, kvm, kvm2 and proxmox never produce Terminated,
and the reconciler treats them the same.
- **Different `ListByTag` partial-failure policies.** gcp leaves failed
parts out, kvm and kvm2 fail the whole list. Both are safe.
parts out, kvm and kvm2 fail the whole list, and proxmox fails it when
its one cluster-wide listing fails. Both are safe.
- **Stale `mock` comments** in `internal/provider/provider.go`,
`internal/provider/registry/registry.go` and `api/v1alpha1/proxy_types.go`
(and so the generated CRD's `spec.provider` description).
@@ -553,10 +747,10 @@ same libvirt.
log lines. kvm2 has them, but only an in-memory recorder checks their
topology. Only kubernetes is tested against a real collector, and that
test checks only the operation spans; nothing checks that gcp's SDK spans,
the kubernetes API spans or kvm2's spans nest under `provider.*` in a
real collector.
the kubernetes API spans, kvm2's spans or proxmox's HTTP spans nest under
`provider.*` in a real collector.
## 9. Proposed features
## 10. Proposed features
Listed only. Nothing below is designed or implemented, except where a
bullet says kvm2 already does it.
@@ -574,7 +768,7 @@ bullet says kvm2 already does it.
backend, so a new provider gets the contract tests for free.
- **Verify the UID when `Create` adopts an instance.** Return `ErrPermanent`
on a mismatch instead of adopting an instance that belongs to someone else.
kvm2 does this; the other three do not.
kvm2 and proxmox do this; the other three do not.
- **Report asynchronous create failures:** gcp keeps the operation name, and
`Get` reports the failure instead of NotFound.
- **Capacity or quota signals** for kvm, such as pool free space and host
+2 -1
View File
@@ -14,12 +14,13 @@ the existing ones happen to work.
open. Copy it to `docs/<type>-provider-spec.md`, fill it in, and hand it to
the implementer with the other two files.
- **[providers-existing.md](providers-existing.md)**: how kubernetes, gcp,
kvm and kvm2 meet the contract, a table mapping each of them to every
kvm, kvm2 and proxmox meet the contract, a table mapping each of them to every
profile, their per-provider reference notes, shared gaps and proposed
features.
Per-provider user guides: [kvm-provider.md](kvm-provider.md),
[kvm2-provider.md](kvm2-provider.md),
[proxmox-provider.md](proxmox-provider.md),
[gcp-vm-validation.md](gcp-vm-validation.md) and
[gcp-in-specific-project.md](gcp-in-specific-project.md).
+961
View File
@@ -0,0 +1,961 @@
# Provider spec: proxmox
Filled from [provider-spec-template.md](provider-spec-template.md) against
[provider-contract.md](provider-contract.md). This is the design the
`proxmox` package is built from. Where an entry names an axis in backticks,
the answer is one of that axis's profiles in contract §4.
Backend facts come from the Phase 0 recon on the user's host, recorded in
[the execution log](plans-executions/2026-09-18-1502-proxmox-provider.md) as
facts F1 to F23, with the raw responses under
`internal/provider/proxmox/testdata/`. An Evidence line citing `F<n>` points
there. Where this spec and the plan
([docs/plans/2026-09-18-1502-proxmox-provider.md](plans/2026-09-18-1502-proxmox-provider.md))
disagree, this spec wins: it folds in the ten corrections Phase 0 found.
## Header
- **Type**: `proxmox`
- **Example configured name**: `proxmox-home`
- **Backend and API version this spec was written against**: Proxmox VE
9.2.20 (`pve-manager/9.2.20/49318c671b82f31e`, kernel `7.0.14-17-pve`,
QEMU 11.0.3) on the single node `pv1-1`, API at
`https://192.168.0.8:8006/api2/json`.
- **Client**: `thin-client`: `net/http` and `encoding/json` from the standard
library, with the transport wrapped by the already-required
`go.opentelemetry.io/contrib/instrumentation/net/http/otelhttp`. No new
dependency. The seed ISO uses `github.com/kdomanski/iso9660`, already
required by kvm2.
- **Author and date**: Claude (orchestrator session), 2026-09-23.
## 1. Identity and providerID
**Answer:** `vms/<vmid>/<name>`, where `<vmid>` is the VM's numeric id and
`<name>` is `req.Name`. The ID is location-free: the node is looked up in
`GET /cluster/resources?type=vm` on every call, so a VM migrated between
nodes stays addressable. Parsing is strict: exactly three segments, the
literal `vms`, a decimal VMID without leading zeros between 100 and
999999999, and a non-empty name without `/`. Anything else is `ErrPermanent`
from `Get` and `Delete`. The name is carried so that `Get` and `Delete` can
tell our VM from another VM that later received the same VMID (entry 2). No
config value is pasted into the ID, so no ID-safety regexp is needed; node
names and storage ids are still validated at load (entry 15) because they
end up in request paths.
**Evidence:** contract §6 and §4.10 (the ID must not depend on the location
when instances can migrate); F8 (`node` is a field of every resource entry).
## 2. Primary key (`primary-key`)
**Answer:** `allocated-id`. Proxmox addresses VMs by VMID; `name` is only a
label and is not unique. The VMID is derived from the Proxy UID so a retry
lands on the same number: `vmidFor(uid, attempt)` is
`min + (fnv1a64(uid) + attempt) mod (max - min + 1)` over `vmidRange`
(entry 15). Create, under the per-name lock (entry 19):
1. Reads `/cluster/resources` once. Entries with `type: qemu`, `template: 0`
and `name == req.Name` are the by-name candidates. Two or more is
`ErrPermanent` naming every VMID: an admin must remove the duplicate.
Exactly one goes to the ownership check below and is adopted or refused.
2. With none, probes `attempt = 0, 1, …` up to 32:
- VMID free in the listing: clone onto it. If the clone fails with
`unable to create VM <vmid>: config file already exists` (F4), someone
took it between the listing and the clone; run the ownership check on
it.
- VMID taken: run the ownership check.
- Exhausted: `ErrTransient` naming the range.
3. The **ownership check** on a VMID: if its resource entry carries a
`lock` or has no `name` (a clone in progress lists neither, F8), wait
until it is unlocked (entry 10, clone budget). Then read its config. It
is **ours** when `name == req.Name` and the description carries the mark
with `uid` equal to `req.UID` or empty: adopt it and resume Create at the
seed step (entry 10). It is **foreign** when the name differs: move to
the next attempt. The same name with no mark, or with a different
non-empty `uid`, is `ErrPermanent` naming the VMID and both UIDs; the
operator never takes over a VM it did not create. A `suspended` lock (a
VM hibernated to disk) is not waited on: it is not a task and never
clears by itself. Ownership is judged at once, as `Get` does (entry 9):
a foreign hibernated VM is probed past, and ours is `ErrPermanent`
(resume or destroy it), like kvm2's paused domain. Container and
template entries at a VMID are foreign without a config read, and so
is a VM whose listed name is not `req.Name`, whatever its lock: only a
clone in progress is nameless, and waiting on another VM's `backup`
lock would hold Create for the whole clone budget on every retry
(review finding P2).
The wait in step 3 is what closes the duplicate-VM bug Phase 0 found: after
a crash mid-clone, the retry's by-name lookup misses the nameless VM, but
attempt 0 lands on the same VMID, waits for the clone to finish, and adopts
it instead of probing past it.
`Get` and `Delete` check the name and mark at the VMID in the ID. A VM there
with another name means ours is gone and the VMID was reused: `Get` answers
`ErrNotFound`, `Delete` succeeds without touching it.
**Evidence:** F4 (taken VMID fails synchronously); F8 (a mid-clone entry has
no name); contract §4.10 (allocate-then-verify, deterministic from the UID)
and §13 (adoption not verified: closed here).
## 3. Placement mapping
**Answer:** three fields are catalogue keys checked before any backend call,
one is ignored:
| Field | Meaning | Required | Unknown value |
|---|---|---|---|
| `Zone` | a node from `nodes` | only when more than one node is configured; with one node an empty zone selects it | `ErrPermanent`: `placement.zone "x" is not in proxmox.nodes` |
| `MachineType` | a key of `machineTypes` (`cores`, `memoryMiB`, optional `diskGiB`) | yes | `ErrPermanent`: `placement.machineType "x" is not in proxmox.machineTypes` |
| `Image` | a key of `templates` (a template VMID) | yes | `ErrPermanent`: `placement.image "x" is not in proxmox.templates` |
| `Region` | ignored | | still part of the spec hash: changing it replaces the VM |
The guide documents the ignored field and that catalogue edits affect only
new VMs.
**Evidence:** contract §2.1 (reject early, MAY ignore with documentation).
## 4. Image source
**Answer:** a template VM prepared by the admin, referenced by a catalogue
key: `templates.<key>.vmid`. Create makes a **full clone** (`full=1`), so a
VM owns copies of its disks and keeps nothing linked to the template: the
template may be edited, replaced or deleted without affecting running VMs.
Before cloning, Create reads the template's config and refuses it with
`ErrPermanent` when it is not a template (`template` is not 1) or when any
disk slot holds a Proxmox cloud-init drive (a value containing `cloudinit`),
which would compete with the provider's seed (entry 5). Because the clone
copies the template's config, what the config or resize step would refuse
on the clone is refused on the template too, before any VM exists: no
`net0` while `bridge` or `vlanTag` is set, and no boot disk, or one
without a readable `size=`, while `diskGiB` is set. A Permanent error
after the clone would leave a VM that fixing the template cannot help. The template must
boot from its first disk, run cloud-init with the NoCloud datasource, and
either ship `qemu-guest-agent` or be able to install it (entry 7).
Clone parameters: `newid`, `name`, `description` (the mark, entry 8),
`full=1`, `storage` when configured (empty clones onto the template's own
disk storage), and `target` when the chosen node differs from the template's
node. A cross-node clone needs the template on shared storage; this host has
one node, so that path is untested (entry 24).
Catalogue edit warning for the guide: changing a key's `vmid` changes only
VMs created afterwards. Add a new key rather than repointing one in use, so
the Proxies that name it keep a meaning you can reason about.
**Evidence:** F19 (template 9001: `scsi0`, `boot: order=scsi0`, `ide2`
free, `agent: 1`; template 9000 carries a cloud-init drive on `ide2`); F15
(a clone onto a storage without image support is refused up front); F16 (a
full clone of 9001 took about two seconds).
## 5. Cloud-init delivery
**Answer:** a NoCloud seed ISO built in-process by the shared
`internal/provider/nocloud` package (extracted from kvm2 in Phase 3):
volume label `cidata`, files `user-data` (the Proxy's `CloudInit`, byte for
byte, never parsed), `meta-data` (`instance-id: <name>` and
`local-hostname: <name>`) and `vendor-data` (the guest agent script, joined
with the configured `vendorData` as a MIME multipart when both are set;
identical to kvm2's rules). The builder stages under `os.TempDir()` (`/tmp`,
the manager's emptyDir) and writes into a `bytes.Buffer`; a seed is tens of
KiB, so buffering is within contract §1.4.
The ISO is uploaded with `POST /nodes/{node}/storage/{isoStorage}/upload`
as multipart (`content=iso`, file part `filename` named
`<name>-seed.iso`) and the returned `imgcopy` task is waited on. An upload
**overwrites** an existing file of the same name, so no delete precedes it;
the seed is rebuilt and re-uploaded on every Create attempt, since a
previous attempt may have been cut off. It is attached as
`ide2: <isoStorage>:iso/<name>-seed.iso,media=cdrom`; Proxmox then appends
`ide2` to the boot order on its own, which is harmless because the boot
disk stays first.
Only the Proxmox API is used: the storage upload endpoint accepts `iso`,
`vztmpl` or `import`, never `snippets`, so `cicustom` would need shell
access to the node. Size: the request timeout (entry 18) bounds the upload;
a seed with a multi-megabyte `vendorData` would need a larger
`requestTimeoutSeconds`.
**Evidence:** F12 (upload fields, UPID, overwrite, `.iso` suffix required);
F19 (`ide2` appended to the boot order); F21 (the seed on `ide2` installed
the agent on both the untagged network and VLAN 4); F23 (the upload is
staged in `/var/tmp` on the node, not the tmpfs `/tmp`); contract §2.1 and
§11 (byte for byte, sentinel test).
## 6. Machine type mapping
**Answer:** a catalogue: `machineTypes.<key>` is `{cores, memoryMiB,
diskGiB}`. `cores` and `memoryMiB` are written in the config step (entry
10). `diskGiB` is optional: when set and larger than the cloned boot disk,
the disk is grown with `PUT /nodes/{node}/qemu/{vmid}/resize` to the
absolute size `<diskGiB>G`, and the returned task is waited on; when unset
or not larger, the disk keeps the template's size and no call is made. The
provider never shrinks a disk. Cloud-init's growpart expands the root
filesystem at first boot.
The boot disk is the first entry of the config's `boot: order=…` that is a
disk key (`scsiN`, `virtioN`, `sataN`, `ideN`) and not a CD-ROM, else the
`bootdisk` key, else the first present of `scsi0`, `virtio0`, `sata0`,
`ide0`. Its current size comes from the `size=` option of that disk's value.
No boot disk, when `diskGiB` is set, is `ErrPermanent`.
**Evidence:** F20 (resize is a task; the same size is an `OK` no-op; a
shrink fails in the task with `shrinking disks is not supported`); F19
(`boot: order=scsi0`, becoming `order=scsi0;ide2` once the seed is
attached).
## 7. Network and IP discovery (`readiness`, `ip-source`)
**Answer:** `readiness`: `provisioning-window`. `ip-source`: `discovered`,
from the QEMU guest agent only (`GET /nodes/{node}/qemu/{vmid}/agent/network-get-interfaces`);
the API offers no ARP or DHCP-lease source. The address reported is the first
IPv4 of the first non-loopback interface, in the agent's order, that is not
loopback, link-local or unspecified.
The NIC is the template's `net0`, rewritten in the config step only when the
config sets `bridge` or `vlanTag`: the value's comma-separated options are
kept in order with the model and MAC untouched, `bridge=` and `tag=` are
replaced or appended. With neither set, `net0` is left as cloned.
The agent answers:
| Agent response | Meaning | Get reports |
|---|---|---|
| 200 with a routable IPv4 | booted | Running with that IP |
| 200 without one | network still coming up | Provisioning |
| 500 `QEMU guest agent is not running` | booting, or the agent install has not finished or failed | Provisioning |
| 500 `VM <vmid> is not running` | the VM stopped between the status read and this call | `ErrTransient` |
| 500 `No QEMU guest agent configured` | `agent: 1` was removed from the config; this VM can never report an address | Stopped, so the reconciler replaces it with a VM that has the agent on |
| anything else | cannot tell | `ErrTransient` |
The guest needs the agent. The vendor-data script installs it at first boot
unless `installGuestAgent: false`, and that install needs a package mirror:
**the proxy network must have a route to the internet**, which an egress
proxy needs anyway. Baking `qemu-guest-agent` into the template and setting
`installGuestAgent: false` removes the mirror from IP discovery. `Port` is
not used by the provider. Reachability is a prerequisite in the guide: the
operator's pod network and the clients must route to the proxy network (on
the user's host, VLAN 4, `192.168.4.0/24`).
**Evidence:** F5 (all three agent messages); F21 (agent response shape; an
address about 50 seconds after start on the untagged network and on VLAN 4);
the Phase 0 log's VLAN 4 section (no internet route meant no agent);
contract §1.3 and §2.2 (Running only with an IP).
## 8. GC marks and CreatedAt (`gc-mark`, `created-at`)
**Answer:** `gc-mark`: `written-metadata`. `created-at`: `written-at-create`.
The mark is three lines in the VM's `description`, passed in the clone call
itself, so there is no mark window:
```
proxy-operator-managed=true
proxy-operator-uid=<req.UID>
proxy-operator-created-at=<RFC3339 UTC>
```
The keys are `provider.LabelManaged`, `provider.LabelUID` and
`proxy-operator-created-at`; the managed value is `provider.LabelManagedYes`.
The parser reads `key=value` lines, tolerates CRLF, blank lines and any
other lines an admin adds, and treats the VM as managed only when the
managed value is exactly `true`. A missing `uid` line gives an empty UID. A
`created-at` that does not parse gives the zero time, which GC treats as
old; that is accepted. The clone's own `meta: …,ctime=…` is not usable:
Proxmox copies it from the template.
The config step also sets the tag `proxy_operator`, for filtering in the
Proxmox UI and as a ListByTag candidate hint (entry 12). It is not the mark:
tags cannot be passed to the clone call. Hyphens are accepted in tags on
9.2.20, but the underscore form is kept because it is valid under every
tag rule Proxmox has had. Uppercase is folded to lowercase by Proxmox.
The seed ISO carries no mark; its name `<name>-seed.iso` identifies it
(entry 11).
**Evidence:** F10 (the description survives the clone verbatim, newlines
included); F11 (the clone's `meta` is the template's); F9 (tag rules);
contract §2.1 (mark in the create call) and §4.5.
## 9. State mapping (`provisioning-state`)
**Answer:** `provisioning-state`: `create-blocks`. Create returns only once
`status/current` reports the VM running (entry 10), so a VM the provider
cloned but never started is only ever seen by a Create retry, which adopts
and starts it; the reconciler has no ID for it to `Get`.
`Get` checks in this order; the first match decides:
| Check | Source | Result |
|---|---|---|
| VMID absent | `/cluster/resources` | `ErrNotFound` |
| `lock` set, other than `suspended` | resources entry (a clone in progress also has no name) | `ErrTransient` |
| `lock: suspended` (hibernated to disk) | resources entry | Stopped |
| `status: unknown` (node not reachable, or a VM seconds old the listing has not reported yet: the first `Get` after Create met this on pv1-1) | resources entry | `ErrTransient` |
| config `does not exist` | `GET …/config` | `ErrNotFound` (deleted between the two calls) |
| config `lock` set | config | `ErrTransient` |
| `name` differs from the ID's name | config | `ErrNotFound`, logged: the VMID was reused |
| no mark in the description | config | `ErrNotFound`, logged: not ours |
| `status: stopped` | `status/current` | Stopped |
| `status: running`, `qmpstatus: running` | `status/current` | the agent decides (entry 7) |
| `status: running`, `qmpstatus: prelaunch` | `status/current` | Provisioning |
| `status: running`, any other `qmpstatus` (`paused`, `suspended`, `io-error`, …) | `status/current` | Stopped |
| any other `status` | `status/current` | Stopped |
A `suspended` lock decides the state but not ownership: the config is still
read, and a foreign name or a missing mark is still `ErrNotFound`, so a
hibernated VM that reused our VMID is never reported as ours. Only
`status/current` and the agent are skipped for it.
`status/current` is read, not only the resources entry, because a paused VM
shows as plain `running` in `/cluster/resources`. `UID` and `CreatedAt` come
from the description on every call. A running `Get` makes four calls:
resources, config, status and agent.
**Evidence:** F8 (resources fields; mid-clone entries lack a name, carry
`lock: clone` and `status: unknown`); F14 (stopped and paused as seen by
`status/current` and by resources); F2 (the `does not exist` message);
contract §2.2 and §4.11.
## 10. Create: blocking and task tracking (`create-return`, `async-ops`)
**Answer:** `create-return`: `blocking`. `async-ops`: `task-handle` and
`backend-lock`. Clone, upload, resize, start, stop, destroy and volume
delete each return a UPID; the provider polls
`GET /nodes/{node}/tasks/{upid}/status` every second until `status` is
`stopped`, then reads `exitstatus`: `OK` is success, and so is
`WARNINGS: <n>`, which Proxmox itself counts as a finished task (logged at
V(1)); anything else is `ErrTransient` carrying that text. The node comes
from the UPID (`UPID:<node>:…`). The handle is held within the call only: Create blocks
until the VM runs, so no handle outlives it, and a failure surfaces as
Create's own error with the task's message. Locks: a VM locked by a task is
waited on in Create (the ownership check) and reported as `ErrTransient` by
`Get` and `Delete`. A start issued while a clone holds the lock queues
behind it rather than failing; the provider never relies on that, because it
waits for the clone task first.
Steps, each idempotent, in order, under the per-name lock (entry 19):
1. Validate placement (entry 3). No backend call yet.
2. `GET /cluster/resources` once, then look the name up (entry 2).
3. When the name finds nothing: find the template's node (a missing
template is `ErrPermanent`), read the template's config and refuse it as
entry 4 says, then allocate the VMID (entry 2). The template is checked
only here, because a VM that already exists no longer depends on it
(entry 4): a template deleted after its clone must not fail that
clone's adoption. An adopted VM, found by name or by the probe, gets one
`status/current` read. Running, it returns its ID at once. Stopped, it
continues at step 5.
4. Clone and wait (clone budget).
5. Build the seed and upload it to the VM's node, overwriting; wait
(upload budget).
6. `PUT /nodes/{node}/qemu/{vmid}/config` (synchronous) with `cores`,
`memory`, `agent=1`, `tags=proxy_operator`,
`ide2=<isoStorage>:iso/<name>-seed.iso,media=cdrom` and, when configured,
the rewritten `net0` (entry 7).
7. Resize when `diskGiB` is larger than the boot disk (entry 6); wait
(resize budget).
8. `POST …/status/start`; wait (start budget).
9. Poll `status/current` every second until `status: running` and
`qmpstatus: running`, up to the running budget; expiry is
`ErrTransient`, and so is a `stopped` status after the start.
10. Return `vms/<vmid>/<name>`.
What a retry finds:
| Previous attempt stopped after | The retry |
|---|---|
| nothing | allocates, clones |
| submitting the clone | finds the VMID locked and nameless, waits, adopts, continues at 5 |
| the clone | adopts the stopped VM, continues at 5 |
| any of steps 5 to 7 | adopts, repeats 5 to 7 (upload overwrites, config is a replace, resize is skipped once large enough) |
| the start | adopts the running VM, returns its ID |
A Proxmox `does not exist` inside Create (a VM or volume vanishing between
steps) is `ErrTransient`, never `ErrNotFound`. The log line for a Create
records `cloudInitBytes`, never the content.
**Evidence:** F3 (a start queues behind the clone lock); F7 (a config write
under the clone lock is refused with `VM is locked (clone)`); F16 (task
status fields); F20 (resize is a task); the Phase 0 log's VLAN 4 rerun
(clone to started VM in about five seconds); contract §2.1, §4.2, §4.12 and
§5 ("do not return NotFound from Create").
## 11. Delete and secondary resources (`delete-return`, `secondary-resources`)
**Answer:** `delete-return`: `waits`. `secondary-resources`: `explicit` for
the seed ISO; the VM's disks are removed by the destroy call itself, with
`purge=1&destroy-unreferenced-disks=1`. Under the per-name lock:
1. Parse the ID (`ErrPermanent` if malformed).
2. `/cluster/resources`. VMID absent, or held by a container: skip to
step 6, trying every configured node because the VM's node is no longer
known.
3. `lock` set, other than `suspended`, or `status: unknown`: `ErrTransient`.
A clone or another task still running finishes first; the reconciler
calls Delete again after `DeletionPoll`.
4. Read the config. `does not exist`: skip to step 6 on every node. A
different name, or no mark: the VM is not ours; log it at V(1) and skip
to step 6 on every node without touching it. Our VM with a `lock` other
than `suspended` (taken since the listing): `ErrTransient`. Our VM with
`lock: suspended` (hibernated) is destroyed like a stopped one: the
listing shows it `stopped`, so step 5 is skipped, and the destroy
accepts it and removes its suspend state volume with the disks
(`destroy_hibernated_response.json`, `task_status_destroy_hibernated.json`,
captured in Phase 10). Until then Delete refused it as `ErrPermanent`
on the belief that the destroy would. A hibernated VM that is not ours
is left alone like any other.
5. Status (from the listing) other than `stopped`: `POST …/status/stop`,
wait (stop budget). Stop works on a paused VM too, which the listing
shows as `running`: the task ends `OK` and the VM is `stopped`
(`stop_paused_response.json`, `task_status_stop_paused.json`,
`status_current_after_stop_paused.json`, captured in Phase 10).
`does not exist` means the VM is gone.
6. `DELETE /nodes/{node}/storage/{isoStorage}/content/<isoStorage>:iso/<name>-seed.iso`,
wait (seed-delete budget), on the VM's node when step 5 was reached on
a VM of ours, and on every configured node in config order otherwise
(then step 7 is skipped). A missing volume is still a successful task,
so no message is matched. The seed goes **before** the destroy: once
the VM is gone `Get` is `ErrNotFound`, the reconciler releases the
finalizer and GC no longer lists it, so a seed left by a failed step
after the destroy would never be deleted (review finding CR1). On the
VM's node a failure fails the call and leaves the VM stopped for the
next Delete. On the every-node path a node whose request or task fails
is logged at V(1) and skipped, unless ctx ended or the failure is
`ErrPermanent` (a 403 for `Datastore.Allocate`, which every node
would give), which fail the call, so the missing privilege is visible
on both paths (review finding P1).
7. `DELETE /nodes/{node}/qemu/{vmid}?purge=1&destroy-unreferenced-disks=1`,
wait (destroy budget). `does not exist` is success. The destroy leaves
the seed alone (Phase 0 deleted it after the destroy), which is why
step 6 exists. That Proxmox deletes an ISO a stopped VM's `ide2` still
references was not in the Phase 0 captures; the Phase 10 real-host run
proved it (imgdel `OK` on the stopped VM's seed, then the destroy).
8. Return nil.
Everything is gone when Delete returns nil, so `Get` afterwards is
`ErrNotFound`. A concurrent `Get` during a Delete keeps seeing the VM (as
Stopped once it is stopped) until the destroy task removes its config,
because destroying sets no lock and leaves the resource entry in place until
the end. A second Delete waits on the lock, then finds nothing.
GC blind spot: a VM destroyed outside the operator leaves its seed ISO, which
ListByTag cannot see. The guide says seeds are named `proxy-*-seed.iso` on
`isoStorage` and any without a matching VM may be removed.
**Evidence:** F2 (a second destroy gives `does not exist`); F13 (volume
delete is an `imgdel` task, needs `Datastore.Allocate`, and succeeds on a
missing volume); the Phase 0 capture of resources and config mid-destroy
(`cluster_resources_during_destroy.json`, `config_during_destroy.json`);
contract §2.3.
## 12. ListByTag scope and partial failure (`list-partial`)
**Answer:** `fail-whole` for the one listing, with single unreadable VMs
skipped. Scope: every VM the cluster knows, from one
`GET /cluster/resources?type=vm`, which covers every node, including
unreachable ones, whose guests Proxmox lists with `status: unknown` (entry
13). Candidates are entries
with `type: qemu`, `template: 0`, and either the tag `proxy_operator` (tags
split on `;`, `,` or space) or a name starting with `proxy-`. A nameless
entry (a clone in progress) is not a candidate: its Create is live and it is
younger than `MinAge`. For each candidate the config is read and the
description parsed; managed VMs become an `Instance` with `ID`, `UID` and
`CreatedAt` only, which is all GC reads (contract §2.4); no status or agent
call is made. A candidate whose config cannot be read (a node down, a VM
destroyed mid-sweep) is skipped with a V(1) line, unless ctx ended, which
fails the call. The resources call failing fails the whole call. Filtering
by the mark is client-side; the tag narrows the candidates. Cost: one
resources call plus one config read per candidate per sweep.
**Evidence:** F8 (resources fields, tags separated by `;`, nameless mid-clone
entries); F9 (tags case-folded); contract §2.4 and §4.7.
## 13. Error classification (`not-found`, `quota`)
**Answer:** `not-found`: `error-code`, the 500 message
`Configuration file 'nodes/<node>/qemu-server/<vmid>.conf' does not exist`,
plus `list-and-miss` on `/cluster/resources`. The listing can miss a VM only
when it is gone: it is built from the cluster filesystem, which holds every
guest's config cluster-wide, and Proxmox lists the guests of an unreachable
node with `status: unknown` rather than dropping them. That last point is
Proxmox's documented design, not something the single-node host could show;
Phase 10 or a second node verifies it. `quota`: `absent`. The thin `local-lvm` storage
overprovisions, so no out-of-space response could be produced; capacity
failures are `ErrTransient`, and the Checker reports free space in
`Details` instead (entry 16).
The client carries each failure as `apiError{Status, Message, Errors}`:
`Message` is the body's `message` with the trailing newline trimmed, or the
reason phrase when the body has none (a 401); `Errors` is the 400 body's
per-parameter map. One `classify(op string, err error)`:
| Backend response | Class | Why |
|---|---|---|
| 401 `Authentication failed!`, 401 `No ticket` | Permanent | the token or its secret is wrong; fixed in the Secret |
| 403 `Permission check failed (<path>, <privilege>)` | Permanent | the token lacks a privilege; the message names it |
| 400 `Parameter verification failed.` with `errors` | Permanent | the request is wrong; the message includes the field errors |
| 400 as above with `storage: No such storage.` | Permanent from a call; the Checker's `StorageNotFound` (entry 16) | the storage does not exist in the cluster |
| other 4xx | Permanent | a request Proxmox will never accept |
| 500 `Configuration file '…' does not exist` | NotFound (Transient when `op == "create"`) | the VM is gone |
| 500 `VM is locked (<lock>)` | Transient | a task holds the VM |
| 500 `unable to create VM <vmid>: config file already exists` | handled by the allocator (entry 2); Transient if it escapes | the VMID is taken |
| 500 `QEMU guest agent is not running`, `No QEMU guest agent configured`, `VM <vmid> is not running` | handled by `Get` (entry 7); Transient if they escape | agent state |
| any other 5xx, including pveproxy's 595 and 596 | Transient | unknown means retry |
| task `exitstatus` other than `OK` or `WARNINGS: <n>` | Transient, with the text | the task failed; the next attempt retries |
| TLS verification failure (`x509.UnknownAuthorityError`, `x509.HostnameError`, `x509.CertificateInvalidError`) | Permanent | wrong URL or CA file; the admin must act |
| token file missing (`fs.ErrNotExist`) | Transient | the Secret has not landed yet |
| token file present but not `<user>@<realm>!<id>=<secret>` | Permanent | wrong content in the Secret |
| `context.Canceled`, `context.DeadlineExceeded`, `net.OpError`, `io.EOF`, `io.ErrUnexpectedEOF` | Transient | network or cancellation |
| an error already wrapped with a class | that class | validation and builders classify at the source |
| anything else | Transient | contract default |
Every matched string is backed by a captured response under `testdata/`;
the table test loads them rather than repeating the text. Matching is on
fixed strings anchored to the 500 status: `does not exist`,
`config file already exists`, and the whole messages
`QEMU guest agent is not running` and `No QEMU guest agent configured`,
plus, on a 400, the `storage` field error `No such storage.` (whole; a
clone onto a storage without image support has another `storage` error).
`VM is locked` and the stopped VM's `VM <vmid> is not running` are not
matched by the provider: both are Transient through the default branch.
The tests keep a matcher for each, with the whole stopped-VM message
against `^VM [0-9]+ is not running$` (its tail is a suffix of the
agent-not-running answer), only to hold the fake's wording to the
captures (review finding CR5).
**Evidence:** F1 (error text in the reason phrase and the body; 401 has no
body); F2, F4, F5, F7 (the 500 messages); F6 (no out-of-space response
could be produced on thin storage); F22 (both 401 texts); fixtures
`err_forbidden.json`, `err_bad_param.json`, `err_tag_invalid.json`,
`err_upload_bad_name.json`, and from Phase 10 `storage_status_missing.json`
(a missing storage is a 400, not the 500 the spec had assumed); the Phase 0 TLS finding (`192.168.2.8` not in
the node certificate); contract §5.
## 14. Credentials and TLS (`credentials`)
**Answer:** `mounted-files`. The config names `tokenFile`, one line
`<user>@<realm>!<tokenid>=<secret>`, sent as
`Authorization: PVEAPIToken=<that line>`. It is expected in the optional
Secret `proxmox-token`, key `token`, mounted at
`/etc/proxy-operator/proxmox`. The file is read on **every request**, so
rotating the Secret takes effect without a restart. `New` stats it and logs
a warning when it is missing, but never fails on it and never dials.
TLS: `ca-file`. `caFile` names a PEM bundle, typically the node's
`/etc/pve/pve-root-ca.pem`, carried in the same Secret as key `ca.pem`. It
is loaded on first use and cached, so rotating the CA needs a restart; a
missing CA file is `ErrTransient` until it lands, one that does not parse is
`ErrPermanent`. Without `caFile` the system roots are used. The URL's host
must be a name or address in the node certificate's SANs: on the user's
host that is `192.168.0.8` or `pv1-1.home.lab`, not `192.168.2.8`.
`insecureSkipTLSVerify: true` disables verification and is logged as a
warning by `New`.
Minimum privileges, as verified on 9.2.20, for `proxy-operator@pve` with a
token without privilege separation:
- a role on `/` with `VM.Allocate VM.Clone VM.Audit VM.Config.CDROM
VM.Config.CPU VM.Config.Memory VM.Config.Disk VM.Config.Network
VM.Config.Options VM.PowerMgmt VM.GuestAgent.Audit Datastore.Audit
Datastore.AllocateSpace Datastore.AllocateTemplate Sys.Audit SDN.Use`;
- a role on `/storage/<isoStorage>` with `Datastore.Allocate
Datastore.AllocateTemplate Datastore.AllocateSpace Datastore.Audit`,
because deleting a volume needs `Datastore.Allocate` and a grant on a more
specific path replaces the inherited one. A dedicated ISO-only storage
keeps that privilege away from backups.
`VM.Monitor` no longer exists in PVE 9 and is not needed.
`VM.Config.Cloudinit` is not needed. `SDN.Use` on `/` covers a VLAN tag on
`vmbr0`.
Hygiene: the token never appears in logs, span attributes or error strings.
`apiError.Error()` carries status and message only, never headers or URLs.
Tested with a sentinel secret that a test server echoes in a 401 body.
**Evidence:** F17 (privileges exercised and the one missing); the Phase 0
log's seed storage section; F22 (auth failure texts); contract §7.
## 15. Config block
**Answer:** a new block `proxmox:` with the Go type `provider.ProxmoxConfig`
in `internal/provider/config_proxmox.go`:
| Field | Required | Default | Validation | Ends up in |
|---|---|---|---|---|
| `url` | yes | | `https` URL with a host and no path other than `/` | request base, `…/api2/json` appended |
| `tokenFile` | yes | | non-empty | `Authorization` header |
| `caFile` | | system roots | non-empty when set | TLS roots |
| `insecureSkipTLSVerify` | | false | | TLS, warning at `New` |
| `nodes` | yes, at least one | | unique, `^[a-z0-9]([a-z0-9-]*[a-z0-9])?$` | request paths, clone `target` |
| `isoStorage` | yes | | Proxmox storage id `^[a-zA-Z][a-zA-Z0-9_.-]*$` | upload and delete paths, `ide2` |
| `storage` | | the template's disk storage | storage id regexp when set | clone `storage` |
| `bridge` | | as cloned | non-empty when set | `net0` `bridge=` |
| `vlanTag` | | as cloned | 1 to 4094 when set | `net0` `tag=` |
| `vmidRange.min`, `vmidRange.max` | | 100000, 199999 | min at least 100, max at most 999999999, min below max | `vmidFor` |
| `templates.<key>.vmid` | at least one template | | at least 100 | clone source |
| `machineTypes.<key>` | at least one | | `cores` and `memoryMiB` above zero, `diskGiB` not negative | config step, resize |
| `requestTimeoutSeconds` | | 30 | not negative | HTTP client timeout |
| `vendorData` | | | | seed `vendor-data` |
| `installGuestAgent` | | true | | seed `vendor-data` |
`isoStorage` has no default on purpose: the token must be allowed to delete
volumes there (entry 14), so the choice must be explicit. `nodes` holds node
names as Proxmox knows them (`pv1-1`), not addresses; the one URL reaches the
whole cluster.
Wiring: `ProviderConfig.Proxmox *ProxmoxConfig` with tag
`proxmox,omitempty`; `setBlocks` reports `proxmox`; the allow-list in
`Config.validate` and the constructor map in `cmd/main.go` both gain
`proxmox` in the same commit; `validateProxmox(prefix, c)` in
`config_proxmox.go` with `config_proxmox_test.go`.
**Evidence:** contract §10; the Phase 0 decisions (dedicated seed storage,
`vlanTag`, URL at `192.168.0.8`); F13 (why `isoStorage` needs
`Datastore.Allocate`).
## 16. Checker
**Answer:** one target per `nodes` entry, `Name` the node, `Address` the
URL, checked concurrently, returned in config order. The check shares the
provider's HTTP client; each request is independent, so a check that runs
out of time cancels only its own requests. Per node:
1. `GET /version` (`pveVersion`).
2. `GET /nodes/{node}/status` (the node answers; `kernel`).
3. `GET /nodes/{node}/storage/{isoStorage}/status`: the storage exists, is
active, and lists `iso` in its content (`isoStorageAvailGiB`).
4. `GET /cluster/resources?type=vm`: every configured template exists and
is a template (`templates` as `<found>/<configured>`).
Reasons, shared first: `Reachable`; `Timeout` (the round's ctx ended,
whichever request was running); `Unreachable` (TCP connect or TLS handshake
failed for a network reason, or pveproxy answered 595 or 596);
`Unauthorized` (401); `Forbidden` (403, the message names the privilege);
`APIError` (any other failure). Provider-specific, each fixed differently by
an admin:
- `CertificateInvalid`: TLS verification failed, or `caFile` is missing or
holds no certificate. Fix `url` or `caFile`, not the host.
- `CredentialsUnavailable`: the token file is missing or malformed. Fix the
Secret, not Proxmox.
- `StorageNotFound`: `isoStorage` is missing, inactive, or has no `iso`
content. Fix the storage on the node. A storage the cluster does not
know answers `400 Parameter verification failed.` with the field error
`storage: No such storage.` (`storage_status_missing.json`, captured in
Phase 10); that exact field error is matched, and the message carries
it. Any 500 from the storage status call counts as missing too,
whatever its wording: Proxmox knows the storage but cannot use it on
that node (not enabled there, or offline). That answer was not
captured, so no text is matched. Any other 4xx, and a 401, 403, 595 or
596, keeps its shared reason.
- `TemplateNotFound`: a configured template VMID does not exist or is not a
template. Fix the template or the catalogue.
Details: `pveVersion`, `kernel`, `tokenID` (the part before `=`, never the
secret), `isoStorage`, `isoStorageAvailGiB`, `templates`, filled as far as
the check got; `isoStorage` is always there. `templates` counts catalogue
entries. The Checker reads `tokenFile` itself, before the first request,
with the client's `readToken`: the seam stays one method per endpoint, and
a missing or malformed file answers `CredentialsUnavailable` without a
request. Only the part before `=` is kept.
`Timeout` also covers a request that runs out of its own timeout while the
round is still running. A dial or a connection cut mid-response is
`Unreachable`, even when the dial ran out of its own 10 s budget, because
the node did not answer. Cost: four requests per node per round. The `status.type`
comment in `api/v1alpha1/providerhealth_types.go` gains `proxmox` and the
CRD is regenerated.
**Evidence:** contract §8; fixtures `version.json`, `node_status.json`,
`storage_status.json`, `cluster_resources.json`.
## 17. Client and child spans (`client-kind`, `child-spans`)
**Answer:** `client-kind`: `thin-client`. `child-spans`: `transport-wrapper`.
The seam is the interface `api`, one method per endpoint, `ctx` first:
| Method | HTTP | Returns |
|---|---|---|
| `Version` | `GET /version` | version info |
| `NodeStatus` | `GET /nodes/{node}/status` | node status |
| `StorageStatus` | `GET /nodes/{node}/storage/{storage}/status` | storage status |
| `ClusterResources` | `GET /cluster/resources?type=vm` | entries |
| `VMConfig` | `GET /nodes/{node}/qemu/{vmid}/config` | config as `map[string]string` |
| `SetVMConfig` | `PUT /nodes/{node}/qemu/{vmid}/config` | nothing (synchronous) |
| `Clone` | `POST /nodes/{node}/qemu/{vmid}/clone` | UPID |
| `Resize` | `PUT /nodes/{node}/qemu/{vmid}/resize` | UPID |
| `Start` | `POST /nodes/{node}/qemu/{vmid}/status/start` | UPID |
| `Stop` | `POST /nodes/{node}/qemu/{vmid}/status/stop` | UPID |
| `VMStatus` | `GET /nodes/{node}/qemu/{vmid}/status/current` | status and qmpstatus |
| `Destroy` | `DELETE /nodes/{node}/qemu/{vmid}?purge=1&destroy-unreferenced-disks=1` | UPID |
| `TaskStatus` | `GET /nodes/{node}/tasks/{upid}/status` | status and exitstatus |
| `AgentNetworkInterfaces` | `GET /nodes/{node}/qemu/{vmid}/agent/network-get-interfaces` | interfaces |
| `UploadISO` | `POST /nodes/{node}/storage/{storage}/upload` (multipart) | UPID |
| `DeleteVolume` | `DELETE /nodes/{node}/storage/{storage}/content/{volid}` | UPID |
`httpAPI` is the production adapter; `fakeAPI` in `fake_test.go` keeps VMs,
tasks and ISOs in memory, simulates locks (a clone leaves a nameless locked
entry until its task completes), task polls, per-method errors and blocking,
and records a call log. Scalars in a config are rendered as strings by the
adapter (`cores` arrives as a JSON number, `memory` as a string).
The transport is `tracing.RestConfigWrapper(opts...)(base)`, a parent-gated
`otelhttp` transport, so each request is an `HTTP <METHOD>` span under
`provider.create`, `provider.get`, `provider.delete`, `provider.list` or the
monitor's `providerhealth.check`, and background traffic makes no root
spans. The `observed` decorator over `api` logs one line per call and starts
no span, so nothing is counted twice. Span attributes are otelhttp's
standard ones; the token is in a header, which otelhttp does not record.
Tests: a span recorder asserts the HTTP spans are children of the operation
span and carry no token; a reflection test asserts every `api` method logs
exactly one line.
**Evidence:** contract §4.15, §4.18, §9; `RestConfigWrapper` and
`parentGatedTransport` in `internal/tracing/transport.go`; F18 (HTTP/1.1
only).
## 18. Timeouts (`timeouts`)
**Answer:** `per-call`.
| Bound | Budget |
|---|---|
| TCP dial | 10 s |
| TLS handshake | 10 s |
| one request, response included (`http.Client.Timeout`) | `requestTimeoutSeconds`, default 30 s |
| clone task, and the wait for a VM's lock during the ownership check | 10 m |
| upload task | 2 m |
| resize task | 2 m |
| start task | 2 m |
| stop task | 2 m |
| destroy task | 5 m |
| seed delete task | 1 m |
| wait for `running` after the start | 60 s |
| task and status poll interval | 1 s |
The budgets live in one `budgets` struct that tests shrink. A task wait that
runs out is `ErrTransient` naming the UPID, and the task keeps running on
Proxmox; the next attempt finds its result through the ownership check or
the lock. A cancelled `ctx` cancels only that request: the transport keeps
its idle connections (`MaxIdleConnsPerHost` 4, `IdleConnTimeout` 90 s), and
no other call is disturbed. HTTP/2 is disabled explicitly
(`ForceAttemptHTTP2: false`, an empty `TLSNextProto`).
**Evidence:** contract §1.5 and §4.16; F16 (a full clone of the 3 GiB
template took about two seconds; tasks finish within a second or two); F21
(about 50 s from start to an address, which the reconciler's polling covers,
not the running wait).
## 19. Concurrency
**Answer:** shared state is the HTTP client (safe for concurrent use), the
lazily loaded CA pool (loaded once under a `sync.Once`-style guard), and a
lock per instance name from the shared `internal/provider/keyedlock` package
(extracted from kvm2 in Phase 3): a one-slot semaphore per key, waitable
with ctx, removed from the map once unused. Create and Delete for one name
run under that name's lock, so the reconciler's Delete and GC's Delete never
race on stop and destroy, and a Delete cannot interleave with a Create's
steps. Get and ListByTag take no lock. Two Creates for different Proxies
that land on the same VMID are safe: one clone wins, the other gets the
`config file already exists` answer, waits in the ownership check for the
winner's clone to finish, sees a different name, and probes on. Three
reconciles, a GC sweep and a check round at once are safe.
**Evidence:** contract §1.2 (concurrency); F4; entry 2.
## 20. Logging (`logging`)
**Answer:** `tiered`. Logger from `logf.FromContext(ctx)` on every call,
named `proxmox`, with `provider` on every line and `vmid`, `name` and
`node` on every line about an instance.
- V(1): one line per operation outcome (`proxmox vm ensured running`,
`proxmox vm fetched` with state, ip and uid, `proxmox vm deleted` with
`destroyed` (false when no VM of ours was left to destroy),
`proxmox vms listed` with counts, `proxmox node checked` with the reason),
one `proxmox api call` line per seam call from the decorator (`call`,
`node`, `vmid`, `storage`, `upid`, `duration`, plus `newid` and `name`
on a clone, `filename` on an upload and `volid` on a volume delete, and
on failure `httpStatus` and `error`), a `proxmox task failed` line with the UPID and
`exitstatus`, and the no-op lines of Delete and ListByTag (VM already
gone, foreign or unmarked VM left alone, seed delete skipped on a node,
unreadable candidate skipped).
- V(2): request parameters (without `description`'s content at V(1) and
never the upload body), config summaries, agent addresses, the resolved
Create parameters with `cloudInitBytes`.
- No V(5) wire tier.
- Never logged at any level: `CloudInit`, the token, the CA content, the
upload body.
**Evidence:** contract §4.17 and §9.
## 21. Tests (`backend-test`)
**Answer:** `backend-test`: `integration-tag`. Unit tests run over `fakeAPI`
behind the seam, the HTTP client over `httptest.NewTLSServer` serving the
captured fixtures, and `classify` over the fixtures directly. The
integration test is `proxmox_integration_test.go` behind
`//go:build integration`, skipped unless `PROXMOX_TEST_URL`,
`PROXMOX_TEST_TOKEN_FILE`, `PROXMOX_TEST_NODE`,
`PROXMOX_TEST_TEMPLATE_VMID` and `PROXMOX_TEST_ISO_STORAGE` are set;
optional `PROXMOX_TEST_CA_FILE`, `PROXMOX_TEST_INSECURE`,
`PROXMOX_TEST_STORAGE`, `PROXMOX_TEST_BRIDGE`, `PROXMOX_TEST_VLAN_TAG`,
`PROXMOX_TEST_VMID_MIN`, `PROXMOX_TEST_VMID_MAX`, `PROXMOX_TEST_TIMEOUT`
(default 20m), `PROXMOX_TEST_KEEP`, `PROXMOX_TEST_LOG_LEVEL`; run with
`make test-integration-proxmox`. It asserts, in order: `New` returns without
a request; `Check` reports the node Ready with details; `Create` returns
`vms/<vmid>/<name>` with the VMID in range; a second `Create` returns the
same ID and the VM count is unchanged; `Get` reaches Running with an IPv4
within the budget and its `UID` and `CreatedAt` equal what Create wrote;
`ListByTag` contains it; `Delete` returns nil; `Get` is `ErrNotFound`;
`Delete` again returns nil; no VM with the name and no `<name>-seed.iso`
remain; an HTTP span is a child of `provider.create` and no span attribute
carries the cloud-init text.
Contract §11 list, with the test for each item. Every name below exists
(Phase 8 checked each with a grep for `func <name>(`). The two integration
tests passed on pv1-1 on 2026-09-24 (timings in
[testing.md](testing.md#the-proxmox-integration-test-real-proxmox-ve-host)).
- [x] repeat `Create`: `TestCreate_repeatReturnsSameIDAndCreatesNothing`
- [x] `Get` after `Delete`: `TestGet_afterDeleteReturnsNotFound`
- [x] `Delete` twice: `TestDelete_twiceSucceeds`
- [x] malformed ID: `TestParseProviderID_malformed`, `TestGet_malformedIDIsPermanent`, `TestDelete_malformedIDIsPermanent`
- [x] unknown state maps to `StateStopped`: `TestMapState_table`, `TestGet_unknownStateIsStopped`
- [x] `StateRunning` requires an IP: `TestGet_runningWithoutIPIsProvisioning`, `TestGet_runningWithIPIsRunning`
- [x] `UID` and `CreatedAt` read back: `TestGet_readsUIDAndCreatedAtFromBackend`
- [x] `ListByTag` excludes unmarked instances: `TestListByTag_excludesUnmarkedVMs`, `TestListByTag_emptyUIDStillListed`
- [x] `Get` during an in-progress delete: `TestGet_duringDeleteStillReturnsInstance`
- [x] `Get` during an asynchronous operation or lock: `TestGet_lockedIsTransientNeverNotFound`, `TestGet_midCloneNamelessEntryIsTransient`
- [x] never started is never `StateStopped`: `TestCreate_returnsOnlyWhenRunning`
- [x] cancelled `ctx` returns promptly: `TestProvider_cancelledContextReturnsPromptly` (every method and `Check`, blocked on the first seam call, plus Create and Delete blocked mid-operation), `TestClient_cancelledContextReturnsPromptly`
- [x] `classify` table with captured responses: `TestClassify_table`
- [x] Checker reasons, expired `ctx`, stable order: `TestCheck_reasonsTable`, `TestCheck_expiredContextReturnsPromptly`, `TestCheck_oneEntryPerNodeInConfigOrder`
- [x] sentinel tests for `CloudInit` and the credential: `TestCreate_cloudInitSentinelNeverLoggedOrInErrors`, `TestClient_tokenNeverInErrorsOrLogs`, and `TestLogging_v0SilentV1OutcomeV2Detail` over a whole lifecycle at every level
- [x] child spans under the operation span: `TestTracing_httpSpansUnderOperationSpan`, `TestClient_childSpanUnderParent`
- [x] real-backend lifecycle test: `TestProxmox_lifecycle` and `TestProxmoxCheck_realHost` in `proxmox_integration_test.go`, passed on pv1-1
Tests that cover the Phase 0 corrections specifically, in addition to the
plan's per-phase lists:
- `TestCreate_resumesMidCloneVMAtDeterministicVMID`: a crash after the clone
was submitted; the retry waits for the lock and adopts, creating nothing.
- `TestCreate_takenVMIDInspectedBeforeProbing`: a foreign VM at attempt 0 is
skipped; ours at attempt 0 is adopted.
- `TestCreate_cloneAlreadyExistsRaceRunsOwnershipCheck`.
- `TestCreate_templateWithCloudInitDriveIsPermanent`.
- `TestCreate_seedUploadOverwritesWithoutDelete`.
- `TestCreate_resizeWaitsForTask`, `TestCreate_resizeSkippedWhenNotLarger`.
- `TestCreate_vlanTagAppendedKeepsMACAndOptions`.
- `TestBootDisk_table`, with a row for `order=scsi0;ide2`.
- `TestGet_pausedIsStoppedFromStatusCurrent`.
- `TestGet_agentMessagesTable` (the three messages and a 200 without IPv4).
- `TestDelete_seedDeleteIsTaskAndIdempotent`.
- `TestClient_x509FailureIsPermanent`.
**Evidence:** contract §11; docs/testing.md conventions.
## 22. Deployment
**Answer:** a new optional Secret and its mount, two Makefile targets, and
samples:
- `config/manager/manager.yaml`: an optional Secret `proxmox-token` mounted
read-only at `/etc/proxy-operator/proxmox`, keys `token` and `ca.pem`.
`/tmp` is already an emptyDir.
- Makefile: `proxmox-token-secret`, creating or updating `proxmox-token`
from `PROXMOX_TOKEN_FILE` (default `$(HOME)/.config/proxmox-token`) and
`PROXMOX_CA_FILE` (default `$(HOME)/.config/proxmox-ca.pem`), shaped like
`kvm-ssh-secret`; and `test-integration-proxmox` (`go test -race
-tags=integration -count=1 -v -timeout 30m -run TestProxmox
./internal/provider/proxmox/`).
- `config/manager/providers_config.yaml` and
`config/samples/providers-config.yaml`: a commented `proxmox-home` entry
for the user's host: `url: https://192.168.0.8:8006`, the two file paths,
`nodes: [pv1-1]`, `isoStorage: proxy-seeds`, `vlanTag: 4`,
`templates: {debian-13: {vmid: 9001}}`, a `small` machine type.
- `config/samples/proxy_proxmox.yaml`, listed in
`config/samples/kustomization.yaml`: provider `proxmox-home`, zone
`pv1-1`, machine type `small`, image `debian-13`, the Squid cloud-init
from `proxy_kvm2.yaml`.
- RBAC: none; the provider does not talk to the cluster's API.
- `hack/providers-dev.yaml`: no entry (needs credentials).
**Evidence:** contract §12.
## 23. Docs
**Answer:** `docs/proxmox-provider.md` with sections: Proxmox prerequisites
(the template recipe and what disqualifies a template, the dedicated seed
storage, the user, both roles and the token, the proxy network with DHCP and
internet, reserving the VMID range), Cluster-side setup (the Secret, the
config entry, the URL-must-match-the-certificate rule), Config reference
(entry 15), Placement (Region ignored, catalogue-edit warning), What lands on
Proxmox (the VMID scheme, the description mark, the tag, the seed ISO, the
config keys written, `qm unlock` for a stuck clone), Logging, Tracing,
Timeouts, Health check (entry 16), Error classes (entry 13), Known gaps
(entry 24), Testing. A fifth column in the three tables of
`docs/providers-existing.md` and a proxmox section. `docs/architecture.md`:
components row, a `proxmox` block in §5, the span note in §10, a §11 row and
the provider count, Decisions bullets (VMID from the UID hash with the
ownership check, the mark in the description, the seed by API upload, the
node resolved per call). A README row. `docs/providers.md` link.
`docs/testing.md` rows and a proxmox integration section. `CLAUDE.md`
Gotchas: `make test-integration-proxmox`. A `CHANGELOG.md` entry after the
user confirms.
**Evidence:** contract §12.
## 24. Known limits accepted at design time
**Answer:** ten limits, each with its consequence and workaround:
- **Create blocks** until the VM runs, holding one of three reconcile
workers. On the user's host that is about five seconds plus the clone;
large templates on slow storage take longer.
- **A clone lock that never clears** (a node crash mid-clone) makes every
Create for that Proxy wait out the clone budget and return
`ErrTransient`. Clear it with `qm unlock <vmid>` or destroy the VM.
- **Seed ISOs of VMs destroyed outside the operator** are invisible to GC.
Remove `proxy-*-seed.iso` files without a matching VM by hand.
- **The VMID range is shared with humans.** A VM created by hand inside it
costs a probe; a hand-made VM named like a proxy is refused as
`ErrPermanent`. Reserve the range.
- **Cross-node clones need shared storage** and are untested, because the
host has one node.
- **A VM whose agent was switched off is replaced**, not repaired (entry 7).
- **The proxy network needs internet** for the agent install, unless the
template ships the agent.
- **The CA is cached** after first use; rotating it needs a restart. The
token rotates freely.
- **A node that stops answering** makes `Get` and `Delete` for its VMs
`ErrTransient` until it returns, so deleting such a Proxy waits.
- Contract §13 limits that apply unchanged: config read once per process,
no installation identity in GC (two operators sharing a cluster and a
VMID range delete each other's VMs), an unconfigured provider latches the
Proxy.
Accepted after the Phase 10 review
([docs/reviews/2026-09-23-proxmox-provider-review/](reviews/2026-09-23-proxmox-provider-review/REPORT.md),
findings P3 and P4), and listed in the guide's Known gaps:
- **A Create that failed permanently after its clone keeps its VM**, and
the next Create adopts it by name even when `placement` has changed.
Destroy the VM before changing `placement` on such a Proxy.
- **A retry can clone a second VM** when a Create is cut off while
cloning onto a later probe VMID, the VM on the earlier one is removed,
and the retry comes before the clone ends. The next Create names both
VMIDs as `ErrPermanent`.
Changed by the Phase 10 captures of the three answers Phase 0 never saw:
- **A hibernated VM of ours** latches only Create's adoption
(`ErrPermanent`, resume or destroy it; entry 2). `Get` reports it
Stopped, so the reconciler replaces it, and Delete destroys it, since
the destroy takes a hibernated VM (entry 11). Before the capture, Delete
refused it as well.
- **A missing `isoStorage`** is a 400 with `storage: No such storage.`,
which the Checker reports as `StorageNotFound` (entry 16); a missing
storage at upload time is Permanent like any 4xx.
- **A paused VM** stops like a running one (entry 11, step 5), as assumed.
**Evidence:** contract §13; entries 2, 7, 10, 11, 14 above.
+667
View File
@@ -0,0 +1,667 @@
# The proxmox provider
Runs Managed proxies as VMs on a Proxmox VE cluster. It is the third
VM-style provider and the second built from
[provider-contract.md](provider-contract.md), from the design in
[proxmox-provider-spec.md](proxmox-provider-spec.md). The operator talks to
the cluster's REST API only (`https://<host>:8006/api2/json`) with an API
token. It needs no SSH, no shell on the nodes, and no library beyond Go's
standard `net/http`.
Each VM is a **full clone of a template** you prepare once. The operator
builds the Proxy's cloud-init into a NoCloud seed ISO in-process, uploads
it to a storage on the VM's node, attaches it as a CD-ROM, sets the size,
starts the VM and learns its address from the QEMU guest agent. Everything
here was proven against **Proxmox VE 9.2.20** (QEMU 11.0) on the single
node `pv1-1`; every error text the provider matches was captured from that
host.
## Proxmox prerequisites
Once per cluster, as root on a node:
1. **Proxmox VE 9.x with the API reachable** from the operator's pods on
port 8006. The operator's pods and the crawler clients must also route to
the network the proxies land on (see step 5).
2. **A template without a Proxmox cloud-init drive.** The provider brings
its own seed on `ide2`; a Proxmox cloud-init drive would compete with it
and the Proxy's user-data would be ignored. Debian 13's generic cloud
image boots UEFI (under BIOS it hangs in GRUB on a headless VM), so the
template does too:
```sh
wget https://cloud.debian.org/images/cloud/trixie/latest/debian-13-generic-amd64.qcow2
qm create 9001 --name debian-13-template --memory 2048 --cores 2 --ostype l26 \
--machine q35 --bios ovmf --efidisk0 local-lvm:1,efitype=4m,pre-enrolled-keys=0 \
--scsihw virtio-scsi-pci --net0 virtio,bridge=vmbr0 \
--serial0 socket --vga serial0 --agent 1
qm set 9001 --scsi0 local-lvm:0,import-from=/root/debian-13-generic-amd64.qcow2
qm set 9001 --boot order=scsi0
qm template 9001
```
The template must boot from its first disk, run cloud-init with the
NoCloud datasource (the Debian and Ubuntu cloud images do), and either
ship `qemu-guest-agent` or be able to install it at first boot. Before
the first clone for a Proxy, `Create` reads the template's config and
refuses it with a permanent error when:
- it is not a template (`qm template` was not run);
- any disk slot holds a Proxmox cloud-init drive (a value containing
`cloudinit`), such as the one `qm set <vmid> --ide2 local-lvm:cloudinit`
adds for Proxmox's own cloud-init support;
- it has no `net0` while the config sets `bridge` or `vlanTag`;
- the machine type sets `diskGiB` and the template has no boot disk, or
one without a readable `size=`.
These are checked on the template because the clone copies its config:
a VM cloned from a bad template could not be fixed by fixing the
template. The template is checked only when a VM is about to be cloned;
a VM that already exists no longer depends on it. The health check sees
only whether each template exists and is a template.
3. **A storage for the seed ISOs**, holding `iso` content. A dedicated
directory storage is best, because the token needs `Datastore.Allocate`
on it to delete seeds, and on a dedicated storage that privilege cannot
touch backups or other ISOs:
```sh
pvesm add dir proxy-seeds --path /var/lib/proxy-seeds --content iso
```
A seed is tens of KiB. Proxmox stages uploads under `/var/tmp`, not the
node's tmpfs `/tmp`. On a cluster, a storage that is not shared keeps
each seed on the node the VM was created on (see *Known gaps*).
4. **A user, two roles and a token.** `VM.Monitor` no longer exists in PVE
9 and `pveum` rejects it; the agent read needs `VM.GuestAgent.Audit`
alone. `VM.Config.Cloudinit` is not needed for the seed on `ide2`.
`SDN.Use` on `/` is what lets the config write put a VLAN tag on
`vmbr0`:
```sh
pveum user add proxy-operator@pve --comment "egress-proxies-operator"
pveum role add ProxyOperator -privs "VM.Allocate VM.Clone VM.Audit VM.Config.CDROM VM.Config.CPU VM.Config.Memory VM.Config.Disk VM.Config.Network VM.Config.Options VM.PowerMgmt VM.GuestAgent.Audit Datastore.Audit Datastore.AllocateSpace Datastore.AllocateTemplate Sys.Audit SDN.Use"
pveum aclmod / -user proxy-operator@pve -role ProxyOperator
pveum role add ProxyOperatorSeeds -privs "Datastore.Allocate Datastore.AllocateTemplate Datastore.AllocateSpace Datastore.Audit"
pveum aclmod /storage/proxy-seeds -user proxy-operator@pve -role ProxyOperatorSeeds
pveum user token add proxy-operator@pve operator --privsep 0
```
The second role repeats three privileges of the first on purpose: an
ACL on a more specific path **replaces** what the user inherits from
`/` for that path, so a role with `Datastore.Allocate` alone would take
the upload's `Datastore.AllocateTemplate` away. The role on `/` covers
clone, config, resize, start, stop, the agent read, the upload and the
destroy; what it lacks is `Datastore.Allocate`, without which deleting
a seed is a 403 (`Permission check failed (/storage/<id>,
Datastore.Allocate)`). With `--privsep 0` the token carries the user's
privileges.
The token's secret is printed once. Save it on your workstation as one
line, `proxy-operator@pve!operator=<secret>`, in
`~/.config/proxmox-token` with mode 0600. Copy the node's CA,
`/etc/pve/pve-root-ca.pem`, to `~/.config/proxmox-ca.pem`.
5. **A proxy network with DHCP and a route to the internet.** Each VM
keeps the template's one NIC, optionally moved to another bridge or
VLAN by the config, and takes its address from that network's DHCP. The
network needs internet twice over: the vendor-data script installs
`qemu-guest-agent` from the distro mirrors at first boot, and the agent
is the only way the operator learns the VM's address; and an egress
proxy needs egress anyway. Baking `qemu-guest-agent` into the template
and setting `installGuestAgent: false` removes the mirror from IP
discovery. The operator's pods and the clients must route to that
network: the health engine probes `http://<ip>:<port>` through the
proxy, and clients get the same address from the discovery API. The
provider does not use `spec.port` and configures no Proxmox firewall.
6. **A reserved VMID range.** VMIDs are allocated from `vmidRange`
(default 100000 to 199999). Keep hand-made VMs out of it: one there
costs a probe, and one named like a proxy is refused (see *What lands
on Proxmox*).
Check from the operator's point of view before wiring it in. This sends
the token through a file descriptor, so it never reaches the command line:
```sh
pve() {
curl -s --cacert ~/.config/proxmox-ca.pem \
-H @<(printf 'Authorization: PVEAPIToken=%s\n' "$(tr -d '[:space:]' < ~/.config/proxmox-token)") \
"https://192.168.0.8:8006/api2/json/$1"; echo
}
pve version # the token works, TLS verifies
pve nodes/pv1-1/storage/proxy-seeds/status # active, content includes iso
pve 'cluster/resources?type=vm' # the template is listed with "template":1
```
## Cluster-side setup
1. Create the Secret. It holds two keys, `token` (the one-line token file)
and `ca.pem` (the node's CA):
```sh
make proxmox-token-secret
# or by hand:
kubectl -n egress-proxies-operator-system create secret generic proxmox-token \
--from-file=token=$HOME/.config/proxmox-token \
--from-file=ca.pem=$HOME/.config/proxmox-ca.pem
```
The make target runs that command as an apply, so it also updates an
existing Secret. Override the paths with
`PROXMOX_TOKEN_FILE=... PROXMOX_CA_FILE=...`. Before it applies, it
checks that both files exist, that the token file is exactly one line
of `<user>@<realm>!<tokenid>=<secret>` (without printing it), and that
the CA file holds a PEM certificate. With `insecureSkipTLSVerify`
instead of a CA, create the Secret by hand with the `token` key only.
`config/manager/manager.yaml` mounts the Secret (optional, read-only)
at `/etc/proxy-operator/proxmox`, so the files are
`/etc/proxy-operator/proxmox/token` and
`/etc/proxy-operator/proxmox/ca.pem`. The deployment starts fine
without it, even with a proxmox provider configured: `New` makes no
request, only checks that the token file exists, and logs a `WARNING:`
line when it does not. The provider's proxies then fail transiently and
recover once the Secret lands; the health check reports
`CredentialsUnavailable` meanwhile.
**Rotating the token needs no restart.** The token file is read on
every request, so the next request after the kubelet refreshes the
mount uses the new token. Keep the old token valid until then: a 401
is a permanent error, so a Proxy whose request lands in the gap
latches until its spec changes. Add the new token
(`pveum user token add`), update the Secret, wait a couple of minutes
for the mount to refresh, and only then remove the old token. A token
created with `--expire` needs the same rotation before it expires.
**Rotating the CA does** need a restart: it is loaded on first use and
cached for the life of the process. A missing CA file is
retried until it lands; one without a certificate is a permanent error.
2. Add a provider entry to the providers config
(`config/manager/providers_config.yaml` in-cluster; the full sample,
commented out, is in `config/samples/providers-config.yaml`):
```yaml
- name: proxmox-home
type: proxmox
proxmox:
url: https://192.168.0.8:8006
tokenFile: /etc/proxy-operator/proxmox/token
caFile: /etc/proxy-operator/proxmox/ca.pem
nodes: [pv1-1]
isoStorage: proxy-seeds
vlanTag: 4
templates:
debian-13: {vmid: 9001}
machineTypes:
small: {cores: 1, memoryMiB: 1024}
```
**The URL's host must be in the node certificate.** The node's
self-signed certificate lists its own addresses and names, and TLS
verification checks the URL's host against them. On `pv1-1` it lists
`192.168.0.8`, `pv1-1` and `pv1-1.home.lab`, so
`https://192.168.2.8:8006` reaches the same node and still fails
verification (`CertificateInvalid` in the health check, a permanent
error on every call). See what yours lists with:
```sh
openssl s_client -connect 192.168.0.8:8006 </dev/null 2>/dev/null | openssl x509 -noout -ext subjectAltName
```
If the address the operator must use is not there, either give the
node a certificate that names it (a custom certificate in the node's
Certificates panel) and put that certificate's CA in `caFile`, or set
`insecureSkipTLSVerify: true`, which `New` logs as a `WARNING:` line.
`nodes` holds node names as Proxmox knows them, not addresses: the one
URL reaches the whole cluster. The providers config is read once per
process, so adding a node, a template or a machine type needs a manager
restart.
3. Apply a Proxy (`config/samples/proxy_proxmox.yaml`) and watch it:
```sh
kubectl get px -w
# NAME MODE PROVIDER PHASE IP HEALTHY
# proxy-proxmox-sample Managed proxmox-home Provisioning
# proxy-proxmox-sample Managed proxmox-home Running 192.168.4.147
# proxy-proxmox-sample Managed proxmox-home Ready 192.168.4.147 True
```
(Illustrative.) `Create` returns once the VM runs, a few seconds after
the clone; the address follows when the guest agent is up, about 50
seconds after the start on `pv1-1` with the agent installed at first
boot.
## Config reference
The block is `proxmox:` and its Go type is `ProxmoxConfig` in
`internal/provider/config_proxmox.go`, validated at load by
`validateProxmox`; unknown keys are rejected. Defaults are applied by
`withDefaults` in `internal/provider/proxmox/proxmox.go`.
| Field | Required | Default | Meaning |
|---|---|---|---|
| `url` | yes | | `https://<host>:8006`, with no path (the client appends `/api2/json`). The host must be in the node certificate |
| `tokenFile` | yes | | file holding one line, `<user>@<realm>!<tokenid>=<secret>`, read on every request and sent as `Authorization: PVEAPIToken=<line>` |
| `caFile` | | system roots | PEM bundle trusted for the API's certificate, typically the node's `/etc/pve/pve-root-ca.pem`. Loaded on first use and cached |
| `insecureSkipTLSVerify` | | `false` | skip certificate verification; `New` logs a `WARNING:` line when it is on |
| `nodes` | yes, at least one | | node names `placement.zone` selects from; unique, lowercase letters, digits and dashes. They end up in request paths and as the clone's `target` |
| `isoStorage` | yes | | storage id the seeds are uploaded to and deleted from. Must hold `iso` content; the token needs `Datastore.Allocate` on it. No default on purpose |
| `storage` | | the template's disk storage | storage id the full clone's disks go to |
| `bridge` | | as cloned | replaces `bridge=` in the cloned `net0` |
| `vlanTag` | | as cloned | 1 to 4094; replaces or appends `tag=` in the cloned `net0` |
| `vmidRange.min`, `vmidRange.max` | | `100000`, `199999` | where VMIDs are allocated; at least 100, at most 999999999, min below max |
| `templates.<key>.vmid` | at least one template | | the key names the template in `placement.image`; the VMID is at least 100 |
| `machineTypes.<key>` | at least one | | `{cores, memoryMiB, diskGiB}` size profile. `cores` and `memoryMiB` above zero. `diskGiB` is optional: when larger than the cloned boot disk the disk is grown to it; it is never shrunk |
| `requestTimeoutSeconds` | | `30` | bound on one API request, response included. The seed upload is one request, so a multi-megabyte `vendorData` needs more |
| `vendorData` | | | extra cloud-init vendor-data placed on every seed next to the Proxy's user-data; a MIME multipart part beside the agent script when `installGuestAgent` is on. Cloud-init drops any top-level key of a vendor-data cloud-config that the user-data also defines |
| `installGuestAgent` | | `true` | add the vendor-data script that installs and starts `qemu-guest-agent`. Turn it off for templates that ship the agent |
## Placement
All three catalogue fields are checked before any request. An unknown
value is `ErrPermanent`, which the reconciler latches as `Failed` with the
message below until the spec changes.
| Proxy field | Meaning for proxmox | Error when unknown |
|---|---|---|
| `placement.zone` | a node from `nodes`. May be omitted when exactly one node is configured; otherwise required | `placement.zone "x" is not in proxmox.nodes`; with several nodes and no zone, `placement.zone is required when proxmox.nodes lists more than one node` |
| `placement.machineType` | a key in `machineTypes` | `placement.machineType "x" is not in proxmox.machineTypes` |
| `placement.image` | a key in `templates` | `placement.image "x" is not in proxmox.templates` |
| `placement.region` | ignored. It is still part of the spec hash, so changing it replaces the VM and does nothing else | |
The messages come from `placement` in `internal/provider/proxmox/create.go`
and `node` in `proxmox.go`. A key that exists but names a VMID that is
missing or not a template fails later, before any clone:
`placement.image "x": template VM 9001 does not exist` or `... is not a
template`.
**Catalogue edits affect only new VMs.** The spec hash covers the
placement values, not what the config maps them to, so repointing
`templates.debian-13.vmid` at another template changes the VMs created
afterwards and leaves the running ones alone. Add a new key rather than
repointing one in use, so the Proxies that name a key keep a meaning you
can reason about. A template may be edited, replaced or deleted without
affecting existing VMs: they are full clones and share nothing with it.
## What lands on Proxmox
For a Proxy whose deterministic name is `proxy-abc…`:
| Object | Name | Lifetime |
|---|---|---|
| VM | name `proxy-abc…`, a VMID from `vmidRange` | with the Proxy |
| its disks | the template's disks, fully copied onto `storage` (or the template's storage) | destroyed with the VM (`purge=1&destroy-unreferenced-disks=1`) |
| seed ISO | `<isoStorage>:iso/proxy-abc…-seed.iso` on the VM's node | with the Proxy |
**The VMID** is derived from the Proxy's UID, so a retried Create lands on
the same number: an FNV-1a hash of the UID placed in `vmidRange`, moving
forward one at a time on collision, up to 32 probes (`vmidFor` in
`id.go`). Before allocating, `Create` looks the name up in the cluster's
resources. A VMID that is taken, or a VM that already has the name, goes
through an ownership check (`inspect` in `create.go`): a VM with a lock
(a clone still running, which also has no name yet) is waited on; then
- another name: someone else's VM, so the next VMID is probed;
- our name and our mark with our UID or an empty one: **adopted**, and
Create carries on from where the previous attempt stopped;
- our name without the mark, or with another UID: **permanent error**
naming the VMID and both UIDs. The operator never takes over a VM it did
not create;
- our name, our mark, hibernated (`lock: suspended`): **permanent error**,
`resume or destroy it in Proxmox` (see *Known gaps*). This only happens
when Create has not recorded the VM yet; a recorded VM that gets
hibernated is reported Stopped and replaced.
Two VMs with the proxy's name are a permanent error naming every VMID;
remove all but one.
**The providerID** is `vms/<vmid>/<name>`. It names no node, because VMs
migrate: the node is looked up in `GET /cluster/resources` on every call.
`Get` and `Delete` check that the VM at that VMID still has that name and
the mark. Another name means ours is gone and the VMID was reused: `Get`
answers NotFound, `Delete` succeeds without touching it.
**The mark** is three lines in the VM's description (Notes in the UI),
passed in the clone call itself, so a VM is never unmarked:
```
proxy-operator-managed=true
proxy-operator-uid=0f3a…
proxy-operator-created-at=2026-09-23T17:42:11Z
```
That is how orphan GC recognises its own VMs and how a repeat `Create`
knows whether a VM of the same name belongs to this Proxy. Lines an admin
adds to the description are kept and ignored. The clone's own
`meta: ctime=` is useless for this: Proxmox copies it from the template.
**The config write.** After the clone and the seed upload, one
`PUT /nodes/{node}/qemu/{vmid}/config` sets:
| Key | Value |
|---|---|
| `cores`, `memory` | from the machine type |
| `agent` | `1` |
| `tags` | `proxy_operator` |
| `ide2` | `<isoStorage>:iso/<name>-seed.iso,media=cdrom` |
| `net0` | only when `bridge` or `vlanTag` is set: the cloned value with `bridge=` and `tag=` replaced or appended, the model, MAC and every other option kept |
The write **replaces the template's `agent` option and tags**: a
template's `agent: 1,fstrim_cloned_disks=1` becomes `agent: 1`, and its
tags give way to `proxy_operator`. The tag is for filtering in the
Proxmox UI and narrows GC's candidates; the mark decides. Proxmox appends
`ide2` to the boot order on its own (`order=scsi0;ide2`), which is
harmless because the boot disk stays first. When the machine type's
`diskGiB` is larger than the boot disk (the first disk in `boot: order=`
that is not a CD-ROM), a resize task grows it; cloud-init's growpart
expands the root filesystem at first boot.
**The seed** (`nocloud.BuildSeedISO`, shared with kvm2 in
`internal/provider/nocloud`) is an ISO 9660 image labelled `cidata`.
`user-data` is the Proxy's `cloudInit` **byte for byte**; the operator
never parses or re-renders it. `meta-data` holds `instance-id: <name>` and
`local-hostname: <name>`. `vendor-data` carries the guest-agent script
(a shell script, not a cloud-config, so a Proxy's own `packages:` cannot
shadow it), joined with `vendorData` as a MIME multipart when both are
set. It is staged under `/tmp` (the manager's emptyDir), built in memory
and uploaded with `POST /nodes/{node}/storage/{isoStorage}/upload`. An
upload overwrites a file of the same name, so it is rebuilt and
re-uploaded on every Create attempt with no delete first: a previous
attempt may have been cut off. Only the API is used; the storage upload
endpoint does not accept `snippets`, so Proxmox's own `cicustom` would
need shell access to the node.
**What a retried Create finds.** Every step is safe to repeat. After a
crash mid-clone, the retry lands on the same VMID, waits for the clone's
lock, and adopts the VM. A stopped VM of ours gets the seed, the config
and the resize again (the upload overwrites, the config is a replace, the
resize is skipped once large enough), then the start. A running one is
returned at once.
**A clone lock that never clears.** A node that crashes mid-clone can
leave the VM locked with `lock: clone` for good. Every Create for that
Proxy then waits out the 10 minute clone budget and returns a transient
error, and the reconciler keeps retrying rather than latching. Clear it
on the node:
```sh
qm unlock <vmid> # then let the next Create adopt and finish it
qm destroy <vmid> --purge # or remove it and let Create clone again
```
**Seeds without a VM.** A VM destroyed outside the operator leaves its
seed ISO, which GC cannot see (it lists VMs). Seeds are named
`proxy-*-seed.iso` on `isoStorage`; any without a matching VM may be
removed by hand:
```sh
pvesm list proxy-seeds --content iso
pvesm free proxy-seeds:iso/proxy-abc…-seed.iso
```
**Delete** runs under a per-name lock shared with Create: stop (a paused
VM too), delete the seed on the VM's node, then destroy with
`purge=1&destroy-unreferenced-disks=1`. Each is a task that is waited on.
The seed goes before the VM because once the VM is gone nothing calls
Delete again; a failed seed delete leaves the VM stopped, and the next
Delete finishes both. A VM that is already gone, not ours, or unmarked is
left alone, and the seed is then looked for on every configured node,
where a node that fails is logged and skipped. A refusal every node would
give fails the call instead: **without `Datastore.Allocate` on
`isoStorage` the 403 makes Delete permanent** on either path, and the
Proxy's deletion stays stuck with that message (a replaced Proxy stays
failed, its VM stopped) until the privilege is granted. When Delete
returns nil, nothing of the Proxy is left.
## Logging
The provider logs through controller-runtime's logger, named `proxmox`,
with `provider` (the configured name) on every line, and `vmid`, `name`
and `node` on every line about an instance. Every line inside a traced
operation also carries `traceID` and `spanID`.
- At startup (always on): `WARNING: token file is not readable; every
call fails until it is` with `path` and `reason`, and `WARNING:
insecureSkipTLSVerify is set`.
- `--zap-log-level=debug` (logr V(1)):
- one line per operation outcome: `proxmox vm ensured running`,
`proxmox vm adopted` with `status` and `qmpstatus`,
`proxmox vm fetched` with `state`, `ip`, `uid`, `status` and
`qmpstatus`, `proxmox vm not found` with `reason`,
`proxmox vm deleted` with `destroyed` (false when no VM of ours was
left to destroy), `proxmox vms listed` with `vms`, `candidates`,
`managed` and `skipped` counts, and `proxmox node checked` with
`ready`, `reason` and `latency`;
- **one `proxmox api call` line per API request**, written by the
decorator in `internal/provider/proxmox/observe.go`: `call` (the seam
method, for example `Clone`), the identifiers it was given (`node`,
`vmid`, `storage`), `upid` for a task, `newid` and `name` on a clone,
`filename` on an upload, `volid` on a volume delete, `duration`, and
on failure `httpStatus` and `error`. A running `Get` is four calls; a
fresh Create is a dozen or more, depending on how many polls each
task takes;
- `proxmox task failed` with `upid` and `exitstatus`, and
`proxmox task finished with warnings`;
- the no-op lines: `proxmox vm already gone`,
`proxmox vm at this vmid is not ours, left alone` with `vmName`,
`proxmox vm carries no proxy-operator mark, left alone`,
`proxmox seed delete failed on a node, skipped`,
`skipping unreadable proxmox vm`, and the probe's
`proxmox vmid taken by another vm, probing on`.
- `--zap-log-level=2`: `proxmox create resolved` with the resolved
parameters (node, machine type, cores, memory, disk, template VMID,
storages, bridge, VLAN tag) and `cloudInitBytes`; the `proxmox api
call` lines also carry the config write's parameters, a summary of each
config read, the clone's description, storage and target, the resize
size, VM and task status, the agent's addresses and the upload's byte
count; `proxmox guest agent answered`; `proxmox listed instance` per
listed VM.
- Never logged at any level: the cloud-init content, the token, the CA
content and the upload body. There is no raw wire tier.
## Tracing
With an OTLP exporter configured ([architecture.md
§10](architecture.md#10-tracing-internaltracing)), every API request is a
child span of the operation span the decorators open (`provider.create`,
`provider.get`, `provider.delete`, `provider.list`) or of the monitor's
`providerhealth.check`:
| Span | Around | Attributes |
|---|---|---|
| `HTTP GET`, `HTTP POST`, `HTTP PUT`, `HTTP DELETE` | one API request, from `otelhttp` on the client's transport | otelhttp's standard HTTP client attributes: method, URL, server address and port, response status |
The transport is `tracing.RestConfigWrapper()`, the same parent-gated
wrapper the kubernetes provider uses, so a request without a parent span
starts none. The logging decorator starts no span of its own, so nothing
is counted twice; its `proxmox api call` line carries the operation
span's `spanID`, not the HTTP span's. The token travels in a header,
which otelhttp does not record, and no attribute carries cloud-init.
## Timeouts
No caller puts a deadline on the context, so every request and every wait
carries its own budget (`defaultBudgets` in
`internal/provider/proxmox/proxmox.go`, and the transport in `client.go`):
| Bound | Budget |
|---|---|
| TCP connect | 10 s |
| TLS handshake | 10 s |
| one request, response included | `requestTimeoutSeconds`, default 30 s |
| clone task, and the wait for a VM's lock in the ownership check | 10 m |
| upload task, resize task, start task, stop task | 2 m each |
| destroy task | 5 m |
| seed delete task | 1 m |
| wait for `running` after the start | 60 s |
| task and status poll interval | 1 s |
A task wait that runs out is a transient error naming the UPID; the task
keeps running on Proxmox, and the next attempt finds its result through
the ownership check or the lock. A task that ends with any `exitstatus`
other than `OK` or `WARNINGS: <n>` is transient too, with Proxmox's text.
Requests share one HTTP client and its idle connections (4 per host,
closed after 90 s idle). A cancelled or overlong request ends only
itself: nothing else in flight is disturbed, and there is no connection
to kill as there is for kvm2. HTTP/2 is disabled, because pveproxy speaks
HTTP/1.1 only. Create and Delete for the same name are serialised on a
per-name lock, so the reconciler's Delete and GC's Delete never race on
stop and destroy; Get and ListByTag take no lock.
## Health check
The operator checks every configured node every
`--provider-check-interval` (default 1m) and publishes the result in the
`ProviderHealth` object named after the provider
(`kubectl get ph proxmox-home -o yaml`) and in the
`proxy_operator_provider_target_up{provider,target}` gauge. One target per
`nodes` entry, named after the node with the URL as its address, in
config order, all checked concurrently.
Each check (`checkNode` in `internal/provider/proxmox/check.go`) first
reads the token file for its id, then makes four requests on the shared
client: `GET /version`, `GET /nodes/{node}/status`,
`GET /nodes/{node}/storage/{isoStorage}/status` (the storage must exist,
be active and hold `iso` content), and `GET /cluster/resources?type=vm`
(every configured template must exist and be a template). A check that
runs out of time cancels only its own requests.
Every target's `details` carry `isoStorage`. As facts arrive they gain
`tokenID` (the part of the token before `=`, never the secret),
`pveVersion`, `kernel`, `isoStorageAvailGiB` and `templates`
(`<found>/<configured>`), so a target that fails late still says what the
node answered before. There is no quota signal; the storage's free space
is here instead.
| Reason | Meaning | Look at |
|---|---|---|
| `Reachable` | ready | |
| `CredentialsUnavailable` | the token file is missing or not `<user>@<realm>!<tokenid>=<secret>`; no request was made | the `proxmox-token` Secret; fixed in the cluster, not in Proxmox |
| `CertificateInvalid` | TLS verification failed, or `caFile` is missing or holds no certificate | `url` (its host must be in the node certificate) or `caFile`; not the node |
| `Unreachable` | the TCP connect or TLS handshake failed for a network reason, the connection was cut mid-response, or pveproxy answered 595 or 596 (it cannot reach the node behind it) | routing and firewalls between the cluster and `url`; `pveproxy` running; the node up |
| `Unauthorized` | 401: the token id or secret is wrong | the token file; `pveum user token list proxy-operator@pve` |
| `Forbidden` | 403; the message names the path and the privilege | the roles in *Proxmox prerequisites* |
| `Timeout` | the monitor's `--provider-check-timeout` (default 15s) ran out, whichever request was running, or a request ran out of `requestTimeoutSeconds` while the round still had time | a slow or hung pveproxy, a saturated link |
| `StorageNotFound` | `isoStorage` answered with a 500 on that node (it does not exist, is not enabled there, or is offline), is not active, or holds no `iso` content; the message carries Proxmox's text | `pvesm status`; the storage's nodes and content types |
| `TemplateNotFound` | a configured template VMID does not exist or is not a template | `qm list`; the `templates` catalogue |
| `APIError` | any other failure | the message carries Proxmox's own text |
The check does not look inside the templates: a cloud-init drive, a
missing `net0` or an unreadable disk size surface only when a Create is
about to clone.
## Error classes
Every error is classified once, in `classify` in
`internal/provider/proxmox/errors.go`, with errors that carry a class from
their source (validation, the token and CA files, the ownership check)
keeping it. The reconciler latches `ErrPermanent` as `Failed` until the
spec changes, recreates on `ErrNotFound` from `Get`, and retries
everything else with backoff. The provider never returns
`ErrQuotaExceeded`: the thin `local-lvm` storage overprovisions, so no
out-of-space answer could be produced to match, and capacity failures are
transient.
| Class | What lands there |
|---|---|
| `ErrPermanent` | placement values not in the catalogue; a template VMID that is missing or not a template, has a cloud-init drive, or lacks `net0` or a sized boot disk when the config needs them; a malformed providerID; any 4xx (401 `Authentication failed!` or `No ticket`: the token; 403 `Permission check failed (<path>, <privilege>)`: a missing privilege, including `Datastore.Allocate` for the seed delete; 400 `Parameter verification failed.` with the field errors); a TLS verification failure; a token file that exists but is not `<user>@<realm>!<tokenid>=<secret>`; a CA file with no certificate; a VM with the proxy's name that carries no mark or another UID; two VMs with the proxy's name; our VM hibernated (`lock: suspended`) found by Create's adoption |
| `ErrNotFound` | from `Get`: the VMID absent from the cluster's resources, the 500 `Configuration file 'nodes/<node>/qemu-server/<vmid>.conf' does not exist`, the VM at the VMID now named differently or unmarked. Inside `Create` the `does not exist` answer is `ErrTransient`, because the contract forbids NotFound from Create; `Delete` treats all of these as done |
| `ErrTransient` | everything else: a missing token or CA file (the Secret has not landed yet); network errors and cancellation; 500 `VM is locked (<lock>)` and any lock seen in the resources or config; a VM whose node does not answer, or one a few seconds old the listing has not reported yet (`status: unknown`); the agent's answers that `Get` cannot place, such as `VM <vmid> is not running`; any other 5xx, including pveproxy's 595 and 596; a task that fails or runs out of its budget; no free VMID after 32 probes; the wait for running expiring, or a VM that stops right after its start |
`Get` turns two agent answers into states rather than errors:
`QEMU guest agent is not running` is Provisioning (booting, or the agent
install has not finished), and `No QEMU guest agent configured` is
Stopped (someone removed `agent: 1`; the reconciler replaces the VM).
Error strings name VMIDs, nodes, storages, UPIDs and Proxmox's own text.
They never contain the token, the CA content or cloud-init.
## Known gaps
- **Create blocks** until the VM runs, holding one of the three reconcile
workers for its duration. On `pv1-1` that is about five seconds plus the
clone; large templates on slow storage take longer. Other Proxies keep
reconciling on the remaining workers.
- **A clone lock that never clears** (a node crash mid-clone) makes every
Create for that Proxy wait out the 10 minute clone budget and return a
transient error. Clear it with `qm unlock <vmid>` or destroy the VM (see
*What lands on Proxmox*).
- **Seed ISOs of VMs destroyed outside the operator are invisible to
GC.** Remove `proxy-*-seed.iso` files without a matching VM by hand.
On a storage that is not shared, a VM migrated to another node leaves
its seed on the original node; Delete then looks on the VM's current
node, finds nothing, and the seed stays. Shared ISO storage avoids it.
- **The VMID range is shared with humans.** A VM created by hand inside
it costs a probe; a hand-made VM named like a proxy is refused as
`ErrPermanent`. Reserve the range.
- **Cross-node clones need shared storage and are untested.** When the
zone's node differs from the template's, the clone is sent with
`target`, which Proxmox allows only when the template's disks are on
shared storage. The host this was built on has one node.
- **A VM whose agent was switched off is replaced**, not repaired: with
`agent: 1` removed it can never report an address, so `Get` reports it
Stopped.
- **The proxy network needs internet** for the agent install, unless the
template ships the agent and `installGuestAgent` is `false`. Without it
the VM boots, gets an address, and stays `Provisioning` forever,
because the agent is the only address source. Check
`systemctl status qemu-guest-agent` and
`/var/log/cloud-init-output.log` on the guest (`qm terminal <vmid>` on
the node, or the console in the UI).
- **The CA is cached** after first use; rotating it needs a manager
restart. The token rotates freely.
- **A node that stops answering** makes `Get` and `Delete` for its VMs
transient until it returns, so deleting such a Proxy waits. Proxmox
lists an unreachable node's guests with `status: unknown` rather than
dropping them, so they are never mistaken for gone. That is Proxmox's
documented behaviour; a single-node host could not show it.
- **Contract limits that apply unchanged** ([provider-contract.md
§13](provider-contract.md)): the providers config is read once per
process; GC has no installation identity, so two operators sharing a
Proxmox cluster and a VMID range delete each other's VMs; and a Proxy
naming an unconfigured provider latches until its spec changes.
Beyond the ten limits accepted in the design:
- **A hibernated VM is replaced, or latches a Create.** `Get` reports a
VM of ours with `lock: suspended` (hibernated to disk) as Stopped, so
the reconciler deletes it (the destroy removes its suspend state volume
too) and creates a new one. Only a Create that finds such a VM before it
was ever recorded, by name, fails permanently with
`resume or destroy it in Proxmox`: resume it by starting it
(`qm start <vmid>` restores the hibernated state) or destroy it on the
node, then touch the Proxy's spec.
- **A Create that failed permanently after its clone keeps its VM.**
The stopped VM is adopted by name on the next Create, even when the
Proxy's `placement` has changed since: the Proxy then runs the old
template on the old node. After changing `placement` on such a Proxy,
destroy the VM on the node (`qm destroy <vmid> --purge`) first, or
delete and recreate the Proxy.
- **A retry can clone a second VM** in one rare sequence: a Create is
cut off while cloning onto a later probe VMID (the first was taken by
someone else's VM), that other VM is removed, and the retry comes
while the clone is still running, so its VM is still nameless. The
retry clones onto the now free first VMID. The next Create then fails
permanently with `VMs … are all named …`; destroy either VM.
- **The seed upload is one request**, bounded by `requestTimeoutSeconds`.
A seed is tens of KiB, but a multi-megabyte `vendorData` would need a
larger timeout.
- **The sample cloud-init is an open proxy.**
`config/samples/proxy_proxmox.yaml` installs Squid with
`http_access allow all`; restrict it before exposing the proxy network
to anything but the cluster.
## Testing
Unit tests run over an in-memory fake of the API seam, and the HTTP
client over a TLS test server replaying the responses captured from
`pv1-1` (`internal/provider/proxmox/testdata/`). The optional integration
test against a real host is described in
[testing.md](testing.md#the-proxmox-integration-test-real-proxmox-ve-host),
which also lists the variables it needs; run it with
`make test-integration-proxmox`. `TestProxmoxCheck_realHost` runs the
health check against the same host and creates nothing:
`go test -tags=integration -count=1 -v -run TestProxmoxCheck ./internal/provider/proxmox/`.
Both passed on `pv1-1` (PVE 9.2.20) on 2026-09-24: Create took 6.2 s,
and the guest reported an address 50 s after Create returned.
@@ -0,0 +1,286 @@
# proxmox provider review: final report
Branch `feat/proxmox-provider` at `7f2073f` (Phases 0 to 9), diffed against
`main` (`2167819`): 123 files, 49 of them Go. The core is
`internal/provider/proxmox/`; also in scope were the `nocloud` and
`keyedlock` extraction from kvm2 (`0a6c862`), the config block in
`internal/provider/config*.go`, registration and the deploy files.
**Inputs.** [docs/proxmox-provider-spec.md](../../proxmox-provider-spec.md)
(the spec the code implements) and
[docs/provider-contract.md](../../provider-contract.md) (the MUST/SHOULD
rules). A disagreement with either counts as a finding.
**Reviewers.** Two independent passes, both read-only:
1. The `go-operator-reviewer` agent (`.claude/agents/go-operator-reviewer.md`),
briefed with the spec and the contract, findings **P1 to P6** and
simplifications **S1 to S3**.
2. `/code-review high` on the branch, findings **CR1 to CR10**.
Where both passes found the same thing, the finding is listed once with both
IDs. Every finding is **fixed** (with a regression test when it is code),
**accepted** (real, not fixed here, documented as a known limit), or
**rejected** (with the reason).
**Toolchain.** `make test` green before and after the fixes. `make lint`
panics inside staticcheck on the local Go 1.27 toolchain, as it did during
kvm2, so lint ran without it, over the proxmox package and
`internal/provider`, with and without the integration tag:
```bash
bin/golangci-lint run --disable staticcheck --max-same-issues 0 --max-issues-per-linter 0 ./internal/provider/proxmox/... ./internal/provider/
bin/golangci-lint run --build-tags integration --disable staticcheck --max-same-issues 0 --max-issues-per-linter 0 ./internal/provider/proxmox/... ./internal/provider/
```
Before the fixes both runs were at the Phase 8 and 9 baseline: logcheck 12,
modernize (mapsloop) 2, unparam 3 in `fake_test.go`, gocyclo 1
(`TestAPITypes_decodeCaptures`), goconst on repeated strings, and the three
goconst hits in `internal/provider`. After the fixes the counts are the same
except goconst, one lower (224, 231 with the tag). No hit is new. The first
version of the P2 test did add one: a third `"backup"` literal in the
package's tests, then `"migrate"`. It now uses `"snapshot-delete"`.
## Summary
| ID | Severity | Finding | Disposition |
|---|---|---|---|
| CR1 + P1 | medium-high | A seed delete that fails after the destroy leaks the seed for good, and the class of the error depends on the attempt | **fixed** |
| CR2 | medium | A 403 on the seed delete latches a replaced Proxy whose VM is already gone | **fixed** (same change as CR1) |
| CR3 + P2 | medium | The ownership check waits out any lock, even on a VM whose listed name is someone else's | **fixed** |
| P3 | low-medium | Create adopts a leftover stopped VM whatever the current placement says | **accepted**, documented |
| P4 | low | A crash mid-clone plus a freed earlier VMID can yield a second VM | **accepted**, documented |
| P5 | low | 401/403 are Permanent, which bites during a token rotation | **fixed** in the guide |
| P6 | low | Get and Delete trust the listing's `status`, which lags | **accepted**, confirmed on the host; error message fixed |
| CR5 | low | `isLocked` and `isVMNotRunning` are called only from tests | **fixed** |
| CR8 | low | `TestPermanentfAndTransientf` breaks the `Test<Function>_<scenario>` rule | **fixed** |
| CR9 | low | `get` repeats `mapState`'s "guest running" rule inline | **fixed** |
| CR4 + S2 | low | The Checker repeats the cluster-wide `/version` and `/cluster/resources` per node | **rejected** |
| CR6 | low | `lockName`, `permanentf` and `transientf` are copies of kvm2's | **rejected** |
| CR7 | low | `readToken`, the CA load and seed staging do file I/O without a ctx | **rejected** |
| CR10 | low | `New` runs `withDefaults` twice | **rejected** |
| S1 | simplification | The `observe.go` decorator could be one log line in `send` | **rejected** |
| S3 | simplification | `nocloud` exports `GuestAgentScript` and `SeedLabel` for tests only | **rejected** |
## Fixed
### CR1 + P1: the seed outlives a Delete that fails after the destroy
`internal/provider/proxmox/delete.go`. Delete stopped the VM, destroyed it,
then deleted the seed. If the seed delete failed (a 595, a task over its
1 minute budget, a 403), Delete returned an error, but the VM was already
gone. The reconciler's next `Get` was `ErrNotFound`, so `reconcileDelete`
released the finalizer without calling Delete again, and GC lists only VMs.
The seed stayed on `isoStorage` for good. That breaks contract §2.3 ("MUST
remove everything Create made ... GC sees only what ListByTag returns"). A
destroy task that outlived its budget but finished in the background had
the same effect.
P1 adds the second half. On the finalizer path the next Delete found the
VMID gone and took `deleteSeedEverywhere`, which logged the same 403 at V(1)
and returned nil. The seed leaked with nothing visible.
**Fix.** The seed now goes between the stop and the destroy, so the VM,
which is what `Get` looks at, is the last thing removed: a failed seed
delete leaves the VM stopped, `Get` keeps finding it, and the next Delete
finishes both. `deleteSeedEverywhere` now fails on an `ErrPermanent`
answer (a 401 or 403 every node would give) instead of skipping it. It
still skips a node that is down or lacks the storage. `runTask`'s "gone"
result lost its last reader and was dropped.
That Proxmox deletes an ISO a stopped VM's `ide2` still references is the
one assumption the fake cannot prove. From memory of the PVE source, not
checked against it: the content delete runs `vdisk_free` without looking
for references to `iso` volumes, and the destroy skips CD-ROM drives.
Phase 0 only ever deleted the seed after the destroy. **The Phase 10
real-host run proved it** on 2026-09-24: the imgdel task on the stopped
VM's seed ended `OK`, the destroy followed, and `TestProxmox_lifecycle`
step 10 found neither the VM nor the seed.
**Tests.** `TestDelete_seedDeleteFailureOnVMNodeFails` (403, 596 and a
failed task on the VM's node) now asserts that no Destroy was sent, that
`Get` still finds the VM, that a second Delete gives the same class, and
that a Delete after the fault clears removes both. It failed on the old
code with `Destroy calls = 1` and `Get = instance not found`.
`TestDelete_seedDeleteNodeFailureSkippedWhenNodeUnknown/a refused privilege
fails the call` covers the every-node 403. Updated for the new order:
`TestDelete_stoppedVMSkipsStop`, `TestDelete_runningVMStoppedFirst`,
`TestDelete_seedDeleteIsTaskAndIdempotent`,
`TestDelete_destroyTaskFailureIsTransient` (the seed is now gone and the VM
kept) and `TestDelete_goneMidwayIsSuccess` (a VM gone at the stop still
gets a Destroy call, which answers `does not exist`).
`TestLogging_noOpLinesAtV1`'s skip case now injects a node-level 500,
since a 403 no longer skips.
Docs: spec entry 11 (steps 5 to 7 reordered, the Permanent exception),
the guide's Delete paragraph, `providers-existing.md`'s Delete bullet and
its `delete-return` cell.
### CR2: a 403 on the seed delete latched a Proxy with no VM
Same code, same fix. On the replace path (`reconcileManaged` sees a Stopped
VM and calls Delete), the old order destroyed the VM, got a 403 on the
seed, and returned Permanent. `providerFailure` latched it, so the Proxy
sat Failed pointing at a VM that no longer existed. The next attempt would
have returned nil. Now the 403 comes before the destroy: the VM stays
stopped, and every attempt gives the same Permanent answer naming
`Datastore.Allocate` until the privilege is granted. The guide says so.
### CR3 + P2: the lock wait on a VM that is plainly not ours
`internal/provider/proxmox/create.go`, `allocate`. Any taken qemu VMID went
through `inspect`, which first waits for the VM's lock to clear (clone
budget, 10 minutes) and only then compares names. A VM on one of the
Proxy's probe VMIDs, listed with another name and holding a long lock (a
`vzdump` backup, or a stale lock a failed backup left behind), made every
Create wait 10 minutes and return Transient. `vmidFor` is deterministic,
so every retry landed on the same VMID. Only a clone in progress is
nameless (F8), so a listed name other than ours settles the question.
**Fix.** `allocate` sends a taken entry to `inspect` only when its listed
name is empty or ours; any other name probes on without a config read.
Spec entry 2's step 3 says so.
**Tests.** `TestCreate_takenVMIDInspectedBeforeProbing` gains "foreign VM
under a lock probes on without waiting" (a `snapshot-delete` lock with a
2 s clone budget; the old code spent the 2 s and failed Transient), and
"foreign VM probes on without a read" now expects one config read (the
template's). `TestCreate_vmidRangeExhaustedIsTransient` expects one
VMConfig call instead of five.
### P5: token rotation order
`docs/proxmox-provider.md`, cluster-side setup. A 401 is Permanent (the
same convention as GCP), so a request made with a revoked token before the
kubelet refreshes the Secret latches that Proxy. The guide now gives the
order: add the new token, update the Secret, wait for the mount, then
remove the old one, and the same before an `--expire` date.
### CR5: matchers used only by tests
`isLocked` and `isVMNotRunning` in `errors.go` had no production caller.
Both answers are Transient through `classify`'s default branch. They moved
to `fake_test.go`, where `TestFakeAPI_errorsMatchCaptures`,
`TestAPIError_matchers` and the fake's own checks still use them to hold the
fake's wording to the captures. Spec entry 13's matching sentence and
`providers-existing.md` now name only the matchers the provider uses.
### CR8: test name
`TestPermanentfAndTransientf` became `TestPermanentf_wrapsClassWithMessage`
and `TestTransientf_wrapsClassWithMessage`. kvm2 has the same name; it is
out of scope here.
### CR9: one rule, one place
`get` asked `status == "running" && qmpstatus in {running, ""}` inline to
decide whether to query the agent, the same rule `mapState` applies. Both
now call `guestRunning(status, qmpStatus)` in `state.go`, so a change to it
cannot leave the two out of step. `TestMapState_*` and the Get tests cover
it unchanged (`TestMapState_table`, `TestMapState_fromCaptures`).
## Accepted
### P3: adoption ignores a placement change
`create.go:94-118`. A Create that fails Permanent after the clone (a 403 on
the upload, say) leaves a stopped VM. If the user then changes
`placement.image` or `zone`, the next Create adopts that VM by name, and
the Proxy runs the old template on the old node. The fix the reviewer
proposed (record the template in the mark, re-clone a never-started VM
whose template differs) is a new mark field and a destroy path inside
Create. That is too much for this MR for a two-step, user-driven sequence.
Documented in the guide's Known gaps and spec entry 24: destroy the VM
before changing `placement` on such a Proxy.
### P4: a second VM after a crash mid-clone
The retry misses a nameless in-progress clone on a later probe VMID when
the VM on the earlier VMID was removed meanwhile, and clones again. It
takes a crash or cut-off during a clone of a few seconds, a foreign VM on
the first probe VMID, and that VM's removal in the same window. The
proposed fix (inspect every nameless `lock: clone` entry in the range
first) would make every Create wait on every other Proxy's clone. The
outcome is visible, not silent: the next Create fails Permanent naming
both VMIDs. Documented in the guide and spec entry 24.
### P6: the listing's `status` may lag
`get.go:51`, `delete.go:60`, `delete.go:90`. This is the reviewer's
inference, not observed: `/cluster/resources` comes from pvestatd, which
may lag a state change by seconds. Both cases heal on retry: `Get`
answering Transient for `unknown` is polled through, and a destroy of a VM
that started behind the listing's back fails its task as Transient and the
next Delete stops it first.
**Confirmed on pv1-1.** The first `Get` came 32 ms after the second Create,
about 6 s after the clone, and was Transient: the listing still showed the
VM as `status: unknown`. The next poll 5 s later saw it running. The
behaviour is right, but the message blamed the node ("its node pv1-1 is not
answering"), which was up. Get and Delete now say "its node … is not
answering, or the VM is too new to be reported yet".
`TestGet_unknownStatusIsTransient` asserts the wording, and spec entry 9
and the guide's error table name both causes. The Delete side (a stop
skipped because the listing lags) was not met: the run's Delete came a
minute after the start.
## Rejected
- **CR4 + S2, Checker repeats cluster-wide calls per node.** Spec entry 16
makes each node's target self-contained, so a failure is reported
against the node whose round saw it. The cost is two extra requests per
extra node per round, and the configured cluster
has one node. Worth revisiting with a multi-node config, not before.
- **CR6, helpers copied from kvm2.** `permanentf`/`transientf` are one
line each; `lockName` is five and wraps `keyedlock.Acquire`'s error in
`provider.ErrTransient`. Moving that into `keyedlock` would couple a
generic lock package to the provider sentinels, and putting the helpers
in `internal/provider` would touch every kvm2 call site in a proxmox MR.
- **CR7, file I/O without a ctx.** `readToken` and the CA load read a
mounted Secret with `os.ReadFile`, which cannot be cancelled; a ctx
parameter would be unused. kvm2's `os.ReadFile` of the SSH key is the
precedent. Seed staging is `nocloud`'s, shared with kvm2 and unchanged by
the extraction.
- **CR10, `withDefaults` twice in `New`.** Startup only, idempotent, a few
small maps. `newProvider` applies it because the tests build providers
through it; defaulting once would need a second constructor shape.
- **S1, replace the `observe.go` decorator with one log line in `send`.**
The decorator is spec entries 17 and 20's `span-decorator` answer: each
seam call gets a child span and a V(1)/V(2) line with node, VMID and
UPID, which `send` does not know. It mirrors kvm2's decorator, and the
reflection test guarantees every seam method is covered. Swapping the
design in the review phase is out of proportion.
- **S3, `nocloud` exports used only by tests.** `SeedLabel` is read by
both kvm2's and proxmox's Create tests to recognise a real seed, and
`GuestAgentScript` by proxmox's `TestBuildSeed_table`, which tests
proxmox's own mapping of its config fields onto the vendor-data, not
`nocloud`'s composition. The exports are the price of cross-package
tests.
## Checked and fine
From the reviewer agent's report, verified in code:
- The kvm2 extraction is a pure move. kvm2 still wraps every seed-build
error as Permanent, as before, and `keyedlock` is byte-identical to the
code it replaced.
- Delete stops or destroys only after the listing shows a qemu entry at
the VMID and the config carries the name and the mark; a reused VMID or
an unmarked VM is left alone.
- A missing VM inside Create is Transient, never NotFound; locks and
`status: unknown` are Transient, never NotFound.
- Create resumes at every step: the VMID comes from the UID, a nameless
locked entry is waited on and adopted, upload/config/resize are
repeatable, and the mark is written in the clone call.
- Secrets: `apiError` carries status and message only, the token never
reaches a `url.Error`, the Checker keeps only the token id, and the
cloud-init text appears only as `cloudInitBytes`.
- Every seam call takes the ctx; `pollUntil` honours ctx and budget;
`keyedlock.Acquire` gives up with ctx.
- Spec entries 1, 3, 4, 9, 11, 12, 13, 15 and 16 match the code (P2 was the
one disagreement, and the spec shared the flaw).
- Every §11 test name in spec entry 21 exists.
The agent also saw kvm2's `TestCheck_hostsCheckedConcurrently` fail once
under `-race` (199.9 ms against its 200 ms floor). That file is unchanged
on this branch; the flake is parked, as before.
+149
View File
@@ -67,6 +67,13 @@ manual runs too.
| | `image_test.go`, `create_test.go`, `get_test.go`, `delete_test.go`, `list_test.go`, `kvm2_test.go`, `provider_test.go` | image cache (fetch with the marker written last, cached, marker missing or without a digest, sha256 change or mismatch, 4xx/5xx, no Content-Length, short read, concurrent callers fetch once, cancel mid-stream, streamed without buffering, upload on a dedicated connection) and the pool-space quota precheck; Create resume after every partial step, returns only when running, adoption guard on UID, seed always rebuilt, cloud-init sentinel never logged; Get state and address mapping incl. agent-before-ARP; every Delete already-gone case, NVRAM flag, per-name lock; ListByTag across hosts, one down fails the whole call, unreadable domain skipped; `New` never dials and warns on missing files; every error wrapped with the configured name; logging tiers V0/V1/V2 | `httptest` image server + in-memory `fakeBackend` behind the `backend` seam |
| | `observe_test.go`, `check_test.go` | every seam method traced and logged exactly once (guarded by reflection), child spans under the parent, detail only at V2, upload bodies never logged, error fields; health check: one entry per host in config order, hosts checked concurrently, expired ctx returns promptly, never touches the pooled connection, the reason table, detail keys | `fakeBackend` + `funcr` capture + span recorder |
| | `kvm2_integration_test.go` (`-tags=integration`) | one real VM: `New` without a dial, `Check` Ready with details, Create, idempotent re-Create with an unchanged domain count, Running with IP and the metadata round-trip, ListByTag, Delete, NotFound, pool state afterwards, `kvm2.*` spans under `provider.create`; `TestKVM2Check_realHost` runs the health check against the host and creates nothing | a **real hypervisor** — see below |
| `internal/provider/nocloud` | `seed_test.go` | the NoCloud seed kvm2 and proxmox share: the ISO reopened (files, label, user-data byte for byte, temp dir removed, an empty vendor-data still a file, an empty name refused) and the vendor-data rules (a shell script, not cloud-config; multipart with configured extra vendor-data; the agent install switched off) | pure |
| `internal/provider/keyedlock` | `keyedlock_test.go` | the per-name lock kvm2 and proxmox share: the entry is removed once uncontended, waiters share one lock, a waiter gives up with its ctx | pure |
| `internal/provider/proxmox` | `id_test.go`, `mark_test.go`, `state_test.go`, `seed_test.go`, `errors_test.go`, `api_test.go` | pure pieces: providerID round-trip and malformed forms, the VMID derived from the UID (deterministic, in range, probing forward and wrapping), the description mark (keys, parsed from the captured clone, admin lines kept), state mapping and the IPv4 filter from captures, seed names and building (empty name Permanent without its payload, staging failure Transient); the `classify` table and the 500-message matchers against every captured error; the API types decoded from the captures | pure, over the responses Phase 0 captured from PVE 9.2.20 (`testdata/`) |
| | `client_test.go` | the thin HTTP client: the token read from its file on every request (missing Transient, unparsable Permanent), `apiError` from status, reason phrase and body, the data envelope, method, path and parameters of every seam call, the clone and destroy query shapes, the volid's slash escaped, HTTP/1.1 only, request timeout and cancel Transient and the client unaffected, a child span under a parent and none without, the token never in errors or logs, the CA file trusted and loaded lazily, an x509 failure Permanent, insecure mode, the upload streamed as multipart | `httptest.NewTLSServer` replaying the captures byte for byte |
| | `create_test.go`, `get_test.go`, `delete_test.go`, `list_test.go`, `task_test.go`, `proxmox_test.go`, `provider_test.go`, `fake_test.go` | Create: resume after every partial step and after a crash mid-clone, the VMID probe and ownership check, the template's shape refused before any clone, resize as a task, the seed overwritten without a delete, the net0 bridge and VLAN rewrite keeping the MAC, returns only when running, cloud-init sentinel never logged; Get: lock checked before name, the agent's messages, paused from `status/current`, hibernated, a reused VMID, four calls when running; Delete: every already-gone and not-ours case, the seed deleted before the destroy on the VM's node or on every node, a hibernated VM destroyed, the per-name lock; ListByTag: candidates by tag or name prefix, the mark decides, an unreadable VM skipped unless ctx ended; task and lock waits; `New` makes no request; the contract sweep: every method returns within a second of a cancel (blocked on its first seam call, and mid-operation), every error wrapped with the configured name, op and id, logging tiers V0/V1/V2 with the no-op lines | in-memory `fakeAPI` behind the `api` seam, its error texts checked against the captures |
| | `observe_test.go`, `check_test.go`, `provider_test.go` | the decorator: every seam method logs exactly one line (guarded by reflection), starts no span, identifiers at V1, detail at V2, upload body never logged, error fields; a Get through `provider.WithTracing` over the real client makes four `HTTP GET` spans, all children of `provider.get` and none carrying the token; health check: one entry per node in config order, nodes checked concurrently (a barrier, not a timer), expired ctx returns promptly, the reasons table at V2 with a sentinel secret, the transport reasons from real `net/http` and `crypto/tls` errors, detail keys | `fakeAPI` + `funcr` capture + span recorder; the real client over `httptest` for the tracing and transport-reason rows |
| | `proxmox_integration_test.go` (`-tags=integration`) | one real VM: `New` without a request, `Check` Ready with details, Create, idempotent re-Create with an unchanged VM count, Running with an IPv4 and the mark read back, ListByTag, Delete, NotFound, no VM and no seed left, HTTP spans under `provider.create`; `TestProxmoxCheck_realHost` runs the health check against the host and creates nothing. **Passed on pv1-1 (PVE 9.2.20), 2026-09-24**: Create 6.2 s, address 50 s later, whole run 61 s | a **real Proxmox VE host** — see below |
| `internal/health` | `probe_test.go` | probes through a **real CONNECT-capable proxy stub** to a real TLS server: tunnel success, refused CONNECT, wrong status, dead proxy, plain-http forward | `httptest` + hijacked bidirectional tunnel |
| | `engine_test.go` | thresholds, first-verdict, latency suppression matrix, dropped-event retry, stale-probe UID guard, tick scheduling/seeding/pruning, `Start` end-to-end | fake `client.Reader` + injected `probeFn` |
| `internal/lease` | `store_test.go` | capacity, `MaxLeases=0`, all three selection tie-breaks, target-scoped vs global cooldowns, expiry via fake clock, report-on-retained-lease, idempotent release, **40 concurrent acquires never exceeding capacity** | injectable clock; the concurrency case is why `-race` matters |
@@ -105,6 +112,27 @@ envtest suite the scaffold generated.
boots and reports an address are provable only on a real host — that is
the opt-in integration test below (one per provider).
- **Live Proxmox, by default**: the proxmox unit tests run over an
in-memory fake of the 16 seam calls, and the HTTP client over
`httptest` replaying responses captured from PVE 9.2.20. Whether a
clone, config write, seed upload and start yield a guest that boots and
reports an address is provable only on a real host (the proxmox
integration test below, which has passed on pv1-1). Two kinds of path
have met only the fake:
- **the cross-node paths**: a clone that targets another node, the seed
deleted on every node when the VM's node is unknown, a Check round over
several nodes. The test host is a single node, so the integration test
cannot reach them either
([proxmox-provider-spec.md](proxmox-provider-spec.md) entry 24);
- **an answer never captured**: the storage status for a storage the
cluster knows but cannot use on a node (disabled there, or offline).
Every 500 there is taken as `StorageNotFound`, and what the fake
answers is its own guess. The three answers Phase 0 missed were
captured on pv1-1 on 2026-09-24: a missing storage is a 400 with
`storage: No such storage.`, a hibernated VM is destroyed without
complaint (its suspend state volume with it), and a stop on a paused
VM ends `OK` with the VM stopped.
## The KVM integration test (real hypervisor)
`internal/provider/kvm/kvm_integration_test.go` compiles only under
@@ -256,6 +284,127 @@ go test -tags=integration -count=1 -v -run TestKVM2Check ./internal/provider/kvm
The test is not part of `make test` and not run in CI. Keep it compiling
with `go vet -tags=integration ./...` when the provider's internals move.
## The proxmox integration test (real Proxmox VE host)
`internal/provider/proxmox/proxmox_integration_test.go` compiles only
under `-tags=integration` and skips itself unless the environment names a
host. It deviates from the testcontainers convention for the same reason
as the kvm tests: Proxmox VE is a hypervisor, not something a container
can stand in for. It needs a node with a cloud-init template without a
Proxmox cloud-init drive and with the guest agent enabled, and a token
with the privileges of [proxmox-provider-spec.md](proxmox-provider-spec.md)
entry 14: the role on `/` from the plan's "Before Phase 0" section
([2026-09-18-1502-proxmox-provider.md](plans/2026-09-18-1502-proxmox-provider.md)),
plus `Datastore.Allocate` on the ISO storage. A passing run leaves
nothing behind: its VM and its seed ISO are gone.
```sh
export PROXMOX_TEST_URL=https://192.168.0.8:8006 # host must be in the node certificate
export PROXMOX_TEST_TOKEN_FILE=$HOME/.config/proxmox-token # one line: user@realm!tokenid=secret
export PROXMOX_TEST_CA_FILE=$HOME/.config/proxmox-ca.pem # the node's pve-root-ca.pem
export PROXMOX_TEST_NODE=pv1-1
export PROXMOX_TEST_TEMPLATE_VMID=9001
export PROXMOX_TEST_ISO_STORAGE=proxy-seeds # holds iso content
export PROXMOX_TEST_VLAN_TAG=4 # template 9001's net0 is untagged; proxies live on VLAN 4
make test-integration-proxmox
```
Required: `PROXMOX_TEST_URL`, `PROXMOX_TEST_TOKEN_FILE`,
`PROXMOX_TEST_NODE`, `PROXMOX_TEST_TEMPLATE_VMID` and
`PROXMOX_TEST_ISO_STORAGE`; the test skips when any is missing. Optional:
`PROXMOX_TEST_CA_FILE` (empty trusts the system roots),
`PROXMOX_TEST_INSECURE` (`true` skips certificate verification; the log
says so), `PROXMOX_TEST_STORAGE` (the clone's target storage; empty keeps
the template's), `PROXMOX_TEST_BRIDGE` and `PROXMOX_TEST_VLAN_TAG` (the
cloned `net0`'s bridge and tag; empty keeps the template's),
`PROXMOX_TEST_VMID_MIN` and `PROXMOX_TEST_VMID_MAX` (default
`100000`..`199999`), `PROXMOX_TEST_TIMEOUT` (default `20m`, the budget for
the whole lifecycle), `PROXMOX_TEST_KEEP` to leave the VM behind, and
`PROXMOX_TEST_LOG_LEVEL` (default `1`): at `1` every request is one
`proxmox api call` line with the call, node, VMID, UPID and duration,
plus the per-operation outcome lines; `2` adds request parameters, config
summaries and agent addresses; `0` leaves only `t.Log`. Provider, node,
name and VMID are stated once and left off every provider line. Without
`Datastore.Allocate` on the ISO storage the seed delete at the end of
Delete is a 403, and Delete fails as Permanent.
What one run asserts, in order, each step logged with its duration:
1. `New` returns within 100 ms and makes no request (no `proxmox api
call` line before the first operation).
2. `Check` reports the node `Ready` with reason `Reachable` and the
`pveVersion`, `kernel`, `tokenID`, `isoStorageAvailGiB` and
`templates` (`1/1`) details, which are logged.
3. Every later call goes through `provider.WithTracing`, with an
in-memory span recorder on the transport as well.
4. `Create` returns `vms/<vmid>/<name>` with the VMID in range. The
machine type is 1 core, 1024 MiB and a 4 GiB disk, so a 3 GiB template
is grown by the resize task.
5. A second `Create` returns the same ID and the cluster's QEMU VM count
is unchanged.
6. `Get`, polled every 5 s, reaches `Running` with an IPv4 within the
budget (a Transient answer while the guest boots is logged and polled
through). Its `UID` and `CreatedAt` equal the mark in the VM's
description, read back from its config, and `CreatedAt` is within a
few minutes of now. The time from Create's return to the first address
is logged.
7. `ListByTag` contains the ID with that UID and CreatedAt.
8. A TCP connect to port 22 on the guest, logged only: the machine
running the test may not route to the VM's network.
9. `Delete` returns nil, `Get` is `ErrNotFound`, a second `Delete` is a
no-op.
10. No VM with the name is left in the cluster's resources, and no
`<isoStorage>:iso/<name>-seed.iso` in the storage's iso content. The
content is read with one extra read-only GET of
`/nodes/{node}/storage/{storage}/content?content=iso` made by the test
itself (`storage_content_iso.json` is its captured shape): the seam
has no method to list storage content, and does not get one for a
test.
11. Among the recorded spans, a `provider.create` span has at least one
`HTTP` child, and no span attribute carries the cloud-init text.
With `PROXMOX_TEST_KEEP` steps 9 and 10 are skipped, and at the end the
log prints what to run as root on the node to clean up:
`qm stop <vmid>`, `qm destroy <vmid> --purge` and
`pvesm free <isoStorage>:iso/<name>-seed.iso`. Otherwise a `t.Cleanup`
runs the Delete even when an earlier assertion failed. It finds the VM by
name, so a Create that failed halfway is cleaned up too.
Cost: one full clone of the template (about 2 s for the 3 GiB template on
pv1-1), one boot to an address (about 50 s), and a few seconds of
destroy; a run takes a couple of minutes. The Makefile target's 30 m
timeout covers the 20 m budget plus the cleanup.
`TestProxmoxCheck_realHost` runs the provider health check against the
same host. It must find the node reachable, the ISO storage usable and
the template present, and it creates nothing. It needs the same
variables; to run only this test:
```sh
go test -tags=integration -count=1 -v -run TestProxmoxCheck ./internal/provider/proxmox/
```
**Last run: 2026-09-24 on pv1-1** (Proxmox VE 9.2.20, kernel
7.0.14-17-pve, template 9001, `proxy-seeds`, VLAN tag 4). Both tests
passed; `go test` reported 60.9 s for the package.
| Step | Time |
|---|---|
| `Check` | 141 ms (125 ms in `TestProxmoxCheck_realHost`) |
| clone task, submit to done | about 3 s (seen done at the third 1 s poll) |
| `Create`, clone to started VM | 6.2 s |
| second `Create` (adoption, nothing created) | 32 ms |
| Create's return to the first address | 50.2 s (the agent install from the internet) |
| `Delete` (stop, seed delete, destroy), `Get`, second `Delete` | 3.1 s |
| `TestProxmox_lifecycle` | 59.7 s |
The first `Get`, 32 ms after the second Create, was Transient: the
listing still showed the new VM with `status: unknown`. The next poll 5 s
later saw it running. The Delete deleted the seed while the stopped VM's
`ide2` still referenced it, then destroyed the VM, and step 10 found
neither left. Like the kvm tests they are not part of `make test` and not
run in CI.
## The scaffolded `make test-e2e` suite
`test/e2e/` is the kubebuilder-generated smoke suite (build image → kind
+10 -1
View File
@@ -25,6 +25,8 @@ type ProviderConfig struct {
// KVM2 is the same block shape as KVM, for the kvm2 provider type: one
// or more libvirt/KVM hypervisors reached over SSH.
KVM2 *KVMConfig `json:"kvm2,omitempty"`
// Proxmox is a Proxmox VE cluster reached through its REST API.
Proxmox *ProxmoxConfig `json:"proxmox,omitempty"`
}
// setBlocks names the type-specific blocks that are non-nil, so validate can
@@ -43,6 +45,9 @@ func (p ProviderConfig) setBlocks() []string {
if p.KVM2 != nil {
out = append(out, "kvm2")
}
if p.Proxmox != nil {
out = append(out, "proxmox")
}
return out
}
@@ -106,7 +111,7 @@ func (c *Config) validate() error {
prefix := fmt.Sprintf("providers[%d] %q", i, p.Name)
switch p.Type {
case "kubernetes", "gcp", "kvm", "kvm2":
case "kubernetes", "gcp", "kvm", "kvm2", "proxmox":
case "":
return fmt.Errorf("%s: type is required", prefix)
default:
@@ -130,6 +135,10 @@ func (c *Config) validate() error {
if err := validateKVMBlock(prefix, "kvm2", p.KVM2); err != nil {
return err
}
case "proxmox":
if err := validateProxmox(prefix, p.Proxmox); err != nil {
return err
}
}
}
return nil
+179
View File
@@ -0,0 +1,179 @@
package provider
import (
"fmt"
"net/url"
"regexp"
)
// ProxmoxConfig configures a proxmox provider instance: one Proxmox VE
// cluster reached through its REST API with an API token. VMs are full
// clones of prepared templates, seeded with a NoCloud ISO uploaded to
// ISOStorage. The design is in docs/proxmox-provider-spec.md.
type ProxmoxConfig struct {
// URL is the API endpoint, https://<host>:8006. The host must be a name
// or address in the node certificate. Required.
URL string `json:"url"`
// TokenFile holds one line, <user>@<realm>!<tokenid>=<secret>, read on
// every request. Required.
TokenFile string `json:"tokenFile"`
// CAFile is a PEM bundle trusted for the API's certificate, typically the
// node's /etc/pve/pve-root-ca.pem. Empty means the system roots.
CAFile string `json:"caFile,omitempty"`
// InsecureSkipTLSVerify disables certificate verification.
InsecureSkipTLSVerify bool `json:"insecureSkipTLSVerify,omitempty"`
// Nodes are the Proxmox node names placement.zone selects from.
// Required, at least one, unique.
Nodes []string `json:"nodes"`
// ISOStorage is the storage seed ISOs are uploaded to and deleted from;
// the token needs Datastore.Allocate on it. Required.
ISOStorage string `json:"isoStorage"`
// Storage is the target storage of the full clone. Empty clones onto the
// template's own disk storage.
Storage string `json:"storage,omitempty"`
// Bridge replaces the bridge of the cloned net0. Empty keeps it.
Bridge string `json:"bridge,omitempty"`
// VLANTag sets the VLAN tag of the cloned net0, 1 to 4094. Zero keeps
// whatever the template has.
VLANTag int `json:"vlanTag,omitempty"`
// VMIDRange is where VMIDs are allocated. Default 100000 to 199999.
VMIDRange *ProxmoxVMIDRange `json:"vmidRange,omitempty"`
// Templates are the clone sources placement.image names. Required, at
// least one.
Templates map[string]ProxmoxTemplate `json:"templates"`
// MachineTypes are the size profiles placement.machineType names.
// Required, at least one.
MachineTypes map[string]ProxmoxMachineType `json:"machineTypes"`
// RequestTimeoutSeconds bounds one API request. Default 30.
RequestTimeoutSeconds int `json:"requestTimeoutSeconds,omitempty"`
// VendorData is optional cloud-init vendor-data placed on every seed next
// to the Proxy's own user-data, as for the kvm providers.
VendorData string `json:"vendorData,omitempty"`
// InstallGuestAgent adds a vendor-data script that installs and starts
// qemu-guest-agent, which is how the provider learns a VM's address.
// Default true. Turn it off for templates that ship the agent.
InstallGuestAgent *bool `json:"installGuestAgent,omitempty"`
}
// ProxmoxVMIDRange bounds the VMIDs the provider allocates, both inclusive.
type ProxmoxVMIDRange struct {
Min int `json:"min"`
Max int `json:"max"`
}
// ProxmoxTemplate is one clone source: a VM converted to a template.
type ProxmoxTemplate struct {
VMID int `json:"vmid"`
}
// ProxmoxMachineType is a VM size profile. DiskGiB grows the cloned boot
// disk when larger than the template's; zero keeps the template's size.
type ProxmoxMachineType struct {
Cores int32 `json:"cores"`
MemoryMiB int64 `json:"memoryMiB"`
DiskGiB int64 `json:"diskGiB,omitempty"`
}
// Proxmox's own limits on VMIDs and VLAN tags.
const (
ProxmoxMinVMID = 100
ProxmoxMaxVMID = 999999999
proxmoxMaxVLANTag = 4094
)
var (
// proxmoxNodeName is a Proxmox node name: a DNS label, as pvecm allows.
proxmoxNodeName = regexp.MustCompile(`^[a-z0-9]([a-z0-9-]*[a-z0-9])?$`)
// proxmoxStorageID is a Proxmox storage identifier.
proxmoxStorageID = regexp.MustCompile(`^[a-zA-Z][a-zA-Z0-9_.-]*$`)
)
func validateProxmox(prefix string, c *ProxmoxConfig) error {
if c == nil {
return fmt.Errorf("%s: proxmox block is required", prefix)
}
if err := validateProxmoxURL(c.URL); err != nil {
return fmt.Errorf("%s: proxmox.url %w", prefix, err)
}
if c.TokenFile == "" {
return fmt.Errorf("%s: proxmox.tokenFile is required", prefix)
}
if len(c.Nodes) == 0 {
return fmt.Errorf("%s: proxmox.nodes must list at least one node", prefix)
}
seen := make(map[string]bool, len(c.Nodes))
for i, n := range c.Nodes {
if !proxmoxNodeName.MatchString(n) {
return fmt.Errorf("%s: proxmox.nodes[%d] %q is not a Proxmox node name (lowercase letters, digits and dashes)", prefix, i, n)
}
if seen[n] {
return fmt.Errorf("%s: proxmox.nodes[%d]: duplicate node %q", prefix, i, n)
}
seen[n] = true
}
if c.ISOStorage == "" {
return fmt.Errorf("%s: proxmox.isoStorage is required: name a storage with iso content where the token holds Datastore.Allocate", prefix)
}
if !proxmoxStorageID.MatchString(c.ISOStorage) {
return fmt.Errorf("%s: proxmox.isoStorage %q is not a Proxmox storage id", prefix, c.ISOStorage)
}
if c.Storage != "" && !proxmoxStorageID.MatchString(c.Storage) {
return fmt.Errorf("%s: proxmox.storage %q is not a Proxmox storage id", prefix, c.Storage)
}
if c.VLANTag < 0 || c.VLANTag > proxmoxMaxVLANTag {
return fmt.Errorf("%s: proxmox.vlanTag must be between 1 and %d (got %d)", prefix, proxmoxMaxVLANTag, c.VLANTag)
}
if r := c.VMIDRange; r != nil {
if r.Min < ProxmoxMinVMID || r.Max > ProxmoxMaxVMID || r.Min >= r.Max {
return fmt.Errorf("%s: proxmox.vmidRange must satisfy %d <= min < max <= %d (got %d to %d)", prefix, ProxmoxMinVMID, ProxmoxMaxVMID, r.Min, r.Max)
}
}
if len(c.Templates) == 0 {
return fmt.Errorf("%s: proxmox.templates must define at least one template", prefix)
}
for key, t := range c.Templates {
if key == "" {
return fmt.Errorf("%s: proxmox.templates has an empty key", prefix)
}
if t.VMID < ProxmoxMinVMID {
return fmt.Errorf("%s: proxmox.templates[%q].vmid must be at least %d (got %d)", prefix, key, ProxmoxMinVMID, t.VMID)
}
}
if len(c.MachineTypes) == 0 {
return fmt.Errorf("%s: proxmox.machineTypes must define at least one profile", prefix)
}
for key, mt := range c.MachineTypes {
if key == "" {
return fmt.Errorf("%s: proxmox.machineTypes has an empty key", prefix)
}
if mt.Cores <= 0 || mt.MemoryMiB <= 0 {
return fmt.Errorf("%s: proxmox.machineTypes[%q] needs cores > 0 and memoryMiB > 0", prefix, key)
}
if mt.DiskGiB < 0 {
return fmt.Errorf("%s: proxmox.machineTypes[%q].diskGiB must not be negative", prefix, key)
}
}
if c.RequestTimeoutSeconds < 0 {
return fmt.Errorf("%s: proxmox.requestTimeoutSeconds must not be negative", prefix)
}
return nil
}
// validateProxmoxURL accepts only an https URL with a host and no path, so
// the client can append /api2/json itself.
func validateProxmoxURL(s string) error {
if s == "" {
return fmt.Errorf("is required")
}
u, err := url.Parse(s)
if err != nil {
return fmt.Errorf("does not parse: %w", err)
}
if u.Scheme != "https" || u.Host == "" {
return fmt.Errorf("must be an https URL with a host (got %q)", s)
}
if u.Path != "" && u.Path != "/" || u.RawQuery != "" || u.Fragment != "" {
return fmt.Errorf("must have no path, query or fragment; the client appends /api2/json (got %q)", s)
}
return nil
}
+250
View File
@@ -0,0 +1,250 @@
package provider
import (
"os"
"slices"
"strings"
"testing"
"sigs.k8s.io/yaml"
)
const validProxmoxYAML = `
providers:
- name: proxmox-home
type: proxmox
proxmox:
url: https://192.168.0.8:8006
tokenFile: /etc/proxy-operator/proxmox/token
caFile: /etc/proxy-operator/proxmox/ca.pem
nodes: [pv1-1, pv1-2]
isoStorage: proxy-seeds
storage: local-lvm
bridge: vmbr0
vlanTag: 4
vmidRange: {min: 150000, max: 150999}
templates:
debian-13: {vmid: 9001}
machineTypes:
small: {cores: 1, memoryMiB: 1024}
large: {cores: 4, memoryMiB: 4096, diskGiB: 20}
requestTimeoutSeconds: 45
installGuestAgent: false
`
func TestLoadConfig_proxmoxValid(t *testing.T) {
t.Parallel()
cfg, err := LoadConfig([]byte(validProxmoxYAML))
if err != nil {
t.Fatalf("LoadConfig() error = %v, want nil", err)
}
c := cfg.Providers[0].Proxmox
if c == nil {
t.Fatal("providers[0].proxmox = nil")
}
if c.URL != "https://192.168.0.8:8006" || c.ISOStorage != "proxy-seeds" || c.VLANTag != 4 {
t.Errorf("proxmox = %+v, want url, isoStorage and vlanTag parsed", c)
}
if len(c.Nodes) != 2 || c.Nodes[1] != "pv1-2" {
t.Errorf("nodes = %v, want [pv1-1 pv1-2]", c.Nodes)
}
if c.VMIDRange == nil || c.VMIDRange.Min != 150000 || c.VMIDRange.Max != 150999 {
t.Errorf("vmidRange = %+v, want 150000 to 150999", c.VMIDRange)
}
if c.Templates["debian-13"].VMID != 9001 {
t.Errorf("templates[debian-13] = %+v, want vmid 9001", c.Templates["debian-13"])
}
if mt := c.MachineTypes["large"]; mt.Cores != 4 || mt.MemoryMiB != 4096 || mt.DiskGiB != 20 {
t.Errorf("machineTypes[large] = %+v, want {4 4096 20}", mt)
}
if c.InstallGuestAgent == nil || *c.InstallGuestAgent {
t.Errorf("installGuestAgent = %v, want explicit false", c.InstallGuestAgent)
}
}
func TestValidateProxmox_table(t *testing.T) {
t.Parallel()
block := func(body string) string {
return "providers:\n - name: proxmox-home\n type: proxmox\n proxmox:\n" + indent(body, 6)
}
const base = `
url: https://192.168.0.8:8006
tokenFile: /t
nodes: [pv1-1]
isoStorage: proxy-seeds
templates: {debian-13: {vmid: 9001}}
machineTypes: {small: {cores: 1, memoryMiB: 1024}}
`
with := func(old, repl string) string {
if !strings.Contains(base, old) {
t.Fatalf("base has no %q", old)
}
return block(strings.Replace(base, old, repl, 1))
}
tests := []struct {
name string
yaml string
wantErrSub string
}{
{name: "valid minimal", yaml: block(base)},
{name: "valid full", yaml: validProxmoxYAML},
{name: "valid with trailing slash", yaml: with("url: https://192.168.0.8:8006", "url: https://192.168.0.8:8006/")},
{name: "missing block", yaml: "providers:\n - name: proxmox-home\n type: proxmox\n", wantErrSub: "proxmox block is required"},
{name: "missing url", yaml: with("url: https://192.168.0.8:8006\n", ""), wantErrSub: "proxmox.url is required"},
{name: "http url", yaml: with("https://", "http://"), wantErrSub: "must be an https URL"},
{name: "url with api path", yaml: with("8006", "8006/api2/json"), wantErrSub: "must have no path"},
{name: "missing tokenFile", yaml: with("tokenFile: /t\n", ""), wantErrSub: "proxmox.tokenFile is required"},
{name: "no nodes", yaml: with("nodes: [pv1-1]", "nodes: []"), wantErrSub: "proxmox.nodes must list at least one node"},
{name: "duplicate node", yaml: with("nodes: [pv1-1]", "nodes: [pv1-1, pv1-1]"), wantErrSub: `duplicate node "pv1-1"`},
{name: "node with a slash", yaml: with("nodes: [pv1-1]", "nodes: [pv1/1]"), wantErrSub: "is not a Proxmox node name"},
{name: "node in uppercase", yaml: with("nodes: [pv1-1]", "nodes: [PV1]"), wantErrSub: "is not a Proxmox node name"},
{name: "missing isoStorage", yaml: with("isoStorage: proxy-seeds\n", ""), wantErrSub: "proxmox.isoStorage is required"},
{name: "bad isoStorage id", yaml: with("isoStorage: proxy-seeds", "isoStorage: 1seeds"), wantErrSub: "proxmox.isoStorage \"1seeds\" is not a Proxmox storage id"},
{name: "bad storage id", yaml: with("isoStorage: proxy-seeds", "isoStorage: proxy-seeds\nstorage: a/b"), wantErrSub: "proxmox.storage \"a/b\""},
{name: "vlanTag too large", yaml: with("isoStorage: proxy-seeds", "isoStorage: proxy-seeds\nvlanTag: 4095"), wantErrSub: "proxmox.vlanTag must be between 1 and 4094"},
{name: "vlanTag negative", yaml: with("isoStorage: proxy-seeds", "isoStorage: proxy-seeds\nvlanTag: -1"), wantErrSub: "proxmox.vlanTag"},
{name: "vmidRange min below 100", yaml: with("isoStorage: proxy-seeds", "isoStorage: proxy-seeds\nvmidRange: {min: 50, max: 500}"), wantErrSub: "proxmox.vmidRange must satisfy"},
{name: "vmidRange min above max", yaml: with("isoStorage: proxy-seeds", "isoStorage: proxy-seeds\nvmidRange: {min: 2000, max: 1000}"), wantErrSub: "proxmox.vmidRange must satisfy"},
{name: "vmidRange max too large", yaml: with("isoStorage: proxy-seeds", "isoStorage: proxy-seeds\nvmidRange: {min: 100, max: 1000000000}"), wantErrSub: "proxmox.vmidRange must satisfy"},
{name: "no templates", yaml: with("templates: {debian-13: {vmid: 9001}}", "templates: {}"), wantErrSub: "proxmox.templates must define at least one template"},
{name: "template vmid below 100", yaml: with("vmid: 9001", "vmid: 99"), wantErrSub: `proxmox.templates["debian-13"].vmid must be at least 100`},
{name: "no machineTypes", yaml: with("machineTypes: {small: {cores: 1, memoryMiB: 1024}}", "machineTypes: {}"), wantErrSub: "proxmox.machineTypes must define at least one profile"},
{name: "cores zero", yaml: with("cores: 1", "cores: 0"), wantErrSub: `proxmox.machineTypes["small"] needs cores > 0`},
{name: "memory zero", yaml: with("memoryMiB: 1024", "memoryMiB: 0"), wantErrSub: "memoryMiB > 0"},
{name: "negative diskGiB", yaml: with("memoryMiB: 1024", "memoryMiB: 1024, diskGiB: -5"), wantErrSub: "diskGiB must not be negative"},
{name: "negative timeout", yaml: with("isoStorage: proxy-seeds", "isoStorage: proxy-seeds\nrequestTimeoutSeconds: -1"), wantErrSub: "proxmox.requestTimeoutSeconds must not be negative"},
{name: "unknown key is rejected", yaml: with("isoStorage: proxy-seeds", "isoStorage: proxy-seeds\nsshKeyFile: /k"), wantErrSub: "parsing providers config"},
}
for _, tc := range tests {
t.Run(tc.name, func(t *testing.T) {
t.Parallel()
_, err := LoadConfig([]byte(tc.yaml))
if tc.wantErrSub == "" {
if err != nil {
t.Fatalf("LoadConfig() error = %v, want nil", err)
}
return
}
if err == nil {
t.Fatalf("LoadConfig() error = nil, want error containing %q", tc.wantErrSub)
}
if !strings.Contains(err.Error(), tc.wantErrSub) {
t.Errorf("LoadConfig() error = %q, want substring %q", err.Error(), tc.wantErrSub)
}
})
}
}
// commentedEntry cuts the commented-out providers entry named name out of a
// config file and strips its "# " at the entry's own indent, so a sample
// that is shipped commented out is still loaded by a test.
func commentedEntry(t *testing.T, path, name string) string {
t.Helper()
data, err := os.ReadFile(path)
if err != nil {
t.Fatalf("reading %s: %v", path, err)
}
var out []string
var prefix string
for line := range strings.SplitSeq(string(data), "\n") {
if prefix == "" {
if strings.TrimSpace(line) == "# - name: "+name {
prefix = line[:strings.Index(line, "#")]
out = append(out, prefix+"- name: "+name)
}
continue
}
if !strings.HasPrefix(line, prefix+"# ") {
break
}
out = append(out, prefix+strings.TrimPrefix(line, prefix+"# "))
}
if prefix == "" {
t.Fatalf("%s has no commented entry %q", path, name)
}
return strings.Join(out, "\n") + "\n"
}
func TestLoadConfig_proxmoxSampleEntry(t *testing.T) {
t.Parallel()
for name, path := range map[string]string{
"sample": "../../config/samples/providers-config.yaml",
"configmap": "../../config/manager/providers_config.yaml",
} {
t.Run(name, func(t *testing.T) {
t.Parallel()
cfg, err := LoadConfig([]byte("providers:\n" + commentedEntry(t, path, "proxmox-home")))
if err != nil {
t.Fatalf("LoadConfig(proxmox-home from %s) error = %v, want nil", path, err)
}
p := cfg.Providers[0]
// LoadConfig already refused a proxmox block under another type.
if p.Name != "proxmox-home" || p.Proxmox == nil {
t.Fatalf("provider %q has proxmox block %v, want proxmox-home with one", p.Name, p.Proxmox)
}
c := p.Proxmox
if c.URL != "https://192.168.0.8:8006" {
t.Errorf("url = %q, want https://192.168.0.8:8006 (the address in the node certificate)", c.URL)
}
if c.TokenFile != "/etc/proxy-operator/proxmox/token" || c.CAFile != "/etc/proxy-operator/proxmox/ca.pem" {
t.Errorf("tokenFile, caFile = %q, %q, want the proxmox-token Secret's mount paths", c.TokenFile, c.CAFile)
}
if c.InsecureSkipTLSVerify {
t.Error("insecureSkipTLSVerify = true, want it left commented out")
}
if len(c.Nodes) != 1 || c.Nodes[0] != "pv1-1" {
t.Errorf("nodes = %v, want [pv1-1]", c.Nodes)
}
if c.ISOStorage != "proxy-seeds" || c.Storage != "" || c.Bridge != "" || c.VLANTag != 4 {
t.Errorf("isoStorage, storage, bridge, vlanTag = %q, %q, %q, %d, want proxy-seeds, \"\", \"\", 4", c.ISOStorage, c.Storage, c.Bridge, c.VLANTag)
}
if len(c.Templates) != 1 || c.Templates["debian-13"].VMID != 9001 {
t.Errorf("templates = %v, want {debian-13: {vmid: 9001}}", c.Templates)
}
if mt, ok := c.MachineTypes["small"]; !ok || mt.Cores != 1 || mt.MemoryMiB != 1024 || mt.DiskGiB != 0 {
t.Errorf("machineTypes[small] = %+v (present %v), want {1 1024 0}", mt, ok)
}
if c.VMIDRange != nil || c.RequestTimeoutSeconds != 0 || c.InstallGuestAgent != nil || c.VendorData != "" {
t.Errorf("optional fields set: %+v, want vmidRange, requestTimeoutSeconds, installGuestAgent and vendorData left commented out", c)
}
})
}
}
func TestLoadConfig_proxmoxSampleProxyMatchesSampleEntry(t *testing.T) {
t.Parallel()
cfg, err := LoadConfig([]byte("providers:\n" + commentedEntry(t, "../../config/samples/providers-config.yaml", "proxmox-home")))
if err != nil {
t.Fatalf("LoadConfig() error = %v, want nil", err)
}
data, err := os.ReadFile("../../config/samples/proxy_proxmox.yaml")
if err != nil {
t.Fatalf("reading the sample Proxy: %v", err)
}
var px struct {
Spec struct {
Provider string `json:"provider"`
Placement struct {
Zone string `json:"zone"`
MachineType string `json:"machineType"`
Image string `json:"image"`
} `json:"placement"`
} `json:"spec"`
}
if err := yaml.Unmarshal(data, &px); err != nil {
t.Fatalf("parsing the sample Proxy: %v", err)
}
c := cfg.Providers[0].Proxmox
if px.Spec.Provider != cfg.Providers[0].Name {
t.Errorf("spec.provider = %q, want %q", px.Spec.Provider, cfg.Providers[0].Name)
}
if !slices.Contains(c.Nodes, px.Spec.Placement.Zone) {
t.Errorf("placement.zone = %q, not in nodes %v", px.Spec.Placement.Zone, c.Nodes)
}
if _, ok := c.MachineTypes[px.Spec.Placement.MachineType]; !ok {
t.Errorf("placement.machineType = %q, not in machineTypes", px.Spec.Placement.MachineType)
}
if _, ok := c.Templates[px.Spec.Placement.Image]; !ok {
t.Errorf("placement.image = %q, not in templates", px.Spec.Placement.Image)
}
}
+20
View File
@@ -138,6 +138,26 @@ providers:
sshKeyFile: /key`,
wantErrSub: "type is kvm2 but a kvm block is set",
},
{
name: "proxmox type with kvm block",
yaml: `
providers:
- name: p1
type: proxmox
kvm:
sshKeyFile: /key`,
wantErrSub: "type is proxmox but a kvm block is set",
},
{
name: "kvm type with proxmox block",
yaml: `
providers:
- name: p1
type: kvm
proxmox:
url: https://pve:8006`,
wantErrSub: "type is kvm but a proxmox block is set",
},
{
name: "strict mode rejects unknown top-level key",
yaml: `
+60
View File
@@ -0,0 +1,60 @@
// Package keyedlock provides one-slot semaphores by key, shared by the
// providers that serialise work per instance name or per cached image
// (kvm2, proxmox).
package keyedlock
import (
"context"
"sync"
)
// Locks is a set of one-slot semaphores by key: a semaphore rather than a
// mutex so a waiter can give up with its ctx. Each entry counts the callers
// holding or waiting for it and is dropped from the map at zero, so the map
// does not grow with every key ever seen while two waiters on one key still
// always share one semaphore. The zero value is not usable; call New.
type Locks struct {
mu sync.Mutex
locks map[string]*lock
}
type lock struct {
sem chan struct{}
refs int
}
func New() *Locks {
return &Locks{locks: make(map[string]*lock)}
}
// Acquire takes the lock for key, or gives up with ctx's error. The
// returned func releases it and must be called exactly once.
func (k *Locks) Acquire(ctx context.Context, key string) (release func(), err error) {
k.mu.Lock()
l, ok := k.locks[key]
if !ok {
l = &lock{sem: make(chan struct{}, 1)}
k.locks[key] = l
}
l.refs++
k.mu.Unlock()
select {
case l.sem <- struct{}{}:
return func() {
<-l.sem
k.unref(key, l)
}, nil
case <-ctx.Done():
k.unref(key, l)
return nil, ctx.Err()
}
}
func (k *Locks) unref(key string, l *lock) {
k.mu.Lock()
defer k.mu.Unlock()
l.refs--
if l.refs == 0 {
delete(k.locks, key)
}
}
@@ -0,0 +1,90 @@
package keyedlock
import (
"context"
"errors"
"sync"
"sync/atomic"
"testing"
"time"
)
func lockCount(k *Locks) int {
k.mu.Lock()
defer k.mu.Unlock()
return len(k.locks)
}
func TestKeyedLocks_entryRemovedWhenUncontended(t *testing.T) {
t.Parallel()
k := New()
release, err := k.Acquire(context.Background(), "a")
if err != nil {
t.Fatalf("acquire error = %v", err)
}
if lockCount(k) != 1 {
t.Errorf("entries while held = %d, want 1", lockCount(k))
}
release()
if lockCount(k) != 0 {
t.Errorf("entries after release = %d, want 0", lockCount(k))
}
if _, err := k.Acquire(context.Background(), "a"); err != nil {
t.Fatalf("acquire after release error = %v", err)
}
}
func TestKeyedLocks_waitersShareOneLock(t *testing.T) {
t.Parallel()
k := New()
const n = 20
var inside, overlaps atomic.Int32
var wg sync.WaitGroup
for range n {
wg.Go(func() {
release, err := k.Acquire(context.Background(), "shared")
if err != nil {
t.Errorf("acquire error = %v", err)
return
}
if inside.Add(1) > 1 {
overlaps.Add(1)
}
time.Sleep(time.Millisecond)
inside.Add(-1)
release()
})
}
wg.Wait()
if overlaps.Load() != 0 {
t.Errorf("%d holders overlapped on one key", overlaps.Load())
}
if lockCount(k) != 0 {
t.Errorf("entries after every holder released = %d, want 0", lockCount(k))
}
}
func TestKeyedLocks_waiterGivesUpWithContext(t *testing.T) {
t.Parallel()
k := New()
release, err := k.Acquire(context.Background(), "a")
if err != nil {
t.Fatalf("acquire error = %v", err)
}
ctx, cancel := context.WithTimeout(context.Background(), 20*time.Millisecond)
defer cancel()
start := time.Now()
if _, err := k.Acquire(ctx, "a"); !errors.Is(err, context.DeadlineExceeded) {
t.Fatalf("waiter error = %v, want DeadlineExceeded", err)
}
if time.Since(start) > time.Second {
t.Errorf("waiter took %v, want to give up with its ctx", time.Since(start))
}
if lockCount(k) != 1 {
t.Errorf("entries with the holder still in = %d, want 1: the waiter's exit must not drop the holder's entry", lockCount(k))
}
release()
if lockCount(k) != 0 {
t.Errorf("entries after the holder released = %d, want 0", lockCount(k))
}
}
+3 -2
View File
@@ -11,6 +11,7 @@ import (
"github.com/go-logr/logr"
"gitea.home.hrajfrisbee.cz/kacerr/egress-proxies-operator/internal/provider"
"gitea.home.hrajfrisbee.cz/kacerr/egress-proxies-operator/internal/provider/nocloud"
)
// The wait after DomainCreate (spec entry 18): Create returns only once the
@@ -191,12 +192,12 @@ func (p *Provider) writeSeed(ctx context.Context, log logr.Logger, be backend, p
return libvirt.StorageVol{}, err
}
installAgent := p.cfg.InstallGuestAgent == nil || *p.cfg.InstallGuestAgent
vendorData, err := buildVendorData(p.cfg.VendorData, installAgent)
vendorData, err := nocloud.BuildVendorData(p.cfg.VendorData, installAgent)
if err != nil {
return libvirt.StorageVol{}, permanentf("building vendor-data for %s: %v", req.Name, err)
}
var iso bytes.Buffer
if err := buildSeedISO(req.Name, req.CloudInit, vendorData, &iso); err != nil {
if err := nocloud.BuildSeedISO(req.Name, req.CloudInit, vendorData, &iso); err != nil {
return libvirt.StorageVol{}, permanentf("building seed ISO for %s: %v", req.Name, err)
}
xmlDoc, err := buildVolumeXML(volumeSpec{Name: name, CapacityBytes: int64(iso.Len()), Format: provider.KVMFormatRaw})
+3 -2
View File
@@ -12,6 +12,7 @@ import (
"github.com/digitalocean/go-libvirt"
"gitea.home.hrajfrisbee.cz/kacerr/egress-proxies-operator/internal/provider"
"gitea.home.hrajfrisbee.cz/kacerr/egress-proxies-operator/internal/provider/nocloud"
)
func TestCreate_repeatReturnsSameIDAndCreatesNothing(t *testing.T) {
@@ -95,7 +96,7 @@ func TestCreate_resumesAfterEachPartialStep(t *testing.T) {
},
check: func(t *testing.T, fx *fixture) {
got := fx.fake.vol(seed)
if got == nil || len(got.content) <= 3 || !bytes.Contains(got.content, []byte(seedLabel)) {
if got == nil || len(got.content) <= 3 || !bytes.Contains(got.content, []byte(nocloud.SeedLabel)) {
t.Errorf("seed volume not rebuilt")
}
},
@@ -461,7 +462,7 @@ func TestCreate_seedAlwaysRebuilt(t *testing.T) {
if got := fx.fake.callCount("StorageVolUpload"); got != uploads+1 {
t.Errorf("StorageVolUpload calls = %d, want %d", got, uploads+1)
}
if len(second.content) != len(first.content) || !bytes.Contains(second.content, []byte(seedLabel)) {
if len(second.content) != len(first.content) || !bytes.Contains(second.content, []byte(nocloud.SeedLabel)) {
t.Errorf("rebuilt seed is %d bytes, first was %d; want a complete seed of the same size", len(second.content), len(first.content))
}
}
+1 -1
View File
@@ -73,7 +73,7 @@ func resolveImage(entry string, cfg provider.KVMConfig, image string) (imageSour
// returns promptly (contract §1.5) instead of waiting out a 20-minute
// transfer.
func (p *Provider) lockImage(ctx context.Context, hostName, volume string) (func(), error) {
release, err := p.imageLocks.acquire(ctx, hostName+"/"+volume)
release, err := p.imageLocks.Acquire(ctx, hostName+"/"+volume)
if err != nil {
return nil, fmt.Errorf("%w: waiting for image %s on host %s: %w", provider.ErrTransient, volume, hostName, err)
}
+6 -57
View File
@@ -9,13 +9,13 @@ import (
"fmt"
"net/http"
"os"
"sync"
"time"
"github.com/go-logr/logr"
logf "sigs.k8s.io/controller-runtime/pkg/log"
"gitea.home.hrajfrisbee.cz/kacerr/egress-proxies-operator/internal/provider"
"gitea.home.hrajfrisbee.cz/kacerr/egress-proxies-operator/internal/provider/keyedlock"
"gitea.home.hrajfrisbee.cz/kacerr/egress-proxies-operator/internal/tracing"
)
@@ -41,12 +41,12 @@ type Provider struct {
runningPoll time.Duration
// imageLocks serialises image fetches per (host, volume), spec entry 19.
imageLocks *keyedLocks
imageLocks *keyedlock.Locks
// nameLocks serialises Create and Delete per instance name, spec entry
// 19, so two Deletes never race on destroy and undefine and a Delete
// cannot interleave with a Create's steps.
nameLocks *keyedLocks
nameLocks *keyedlock.Locks
}
// providerOption adjusts what newProvider builds; tests inject an HTTP
@@ -111,8 +111,8 @@ func newProvider(name string, cfg provider.KVMConfig, d dialer, opts ...provider
http: http.DefaultClient,
runningWait: defaultRunningWait,
runningPoll: defaultRunningPoll,
imageLocks: newKeyedLocks(),
nameLocks: newKeyedLocks(),
imageLocks: keyedlock.New(),
nameLocks: keyedlock.New(),
}
for _, opt := range opts {
opt(p)
@@ -187,60 +187,9 @@ func (p *Provider) wrapErr(op, id string, err error) error {
// so a caller whose ctx ends while another caller's Create is downloading
// an image returns promptly.
func (p *Provider) lockName(ctx context.Context, name string) (func(), error) {
release, err := p.nameLocks.acquire(ctx, name)
release, err := p.nameLocks.Acquire(ctx, name)
if err != nil {
return nil, fmt.Errorf("%w: waiting for the lock on %s: %w", provider.ErrTransient, name, err)
}
return release, nil
}
// keyedLocks is a set of one-slot semaphores by key (spec entry 19): a
// semaphore rather than a mutex so a waiter can give up with its ctx. Each
// entry counts the callers holding or waiting for it and is dropped from
// the map at zero, so the map does not grow with every name ever seen while
// two waiters on one key still always share one semaphore.
type keyedLocks struct {
mu sync.Mutex
locks map[string]*keyedLock
}
type keyedLock struct {
sem chan struct{}
refs int
}
func newKeyedLocks() *keyedLocks {
return &keyedLocks{locks: make(map[string]*keyedLock)}
}
// acquire takes the lock for key, or gives up with ctx's error. The
// returned func releases it and must be called exactly once.
func (k *keyedLocks) acquire(ctx context.Context, key string) (func(), error) {
k.mu.Lock()
l, ok := k.locks[key]
if !ok {
l = &keyedLock{sem: make(chan struct{}, 1)}
k.locks[key] = l
}
l.refs++
k.mu.Unlock()
select {
case l.sem <- struct{}{}:
return func() {
<-l.sem
k.unref(key, l)
}, nil
case <-ctx.Done():
k.unref(key, l)
return nil, ctx.Err()
}
}
func (k *keyedLocks) unref(key string, l *keyedLock) {
k.mu.Lock()
defer k.mu.Unlock()
l.refs--
if l.refs == 0 {
delete(k.locks, key)
}
}
-81
View File
@@ -5,7 +5,6 @@ import (
"errors"
"strings"
"sync"
"sync/atomic"
"testing"
"time"
@@ -267,86 +266,6 @@ func TestWithDefaults_doesNotAliasInput(t *testing.T) {
}
}
func lockCount(k *keyedLocks) int {
k.mu.Lock()
defer k.mu.Unlock()
return len(k.locks)
}
func TestKeyedLocks_entryRemovedWhenUncontended(t *testing.T) {
t.Parallel()
k := newKeyedLocks()
release, err := k.acquire(context.Background(), "a")
if err != nil {
t.Fatalf("acquire error = %v", err)
}
if lockCount(k) != 1 {
t.Errorf("entries while held = %d, want 1", lockCount(k))
}
release()
if lockCount(k) != 0 {
t.Errorf("entries after release = %d, want 0", lockCount(k))
}
if _, err := k.acquire(context.Background(), "a"); err != nil {
t.Fatalf("acquire after release error = %v", err)
}
}
func TestKeyedLocks_waitersShareOneLock(t *testing.T) {
t.Parallel()
k := newKeyedLocks()
const n = 20
var inside, overlaps atomic.Int32
var wg sync.WaitGroup
for range n {
wg.Go(func() {
release, err := k.acquire(context.Background(), "shared")
if err != nil {
t.Errorf("acquire error = %v", err)
return
}
if inside.Add(1) > 1 {
overlaps.Add(1)
}
time.Sleep(time.Millisecond)
inside.Add(-1)
release()
})
}
wg.Wait()
if overlaps.Load() != 0 {
t.Errorf("%d holders overlapped on one key", overlaps.Load())
}
if lockCount(k) != 0 {
t.Errorf("entries after every holder released = %d, want 0", lockCount(k))
}
}
func TestKeyedLocks_waiterGivesUpWithContext(t *testing.T) {
t.Parallel()
k := newKeyedLocks()
release, err := k.acquire(context.Background(), "a")
if err != nil {
t.Fatalf("acquire error = %v", err)
}
ctx, cancel := context.WithTimeout(context.Background(), 20*time.Millisecond)
defer cancel()
start := time.Now()
if _, err := k.acquire(ctx, "a"); !errors.Is(err, context.DeadlineExceeded) {
t.Fatalf("waiter error = %v, want DeadlineExceeded", err)
}
if time.Since(start) > time.Second {
t.Errorf("waiter took %v, want to give up with its ctx", time.Since(start))
}
if lockCount(k) != 1 {
t.Errorf("entries with the holder still in = %d, want 1: the waiter's exit must not drop the holder's entry", lockCount(k))
}
release()
if lockCount(k) != 0 {
t.Errorf("entries after the holder released = %d, want 0", lockCount(k))
}
}
func TestWrapErr_carriesProviderNameOpAndID(t *testing.T) {
t.Parallel()
p := newProvider("kvm2-home", twoHosts(), &stubDialer{})
@@ -1,4 +1,7 @@
package kvm2
// Package nocloud builds cloud-init NoCloud seed ISOs in process, shared by
// the providers that attach a seed as a CD-ROM (kvm2, proxmox). It never
// parses or logs the payload.
package nocloud
import (
"bytes"
@@ -12,19 +15,22 @@ import (
"github.com/kdomanski/iso9660"
)
// seedLabel is the volume label cloud-init's NoCloud datasource looks for.
const seedLabel = "cidata"
// SeedLabel is the volume label cloud-init's NoCloud datasource looks for.
const SeedLabel = "cidata"
// guestAgentScript is the vendor-data payload that gets qemu-guest-agent
// running so the provider can read the VM's address through libvirt. It is
// ErrEmptyName is returned by BuildSeedISO for an empty instance name.
var ErrEmptyName = errors.New("seed ISO: instance name is empty")
// GuestAgentScript is the vendor-data payload that gets qemu-guest-agent
// running so the provider can read the VM's address from the hypervisor. It is
// a shell script and not a cloud-config on purpose: cloud-init merges a
// vendor-data cloud-config under the user-data with mergers that keep the
// user-data's value for every top-level key and never append lists, so a
// Proxy with its own packages: would silently lose the agent. Scripts
// bypass the merge.
const guestAgentScript = `#!/bin/sh
const GuestAgentScript = `#!/bin/sh
# Added by egress-proxies-operator: installs and starts qemu-guest-agent so
# the operator can learn this VM's IP address through libvirt.
# the operator can learn this VM's IP address from the hypervisor.
# To leave this out, set installGuestAgent: false in the provider's config.
set -e
if command -v qemu-ga >/dev/null 2>&1 || [ -x /usr/sbin/qemu-ga ]; then
@@ -53,15 +59,15 @@ else
fi
`
// buildSeedISO writes a NoCloud seed to w: user-data byte for byte,
// BuildSeedISO writes a NoCloud seed to w: user-data byte for byte,
// meta-data naming the instance, and vendor-data (an empty file when there
// is none, so cloud-init does not look elsewhere). iso9660.NewWriter stages
// the files in os.MkdirTemp("", ""), that is under os.TempDir(): $TMPDIR
// when set, else /tmp, the manager's 64Mi emptyDir. The staging dir is
// removed before returning. Nothing about the payload is logged here.
func buildSeedISO(name, userData, vendorData string, w io.Writer) (err error) {
func BuildSeedISO(name, userData, vendorData string, w io.Writer) (err error) {
if name == "" {
return permanentf("seed ISO: instance name is empty")
return ErrEmptyName
}
iw, err := iso9660.NewWriter()
if err != nil {
@@ -83,30 +89,30 @@ func buildSeedISO(name, userData, vendorData string, w io.Writer) (err error) {
return fmt.Errorf("adding %s to seed ISO: %w", f.name, err)
}
}
if err := iw.WriteTo(w, seedLabel); err != nil {
if err := iw.WriteTo(w, SeedLabel); err != nil {
return fmt.Errorf("writing seed ISO: %w", err)
}
return nil
}
// buildVendorData composes the seed's vendor-data from the configured
// BuildVendorData composes the seed's vendor-data from the configured
// document and the guest agent script. With both, they travel as separate
// parts of a MIME multipart, so cloud-init handles each by its own type; the
// MIME-Version header comes first because cloud-init detects multipart by
// finding "mime-version:" within the first 4096 bytes.
func buildVendorData(configured string, installAgent bool) (string, error) {
func BuildVendorData(configured string, installAgent bool) (string, error) {
switch {
case !installAgent:
return configured, nil
case strings.TrimSpace(configured) == "":
return guestAgentScript, nil
return GuestAgentScript, nil
}
var buf bytes.Buffer
mw := multipart.NewWriter(&buf)
fmt.Fprintf(&buf, "MIME-Version: 1.0\r\nContent-Type: multipart/mixed; boundary=%q\r\n\r\n", mw.Boundary())
parts := []struct{ contentType, body string }{
{"text/plain", configured},
{"text/x-shellscript", guestAgentScript},
{"text/x-shellscript", GuestAgentScript},
}
for _, p := range parts {
pw, err := mw.CreatePart(textproto.MIMEHeader{"Content-Type": {p.contentType}})
@@ -1,8 +1,9 @@
package kvm2
package nocloud
import (
"bufio"
"bytes"
"errors"
"io"
"mime"
"mime/multipart"
@@ -18,8 +19,8 @@ import (
func readSeed(t *testing.T, name, userData, vendorData string) (map[string]string, string) {
t.Helper()
var buf bytes.Buffer
if err := buildSeedISO(name, userData, vendorData, &buf); err != nil {
t.Fatalf("buildSeedISO: %v", err)
if err := BuildSeedISO(name, userData, vendorData, &buf); err != nil {
t.Fatalf("BuildSeedISO: %v", err)
}
img, err := iso9660.OpenImage(bytes.NewReader(buf.Bytes()))
if err != nil {
@@ -57,8 +58,8 @@ func TestSeedISO_filesAndLabel(t *testing.T) {
userData := "#cloud-config\npackages: [squid]\n"
vendorData := "#!/bin/sh\necho hi\n"
files, label := readSeed(t, "proxy-abc", userData, vendorData)
if label != seedLabel {
t.Errorf("label = %q, want %q", label, seedLabel)
if label != SeedLabel {
t.Errorf("label = %q, want %q", label, SeedLabel)
}
want := map[string]string{
"user-data": userData,
@@ -117,8 +118,8 @@ func TestSeedISO_tempDirRemoved(t *testing.T) {
t.Fatalf("os.TempDir() = %q, want %q", got, tmp)
}
var buf bytes.Buffer
if err := buildSeedISO("proxy-abc", "#cloud-config\n", guestAgentScript, &buf); err != nil {
t.Fatalf("buildSeedISO: %v", err)
if err := BuildSeedISO("proxy-abc", "#cloud-config\n", GuestAgentScript, &buf); err != nil {
t.Fatalf("BuildSeedISO: %v", err)
}
if buf.Len() == 0 {
t.Fatal("seed ISO is empty")
@@ -151,8 +152,8 @@ func TestSeedISO_emptyVendorDataStillHasFile(t *testing.T) {
func TestSeedISO_rejectsEmptyName(t *testing.T) {
t.Parallel()
var buf bytes.Buffer
if err := buildSeedISO("", "#cloud-config\n", "", &buf); err == nil {
t.Fatal("buildSeedISO with empty name succeeded")
if err := BuildSeedISO("", "#cloud-config\n", "", &buf); !errors.Is(err, ErrEmptyName) {
t.Fatalf("BuildSeedISO with empty name error = %v, want ErrEmptyName", err)
}
if buf.Len() != 0 {
t.Errorf("wrote %d bytes alongside the error", buf.Len())
@@ -161,7 +162,7 @@ func TestSeedISO_rejectsEmptyName(t *testing.T) {
func TestVendorData_isShellScriptNotCloudConfig(t *testing.T) {
t.Parallel()
got, err := buildVendorData("", true)
got, err := BuildVendorData("", true)
if err != nil {
t.Fatal(err)
}
@@ -193,7 +194,7 @@ func bufioReader(s string) *bufio.Reader {
func TestVendorData_multipartWhenExtraProvided(t *testing.T) {
t.Parallel()
configured := "#cloud-config\nntp:\n enabled: true\n"
got, err := buildVendorData(configured, true)
got, err := BuildVendorData(configured, true)
if err != nil {
t.Fatal(err)
}
@@ -229,7 +230,7 @@ func TestVendorData_multipartWhenExtraProvided(t *testing.T) {
}
parts = append(parts, part{p.Header.Get("Content-Type"), string(body)})
}
want := []part{{"text/plain", configured}, {"text/x-shellscript", guestAgentScript}}
want := []part{{"text/plain", configured}, {"text/x-shellscript", GuestAgentScript}}
if len(parts) != len(want) {
t.Fatalf("got %d parts, want %d", len(parts), len(want))
}
@@ -242,7 +243,7 @@ func TestVendorData_multipartWhenExtraProvided(t *testing.T) {
func TestVendorData_offWithoutConfiguredIsEmpty(t *testing.T) {
t.Parallel()
got, err := buildVendorData("", false)
got, err := BuildVendorData("", false)
if err != nil {
t.Fatal(err)
}
@@ -254,7 +255,7 @@ func TestVendorData_offWithoutConfiguredIsEmpty(t *testing.T) {
func TestVendorData_offWithConfiguredPassesThrough(t *testing.T) {
t.Parallel()
configured := "#cloud-config\nntp:\n enabled: true"
got, err := buildVendorData(configured, false)
got, err := BuildVendorData(configured, false)
if err != nil {
t.Fatal(err)
}
+119
View File
@@ -0,0 +1,119 @@
package proxmox
import (
"context"
"io"
)
// api is the seam over the Proxmox VE REST API: one method per endpoint the
// provider uses, ctx first (spec entry 17). Calls that start a Proxmox task
// return its UPID; the caller waits on it with TaskStatus.
type api interface {
Version(ctx context.Context) (versionInfo, error)
NodeStatus(ctx context.Context, node string) (nodeStatus, error)
StorageStatus(ctx context.Context, node, storage string) (storageStatus, error)
ClusterResources(ctx context.Context) ([]vmResource, error)
VMConfig(ctx context.Context, node string, vmid int) (vmConfig, error)
SetVMConfig(ctx context.Context, node string, vmid int, params map[string]string) error
Clone(ctx context.Context, node string, vmid int, p cloneParams) (upid string, err error)
Resize(ctx context.Context, node string, vmid int, disk, size string) (upid string, err error)
Start(ctx context.Context, node string, vmid int) (upid string, err error)
Stop(ctx context.Context, node string, vmid int) (upid string, err error)
VMStatus(ctx context.Context, node string, vmid int) (vmStatus, error)
Destroy(ctx context.Context, node string, vmid int) (upid string, err error)
TaskStatus(ctx context.Context, node, upid string) (taskStatus, error)
AgentNetworkInterfaces(ctx context.Context, node string, vmid int) ([]agentInterface, error)
UploadISO(ctx context.Context, node, storage, filename string, r io.Reader, size int64) (upid string, err error)
DeleteVolume(ctx context.Context, node, storage, volid string) (upid string, err error)
}
// versionInfo is GET /version.
type versionInfo struct {
Version string `json:"version"`
Release string `json:"release"`
RepoID string `json:"repoid"`
}
// nodeStatus is the part of GET /nodes/{node}/status the Checker reports.
type nodeStatus struct {
PVEVersion string `json:"pveversion"`
CurrentKernel struct {
Release string `json:"release"`
} `json:"current-kernel"`
}
// storageStatus is GET /nodes/{node}/storage/{storage}/status. Content is a
// comma-separated list such as "iso,import,backup,vztmpl".
type storageStatus struct {
Type string `json:"type"`
Content string `json:"content"`
Active int `json:"active"`
Enabled int `json:"enabled"`
Shared int `json:"shared"`
Total int64 `json:"total"`
Used int64 `json:"used"`
Avail int64 `json:"avail"`
}
// vmResource is one entry of GET /cluster/resources?type=vm. A VM being
// cloned is listed without a name, with lock "clone" and status "unknown"
// (spec entry 9). Tags are separated by ";".
type vmResource struct {
VMID int `json:"vmid"`
Name string `json:"name"`
Node string `json:"node"`
Type string `json:"type"`
Status string `json:"status"`
Lock string `json:"lock"`
Tags string `json:"tags"`
Template int `json:"template"`
}
// vmConfig is GET /nodes/{node}/qemu/{vmid}/config with every value
// rendered as a string: Proxmox sends some scalars as JSON numbers (cores,
// template) and others as strings (memory).
type vmConfig map[string]string
// vmStatus is GET /nodes/{node}/qemu/{vmid}/status/current. QMPStatus tells
// a paused VM from a running one; the resources listing cannot.
type vmStatus struct {
Status string `json:"status"`
QMPStatus string `json:"qmpstatus"`
Name string `json:"name"`
Lock string `json:"lock"`
Uptime int64 `json:"uptime"`
}
// taskStatus is GET /nodes/{node}/tasks/{upid}/status. ExitStatus is "OK"
// or the failure's text once Status is "stopped".
type taskStatus struct {
UPID string `json:"upid"`
Type string `json:"type"`
ID string `json:"id"`
Node string `json:"node"`
Status string `json:"status"`
ExitStatus string `json:"exitstatus"`
}
// agentInterface is one entry of the guest agent's network-get-interfaces
// result.
type agentInterface struct {
Name string `json:"name"`
IPAddresses []agentIP `json:"ip-addresses"`
}
type agentIP struct {
Address string `json:"ip-address"`
Type string `json:"ip-address-type"`
Prefix int `json:"prefix"`
}
// cloneParams are the parameters of POST .../clone the provider sets.
// Storage and Target are omitted when empty.
type cloneParams struct {
NewID int
Name string
Description string
Storage string
Target string
}
+97
View File
@@ -0,0 +1,97 @@
package proxmox
import (
"strings"
"testing"
)
// TestAPITypes_decodeCaptures decodes the Phase 0 responses into the seam's
// types, so a misspelled JSON tag fails here and not in a later phase.
func TestAPITypes_decodeCaptures(t *testing.T) {
t.Parallel()
t.Run("version", func(t *testing.T) {
t.Parallel()
if v := decodeData[versionInfo](t, "version"); v.Version != "9.2.20" || v.Release != "9.2" || v.RepoID == "" {
t.Errorf("version = %+v", v)
}
})
t.Run("node status", func(t *testing.T) {
t.Parallel()
n := decodeData[nodeStatus](t, "node_status")
if !strings.HasPrefix(n.PVEVersion, "pve-manager/9.2.20") || n.CurrentKernel.Release != "7.0.14-17-pve" {
t.Errorf("node status = %+v", n)
}
})
t.Run("storage status", func(t *testing.T) {
t.Parallel()
s := decodeData[storageStatus](t, "storage_status")
if s.Type != "dir" || !strings.Contains(s.Content, "iso") || s.Active != 1 || s.Avail <= 0 || s.Total < s.Avail {
t.Errorf("storage status = %+v", s)
}
})
t.Run("cluster resources", func(t *testing.T) {
t.Parallel()
rs := decodeData[[]vmResource](t, "cluster_resources")
byID := map[int]vmResource{}
for _, r := range rs {
byID[r.VMID] = r
}
if r := byID[9001]; r.Name != "debian-13-template" || r.Template != 1 || r.Node != "pv1-1" || r.Type != "qemu" || r.Status != "stopped" {
t.Errorf("resources[9001] = %+v", r)
}
if r := byID[100]; r.Template != 0 || r.Name != "ubuntu-vm-1" {
t.Errorf("resources[100] = %+v", r)
}
})
t.Run("cluster resources during clone", func(t *testing.T) {
t.Parallel()
for _, r := range decodeData[[]vmResource](t, "cluster_resources_during_clone") {
if r.VMID != 199999 {
continue
}
if r.Name != "" || r.Lock != "clone" || r.Status != "unknown" {
t.Errorf("mid-clone entry = %+v, want no name, lock clone, status unknown", r)
}
return
}
t.Error("no entry for VMID 199999 in the mid-clone capture")
})
t.Run("cluster resources tags", func(t *testing.T) {
t.Parallel()
for _, r := range decodeData[[]vmResource](t, "cluster_resources_during_destroy") {
if r.VMID == 199999 && r.Tags != "proxy_operator" {
t.Errorf("tags = %q, want proxy_operator", r.Tags)
}
}
})
t.Run("status current", func(t *testing.T) {
t.Parallel()
if s := decodeData[vmStatus](t, "status_current_paused"); s.Status != "running" || s.QMPStatus != "paused" {
t.Errorf("paused status = %+v", s)
}
if s := decodeData[vmStatus](t, "status_current_stopped"); s.Status != "stopped" || s.QMPStatus != "stopped" {
t.Errorf("stopped status = %+v", s)
}
})
t.Run("task status", func(t *testing.T) {
t.Parallel()
if s := decodeData[taskStatus](t, "task_status_stopped_error"); s.Status != "stopped" || s.ExitStatus != "shrinking disks is not supported" || s.Type != "resize" {
t.Errorf("failed task = %+v", s)
}
if s := decodeData[taskStatus](t, "task_status_running"); s.Status != "running" || s.ExitStatus != "" || s.Node != "pv1-1" {
t.Errorf("running task = %+v", s)
}
if s := decodeData[taskStatus](t, "task_status_volume_missing"); s.ExitStatus != "OK" {
t.Errorf("deleting a missing volume = %+v, want exitstatus OK", s)
}
})
t.Run("UPID responses", func(t *testing.T) {
t.Parallel()
for _, c := range []string{"clone_response", "start_response", "stop_response", "destroy_response", "resize_response", "upload_response", "content_delete_response"} {
if upid := decodeData[string](t, c); !strings.HasPrefix(upid, "UPID:pv1-1:") {
t.Errorf("%s = %q, want a UPID of node pv1-1", c, upid)
}
}
})
}
+224
View File
@@ -0,0 +1,224 @@
package proxmox
import (
"context"
"errors"
"fmt"
"io"
"maps"
"net"
"net/http"
"slices"
"strconv"
"strings"
"sync"
"time"
"gitea.home.hrajfrisbee.cz/kacerr/egress-proxies-operator/internal/provider"
)
var _ provider.Checker = (*Provider)(nil)
// Reasons only this provider can tell apart (spec entry 16). Each names a
// failure an admin fixes somewhere other than where the shared reasons
// point.
const (
// reasonCertificateInvalid: TLS verification failed or the CA file is
// unusable; fix url or caFile, not the host.
reasonCertificateInvalid = "CertificateInvalid"
// reasonCredentialsUnavailable: the token file is missing or malformed;
// fix the Secret, not Proxmox.
reasonCredentialsUnavailable = "CredentialsUnavailable"
// reasonStorageNotFound: isoStorage is missing, inactive, or holds no
// iso content on the node; fix the storage on the node.
reasonStorageNotFound = "StorageNotFound"
// reasonTemplateNotFound: a configured template VMID does not exist or
// is not a template; fix the template or the catalogue.
reasonTemplateNotFound = "TemplateNotFound"
)
// pveproxy's own answers when it cannot reach the node behind it.
const (
statusProxyConnect = 595
statusProxyBroken = 596
)
// Check verifies every configured node concurrently and returns one verdict
// per node in config order. The requests share the provider's HTTP client,
// but each is its own request, so a round that runs out of time cancels
// only its own requests and never a reconcile's.
func (p *Provider) Check(ctx context.Context) []provider.TargetStatus {
out := make([]provider.TargetStatus, len(p.cfg.Nodes))
var wg sync.WaitGroup
for i, node := range p.cfg.Nodes {
wg.Go(func() { out[i] = p.checkNode(ctx, node) })
}
wg.Wait()
return out
}
func (p *Provider) checkNode(ctx context.Context, node string) provider.TargetStatus {
start := time.Now()
log := p.logger(ctx).WithValues("node", node)
ts := provider.TargetStatus{
Name: node,
Address: p.cfg.URL,
Details: map[string]string{"isoStorage": p.cfg.ISOStorage},
}
err := p.probeNode(ctx, node, ts.Details)
ts.Latency = time.Since(start)
switch {
case err == nil:
ts.Ready = true
ts.Reason = provider.ReasonReachable
case ctx.Err() != nil:
// The round ran out of time, whichever request was running.
ts.Reason = provider.ReasonTimeout
ts.Message = err.Error()
default:
ts.Reason = checkReason(err)
ts.Message = err.Error()
}
log.V(1).Info("proxmox node checked", "ready", ts.Ready, "reason", ts.Reason, "latency", ts.Latency)
return ts
}
// probeNode runs spec entry 16's four requests and fills details as facts
// arrive, so a failure late in the sequence still reports what came before.
func (p *Provider) probeNode(ctx context.Context, node string, details map[string]string) error {
// The token file is read here only for its id: the secret after "="
// is dropped at once and never reaches details, logs or errors.
token, err := readToken(p.cfg.TokenFile)
if err != nil {
return err
}
tokenID, _, _ := strings.Cut(token, "=")
details["tokenID"] = tokenID
v, err := p.api.Version(ctx)
if err != nil {
return fmt.Errorf("reading the API version: %w", err)
}
details["pveVersion"] = v.Version
ns, err := p.api.NodeStatus(ctx, node)
if err != nil {
return fmt.Errorf("reading the status of node %s: %w", node, err)
}
details["kernel"] = ns.CurrentKernel.Release
if err := p.checkStorage(ctx, node, details); err != nil {
return err
}
return p.checkTemplates(ctx, details)
}
// checkStorage reads isoStorage on the node. A storage the cluster does
// not know is a 400 on the storage parameter (captured). A 500 is taken
// the same way, whatever its wording: Proxmox cannot use that storage on
// this node (not enabled here, or offline; not captured). The message
// carries Proxmox's text.
func (p *Provider) checkStorage(ctx context.Context, node string, details map[string]string) error {
storage := p.cfg.ISOStorage
st, err := p.api.StorageStatus(ctx, node, storage)
var ae *apiError
if errors.As(err, &ae) {
switch {
case isNoSuchStorage(ae):
return &checkFailure{reason: reasonStorageNotFound, msg: fmt.Sprintf("storage %s on node %s: %s", storage, node, ae.Errors["storage"])}
case ae.Status == http.StatusInternalServerError:
return &checkFailure{reason: reasonStorageNotFound, msg: fmt.Sprintf("storage %s on node %s: %s", storage, node, ae.Message)}
}
}
if err != nil {
return fmt.Errorf("reading storage %s on node %s: %w", storage, node, err)
}
details["isoStorageAvailGiB"] = formatGiB(st.Avail)
if st.Active == 0 {
return &checkFailure{reason: reasonStorageNotFound, msg: fmt.Sprintf("storage %s is not active on node %s", storage, node)}
}
if !slices.Contains(strings.Split(st.Content, ","), "iso") {
return &checkFailure{reason: reasonStorageNotFound, msg: fmt.Sprintf("storage %s on node %s does not hold iso content (content: %s)", storage, node, st.Content)}
}
return nil
}
// checkTemplates looks every catalogue entry up in the cluster's resources:
// a template may sit on any node, since a clone can target another.
func (p *Provider) checkTemplates(ctx context.Context, details map[string]string) error {
rs, err := p.api.ClusterResources(ctx)
if err != nil {
return fmt.Errorf("listing cluster resources: %w", err)
}
isTemplate := make(map[int]bool)
for _, r := range rs {
if r.Type == "qemu" {
isTemplate[r.VMID] = r.Template == 1
}
}
var problems []string
for _, key := range slices.Sorted(maps.Keys(p.cfg.Templates)) {
vmid := p.cfg.Templates[key].VMID
switch tmpl, found := isTemplate[vmid]; {
case !found:
problems = append(problems, fmt.Sprintf("template %q: VM %d does not exist", key, vmid))
case !tmpl:
problems = append(problems, fmt.Sprintf("template %q: VM %d is not a template", key, vmid))
}
}
details["templates"] = fmt.Sprintf("%d/%d", len(p.cfg.Templates)-len(problems), len(p.cfg.Templates))
if len(problems) > 0 {
return &checkFailure{reason: reasonTemplateNotFound, msg: strings.Join(problems, "; ")}
}
return nil
}
// checkFailure is a verdict the backend answered but the check refuses: the
// storage or a template is not usable.
type checkFailure struct {
reason string
msg string
}
func (e *checkFailure) Error() string { return e.msg }
// checkReason maps a failed probe to a reason once checkNode has ruled out
// the round's ctx ending. A failed connect or a cut connection is
// Unreachable; a request timeout with the round still running is Timeout.
func checkReason(err error) string {
var (
cf *checkFailure
ae *apiError
op *net.OpError
ne net.Error
)
switch {
case errors.As(err, &cf):
return cf.reason
case errors.Is(err, errCredentials):
return reasonCredentialsUnavailable
case errors.Is(err, errCAFile), isTLSVerification(err):
return reasonCertificateInvalid
case errors.As(err, &ae):
switch ae.Status {
case http.StatusUnauthorized:
return provider.ReasonUnauthorized
case http.StatusForbidden:
return provider.ReasonForbidden
case statusProxyConnect, statusProxyBroken:
return provider.ReasonUnreachable
}
return provider.ReasonAPIError
case errors.As(err, &op):
// Checked before the deadline: a dial that runs out of its own
// budget is a node that did not answer.
return provider.ReasonUnreachable
case errors.Is(err, context.DeadlineExceeded), errors.Is(err, context.Canceled):
return provider.ReasonTimeout
case errors.As(err, &ne), errors.Is(err, io.EOF), errors.Is(err, io.ErrUnexpectedEOF):
return provider.ReasonUnreachable
}
return provider.ReasonAPIError
}
func formatGiB(bytes int64) string {
return strconv.FormatFloat(float64(bytes)/(1<<30), 'f', 1, 64)
}
+593
View File
@@ -0,0 +1,593 @@
package proxmox
import (
"context"
"crypto/x509"
"errors"
"fmt"
"io"
"maps"
"net"
"net/http"
"net/url"
"path/filepath"
"slices"
"strings"
"sync"
"syscall"
"testing"
"time"
"gitea.home.hrajfrisbee.cz/kacerr/egress-proxies-operator/internal/provider"
)
// checkSecret is the secret half of the Checker's token file; it must never
// reach a verdict or a log line.
const checkSecret = "SENTINEL-token-secret-4c1d"
// newCheckFixture is an observed create fixture at V(2) with a token file
// whose secret is checkSecret.
func newCheckFixture(t *testing.T) *fixture {
t.Helper()
fx := newCreateFixture(t, 2)
fx.p.api = observe(fx.fake, testProvider)
fx.p.cfg.TokenFile = filepath.Join(t.TempDir(), "token")
writeFile(t, fx.p.cfg.TokenFile, "proxy-operator@pve!operator="+checkSecret+"\n")
return fx
}
// assertNoSecret checks every verdict and every log line for checkSecret.
func assertNoSecret(t *testing.T, fx *fixture, got []provider.TargetStatus) {
t.Helper()
for _, ts := range got {
if strings.Contains(ts.Message, checkSecret) || slices.ContainsFunc(slices.Collect(maps.Values(ts.Details)), func(v string) bool { return strings.Contains(v, checkSecret) }) {
t.Errorf("verdict carries the token secret: %+v", ts)
}
}
for _, l := range fx.logs.all() {
if strings.Contains(l, checkSecret) {
t.Errorf("log line carries the token secret: %s", l)
}
}
}
func TestCheck_oneEntryPerNodeInConfigOrder(t *testing.T) {
t.Parallel()
fx := newCheckFixture(t)
fx.p.cfg.Nodes = []string{"pv1-3", fakeNode, "pv1-2"}
fx.fake.nodes["pv1-3"] = fx.fake.nodes[fakeNode]
got := fx.p.Check(fx.ctx)
if len(got) != len(fx.p.cfg.Nodes) {
t.Fatalf("Check() returned %d targets, want %d", len(got), len(fx.p.cfg.Nodes))
}
for i, node := range fx.p.cfg.Nodes {
ts := got[i]
if ts.Name != node || ts.Address != fx.p.cfg.URL {
t.Errorf("target %d = %s (%s), want %s (%s)", i, ts.Name, ts.Address, node, fx.p.cfg.URL)
}
if wantReady := node != "pv1-2"; ts.Ready != wantReady {
t.Errorf("target %s ready = %v, want %v (%s: %s)", node, ts.Ready, wantReady, ts.Reason, ts.Message)
}
}
if got[2].Reason != provider.ReasonAPIError || !strings.Contains(got[2].Message, "pv1-2") {
t.Errorf("pv1-2 verdict = %s: %s, want APIError naming the node", got[2].Reason, got[2].Message)
}
}
// TestCheck_nodesCheckedConcurrently holds every node's first request
// until all of them have arrived; checked one at a time, the first would
// give up and fail its verdict.
func TestCheck_nodesCheckedConcurrently(t *testing.T) {
t.Parallel()
fx := newCheckFixture(t)
fx.p.cfg.Nodes = []string{fakeNode, "pv1-2", "pv1-3"}
for _, n := range fx.p.cfg.Nodes[1:] {
fx.fake.nodes[n] = fx.fake.nodes[fakeNode]
}
var (
mu sync.Mutex
arrived int
)
all := make(chan struct{})
fx.fake.hookOn("Version", func() error {
mu.Lock()
if arrived++; arrived == len(fx.p.cfg.Nodes) {
close(all)
}
mu.Unlock()
select {
case <-all:
return nil
case <-time.After(2 * time.Second):
return errors.New("barrier: the nodes were not checked concurrently")
}
})
for _, ts := range fx.p.Check(fx.ctx) {
if !ts.Ready {
t.Errorf("target %s = %s: %s, want Ready", ts.Name, ts.Reason, ts.Message)
}
}
}
func TestCheck_expiredContextReturnsPromptly(t *testing.T) {
t.Parallel()
fx := newCheckFixture(t)
fx.p.cfg.Nodes = []string{fakeNode, "pv1-2"}
fx.fake.blockOn("Version")
ctx, cancel := context.WithTimeout(fx.ctx, 0)
defer cancel()
start := time.Now()
var got []provider.TargetStatus
within(t, 5*time.Second, func() { got = fx.p.Check(ctx) })
if elapsed := time.Since(start); elapsed > time.Second {
t.Errorf("Check() took %v with an expired ctx, want under 1s", elapsed)
}
if len(got) != 2 {
t.Fatalf("Check() returned %d targets, want 2", len(got))
}
for _, ts := range got {
if ts.Ready || ts.Reason != provider.ReasonTimeout || !strings.Contains(ts.Message, context.DeadlineExceeded.Error()) {
t.Errorf("target %s = ready %v reason %s (%s), want Timeout naming the deadline", ts.Name, ts.Ready, ts.Reason, ts.Message)
}
}
}
func TestCheck_reasonsTable(t *testing.T) {
t.Parallel()
urlErr := func(err error) error {
return &url.Error{Op: "Get", URL: "https://192.0.2.8:8006/api2/json/version", Err: err}
}
tests := []struct {
name string
prep func(fx *fixture)
timeout time.Duration
cancel bool
wantReady bool
wantReason string
wantMessage string
}{
{name: "ok", wantReady: true, wantReason: provider.ReasonReachable},
{
name: "token file missing",
prep: func(fx *fixture) { fx.p.cfg.TokenFile = filepath.Join(fx.t.TempDir(), "absent") },
wantReason: reasonCredentialsUnavailable,
wantMessage: "does not exist yet",
},
{
name: "token file malformed",
prep: func(fx *fixture) { writeFile(fx.t, fx.p.cfg.TokenFile, checkSecret+"\n") },
wantReason: reasonCredentialsUnavailable,
wantMessage: "is not <user>@<realm>!<tokenid>=<secret>",
},
{
name: "token file gone by the time the client reads it",
prep: func(fx *fixture) {
fx.fake.failWith("NodeStatus", fmt.Errorf("%w: %w: /etc/proxy-operator/proxmox/token does not exist yet", provider.ErrTransient, errCredentials))
},
wantReason: reasonCredentialsUnavailable,
},
{
name: "unknown certificate authority",
prep: func(fx *fixture) { fx.fake.failWith("Version", urlErr(x509.UnknownAuthorityError{})) },
wantReason: reasonCertificateInvalid,
},
{
name: "host not in the certificate",
prep: func(fx *fixture) { fx.fake.failWith("Version", urlErr(x509.HostnameError{Host: "192.168.2.8"})) },
wantReason: reasonCertificateInvalid,
},
{
name: "CA file unusable",
prep: func(fx *fixture) {
fx.fake.failWith("Version", fmt.Errorf("%w: %w: reading CA file: %w", provider.ErrTransient, errCAFile, syscall.ENOENT))
},
wantReason: reasonCertificateInvalid,
},
{
name: "wrong secret",
prep: func(fx *fixture) { fx.fake.failWith("Version", loadAPIError(fx.t, "err_unauthorized")) },
wantReason: provider.ReasonUnauthorized,
},
{
name: "no credentials sent",
prep: func(fx *fixture) { fx.fake.failWith("Version", loadAPIError(fx.t, "err_no_credentials")) },
wantReason: provider.ReasonUnauthorized,
},
{
name: "privilege missing",
prep: func(fx *fixture) { fx.fake.failWith("NodeStatus", loadAPIError(fx.t, "err_forbidden")) },
wantReason: provider.ReasonForbidden,
wantMessage: "Sys.Syslog",
},
{
name: "pveproxy cannot reach the node",
prep: func(fx *fixture) { fx.fake.failWith("NodeStatus", fakeErr(595, "Connection refused")) },
wantReason: provider.ReasonUnreachable,
},
{
name: "pveproxy broken pipe",
prep: func(fx *fixture) { fx.fake.failWith("NodeStatus", fakeErr(596, "Broken pipe")) },
wantReason: provider.ReasonUnreachable,
},
{
name: "connection refused",
prep: func(fx *fixture) {
fx.fake.failWith("Version", urlErr(&net.OpError{Op: "dial", Net: "tcp", Err: syscall.ECONNREFUSED}))
},
wantReason: provider.ReasonUnreachable,
},
{
name: "node status 500",
prep: func(fx *fixture) { fx.fake.failWith("NodeStatus", fakeErr(500, "fake: pve-cluster not running")) },
wantReason: provider.ReasonAPIError,
},
{
name: "iso storage missing",
prep: func(fx *fixture) {
fx.fake.mu.Lock()
delete(fx.fake.storages, fakeISOStorage)
fx.fake.mu.Unlock()
},
wantReason: reasonStorageNotFound,
wantMessage: "storage proxy-seeds on node pv1-1: No such storage.",
},
{
// Not captured: a storage Proxmox knows but cannot use here.
name: "iso storage unusable on the node",
prep: func(fx *fixture) {
fx.fake.failWith("StorageStatus", fakeErr(500, "fake: storage 'proxy-seeds' is not online"))
},
wantReason: reasonStorageNotFound,
wantMessage: "storage proxy-seeds on node pv1-1: fake: storage 'proxy-seeds' is not online",
},
{
name: "another bad parameter is not a missing storage",
prep: func(fx *fixture) { fx.fake.failWith("StorageStatus", loadAPIError(fx.t, "err_bad_param")) },
wantReason: provider.ReasonAPIError,
},
{
name: "iso storage inactive",
prep: func(fx *fixture) { setStorage(fx, func(s *storageStatus) { s.Active = 0 }) },
wantReason: reasonStorageNotFound,
wantMessage: "not active",
},
{
name: "iso storage without iso content",
prep: func(fx *fixture) { setStorage(fx, func(s *storageStatus) { s.Content = "images,rootdir,isoish" }) },
wantReason: reasonStorageNotFound,
wantMessage: "does not hold iso content (content: images,rootdir,isoish)",
},
{
name: "iso storage forbidden",
prep: func(fx *fixture) { fx.fake.failWith("StorageStatus", loadAPIError(fx.t, "err_forbidden")) },
wantReason: provider.ReasonForbidden,
},
{
name: "template missing",
prep: func(fx *fixture) {
fx.fake.mu.Lock()
delete(fx.fake.vms, testTemplateVMID)
fx.fake.mu.Unlock()
},
wantReason: reasonTemplateNotFound,
wantMessage: `template "debian-13": VM 9001 does not exist`,
},
{
name: "template VMID holds a plain VM",
prep: func(fx *fixture) { fx.fake.vms[testTemplateVMID].template = false },
wantReason: reasonTemplateNotFound,
wantMessage: `template "debian-13": VM 9001 is not a template`,
},
{
name: "template VMID holds a container",
prep: func(fx *fixture) { fx.fake.vms[testTemplateVMID].kind = "lxc" },
wantReason: reasonTemplateNotFound,
wantMessage: "does not exist",
},
{
name: "resources listing fails",
prep: func(fx *fixture) {
fx.fake.failWith("ClusterResources", fakeErr(500, "fake: cluster filesystem not ready"))
},
wantReason: provider.ReasonAPIError,
},
{
name: "request outlives the deadline",
prep: func(fx *fixture) { fx.fake.blockOn("StorageStatus") },
timeout: 50 * time.Millisecond,
wantReason: provider.ReasonTimeout,
},
{
// The real client's dial sees the round's ctx end and fails as
// a dial error, which checkReason alone would call Unreachable.
name: "dial cut short by the round's deadline",
prep: func(fx *fixture) {
fx.fake.hookOn("Version", func() error {
time.Sleep(200 * time.Millisecond)
return urlErr(&net.OpError{Op: "dial", Net: "tcp", Err: context.DeadlineExceeded})
})
},
timeout: 20 * time.Millisecond,
wantReason: provider.ReasonTimeout,
},
{
name: "ctx cancelled mid check",
prep: func(fx *fixture) { fx.fake.blockOn("ClusterResources") },
cancel: true,
wantReason: provider.ReasonTimeout,
},
}
for _, tc := range tests {
t.Run(tc.name, func(t *testing.T) {
t.Parallel()
fx := newCheckFixture(t)
if tc.prep != nil {
tc.prep(fx)
}
ctx := fx.ctx
switch {
case tc.timeout > 0:
var cancel context.CancelFunc
ctx, cancel = context.WithTimeout(ctx, tc.timeout)
defer cancel()
case tc.cancel:
var cancel context.CancelFunc
ctx, cancel = context.WithCancel(ctx)
time.AfterFunc(50*time.Millisecond, cancel)
}
var got []provider.TargetStatus
within(t, 5*time.Second, func() { got = fx.p.Check(ctx) })
if len(got) != 1 {
t.Fatalf("Check() returned %d targets, want 1", len(got))
}
ts := got[0]
if ts.Ready != tc.wantReady || ts.Reason != tc.wantReason {
t.Errorf("verdict = ready %v reason %s (%s), want ready %v reason %s", ts.Ready, ts.Reason, ts.Message, tc.wantReady, tc.wantReason)
}
if !tc.wantReady && ts.Message == "" {
t.Error("failed check has no message")
}
if tc.wantReady && ts.Message != "" {
t.Errorf("successful check has a message: %q", ts.Message)
}
if tc.wantMessage != "" && !strings.Contains(ts.Message, tc.wantMessage) {
t.Errorf("message = %q, want it to contain %q", ts.Message, tc.wantMessage)
}
if ts.Details["isoStorage"] != fakeISOStorage {
t.Errorf("details = %v, want isoStorage whatever failed", ts.Details)
}
assertNoSecret(t, fx, got)
})
}
}
func setStorage(fx *fixture, fn func(s *storageStatus)) {
fx.fake.mu.Lock()
defer fx.fake.mu.Unlock()
s := fx.fake.storages[fakeISOStorage]
fn(&s)
fx.fake.storages[fakeISOStorage] = s
}
// TestCheck_reasonsOverHTTPClient takes the transport-level reasons from
// the real client, so the errors checkReason sees are the ones net/http and
// crypto/tls produce, and the Ready row replays the Phase 0 captures.
func TestCheck_reasonsOverHTTPClient(t *testing.T) {
t.Parallel()
captures := func(t *testing.T) http.HandlerFunc {
return func(w http.ResponseWriter, r *http.Request) {
path := strings.TrimPrefix(r.URL.Path, "/api2/json/")
name := map[string]string{
"version": "version",
"nodes/pv1-1/status": "node_status",
"nodes/pv1-1/storage/local/status": "storage_status",
"cluster/resources": "cluster_resources",
}[path]
if name == "" {
t.Errorf("unexpected request %s", path)
http.NotFound(w, r)
return
}
serveJSON(w, readCapture(t, name).Body)
}
}
tests := []struct {
name string
handler func(t *testing.T) http.HandlerFunc
mutate func(t *testing.T, cfx *clientFixture)
wantReason string
}{
{name: "captures", handler: captures, wantReason: provider.ReasonReachable},
{name: "unknown authority", handler: captures, mutate: func(_ *testing.T, cfx *clientFixture) { cfx.cfg.CAFile = "" }, wantReason: reasonCertificateInvalid},
{
name: "host not in the certificate",
handler: captures,
mutate: func(_ *testing.T, cfx *clientFixture) {
cfx.cfg.URL = strings.Replace(cfx.srv.URL, "127.0.0.1", "localhost", 1)
},
wantReason: reasonCertificateInvalid,
},
{
name: "CA file missing",
handler: captures,
mutate: func(t *testing.T, cfx *clientFixture) { cfx.cfg.CAFile = filepath.Join(t.TempDir(), "ca.pem") },
wantReason: reasonCertificateInvalid,
},
{
name: "401 from pveproxy",
handler: func(t *testing.T) http.HandlerFunc {
return func(w http.ResponseWriter, _ *http.Request) { serveCapture(t, w, readCapture(t, "err_unauthorized")) }
},
wantReason: provider.ReasonUnauthorized,
},
{
name: "596 from pveproxy",
handler: func(t *testing.T) http.HandlerFunc {
return func(w http.ResponseWriter, _ *http.Request) {
serveCapture(t, w, capture{Status: 596, Reason: "Broken pipe"})
}
},
wantReason: provider.ReasonUnreachable,
},
{
name: "server gone",
handler: captures,
mutate: func(_ *testing.T, cfx *clientFixture) { cfx.srv.Close() },
wantReason: provider.ReasonUnreachable,
},
{
name: "connection cut mid response",
handler: func(t *testing.T) http.HandlerFunc {
return func(w http.ResponseWriter, _ *http.Request) {
conn, _, err := w.(http.Hijacker).Hijack()
if err != nil {
t.Errorf("hijack: %v", err)
return
}
_, _ = io.WriteString(conn, "HTTP/1.1 200 OK\r\nContent-Length: 100\r\n\r\n{\"data\":")
_ = conn.Close()
}
},
wantReason: provider.ReasonUnreachable,
},
}
for _, tc := range tests {
t.Run(tc.name, func(t *testing.T) {
t.Parallel()
cfx := newClientFixture(t, tc.handler(t))
if tc.mutate != nil {
tc.mutate(t, cfx)
}
cfg := testConfig()
cfg.ISOStorage = "local"
cfg.URL, cfg.TokenFile, cfg.CAFile = cfx.cfg.URL, cfx.cfg.TokenFile, cfx.cfg.CAFile
p := newProvider(testProvider, cfg, cfx.api(t))
got := p.Check(context.Background())
if len(got) != 1 {
t.Fatalf("Check() returned %d targets, want 1", len(got))
}
if ts := got[0]; ts.Reason != tc.wantReason || ts.Ready != (tc.wantReason == provider.ReasonReachable) {
t.Errorf("verdict = ready %v reason %s (%s), want %s", ts.Ready, ts.Reason, ts.Message, tc.wantReason)
}
})
}
}
func TestCheck_detailsKeys(t *testing.T) {
t.Parallel()
full := map[string]string{
"tokenID": "proxy-operator@pve!operator", "isoStorage": fakeISOStorage, "pveVersion": "9.2.20",
"kernel": "7.0.14-17-pve", "isoStorageAvailGiB": "90.0", "templates": "1/1",
}
tests := []struct {
name string
prep func(fx *fixture)
want []string
}{
{name: "success", want: slices.Sorted(maps.Keys(full))},
{name: "token unusable", prep: func(fx *fixture) { writeFile(fx.t, fx.p.cfg.TokenFile, "junk") }, want: []string{"isoStorage"}},
{name: "node status fails", prep: func(fx *fixture) { fx.fake.failWith("NodeStatus", fakeErr(596, "Broken pipe")) }, want: []string{"isoStorage", "pveVersion", "tokenID"}},
{name: "storage inactive", prep: func(fx *fixture) { setStorage(fx, func(s *storageStatus) { s.Active = 0 }) }, want: []string{"isoStorage", "isoStorageAvailGiB", "kernel", "pveVersion", "tokenID"}},
{name: "template missing", prep: func(fx *fixture) { fx.fake.vms[testTemplateVMID].template = false }, want: slices.Sorted(maps.Keys(full))},
}
for _, tc := range tests {
t.Run(tc.name, func(t *testing.T) {
t.Parallel()
fx := newCheckFixture(t)
if tc.prep != nil {
tc.prep(fx)
}
got := fx.p.Check(fx.ctx)[0]
if keys := slices.Sorted(maps.Keys(got.Details)); !slices.Equal(keys, tc.want) {
t.Errorf("details keys = %v, want %v", keys, tc.want)
}
for k, v := range got.Details {
if want, ok := full[k]; ok && tc.name == "success" && v != want {
t.Errorf("details[%s] = %q, want %q", k, v, want)
}
if v == "" {
t.Errorf("details[%s] is empty", k)
}
}
})
}
t.Run("templates counts found over configured", func(t *testing.T) {
t.Parallel()
fx := newCheckFixture(t)
fx.p.cfg.Templates["ubuntu"] = provider.ProxmoxTemplate{VMID: 9002}
fx.p.cfg.Templates["debian-again"] = provider.ProxmoxTemplate{VMID: testTemplateVMID}
got := fx.p.Check(fx.ctx)[0]
if got.Details["templates"] != "2/3" || got.Reason != reasonTemplateNotFound || !strings.Contains(got.Message, `template "ubuntu": VM 9002`) {
t.Errorf("verdict = %s %q, details %v; want TemplateNotFound naming ubuntu, templates 2/3", got.Reason, got.Message, got.Details)
}
})
}
func TestCheck_logsVerdictPerNode(t *testing.T) {
t.Parallel()
fx := newCheckFixture(t)
fx.p.cfg.Nodes = []string{fakeNode, "pv1-2"}
fx.p.Check(fx.ctx)
checked := fx.logs.matching(`"msg"="proxmox node checked"`)
if len(checked) != 2 {
t.Fatalf("got %d verdict lines, want 2: %v", len(checked), checked)
}
for _, l := range checked {
switch {
case strings.Contains(l, `"node"="pv1-1"`):
if !strings.Contains(l, `"ready"=true`) || !strings.Contains(l, `"reason"="Reachable"`) || !strings.Contains(l, `"latency"=`) {
t.Errorf("pv1-1 verdict line = %s", l)
}
case strings.Contains(l, `"node"="pv1-2"`):
if !strings.Contains(l, `"ready"=false`) || !strings.Contains(l, `"reason"="APIError"`) {
t.Errorf("pv1-2 verdict line = %s", l)
}
default:
t.Errorf("verdict line without a node: %s", l)
}
if !strings.Contains(l, `"provider"="`+testProvider+`"`) || !strings.Contains(l, `"level"=1`) {
t.Errorf("verdict line lacks the provider or is not V(1): %s", l)
}
}
}
func TestCheckReason_table(t *testing.T) {
t.Parallel()
dial := &url.Error{Op: "Get", URL: "https://192.0.2.8:8006/api2/json/version", Err: &net.OpError{Op: "dial", Net: "tcp", Err: context.DeadlineExceeded}}
tests := []struct {
name string
err error
want string
}{
{name: "storage verdict", err: &checkFailure{reason: reasonStorageNotFound, msg: "x"}, want: reasonStorageNotFound},
{name: "template verdict wrapped", err: fmt.Errorf("check: %w", &checkFailure{reason: reasonTemplateNotFound, msg: "x"}), want: reasonTemplateNotFound},
{name: "token missing", err: fmt.Errorf("%w: %w: /t does not exist yet", provider.ErrTransient, errCredentials), want: reasonCredentialsUnavailable},
{name: "token malformed", err: fmt.Errorf("%w: %w: /t is not a token", provider.ErrPermanent, errCredentials), want: reasonCredentialsUnavailable},
{name: "CA file", err: fmt.Errorf("%w: %w: /ca holds no PEM certificate", provider.ErrPermanent, errCAFile), want: reasonCertificateInvalid},
{name: "unknown authority", err: &url.Error{Err: x509.UnknownAuthorityError{}}, want: reasonCertificateInvalid},
{name: "certificate invalid", err: &url.Error{Err: x509.CertificateInvalidError{Reason: x509.Expired}}, want: reasonCertificateInvalid},
{name: "401", err: loadAPIError(t, "err_unauthorized"), want: provider.ReasonUnauthorized},
{name: "403 wrapped", err: fmt.Errorf("reading: %w", loadAPIError(t, "err_forbidden_volume_delete")), want: provider.ReasonForbidden},
{name: "595", err: fakeErr(595, "Connection refused"), want: provider.ReasonUnreachable},
{name: "596", err: fakeErr(596, "Broken pipe"), want: provider.ReasonUnreachable},
{name: "400", err: loadAPIError(t, "err_bad_param"), want: provider.ReasonAPIError},
{name: "500", err: loadAPIError(t, "err_vm_missing"), want: provider.ReasonAPIError},
{name: "dial ran out of its own budget", err: dial, want: provider.ReasonUnreachable},
{name: "connection reset", err: &url.Error{Err: &net.OpError{Op: "read", Net: "tcp", Err: syscall.ECONNRESET}}, want: provider.ReasonUnreachable},
{name: "request timeout", err: fmt.Errorf("reading: %w", &url.Error{Err: context.DeadlineExceeded}), want: provider.ReasonTimeout},
{name: "cancelled", err: context.Canceled, want: provider.ReasonTimeout},
{name: "server closed mid response", err: &url.Error{Err: io.ErrUnexpectedEOF}, want: provider.ReasonUnreachable},
{name: "eof", err: io.EOF, want: provider.ReasonUnreachable},
{name: "decode failure", err: errors.New("decoding GET version: invalid character '<'"), want: provider.ReasonAPIError},
{name: "nil", err: nil, want: provider.ReasonAPIError},
}
for _, tc := range tests {
t.Run(tc.name, func(t *testing.T) {
t.Parallel()
if got := checkReason(tc.err); got != tc.want {
t.Errorf("checkReason(%v) = %s, want %s", tc.err, got, tc.want)
}
})
}
}
+382
View File
@@ -0,0 +1,382 @@
package proxmox
import (
"bytes"
"context"
"crypto/tls"
"crypto/x509"
"encoding/json"
"errors"
"fmt"
"io"
"io/fs"
"mime/multipart"
"net"
"net/http"
"net/url"
"os"
"regexp"
"strconv"
"strings"
"sync"
"time"
"gitea.home.hrajfrisbee.cz/kacerr/egress-proxies-operator/internal/provider"
"gitea.home.hrajfrisbee.cz/kacerr/egress-proxies-operator/internal/tracing"
)
const (
dialTimeout = 10 * time.Second
tlsHandshakeTimeout = 10 * time.Second
idleConnTimeout = 90 * time.Second
maxIdleConnsPerHost = 4
// maxErrorBody caps how much of a failed response is read for its
// message; Proxmox error bodies are a few hundred bytes.
maxErrorBody = 64 << 10
)
// errCredentials marks a token file that is missing or unusable, so the
// Checker can report CredentialsUnavailable instead of an API failure.
var errCredentials = errors.New("proxmox token file unusable")
// errCAFile marks a CA file that is missing or holds no certificate, so the
// Checker can report CertificateInvalid: the admin fixes caFile.
var errCAFile = errors.New("proxmox CA file unusable")
// tokenPattern is <user>@<realm>!<tokenid>=<secret> (spec entry 14).
var tokenPattern = regexp.MustCompile(`^[^@\s]+@[^!\s]+![^=\s]+=\S+$`)
// httpAPI is the production adapter behind the api seam: a thin net/http
// client for /api2/json (spec entry 17). It is safe for concurrent use.
type httpAPI struct {
base *url.URL
tokenFile string
caFile string
insecure bool
// timeout bounds one request, response included. Read when the client
// is built; tests shrink it before the first call.
timeout time.Duration
wrap func(http.RoundTripper) http.RoundTripper
// mu guards client, built on first use because the CA is loaded lazily
// and only cached once it loads (spec entry 14).
mu sync.Mutex
client *http.Client
}
var _ api = (*httpAPI)(nil)
// newHTTPAPI builds the adapter without touching the network or any file.
func newHTTPAPI(cfg provider.ProxmoxConfig, opts ...tracing.Option) (*httpAPI, error) {
u, err := url.Parse(cfg.URL)
if err != nil {
return nil, fmt.Errorf("proxmox.url: %w", err)
}
return &httpAPI{
base: u.JoinPath("api2", "json"),
tokenFile: cfg.TokenFile,
caFile: cfg.CAFile,
insecure: cfg.InsecureSkipTLSVerify,
timeout: time.Duration(cfg.RequestTimeoutSeconds) * time.Second,
wrap: tracing.RestConfigWrapper(opts...),
}, nil
}
// httpClient returns the shared client, building it once the CA loads. A
// missing CA file is Transient and retried on the next call; one without a
// certificate is Permanent.
func (a *httpAPI) httpClient() (*http.Client, error) {
a.mu.Lock()
defer a.mu.Unlock()
if a.client != nil {
return a.client, nil
}
tlsCfg := &tls.Config{MinVersion: tls.VersionTLS12, InsecureSkipVerify: a.insecure} //nolint:gosec // opt-in, warned about at startup
if a.caFile != "" && !a.insecure {
pem, err := os.ReadFile(a.caFile)
if err != nil {
return nil, fmt.Errorf("%w: %w: reading CA file: %w", provider.ErrTransient, errCAFile, err)
}
pool := x509.NewCertPool()
if !pool.AppendCertsFromPEM(pem) {
return nil, fmt.Errorf("%w: %w: %s holds no PEM certificate", provider.ErrPermanent, errCAFile, a.caFile)
}
tlsCfg.RootCAs = pool
}
transport := &http.Transport{
Proxy: http.ProxyFromEnvironment,
DialContext: (&net.Dialer{Timeout: dialTimeout, KeepAlive: 30 * time.Second}).DialContext,
TLSClientConfig: tlsCfg,
TLSHandshakeTimeout: tlsHandshakeTimeout,
ResponseHeaderTimeout: a.timeout,
IdleConnTimeout: idleConnTimeout,
MaxIdleConnsPerHost: maxIdleConnsPerHost,
// pveproxy speaks HTTP/1.1 only (F18); a non-nil empty map keeps
// the transport from offering h2.
ForceAttemptHTTP2: false,
TLSNextProto: map[string]func(string, *tls.Conn) http.RoundTripper{},
}
a.client = &http.Client{Transport: a.wrap(transport), Timeout: a.timeout}
return a.client, nil
}
// readToken reads the token file on every request, so a rotated Secret
// takes effect at once. Neither error names the file's content.
func readToken(path string) (string, error) {
raw, err := os.ReadFile(path)
if errors.Is(err, fs.ErrNotExist) {
return "", fmt.Errorf("%w: %w: %s does not exist yet", provider.ErrTransient, errCredentials, path)
}
if err != nil {
return "", fmt.Errorf("%w: reading %s: %w", errCredentials, path, err)
}
token := strings.TrimSpace(string(raw))
if !tokenPattern.MatchString(token) {
return "", fmt.Errorf("%w: %w: %s is not <user>@<realm>!<tokenid>=<secret>", provider.ErrPermanent, errCredentials, path)
}
return token, nil
}
// apiPath joins path segments, escaping each, so a volid's "/" travels as
// %2F as Phase 0 sent it.
func apiPath(segments ...string) string {
escaped := make([]string, len(segments))
for i, s := range segments {
escaped[i] = url.PathEscape(s)
}
return strings.Join(escaped, "/")
}
func vmPath(node string, vmid int, rest ...string) string {
return apiPath(append([]string{"nodes", node, "qemu", strconv.Itoa(vmid)}, rest...)...)
}
// do sends one request and decodes the response's data member into out,
// which may be nil. POST and PUT send form as the body; every method sends
// query in the URL. Any non-2xx answer is an *apiError.
func (a *httpAPI) do(ctx context.Context, method, path string, query, form url.Values, out any) error {
var body io.Reader
contentType := ""
if form != nil {
body = strings.NewReader(form.Encode())
contentType = "application/x-www-form-urlencoded"
}
return a.send(ctx, method, path, query, body, -1, contentType, out)
}
// send is do with a caller-built body. length is the body's size, or -1
// when do built it from a string the request can measure itself.
func (a *httpAPI) send(ctx context.Context, method, path string, query url.Values, body io.Reader, length int64, contentType string, out any) error {
token, err := readToken(a.tokenFile)
if err != nil {
return err
}
client, err := a.httpClient()
if err != nil {
return err
}
u := a.base.JoinPath(path)
u.RawQuery = query.Encode()
req, err := http.NewRequestWithContext(ctx, method, u.String(), body)
if err != nil {
return fmt.Errorf("building %s %s: %w", method, path, err)
}
if length >= 0 {
req.ContentLength = length
}
if contentType != "" {
req.Header.Set("Content-Type", contentType)
}
req.Header.Set("Accept", "application/json")
req.Header.Set("Authorization", "PVEAPIToken="+token)
resp, err := client.Do(req)
if err != nil {
return err
}
defer func() {
_, _ = io.Copy(io.Discard, io.LimitReader(resp.Body, maxErrorBody))
_ = resp.Body.Close()
}()
if resp.StatusCode < 200 || resp.StatusCode > 299 {
raw, _ := io.ReadAll(io.LimitReader(resp.Body, maxErrorBody))
reason := strings.TrimPrefix(resp.Status, strconv.Itoa(resp.StatusCode))
return redact(newAPIError(resp.StatusCode, reason, raw), token)
}
var env struct {
Data json.RawMessage `json:"data"`
}
if err := json.NewDecoder(resp.Body).Decode(&env); err != nil {
return fmt.Errorf("decoding %s %s: %w", method, path, err)
}
if out == nil || len(env.Data) == 0 {
return nil
}
if err := json.Unmarshal(env.Data, out); err != nil {
return fmt.Errorf("decoding %s %s data: %w", method, path, err)
}
return nil
}
// redact removes the token's secret from an error Proxmox echoed back, so
// it cannot reach a log line, a span or a Proxy's status (spec entry 14).
func redact(e *apiError, token string) *apiError {
_, secret, _ := strings.Cut(token, "=")
if secret == "" {
return e
}
e.Message = strings.ReplaceAll(e.Message, secret, "<redacted>")
for k, v := range e.Errors {
e.Errors[k] = strings.ReplaceAll(v, secret, "<redacted>")
}
return e
}
// decodeConfig renders every config value as a string: Proxmox sends some
// scalars as JSON numbers (cores) and others as strings (memory). A null
// value is dropped.
func decodeConfig(raw map[string]json.RawMessage) (vmConfig, error) {
cfg := make(vmConfig, len(raw))
for k, v := range raw {
switch {
case len(v) == 0 || string(v) == "null":
continue
case v[0] == '"':
var s string
if err := json.Unmarshal(v, &s); err != nil {
return nil, fmt.Errorf("config key %s: %w", k, err)
}
cfg[k] = s
default:
cfg[k] = string(v)
}
}
return cfg, nil
}
func (a *httpAPI) Version(ctx context.Context) (v versionInfo, err error) {
err = a.do(ctx, http.MethodGet, "version", nil, nil, &v)
return v, err
}
func (a *httpAPI) NodeStatus(ctx context.Context, node string) (s nodeStatus, err error) {
err = a.do(ctx, http.MethodGet, apiPath("nodes", node, "status"), nil, nil, &s)
return s, err
}
func (a *httpAPI) StorageStatus(ctx context.Context, node, storage string) (s storageStatus, err error) {
err = a.do(ctx, http.MethodGet, apiPath("nodes", node, "storage", storage, "status"), nil, nil, &s)
return s, err
}
func (a *httpAPI) ClusterResources(ctx context.Context) (rs []vmResource, err error) {
err = a.do(ctx, http.MethodGet, "cluster/resources", url.Values{"type": {"vm"}}, nil, &rs)
return rs, err
}
func (a *httpAPI) VMConfig(ctx context.Context, node string, vmid int) (vmConfig, error) {
var raw map[string]json.RawMessage
if err := a.do(ctx, http.MethodGet, vmPath(node, vmid, "config"), nil, nil, &raw); err != nil {
return nil, err
}
return decodeConfig(raw)
}
func (a *httpAPI) SetVMConfig(ctx context.Context, node string, vmid int, params map[string]string) error {
form := url.Values{}
for k, v := range params {
form.Set(k, v)
}
return a.do(ctx, http.MethodPut, vmPath(node, vmid, "config"), nil, form, nil)
}
func (a *httpAPI) Clone(ctx context.Context, node string, vmid int, p cloneParams) (upid string, err error) {
form := url.Values{
"newid": {strconv.Itoa(p.NewID)},
"name": {p.Name},
"description": {p.Description},
"full": {"1"},
}
if p.Storage != "" {
form.Set("storage", p.Storage)
}
if p.Target != "" {
form.Set("target", p.Target)
}
err = a.do(ctx, http.MethodPost, vmPath(node, vmid, "clone"), nil, form, &upid)
return upid, err
}
func (a *httpAPI) Resize(ctx context.Context, node string, vmid int, disk, size string) (upid string, err error) {
form := url.Values{"disk": {disk}, "size": {size}}
err = a.do(ctx, http.MethodPut, vmPath(node, vmid, "resize"), nil, form, &upid)
return upid, err
}
func (a *httpAPI) Start(ctx context.Context, node string, vmid int) (upid string, err error) {
err = a.do(ctx, http.MethodPost, vmPath(node, vmid, "status", "start"), nil, url.Values{}, &upid)
return upid, err
}
func (a *httpAPI) Stop(ctx context.Context, node string, vmid int) (upid string, err error) {
err = a.do(ctx, http.MethodPost, vmPath(node, vmid, "status", "stop"), nil, url.Values{}, &upid)
return upid, err
}
func (a *httpAPI) VMStatus(ctx context.Context, node string, vmid int) (s vmStatus, err error) {
err = a.do(ctx, http.MethodGet, vmPath(node, vmid, "status", "current"), nil, nil, &s)
return s, err
}
// Destroy purges the VM from backup jobs and HA and removes disks no config
// references, so nothing of it is left behind (spec entry 11).
func (a *httpAPI) Destroy(ctx context.Context, node string, vmid int) (upid string, err error) {
query := url.Values{"purge": {"1"}, "destroy-unreferenced-disks": {"1"}}
err = a.do(ctx, http.MethodDelete, vmPath(node, vmid), query, nil, &upid)
return upid, err
}
func (a *httpAPI) TaskStatus(ctx context.Context, node, upid string) (s taskStatus, err error) {
err = a.do(ctx, http.MethodGet, apiPath("nodes", node, "tasks", upid, "status"), nil, nil, &s)
return s, err
}
func (a *httpAPI) AgentNetworkInterfaces(ctx context.Context, node string, vmid int) ([]agentInterface, error) {
var out struct {
Result []agentInterface `json:"result"`
}
err := a.do(ctx, http.MethodGet, vmPath(node, vmid, "agent", "network-get-interfaces"), nil, nil, &out)
return out.Result, err
}
// UploadISO streams r as the file part of a multipart upload (F12). The
// multipart framing is rendered up front, so the request carries an exact
// Content-Length, as Phase 0's curl -F upload did, while r itself is read
// in the transport's chunks and never held in memory.
func (a *httpAPI) UploadISO(ctx context.Context, node, storage, filename string, r io.Reader, size int64) (upid string, err error) {
var frame bytes.Buffer
mw := multipart.NewWriter(&frame)
if err := mw.WriteField("content", "iso"); err != nil {
return "", fmt.Errorf("framing upload: %w", err)
}
if _, err := mw.CreateFormFile("filename", filename); err != nil {
return "", fmt.Errorf("framing upload: %w", err)
}
head := bytes.Clone(frame.Bytes())
frame.Reset()
if err := mw.Close(); err != nil {
return "", fmt.Errorf("framing upload: %w", err)
}
tail := frame.Bytes()
if r == nil {
r = bytes.NewReader(nil)
}
body := io.MultiReader(bytes.NewReader(head), io.LimitReader(r, size), bytes.NewReader(tail))
length := int64(len(head)) + size + int64(len(tail))
err = a.send(ctx, http.MethodPost, apiPath("nodes", node, "storage", storage, "upload"), nil, body, length, mw.FormDataContentType(), &upid)
return upid, err
}
func (a *httpAPI) DeleteVolume(ctx context.Context, node, storage, volid string) (upid string, err error) {
err = a.do(ctx, http.MethodDelete, apiPath("nodes", node, "storage", storage, "content", volid), nil, nil, &upid)
return upid, err
}
+874
View File
@@ -0,0 +1,874 @@
package proxmox
import (
"bytes"
"context"
"encoding/json"
"encoding/pem"
"errors"
"fmt"
"io"
stdlog "log"
"net"
"net/http"
"net/http/httptest"
"net/url"
"os"
"path/filepath"
"reflect"
"strings"
"sync"
"sync/atomic"
"testing"
"time"
sdktrace "go.opentelemetry.io/otel/sdk/trace"
"go.opentelemetry.io/otel/sdk/trace/tracetest"
"gitea.home.hrajfrisbee.cz/kacerr/egress-proxies-operator/internal/provider"
"gitea.home.hrajfrisbee.cz/kacerr/egress-proxies-operator/internal/tracing"
)
const testToken = "proxy-operator@pve!operator=0b4e1c2a-5d6f-4a7b-8c9d-0e1f2a3b4c5d"
// clientFixture is a TLS test server, a CA file trusting it, a token file
// and the config pointing at them.
type clientFixture struct {
srv *httptest.Server
cfg provider.ProxmoxConfig
tokenFile string
caFile string
}
// newClientFixture starts h on a TLS server whose error log is discarded:
// the tests that fail a handshake on purpose would otherwise print it.
func newClientFixture(t *testing.T, h http.HandlerFunc) *clientFixture {
t.Helper()
srv := httptest.NewUnstartedServer(h)
srv.Config.ErrorLog = stdlog.New(io.Discard, "", 0)
srv.StartTLS()
t.Cleanup(srv.Close)
return fixtureFor(t, srv)
}
func fixtureFor(t *testing.T, srv *httptest.Server) *clientFixture {
t.Helper()
dir := t.TempDir()
fx := &clientFixture{srv: srv, tokenFile: filepath.Join(dir, "token"), caFile: filepath.Join(dir, "ca.pem")}
writeFile(t, fx.tokenFile, testToken+"\n")
writeFile(t, fx.caFile, string(pem.EncodeToMemory(&pem.Block{Type: "CERTIFICATE", Bytes: srv.Certificate().Raw})))
fx.cfg = provider.ProxmoxConfig{URL: srv.URL, TokenFile: fx.tokenFile, CAFile: fx.caFile, RequestTimeoutSeconds: 30}
return fx
}
func (fx *clientFixture) api(t *testing.T, opts ...tracing.Option) *httpAPI {
t.Helper()
a, err := newHTTPAPI(fx.cfg, opts...)
if err != nil {
t.Fatalf("newHTTPAPI error = %v", err)
}
return a
}
func writeFile(t *testing.T, path, content string) {
t.Helper()
if err := os.WriteFile(path, []byte(content), 0o600); err != nil {
t.Fatalf("writing %s: %v", path, err)
}
}
// serveCapture writes a Phase 0 capture back byte for byte, reason phrase
// included, which net/http's ResponseWriter cannot do.
func serveCapture(t *testing.T, w http.ResponseWriter, c capture) {
t.Helper()
conn, buf, err := w.(http.Hijacker).Hijack()
if err != nil {
t.Errorf("hijack: %v", err)
return
}
defer func() { _ = conn.Close() }()
_, _ = fmt.Fprintf(buf, "HTTP/1.1 %d %s\r\nContent-Type: application/json;charset=UTF-8\r\nContent-Length: %d\r\nConnection: close\r\n\r\n%s",
c.Status, c.Reason, len(c.Body), c.Body)
_ = buf.Flush()
}
func serveJSON(w http.ResponseWriter, body string) {
w.Header().Set("Content-Type", "application/json;charset=UTF-8")
_, _ = io.WriteString(w, body)
}
func assertClass(t *testing.T, err, class error) {
t.Helper()
if got := classify("get", err); got != class {
t.Errorf("classify(%v) = %v, want %v", err, got, class)
}
}
func TestClient_sendsTokenHeaderFromFileOnEveryRequest(t *testing.T) {
t.Parallel()
var (
mu sync.Mutex
headers []string
)
fx := newClientFixture(t, func(w http.ResponseWriter, r *http.Request) {
mu.Lock()
headers = append(headers, r.Header.Get("Authorization"))
mu.Unlock()
serveJSON(w, readCapture(t, "version").Body)
})
a := fx.api(t)
const rotated = "proxy-operator@pve!operator=11111111-2222-3333-4444-555555555555"
for i, token := range []string{testToken, rotated} {
writeFile(t, fx.tokenFile, token+"\n")
v, err := a.Version(context.Background())
if err != nil || v.Version != "9.2.20" {
t.Fatalf("call %d: Version() = %+v, %v", i, v, err)
}
}
want := []string{"PVEAPIToken=" + testToken, "PVEAPIToken=" + rotated}
mu.Lock()
defer mu.Unlock()
if !reflect.DeepEqual(headers, want) {
t.Errorf("Authorization headers = %q, want %q", headers, want)
}
}
func TestClient_missingTokenFileIsTransient(t *testing.T) {
t.Parallel()
var requests atomic.Int32
fx := newClientFixture(t, func(http.ResponseWriter, *http.Request) { requests.Add(1) })
fx.cfg.TokenFile = filepath.Join(t.TempDir(), "absent")
_, err := fx.api(t).Version(context.Background())
if !errors.Is(err, provider.ErrTransient) || !errors.Is(err, errCredentials) {
t.Errorf("error = %v, want ErrTransient and errCredentials", err)
}
if n := requests.Load(); n != 0 {
t.Errorf("server saw %d requests, want none without a token", n)
}
}
func TestClient_unparsableTokenFileIsPermanent(t *testing.T) {
t.Parallel()
const sentinel = "SENTINEL-7f3e"
tests := []struct{ name, content string }{
{name: "empty", content: ""},
{name: "secret only", content: sentinel},
{name: "no secret", content: "proxy-operator@pve!operator"},
{name: "no token id", content: "proxy-operator@pve=" + sentinel},
{name: "no realm", content: "proxy-operator!operator=" + sentinel},
{name: "two lines", content: "proxy-operator@pve!operator=" + sentinel + "\nsecond=line"},
}
for _, tc := range tests {
t.Run(tc.name, func(t *testing.T) {
t.Parallel()
var requests atomic.Int32
fx := newClientFixture(t, func(http.ResponseWriter, *http.Request) { requests.Add(1) })
writeFile(t, fx.tokenFile, tc.content)
_, err := fx.api(t).Version(context.Background())
if !errors.Is(err, provider.ErrPermanent) || !errors.Is(err, errCredentials) {
t.Errorf("error = %v, want ErrPermanent and errCredentials", err)
}
if err != nil && strings.Contains(err.Error(), sentinel) {
t.Errorf("error carries the file's content: %v", err)
}
if n := requests.Load(); n != 0 {
t.Errorf("server saw %d requests, want none", n)
}
})
}
}
// TestClient_apiErrorFromStatusReasonAndBody serves every captured error
// verbatim and expects the apiError the captures describe (F1): the body's
// message, or the reason phrase when the body is empty.
func TestClient_apiErrorFromStatusReasonAndBody(t *testing.T) {
t.Parallel()
paths, err := filepath.Glob(filepath.Join("testdata", "err_*.json"))
if err != nil || len(paths) == 0 {
t.Fatalf("no error captures: %v", err)
}
for _, path := range paths {
name := strings.TrimSuffix(filepath.Base(path), ".json")
if readCapture(t, name).Status < 400 {
continue // err_tag_hyphen: hyphens turned out to be accepted (F9)
}
t.Run(name, func(t *testing.T) {
t.Parallel()
c := readCapture(t, name)
fx := newClientFixture(t, func(w http.ResponseWriter, _ *http.Request) { serveCapture(t, w, c) })
_, err := fx.api(t).Version(context.Background())
var ae *apiError
if !errors.As(err, &ae) {
t.Fatalf("error = %v (%T), want *apiError", err, err)
}
want := loadAPIError(t, name)
if ae.Status != want.Status || ae.Message != want.Message || !reflect.DeepEqual(ae.Errors, want.Errors) {
t.Errorf("apiError = %+v, want %+v", ae, want)
}
if ae.Message == "" || strings.HasSuffix(ae.Message, "\n") {
t.Errorf("message %q is empty or untrimmed", ae.Message)
}
})
}
}
func TestClient_decodesDataEnvelope(t *testing.T) {
t.Parallel()
ctx := context.Background()
tests := []struct {
capture string
call func(a *httpAPI) (any, error)
check func(t *testing.T, got any)
}{
{capture: "version", call: func(a *httpAPI) (any, error) { return a.Version(ctx) }, check: func(t *testing.T, got any) {
if v := got.(versionInfo); v.Version != "9.2.20" {
t.Errorf("version = %+v", v)
}
}},
{capture: "node_status", call: func(a *httpAPI) (any, error) { return a.NodeStatus(ctx, "pv1-1") }, check: func(t *testing.T, got any) {
if s := got.(nodeStatus); s.CurrentKernel.Release != "7.0.14-17-pve" {
t.Errorf("node status = %+v", s)
}
}},
{capture: "storage_status", call: func(a *httpAPI) (any, error) { return a.StorageStatus(ctx, "pv1-1", "local") }, check: func(t *testing.T, got any) {
if s := got.(storageStatus); s.Avail <= 0 || !strings.Contains(s.Content, "iso") {
t.Errorf("storage status = %+v", s)
}
}},
{capture: "cluster_resources_during_clone", call: func(a *httpAPI) (any, error) { return a.ClusterResources(ctx) }, check: func(t *testing.T, got any) {
rs := got.([]vmResource)
for _, r := range rs {
if r.VMID == 199999 && r.Lock == "clone" && r.Name == "" {
return
}
}
t.Errorf("resources = %+v, want the nameless locked clone", rs)
}},
{capture: "vm_config_clone", call: func(a *httpAPI) (any, error) { return a.VMConfig(ctx, "pv1-1", 199999) }, check: func(t *testing.T, got any) {
cfg := got.(vmConfig)
if cfg["cores"] != "2" || cfg["memory"] != "2048" || cfg["agent"] != "1" || cfg["boot"] != "order=scsi0" {
t.Errorf("config scalars = cores %q memory %q agent %q boot %q", cfg["cores"], cfg["memory"], cfg["agent"], cfg["boot"])
}
if m := parseDescription(cfg["description"]); !m.Managed || m.UID == "" {
t.Errorf("description %q did not survive decoding", cfg["description"])
}
}},
{capture: "status_current_paused", call: func(a *httpAPI) (any, error) { return a.VMStatus(ctx, "pv1-1", 199999) }, check: func(t *testing.T, got any) {
if s := got.(vmStatus); s.Status != "running" || s.QMPStatus != "paused" {
t.Errorf("status = %+v", s)
}
}},
{capture: "task_status_stopped_error", call: func(a *httpAPI) (any, error) { return a.TaskStatus(ctx, "pv1-1", "UPID:pv1-1:x") }, check: func(t *testing.T, got any) {
if s := got.(taskStatus); s.Status != "stopped" || s.ExitStatus != "shrinking disks is not supported" {
t.Errorf("task = %+v", s)
}
}},
{capture: "agent_network_get_interfaces", call: func(a *httpAPI) (any, error) { return a.AgentNetworkInterfaces(ctx, "pv1-1", 199999) }, check: func(t *testing.T, got any) {
if ip := firstIPv4(got.([]agentInterface)); ip != "192.168.0.150" {
t.Errorf("first IPv4 = %q", ip)
}
}},
{capture: "set_config_response", call: func(a *httpAPI) (any, error) {
return nil, a.SetVMConfig(ctx, "pv1-1", 199999, map[string]string{"cores": "1"})
}, check: func(*testing.T, any) {}},
{capture: "clone_response", call: func(a *httpAPI) (any, error) { return a.Clone(ctx, "pv1-1", 9001, cloneParams{NewID: 199999}) }, check: wantUPID("qmclone")},
{capture: "resize_response", call: func(a *httpAPI) (any, error) { return a.Resize(ctx, "pv1-1", 199999, "scsi0", "4G") }, check: wantUPID("resize")},
{capture: "start_response", call: func(a *httpAPI) (any, error) { return a.Start(ctx, "pv1-1", 199999) }, check: wantUPID("qmstart")},
{capture: "stop_response", call: func(a *httpAPI) (any, error) { return a.Stop(ctx, "pv1-1", 199999) }, check: wantUPID("qmstop")},
{capture: "destroy_response", call: func(a *httpAPI) (any, error) { return a.Destroy(ctx, "pv1-1", 199999) }, check: wantUPID("qmdestroy")},
{capture: "upload_response", call: func(a *httpAPI) (any, error) {
return a.UploadISO(ctx, "pv1-1", "local", "x.iso", strings.NewReader("iso"), 3)
}, check: wantUPID("imgcopy")},
{capture: "content_delete_response", call: func(a *httpAPI) (any, error) {
return a.DeleteVolume(ctx, "pv1-1", "proxy-seeds", "proxy-seeds:iso/x.iso")
}, check: wantUPID("imgdel")},
}
for _, tc := range tests {
t.Run(tc.capture, func(t *testing.T) {
t.Parallel()
c := readCapture(t, tc.capture)
fx := newClientFixture(t, func(w http.ResponseWriter, r *http.Request) {
_, _ = io.Copy(io.Discard, r.Body)
serveCapture(t, w, c)
})
got, err := tc.call(fx.api(t))
if err != nil {
t.Fatalf("call error = %v", err)
}
tc.check(t, got)
})
}
}
func wantUPID(kind string) func(t *testing.T, got any) {
return func(t *testing.T, got any) {
t.Helper()
if upid := got.(string); !strings.HasPrefix(upid, "UPID:pv1-1:") || !strings.Contains(upid, ":"+kind+":") {
t.Errorf("upid = %q, want a %s UPID of pv1-1", upid, kind)
}
}
}
// seenRequest is what a test server records of one request.
type seenRequest struct {
method, path, escapedPath string
query, form url.Values
contentType string
}
func recordingServer(t *testing.T, seen chan<- seenRequest) *clientFixture {
t.Helper()
return newClientFixture(t, func(w http.ResponseWriter, r *http.Request) {
req := seenRequest{method: r.Method, path: r.URL.Path, escapedPath: r.URL.EscapedPath(), query: r.URL.Query(), contentType: r.Header.Get("Content-Type")}
if r.Method == http.MethodPost || r.Method == http.MethodPut {
if err := r.ParseForm(); err != nil {
t.Errorf("parsing form: %v", err)
}
req.form = r.PostForm
}
seen <- req
serveJSON(w, `{"data":null}`)
})
}
// TestClient_methodPathAndParams pins the HTTP method, path and parameters
// of every seam method to the endpoint map of spec entry 17.
func TestClient_methodPathAndParams(t *testing.T) {
t.Parallel()
ctx := context.Background()
const upid = "UPID:pv1-1:0010FBF1:02804D0D:6AB378EA:qmstart:199999:proxy-operator@pve!operator:"
tests := map[string]struct {
call func(a *httpAPI) error
method string
path string
query url.Values
form url.Values
}{
"Version": {call: func(a *httpAPI) error { _, err := a.Version(ctx); return err }, method: "GET", path: "/api2/json/version"},
"NodeStatus": {call: func(a *httpAPI) error { _, err := a.NodeStatus(ctx, "pv1-1"); return err },
method: "GET", path: "/api2/json/nodes/pv1-1/status"},
"StorageStatus": {call: func(a *httpAPI) error { _, err := a.StorageStatus(ctx, "pv1-1", "proxy-seeds"); return err },
method: "GET", path: "/api2/json/nodes/pv1-1/storage/proxy-seeds/status"},
"ClusterResources": {call: func(a *httpAPI) error { _, err := a.ClusterResources(ctx); return err },
method: "GET", path: "/api2/json/cluster/resources", query: url.Values{"type": {"vm"}}},
"VMConfig": {call: func(a *httpAPI) error { _, err := a.VMConfig(ctx, "pv1-1", 150042); return err },
method: "GET", path: "/api2/json/nodes/pv1-1/qemu/150042/config"},
"SetVMConfig": {call: func(a *httpAPI) error {
return a.SetVMConfig(ctx, "pv1-1", 150042, map[string]string{"cores": "1", "ide2": "proxy-seeds:iso/proxy-x-seed.iso,media=cdrom"})
}, method: "PUT", path: "/api2/json/nodes/pv1-1/qemu/150042/config",
form: url.Values{"cores": {"1"}, "ide2": {"proxy-seeds:iso/proxy-x-seed.iso,media=cdrom"}}},
"Clone": {call: func(a *httpAPI) error {
_, err := a.Clone(ctx, "pv1-1", 9001, cloneParams{NewID: 150042, Name: "proxy-x", Description: "a=1\nb=2", Storage: "local-lvm", Target: "pv1-2"})
return err
}, method: "POST", path: "/api2/json/nodes/pv1-1/qemu/9001/clone",
form: url.Values{"newid": {"150042"}, "name": {"proxy-x"}, "description": {"a=1\nb=2"}, "full": {"1"}, "storage": {"local-lvm"}, "target": {"pv1-2"}}},
"Resize": {call: func(a *httpAPI) error { _, err := a.Resize(ctx, "pv1-1", 150042, "scsi0", "8G"); return err },
method: "PUT", path: "/api2/json/nodes/pv1-1/qemu/150042/resize", form: url.Values{"disk": {"scsi0"}, "size": {"8G"}}},
"Start": {call: func(a *httpAPI) error { _, err := a.Start(ctx, "pv1-1", 150042); return err },
method: "POST", path: "/api2/json/nodes/pv1-1/qemu/150042/status/start", form: url.Values{}},
"Stop": {call: func(a *httpAPI) error { _, err := a.Stop(ctx, "pv1-1", 150042); return err },
method: "POST", path: "/api2/json/nodes/pv1-1/qemu/150042/status/stop", form: url.Values{}},
"VMStatus": {call: func(a *httpAPI) error { _, err := a.VMStatus(ctx, "pv1-1", 150042); return err },
method: "GET", path: "/api2/json/nodes/pv1-1/qemu/150042/status/current"},
"Destroy": {call: func(a *httpAPI) error { _, err := a.Destroy(ctx, "pv1-1", 150042); return err },
method: "DELETE", path: "/api2/json/nodes/pv1-1/qemu/150042", query: url.Values{"purge": {"1"}, "destroy-unreferenced-disks": {"1"}}},
"TaskStatus": {call: func(a *httpAPI) error { _, err := a.TaskStatus(ctx, "pv1-1", upid); return err },
method: "GET", path: "/api2/json/nodes/pv1-1/tasks/" + upid + "/status"},
"AgentNetworkInterfaces": {call: func(a *httpAPI) error { _, err := a.AgentNetworkInterfaces(ctx, "pv1-1", 150042); return err },
method: "GET", path: "/api2/json/nodes/pv1-1/qemu/150042/agent/network-get-interfaces"},
"UploadISO": {call: func(a *httpAPI) error {
_, err := a.UploadISO(ctx, "pv1-1", "proxy-seeds", "proxy-x-seed.iso", strings.NewReader("iso"), 3)
return err
}, method: "POST", path: "/api2/json/nodes/pv1-1/storage/proxy-seeds/upload"}, // multipart, see TestClient_uploadStreamsMultipart
"DeleteVolume": {call: func(a *httpAPI) error {
_, err := a.DeleteVolume(ctx, "pv1-1", "proxy-seeds", "proxy-seeds:iso/proxy-x-seed.iso")
return err
}, method: "DELETE", path: "/api2/json/nodes/pv1-1/storage/proxy-seeds/content/proxy-seeds:iso/proxy-x-seed.iso"},
}
for m := range reflect.TypeFor[api]().Methods() {
if _, ok := tests[m.Name]; !ok {
t.Errorf("no row for seam method %s", m.Name)
}
}
for name, tc := range tests {
t.Run(name, func(t *testing.T) {
t.Parallel()
seen := make(chan seenRequest, 1)
fx := recordingServer(t, seen)
if err := tc.call(fx.api(t)); err != nil {
t.Fatalf("call error = %v", err)
}
got := <-seen
if got.method != tc.method || got.path != tc.path {
t.Errorf("request = %s %s, want %s %s", got.method, got.path, tc.method, tc.path)
}
if len(got.query) != 0 || len(tc.query) != 0 {
if !reflect.DeepEqual(got.query, tc.query) {
t.Errorf("query = %v, want %v", got.query, tc.query)
}
}
if tc.form == nil {
if got.form != nil && len(got.form) != 0 {
t.Errorf("form = %v, want none", got.form)
}
return
}
if got.contentType != "application/x-www-form-urlencoded" {
t.Errorf("content type = %q, want a urlencoded form", got.contentType)
}
for k, want := range tc.form {
if !reflect.DeepEqual(got.form[k], want) {
t.Errorf("form %s = %q, want %q", k, got.form[k], want)
}
}
if len(got.form) != len(tc.form) {
t.Errorf("form = %v, want exactly %v", got.form, tc.form)
}
})
}
}
func TestClient_cloneOmitsEmptyStorageAndTarget(t *testing.T) {
t.Parallel()
seen := make(chan seenRequest, 1)
fx := recordingServer(t, seen)
if _, err := fx.api(t).Clone(context.Background(), "pv1-1", 9001, cloneParams{NewID: 150042, Name: "proxy-x"}); err != nil {
t.Fatalf("Clone error = %v", err)
}
got := <-seen
if _, ok := got.form["storage"]; ok {
t.Errorf("form = %v, want no storage", got.form)
}
if _, ok := got.form["target"]; ok {
t.Errorf("form = %v, want no target", got.form)
}
}
// TestClient_destroyQueryFlags checks that destroy leaves nothing behind:
// the purge and unreferenced-disk flags travel in the query of the DELETE
// (spec entry 11).
func TestClient_destroyQueryFlags(t *testing.T) {
t.Parallel()
seen := make(chan seenRequest, 1)
fx := recordingServer(t, seen)
if _, err := fx.api(t).Destroy(context.Background(), "pv1-1", 150042); err != nil {
t.Fatalf("Destroy error = %v", err)
}
got := <-seen
if got.method != http.MethodDelete || got.query.Get("purge") != "1" || got.query.Get("destroy-unreferenced-disks") != "1" {
t.Errorf("request = %s %s?%s, want DELETE with purge=1 and destroy-unreferenced-disks=1", got.method, got.path, got.query.Encode())
}
}
// TestClient_volIDSlashIsEscaped sends the volid's "/" as %2F, the way
// Phase 0 deleted volumes, so the router sees one path segment.
func TestClient_volIDSlashIsEscaped(t *testing.T) {
t.Parallel()
seen := make(chan seenRequest, 1)
fx := recordingServer(t, seen)
if _, err := fx.api(t).DeleteVolume(context.Background(), "pv1-1", "proxy-seeds", "proxy-seeds:iso/proxy-x-seed.iso"); err != nil {
t.Fatalf("DeleteVolume error = %v", err)
}
if got := <-seen; !strings.HasSuffix(got.escapedPath, "/content/proxy-seeds:iso%2Fproxy-x-seed.iso") {
t.Errorf("escaped path = %q, want the volid as one segment with %%2F", got.escapedPath)
}
}
func TestClient_http1Only(t *testing.T) {
t.Parallel()
protos := make(chan int, 2)
srv := httptest.NewUnstartedServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
protos <- r.ProtoMajor
serveJSON(w, readCapture(t, "version").Body)
}))
srv.EnableHTTP2 = true
srv.StartTLS()
t.Cleanup(srv.Close)
resp, err := srv.Client().Get(srv.URL)
if err != nil {
t.Fatalf("sanity request error = %v", err)
}
_ = resp.Body.Close()
if p := <-protos; p != 2 {
t.Fatalf("sanity: server spoke HTTP/%d to an h2 client, want 2", p)
}
if _, err := fixtureFor(t, srv).api(t).Version(context.Background()); err != nil {
t.Fatalf("Version error = %v", err)
}
if p := <-protos; p != 1 {
t.Errorf("client spoke HTTP/%d, want HTTP/1.1 (F18)", p)
}
}
func TestClient_requestTimeoutIsTransient(t *testing.T) {
t.Parallel()
fx := newClientFixture(t, func(_ http.ResponseWriter, r *http.Request) {
select {
case <-r.Context().Done():
case <-time.After(5 * time.Second):
}
})
a := fx.api(t)
a.timeout = 100 * time.Millisecond
start := time.Now()
_, err := a.Version(context.Background())
if elapsed := time.Since(start); elapsed > 2*time.Second {
t.Errorf("Version took %v, want the 100ms request timeout to cut it", elapsed)
}
var ne net.Error
if !errors.As(err, &ne) || !ne.Timeout() {
t.Errorf("error = %v, want a timeout", err)
}
assertClass(t, err, provider.ErrTransient)
}
// TestClient_cancelledContextReturnsPromptly also checks that the
// cancellation touches only that request: the next call on the same client
// succeeds.
func TestClient_cancelledContextReturnsPromptly(t *testing.T) {
t.Parallel()
fx := newClientFixture(t, func(w http.ResponseWriter, r *http.Request) {
if strings.HasSuffix(r.URL.Path, "/status") {
select {
case <-r.Context().Done():
case <-time.After(5 * time.Second):
}
return
}
serveJSON(w, readCapture(t, "version").Body)
})
a := fx.api(t)
ctx, cancel := context.WithCancel(context.Background())
time.AfterFunc(50*time.Millisecond, cancel)
start := time.Now()
_, err := a.NodeStatus(ctx, "pv1-1")
if elapsed := time.Since(start); elapsed > time.Second {
t.Errorf("NodeStatus took %v after cancel, want prompt", elapsed)
}
if !errors.Is(err, context.Canceled) {
t.Errorf("error = %v, want context.Canceled", err)
}
assertClass(t, err, provider.ErrTransient)
if _, err := a.Version(context.Background()); err != nil {
t.Errorf("next call error = %v, want the client unaffected", err)
}
}
func tracedFixture(t *testing.T) (*clientFixture, *tracetest.SpanRecorder, *sdktrace.TracerProvider, func() []string) {
t.Helper()
var (
mu sync.Mutex
traceparent []string
)
fx := newClientFixture(t, func(w http.ResponseWriter, r *http.Request) {
mu.Lock()
traceparent = append(traceparent, r.Header.Get("Traceparent"))
mu.Unlock()
serveJSON(w, readCapture(t, "version").Body)
})
sr := tracetest.NewSpanRecorder()
tp := sdktrace.NewTracerProvider(sdktrace.WithSpanProcessor(sr))
t.Cleanup(func() { _ = tp.Shutdown(context.Background()) })
sent := func() []string {
mu.Lock()
defer mu.Unlock()
return append([]string(nil), traceparent...)
}
return fx, sr, tp, sent
}
func TestClient_childSpanUnderParent(t *testing.T) {
t.Parallel()
fx, sr, tp, traceparent := tracedFixture(t)
a := fx.api(t, tracing.WithTracerProvider(tp))
ctx, parent := tp.Tracer("test").Start(context.Background(), "provider.get")
if _, err := a.Version(ctx); err != nil {
t.Fatalf("Version error = %v", err)
}
parent.End()
var child sdktrace.ReadOnlySpan
for _, s := range sr.Ended() {
if s.Name() != "provider.get" {
child = s
}
}
if child == nil {
t.Fatalf("no HTTP span recorded; spans: %d", len(sr.Ended()))
}
if child.Parent().SpanID() != parent.SpanContext().SpanID() || child.SpanContext().TraceID() != parent.SpanContext().TraceID() {
t.Errorf("HTTP span %q is not a child of provider.get", child.Name())
}
secret := testToken[strings.Index(testToken, "=")+1:]
for _, kv := range child.Attributes() {
if strings.Contains(kv.Value.Emit(), secret) {
t.Errorf("span attribute %s carries the token", kv.Key)
}
}
if sent := traceparent(); len(sent) != 1 || !strings.Contains(sent[0], child.SpanContext().TraceID().String()) {
t.Errorf("traceparent sent = %q, want the trace id", sent)
}
}
func TestClient_noSpanWithoutParent(t *testing.T) {
t.Parallel()
fx, sr, tp, traceparent := tracedFixture(t)
if _, err := fx.api(t, tracing.WithTracerProvider(tp)).Version(context.Background()); err != nil {
t.Fatalf("Version error = %v", err)
}
if n := len(sr.Ended()); n != 0 {
t.Errorf("got %d spans without a parent, want none", n)
}
if sent := traceparent(); len(sent) != 1 || sent[0] != "" {
t.Errorf("traceparent sent = %q, want none", sent)
}
}
// TestClient_tokenNeverInErrorsOrLogs has the server echo the secret back
// in every place an error's text comes from; neither the error nor the
// decorator's V(2) lines may carry it.
func TestClient_tokenNeverInErrorsOrLogs(t *testing.T) {
t.Parallel()
const secret = "SENTINEL-SECRET-9d41"
token := "proxy-operator@pve!operator=" + secret
tests := []struct {
name string
resp func(auth string) capture
}{
{name: "401 body echoes the header", resp: func(auth string) capture {
return capture{Status: 401, Reason: "Authentication failed!", Body: `{"data":null,"message":"bad credentials: ` + auth + `\n"}`}
}},
{name: "401 reason echoes the header", resp: func(auth string) capture {
return capture{Status: 401, Reason: "bad credentials " + auth}
}},
{name: "400 errors echo the header", resp: func(auth string) capture {
return capture{Status: 400, Reason: "Parameter verification failed.", Body: `{"data":null,"message":"Parameter verification failed.\n","errors":{"token":"` + auth + `"}}`}
}},
}
for _, tc := range tests {
t.Run(tc.name, func(t *testing.T) {
t.Parallel()
fx := newClientFixture(t, func(w http.ResponseWriter, r *http.Request) {
serveCapture(t, w, tc.resp(r.Header.Get("Authorization")))
})
writeFile(t, fx.tokenFile, token)
logs := &logCapture{}
ctx := logs.context(2)
_, err := observe(fx.api(t), "proxmox-home").Version(ctx)
var ae *apiError
if !errors.As(err, &ae) {
t.Fatalf("error = %v, want an apiError", err)
}
if !strings.Contains(err.Error(), "<redacted>") {
t.Errorf("error = %v, want the echoed secret redacted", err)
}
for _, s := range append([]string{err.Error(), fmt.Sprintf("%+v", err), fmt.Sprintf("%#v", ae)}, logs.all()...) {
if strings.Contains(s, secret) {
t.Errorf("secret leaked: %s", s)
}
}
if len(logs.all()) != 1 {
t.Errorf("got %d log lines, want the one call line", len(logs.all()))
}
})
}
}
func TestClient_caFileTrustsPrivateCA(t *testing.T) {
t.Parallel()
fx := newClientFixture(t, func(w http.ResponseWriter, _ *http.Request) { serveJSON(w, readCapture(t, "version").Body) })
if _, err := fx.api(t).Version(context.Background()); err != nil {
t.Errorf("with the CA file: error = %v, want nil", err)
}
fx.cfg.CAFile = ""
_, err := fx.api(t).Version(context.Background())
if !isTLSVerification(err) {
t.Errorf("with system roots: error = %v, want a verification failure", err)
}
}
// TestClient_caFileLoadedLazily covers spec entry 14: a missing CA file is
// Transient and retried until it lands; one without a certificate is
// Permanent.
func TestClient_caFileLoadedLazily(t *testing.T) {
t.Parallel()
fx := newClientFixture(t, func(w http.ResponseWriter, _ *http.Request) { serveJSON(w, readCapture(t, "version").Body) })
pemBytes, err := os.ReadFile(fx.caFile)
if err != nil {
t.Fatal(err)
}
fx.cfg.CAFile = filepath.Join(t.TempDir(), "ca.pem")
a := fx.api(t)
if _, err := a.Version(context.Background()); !errors.Is(err, provider.ErrTransient) {
t.Errorf("missing CA file: error = %v, want ErrTransient", err)
}
writeFile(t, fx.cfg.CAFile, "not a certificate")
if _, err := a.Version(context.Background()); !errors.Is(err, provider.ErrPermanent) {
t.Errorf("CA file without a certificate: error = %v, want ErrPermanent", err)
}
writeFile(t, fx.cfg.CAFile, string(pemBytes))
if _, err := a.Version(context.Background()); err != nil {
t.Errorf("CA file landed: error = %v, want nil", err)
}
}
func TestClient_x509FailureIsPermanent(t *testing.T) {
t.Parallel()
tests := []struct {
name string
mutate func(fx *clientFixture)
}{
{name: "unknown authority", mutate: func(fx *clientFixture) { fx.cfg.CAFile = "" }},
{name: "host not in the certificate", mutate: func(fx *clientFixture) {
fx.cfg.URL = strings.Replace(fx.srv.URL, "127.0.0.1", "localhost", 1)
}},
}
for _, tc := range tests {
t.Run(tc.name, func(t *testing.T) {
t.Parallel()
var requests atomic.Int32
fx := newClientFixture(t, func(http.ResponseWriter, *http.Request) { requests.Add(1) })
tc.mutate(fx)
_, err := fx.api(t).Version(context.Background())
if !isTLSVerification(err) {
t.Errorf("error = %v, want a verification failure", err)
}
assertClass(t, err, provider.ErrPermanent)
if n := requests.Load(); n != 0 {
t.Errorf("server saw %d requests, want none past the handshake", n)
}
})
}
}
func TestClient_insecureSkipsVerification(t *testing.T) {
t.Parallel()
fx := newClientFixture(t, func(w http.ResponseWriter, _ *http.Request) { serveJSON(w, readCapture(t, "version").Body) })
fx.cfg.CAFile = ""
fx.cfg.InsecureSkipTLSVerify = true
if _, err := fx.api(t).Version(context.Background()); err != nil {
t.Errorf("error = %v, want nil with verification off", err)
}
}
// gatedReader hands out first, then waits for release before handing out
// rest. A client that read the whole body before sending would never get
// past first.
type gatedReader struct {
first, rest []byte
release <-chan struct{}
waited bool
}
func (g *gatedReader) Read(p []byte) (int, error) {
if len(g.first) > 0 {
n := copy(p, g.first)
g.first = g.first[n:]
return n, nil
}
if !g.waited {
select {
case <-g.release:
case <-time.After(5 * time.Second):
return 0, errors.New("the server saw nothing of the file before the reader was drained: the body is not streamed")
}
g.waited = true
}
if len(g.rest) == 0 {
return 0, io.EOF
}
n := copy(p, g.rest)
g.rest = g.rest[n:]
return n, nil
}
func TestClient_uploadStreamsMultipart(t *testing.T) {
t.Parallel()
first := bytes.Repeat([]byte("a"), 256<<10)
rest := bytes.Repeat([]byte("b"), 256<<10)
sawFirst := make(chan struct{})
type upload struct {
contentLength int64
chunked bool
content, name string
fileName, bytes string
}
got := make(chan upload, 1)
fx := newClientFixture(t, func(w http.ResponseWriter, r *http.Request) {
u := upload{contentLength: r.ContentLength, chunked: len(r.TransferEncoding) > 0}
mr, err := r.MultipartReader()
if err != nil {
t.Errorf("multipart reader: %v", err)
return
}
for {
part, err := mr.NextPart()
if err == io.EOF {
break
}
if err != nil {
t.Errorf("next part: %v", err)
return
}
switch part.FormName() {
case "content":
b, _ := io.ReadAll(part)
u.content = string(b)
case "filename":
u.name, u.fileName = part.FormName(), part.FileName()
var buf bytes.Buffer
if _, err := io.CopyN(&buf, part, 1); err != nil {
t.Errorf("reading the file part: %v", err)
return
}
close(sawFirst)
_, _ = io.Copy(&buf, part)
u.bytes = buf.String()
}
}
got <- u
serveJSON(w, readCapture(t, "upload_response").Body)
})
body := &gatedReader{first: first, rest: rest, release: sawFirst}
size := int64(len(first) + len(rest))
upid, err := fx.api(t).UploadISO(context.Background(), "pv1-1", "proxy-seeds", "proxy-x-seed.iso", body, size)
if err != nil {
t.Fatalf("UploadISO error = %v", err)
}
if !strings.Contains(upid, ":imgcopy:") {
t.Errorf("upid = %q, want the imgcopy task", upid)
}
u := <-got
if u.chunked || u.contentLength <= size {
t.Errorf("Content-Length %d (chunked %v), want an exact length above the file's %d bytes", u.contentLength, u.chunked, size)
}
if u.content != "iso" || u.name != "filename" || u.fileName != "proxy-x-seed.iso" {
t.Errorf("fields content=%q part=%q file=%q, want iso, filename, proxy-x-seed.iso (F12)", u.content, u.name, u.fileName)
}
if u.bytes != string(first)+string(rest) {
t.Errorf("server got %d file bytes, want the %d sent", len(u.bytes), size)
}
}
func TestDecodeConfig_scalarsAsStrings(t *testing.T) {
t.Parallel()
in := map[string]json.RawMessage{"cores": json.RawMessage(`2`), "memory": json.RawMessage(`"2048"`), "template": json.RawMessage(`1`), "balloon": json.RawMessage(`null`), "description": json.RawMessage(`"a\nb"`)}
got, err := decodeConfig(in)
if err != nil {
t.Fatalf("decodeConfig error = %v", err)
}
want := vmConfig{"cores": "2", "memory": "2048", "template": "1", "description": "a\nb"}
if !reflect.DeepEqual(got, want) {
t.Errorf("decodeConfig = %v, want %v", got, want)
}
}
func TestReadToken_trimsTrailingNewline(t *testing.T) {
t.Parallel()
path := filepath.Join(t.TempDir(), "token")
writeFile(t, path, " "+testToken+"\r\n")
if got, err := readToken(path); err != nil || got != testToken {
t.Errorf("readToken = %q, %v; want the token without whitespace", got, err)
}
}
+428
View File
@@ -0,0 +1,428 @@
package proxmox
import (
"context"
"fmt"
"maps"
"regexp"
"slices"
"strconv"
"strings"
"github.com/go-logr/logr"
"gitea.home.hrajfrisbee.cz/kacerr/egress-proxies-operator/internal/provider"
)
// maxProbes bounds the VMID probe of spec entry 2.
const maxProbes = 32
// diskKey is a Proxmox disk slot: the keys a boot disk, a CD-ROM or a
// cloud-init drive can sit on.
var diskKey = regexp.MustCompile(`^(ide|sata|scsi|virtio)[0-9]+$`)
// ownedVM is a VM Create has cloned or adopted: where it lives and its
// config as last read, which the config and resize steps start from.
type ownedVM struct {
node string
vmid int
cfg vmConfig
}
// Create is spec entry 10: validate placement, then, under the per-name
// lock, adopt the VM a previous attempt left or clone a new one, and carry
// it through seed, config, resize and start until it runs.
func (p *Provider) Create(ctx context.Context, req provider.CreateRequest) (string, error) {
node, mt, tmpl, err := p.placement(req)
if err != nil {
return "", p.wrapErr("create", "", err)
}
log := p.logger(ctx).WithValues("name", req.Name)
log.V(2).Info("proxmox create resolved",
"zone", node, "machineType", req.Placement.MachineType, "cores", mt.Cores, "memoryMiB", mt.MemoryMiB,
"diskGiB", mt.DiskGiB, "image", req.Placement.Image, "templateVMID", tmpl.VMID, "storage", p.cfg.Storage,
"isoStorage", p.cfg.ISOStorage, "bridge", p.cfg.Bridge, "vlanTag", p.cfg.VLANTag, "cloudInitBytes", len(req.CloudInit))
unlock, err := p.lockName(ctx, req.Name)
if err != nil {
return "", p.wrapErr("create", "", err)
}
defer unlock()
vm, err := p.create(ctx, log, node, mt, tmpl, req)
if err != nil {
return "", p.wrapErr("create", "", err)
}
log.V(1).Info("proxmox vm ensured running", "vmid", vm.vmid, "node", vm.node)
return formatProviderID(vm.vmid, req.Name), nil
}
// placement is step 1: every catalogue key is checked before any backend
// call, and an unknown one is Permanent naming the field and the value
// (spec entry 3). Region is ignored.
func (p *Provider) placement(req provider.CreateRequest) (string, provider.ProxmoxMachineType, provider.ProxmoxTemplate, error) {
var (
mt provider.ProxmoxMachineType
tmpl provider.ProxmoxTemplate
)
if req.Name == "" {
return "", mt, tmpl, permanentf("the request has no instance name")
}
node, err := p.node(req.Placement.Zone)
if err != nil {
return "", mt, tmpl, err
}
mt, ok := p.cfg.MachineTypes[req.Placement.MachineType]
if !ok {
return "", mt, tmpl, permanentf("placement.machineType %q is not in proxmox.machineTypes", req.Placement.MachineType)
}
tmpl, ok = p.cfg.Templates[req.Placement.Image]
if !ok {
return "", mt, tmpl, permanentf("placement.image %q is not in proxmox.templates", req.Placement.Image)
}
return node, mt, tmpl, nil
}
// create is steps 2 to 9. The template is checked only when a clone is
// needed: a VM a previous attempt created no longer depends on it (spec
// entry 4), so a template removed since must not fail its adoption.
func (p *Provider) create(ctx context.Context, log logr.Logger, node string, mt provider.ProxmoxMachineType, tmpl provider.ProxmoxTemplate, req provider.CreateRequest) (ownedVM, error) {
rs, err := p.api.ClusterResources(ctx)
if err != nil {
return ownedVM{}, fmt.Errorf("listing cluster resources: %w", err)
}
vm, adopted, err := p.adoptByName(ctx, rs, req)
if err != nil {
return ownedVM{}, err
}
if !adopted {
src, err := p.template(ctx, rs, req.Placement.Image, tmpl.VMID, mt)
if err != nil {
return ownedVM{}, err
}
if vm, adopted, err = p.allocate(ctx, log, rs, node, src, req); err != nil {
return ownedVM{}, err
}
}
log = log.WithValues("vmid", vm.vmid, "node", vm.node)
if adopted {
st, err := p.api.VMStatus(ctx, vm.node, vm.vmid)
if err != nil {
return ownedVM{}, fmt.Errorf("reading status of VM %d: %w", vm.vmid, err)
}
log.V(1).Info("proxmox vm adopted", "status", st.Status, "qmpstatus", st.QMPStatus)
if st.Status == "running" {
return vm, nil
}
}
return vm, p.provision(ctx, log, vm, mt, req)
}
// adoptByName is the by-name half of spec entry 2: two VMs with the name
// are for an admin to sort out, one goes through the ownership check.
func (p *Provider) adoptByName(ctx context.Context, rs []vmResource, req provider.CreateRequest) (ownedVM, bool, error) {
var found []vmResource
for _, r := range rs {
if r.Type == "qemu" && r.Template == 0 && r.Name == req.Name {
found = append(found, r)
}
}
switch len(found) {
case 0:
return ownedVM{}, false, nil
case 1:
return p.inspect(ctx, found[0], req)
}
vmids := make([]string, len(found))
for i, r := range found {
vmids[i] = strconv.Itoa(r.VMID)
}
return ownedVM{}, false, permanentf("VMs %s are all named %s; remove all but one", strings.Join(vmids, ", "), req.Name)
}
// inspect is the ownership check of spec entry 2 on a VMID that is taken.
// It waits out a task's lock (a clone in progress is also nameless), then
// reads the name and the mark: another name is someone else's VM, and our
// name without our mark is never taken over.
func (p *Provider) inspect(ctx context.Context, r vmResource, req provider.CreateRequest) (ownedVM, bool, error) {
cfg, err := p.waitUnlocked(ctx, r.Node, r.VMID, p.budgets.clone)
if err != nil {
return ownedVM{}, false, err
}
if cfg["name"] != req.Name {
return ownedVM{}, false, nil
}
mark := parseDescription(cfg["description"])
switch {
case !mark.Managed:
return ownedVM{}, false, permanentf("VM %d is named %s but carries no proxy-operator mark; it was not created by this operator", r.VMID, req.Name)
case mark.UID != "" && mark.UID != req.UID:
return ownedVM{}, false, permanentf("VM %d named %s belongs to proxy uid %s, not %s", r.VMID, req.Name, mark.UID, req.UID)
case cfg["lock"] == lockSuspended:
return ownedVM{}, false, permanentf("VM %d named %s is hibernated (lock suspended); resume or destroy it in Proxmox", r.VMID, req.Name)
}
return ownedVM{node: r.Node, vmid: r.VMID, cfg: cfg}, true, nil
}
// template finds the clone source and refuses one that is not a template
// or brings a Proxmox cloud-init drive, which would compete with the seed
// (spec entry 4). The clone copies the template's config, so what the
// config or resize step would refuse on the clone is refused here, while no
// VM exists that a fixed template could no longer help.
func (p *Provider) template(ctx context.Context, rs []vmResource, image string, vmid int, mt provider.ProxmoxMachineType) (vmResource, error) {
i := slices.IndexFunc(rs, func(r vmResource) bool { return r.VMID == vmid && r.Type == "qemu" })
if i < 0 {
return vmResource{}, permanentf("placement.image %q: template VM %d does not exist", image, vmid)
}
src := rs[i]
cfg, err := p.api.VMConfig(ctx, src.Node, vmid)
if err != nil {
return vmResource{}, fmt.Errorf("reading config of template %d: %w", vmid, err)
}
if cfg["template"] != "1" {
return vmResource{}, permanentf("placement.image %q: VM %d is not a template", image, vmid)
}
for _, k := range slices.Sorted(maps.Keys(cfg)) {
if diskKey.MatchString(k) && strings.Contains(cfg[k], "cloudinit") {
return vmResource{}, permanentf("placement.image %q: template %d has a Proxmox cloud-init drive on %s; remove it, the provider attaches its own seed", image, vmid, k)
}
}
shape := ownedVM{node: src.Node, vmid: vmid, cfg: cfg}
if _, err := p.configParams(shape, mt, ""); err != nil {
return vmResource{}, err
}
if _, _, _, err := resizeTarget(shape, mt.DiskGiB); err != nil {
return vmResource{}, err
}
return src, nil
}
// allocate is the probe of spec entry 2: the attempt-th VMID for the UID
// is cloned onto when the listing shows it free, and put through the
// ownership check when it is taken, or when the clone finds it taken after
// all (F4), by a VM that is nameless or carries our name. The boolean
// reports an adoption rather than a fresh clone.
func (p *Provider) allocate(ctx context.Context, log logr.Logger, rs []vmResource, node string, src vmResource, req provider.CreateRequest) (ownedVM, bool, error) {
taken := make(map[int]vmResource, len(rs))
for _, r := range rs {
taken[r.VMID] = r
}
rng := p.cfg.VMIDRange
probes := min(maxProbes, rng.Max-rng.Min+1)
for attempt := range probes {
vmid := vmidFor(req.UID, rng.Min, rng.Max, attempt)
r, isTaken := taken[vmid]
if !isTaken {
vm, err := p.clone(ctx, log, node, src, vmid, req)
if !isAlreadyExists(err) {
return vm, false, err
}
var found bool
if r, found, err = p.resolve(ctx, vmid); err != nil {
return ownedVM{}, false, err
}
if !found {
log.V(1).Info("proxmox vmid taken by something other than a VM, probing on", "vmid", vmid, "attempt", attempt)
continue
}
}
// A listed name other than ours is someone else's VM whatever its
// lock: only a clone in progress is nameless.
if r.Type == "qemu" && r.Template == 0 && (r.Name == "" || r.Name == req.Name) {
vm, ours, err := p.inspect(ctx, r, req)
if err != nil || ours {
return vm, ours, err
}
}
log.V(1).Info("proxmox vmid taken by another vm, probing on", "vmid", vmid, "attempt", attempt)
}
return ownedVM{}, false, transientf("no free VMID for %s after %d probes in %d..%d", req.Name, probes, rng.Min, rng.Max)
}
// clone is step 4: a full clone carrying the mark, so the VM is never
// unmarked (spec entry 8), then its unlocked config. A taken VMID comes
// back as the API's error for allocate to judge.
func (p *Provider) clone(ctx context.Context, log logr.Logger, node string, src vmResource, vmid int, req provider.CreateRequest) (ownedVM, error) {
params := cloneParams{
NewID: vmid,
Name: req.Name,
Description: formatDescription(req.UID, p.now()),
Storage: p.cfg.Storage,
}
if node != src.Node {
params.Target = node
}
upid, err := p.api.Clone(ctx, src.Node, src.VMID, params)
if err != nil {
return ownedVM{}, fmt.Errorf("cloning template %d to VM %d: %w", src.VMID, vmid, err)
}
if err := p.waitTask(ctx, log, upid, p.budgets.clone); err != nil {
return ownedVM{}, fmt.Errorf("cloning template %d to VM %d: %w", src.VMID, vmid, err)
}
cfg, err := p.waitUnlocked(ctx, node, vmid, p.budgets.clone)
if err != nil {
return ownedVM{}, err
}
return ownedVM{node: node, vmid: vmid, cfg: cfg}, nil
}
// provision is steps 5 to 9, each safe to repeat on a VM a previous attempt
// left stopped: the upload overwrites, the config write replaces, and the
// resize is skipped once the disk is large enough. Everything that can be
// refused from the config alone is checked before the first write.
func (p *Provider) provision(ctx context.Context, log logr.Logger, vm ownedVM, mt provider.ProxmoxMachineType, req provider.CreateRequest) error {
params, err := p.configParams(vm, mt, req.Name)
if err != nil {
return err
}
disk, size, grow, err := resizeTarget(vm, mt.DiskGiB)
if err != nil {
return err
}
if err := p.uploadSeed(ctx, log, vm.node, req); err != nil {
return err
}
if err := p.api.SetVMConfig(ctx, vm.node, vm.vmid, params); err != nil {
return fmt.Errorf("configuring VM %d: %w", vm.vmid, err)
}
if grow {
upid, err := p.api.Resize(ctx, vm.node, vm.vmid, disk, size)
if err != nil {
return fmt.Errorf("resizing %s of VM %d to %s: %w", disk, vm.vmid, size, err)
}
if err := p.waitTask(ctx, log, upid, p.budgets.resize); err != nil {
return fmt.Errorf("resizing %s of VM %d to %s: %w", disk, vm.vmid, size, err)
}
}
upid, err := p.api.Start(ctx, vm.node, vm.vmid)
if err != nil {
return fmt.Errorf("starting VM %d: %w", vm.vmid, err)
}
if err := p.waitTask(ctx, log, upid, p.budgets.start); err != nil {
return fmt.Errorf("starting VM %d: %w", vm.vmid, err)
}
return p.waitRunning(ctx, vm.node, vm.vmid)
}
// configParams is step 6's write: size, agent, tag and seed, and net0 only
// when the config sets a bridge or a VLAN tag (spec entry 7).
func (p *Provider) configParams(vm ownedVM, mt provider.ProxmoxMachineType, name string) (map[string]string, error) {
params := map[string]string{
"cores": strconv.Itoa(int(mt.Cores)),
"memory": strconv.FormatInt(mt.MemoryMiB, 10),
"agent": "1",
"tags": proxyTag,
"ide2": seedVolID(p.cfg.ISOStorage, name) + ",media=cdrom",
}
if p.cfg.Bridge == "" && p.cfg.VLANTag == 0 {
return params, nil
}
net0 := vm.cfg["net0"]
if net0 == "" {
return nil, permanentf("VM %d has no net0 to apply proxmox.bridge or proxmox.vlanTag to; the template needs a network device", vm.vmid)
}
params["net0"] = rewriteNet0(net0, p.cfg.Bridge, p.cfg.VLANTag)
return params, nil
}
// rewriteNet0 replaces or appends bridge= and tag= in a net0 value, keeping
// every other option, the model and the MAC in place. An empty bridge or a
// zero tag leaves that option alone.
func rewriteNet0(net0, bridge string, tag int) string {
want := map[string]string{}
if bridge != "" {
want["bridge"] = bridge
}
if tag != 0 {
want["tag"] = strconv.Itoa(tag)
}
opts := strings.Split(net0, ",")
for i, opt := range opts {
key, _, _ := strings.Cut(opt, "=")
if v, ok := want[key]; ok {
opts[i] = key + "=" + v
delete(want, key)
}
}
for _, key := range []string{"bridge", "tag"} {
if v, ok := want[key]; ok {
opts = append(opts, key+"="+v)
}
}
return strings.Join(opts, ",")
}
// resizeTarget decides step 7: grow the boot disk to diskGiB when that is
// larger than its size= (spec entry 6). A disk is never shrunk.
func resizeTarget(vm ownedVM, diskGiB int64) (disk, size string, grow bool, err error) {
if diskGiB == 0 {
return "", "", false, nil
}
disk, ok := bootDisk(vm.cfg)
if !ok {
return "", "", false, permanentf("VM %d has no boot disk to grow to %dG", vm.vmid, diskGiB)
}
cur, ok := diskBytes(vm.cfg[disk])
if !ok {
return "", "", false, permanentf("boot disk %s of VM %d has no readable size in %q", disk, vm.vmid, vm.cfg[disk])
}
if cur >= diskGiB<<30 {
return "", "", false, nil
}
return disk, strconv.FormatInt(diskGiB, 10) + "G", true, nil
}
// bootDisk is the first disk of boot: order=… that is not a CD-ROM, else
// the legacy bootdisk key, else the first present of scsi0, virtio0,
// sata0 and ide0. Proxmox appends the seed's ide2 to the order (F19), so
// CD-ROMs are skipped.
func bootDisk(cfg vmConfig) (string, bool) {
isDisk := func(k string) bool {
v, ok := cfg[k]
return ok && diskKey.MatchString(k) && diskOption(v, "media") != "cdrom"
}
for dev := range strings.SplitSeq(diskOption(cfg["boot"], "order"), ";") {
if isDisk(dev) {
return dev, true
}
}
if k := cfg["bootdisk"]; isDisk(k) {
return k, true
}
for _, k := range []string{"scsi0", "virtio0", "sata0", "ide0"} {
if isDisk(k) {
return k, true
}
}
return "", false
}
// diskOption returns the value of key= among a config value's
// comma-separated options.
func diskOption(value, key string) string {
for opt := range strings.SplitSeq(value, ",") {
if v, ok := strings.CutPrefix(opt, key+"="); ok {
return v
}
}
return ""
}
// diskBytes reads a disk value's size=, in Proxmox's <n>[KMGT] notation
// (bytes without a suffix).
func diskBytes(disk string) (int64, bool) {
s := diskOption(disk, "size")
if s == "" {
return 0, false
}
shift := map[byte]uint{'K': 10, 'M': 20, 'G': 30, 'T': 40}
unit, ok := shift[s[len(s)-1]]
if ok {
s = s[:len(s)-1]
}
n, err := strconv.ParseFloat(s, 64)
if err != nil || n < 0 {
return 0, false
}
return int64(n * float64(int64(1)<<unit)), true
}
File diff suppressed because it is too large Load Diff
+152
View File
@@ -0,0 +1,152 @@
package proxmox
import (
"context"
"errors"
"fmt"
"time"
"github.com/go-logr/logr"
"gitea.home.hrajfrisbee.cz/kacerr/egress-proxies-operator/internal/provider"
)
// Delete is spec entry 11: under the per-name lock, stop the VM when it is
// ours, delete its seed, then destroy it. Everything already gone is
// success, and nothing of ours is left when nil is returned, so Get
// afterwards is NotFound.
func (p *Provider) Delete(ctx context.Context, id string) error {
vmid, name, err := parseProviderID(id)
if err != nil {
return p.wrapErr("delete", id, err)
}
log := p.logger(ctx).WithValues("vmid", vmid, "name", name)
unlock, err := p.lockName(ctx, name)
if err != nil {
return p.wrapErr("delete", id, err)
}
defer unlock()
node, err := p.destroy(ctx, log, vmid, name)
if err != nil {
return p.wrapErr("delete", id, err)
}
if node == "" {
if err := p.deleteSeedEverywhere(ctx, log, name); err != nil {
return p.wrapErr("delete", id, err)
}
} else {
log = log.WithValues("node", node)
}
log.V(1).Info("proxmox vm deleted", "destroyed", node != "")
return nil
}
// destroy is steps 2 to 7. It returns the node of the VM it destroyed, or
// "" when there was no VM of ours to destroy: then the seed's node is not
// known, and the seed is looked for on every node.
func (p *Provider) destroy(ctx context.Context, log logr.Logger, vmid int, name string) (string, error) {
r, ok, err := p.resolve(ctx, vmid)
if err != nil {
return "", err
}
if !ok {
log.V(1).Info("proxmox vm already gone")
return "", nil
}
log = log.WithValues("node", r.Node)
if r.Lock != "" && r.Lock != lockSuspended {
return "", transientf("VM %d is locked (%s)", vmid, r.Lock)
}
if r.Status == "unknown" {
return "", transientf("VM %d has status unknown: its node %s is not answering, or the VM is too new to be reported yet", vmid, r.Node)
}
cfg, err := p.api.VMConfig(ctx, r.Node, vmid)
if isNotExist(err) {
log.V(1).Info("proxmox vm already gone")
return "", nil
}
if err != nil {
return "", fmt.Errorf("reading config of VM %d: %w", vmid, err)
}
if cfg["name"] != name {
log.V(1).Info("proxmox vm at this vmid is not ours, left alone", "vmName", cfg["name"])
return "", nil
}
if !parseDescription(cfg["description"]).Managed {
log.V(1).Info("proxmox vm carries no proxy-operator mark, left alone")
return "", nil
}
// A hibernated VM is stopped with lock suspended, and the destroy
// takes it, state volume included (captured on pv1-1).
if lock := cfg["lock"]; lock != "" && lock != lockSuspended {
return "", transientf("VM %d is locked (%s)", vmid, lock)
}
// The listing shows a paused VM as running, and Stop ends it too.
if r.Status != "stopped" {
if err := p.runTask(ctx, log, "stopping", vmid, p.budgets.stop, func() (string, error) { return p.api.Stop(ctx, r.Node, vmid) }); err != nil {
return "", err
}
}
// The seed goes while the VM is still there: once it is gone, Get is
// NotFound and nothing calls Delete again to finish the job.
if err := p.deleteSeed(ctx, log, r.Node, name); err != nil {
return "", err
}
if err := p.runTask(ctx, log, "destroying", vmid, p.budgets.destroy, func() (string, error) { return p.api.Destroy(ctx, r.Node, vmid) }); err != nil {
return "", err
}
return r.Node, nil
}
// runTask submits a task on a VM and waits for it. A VM that is gone by the
// time the task is submitted is success: it is what the task was for.
func (p *Provider) runTask(ctx context.Context, log logr.Logger, what string, vmid int, budget time.Duration, submit func() (string, error)) error {
upid, err := submit()
if isNotExist(err) {
log.V(1).Info("proxmox vm already gone")
return nil
}
if err != nil {
return fmt.Errorf("%s VM %d: %w", what, vmid, err)
}
if err := p.waitTask(ctx, log, upid, budget); err != nil {
return fmt.Errorf("%s VM %d: %w", what, vmid, err)
}
return nil
}
// deleteSeed is step 6 on one node. A missing volume still ends in a
// successful task (F13), so nothing is matched.
func (p *Provider) deleteSeed(ctx context.Context, log logr.Logger, node, name string) error {
volid := seedVolID(p.cfg.ISOStorage, name)
upid, err := p.api.DeleteVolume(ctx, node, p.cfg.ISOStorage, volid)
if err != nil {
return fmt.Errorf("deleting seed %s on %s: %w", volid, node, err)
}
if err := p.waitTask(ctx, log, upid, p.budgets.seedDelete); err != nil {
return fmt.Errorf("deleting seed %s on %s: %w", volid, node, err)
}
return nil
}
// deleteSeedEverywhere is step 6 when the VM's node is not known: every
// configured node is tried, and one that fails is logged and skipped, so a
// node that is down or lacks the storage cannot hold the Delete up. An
// ended ctx fails the call instead, and so does a Permanent refusal such as
// a 403 for Datastore.Allocate, which every node would give and which
// would otherwise leave the seed behind unseen.
func (p *Provider) deleteSeedEverywhere(ctx context.Context, log logr.Logger, name string) error {
for _, node := range p.cfg.Nodes {
err := p.deleteSeed(ctx, log, node, name)
if err == nil {
continue
}
if ctx.Err() != nil || errors.Is(classify("delete", err), provider.ErrPermanent) {
return err
}
log.V(1).Info("proxmox seed delete failed on a node, skipped", "node", node, "reason", err.Error())
}
return nil
}
+760
View File
@@ -0,0 +1,760 @@
package proxmox
import (
"context"
"errors"
"net/http"
"slices"
"strconv"
"strings"
"sync"
"testing"
"time"
"gitea.home.hrajfrisbee.cz/kacerr/egress-proxies-operator/internal/provider"
)
// putSeed uploads testName's seed to the fake's ISO storage, as Create
// leaves it.
func (fx *fixture) putSeed() {
fx.t.Helper()
upid, err := fx.fake.UploadISO(context.Background(), fakeNode, fakeISOStorage, seedISOName(testName), strings.NewReader("seed"), 4)
if err != nil {
fx.t.Fatalf("UploadISO error = %v", err)
}
pollTask(fx.t, fx.fake, upid)
}
func (fx *fixture) seedPresent() bool {
_, ok := fx.fake.iso(fakeISOStorage, seedISOName(testName))
return ok
}
// deleteFails runs Delete on fx.id(), checks the error's class and returns
// its text.
func (fx *fixture) deleteFails(class error) string {
fx.t.Helper()
err := fx.p.Delete(fx.ctx, fx.id())
assertProviderError(fx.t, err, "delete", class)
return err.Error()
}
func (fx *fixture) delete() {
fx.t.Helper()
if err := fx.p.Delete(fx.ctx, fx.id()); err != nil {
fx.t.Fatalf("Delete error = %v", err)
}
}
// seedDeleteNodes returns the node of every DeleteVolume call, read from
// the decorator's lines; the fixture's api must be observed.
func (fx *fixture) seedDeleteNodes() []string {
lines := fx.logs.matching(`"call"="DeleteVolume"`)
nodes := make([]string, 0, len(lines))
for _, l := range lines {
_, rest, _ := strings.Cut(l, `"node"="`)
node, _, _ := strings.Cut(rest, `"`)
nodes = append(nodes, node)
}
return nodes
}
// twoNodeFixture is an observed fixture at V(1) with nodes pv1-1 and pv1-2.
func twoNodeFixture(t *testing.T) *fixture {
t.Helper()
fx := newFixture(t, 1)
fx.p.api = observe(fx.fake, testProvider)
fx.p.cfg.Nodes = []string{fakeNode, "pv1-2"}
return fx
}
func TestDelete_malformedIDIsPermanent(t *testing.T) {
t.Parallel()
for _, id := range []string{"", "vms/150042", "vms/0150042/" + testName, "vm/150042/" + testName, "vms/abc/" + testName, "vms/150042/a/b", "vms/99/" + testName} {
t.Run(id, func(t *testing.T) {
t.Parallel()
fx := newFixture(t, 1)
assertProviderError(t, fx.p.Delete(fx.ctx, id), "delete", provider.ErrPermanent)
if calls := fx.fake.callLog(); len(calls) != 0 {
t.Errorf("calls = %v, want none", calls)
}
})
}
}
func TestDelete_twiceSucceeds(t *testing.T) {
t.Parallel()
fx := newFixture(t, 1)
fx.addOurVM(testVMID, testName, "running")
fx.putSeed()
fx.delete()
fx.delete()
if _, ok := fx.fake.vmSnapshot(testVMID); ok {
t.Error("VM still exists")
}
if fx.seedPresent() {
t.Error("seed still exists")
}
for method, want := range map[string]int{"Stop": 1, "Destroy": 1, "DeleteVolume": 2} {
if got := fx.fake.callCount(method); got != want {
t.Errorf("%s calls = %d, want %d", method, got, want)
}
}
}
// TestDelete_absentStillDeletesSeedOnEveryNode replaces the plan's
// TestDelete_absentIsNoOp: spec entry 11 deletes the seed even when the VM
// is gone, on every node, because the VM's node is no longer known.
func TestDelete_absentStillDeletesSeedOnEveryNode(t *testing.T) {
t.Parallel()
fx := twoNodeFixture(t)
fx.putSeed()
fx.delete()
if got := fx.seedDeleteNodes(); !slices.Equal(got, []string{fakeNode, "pv1-2"}) {
t.Errorf("seed deleted on %v, want every node in config order", got)
}
if fx.seedPresent() {
t.Error("seed still exists")
}
for _, method := range []string{"VMConfig", "Stop", "Destroy"} {
if n := fx.fake.callCount(method); n != 0 {
t.Errorf("%s calls = %d, want 0 for an absent VM", method, n)
}
}
if lines := fx.logs.matching(`"msg"="proxmox vm already gone"`); len(lines) != 1 {
t.Errorf("already-gone lines = %v, want one", lines)
}
}
func TestDelete_containerAtVMIDIsLeftAlone(t *testing.T) {
t.Parallel()
fx := twoNodeFixture(t)
fx.addOurVM(testVMID, testName, "running").kind = "lxc"
fx.delete()
if vm, ok := fx.fake.vmSnapshot(testVMID); !ok || vm.status != "running" {
t.Errorf("container = %+v, %v; want it untouched", vm, ok)
}
if got := fx.seedDeleteNodes(); len(got) != 2 {
t.Errorf("seed deleted on %v, want every node", got)
}
}
// TestDelete_foreignNameLeavesVMAlone replaces the plan's
// TestDelete_foreignNameIsNoOp: the VM is not touched, but our seed is
// still deleted, on every node since ours is gone.
func TestDelete_foreignNameLeavesVMAlone(t *testing.T) {
t.Parallel()
fx := twoNodeFixture(t)
fx.addOurVM(testVMID, "proxy-someoneelse00000", "running")
fx.putSeed()
fx.delete()
if vm, ok := fx.fake.vmSnapshot(testVMID); !ok || vm.status != "running" {
t.Errorf("foreign VM = %+v, %v; want it running and untouched", vm, ok)
}
for _, method := range []string{"Stop", "Destroy", "SetVMConfig"} {
if n := fx.fake.callCount(method); n != 0 {
t.Errorf("%s calls = %d, want 0", method, n)
}
}
if got := fx.seedDeleteNodes(); !slices.Equal(got, []string{fakeNode, "pv1-2"}) {
t.Errorf("seed deleted on %v, want every node", got)
}
lines := fx.logs.matching(`"msg"="proxmox vm at this vmid is not ours, left alone"`)
if len(lines) != 1 || !strings.Contains(lines[0], `"vmName"="proxy-someoneelse00000"`) {
t.Errorf("no-op lines = %v, want one naming the other VM", lines)
}
}
// TestDelete_unmarkedLeavesVMAlone replaces the plan's
// TestDelete_unmarkedIsNoOp, for the same reason.
func TestDelete_unmarkedLeavesVMAlone(t *testing.T) {
t.Parallel()
for name, desc := range map[string]string{
"no description": "",
"managed false": "proxy-operator-managed=false\nproxy-operator-uid=" + testUID,
} {
t.Run(name, func(t *testing.T) {
t.Parallel()
fx := twoNodeFixture(t)
vm := fx.fake.addVM(testVMID, testName, "stopped")
if desc != "" {
vm.config["description"] = desc
}
fx.delete()
if _, ok := fx.fake.vmSnapshot(testVMID); !ok {
t.Error("unmarked VM was destroyed")
}
if n := fx.fake.callCount("Destroy"); n != 0 {
t.Errorf("Destroy calls = %d, want 0", n)
}
if got := fx.seedDeleteNodes(); len(got) != 2 {
t.Errorf("seed deleted on %v, want every node", got)
}
if lines := fx.logs.matching(`"msg"="proxmox vm carries no proxy-operator mark, left alone"`); len(lines) != 1 {
t.Errorf("no-op lines = %v, want one", lines)
}
})
}
}
// TestDelete_runningVMStoppedFirst covers a paused VM too: the listing
// shows it as running (F14), and Stop ends it. The fake refuses to destroy
// a running VM, and each task takes two polls, so a Destroy that does not
// wait for the stop fails.
func TestDelete_runningVMStoppedFirst(t *testing.T) {
t.Parallel()
// pv1-1 took a stop on a paused VM as a task that ended OK with the VM
// stopped (Phase 10 captures), which is what the fake does for all three.
accepted := readCapture(t, "stop_paused_response")
done := decodeData[taskStatus](t, "task_status_stop_paused")
after := decodeData[vmStatus](t, "status_current_after_stop_paused")
if accepted.Status != http.StatusOK || done.Type != "qmstop" || done.ExitStatus != "OK" || after.Status != "stopped" || after.QMPStatus != "stopped" {
t.Fatalf("captures = %d, %+v, %+v; want the stop accepted, OK, and the VM stopped", accepted.Status, done, after)
}
for _, qmp := range []string{"running", "paused", "prelaunch"} {
t.Run(qmp, func(t *testing.T) {
t.Parallel()
fx := newFixture(t, 1)
fx.fake.taskPolls = 2
fx.addOurVM(testVMID, testName, "running").qmpstatus = qmp
fx.delete()
log := fx.fake.callLog()
stop, destroy := slices.Index(log, "Stop"), slices.Index(log, "Destroy")
if stop < 0 || destroy < stop {
t.Fatalf("calls = %v, want Stop before Destroy", log)
}
if n := callsBefore(log[stop:], "TaskStatus", "DeleteVolume"); n != fx.fake.taskPolls+1 {
t.Errorf("stop task polled %d times before the seed delete, want %d", n, fx.fake.taskPolls+1)
}
if _, ok := fx.fake.vmSnapshot(testVMID); ok {
t.Error("VM still exists")
}
})
}
}
func TestDelete_stoppedVMSkipsStop(t *testing.T) {
t.Parallel()
fx := newFixture(t, 1)
fx.addOurVM(testVMID, testName, "stopped")
fx.delete()
want := []string{"ClusterResources", "VMConfig", "DeleteVolume", "TaskStatus", "Destroy", "TaskStatus"}
if got := fx.fake.callLog(); !slices.Equal(got, want) {
t.Errorf("calls = %v, want %v", got, want)
}
}
func TestDelete_lockedIsTransient(t *testing.T) {
t.Parallel()
for _, lock := range []string{"backup", "migrate", "snapshot", "rollback", "create"} {
t.Run("resources lock "+lock, func(t *testing.T) {
t.Parallel()
fx := newFixture(t, 1)
fx.addOurVM(testVMID, testName, "running").lock = lock
fx.putSeed()
if msg := fx.deleteFails(provider.ErrTransient); !strings.Contains(msg, "locked ("+lock+")") {
t.Errorf("error = %s, want it to name the lock", msg)
}
if got := fx.fake.callLog(); !slices.Equal(got, []string{"UploadISO", "TaskStatus", "ClusterResources"}) {
t.Errorf("calls = %v, want only the listing after the seed setup", got)
}
if !fx.seedPresent() {
t.Error("seed deleted while the VM is locked")
}
})
}
t.Run("mid-clone nameless entry", func(t *testing.T) {
t.Parallel()
fx := newFixture(t, 1)
fx.fake.addTemplate(9001, "debian-13-template")
if _, err := fx.fake.Clone(fx.ctx, fakeNode, 9001, cloneParams{NewID: testVMID, Name: testName, Description: formatDescription(testUID, testCreatedAt)}); err != nil {
t.Fatalf("Clone error = %v", err)
}
fx.deleteFails(provider.ErrTransient)
if _, ok := fx.fake.vmSnapshot(testVMID); !ok {
t.Error("mid-clone VM is gone")
}
})
t.Run("lock taken between the resources and config reads", func(t *testing.T) {
t.Parallel()
fx := newFixture(t, 1)
vm := fx.addOurVM(testVMID, testName, "running")
mutateOn(fx.fake, "VMConfig", func() { vm.lock = "backup" })
fx.deleteFails(provider.ErrTransient)
if n := fx.fake.callCount("Stop"); n != 0 {
t.Errorf("Stop calls = %d, want 0", n)
}
})
}
func TestDelete_unknownStatusIsTransient(t *testing.T) {
t.Parallel()
fx := newFixture(t, 1)
fx.addOurVM(testVMID, testName, "running").status = "unknown"
if msg := fx.deleteFails(provider.ErrTransient); !strings.Contains(msg, fakeNode) {
t.Errorf("error = %s, want it to name the node", msg)
}
if n := fx.fake.callCount("VMConfig"); n != 0 {
t.Errorf("VMConfig calls = %d, want 0 while the node is unknown", n)
}
}
// TestDelete_hibernatedVMIsDestroyedIfOurs: on pv1-1 the destroy took a
// hibernated VM, suspend state volume included, and its task ended OK.
// Delete used to refuse it as Permanent on the belief that the destroy
// would. A hibernated VM that is not ours is still left alone.
func TestDelete_hibernatedVMIsDestroyedIfOurs(t *testing.T) {
t.Parallel()
t.Run("ours", func(t *testing.T) {
t.Parallel()
accepted := readCapture(t, "destroy_hibernated_response")
done := decodeData[taskStatus](t, "task_status_destroy_hibernated")
if accepted.Status != http.StatusOK || done.Type != "qmdestroy" || done.ExitStatus != "OK" {
t.Fatalf("captures = %d, %+v; want the destroy accepted and its task OK", accepted.Status, done)
}
fx := newFixture(t, 1)
fx.addOurVM(testVMID, testName, "stopped").lock = lockSuspended
fx.putSeed()
fx.delete()
want := []string{"ClusterResources", "VMConfig", "DeleteVolume", "TaskStatus", "Destroy", "TaskStatus"}
if got := fx.fake.callLog()[2:]; !slices.Equal(got, want) {
t.Errorf("calls = %v, want %v: no stop, since the VM is not running", got, want)
}
if _, ok := fx.fake.vmSnapshot(testVMID); ok {
t.Error("hibernated VM still exists")
}
if fx.seedPresent() {
t.Error("seed still exists")
}
})
t.Run("someone else's", func(t *testing.T) {
t.Parallel()
fx := newFixture(t, 1)
fx.addOurVM(testVMID, "proxy-someoneelse00000", "stopped").lock = lockSuspended
fx.delete()
if vm, ok := fx.fake.vmSnapshot(testVMID); !ok || vm.lock != lockSuspended {
t.Errorf("foreign hibernated VM = %+v, %v; want it untouched", vm, ok)
}
})
}
func TestDelete_removesSeedISO(t *testing.T) {
t.Parallel()
fx := twoNodeFixture(t)
fx.addOurVM(testVMID, testName, "stopped").node = "pv1-2"
fx.putSeed()
fx.delete()
if fx.seedPresent() {
t.Error("seed still exists")
}
if got := fx.seedDeleteNodes(); !slices.Equal(got, []string{"pv1-2"}) {
t.Errorf("seed deleted on %v, want only the VM's node pv1-2", got)
}
lines := fx.logs.matching(`"call"="DeleteVolume"`)
if len(lines) != 1 || !strings.Contains(lines[0], `"volid"="`+seedVolID(fakeISOStorage, testName)+`"`) || !strings.Contains(lines[0], `"storage"="`+fakeISOStorage+`"`) {
t.Errorf("DeleteVolume lines = %v, want the seed's volid on the ISO storage", lines)
}
}
// TestDelete_seedDeleteIsTaskAndIdempotent replaces the plan's
// TestDelete_toleratesMissingISO: Proxmox ends a delete of a missing volume
// with a successful imgdel task (F13), so nothing is matched, but the task
// is still waited on.
func TestDelete_seedDeleteIsTaskAndIdempotent(t *testing.T) {
t.Parallel()
missing := decodeData[taskStatus](t, "task_status_volume_missing")
if missing.Type != "imgdel" || missing.ExitStatus != "OK" {
t.Fatalf("capture task_status_volume_missing = %+v, want an imgdel task ending OK", missing)
}
for _, withSeed := range []bool{true, false} {
t.Run("seed present "+strconv.FormatBool(withSeed), func(t *testing.T) {
t.Parallel()
fx := newFixture(t, 1)
fx.addOurVM(testVMID, testName, "stopped")
if withSeed {
fx.putSeed()
}
fx.fake.taskPolls = 2
fx.delete()
log := fx.fake.callLog()
last := len(log) - 1
for log[last] != "DeleteVolume" {
last--
}
if n := callsBefore(log[last:], "TaskStatus", "Destroy"); n != fx.fake.taskPolls+1 {
t.Errorf("seed delete task polled %d times, want %d: %v", n, fx.fake.taskPolls+1, log)
}
if fx.seedPresent() {
t.Error("seed still exists")
}
})
}
}
// TestDelete_seedDeleteFailureOnVMNodeFails: the VM outlives a failed seed
// delete, so Get keeps finding it and the next Delete finishes the job with
// the same class of answer (review findings CR1, CR2, P1).
func TestDelete_seedDeleteFailureOnVMNodeFails(t *testing.T) {
t.Parallel()
tests := []struct {
name string
setup func(fx *fixture)
class error
}{
{name: "token lacks Datastore.Allocate", setup: func(fx *fixture) {
fx.fake.failWith("DeleteVolume", loadAPIError(t, "err_forbidden_volume_delete"))
}, class: provider.ErrPermanent},
{name: "pveproxy 596", setup: func(fx *fixture) { fx.fake.failWith("DeleteVolume", fakeErr(596, "Broken pipe")) }, class: provider.ErrTransient},
{name: "task failed", setup: func(fx *fixture) {
fx.fake.hookOn("TaskStatus", func() error {
if fx.fake.callCount("DeleteVolume") > 0 {
return fakeErr(500, "fake: task log unreadable")
}
return nil
})
}, class: provider.ErrTransient},
}
for _, tc := range tests {
t.Run(tc.name, func(t *testing.T) {
t.Parallel()
fx := newFixture(t, 1)
fx.addOurVM(testVMID, testName, "stopped")
tc.setup(fx)
fx.putSeed()
fx.deleteFails(tc.class)
if n := fx.fake.callCount("Destroy"); n != 0 {
t.Errorf("Destroy calls = %d, want 0 while the seed is still there", n)
}
if _, err := fx.p.Get(fx.ctx, fx.id()); errors.Is(err, provider.ErrNotFound) {
t.Fatalf("Get = %v, want the VM still found so Delete runs again", err)
}
fx.deleteFails(tc.class)
fx.fake.clearHooks()
fx.delete()
if _, ok := fx.fake.vmSnapshot(testVMID); ok {
t.Error("VM still exists after the retry")
}
if fx.seedPresent() {
t.Error("seed still exists after the retry")
}
})
}
}
func TestDelete_seedDeleteNodeFailureSkippedWhenNodeUnknown(t *testing.T) {
t.Parallel()
t.Run("one node refuses", func(t *testing.T) {
t.Parallel()
fx := twoNodeFixture(t)
fx.putSeed()
var calls int
fx.fake.hookOn("DeleteVolume", func() error {
if calls++; calls == 1 {
return fakeErr(500, "fake: storage 'proxy-seeds' is not available on node 'pv1-1'")
}
return nil
})
fx.delete()
if got := fx.seedDeleteNodes(); !slices.Equal(got, []string{fakeNode, "pv1-2"}) {
t.Errorf("seed deleted on %v, want both nodes tried", got)
}
lines := fx.logs.matching(`"msg"="proxmox seed delete failed on a node, skipped"`)
if len(lines) != 1 || !strings.Contains(lines[0], `"node"="pv1-1"`) || !strings.Contains(lines[0], "not available") {
t.Errorf("skip lines = %v, want one naming pv1-1 and the reason", lines)
}
if fx.seedPresent() {
t.Error("seed still exists after pv1-2's delete")
}
})
t.Run("a refused privilege fails the call", func(t *testing.T) {
t.Parallel()
fx := twoNodeFixture(t)
fx.putSeed()
fx.fake.failWith("DeleteVolume", loadAPIError(t, "err_forbidden_volume_delete"))
if msg := fx.deleteFails(provider.ErrPermanent); !strings.Contains(msg, "Datastore.Allocate") {
t.Errorf("error = %s, want the missing privilege named", msg)
}
if n := fx.fake.callCount("DeleteVolume"); n != 1 {
t.Errorf("DeleteVolume calls = %d, want 1: every node would refuse the same", n)
}
})
t.Run("ended ctx fails the call", func(t *testing.T) {
t.Parallel()
fx := twoNodeFixture(t)
ctx, cancel := context.WithCancel(fx.ctx)
defer cancel()
fx.fake.hookOn("DeleteVolume", func() error {
cancel()
return context.Canceled
})
err := fx.p.Delete(ctx, fx.id())
assertProviderError(t, err, "delete", provider.ErrTransient)
if n := fx.fake.callCount("DeleteVolume"); n != 1 {
t.Errorf("DeleteVolume calls = %d, want 1: the ended ctx stops the loop", n)
}
})
}
func TestDelete_destroyTaskFailureIsTransient(t *testing.T) {
t.Parallel()
fx := newFixture(t, 1)
fx.addOurVM(testVMID, testName, "stopped")
fx.putSeed()
fx.fake.destroyFailsWith = "fake: unable to remove disk"
if msg := fx.deleteFails(provider.ErrTransient); !strings.Contains(msg, "unable to remove disk") {
t.Errorf("error = %s, want the task's exitstatus", msg)
}
if _, ok := fx.fake.vmSnapshot(testVMID); !ok {
t.Error("VM gone after a failed destroy task")
}
if fx.seedPresent() {
t.Error("seed still exists: it goes before the destroy")
}
if lines := fx.logs.matching(`"msg"="proxmox task failed"`); len(lines) != 1 {
t.Errorf("task failed lines = %v, want one", lines)
}
}
// TestDelete_goneMidwayIsSuccess: the VM vanishing at any step counts as
// done, and F2's does-not-exist answer is what each step then sees.
func TestDelete_goneMidwayIsSuccess(t *testing.T) {
t.Parallel()
tests := []struct {
name string
status string
method string
wantDestroy int
wantNodes []string
}{
{name: "before the config read", status: "stopped", method: "VMConfig", wantNodes: []string{fakeNode, "pv1-2"}},
{name: "before the stop", status: "running", method: "Stop", wantDestroy: 1, wantNodes: []string{fakeNode}},
{name: "before the destroy", status: "stopped", method: "Destroy", wantDestroy: 1, wantNodes: []string{fakeNode}},
}
for _, tc := range tests {
t.Run(tc.name, func(t *testing.T) {
t.Parallel()
fx := twoNodeFixture(t)
fx.addOurVM(testVMID, testName, tc.status)
mutateOn(fx.fake, tc.method, func() { delete(fx.fake.vms, testVMID) })
fx.delete()
if n := fx.fake.callCount("Destroy"); n != tc.wantDestroy {
t.Errorf("Destroy calls = %d, want %d", n, tc.wantDestroy)
}
if got := fx.seedDeleteNodes(); !slices.Equal(got, tc.wantNodes) {
t.Errorf("seed deleted on %v, want %v", got, tc.wantNodes)
}
})
}
}
func TestDelete_resourcesFailureIsTransient(t *testing.T) {
t.Parallel()
fx := newFixture(t, 1)
fx.addOurVM(testVMID, testName, "running")
fx.fake.failWith("ClusterResources", fakeErr(595, "Connection refused"))
fx.deleteFails(provider.ErrTransient)
if n := fx.fake.callCount("DeleteVolume"); n != 0 {
t.Errorf("DeleteVolume calls = %d, want 0 when the VM's state is unknown", n)
}
}
// TestDelete_usesPurgeAndDestroyUnreferencedDisks runs Delete over the real
// client against Phase 0's destroy captures: the DELETE carries both flags
// and the seed delete sends the volid as one escaped segment.
func TestDelete_usesPurgeAndDestroyUnreferencedDisks(t *testing.T) {
t.Parallel()
const (
vmid = 199999
name = "proxy-reconaaaaaaaaaaa"
)
var (
mu sync.Mutex
seen []string
)
cfx := newClientFixture(t, func(w http.ResponseWriter, r *http.Request) {
path := strings.TrimPrefix(r.URL.EscapedPath(), "/api2/json/")
mu.Lock()
seen = append(seen, r.Method+" "+path+"?"+r.URL.RawQuery)
mu.Unlock()
capture := ""
switch {
case r.Method == http.MethodGet && path == "cluster/resources":
capture = "cluster_resources_during_destroy"
case r.Method == http.MethodGet && path == "nodes/pv1-1/qemu/199999/config":
capture = "config_during_destroy"
case r.Method == http.MethodDelete && path == "nodes/pv1-1/qemu/199999":
capture = "destroy_response"
case r.Method == http.MethodDelete && path == "nodes/pv1-1/storage/proxy-seeds/content/proxy-seeds:iso%2F"+name+"-seed.iso":
capture = "content_delete_response"
case r.Method == http.MethodGet && strings.Contains(path, ":imgdel:"):
capture = "task_status_volume_missing"
case r.Method == http.MethodGet && strings.Contains(path, ":qmdestroy:"):
capture = "task_status_stopped_ok"
default:
t.Errorf("unexpected request %s %s", r.Method, path)
http.Error(w, "unexpected", http.StatusNotImplemented)
return
}
serveJSON(w, readCapture(t, capture).Body)
})
cfg := testConfig()
cfg.URL, cfg.TokenFile, cfg.CAFile = cfx.cfg.URL, cfx.cfg.TokenFile, cfx.cfg.CAFile
p := newProvider(testProvider, cfg, cfx.api(t))
p.budgets = testBudgets()
if err := p.Delete(context.Background(), formatProviderID(vmid, name)); err != nil {
t.Fatalf("Delete error = %v", err)
}
mu.Lock()
defer mu.Unlock()
if !slices.Contains(seen, "DELETE nodes/pv1-1/qemu/199999?destroy-unreferenced-disks=1&purge=1") {
t.Errorf("requests = %v, want the destroy with purge=1 and destroy-unreferenced-disks=1", seen)
}
if !slices.ContainsFunc(seen, func(s string) bool { return strings.HasPrefix(s, "DELETE nodes/pv1-1/storage/proxy-seeds/content/") }) {
t.Errorf("requests = %v, want the seed delete", seen)
}
if slices.ContainsFunc(seen, func(s string) bool { return strings.Contains(s, "status/stop") }) {
t.Errorf("requests = %v, want no stop for the captured stopped VM", seen)
}
}
func TestDelete_concurrentSameNameSafe(t *testing.T) {
t.Parallel()
fx := newFixture(t, 1)
fx.addOurVM(testVMID, testName, "running")
release := fx.fake.blockOn("Stop")
errs := make([]error, 2)
first := make(chan struct{})
var wg sync.WaitGroup
wg.Go(func() {
defer close(first)
errs[0] = fx.p.Delete(fx.ctx, fx.id())
})
for fx.fake.callCount("Stop") == 0 {
select {
case <-first:
t.Fatalf("first Delete returned %v without stopping the VM", errs[0])
case <-time.After(time.Millisecond):
}
}
wg.Go(func() { errs[1] = fx.p.Delete(fx.ctx, fx.id()) })
close(release)
wg.Wait()
for i, err := range errs {
if err != nil {
t.Errorf("Delete %d error = %v", i, err)
}
}
for method, want := range map[string]int{"Stop": 1, "Destroy": 1} {
if got := fx.fake.callCount(method); got != want {
t.Errorf("%s calls = %d, want %d", method, got, want)
}
}
}
// TestDelete_waitsOnTheNameLock: Delete takes the per-name lock before any
// call, so it cannot interleave with a Create or another Delete of that
// name (spec entry 19).
func TestDelete_waitsOnTheNameLock(t *testing.T) {
t.Parallel()
fx := newFixture(t, 1)
fx.addOurVM(testVMID, testName, "running")
release, err := fx.p.lockName(fx.ctx, testName)
if err != nil {
t.Fatalf("lockName error = %v", err)
}
defer release()
ctx, cancel := context.WithTimeout(fx.ctx, 20*time.Millisecond)
defer cancel()
within(t, 5*time.Second, func() { err = fx.p.Delete(ctx, fx.id()) })
assertProviderError(t, err, "delete", provider.ErrTransient)
if !errors.Is(err, context.DeadlineExceeded) {
t.Errorf("error = %v, want it to wrap DeadlineExceeded", err)
}
if calls := fx.fake.callLog(); len(calls) != 0 {
t.Errorf("calls = %v, want none while the name is locked", calls)
}
}
func TestDelete_cancelledMidWaitReturnsPromptly(t *testing.T) {
t.Parallel()
fx := newFixture(t, 1)
fx.p.budgets = defaultBudgets()
fx.p.budgets.poll = time.Millisecond
fx.fake.taskPolls = 1 << 30
fx.addOurVM(testVMID, testName, "running")
ctx, cancel := context.WithCancel(fx.ctx)
defer cancel()
fx.fake.hookOn("TaskStatus", func() error {
cancel()
return nil
})
start := time.Now()
var err error
within(t, 10*time.Second, func() { err = fx.p.Delete(ctx, fx.id()) })
if elapsed := time.Since(start); elapsed > 5*time.Second {
t.Errorf("Delete took %v after cancellation", elapsed)
}
assertProviderError(t, err, "delete", provider.ErrTransient)
if !errors.Is(err, context.Canceled) {
t.Errorf("error = %v, want it to wrap context.Canceled", err)
}
}
func TestDelete_outcomeLine(t *testing.T) {
t.Parallel()
tests := []struct {
name string
setup func(fx *fixture)
want []string
never string
}{
{
name: "destroyed",
setup: func(fx *fixture) { fx.addOurVM(testVMID, testName, "stopped") },
want: []string{`"node"="` + fakeNode + `"`, `"destroyed"=true`},
},
{name: "already gone", setup: func(*fixture) {}, want: []string{`"destroyed"=false`}, never: `"node"=`},
}
for _, tc := range tests {
t.Run(tc.name, func(t *testing.T) {
t.Parallel()
fx := newFixture(t, 1)
tc.setup(fx)
fx.delete()
lines := fx.logs.matching(`"msg"="proxmox vm deleted"`)
if len(lines) != 1 {
t.Fatalf("outcome lines = %d, want 1:\n%s", len(lines), strings.Join(fx.logs.all(), "\n"))
}
for _, want := range append(tc.want, `"vmid"=150042`, `"name"="`+testName+`"`, `"provider"="`+testProvider+`"`) {
if !strings.Contains(lines[0], want) {
t.Errorf("outcome line lacks %s: %s", want, lines[0])
}
}
if tc.never != "" && strings.Contains(lines[0], tc.never) {
t.Errorf("outcome line has %s: %s", tc.never, lines[0])
}
})
}
}
func TestDelete_noOpLinesAreSilentAtV0(t *testing.T) {
t.Parallel()
tests := map[string]func(fx *fixture){
"absent": func(*fixture) {},
"foreign": func(fx *fixture) { fx.addOurVM(testVMID, "proxy-someoneelse00000", "running") },
"unmarked": func(fx *fixture) { fx.fake.addVM(testVMID, testName, "running") },
"ours": func(fx *fixture) { fx.addOurVM(testVMID, testName, "running") },
}
for name, setup := range tests {
t.Run(name, func(t *testing.T) {
t.Parallel()
fx := newFixture(t, 0)
fx.p.api = observe(fx.fake, testProvider)
setup(fx)
fx.delete()
if lines := fx.logs.all(); len(lines) != 0 {
t.Errorf("V(0) lines = %v, want none", lines)
}
})
}
}
+185
View File
@@ -0,0 +1,185 @@
package proxmox
import (
"crypto/tls"
"crypto/x509"
"encoding/json"
"errors"
"fmt"
"maps"
"net/http"
"regexp"
"slices"
"strings"
"gitea.home.hrajfrisbee.cz/kacerr/egress-proxies-operator/internal/provider"
)
// permanentf and transientf classify an error at the source, so classify
// keeps that class no matter what wraps it later.
func permanentf(format string, args ...any) error {
return fmt.Errorf("%w: %s", provider.ErrPermanent, fmt.Sprintf(format, args...))
}
func transientf(format string, args ...any) error {
return fmt.Errorf("%w: %s", provider.ErrTransient, fmt.Sprintf(format, args...))
}
// apiError is a non-2xx answer from the API. It carries the status and the
// message only, never headers or the request URL, so no credential can leak
// through it (spec entry 14).
type apiError struct {
Status int
Message string
// Errors is a 400's per-parameter map.
Errors map[string]string
}
func (e *apiError) Error() string {
msg := fmt.Sprintf("proxmox API %d: %s", e.Status, e.Message)
if len(e.Errors) == 0 {
return msg
}
fields := make([]string, 0, len(e.Errors))
for _, k := range slices.Sorted(maps.Keys(e.Errors)) {
fields = append(fields, k+": "+e.Errors[k])
}
return msg + " (" + strings.Join(fields, "; ") + ")"
}
// newAPIError builds an apiError from a response: the body's message with
// its trailing newline trimmed, or the status reason phrase when the body
// has none, as for a 401 (spec entry 13, fact F1).
func newAPIError(status int, reason string, body []byte) *apiError {
var b struct {
Message string `json:"message"`
Errors map[string]string `json:"errors"`
}
_ = json.Unmarshal(body, &b)
e := &apiError{Status: status, Message: strings.TrimSpace(b.Message)}
if e.Message == "" {
e.Message = strings.TrimSpace(reason)
}
if e.Message == "" {
e.Message = http.StatusText(status)
}
if len(b.Errors) > 0 {
e.Errors = make(map[string]string, len(b.Errors))
for k, v := range b.Errors {
e.Errors[k] = strings.TrimSpace(v)
}
}
return e
}
// The 500 messages the provider acts on, each backed by a captured response
// under testdata/.
var (
vmConfigMissing = regexp.MustCompile(`^Configuration file '[^']*' does not exist$`)
)
const (
msgAlreadyExists = "config file already exists"
msgAgentNotRunning = "QEMU guest agent is not running"
msgAgentNotConfigured = "No QEMU guest agent configured"
msgNoSuchStorage = "No such storage."
)
// server500 returns the message of err when it is, or wraps, an apiError
// with status 500.
func server500(err error) (string, bool) {
var ae *apiError
if !errors.As(err, &ae) || ae.Status != http.StatusInternalServerError {
return "", false
}
return ae.Message, true
}
// isNotExist reports the missing-VM answer: the VM's config file is gone.
func isNotExist(err error) bool {
msg, ok := server500(err)
return ok && vmConfigMissing.MatchString(msg)
}
// isNoSuchStorage reports a storage the cluster does not know: a 400 whose
// storage field error is exactly msgNoSuchStorage. Other storage field
// errors, such as a clone onto a storage without image support, are not.
func isNoSuchStorage(err error) bool {
var ae *apiError
return errors.As(err, &ae) && ae.Status == http.StatusBadRequest && ae.Errors["storage"] == msgNoSuchStorage
}
// isAlreadyExists reports a clone onto a VMID that is taken.
func isAlreadyExists(err error) bool {
msg, ok := server500(err)
return ok && strings.HasSuffix(msg, msgAlreadyExists)
}
// isAgentNotRunning reports a VM with the agent enabled whose agent does not
// answer yet.
func isAgentNotRunning(err error) bool {
msg, ok := server500(err)
return ok && msg == msgAgentNotRunning
}
// isAgentNotConfigured reports a VM whose config no longer enables the agent.
func isAgentNotConfigured(err error) bool {
msg, ok := server500(err)
return ok && msg == msgAgentNotConfigured
}
// classify maps any error reaching the provider's boundary to one of the
// four contract sentinels, per spec entry 13. op is the contract operation:
// a vanished VM inside Create is Transient, because the contract forbids
// NotFound from Create.
func classify(op string, err error) error {
if err == nil {
return provider.ErrTransient
}
for _, class := range []error{provider.ErrNotFound, provider.ErrQuotaExceeded, provider.ErrPermanent, provider.ErrTransient} {
if errors.Is(err, class) {
return class
}
}
var ae *apiError
if errors.As(err, &ae) {
return classifyAPI(op, err, ae)
}
if isTLSVerification(err) {
return provider.ErrPermanent
}
// Everything else retries: context cancellation and deadlines,
// net.OpError, io.EOF, and anything unrecognised.
return provider.ErrTransient
}
func classifyAPI(op string, err error, ae *apiError) error {
switch {
case ae.Status >= 400 && ae.Status < 500:
// 401 and 403 are credentials or privileges, 400 a request Proxmox
// will never accept: all fixed by an admin, not by retrying.
return provider.ErrPermanent
case isNotExist(err):
if op == "create" {
return provider.ErrTransient
}
return provider.ErrNotFound
default:
// Other 500s (a lock, a taken VMID escaping the allocator, agent
// states escaping Get), pveproxy's 595 and 596, and any other status.
return provider.ErrTransient
}
}
// isTLSVerification reports a certificate the client refused: the URL does
// not match the certificate, or the CA is wrong. An admin fixes the URL or
// caFile; retrying cannot.
func isTLSVerification(err error) bool {
var (
verr *tls.CertificateVerificationError
unknown x509.UnknownAuthorityError
hostname x509.HostnameError
invalid x509.CertificateInvalidError
)
return errors.As(err, &verr) || errors.As(err, &unknown) || errors.As(err, &hostname) || errors.As(err, &invalid)
}
+203
View File
@@ -0,0 +1,203 @@
package proxmox
import (
"context"
"crypto/tls"
"crypto/x509"
"errors"
"fmt"
"io"
"net"
"net/url"
"strings"
"testing"
"gitea.home.hrajfrisbee.cz/kacerr/egress-proxies-operator/internal/provider"
)
func TestClassify_table(t *testing.T) {
t.Parallel()
fixture := func(name string) error { return loadAPIError(t, name) }
tests := []struct {
name string
op string
err error
want error
}{
{name: "nil is transient", op: "get", err: nil, want: provider.ErrTransient},
{name: "missing VM config on get", op: "get", err: fixture("err_vm_missing"), want: provider.ErrNotFound},
{name: "missing VM status on get", op: "get", err: fixture("err_vm_missing_status"), want: provider.ErrNotFound},
{name: "missing VM on delete", op: "delete", err: fixture("err_destroy_missing"), want: provider.ErrNotFound},
{name: "missing VM on list", op: "list", err: fixture("err_vm_missing"), want: provider.ErrNotFound},
{name: "missing VM inside create is transient", op: "create", err: fixture("err_vm_missing"), want: provider.ErrTransient},
{name: "wrapped missing VM keeps its class", op: "get", err: fmt.Errorf("reading config: %w", fixture("err_vm_missing")), want: provider.ErrNotFound},
{name: "locked by a clone", op: "create", err: fixture("err_config_while_clone_locked"), want: provider.ErrTransient},
{name: "VMID taken escaping the allocator", op: "create", err: fixture("err_newid_in_use"), want: provider.ErrTransient},
{name: "agent not running escaping Get", op: "get", err: fixture("err_agent_not_running"), want: provider.ErrTransient},
{name: "agent not configured escaping Get", op: "get", err: fixture("err_agent_vm_stopped"), want: provider.ErrTransient},
{name: "VM not running escaping Get", op: "get", err: fixture("err_agent_vm_stopped_agent_enabled"), want: provider.ErrTransient},
{name: "wrong secret", op: "get", err: fixture("err_unauthorized"), want: provider.ErrPermanent},
{name: "unknown token id", op: "list", err: fixture("err_unauthorized_unknown_token"), want: provider.ErrPermanent},
{name: "no credentials", op: "list", err: fixture("err_no_credentials"), want: provider.ErrPermanent},
{name: "missing privilege", op: "get", err: fixture("err_forbidden"), want: provider.ErrPermanent},
{name: "missing Datastore.Allocate", op: "delete", err: fixture("err_forbidden_volume_delete"), want: provider.ErrPermanent},
{name: "parameter verification", op: "create", err: fixture("err_bad_param"), want: provider.ErrPermanent},
{name: "invalid tag", op: "create", err: fixture("err_tag_invalid"), want: provider.ErrPermanent},
{name: "upload without .iso", op: "create", err: fixture("err_upload_bad_name"), want: provider.ErrPermanent},
{name: "clone onto a storage without images", op: "create", err: fixture("err_clone_bad_storage"), want: provider.ErrPermanent},
{name: "any other 4xx", op: "get", err: &apiError{Status: 404, Message: "Not Found"}, want: provider.ErrPermanent},
{name: "other 500", op: "get", err: &apiError{Status: 500, Message: "got timeout"}, want: provider.ErrTransient},
{name: "storage does not exist is not a missing VM", op: "delete", err: &apiError{Status: 500, Message: "storage 'x' does not exist"}, want: provider.ErrTransient},
{name: "502", op: "get", err: &apiError{Status: 502, Message: "Bad Gateway"}, want: provider.ErrTransient},
{name: "pveproxy 595", op: "get", err: &apiError{Status: 595, Message: "No route to host"}, want: provider.ErrTransient},
{name: "pveproxy 596", op: "list", err: &apiError{Status: 596, Message: "Connection timed out"}, want: provider.ErrTransient},
{name: "unknown authority", op: "get", err: x509.UnknownAuthorityError{}, want: provider.ErrPermanent},
{name: "hostname mismatch", op: "create", err: x509.HostnameError{Certificate: &x509.Certificate{}, Host: "192.168.2.8"}, want: provider.ErrPermanent},
{name: "expired certificate", op: "list", err: x509.CertificateInvalidError{Reason: x509.Expired}, want: provider.ErrPermanent},
{name: "tls verification wrapped in url.Error", op: "get", err: &url.Error{Op: "Get", URL: "https://pve", Err: &tls.CertificateVerificationError{Err: x509.UnknownAuthorityError{}}}, want: provider.ErrPermanent},
{name: "context canceled", op: "get", err: context.Canceled, want: provider.ErrTransient},
{name: "context deadline", op: "get", err: context.DeadlineExceeded, want: provider.ErrTransient},
{name: "net.OpError", op: "get", err: &net.OpError{Op: "dial", Net: "tcp", Err: errors.New("connection refused")}, want: provider.ErrTransient},
{name: "io.EOF", op: "list", err: io.EOF, want: provider.ErrTransient},
{name: "io.ErrUnexpectedEOF", op: "list", err: io.ErrUnexpectedEOF, want: provider.ErrTransient},
{name: "anything else is transient", op: "get", err: errors.New("anything"), want: provider.ErrTransient},
{name: "pre-classified NotFound stays", op: "create", err: fmt.Errorf("x: %w", provider.ErrNotFound), want: provider.ErrNotFound},
{name: "pre-classified Quota stays", op: "create", err: fmt.Errorf("x: %w", provider.ErrQuotaExceeded), want: provider.ErrQuotaExceeded},
{name: "pre-classified Permanent stays", op: "get", err: permanentf("bad %s", "thing"), want: provider.ErrPermanent},
{name: "pre-classified Transient beats a 4xx underneath", op: "get", err: fmt.Errorf("%w: %w", provider.ErrTransient, fixture("err_forbidden")), want: provider.ErrTransient},
}
for _, tc := range tests {
t.Run(tc.name, func(t *testing.T) {
t.Parallel()
if got := classify(tc.op, tc.err); !errors.Is(got, tc.want) {
t.Errorf("classify(%q, %v) = %v, want %v", tc.op, tc.err, got, tc.want)
}
})
}
}
// TestAPIError_matchers runs every matcher over every captured error, so a
// matcher that also fires on another message fails here.
func TestAPIError_matchers(t *testing.T) {
t.Parallel()
matchers := map[string]func(error) bool{
"isNotExist": isNotExist,
"isLocked": isLocked,
"isAlreadyExists": isAlreadyExists,
"isAgentNotRunning": isAgentNotRunning,
"isAgentNotConfigured": isAgentNotConfigured,
"isNoSuchStorage": isNoSuchStorage,
"isVMNotRunning": isVMNotRunning,
}
tests := []struct {
capture string
want string // the one matcher that fires, or "" for none
}{
{"err_vm_missing", "isNotExist"},
{"err_vm_missing_status", "isNotExist"},
{"err_destroy_missing", "isNotExist"},
{"err_config_while_clone_locked", "isLocked"},
{"err_newid_in_use", "isAlreadyExists"},
{"storage_status_missing", "isNoSuchStorage"},
{"err_agent_not_running", "isAgentNotRunning"},
{"err_agent_vm_stopped", "isAgentNotConfigured"},
{"err_agent_vm_stopped_agent_enabled", "isVMNotRunning"},
{"err_unauthorized", ""},
{"err_forbidden", ""},
{"err_bad_param", ""},
{"err_clone_bad_storage", ""},
}
for _, tc := range tests {
t.Run(tc.capture, func(t *testing.T) {
t.Parallel()
err := fmt.Errorf("wrapped: %w", loadAPIError(t, tc.capture))
for name, match := range matchers {
if got, want := match(err), name == tc.want; got != want {
t.Errorf("%s(%v) = %v, want %v", name, err, got, want)
}
}
})
}
if isNotExist(errors.New("Configuration file 'x' does not exist")) {
t.Error("isNotExist matched an error that is not an apiError")
}
}
func TestNewAPIError_fromCaptures(t *testing.T) {
t.Parallel()
tests := []struct {
capture string
wantStatus int
wantMessage string
wantErrors map[string]string
}{
{capture: "err_vm_missing", wantStatus: 500, wantMessage: "Configuration file 'nodes/pv1-1/qemu-server/999999.conf' does not exist"},
{capture: "err_unauthorized", wantStatus: 401, wantMessage: "Authentication failed!"},
{capture: "err_no_credentials", wantStatus: 401, wantMessage: "No ticket"},
{capture: "err_forbidden", wantStatus: 403, wantMessage: "Permission check failed (/nodes/pv1-1, Sys.Syslog)"},
{capture: "err_bad_param", wantStatus: 400, wantMessage: "Parameter verification failed.", wantErrors: map[string]string{"newid": "type check ('integer') failed - got 'abc'"}},
}
for _, tc := range tests {
t.Run(tc.capture, func(t *testing.T) {
t.Parallel()
got := loadAPIError(t, tc.capture)
if got.Status != tc.wantStatus || got.Message != tc.wantMessage {
t.Errorf("newAPIError = {%d %q}, want {%d %q}", got.Status, got.Message, tc.wantStatus, tc.wantMessage)
}
if len(got.Errors) != len(tc.wantErrors) {
t.Fatalf("Errors = %v, want %v", got.Errors, tc.wantErrors)
}
for k, v := range tc.wantErrors {
if got.Errors[k] != v {
t.Errorf("Errors[%q] = %q, want %q", k, got.Errors[k], v)
}
}
})
}
}
func TestNewAPIError_fallbacks(t *testing.T) {
t.Parallel()
if got := newAPIError(503, "", []byte("<html>")); got.Message != "Service Unavailable" {
t.Errorf("no reason and no JSON body: Message = %q, want the status text", got.Message)
}
if got := newAPIError(500, "reason text", []byte(`{"data":null}`)); got.Message != "reason text" {
t.Errorf("body without message: Message = %q, want the reason phrase", got.Message)
}
}
func TestAPIError_errorString(t *testing.T) {
t.Parallel()
got := loadAPIError(t, "err_bad_param").Error()
want := "proxmox API 400: Parameter verification failed. (newid: type check ('integer') failed - got 'abc')"
if got != want {
t.Errorf("Error() = %q, want %q", got, want)
}
e := &apiError{Status: 400, Message: "m", Errors: map[string]string{"b": "2", "a": "1"}}
if got := e.Error(); !strings.HasSuffix(got, "(a: 1; b: 2)") {
t.Errorf("Error() = %q, want field errors sorted by key", got)
}
}
func TestPermanentf_wrapsClassWithMessage(t *testing.T) {
t.Parallel()
perm := permanentf("node %q missing", "x")
if !errors.Is(perm, provider.ErrPermanent) || perm.Error() != `provider: permanent error: node "x" missing` {
t.Errorf("permanentf() = %q, want ErrPermanent with the message", perm)
}
}
func TestTransientf_wrapsClassWithMessage(t *testing.T) {
t.Parallel()
trans := transientf("retry %d", 3)
if !errors.Is(trans, provider.ErrTransient) || trans.Error() != "provider: transient error: retry 3" {
t.Errorf("transientf() = %q, want ErrTransient with the message", trans)
}
}
+943
View File
@@ -0,0 +1,943 @@
package proxmox
import (
"context"
"errors"
"fmt"
"io"
"maps"
"regexp"
"slices"
"strconv"
"strings"
"sync"
"testing"
)
// fakeAPI is the in-memory cluster of spec entry 17: VMs, tasks and ISOs,
// with lock simulation (a clone leaves a nameless locked entry until its
// task completes), task polls, per-method errors and blocking, and a call
// log. Error texts Proxmox was seen to send are reproduced exactly and
// checked against the captures by TestFakeAPI_errorsMatchCaptures; texts
// recon never produced start with "fake:".
type fakeAPI struct {
mu sync.Mutex
vms map[int]*fakeVM
tasks map[string]*fakeTask
isos map[string]map[string][]byte // storage, then file name
nodes map[string]nodeStatus
storages map[string]storageStatus
version versionInfo
taskSeq int
// hooks returns the error a method fails with before touching state.
hooks map[string]func() error
// block makes a method wait on its channel (or ctx) before running.
block map[string]chan struct{}
calls map[string]int
log []string
// taskPolls is how many TaskStatus polls answer "running" before a
// task stops.
taskPolls int
// cloneFailsWith, startFailsWith and destroyFailsWith, when set, are
// the exitstatus of every clone, start or destroy task. A failed clone
// removes its VM; a failed destroy keeps it.
cloneFailsWith, startFailsWith, destroyFailsWith string
// agentNotRunning makes the agent of every running VM with agent=1
// answer "QEMU guest agent is not running".
agentNotRunning bool
// cloneLeavesLock keeps lock=clone after a clone task ends, as an
// interrupted clone would (F15).
cloneLeavesLock bool
}
type fakeVM struct {
node, name, status, qmpstatus, lock string
// kind is the resource type; empty is "qemu". Any other kind, such as
// "lxc", is listed but invisible to the qemu endpoints.
kind string
template bool
config map[string]string
ifaces []agentInterface
}
type fakeTask struct {
node string
polls int
exit string
onDone func()
kind string
id string
done bool
}
const (
fakeNode = "pv1-1"
fakeISOStorage = "proxy-seeds"
)
var _ api = (*fakeAPI)(nil)
func newFakeAPI() *fakeAPI {
var node nodeStatus
node.PVEVersion = "pve-manager/9.2.20/49318c671b82f31e"
node.CurrentKernel.Release = "7.0.14-17-pve"
return &fakeAPI{
vms: make(map[int]*fakeVM),
tasks: make(map[string]*fakeTask),
isos: make(map[string]map[string][]byte),
nodes: map[string]nodeStatus{fakeNode: node},
storages: map[string]storageStatus{fakeISOStorage: {Type: "dir", Content: "iso", Active: 1, Enabled: 1, Total: 100 << 30, Used: 10 << 30, Avail: 90 << 30}},
version: versionInfo{Version: "9.2.20", Release: "9.2", RepoID: "49318c671b82f31e"},
hooks: make(map[string]func() error),
block: make(map[string]chan struct{}),
calls: make(map[string]int),
}
}
// addVM places a stopped or running VM cloned from a template like 9001 on
// fakeNode, without a description; tests add the mark they need.
func (f *fakeAPI) addVM(vmid int, name, status string) *fakeVM {
f.mu.Lock()
defer f.mu.Unlock()
vm := &fakeVM{
node: fakeNode,
name: name,
status: status,
qmpstatus: status,
config: map[string]string{
"agent": "1",
"boot": "order=scsi0",
"cores": "2",
"memory": "2048",
"net0": "virtio=BC:24:11:48:11:46,bridge=vmbr0",
"scsi0": fmt.Sprintf("local-lvm:vm-%d-disk-1,size=3G", vmid),
},
}
f.vms[vmid] = vm
return vm
}
// addTemplate places a template shaped like 9001 (F19) on fakeNode.
func (f *fakeAPI) addTemplate(vmid int, name string) *fakeVM {
vm := f.addVM(vmid, name, "stopped")
f.mu.Lock()
defer f.mu.Unlock()
vm.template = true
vm.config["scsi0"] = fmt.Sprintf("local-lvm:base-%d-disk-1,size=3G", vmid)
return vm
}
func (f *fakeAPI) failWith(method string, err error) {
f.hookOn(method, func() error { return err })
}
// hookOn runs fn at the start of every call to method, without f.mu held;
// a non-nil result fails the call. A hook that changes state takes f.mu.
func (f *fakeAPI) hookOn(method string, fn func() error) {
f.mu.Lock()
defer f.mu.Unlock()
f.hooks[method] = fn
}
// clearHooks drops every hook, so the faults they injected stop.
func (f *fakeAPI) clearHooks() {
f.mu.Lock()
defer f.mu.Unlock()
clear(f.hooks)
}
func (f *fakeAPI) blockOn(method string) chan struct{} {
f.mu.Lock()
defer f.mu.Unlock()
ch := make(chan struct{})
f.block[method] = ch
return ch
}
func (f *fakeAPI) callCount(method string) int {
f.mu.Lock()
defer f.mu.Unlock()
return f.calls[method]
}
func (f *fakeAPI) callLog() []string {
f.mu.Lock()
defer f.mu.Unlock()
return slices.Clone(f.log)
}
// iso returns an uploaded file's content.
func (f *fakeAPI) iso(storage, filename string) ([]byte, bool) {
f.mu.Lock()
defer f.mu.Unlock()
b, ok := f.isos[storage][filename]
return b, ok
}
// vmSnapshot returns a copy of the VM at vmid, for assertions.
func (f *fakeAPI) vmSnapshot(vmid int) (fakeVM, bool) {
f.mu.Lock()
defer f.mu.Unlock()
vm, ok := f.vms[vmid]
if !ok {
return fakeVM{}, false
}
c := *vm
c.config = maps.Clone(vm.config)
c.ifaces = slices.Clone(vm.ifaces)
return c, true
}
func fakeErr(status int, msg string) *apiError {
return &apiError{Status: status, Message: msg}
}
// The 500s recon captured (F2, F4, F5, F7).
func errVMMissing(node string, vmid int) *apiError {
return fakeErr(500, fmt.Sprintf("Configuration file 'nodes/%s/qemu-server/%d.conf' does not exist", node, vmid))
}
// errNoSuchStorage is the answer for a storage the cluster does not know
// (Phase 10 capture storage_status_missing).
func errNoSuchStorage() *apiError {
return newAPIError(400, "", []byte(`{"message":"Parameter verification failed.\n","data":null,"errors":{"storage":"No such storage."}}`))
}
func errLocked(lock string) *apiError { return fakeErr(500, "VM is locked ("+lock+")") }
// isLocked and isVMNotRunning recognise two answers the provider leaves to
// classify's Transient default. They live here only to hold the fake's
// wording to the captures. The stopped-VM message is matched whole because
// " is not running" is also a suffix of the agent-not-running answer.
var vmNotRunning = regexp.MustCompile(`^VM [0-9]+ is not running$`)
func isLocked(err error) bool {
msg, ok := server500(err)
return ok && strings.HasPrefix(msg, "VM is locked")
}
func isVMNotRunning(err error) bool {
msg, ok := server500(err)
return ok && vmNotRunning.MatchString(msg)
}
func errNewIDInUse(vmid int) *apiError {
return fakeErr(500, fmt.Sprintf("unable to create VM %d: config file already exists", vmid))
}
func errVMNotRunning(vmid int) *apiError {
return fakeErr(500, fmt.Sprintf("VM %d is not running", vmid))
}
// enter records the call, honours a block and runs the hook. It is called
// without f.mu held; state access happens afterwards under the lock.
func (f *fakeAPI) enter(ctx context.Context, method string) error {
f.mu.Lock()
f.calls[method]++
f.log = append(f.log, method)
block := f.block[method]
hook := f.hooks[method]
f.mu.Unlock()
if block != nil {
select {
case <-block:
case <-ctx.Done():
return ctx.Err()
}
}
if hook != nil {
return hook()
}
return nil
}
// vm looks a VM up the way Proxmox does, by node and VMID: a VM on another
// node has no config file on this one.
func (f *fakeAPI) vm(node string, vmid int) (*fakeVM, error) {
vm, ok := f.vms[vmid]
if !ok || vm.node != node || vm.kind != "" {
return nil, errVMMissing(node, vmid)
}
return vm, nil
}
// newTask registers a task that stops after f.taskPolls running answers,
// then runs onDone under f.mu. The UPID has the shape parseUPID reads.
func (f *fakeAPI) newTask(node, kind, id, exit string, onDone func()) string {
f.taskSeq++
upid := fmt.Sprintf("UPID:%s:%08X:%08X:%08X:%s:%s:proxy-operator@pve!operator:", node, 0x100000+f.taskSeq, 0x2800000+f.taskSeq, 0x6AB37800+f.taskSeq, kind, id)
f.tasks[upid] = &fakeTask{node: node, exit: exit, onDone: onDone, kind: kind, id: id}
return upid
}
func (f *fakeAPI) Version(ctx context.Context) (versionInfo, error) {
if err := f.enter(ctx, "Version"); err != nil {
return versionInfo{}, err
}
f.mu.Lock()
defer f.mu.Unlock()
return f.version, nil
}
func (f *fakeAPI) NodeStatus(ctx context.Context, node string) (nodeStatus, error) {
if err := f.enter(ctx, "NodeStatus"); err != nil {
return nodeStatus{}, err
}
f.mu.Lock()
defer f.mu.Unlock()
s, ok := f.nodes[node]
if !ok {
return nodeStatus{}, fakeErr(500, "fake: no such node '"+node+"'")
}
return s, nil
}
func (f *fakeAPI) StorageStatus(ctx context.Context, node, storage string) (storageStatus, error) {
if err := f.enter(ctx, "StorageStatus"); err != nil {
return storageStatus{}, err
}
f.mu.Lock()
defer f.mu.Unlock()
s, ok := f.storages[storage]
if !ok {
return storageStatus{}, errNoSuchStorage()
}
return s, nil
}
// ClusterResources lists VMs by VMID. A VM being cloned is listed as F8
// saw it: no name, no tags, lock clone and status unknown.
func (f *fakeAPI) ClusterResources(ctx context.Context) ([]vmResource, error) {
if err := f.enter(ctx, "ClusterResources"); err != nil {
return nil, err
}
f.mu.Lock()
defer f.mu.Unlock()
out := make([]vmResource, 0, len(f.vms))
for _, vmid := range slices.Sorted(maps.Keys(f.vms)) {
vm := f.vms[vmid]
r := vmResource{VMID: vmid, Name: vm.name, Node: vm.node, Type: "qemu", Status: vm.status, Lock: vm.lock, Tags: vm.config["tags"]}
if vm.kind != "" {
r.Type = vm.kind
}
if vm.template {
r.Template = 1
}
if vm.lock == "clone" {
r.Name, r.Tags, r.Status = "", "", "unknown"
}
out = append(out, r)
}
return out, nil
}
func (f *fakeAPI) VMConfig(ctx context.Context, node string, vmid int) (vmConfig, error) {
if err := f.enter(ctx, "VMConfig"); err != nil {
return nil, err
}
f.mu.Lock()
defer f.mu.Unlock()
vm, err := f.vm(node, vmid)
if err != nil {
return nil, err
}
cfg := vmConfig(maps.Clone(vm.config))
if vm.name != "" {
cfg["name"] = vm.name
}
if vm.lock != "" {
cfg["lock"] = vm.lock
}
if vm.template {
cfg["template"] = "1"
}
return cfg, nil
}
// SetVMConfig is refused while the VM is locked (F7). "delete" removes a
// comma-separated list of keys; "name" renames.
func (f *fakeAPI) SetVMConfig(ctx context.Context, node string, vmid int, params map[string]string) error {
if err := f.enter(ctx, "SetVMConfig"); err != nil {
return err
}
f.mu.Lock()
defer f.mu.Unlock()
vm, err := f.vm(node, vmid)
if err != nil {
return err
}
if vm.lock != "" {
return errLocked(vm.lock)
}
for k, v := range params {
switch k {
case "delete":
for key := range strings.SplitSeq(v, ",") {
delete(vm.config, key)
}
case "name":
vm.name = v
default:
vm.config[k] = v
}
}
return nil
}
// Clone creates the VM at once, locked and nameless in the listing, and a
// task that clears the lock, or removes the VM when cloneFailsWith is set.
// A taken newid fails synchronously (F4).
func (f *fakeAPI) Clone(ctx context.Context, node string, vmid int, p cloneParams) (string, error) {
if err := f.enter(ctx, "Clone"); err != nil {
return "", err
}
f.mu.Lock()
defer f.mu.Unlock()
src, err := f.vm(node, vmid)
if err != nil {
return "", err
}
if _, taken := f.vms[p.NewID]; taken {
return "", errNewIDInUse(p.NewID)
}
target := node
if p.Target != "" {
target = p.Target
}
cfg := maps.Clone(src.config)
for k, v := range cfg {
cfg[k] = strings.ReplaceAll(v, fmt.Sprintf("base-%d-", vmid), fmt.Sprintf("vm-%d-", p.NewID))
}
if p.Description != "" {
cfg["description"] = p.Description
}
vm := &fakeVM{node: target, name: p.Name, status: "stopped", qmpstatus: "stopped", lock: "clone", config: cfg}
f.vms[p.NewID] = vm
exit := "OK"
if f.cloneFailsWith != "" {
exit = f.cloneFailsWith
}
leaveLock := f.cloneLeavesLock
return f.newTask(node, "qmclone", strconv.Itoa(vmid), exit, func() {
switch {
case exit != "OK":
if f.vms[p.NewID] == vm {
delete(f.vms, p.NewID)
}
case !leaveLock:
vm.lock = ""
}
}), nil
}
// Resize runs as a task (F20): the same size is a no-op, a smaller one
// fails inside the task with the captured exitstatus.
func (f *fakeAPI) Resize(ctx context.Context, node string, vmid int, disk, size string) (string, error) {
if err := f.enter(ctx, "Resize"); err != nil {
return "", err
}
f.mu.Lock()
defer f.mu.Unlock()
vm, err := f.vm(node, vmid)
if err != nil {
return "", err
}
if vm.lock != "" {
return "", errLocked(vm.lock)
}
cur, ok := vm.config[disk]
if !ok {
return "", fakeErr(500, "fake: disk '"+disk+"' does not exist")
}
curBytes, ok := fakeParseSize(diskSizeOption(cur))
if !ok {
return "", fakeErr(500, "fake: disk '"+disk+"' has no size")
}
want, ok := fakeParseSize(strings.TrimPrefix(size, "+"))
if !ok {
return "", fakeErr(400, "fake: bad size '"+size+"'")
}
if strings.HasPrefix(size, "+") {
want += curBytes
}
exit := "OK"
if want < curBytes {
exit = "shrinking disks is not supported"
}
return f.newTask(node, "resize", strconv.Itoa(vmid), exit, func() {
if exit == "OK" {
vm.config[disk] = strings.Replace(cur, "size="+diskSizeOption(cur), "size="+fakeFormatSize(want), 1)
}
}), nil
}
// Start is accepted even while a clone holds the lock (F3).
func (f *fakeAPI) Start(ctx context.Context, node string, vmid int) (string, error) {
if err := f.enter(ctx, "Start"); err != nil {
return "", err
}
f.mu.Lock()
defer f.mu.Unlock()
vm, err := f.vm(node, vmid)
if err != nil {
return "", err
}
exit := "OK"
if f.startFailsWith != "" {
exit = f.startFailsWith
}
return f.newTask(node, "qmstart", strconv.Itoa(vmid), exit, func() {
if exit == "OK" {
vm.status, vm.qmpstatus = "running", "running"
}
}), nil
}
func (f *fakeAPI) Stop(ctx context.Context, node string, vmid int) (string, error) {
if err := f.enter(ctx, "Stop"); err != nil {
return "", err
}
f.mu.Lock()
defer f.mu.Unlock()
vm, err := f.vm(node, vmid)
if err != nil {
return "", err
}
return f.newTask(node, "qmstop", strconv.Itoa(vmid), "OK", func() {
vm.status, vm.qmpstatus = "stopped", "stopped"
}), nil
}
func (f *fakeAPI) VMStatus(ctx context.Context, node string, vmid int) (vmStatus, error) {
if err := f.enter(ctx, "VMStatus"); err != nil {
return vmStatus{}, err
}
f.mu.Lock()
defer f.mu.Unlock()
vm, err := f.vm(node, vmid)
if err != nil {
return vmStatus{}, err
}
return vmStatus{Status: vm.status, QMPStatus: vm.qmpstatus, Name: vm.name, Lock: vm.lock}, nil
}
// Destroy refuses a locked or running VM and removes it when its task
// ends.
func (f *fakeAPI) Destroy(ctx context.Context, node string, vmid int) (string, error) {
if err := f.enter(ctx, "Destroy"); err != nil {
return "", err
}
f.mu.Lock()
defer f.mu.Unlock()
vm, err := f.vm(node, vmid)
if err != nil {
return "", err
}
// A hibernated VM is destroyed like a stopped one (Phase 10 captures
// destroy_hibernated_response and task_status_destroy_hibernated).
if vm.lock != "" && vm.lock != lockSuspended {
return "", errLocked(vm.lock)
}
if vm.status == "running" {
return "", fakeErr(500, fmt.Sprintf("fake: VM %d is running - destroy failed", vmid))
}
exit := "OK"
if f.destroyFailsWith != "" {
exit = f.destroyFailsWith
}
return f.newTask(node, "qmdestroy", strconv.Itoa(vmid), exit, func() {
if exit == "OK" && f.vms[vmid] == vm {
delete(f.vms, vmid)
}
}), nil
}
// TaskStatus answers running for f.taskPolls polls, then stopped with the
// task's exitstatus, running its effect once.
func (f *fakeAPI) TaskStatus(ctx context.Context, node, upid string) (taskStatus, error) {
if err := f.enter(ctx, "TaskStatus"); err != nil {
return taskStatus{}, err
}
f.mu.Lock()
defer f.mu.Unlock()
t, ok := f.tasks[upid]
if !ok || t.node != node {
return taskStatus{}, fakeErr(500, "fake: no such task '"+upid+"'")
}
t.polls++
s := taskStatus{UPID: upid, Type: t.kind, ID: t.id, Node: t.node, Status: "running"}
if !t.done && t.polls > f.taskPolls {
t.done = true
if t.onDone != nil {
t.onDone()
}
}
if t.done {
s.Status, s.ExitStatus = "stopped", t.exit
}
return s, nil
}
// AgentNetworkInterfaces answers with the three messages of F5 in the
// order Proxmox checks them.
func (f *fakeAPI) AgentNetworkInterfaces(ctx context.Context, node string, vmid int) ([]agentInterface, error) {
if err := f.enter(ctx, "AgentNetworkInterfaces"); err != nil {
return nil, err
}
f.mu.Lock()
defer f.mu.Unlock()
vm, err := f.vm(node, vmid)
if err != nil {
return nil, err
}
if agent := vm.config["agent"]; agent != "1" && !strings.HasPrefix(agent, "1,") && !strings.Contains(agent, "enabled=1") {
return nil, fakeErr(500, msgAgentNotConfigured)
}
if vm.status != "running" {
return nil, errVMNotRunning(vmid)
}
if f.agentNotRunning {
return nil, fakeErr(500, msgAgentNotRunning)
}
return slices.Clone(vm.ifaces), nil
}
// UploadISO records the bytes and overwrites an existing file silently
// (F12). A body that disagrees with size fails, as the real Content-Length
// would.
func (f *fakeAPI) UploadISO(ctx context.Context, node, storage, filename string, r io.Reader, size int64) (string, error) {
if err := f.enter(ctx, "UploadISO"); err != nil {
return "", err
}
var content []byte
if r != nil {
var err error
if content, err = io.ReadAll(r); err != nil {
return "", err
}
}
f.mu.Lock()
defer f.mu.Unlock()
if _, ok := f.storages[storage]; !ok {
return "", fakeErr(500, "fake: storage '"+storage+"' does not exist")
}
if int64(len(content)) != size {
return "", fakeErr(400, fmt.Sprintf("fake: body has %d bytes, size says %d", len(content), size))
}
if f.isos[storage] == nil {
f.isos[storage] = make(map[string][]byte)
}
f.isos[storage][filename] = content
return f.newTask(node, "imgcopy", "", "OK", nil), nil
}
// DeleteVolume is a task that ends OK whether or not the volume exists
// (F13).
func (f *fakeAPI) DeleteVolume(ctx context.Context, node, storage, volid string) (string, error) {
if err := f.enter(ctx, "DeleteVolume"); err != nil {
return "", err
}
f.mu.Lock()
defer f.mu.Unlock()
if _, ok := f.storages[storage]; !ok {
return "", fakeErr(500, "fake: storage '"+storage+"' does not exist")
}
filename := strings.TrimPrefix(volid, storage+":iso/")
return f.newTask(node, "imgdel", storage, "OK", func() {
delete(f.isos[storage], filename)
}), nil
}
// diskSizeOption returns the size= option of a disk value.
func diskSizeOption(disk string) string {
for opt := range strings.SplitSeq(disk, ",") {
if v, ok := strings.CutPrefix(opt, "size="); ok {
return v
}
}
return ""
}
// fakeParseSize reads Proxmox's <n>[KMGT] sizes into bytes.
func fakeParseSize(s string) (int64, bool) {
if s == "" {
return 0, false
}
shift := map[byte]uint{'K': 10, 'M': 20, 'G': 30, 'T': 40}
unit, ok := shift[s[len(s)-1]]
if ok {
s = s[:len(s)-1]
}
n, err := strconv.ParseInt(s, 10, 64)
if err != nil || n < 0 {
return 0, false
}
return n << unit, true
}
func fakeFormatSize(b int64) string {
if b%(1<<30) == 0 {
return strconv.FormatInt(b>>30, 10) + "G"
}
return strconv.FormatInt(b>>20, 10) + "M"
}
// TestFakeAPI_errorsMatchCaptures keeps the fake's error texts equal to
// what Proxmox sent, so the provider's matchers are exercised against real
// wording in every test that goes through the fake.
func TestFakeAPI_errorsMatchCaptures(t *testing.T) {
t.Parallel()
tests := []struct {
capture string
fake *apiError
matcher func(error) bool
}{
{capture: "err_vm_missing", fake: errVMMissing("pv1-1", 999999), matcher: isNotExist},
{capture: "err_destroy_missing", fake: errVMMissing("pv1-1", 199999), matcher: isNotExist},
{capture: "err_config_while_clone_locked", fake: errLocked("clone"), matcher: isLocked},
{capture: "err_newid_in_use", fake: errNewIDInUse(199999), matcher: isAlreadyExists},
{capture: "storage_status_missing", fake: errNoSuchStorage(), matcher: isNoSuchStorage},
{capture: "err_agent_vm_stopped_agent_enabled", fake: errVMNotRunning(199999), matcher: isVMNotRunning},
{capture: "err_agent_not_running", fake: fakeErr(500, msgAgentNotRunning), matcher: isAgentNotRunning},
{capture: "err_agent_vm_stopped", fake: fakeErr(500, msgAgentNotConfigured), matcher: isAgentNotConfigured},
}
for _, tc := range tests {
t.Run(tc.capture, func(t *testing.T) {
t.Parallel()
want := loadAPIError(t, tc.capture)
if tc.fake.Status != want.Status || tc.fake.Message != want.Message || !maps.Equal(tc.fake.Errors, want.Errors) {
t.Errorf("fake error = %d %q %v, capture = %d %q %v", tc.fake.Status, tc.fake.Message, tc.fake.Errors, want.Status, want.Message, want.Errors)
}
if !tc.matcher(tc.fake) {
t.Errorf("matcher does not accept the fake's %q", tc.fake.Message)
}
})
}
}
// pollTask polls a fake task until it stops and returns its exitstatus.
func pollTask(t *testing.T, f *fakeAPI, upid string) string {
t.Helper()
for range 10 {
s, err := f.TaskStatus(context.Background(), fakeNode, upid)
if err != nil {
t.Fatalf("TaskStatus(%s) error = %v", upid, err)
}
if s.Status == "stopped" {
return s.ExitStatus
}
}
t.Fatalf("task %s still running after 10 polls", upid)
return ""
}
func resourceByID(t *testing.T, f *fakeAPI, vmid int) (vmResource, bool) {
t.Helper()
rs, err := f.ClusterResources(context.Background())
if err != nil {
t.Fatalf("ClusterResources error = %v", err)
}
for _, r := range rs {
if r.VMID == vmid {
return r, true
}
}
return vmResource{}, false
}
func TestFakeAPI_cloneLocksUntilTaskEnds(t *testing.T) {
t.Parallel()
ctx := context.Background()
f := newFakeAPI()
f.addTemplate(9001, "debian-13-template")
f.taskPolls = 1
upid, err := f.Clone(ctx, fakeNode, 9001, cloneParams{NewID: 150042, Name: "proxy-x", Description: "d"})
if err != nil {
t.Fatalf("Clone error = %v", err)
}
if r, _ := resourceByID(t, f, 150042); r.Name != "" || r.Lock != "clone" || r.Status != "unknown" {
t.Errorf("mid-clone entry = %+v, want nameless, lock clone, status unknown (F8)", r)
}
if err := f.SetVMConfig(ctx, fakeNode, 150042, map[string]string{"cores": "1"}); !isLocked(err) {
t.Errorf("SetVMConfig mid-clone error = %v, want the lock answer (F7)", err)
}
if _, err := f.Clone(ctx, fakeNode, 9001, cloneParams{NewID: 150042, Name: "proxy-y"}); !isAlreadyExists(err) {
t.Errorf("second Clone onto 150042 error = %v, want config file already exists (F4)", err)
}
if s, _ := f.TaskStatus(ctx, fakeNode, upid); s.Status != "running" {
t.Errorf("first poll = %+v, want running with taskPolls 1", s)
}
if exit := pollTask(t, f, upid); exit != "OK" {
t.Errorf("clone exitstatus = %q, want OK", exit)
}
r, _ := resourceByID(t, f, 150042)
if r.Name != "proxy-x" || r.Lock != "" || r.Status != "stopped" || r.Template != 0 {
t.Errorf("cloned entry = %+v, want named, unlocked, stopped", r)
}
cfg, err := f.VMConfig(ctx, fakeNode, 150042)
if err != nil || cfg["description"] != "d" || cfg["scsi0"] != "local-lvm:vm-150042-disk-1,size=3G" || cfg["template"] != "" {
t.Errorf("clone config = %v, %v", cfg, err)
}
}
func TestFakeAPI_cloneFailureAndLeftLock(t *testing.T) {
t.Parallel()
ctx := context.Background()
t.Run("failed clone removes the VM", func(t *testing.T) {
t.Parallel()
f := newFakeAPI()
f.addTemplate(9001, "debian-13-template")
f.cloneFailsWith = "clone failed: out of space"
upid, _ := f.Clone(ctx, fakeNode, 9001, cloneParams{NewID: 150042, Name: "proxy-x"})
if exit := pollTask(t, f, upid); exit != "clone failed: out of space" {
t.Errorf("exitstatus = %q", exit)
}
if _, ok := resourceByID(t, f, 150042); ok {
t.Error("failed clone left its VM")
}
})
t.Run("interrupted clone keeps the lock", func(t *testing.T) {
t.Parallel()
f := newFakeAPI()
f.addTemplate(9001, "debian-13-template")
f.cloneLeavesLock = true
upid, _ := f.Clone(ctx, fakeNode, 9001, cloneParams{NewID: 150042, Name: "proxy-x"})
pollTask(t, f, upid)
if r, _ := resourceByID(t, f, 150042); r.Lock != "clone" {
t.Errorf("entry = %+v, want lock clone kept", r)
}
})
}
func TestFakeAPI_powerAgentAndDestroy(t *testing.T) {
t.Parallel()
ctx := context.Background()
f := newFakeAPI()
vm := f.addVM(150042, "proxy-x", "stopped")
vm.ifaces = []agentInterface{{Name: "ens18", IPAddresses: []agentIP{{Address: "192.168.4.147", Type: "ipv4", Prefix: 24}}}}
if _, err := f.AgentNetworkInterfaces(ctx, fakeNode, 150042); !isVMNotRunning(err) {
t.Errorf("agent on a stopped VM error = %v, want VM is not running", err)
}
upid, _ := f.Start(ctx, fakeNode, 150042)
pollTask(t, f, upid)
if s, _ := f.VMStatus(ctx, fakeNode, 150042); s.Status != "running" || s.QMPStatus != "running" {
t.Errorf("status after start = %+v", s)
}
f.agentNotRunning = true
if _, err := f.AgentNetworkInterfaces(ctx, fakeNode, 150042); !isAgentNotRunning(err) {
t.Errorf("agent error = %v, want agent not running", err)
}
f.agentNotRunning = false
if ifaces, err := f.AgentNetworkInterfaces(ctx, fakeNode, 150042); err != nil || firstIPv4(ifaces) != "192.168.4.147" {
t.Errorf("agent = %v, %v", ifaces, err)
}
if _, err := f.Destroy(ctx, fakeNode, 150042); err == nil {
t.Error("Destroy of a running VM succeeded")
}
upid, _ = f.Stop(ctx, fakeNode, 150042)
pollTask(t, f, upid)
upid, err := f.Destroy(ctx, fakeNode, 150042)
if err != nil {
t.Fatalf("Destroy error = %v", err)
}
pollTask(t, f, upid)
if _, err := f.VMConfig(ctx, fakeNode, 150042); !isNotExist(err) {
t.Errorf("VMConfig after destroy error = %v, want does not exist", err)
}
if _, err := f.VMStatus(ctx, "pv1-2", 150042); !isNotExist(err) {
t.Errorf("lookup on another node error = %v, want does not exist", err)
}
}
func TestFakeAPI_resizeIsATask(t *testing.T) {
t.Parallel()
tests := []struct {
name string
size string
wantExit string
wantDisk string
}{
{name: "grow absolute", size: "8G", wantExit: "OK", wantDisk: "local-lvm:vm-150042-disk-1,size=8G"},
{name: "grow relative", size: "+1G", wantExit: "OK", wantDisk: "local-lvm:vm-150042-disk-1,size=4G"},
{name: "same size", size: "3G", wantExit: "OK", wantDisk: "local-lvm:vm-150042-disk-1,size=3G"},
{name: "shrink fails in the task", size: "2G", wantExit: loadTaskExit(t, "task_status_stopped_error"), wantDisk: "local-lvm:vm-150042-disk-1,size=3G"},
}
for _, tc := range tests {
t.Run(tc.name, func(t *testing.T) {
t.Parallel()
f := newFakeAPI()
f.addVM(150042, "proxy-x", "stopped")
upid, err := f.Resize(context.Background(), fakeNode, 150042, "scsi0", tc.size)
if err != nil {
t.Fatalf("Resize error = %v", err)
}
if exit := pollTask(t, f, upid); exit != tc.wantExit {
t.Errorf("exitstatus = %q, want %q", exit, tc.wantExit)
}
if vm, _ := f.vmSnapshot(150042); vm.config["scsi0"] != tc.wantDisk {
t.Errorf("scsi0 = %q, want %q", vm.config["scsi0"], tc.wantDisk)
}
})
}
}
func loadTaskExit(t *testing.T, name string) string {
t.Helper()
return decodeData[taskStatus](t, name).ExitStatus
}
func TestFakeAPI_uploadOverwritesAndDeleteIsIdempotent(t *testing.T) {
t.Parallel()
ctx := context.Background()
f := newFakeAPI()
for _, body := range []string{"first", "second"} {
upid, err := f.UploadISO(ctx, fakeNode, fakeISOStorage, "proxy-x-seed.iso", strings.NewReader(body), int64(len(body)))
if err != nil {
t.Fatalf("UploadISO error = %v", err)
}
pollTask(t, f, upid)
}
if b, _ := f.iso(fakeISOStorage, "proxy-x-seed.iso"); string(b) != "second" {
t.Errorf("iso = %q, want the second upload (F12 overwrite)", b)
}
if _, err := f.UploadISO(ctx, fakeNode, fakeISOStorage, "x.iso", strings.NewReader("abc"), 4); err == nil {
t.Error("UploadISO with a wrong size succeeded")
}
for range 2 {
upid, err := f.DeleteVolume(ctx, fakeNode, fakeISOStorage, seedVolID(fakeISOStorage, "proxy-x"))
if err != nil {
t.Fatalf("DeleteVolume error = %v", err)
}
if exit := pollTask(t, f, upid); exit != "OK" {
t.Errorf("delete exitstatus = %q, want OK even when absent (F13)", exit)
}
}
if _, ok := f.iso(fakeISOStorage, "proxy-x-seed.iso"); ok {
t.Error("iso still present after delete")
}
}
func TestFakeAPI_hooksBlocksAndLog(t *testing.T) {
t.Parallel()
f := newFakeAPI()
boom := errors.New("boom")
f.failWith("Version", boom)
if _, err := f.Version(context.Background()); !errors.Is(err, boom) {
t.Errorf("Version error = %v, want the hook's", err)
}
f.blockOn("ClusterResources")
ctx, cancel := context.WithCancel(context.Background())
cancel()
if _, err := f.ClusterResources(ctx); !errors.Is(err, context.Canceled) {
t.Errorf("blocked call with ended ctx error = %v, want context.Canceled", err)
}
if got := f.callLog(); !slices.Equal(got, []string{"Version", "ClusterResources"}) {
t.Errorf("call log = %v", got)
}
if f.callCount("Version") != 1 {
t.Errorf("Version count = %d, want 1", f.callCount("Version"))
}
}
@@ -0,0 +1,57 @@
package proxmox
import (
"encoding/json"
"os"
"path/filepath"
"testing"
)
// capture is one response recorded from the real host in Phase 0: the
// status, the reason phrase after the status code, and the raw body.
type capture struct {
Status int `json:"status"`
Reason string `json:"reason"`
Body string `json:"body"`
}
// readCapture loads testdata/<name>.json and fails the test when it is
// missing, so every string a test relies on is backed by a real response.
func readCapture(t *testing.T, name string) capture {
t.Helper()
raw, err := os.ReadFile(filepath.Join("testdata", name+".json"))
if err != nil {
t.Fatalf("reading capture %s: %v", name, err)
}
var c capture
if err := json.Unmarshal(raw, &c); err != nil {
t.Fatalf("parsing capture %s: %v", name, err)
}
return c
}
// loadAPIError turns an error capture into the apiError the client builds.
func loadAPIError(t *testing.T, name string) *apiError {
t.Helper()
c := readCapture(t, name)
if c.Status < 400 {
t.Fatalf("capture %s has status %d, want an error status", name, c.Status)
}
return newAPIError(c.Status, c.Reason, []byte(c.Body))
}
// decodeData decodes the "data" member of a successful capture into T.
func decodeData[T any](t *testing.T, name string) T {
t.Helper()
c := readCapture(t, name)
if c.Status != 200 {
t.Fatalf("capture %s has status %d, want 200", name, c.Status)
}
var env struct {
Data T `json:"data"`
}
if err := json.Unmarshal([]byte(c.Body), &env); err != nil {
t.Fatalf("decoding capture %s: %v", name, err)
}
return env.Data
}
+132
View File
@@ -0,0 +1,132 @@
package proxmox
import (
"context"
"errors"
"fmt"
"github.com/go-logr/logr"
"gitea.home.hrajfrisbee.cz/kacerr/egress-proxies-operator/internal/provider"
)
// lockSuspended is the lock of a VM hibernated to disk: it is not a task in
// progress, and the VM stays that way until someone resumes it.
const lockSuspended = "suspended"
// Get is spec entries 7 to 9: the VMID's resource entry, then its config
// for the name and the mark, then status/current and, for a running VM, the
// guest agent. A running Get makes four calls.
func (p *Provider) Get(ctx context.Context, id string) (*provider.Instance, error) {
vmid, name, err := parseProviderID(id)
if err != nil {
return nil, p.wrapErr("get", id, err)
}
log := p.logger(ctx).WithValues("vmid", vmid, "name", name)
inst, err := p.get(ctx, log, vmid, name)
if err != nil {
err = p.wrapErr("get", id, err)
if errors.Is(err, provider.ErrNotFound) {
log.V(1).Info("proxmox vm not found", "reason", err.Error())
}
return nil, err
}
return inst, nil
}
func (p *Provider) get(ctx context.Context, log logr.Logger, vmid int, name string) (*provider.Instance, error) {
r, ok, err := p.resolve(ctx, vmid)
if err != nil {
return nil, err
}
if !ok {
return nil, fmt.Errorf("%w: VM %d is not in the cluster's resources", provider.ErrNotFound, vmid)
}
log = log.WithValues("node", r.Node)
// A mid-clone entry has no name and status unknown, so the lock is
// checked before anything that would read it as absent or foreign.
if r.Lock != "" && r.Lock != lockSuspended {
return nil, transientf("VM %d is locked (%s)", vmid, r.Lock)
}
if r.Status == "unknown" {
return nil, transientf("VM %d has status unknown: its node %s is not answering, or the VM is too new to be reported yet", vmid, r.Node)
}
cfg, err := p.api.VMConfig(ctx, r.Node, vmid)
if err != nil {
return nil, fmt.Errorf("reading config of VM %d: %w", vmid, err)
}
if lock := cfg["lock"]; lock != "" && lock != lockSuspended {
return nil, transientf("VM %d is locked (%s)", vmid, lock)
}
if cfg["name"] != name {
return nil, fmt.Errorf("%w: VM %d is now named %q: the VMID was reused", provider.ErrNotFound, vmid, cfg["name"])
}
mark := parseDescription(cfg["description"])
if !mark.Managed {
return nil, fmt.Errorf("%w: VM %d carries no proxy-operator mark", provider.ErrNotFound, vmid)
}
inst := &provider.Instance{ID: formatProviderID(vmid, name), UID: mark.UID, CreatedAt: mark.CreatedAt}
// A hibernated VM is Stopped whatever status/current says; it needs no
// further call.
if r.Lock == lockSuspended || cfg["lock"] == lockSuspended {
inst.State = provider.StateStopped
log.V(1).Info("proxmox vm fetched", "state", inst.State, "lock", lockSuspended, "uid", inst.UID)
return inst, nil
}
st, err := p.api.VMStatus(ctx, r.Node, vmid)
if err != nil {
return nil, fmt.Errorf("reading status of VM %d: %w", vmid, err)
}
if guestRunning(st.Status, st.QMPStatus) {
ip, state, err := p.agentAddress(ctx, log, r.Node, vmid)
if err != nil {
return nil, err
}
inst.IP = ip
inst.State = state
} else {
inst.State = mapState(st.Status, st.QMPStatus, "")
}
log.V(1).Info("proxmox vm fetched", "state", inst.State, "ip", inst.IP, "uid", inst.UID, "status", st.Status, "qmpstatus", st.QMPStatus)
return inst, nil
}
// agentAddress asks the guest agent of a running VM for its address and
// turns its answers into a state (spec entry 7). Any answer it cannot place
// is Transient, whatever its status: an address read never latches or
// loses a Proxy.
func (p *Provider) agentAddress(ctx context.Context, log logr.Logger, node string, vmid int) (string, provider.InstanceState, error) {
ifaces, err := p.api.AgentNetworkInterfaces(ctx, node, vmid)
switch {
case err == nil:
ip := firstIPv4(ifaces)
log.V(2).Info("proxmox guest agent answered", "interfaces", len(ifaces), "ip", ip)
return ip, mapState("running", "running", ip), nil
case isAgentNotRunning(err):
return "", provider.StateProvisioning, nil
case isAgentNotConfigured(err):
// agent=1 was removed: this VM can never report an address, so it
// is replaced by one that can.
return "", provider.StateStopped, nil
default:
// Includes "VM <vmid> is not running": it stopped since the status
// read, and the next Get sees that directly.
return "", "", fmt.Errorf("%w: reading the guest agent of VM %d: %w", provider.ErrTransient, vmid, err)
}
}
// resolve finds a VMID in the cluster's resources. Only QEMU VMs count: a
// container at the VMID means ours is gone.
func (p *Provider) resolve(ctx context.Context, vmid int) (vmResource, bool, error) {
rs, err := p.api.ClusterResources(ctx)
if err != nil {
return vmResource{}, false, fmt.Errorf("listing cluster resources: %w", err)
}
for _, r := range rs {
if r.VMID == vmid && r.Type == "qemu" {
return r, true, nil
}
}
return vmResource{}, false, nil
}
+536
View File
@@ -0,0 +1,536 @@
package proxmox
import (
"errors"
"slices"
"strings"
"sync"
"testing"
"time"
"gitea.home.hrajfrisbee.cz/kacerr/egress-proxies-operator/internal/provider"
)
// mutateOn changes the fake's state under its lock at the start of every
// call to method: a VM changing between two of Get's calls.
func mutateOn(f *fakeAPI, method string, fn func()) {
f.hookOn(method, func() error {
f.mu.Lock()
defer f.mu.Unlock()
fn()
return nil
})
}
// capturedIfaces is the agent's answer from the Phase 0 guest: lo and ens18
// with 192.168.0.150.
func capturedIfaces(t *testing.T) []agentInterface {
t.Helper()
return decodeData[struct {
Result []agentInterface `json:"result"`
}](t, "agent_network_get_interfaces").Result
}
// assertGetFails runs Get on fx.id(), checks the error's class and returns
// its text.
func assertGetFails(t *testing.T, fx *fixture, class error) string {
t.Helper()
inst, err := fx.p.Get(fx.ctx, fx.id())
if inst != nil {
t.Errorf("Get = %+v, want nil", inst)
}
assertProviderError(t, err, "get", class)
return err.Error()
}
func TestGet_missingVMIsNotFound(t *testing.T) {
t.Parallel()
fx := newFixture(t, 1)
fx.addOurVM(testVMID+1, testName, "running")
assertGetFails(t, fx, provider.ErrNotFound)
lines := fx.logs.matching(`"msg"="proxmox vm not found"`)
if len(lines) != 1 || !strings.Contains(lines[0], `"vmid"=150042`) || !strings.Contains(lines[0], `"name"="`+testName+`"`) {
t.Errorf("not-found lines = %v, want one with vmid and name", lines)
}
if got := fx.fake.callLog(); !slices.Equal(got, []string{"ClusterResources"}) {
t.Errorf("calls = %v, want only the resources listing", got)
}
}
func TestGet_containerAtVMIDIsNotFound(t *testing.T) {
t.Parallel()
fx := newFixture(t, 1)
fx.addOurVM(testVMID, testName, "running").kind = "lxc"
assertGetFails(t, fx, provider.ErrNotFound)
}
func TestGet_nameMismatchIsNotFound(t *testing.T) {
t.Parallel()
fx := newFixture(t, 1)
fx.addOurVM(testVMID, "proxy-someoneelse00000", "running")
msg := assertGetFails(t, fx, provider.ErrNotFound)
if !strings.Contains(msg, "reused") {
t.Errorf("error = %s, want it to say the VMID was reused", msg)
}
if n := fx.fake.callCount("VMStatus"); n != 0 {
t.Errorf("VMStatus calls = %d, want 0 for a VM that is not ours", n)
}
if lines := fx.logs.matching(`"msg"="proxmox vm not found"`); len(lines) != 1 {
t.Errorf("not-found lines = %v, want one", lines)
}
}
func TestGet_unmarkedIsNotFound(t *testing.T) {
t.Parallel()
tests := []struct {
name string
description string
}{
{name: "no description"},
{name: "managed false", description: "proxy-operator-managed=false\nproxy-operator-uid=" + testUID},
{name: "admin notes only", description: "hand made, do not touch"},
}
for _, tc := range tests {
t.Run(tc.name, func(t *testing.T) {
t.Parallel()
fx := newFixture(t, 1)
vm := fx.fake.addVM(testVMID, testName, "running")
if tc.description != "" {
vm.config["description"] = tc.description
}
msg := assertGetFails(t, fx, provider.ErrNotFound)
if !strings.Contains(msg, "no proxy-operator mark") {
t.Errorf("error = %s, want it to name the missing mark", msg)
}
})
}
}
// TestGet_otherVMWithOurNameIsIgnored replaces the plan's
// TestGet_twoVMsWithOurNameIsPermanent: spec entries 2 and 9 address Get by
// VMID, and only Create treats a duplicate name as Permanent.
func TestGet_otherVMWithOurNameIsIgnored(t *testing.T) {
t.Parallel()
fx := newFixture(t, 1)
fx.addOurVM(testVMID, testName, "stopped")
fx.addOurVM(testVMID+7, testName, "running")
if inst := fx.get(fx.id()); inst.State != provider.StateStopped || inst.ID != fx.id() {
t.Errorf("Get = %+v, want the VM at the ID's VMID, Stopped", inst)
}
}
func TestGet_vmOnAnotherNodeIsFound(t *testing.T) {
t.Parallel()
fx := newFixture(t, 1)
vm := fx.addOurVM(testVMID, testName, "running")
vm.node = "pv1-2"
vm.ifaces = capturedIfaces(t)
inst := fx.get(fx.id())
if inst.State != provider.StateRunning || inst.IP != "192.168.0.150" {
t.Errorf("Get = %+v, want Running with 192.168.0.150 read from pv1-2", inst)
}
if lines := fx.logs.matching(`"msg"="proxmox vm fetched"`); len(lines) != 1 || !strings.Contains(lines[0], `"node"="pv1-2"`) {
t.Errorf("fetched lines = %v, want the resolved node", lines)
}
}
func TestGet_runningWithoutIPIsProvisioning(t *testing.T) {
t.Parallel()
fx := newFixture(t, 1)
fx.addOurVM(testVMID, testName, "running")
if inst := fx.get(fx.id()); inst.State != provider.StateProvisioning || inst.IP != "" {
t.Errorf("Get = %+v, want Provisioning with no IP", inst)
}
}
func TestGet_runningWithIPIsRunning(t *testing.T) {
t.Parallel()
fx := newFixture(t, 1)
fx.addOurVM(testVMID, testName, "running").ifaces = capturedIfaces(t)
inst := fx.get(fx.id())
if inst.State != provider.StateRunning || inst.IP != "192.168.0.150" || inst.ID != fx.id() {
t.Errorf("Get = %+v, want Running with 192.168.0.150", inst)
}
lines := fx.logs.matching(`"msg"="proxmox vm fetched"`)
if len(lines) != 1 {
t.Fatalf("fetched lines = %v, want one", lines)
}
for _, kv := range []string{`"provider"="proxmox-home"`, `"vmid"=150042`, `"name"="` + testName + `"`, `"node"="pv1-1"`, `"state"="Running"`, `"ip"="192.168.0.150"`, `"uid"="` + testUID + `"`} {
if !strings.Contains(lines[0], kv) {
t.Errorf("fetched line lacks %s: %s", kv, lines[0])
}
}
}
func TestGet_runningMakesFourCalls(t *testing.T) {
t.Parallel()
fx := newFixture(t, 1)
fx.addOurVM(testVMID, testName, "running").ifaces = capturedIfaces(t)
fx.get(fx.id())
want := []string{"ClusterResources", "VMConfig", "VMStatus", "AgentNetworkInterfaces"}
if got := fx.fake.callLog(); !slices.Equal(got, want) {
t.Errorf("calls = %v, want %v", got, want)
}
}
func TestGet_agentNotRunningIsProvisioning(t *testing.T) {
t.Parallel()
fx := newFixture(t, 1)
fx.addOurVM(testVMID, testName, "running").ifaces = capturedIfaces(t)
fx.fake.agentNotRunning = true
if inst := fx.get(fx.id()); inst.State != provider.StateProvisioning || inst.IP != "" {
t.Errorf("Get = %+v, want Provisioning while the agent is not running", inst)
}
}
// TestGet_agentMessagesTable is spec entry 7's table, with the agent's
// answers taken from the captures.
func TestGet_agentMessagesTable(t *testing.T) {
t.Parallel()
tests := []struct {
name string
agentErr error
ifaces []agentInterface
wantState provider.InstanceState
wantClass error
}{
{name: "200 with a routable IPv4", ifaces: capturedIfaces(t), wantState: provider.StateRunning},
{
name: "200 without an IPv4",
ifaces: []agentInterface{{Name: "lo", IPAddresses: []agentIP{{Address: "127.0.0.1", Type: "ipv4"}}}, {Name: "ens18", IPAddresses: []agentIP{{Address: "fe80::1", Type: "ipv6"}}}},
wantState: provider.StateProvisioning,
},
{name: "agent not running", agentErr: loadAPIError(t, "err_agent_not_running"), wantState: provider.StateProvisioning},
{name: "no agent configured", agentErr: loadAPIError(t, "err_agent_vm_stopped"), wantState: provider.StateStopped},
{name: "VM stopped since the status read", agentErr: loadAPIError(t, "err_agent_vm_stopped_agent_enabled"), wantClass: provider.ErrTransient},
}
for _, tc := range tests {
t.Run(tc.name, func(t *testing.T) {
t.Parallel()
fx := newFixture(t, 1)
fx.addOurVM(testVMID, testName, "running").ifaces = tc.ifaces
if tc.agentErr != nil {
fx.fake.failWith("AgentNetworkInterfaces", tc.agentErr)
}
if tc.wantClass != nil {
assertGetFails(t, fx, tc.wantClass)
return
}
inst := fx.get(fx.id())
if inst.State != tc.wantState {
t.Errorf("Get state = %s, want %s", inst.State, tc.wantState)
}
if (inst.State == provider.StateRunning) != (inst.IP != "") {
t.Errorf("Get = %+v: an IP goes with Running and only with Running", inst)
}
})
}
}
func TestGet_agentOtherErrorIsTransient(t *testing.T) {
t.Parallel()
tests := []struct {
name string
err error
}{
{name: "forbidden", err: loadAPIError(t, "err_forbidden")},
{name: "VM vanished since the status read", err: loadAPIError(t, "err_vm_missing")},
{name: "pveproxy 596", err: fakeErr(596, "Broken pipe")},
{name: "other 500", err: fakeErr(500, "fake: guest-network-get-interfaces timed out")},
{name: "not an api error", err: errors.New("read tcp: connection reset by peer")},
}
for _, tc := range tests {
t.Run(tc.name, func(t *testing.T) {
t.Parallel()
fx := newFixture(t, 1)
fx.addOurVM(testVMID, testName, "running")
fx.fake.failWith("AgentNetworkInterfaces", tc.err)
assertGetFails(t, fx, provider.ErrTransient)
})
}
}
func TestGet_ipFilterSkipsLoopbackAndLinkLocal(t *testing.T) {
t.Parallel()
fx := newFixture(t, 1)
fx.addOurVM(testVMID, testName, "running").ifaces = []agentInterface{
{Name: "lo", IPAddresses: []agentIP{{Address: "127.0.0.1", Type: "ipv4"}}},
{Name: "ens18", IPAddresses: []agentIP{{Address: "169.254.10.2", Type: "ipv4"}, {Address: "fe80::be24:11ff:fe48:1146", Type: "ipv6"}}},
{Name: "ens19", IPAddresses: []agentIP{{Address: "2001:db8::5", Type: "ipv6"}, {Address: "192.168.4.147", Type: "ipv4"}}},
}
if inst := fx.get(fx.id()); inst.State != provider.StateRunning || inst.IP != "192.168.4.147" {
t.Errorf("Get = %+v, want Running with 192.168.4.147", inst)
}
}
func TestGet_stoppedIsStopped(t *testing.T) {
t.Parallel()
fx := newFixture(t, 1)
fx.addOurVM(testVMID, testName, "stopped")
if inst := fx.get(fx.id()); inst.State != provider.StateStopped || inst.IP != "" || inst.UID != testUID {
t.Errorf("Get = %+v, want Stopped with uid %s", inst, testUID)
}
if n := fx.fake.callCount("AgentNetworkInterfaces"); n != 0 {
t.Errorf("agent calls = %d, want 0 for a stopped VM", n)
}
}
func TestGet_pausedIsStopped(t *testing.T) {
t.Parallel()
for _, qmp := range []string{"paused", "suspended", "io-error", "postmigrate", "internal-error"} {
t.Run(qmp, func(t *testing.T) {
t.Parallel()
fx := newFixture(t, 1)
fx.addOurVM(testVMID, testName, "running").qmpstatus = qmp
if inst := fx.get(fx.id()); inst.State != provider.StateStopped {
t.Errorf("Get state with qmpstatus %s = %s, want Stopped", qmp, inst.State)
}
if n := fx.fake.callCount("AgentNetworkInterfaces"); n != 0 {
t.Errorf("agent calls = %d, want 0", n)
}
})
}
}
// TestGet_pausedIsStoppedFromStatusCurrent uses F14's paused VM: the
// resources listing shows plain running, only status/current tells.
func TestGet_pausedIsStoppedFromStatusCurrent(t *testing.T) {
t.Parallel()
st := decodeData[vmStatus](t, "status_current_paused")
if st.Status != "running" || st.QMPStatus != "paused" {
t.Fatalf("capture status_current_paused = %+v, want running and paused", st)
}
fx := newFixture(t, 1)
vm := fx.addOurVM(testVMID, testName, st.Status)
vm.qmpstatus = st.QMPStatus
vm.ifaces = capturedIfaces(t)
if r, _ := resourceByID(t, fx.fake, testVMID); r.Status != "running" {
t.Fatalf("resources status = %q, want running as F14 saw", r.Status)
}
if inst := fx.get(fx.id()); inst.State != provider.StateStopped || inst.IP != "" {
t.Errorf("Get = %+v, want Stopped with no IP", inst)
}
}
func TestGet_prelaunchIsProvisioning(t *testing.T) {
t.Parallel()
fx := newFixture(t, 1)
vm := fx.addOurVM(testVMID, testName, "running")
vm.qmpstatus = "prelaunch"
vm.ifaces = capturedIfaces(t)
if inst := fx.get(fx.id()); inst.State != provider.StateProvisioning || inst.IP != "" {
t.Errorf("Get = %+v, want Provisioning", inst)
}
}
func TestGet_suspendedLockIsStopped(t *testing.T) {
t.Parallel()
fx := newFixture(t, 1)
fx.addOurVM(testVMID, testName, "stopped").lock = lockSuspended
inst := fx.get(fx.id())
if inst.State != provider.StateStopped || inst.UID != testUID {
t.Errorf("Get = %+v, want Stopped with uid", inst)
}
if got := fx.fake.callLog(); !slices.Equal(got, []string{"ClusterResources", "VMConfig"}) {
t.Errorf("calls = %v, want resources and config only", got)
}
}
func TestGet_unknownStatusIsTransient(t *testing.T) {
t.Parallel()
fx := newFixture(t, 1)
fx.addOurVM(testVMID, testName, "running").status = "unknown"
msg := assertGetFails(t, fx, provider.ErrTransient)
// On pv1-1 the first Get after Create met this with the node up: the
// listing had not reported the new VM yet (Phase 10 run, finding P6).
if !strings.Contains(msg, "pv1-1") || !strings.Contains(msg, "too new to be reported yet") {
t.Errorf("error = %s, want it to name the node and both causes", msg)
}
if n := fx.fake.callCount("VMConfig"); n != 0 {
t.Errorf("VMConfig calls = %d, want 0 while the node is unknown", n)
}
}
func TestGet_lockedIsTransientNeverNotFound(t *testing.T) {
t.Parallel()
for _, lock := range []string{"backup", "migrate", "snapshot", "rollback", "create"} {
t.Run("resources lock "+lock, func(t *testing.T) {
t.Parallel()
fx := newFixture(t, 1)
fx.addOurVM(testVMID, testName, "running").lock = lock
msg := assertGetFails(t, fx, provider.ErrTransient)
if !strings.Contains(msg, "locked ("+lock+")") {
t.Errorf("error = %s, want it to name the lock", msg)
}
})
}
t.Run("lock taken between the resources and config reads", func(t *testing.T) {
t.Parallel()
fx := newFixture(t, 1)
vm := fx.addOurVM(testVMID, testName, "running")
mutateOn(fx.fake, "VMConfig", func() { vm.lock = "backup" })
assertGetFails(t, fx, provider.ErrTransient)
if n := fx.fake.callCount("VMStatus"); n != 0 {
t.Errorf("VMStatus calls = %d, want 0", n)
}
})
}
// TestGet_midCloneNamelessEntryIsTransient drives a real fake clone: while
// it runs the entry has no name (F8), which must not read as NotFound.
func TestGet_midCloneNamelessEntryIsTransient(t *testing.T) {
t.Parallel()
fx := newFixture(t, 1)
fx.fake.addTemplate(9001, "debian-13-template")
upid, err := fx.fake.Clone(fx.ctx, fakeNode, 9001, cloneParams{NewID: testVMID, Name: testName, Description: formatDescription(testUID, testCreatedAt)})
if err != nil {
t.Fatalf("Clone error = %v", err)
}
if r, _ := resourceByID(t, fx.fake, testVMID); r.Name != "" || r.Lock != "clone" {
t.Fatalf("mid-clone entry = %+v, want nameless and locked", r)
}
assertGetFails(t, fx, provider.ErrTransient)
pollTask(t, fx.fake, upid)
if inst := fx.get(fx.id()); inst.State != provider.StateStopped || inst.UID != testUID {
t.Errorf("Get after the clone = %+v, want Stopped with uid", inst)
}
}
func TestGet_vanishedBetweenCallsIsNotFound(t *testing.T) {
t.Parallel()
for _, method := range []string{"VMConfig", "VMStatus"} {
t.Run("before "+method, func(t *testing.T) {
t.Parallel()
fx := newFixture(t, 1)
fx.addOurVM(testVMID, testName, "running")
mutateOn(fx.fake, method, func() { delete(fx.fake.vms, testVMID) })
assertGetFails(t, fx, provider.ErrNotFound)
})
}
}
func TestGet_unknownStateIsStopped(t *testing.T) {
t.Parallel()
fx := newFixture(t, 1)
vm := fx.addOurVM(testVMID, testName, "running")
vm.status, vm.qmpstatus = "hibernating-v2", "hibernating-v2"
if inst := fx.get(fx.id()); inst.State != provider.StateStopped {
t.Errorf("Get state = %s, want Stopped for an unrecognised status", inst.State)
}
}
func TestGet_readsUIDAndCreatedAtFromBackend(t *testing.T) {
t.Parallel()
fx := newFixture(t, 1)
created := time.Date(2026, 9, 1, 8, 30, 0, 0, time.UTC)
fx.addOurVM(testVMID, testName, "stopped").config["description"] = formatDescription("uid-from-backend", created)
inst := fx.get(fx.id())
if inst.UID != "uid-from-backend" || !inst.CreatedAt.Equal(created) {
t.Errorf("Get = uid %q createdAt %v, want uid-from-backend and %v", inst.UID, inst.CreatedAt, created)
}
}
// TestGet_unmanagedDescriptionHasEmptyUID: a mark without a uid line is
// still ours, with an empty UID, which callers must tolerate.
func TestGet_unmanagedDescriptionHasEmptyUID(t *testing.T) {
t.Parallel()
fx := newFixture(t, 1)
fx.addOurVM(testVMID, testName, "stopped").config["description"] = "proxy-operator-managed=true\r\nadmin note\r\n"
inst := fx.get(fx.id())
if inst.UID != "" || !inst.CreatedAt.IsZero() || inst.State != provider.StateStopped {
t.Errorf("Get = %+v, want empty uid, zero createdAt, Stopped", inst)
}
}
func TestGet_malformedIDIsPermanent(t *testing.T) {
t.Parallel()
for _, id := range []string{"", "vms/150042", "vms/0150042/" + testName, "vm/150042/" + testName, "vms/abc/" + testName, "vms/150042/a/b"} {
t.Run(id, func(t *testing.T) {
t.Parallel()
fx := newFixture(t, 1)
_, err := fx.p.Get(fx.ctx, id)
assertProviderError(t, err, "get", provider.ErrPermanent)
if calls := fx.fake.callLog(); len(calls) != 0 {
t.Errorf("calls = %v, want none", calls)
}
})
}
}
func TestGet_resourcesFailureIsTransient(t *testing.T) {
t.Parallel()
tests := []struct {
name string
err error
}{
{name: "pveproxy 595", err: fakeErr(595, "Connection refused")},
{name: "other 500", err: fakeErr(500, "fake: cluster filesystem not ready")},
{name: "network", err: errors.New("dial tcp 192.168.0.8:8006: connect: no route to host")},
}
for _, tc := range tests {
t.Run(tc.name, func(t *testing.T) {
t.Parallel()
fx := newFixture(t, 1)
fx.addOurVM(testVMID, testName, "running")
fx.fake.failWith("ClusterResources", tc.err)
assertGetFails(t, fx, provider.ErrTransient)
})
}
}
func TestGet_afterDeleteReturnsNotFound(t *testing.T) {
t.Parallel()
fx := newFixture(t, 1)
fx.addOurVM(testVMID, testName, "running").ifaces = capturedIfaces(t)
if inst := fx.get(fx.id()); inst.State != provider.StateRunning {
t.Fatalf("Get before Delete = %+v, want Running", inst)
}
if err := fx.p.Delete(fx.ctx, fx.id()); err != nil {
t.Fatalf("Delete error = %v", err)
}
assertGetFails(t, fx, provider.ErrNotFound)
}
// TestGet_duringDeleteStillReturnsInstance holds the destroy task running:
// destroying sets no lock and keeps the VM listed until the task ends, as
// Phase 0's mid-destroy captures show, so Get sees it Stopped, not gone.
func TestGet_duringDeleteStillReturnsInstance(t *testing.T) {
t.Parallel()
if _, locked := capturedConfig(t, "config_during_destroy")["lock"]; locked {
t.Fatal("capture config_during_destroy has a lock; the fake models none")
}
listed := decodeData[[]vmResource](t, "cluster_resources_during_destroy")
if !slices.ContainsFunc(listed, func(r vmResource) bool { return r.VMID == 199999 && r.Status == "stopped" && r.Lock == "" }) {
t.Fatal("capture cluster_resources_during_destroy does not list VM 199999 stopped and unlocked")
}
fx := newFixture(t, 1)
fx.addOurVM(testVMID, testName, "running")
inTask, release := make(chan struct{}), make(chan struct{})
var once sync.Once
fx.fake.hookOn("TaskStatus", func() error {
if fx.fake.callCount("Destroy") > 0 {
once.Do(func() {
close(inTask)
<-release
})
}
return nil
})
done := make(chan error, 1)
go func() { done <- fx.p.Delete(fx.ctx, fx.id()) }()
select {
case <-inTask:
case err := <-done:
t.Fatalf("Delete returned %v before its destroy task was polled", err)
}
inst := fx.get(fx.id())
if inst.State != provider.StateStopped || inst.UID != testUID || inst.ID != fx.id() {
t.Errorf("Get during the destroy = %+v, want the instance Stopped with its uid", inst)
}
close(release)
if err := <-done; err != nil {
t.Fatalf("Delete error = %v", err)
}
assertGetFails(t, fx, provider.ErrNotFound)
}
+42
View File
@@ -0,0 +1,42 @@
package proxmox
import (
"hash/fnv"
"strconv"
"strings"
"gitea.home.hrajfrisbee.cz/kacerr/egress-proxies-operator/internal/provider"
)
const idPrefix = "vms"
// formatProviderID gives vms/<vmid>/<name>: location-free, because VMs can
// migrate between nodes (spec entry 1).
func formatProviderID(vmid int, name string) string {
return idPrefix + "/" + strconv.Itoa(vmid) + "/" + name
}
// parseProviderID is strict: exactly three segments, the literal "vms", a
// decimal VMID without leading zeros in Proxmox's range, and a non-empty
// name. Anything else is ErrPermanent.
func parseProviderID(id string) (int, string, error) {
parts := strings.Split(id, "/")
if len(parts) != 3 || parts[0] != idPrefix || parts[2] == "" {
return 0, "", permanentf("malformed providerID %q, want vms/<vmid>/<name>", id)
}
vmid, err := strconv.Atoi(parts[1])
if err != nil || strconv.Itoa(vmid) != parts[1] || vmid < provider.ProxmoxMinVMID || vmid > provider.ProxmoxMaxVMID {
return 0, "", permanentf("malformed providerID %q: %q is not a VMID", id, parts[1])
}
return vmid, parts[2], nil
}
// vmidFor is the attempt-th VMID candidate for a Proxy UID: a stable hash of
// the UID placed in [min, max], moving forward and wrapping as attempt grows,
// so a retry of the same Create lands on the same VMID (spec entry 2).
func vmidFor(uid string, min, max, attempt int) int {
h := fnv.New64a()
_, _ = h.Write([]byte(uid))
span := uint64(max - min + 1)
return min + int((h.Sum64()%span+uint64(attempt)%span)%span)
}
+89
View File
@@ -0,0 +1,89 @@
package proxmox
import (
"errors"
"testing"
"gitea.home.hrajfrisbee.cz/kacerr/egress-proxies-operator/internal/provider"
)
func TestFormatProviderID_roundTrip(t *testing.T) {
t.Parallel()
for _, vmid := range []int{100, 150042, 999999999} {
id := formatProviderID(vmid, "proxy-abcdefghijklmnop")
gotVMID, gotName, err := parseProviderID(id)
if err != nil || gotVMID != vmid || gotName != "proxy-abcdefghijklmnop" {
t.Errorf("parseProviderID(%q) = %d, %q, %v; want %d, proxy-abcdefghijklmnop, nil", id, gotVMID, gotName, err, vmid)
}
}
if got := formatProviderID(150042, "proxy-x"); got != "vms/150042/proxy-x" {
t.Errorf("formatProviderID() = %q, want vms/150042/proxy-x", got)
}
}
func TestParseProviderID_malformed(t *testing.T) {
t.Parallel()
for _, id := range []string{
"",
"vms",
"vms/150042",
"vms/150042/",
"vms//proxy-x",
"vm/150042/proxy-x",
"hosts/beelink/domains/proxy-x",
"vms/150042/proxy-x/extra",
"vms/abc/proxy-x",
"vms/0150042/proxy-x",
"vms/+150042/proxy-x",
"vms/-5/proxy-x",
"vms/99/proxy-x",
"vms/1000000000/proxy-x",
} {
t.Run(id, func(t *testing.T) {
t.Parallel()
_, _, err := parseProviderID(id)
if !errors.Is(err, provider.ErrPermanent) {
t.Errorf("parseProviderID(%q) error = %v, want ErrPermanent", id, err)
}
})
}
}
func TestVMIDFor_deterministicAndInRange(t *testing.T) {
t.Parallel()
uids := []string{"3f1c2d7e-8a44-4f1b-9a6c-1d2e3f4a5b6c", "00000000-0000-0000-0000-000000000000", "x"}
for _, uid := range uids {
first := vmidFor(uid, 100000, 199999, 0)
if first < 100000 || first > 199999 {
t.Errorf("vmidFor(%q) = %d, want within 100000..199999", uid, first)
}
if again := vmidFor(uid, 100000, 199999, 0); again != first {
t.Errorf("vmidFor(%q) = %d then %d, want the same VMID every time", uid, first, again)
}
}
if vmidFor(uids[0], 100000, 199999, 0) == vmidFor(uids[1], 100000, 199999, 0) {
t.Error("two different UIDs got the same first VMID; the hash does not spread")
}
}
func TestVMIDFor_probesForwardAndWraps(t *testing.T) {
t.Parallel()
const min, max = 500, 509
uid := "3f1c2d7e-8a44-4f1b-9a6c-1d2e3f4a5b6c"
start := vmidFor(uid, min, max, 0)
seen := map[int]bool{}
for attempt := range 10 {
got := vmidFor(uid, min, max, attempt)
want := min + (start-min+attempt)%10
if got != want {
t.Errorf("attempt %d: vmidFor = %d, want %d (one step forward, wrapping at max)", attempt, got, want)
}
seen[got] = true
}
if len(seen) != 10 {
t.Errorf("10 attempts over a range of 10 visited %d VMIDs, want all 10", len(seen))
}
if got := vmidFor(uid, min, max, 10); got != start {
t.Errorf("attempt 10 = %d, want to wrap back to %d", got, start)
}
}
+81
View File
@@ -0,0 +1,81 @@
package proxmox
import (
"context"
"strings"
"gitea.home.hrajfrisbee.cz/kacerr/egress-proxies-operator/internal/provider"
)
// proxyTag is the tag Create sets for the Proxmox UI; ListByTag uses it,
// with the name prefix provider.NameFromUID gives, only to pick candidates.
// The mark decides.
const (
proxyTag = "proxy_operator"
proxyNamePrefix = "proxy-"
)
// ListByTag is spec entry 12: one resources call covers every node, then
// one config read per candidate for the mark. GC reads only ID, UID and
// CreatedAt (contract §2.4), so no status or agent call is made. The
// resources call failing fails the list; one unreadable candidate is
// skipped, unless ctx ended, which fails the list rather than pass off a
// partial view as complete.
func (p *Provider) ListByTag(ctx context.Context) ([]provider.Instance, error) {
log := p.logger(ctx)
rs, err := p.api.ClusterResources(ctx)
if err != nil {
return nil, p.wrapErr("list", "", err)
}
var (
out []provider.Instance
candidates, skipped int
)
for _, r := range rs {
if !isCandidate(r) {
continue
}
candidates++
vlog := log.WithValues("vmid", r.VMID, "name", r.Name, "node", r.Node)
cfg, err := p.api.VMConfig(ctx, r.Node, r.VMID)
if err != nil {
if ctx.Err() != nil {
return nil, p.wrapErr("list", "", err)
}
skipped++
vlog.V(1).Info("skipping unreadable proxmox vm", "error", err.Error())
continue
}
mark := parseDescription(cfg["description"])
if !mark.Managed {
continue
}
inst := provider.Instance{ID: formatProviderID(r.VMID, r.Name), UID: mark.UID, CreatedAt: mark.CreatedAt}
vlog.V(2).Info("proxmox listed instance", "id", inst.ID, "uid", inst.UID, "createdAt", inst.CreatedAt)
out = append(out, inst)
}
log.V(1).Info("proxmox vms listed", "vms", len(rs), "candidates", candidates, "managed", len(out), "skipped", skipped)
return out, nil
}
// isCandidate picks the resource entries worth a config read: QEMU VMs, not
// templates, named, and either tagged or named like a proxy. A nameless
// entry is a clone in progress, whose Create is still running.
func isCandidate(r vmResource) bool {
if r.Type != "qemu" || r.Template != 0 || r.Name == "" {
return false
}
return strings.HasPrefix(r.Name, proxyNamePrefix) || hasTag(r.Tags, proxyTag)
}
// hasTag reports whether tags, as Proxmox lists them (";"-separated, though
// "," and spaces are accepted on input), contains tag. Proxmox folds tags to
// lower case.
func hasTag(tags, tag string) bool {
for _, t := range strings.FieldsFunc(tags, func(r rune) bool { return r == ';' || r == ',' || r == ' ' }) {
if strings.EqualFold(t, tag) {
return true
}
}
return false
}
+227
View File
@@ -0,0 +1,227 @@
package proxmox
import (
"context"
"errors"
"slices"
"strings"
"testing"
"gitea.home.hrajfrisbee.cz/kacerr/egress-proxies-operator/internal/provider"
)
func listIDs(list []provider.Instance) []string {
ids := make([]string, len(list))
for i, inst := range list {
ids[i] = inst.ID
}
return ids
}
func (fx *fixture) list() []provider.Instance {
fx.t.Helper()
list, err := fx.p.ListByTag(fx.ctx)
if err != nil {
fx.t.Fatalf("ListByTag() error = %v", err)
}
return list
}
func TestListByTag_excludesUnmarkedVMs(t *testing.T) {
t.Parallel()
fx := newFixture(t, 1)
fx.addOurVM(testVMID, testName, "running")
fx.fake.addVM(150100, "proxy-nomark0000000000", "running")
fx.fake.addVM(150101, "proxy-managedfalse0000", "stopped").config["description"] = "proxy-operator-managed=false\nproxy-operator-uid=x"
fx.fake.addVM(150102, "tagged-by-hand", "running").config["tags"] = proxyTag
list := fx.list()
if got := listIDs(list); !slices.Equal(got, []string{fx.id()}) {
t.Fatalf("ListByTag = %v, want only %s", got, fx.id())
}
if list[0].UID != testUID || !list[0].CreatedAt.Equal(testCreatedAt) || list[0].State != "" || list[0].IP != "" {
t.Errorf("listed = %+v, want uid and createdAt only", list[0])
}
lines := fx.logs.matching(`"msg"="proxmox vms listed"`)
if len(lines) != 1 || !strings.Contains(lines[0], `"candidates"=4`) || !strings.Contains(lines[0], `"managed"=1`) || !strings.Contains(lines[0], `"skipped"=0`) {
t.Errorf("listed line = %v, want candidates 4, managed 1, skipped 0", lines)
}
}
func TestListByTag_emptyUIDStillListed(t *testing.T) {
t.Parallel()
fx := newFixture(t, 1)
fx.addOurVM(testVMID, testName, "stopped").config["description"] = "proxy-operator-managed=true"
list := fx.list()
if len(list) != 1 || list[0].UID != "" || list[0].ID != fx.id() {
t.Errorf("ListByTag = %+v, want the VM listed with an empty UID", list)
}
}
func TestListByTag_candidatesByTagOrNamePrefix(t *testing.T) {
t.Parallel()
fx := newFixture(t, 1)
// Named like a proxy, no tag: a candidate.
fx.addOurVM(150001, "proxy-notag00000000000", "running").config["tags"] = ""
// Tagged, renamed by an admin: a candidate. Tags as Proxmox lists them.
fx.addOurVM(150002, "renamed", "running").config["tags"] = "web;proxy_operator"
// Marked but neither tagged nor named like a proxy: never read.
fx.addOurVM(150003, "unrelated", "running").config["tags"] = "web"
got := listIDs(fx.list())
want := []string{formatProviderID(150001, "proxy-notag00000000000"), formatProviderID(150002, "renamed")}
if !slices.Equal(got, want) {
t.Errorf("ListByTag = %v, want %v", got, want)
}
if n := fx.fake.callCount("VMConfig"); n != 2 {
t.Errorf("VMConfig calls = %d, want 2, one per candidate", n)
}
}
func TestHasTag_table(t *testing.T) {
t.Parallel()
tests := []struct {
tags string
want bool
}{
{tags: "proxy_operator", want: true},
{tags: "a;proxy_operator;b", want: true},
{tags: "a,proxy_operator", want: true},
{tags: "a proxy_operator", want: true},
{tags: "Proxy_Operator", want: true},
{tags: ""},
{tags: "proxy_operator2"},
{tags: "proxy-operator"},
}
for _, tc := range tests {
if got := hasTag(tc.tags, proxyTag); got != tc.want {
t.Errorf("hasTag(%q) = %v, want %v", tc.tags, got, tc.want)
}
}
}
func TestListByTag_skipsLXCAndTemplates(t *testing.T) {
t.Parallel()
fx := newFixture(t, 1)
fx.addOurVM(150001, "proxy-container0000000", "running").kind = "lxc"
fx.addOurVM(150002, "proxy-template00000000", "stopped").template = true
if list := fx.list(); len(list) != 0 {
t.Errorf("ListByTag = %+v, want nothing", list)
}
if n := fx.fake.callCount("VMConfig"); n != 0 {
t.Errorf("VMConfig calls = %d, want 0", n)
}
}
func TestListByTag_midCloneEntryIsNotACandidate(t *testing.T) {
t.Parallel()
fx := newFixture(t, 1)
fx.fake.addTemplate(9001, "debian-13-template")
if _, err := fx.fake.Clone(fx.ctx, fakeNode, 9001, cloneParams{NewID: testVMID, Name: testName, Description: formatDescription(testUID, testCreatedAt)}); err != nil {
t.Fatalf("Clone error = %v", err)
}
if list := fx.list(); len(list) != 0 {
t.Errorf("ListByTag = %+v, want nothing while the clone runs", list)
}
}
func TestListByTag_resourcesFailureFailsWhole(t *testing.T) {
t.Parallel()
fx := newFixture(t, 1)
fx.addOurVM(testVMID, testName, "running")
fx.fake.failWith("ClusterResources", fakeErr(595, "Connection refused"))
list, err := fx.p.ListByTag(fx.ctx)
if list != nil {
t.Errorf("ListByTag = %+v, want nil", list)
}
assertProviderError(t, err, "list", provider.ErrTransient)
}
func TestListByTag_unreadableVMSkipped(t *testing.T) {
t.Parallel()
fx := newFixture(t, 1)
fx.addOurVM(150001, "proxy-destroyedmidswee", "running")
fx.addOurVM(testVMID, testName, "running")
// The first candidate is destroyed between the listing and its config
// read.
mutateOn(fx.fake, "VMConfig", func() { delete(fx.fake.vms, 150001) })
if got := listIDs(fx.list()); !slices.Equal(got, []string{fx.id()}) {
t.Errorf("ListByTag = %v, want only %s", got, fx.id())
}
lines := fx.logs.matching(`"msg"="skipping unreadable proxmox vm"`)
if len(lines) != 1 || !strings.Contains(lines[0], `"vmid"=150001`) || !strings.Contains(lines[0], "does not exist") {
t.Errorf("skip lines = %v, want one naming 150001 and the error", lines)
}
if lines := fx.logs.matching(`"skipped"=1`); len(lines) != 1 {
t.Errorf("listed line with skipped 1 = %v", lines)
}
}
func TestListByTag_unreadableVMWithEndedContextFails(t *testing.T) {
t.Parallel()
fx := newFixture(t, 1)
fx.addOurVM(testVMID, testName, "running")
ctx, cancel := context.WithCancel(fx.ctx)
defer cancel()
fx.fake.hookOn("VMConfig", func() error {
cancel()
return context.Canceled
})
list, err := fx.p.ListByTag(ctx)
if list != nil {
t.Errorf("ListByTag = %+v, want nil", list)
}
assertProviderError(t, err, "list", provider.ErrTransient)
if !errors.Is(err, context.Canceled) {
t.Errorf("error = %v, want it to wrap context.Canceled", err)
}
}
func TestListByTag_unknownStatusStillAttempted(t *testing.T) {
t.Parallel()
fx := newFixture(t, 1)
fx.addOurVM(testVMID, testName, "running").status = "unknown"
if got := listIDs(fx.list()); !slices.Equal(got, []string{fx.id()}) {
t.Errorf("ListByTag = %v, want %s despite status unknown", got, fx.id())
}
}
func TestListByTag_doesNotReadStatusOrAgent(t *testing.T) {
t.Parallel()
fx := newFixture(t, 1)
fx.addOurVM(testVMID, testName, "running").ifaces = capturedIfaces(t)
fx.addOurVM(150001, "proxy-second0000000000", "stopped")
fx.list()
want := []string{"ClusterResources", "VMConfig", "VMConfig"}
if got := fx.fake.callLog(); !slices.Equal(got, want) {
t.Errorf("calls = %v, want %v", got, want)
}
}
func TestListByTag_idsAcceptedByParse(t *testing.T) {
t.Parallel()
fx := newFixture(t, 1)
fx.addOurVM(testVMID, testName, "running")
fx.addOurVM(999999999, "proxy-topofrange000000", "stopped")
fx.addOurVM(100, "proxy-bottomofrange000", "stopped")
list := fx.list()
if len(list) != 3 {
t.Fatalf("ListByTag = %+v, want 3", list)
}
for _, inst := range list {
vmid, name, err := parseProviderID(inst.ID)
if err != nil || formatProviderID(vmid, name) != inst.ID {
t.Errorf("parseProviderID(%q) = %d, %q, %v; want a round trip", inst.ID, vmid, name, err)
}
}
}
func TestListByTag_quietAtV0(t *testing.T) {
t.Parallel()
fx := newFixture(t, 0)
fx.addOurVM(testVMID, testName, "running")
fx.addOurVM(150001, "proxy-destroyedmidswee", "running")
mutateOn(fx.fake, "VMConfig", func() { delete(fx.fake.vms, 150001) })
fx.list()
if lines := fx.logs.all(); len(lines) != 0 {
t.Errorf("V(0) lines = %v, want none", lines)
}
}
+54
View File
@@ -0,0 +1,54 @@
package proxmox
import (
"strings"
"time"
"gitea.home.hrajfrisbee.cz/kacerr/egress-proxies-operator/internal/provider"
)
// createdAtKey is the third line of the mark; the first two use the
// contract's label names.
const createdAtKey = "proxy-operator-created-at"
// instanceMark is the GC mark as read back from a VM's description (spec
// entry 8).
type instanceMark struct {
Managed bool
UID string
CreatedAt time.Time
}
// formatDescription renders the mark written in the clone call.
func formatDescription(uid string, createdAt time.Time) string {
return provider.LabelManaged + "=" + provider.LabelManagedYes + "\n" +
provider.LabelUID + "=" + uid + "\n" +
createdAtKey + "=" + createdAt.UTC().Format(time.RFC3339)
}
// parseDescription reads key=value lines, tolerating CRLF, blank lines and
// lines an admin added. The first occurrence of a key wins. Managed only
// when the value is exactly LabelManagedYes; an unparsable created-at gives
// the zero time, which GC treats as old.
func parseDescription(desc string) instanceMark {
var m instanceMark
seen := map[string]bool{}
for line := range strings.SplitSeq(desc, "\n") {
key, value, ok := strings.Cut(strings.TrimSpace(line), "=")
if !ok || seen[key] {
continue
}
seen[key] = true
switch key {
case provider.LabelManaged:
m.Managed = value == provider.LabelManagedYes
case provider.LabelUID:
m.UID = value
case createdAtKey:
if t, err := time.Parse(time.RFC3339, value); err == nil {
m.CreatedAt = t
}
}
}
return m
}
+64
View File
@@ -0,0 +1,64 @@
package proxmox
import (
"testing"
"time"
)
func TestFormatDescription_keys(t *testing.T) {
t.Parallel()
created := time.Date(2026, 9, 23, 8, 0, 0, 0, time.FixedZone("CEST", 2*3600))
got := formatDescription("3f1c2d7e-uid", created)
want := "proxy-operator-managed=true\nproxy-operator-uid=3f1c2d7e-uid\nproxy-operator-created-at=2026-09-23T06:00:00Z"
if got != want {
t.Errorf("formatDescription() =\n%s\nwant\n%s", got, want)
}
m := parseDescription(got)
if !m.Managed || m.UID != "3f1c2d7e-uid" || !m.CreatedAt.Equal(created) {
t.Errorf("parseDescription(formatDescription()) = %+v, want managed, the UID and the time back", m)
}
}
// TestParseDescription_fromClone reads the mark back from the config of the
// VM cloned in Phase 0, as Proxmox stored it.
func TestParseDescription_fromClone(t *testing.T) {
t.Parallel()
cfg := decodeData[map[string]any](t, "vm_config_clone")
desc, _ := cfg["description"].(string)
m := parseDescription(desc)
want := time.Date(2026, 9, 23, 7, 0, 0, 0, time.UTC)
if !m.Managed || m.UID != "00000000-0000-0000-0000-00000000recon" || !m.CreatedAt.Equal(want) {
t.Errorf("parseDescription(captured) = %+v, want the recon mark", m)
}
}
func TestParseDescription_table(t *testing.T) {
t.Parallel()
ts := time.Date(2026, 9, 23, 7, 0, 0, 0, time.UTC)
tests := []struct {
name string
desc string
want instanceMark
}{
{name: "empty", desc: "", want: instanceMark{}},
{name: "unmarked free text", desc: "Ubuntu VM for testing", want: instanceMark{}},
{name: "managed with UID", desc: "proxy-operator-managed=true\nproxy-operator-uid=u1\nproxy-operator-created-at=2026-09-23T07:00:00Z", want: instanceMark{Managed: true, UID: "u1", CreatedAt: ts}},
{name: "managed with empty UID", desc: "proxy-operator-managed=true\nproxy-operator-uid=", want: instanceMark{Managed: true}},
{name: "managed without UID line", desc: "proxy-operator-managed=true", want: instanceMark{Managed: true}},
{name: "managed value must be exactly true", desc: "proxy-operator-managed=yes\nproxy-operator-uid=u1", want: instanceMark{UID: "u1"}},
{name: "unparsable created-at gives zero time", desc: "proxy-operator-managed=true\nproxy-operator-created-at=yesterday", want: instanceMark{Managed: true}},
{name: "CRLF line endings", desc: "proxy-operator-managed=true\r\nproxy-operator-uid=u1\r\n", want: instanceMark{Managed: true, UID: "u1"}},
{name: "admin lines kept around the mark", desc: "owner: team-crawl\n\nproxy-operator-managed=true\nnote=do not touch\nproxy-operator-uid=u1", want: instanceMark{Managed: true, UID: "u1"}},
{name: "first occurrence of a key wins", desc: "proxy-operator-uid=u1\nproxy-operator-uid=u2\nproxy-operator-managed=true", want: instanceMark{Managed: true, UID: "u1"}},
{name: "surrounding spaces trimmed", desc: " proxy-operator-managed=true \n proxy-operator-uid=u1", want: instanceMark{Managed: true, UID: "u1"}},
}
for _, tc := range tests {
t.Run(tc.name, func(t *testing.T) {
t.Parallel()
got := parseDescription(tc.desc)
if got.Managed != tc.want.Managed || got.UID != tc.want.UID || !got.CreatedAt.Equal(tc.want.CreatedAt) {
t.Errorf("parseDescription(%q) = %+v, want %+v", tc.desc, got, tc.want)
}
})
}
}
+259
View File
@@ -0,0 +1,259 @@
package proxmox
import (
"context"
"errors"
"io"
"strconv"
"time"
logf "sigs.k8s.io/controller-runtime/pkg/log"
)
// observed is the logging decorator of spec entry 20: one V(1)
// "proxmox api call" line per seam call, with request and response detail
// at V(2). It starts no span: the transport already traces each request
// (spec entry 17), so nothing is counted twice.
type observed struct {
inner api
provider string
}
var _ api = (*observed)(nil)
func observe(inner api, providerName string) api {
return &observed{inner: inner, provider: providerName}
}
// target is what a call names at V(1); zero fields are left out.
type target struct {
node string
vmid int
storage string
upid string
extra []any
}
// detail returns V(2)-only key/value pairs. It runs only when V(2) is on.
type detail func() []any
// call wraps one seam call. fn returns the V(2) detail and may fill in
// t.upid from the call's answer.
func (o *observed) call(ctx context.Context, name string, t *target, fn func() (detail, error)) error {
log := logf.FromContext(ctx).WithName("proxmox").WithValues("provider", o.provider)
start := time.Now()
more, err := fn()
kv := make([]any, 0, 16)
kv = append(kv, "call", name)
if t.node != "" {
kv = append(kv, "node", t.node)
}
if t.vmid != 0 {
kv = append(kv, "vmid", t.vmid)
}
if t.storage != "" {
kv = append(kv, "storage", t.storage)
}
if t.upid != "" {
kv = append(kv, "upid", t.upid)
}
kv = append(kv, t.extra...)
kv = append(kv, "duration", time.Since(start))
if err != nil {
var ae *apiError
if errors.As(err, &ae) {
kv = append(kv, "httpStatus", ae.Status)
}
kv = append(kv, "error", err.Error())
}
if more != nil && log.V(2).Enabled() {
kv = append(kv, more()...)
}
log.V(1).Info("proxmox api call", kv...)
return err
}
func (o *observed) Version(ctx context.Context) (v versionInfo, err error) {
err = o.call(ctx, "Version", &target{}, func() (detail, error) {
v, err = o.inner.Version(ctx)
return func() []any { return []any{"version", v.Version} }, err
})
return v, err
}
func (o *observed) NodeStatus(ctx context.Context, node string) (s nodeStatus, err error) {
err = o.call(ctx, "NodeStatus", &target{node: node}, func() (detail, error) {
s, err = o.inner.NodeStatus(ctx, node)
return func() []any { return []any{"pveVersion", s.PVEVersion, "kernel", s.CurrentKernel.Release} }, err
})
return s, err
}
func (o *observed) StorageStatus(ctx context.Context, node, storage string) (s storageStatus, err error) {
err = o.call(ctx, "StorageStatus", &target{node: node, storage: storage}, func() (detail, error) {
s, err = o.inner.StorageStatus(ctx, node, storage)
return func() []any {
return []any{"type", s.Type, "content", s.Content, "active", s.Active, "avail", s.Avail}
}, err
})
return s, err
}
func (o *observed) ClusterResources(ctx context.Context) (rs []vmResource, err error) {
err = o.call(ctx, "ClusterResources", &target{}, func() (detail, error) {
rs, err = o.inner.ClusterResources(ctx)
return func() []any { return []any{"count", len(rs), "vms", resourceSummary(rs)} }, err
})
return rs, err
}
func (o *observed) VMConfig(ctx context.Context, node string, vmid int) (cfg vmConfig, err error) {
err = o.call(ctx, "VMConfig", &target{node: node, vmid: vmid}, func() (detail, error) {
cfg, err = o.inner.VMConfig(ctx, node, vmid)
return func() []any { return []any{"config", configSummary(cfg)} }, err
})
return cfg, err
}
func (o *observed) SetVMConfig(ctx context.Context, node string, vmid int, params map[string]string) error {
return o.call(ctx, "SetVMConfig", &target{node: node, vmid: vmid}, func() (detail, error) {
return func() []any { return []any{"params", params} }, o.inner.SetVMConfig(ctx, node, vmid, params)
})
}
// Clone names the new VM at V(1); the description, which holds the mark,
// only at V(2).
func (o *observed) Clone(ctx context.Context, node string, vmid int, p cloneParams) (upid string, err error) {
t := &target{node: node, vmid: vmid, extra: []any{"newid", p.NewID, "name", p.Name}}
err = o.call(ctx, "Clone", t, func() (detail, error) {
upid, err = o.inner.Clone(ctx, node, vmid, p)
t.upid = upid
return func() []any {
return []any{"description", p.Description, "storage", p.Storage, "target", p.Target}
}, err
})
return upid, err
}
func (o *observed) Resize(ctx context.Context, node string, vmid int, disk, size string) (upid string, err error) {
t := &target{node: node, vmid: vmid}
err = o.call(ctx, "Resize", t, func() (detail, error) {
upid, err = o.inner.Resize(ctx, node, vmid, disk, size)
t.upid = upid
return func() []any { return []any{"disk", disk, "size", size} }, err
})
return upid, err
}
func (o *observed) Start(ctx context.Context, node string, vmid int) (upid string, err error) {
t := &target{node: node, vmid: vmid}
err = o.call(ctx, "Start", t, func() (detail, error) {
upid, err = o.inner.Start(ctx, node, vmid)
t.upid = upid
return nil, err
})
return upid, err
}
func (o *observed) Stop(ctx context.Context, node string, vmid int) (upid string, err error) {
t := &target{node: node, vmid: vmid}
err = o.call(ctx, "Stop", t, func() (detail, error) {
upid, err = o.inner.Stop(ctx, node, vmid)
t.upid = upid
return nil, err
})
return upid, err
}
func (o *observed) VMStatus(ctx context.Context, node string, vmid int) (s vmStatus, err error) {
err = o.call(ctx, "VMStatus", &target{node: node, vmid: vmid}, func() (detail, error) {
s, err = o.inner.VMStatus(ctx, node, vmid)
return func() []any { return []any{"status", s.Status, "qmpstatus", s.QMPStatus, "lock", s.Lock} }, err
})
return s, err
}
func (o *observed) Destroy(ctx context.Context, node string, vmid int) (upid string, err error) {
t := &target{node: node, vmid: vmid}
err = o.call(ctx, "Destroy", t, func() (detail, error) {
upid, err = o.inner.Destroy(ctx, node, vmid)
t.upid = upid
return nil, err
})
return upid, err
}
func (o *observed) TaskStatus(ctx context.Context, node, upid string) (s taskStatus, err error) {
err = o.call(ctx, "TaskStatus", &target{node: node, upid: upid}, func() (detail, error) {
s, err = o.inner.TaskStatus(ctx, node, upid)
return func() []any { return []any{"status", s.Status, "exitstatus", s.ExitStatus} }, err
})
return s, err
}
func (o *observed) AgentNetworkInterfaces(ctx context.Context, node string, vmid int) (ifaces []agentInterface, err error) {
err = o.call(ctx, "AgentNetworkInterfaces", &target{node: node, vmid: vmid}, func() (detail, error) {
ifaces, err = o.inner.AgentNetworkInterfaces(ctx, node, vmid)
return func() []any { return []any{"addresses", agentAddresses(ifaces)} }, err
})
return ifaces, err
}
// UploadISO logs the file name and size, never the body: a seed carries
// CloudInit.
func (o *observed) UploadISO(ctx context.Context, node, storage, filename string, r io.Reader, size int64) (upid string, err error) {
t := &target{node: node, storage: storage, extra: []any{"filename", filename}}
err = o.call(ctx, "UploadISO", t, func() (detail, error) {
upid, err = o.inner.UploadISO(ctx, node, storage, filename, r, size)
t.upid = upid
return func() []any { return []any{"bytes", size} }, err
})
return upid, err
}
func (o *observed) DeleteVolume(ctx context.Context, node, storage, volid string) (upid string, err error) {
t := &target{node: node, storage: storage, extra: []any{"volid", volid}}
err = o.call(ctx, "DeleteVolume", t, func() (detail, error) {
upid, err = o.inner.DeleteVolume(ctx, node, storage, volid)
t.upid = upid
return nil, err
})
return upid, err
}
// resourceSummary is one "<vmid>:<name>:<status>" per entry, with the lock
// appended when set.
func resourceSummary(rs []vmResource) []string {
out := make([]string, len(rs))
for i, r := range rs {
s := strconv.Itoa(r.VMID) + ":" + r.Name + ":" + r.Status
if r.Lock != "" {
s += ":lock=" + r.Lock
}
out[i] = s
}
return out
}
// summaryKeys are the config keys the provider reads or writes.
var summaryKeys = []string{"name", "lock", "tags", "description", "cores", "memory", "agent", "net0", "ide2", "boot", "bootdisk", "scsi0", "virtio0", "sata0", "ide0", "template"}
func configSummary(cfg vmConfig) map[string]string {
out := make(map[string]string, len(summaryKeys))
for _, k := range summaryKeys {
if v, ok := cfg[k]; ok {
out[k] = v
}
}
return out
}
func agentAddresses(ifaces []agentInterface) []string {
var out []string
for _, iface := range ifaces {
for _, a := range iface.IPAddresses {
out = append(out, iface.Name+"="+a.Address)
}
}
return out
}
+229
View File
@@ -0,0 +1,229 @@
package proxmox
import (
"context"
"errors"
"reflect"
"strconv"
"strings"
"sync"
"testing"
"github.com/go-logr/logr/funcr"
sdktrace "go.opentelemetry.io/otel/sdk/trace"
"go.opentelemetry.io/otel/sdk/trace/tracetest"
logf "sigs.k8s.io/controller-runtime/pkg/log"
)
// logCapture collects funcr output, safe for concurrent use.
type logCapture struct {
mu sync.Mutex
lines []string
}
// context returns a ctx whose logger writes into c at verbosity.
func (c *logCapture) context(verbosity int) context.Context {
log := funcr.New(func(prefix, args string) {
c.mu.Lock()
defer c.mu.Unlock()
c.lines = append(c.lines, prefix+" "+args)
}, funcr.Options{Verbosity: verbosity})
return logf.IntoContext(context.Background(), log)
}
func (c *logCapture) all() []string {
c.mu.Lock()
defer c.mu.Unlock()
return append([]string(nil), c.lines...)
}
func (c *logCapture) matching(substr string) []string {
var out []string
for _, l := range c.all() {
if strings.Contains(l, substr) {
out = append(out, l)
}
}
return out
}
const callLine = `"msg"="proxmox api call"`
func newObserveFixture(verbosity int) (*fakeAPI, api, *logCapture, context.Context) {
fake := newFakeAPI()
logs := &logCapture{}
return fake, observe(fake, "proxmox-home"), logs, logs.context(verbosity)
}
func TestObserve_everySeamMethodLogsOneLine(t *testing.T) {
t.Parallel()
for m := range reflect.TypeFor[api]().Methods() {
t.Run(m.Name, func(t *testing.T) {
t.Parallel()
fake, a, logs, ctx := newObserveFixture(1)
args := make([]reflect.Value, m.Type.NumIn())
args[0] = reflect.ValueOf(ctx)
for j := 1; j < m.Type.NumIn(); j++ {
args[j] = reflect.Zero(m.Type.In(j))
}
reflect.ValueOf(a).MethodByName(m.Name).Call(args)
lines := logs.matching(callLine)
if len(lines) != 1 || len(logs.all()) != 1 {
t.Fatalf("got %d call lines of %d, want exactly 1:\n%s", len(lines), len(logs.all()), strings.Join(logs.all(), "\n"))
}
if !strings.HasPrefix(lines[0], "proxmox ") {
t.Errorf("line is not from the proxmox logger: %s", lines[0])
}
for _, want := range []string{`"call"="` + m.Name + `"`, `"provider"="proxmox-home"`, `"duration"=`, `"level"=1`} {
if !strings.Contains(lines[0], want) {
t.Errorf("line lacks %s: %s", want, lines[0])
}
}
if n := fake.callCount(m.Name); n != 1 {
t.Errorf("inner %s called %d times, want 1", m.Name, n)
}
})
}
}
// TestObserve_startsNoSpan keeps the decorator from duplicating the
// transport's HTTP spans (spec entry 17).
func TestObserve_startsNoSpan(t *testing.T) {
t.Parallel()
sr := tracetest.NewSpanRecorder()
tp := sdktrace.NewTracerProvider(sdktrace.WithSpanProcessor(sr))
t.Cleanup(func() { _ = tp.Shutdown(context.Background()) })
fake, a, _, ctx := newObserveFixture(1)
fake.addVM(150042, "proxy-x", "running")
ctx, parent := tp.Tracer("test").Start(ctx, "provider.get")
_, _ = a.ClusterResources(ctx)
_, _ = a.VMConfig(ctx, fakeNode, 150042)
parent.End()
if ended := sr.Ended(); len(ended) != 1 || ended[0].Name() != "provider.get" {
t.Errorf("spans = %d, want only the parent", len(ended))
}
}
func TestObserve_identifiersAtV1(t *testing.T) {
t.Parallel()
fake, a, logs, ctx := newObserveFixture(1)
fake.addTemplate(9001, "debian-13-template")
upid, err := a.Clone(ctx, fakeNode, 9001, cloneParams{NewID: 150042, Name: "proxy-x", Description: "proxy-operator-managed=true"})
if err != nil {
t.Fatalf("Clone error = %v", err)
}
_, _ = a.TaskStatus(ctx, fakeNode, upid)
_, _ = a.DeleteVolume(ctx, fakeNode, fakeISOStorage, seedVolID(fakeISOStorage, "proxy-x"))
lines := logs.matching(callLine)
if len(lines) != 3 {
t.Fatalf("got %d lines, want 3:\n%s", len(lines), strings.Join(logs.all(), "\n"))
}
wants := [][]string{
{`"call"="Clone" "node"="pv1-1" "vmid"=9001 "upid"="` + upid + `" "newid"=150042 "name"="proxy-x"`},
{`"call"="TaskStatus" "node"="pv1-1" "upid"="` + upid + `"`},
{`"call"="DeleteVolume" "node"="pv1-1" "storage"="proxy-seeds"`, `"volid"="proxy-seeds:iso/proxy-x-seed.iso"`, `"upid"="UPID:pv1-1:`},
}
for i, want := range wants {
for _, w := range want {
if !strings.Contains(lines[i], w) {
t.Errorf("line %d lacks %s:\n%s", i, w, lines[i])
}
}
}
if strings.Contains(lines[0], "description") {
t.Errorf("the description reached V(1): %s", lines[0])
}
}
func TestObserve_detailOnlyAtV2(t *testing.T) {
t.Parallel()
tests := []struct {
name string
verbosity int
want bool
}{
{name: "V1 has no detail", verbosity: 1, want: false},
{name: "V2 has detail", verbosity: 2, want: true},
}
for _, tc := range tests {
t.Run(tc.name, func(t *testing.T) {
t.Parallel()
fake, a, logs, ctx := newObserveFixture(tc.verbosity)
fake.addTemplate(9001, "debian-13-template")
vm := fake.addVM(150042, "proxy-x", "running")
vm.ifaces = []agentInterface{{Name: "ens18", IPAddresses: []agentIP{{Address: "192.168.4.147", Type: "ipv4"}}}}
_, _ = a.ClusterResources(ctx)
_, _ = a.VMConfig(ctx, fakeNode, 150042)
_ = a.SetVMConfig(ctx, fakeNode, 9001, map[string]string{"cores": "3"})
_, _ = a.Clone(ctx, fakeNode, 9001, cloneParams{NewID: 150043, Name: "proxy-y", Description: "mark-line"})
_, _ = a.AgentNetworkInterfaces(ctx, fakeNode, 150042)
all := strings.Join(logs.all(), "\n")
for _, detail := range []string{`"vms"=["9001:debian-13-template:stopped" "150042:proxy-x:running"]`, `"config"={`, `"params"={"cores"="3"}`, `"description"="mark-line"`, `"addresses"=["ens18=192.168.4.147"]`} {
if got := strings.Contains(all, detail); got != tc.want {
t.Errorf("%s logged = %v, want %v:\n%s", detail, got, tc.want, all)
}
}
if got := len(logs.matching(callLine)); got != 5 {
t.Errorf("got %d call lines, want 5", got)
}
})
}
}
func TestObserve_uploadBodyNeverLogged(t *testing.T) {
t.Parallel()
const sentinel = "SENTINEL-CLOUDINIT-51c9"
fake, a, logs, ctx := newObserveFixture(2)
body := "#cloud-config\npassword: " + sentinel + "\n"
if _, err := a.UploadISO(ctx, fakeNode, fakeISOStorage, "proxy-x-seed.iso", strings.NewReader(body), int64(len(body))); err != nil {
t.Fatalf("UploadISO error = %v", err)
}
if got, _ := fake.iso(fakeISOStorage, "proxy-x-seed.iso"); string(got) != body {
t.Errorf("fake got %q, want the body passed through", got)
}
all := strings.Join(logs.all(), "\n")
if strings.Contains(all, sentinel) {
t.Errorf("upload body leaked into logs:\n%s", all)
}
lines := logs.matching(`"call"="UploadISO"`)
if len(lines) != 1 || !strings.Contains(lines[0], `"filename"="proxy-x-seed.iso"`) || !strings.Contains(lines[0], `"bytes"=`+strconv.Itoa(len(body))) {
t.Errorf("upload line lacks filename or bytes: %v", lines)
}
}
func TestObserve_errorFieldsOnFailure(t *testing.T) {
t.Parallel()
fake, a, logs, ctx := newObserveFixture(1)
_, err := a.VMConfig(ctx, fakeNode, 999999)
if !isNotExist(err) {
t.Fatalf("error = %v, want the does-not-exist apiError through the decorator", err)
}
lines := logs.matching(callLine)
if len(lines) != 1 {
t.Fatalf("got %d lines, want 1", len(lines))
}
for _, want := range []string{`"httpStatus"=500`, `"error"="proxmox API 500: Configuration file 'nodes/pv1-1/qemu-server/999999.conf' does not exist"`} {
if !strings.Contains(lines[0], want) {
t.Errorf("line lacks %s: %s", want, lines[0])
}
}
fake.failWith("Version", errors.New("dial tcp: connection refused"))
_, _ = a.Version(ctx)
lines = logs.matching(`"call"="Version"`)
if len(lines) != 1 || strings.Contains(lines[0], "httpStatus") || !strings.Contains(lines[0], `"error"="dial tcp: connection refused"`) {
t.Errorf("non-API failure line = %v, want the error without httpStatus", lines)
}
}
func TestObserve_v0Silent(t *testing.T) {
t.Parallel()
fake, a, logs, ctx := newObserveFixture(0)
fake.addVM(150042, "proxy-x", "running")
_, _ = a.VMConfig(ctx, fakeNode, 150042)
_, _ = a.VMConfig(ctx, fakeNode, 999999)
if lines := logs.all(); len(lines) != 0 {
t.Errorf("got %d lines at V(0), want none:\n%s", len(lines), strings.Join(lines, "\n"))
}
}
+691
View File
@@ -0,0 +1,691 @@
package proxmox
import (
"context"
"errors"
"net/http"
"path/filepath"
"slices"
"strconv"
"strings"
"testing"
"time"
sdktrace "go.opentelemetry.io/otel/sdk/trace"
"go.opentelemetry.io/otel/sdk/trace/tracetest"
"gitea.home.hrajfrisbee.cz/kacerr/egress-proxies-operator/internal/provider"
"gitea.home.hrajfrisbee.cz/kacerr/egress-proxies-operator/internal/tracing"
)
const (
testProvider = "proxmox-home"
testUID = "0b9c6f3e-2d4a-4e1b-9c7f-5a8d3e2f1b60"
testVMID = 150042
)
var (
testName = provider.NameFromUID(testUID)
testCreatedAt = time.Date(2026, 9, 23, 10, 0, 0, 0, time.UTC)
)
// fixture is a Provider over fakeAPI, logging into a capture, with every
// budget shrunk so waits take milliseconds.
type fixture struct {
t *testing.T
p *Provider
fake *fakeAPI
logs *logCapture
ctx context.Context
}
func newFixture(t *testing.T, verbosity int) *fixture {
t.Helper()
fx := &fixture{t: t, fake: newFakeAPI(), logs: &logCapture{}}
fx.p = newProvider(testProvider, testConfig(), fx.fake)
fx.p.budgets = testBudgets()
fx.ctx = fx.logs.context(verbosity)
return fx
}
func testBudgets() budgets {
return budgets{
clone: time.Second,
upload: time.Second,
resize: time.Second,
start: time.Second,
stop: time.Second,
destroy: time.Second,
seedDelete: time.Second,
running: time.Second,
poll: time.Millisecond,
}
}
// addOurVM places a VM carrying the mark for testUID and the tag, as
// Create leaves it.
func (fx *fixture) addOurVM(vmid int, name, status string) *fakeVM {
fx.t.Helper()
vm := fx.fake.addVM(vmid, name, status)
vm.config["description"] = formatDescription(testUID, testCreatedAt)
vm.config["tags"] = proxyTag
return vm
}
// id is the providerID of the VM addOurVM(testVMID, testName, …) places.
func (fx *fixture) id() string { return formatProviderID(testVMID, testName) }
func (fx *fixture) get(id string) *provider.Instance {
fx.t.Helper()
inst, err := fx.p.Get(fx.ctx, id)
if err != nil {
fx.t.Fatalf("Get(%s) error = %v", id, err)
}
return inst
}
// assertProviderError checks that err is a *provider.Error from this
// provider for op, in class and in no other class.
func assertProviderError(t *testing.T, err error, op string, class error) {
t.Helper()
if err == nil {
t.Fatalf("error = nil, want op %s class %v", op, class)
}
var pe *provider.Error
if !errors.As(err, &pe) {
t.Fatalf("error = %T (%v), want *provider.Error", err, err)
}
if pe.Provider != testProvider {
t.Errorf("error.Provider = %q, want %q", pe.Provider, testProvider)
}
if pe.Op != op {
t.Errorf("error.Op = %q, want %q", pe.Op, op)
}
if !errors.Is(err, class) {
t.Errorf("error = %v, want class %v", err, class)
}
for _, other := range []error{provider.ErrNotFound, provider.ErrQuotaExceeded, provider.ErrPermanent, provider.ErrTransient} {
if other != class && errors.Is(err, other) {
t.Errorf("error = %v, also matches %v", err, other)
}
}
}
// within runs fn and fails the test when it has not returned after d, so
// a call that ignores its ctx fails by name instead of hanging the package
// until the test binary's timeout.
func within(t *testing.T, d time.Duration, fn func()) {
t.Helper()
done := make(chan struct{})
go func() {
defer close(done)
fn()
}()
select {
case <-done:
case <-time.After(d):
t.Fatalf("did not return within %v", d)
}
}
// cancelWhileBlocked runs call with a ctx it cancels once call has reached
// method, which blocks until then, and fails unless call returns within a
// second of the cancel. Every wait is bounded, so a call that never reaches
// method or never returns fails instead of hanging the package.
func cancelWhileBlocked(t *testing.T, fx *fixture, method string, call func(ctx context.Context)) {
t.Helper()
release := fx.fake.blockOn(method)
defer close(release)
ctx, cancel := context.WithCancel(fx.ctx)
defer cancel()
returned := make(chan struct{})
go func() {
defer close(returned)
call(ctx)
}()
deadline := time.After(5 * time.Second)
for fx.fake.callCount(method) == 0 {
select {
case <-returned:
t.Fatalf("returned before calling %s; calls: %v", method, fx.fake.callLog())
case <-deadline:
t.Fatalf("%s was not called within 5s; calls: %v", method, fx.fake.callLog())
case <-time.After(time.Millisecond):
}
}
cancel()
select {
case <-returned:
case <-time.After(time.Second):
t.Fatalf("did not return within 1s of the cancel while blocked in %s", method)
}
}
func TestProvider_cancelledContextReturnsPromptly(t *testing.T) {
t.Parallel()
tests := []struct {
name string
blockOn string
// first: blockOn is the operation's first seam call.
first bool
setup func(fx *fixture)
op string
call func(ctx context.Context, fx *fixture) error
}{
{
name: "create", blockOn: "ClusterResources", first: true, op: "create",
call: func(ctx context.Context, fx *fixture) error { _, err := fx.p.Create(ctx, fx.req()); return err },
},
{
name: "create mid upload", blockOn: "UploadISO", op: "create",
call: func(ctx context.Context, fx *fixture) error { _, err := fx.p.Create(ctx, fx.req()); return err },
},
{
name: "get", blockOn: "ClusterResources", first: true, op: "get",
setup: func(fx *fixture) { fx.addOurVM(testVMID, testName, "running") },
call: func(ctx context.Context, fx *fixture) error { _, err := fx.p.Get(ctx, fx.id()); return err },
},
{
name: "delete", blockOn: "ClusterResources", first: true, op: "delete",
setup: func(fx *fixture) { fx.addOurVM(testVMID, testName, "running") },
call: func(ctx context.Context, fx *fixture) error { return fx.p.Delete(ctx, fx.id()) },
},
{
name: "delete mid stop", blockOn: "Stop", op: "delete",
setup: func(fx *fixture) { fx.addOurVM(testVMID, testName, "running") },
call: func(ctx context.Context, fx *fixture) error { return fx.p.Delete(ctx, fx.id()) },
},
{
name: "list", blockOn: "ClusterResources", first: true, op: "list",
setup: func(fx *fixture) { fx.addOurVM(testVMID, testName, "running") },
call: func(ctx context.Context, fx *fixture) error { _, err := fx.p.ListByTag(ctx); return err },
},
}
for _, tc := range tests {
t.Run(tc.name, func(t *testing.T) {
t.Parallel()
fx := newCreateFixture(t, 1)
if tc.setup != nil {
tc.setup(fx)
}
var err error
cancelWhileBlocked(t, fx, tc.blockOn, func(ctx context.Context) { err = tc.call(ctx, fx) })
if calls := fx.fake.callLog(); tc.first && calls[0] != tc.blockOn {
t.Errorf("first call = %s, want %s", calls[0], tc.blockOn)
}
assertProviderError(t, err, tc.op, provider.ErrTransient)
if !errors.Is(err, context.Canceled) {
t.Errorf("error = %v, want it to carry context.Canceled", err)
}
})
}
t.Run("check", func(t *testing.T) {
t.Parallel()
fx := newCheckFixture(t)
var got []provider.TargetStatus
cancelWhileBlocked(t, fx, "Version", func(ctx context.Context) { got = fx.p.Check(ctx) })
if calls := fx.fake.callLog(); calls[0] != "Version" {
t.Errorf("first call = %s, want Version", calls[0])
}
if len(got) != 1 || got[0].Ready || got[0].Reason != provider.ReasonTimeout {
t.Errorf("Check() = %+v, want one target not ready with reason %s", got, provider.ReasonTimeout)
}
})
}
func TestProvider_everyErrorIsWrappedWithConfiguredName(t *testing.T) {
t.Parallel()
createWith := func(mutate func(*provider.CreateRequest)) func(context.Context, *fixture) error {
return func(ctx context.Context, fx *fixture) error {
req := fx.req()
if mutate != nil {
mutate(&req)
}
_, err := fx.p.Create(ctx, req)
return err
}
}
get := func(ctx context.Context, fx *fixture) error { _, err := fx.p.Get(ctx, fx.id()); return err }
del := func(ctx context.Context, fx *fixture) error { return fx.p.Delete(ctx, fx.id()) }
list := func(ctx context.Context, fx *fixture) error { _, err := fx.p.ListByTag(ctx); return err }
holdNameLock := func(fx *fixture, cancel context.CancelFunc) {
release, err := fx.p.nameLocks.Acquire(context.Background(), testName)
if err != nil {
fx.t.Fatalf("Acquire error = %v", err)
}
fx.t.Cleanup(release)
cancel()
}
broken := fakeErr(596, "Broken pipe")
tests := []struct {
name string
setup func(fx *fixture, cancel context.CancelFunc)
call func(ctx context.Context, fx *fixture) error
op string
id string
class error
}{
{name: "create without a name", call: createWith(func(r *provider.CreateRequest) { r.Name = "" }), op: "create", class: provider.ErrPermanent},
{name: "create unknown zone", call: createWith(func(r *provider.CreateRequest) { r.Placement.Zone = "pv9-9" }), op: "create", class: provider.ErrPermanent},
{name: "create unknown image", call: createWith(func(r *provider.CreateRequest) { r.Placement.Image = "arch" }), op: "create", class: provider.ErrPermanent},
{name: "create waiting on the name lock", setup: holdNameLock, call: createWith(nil), op: "create", class: provider.ErrTransient},
{
name: "create resources unreadable",
setup: func(fx *fixture, _ context.CancelFunc) { fx.fake.failWith("ClusterResources", broken) },
call: createWith(nil), op: "create", class: provider.ErrTransient,
},
{
name: "create clone forbidden",
setup: func(fx *fixture, _ context.CancelFunc) {
fx.fake.failWith("Clone", loadAPIError(fx.t, "err_forbidden"))
},
call: createWith(nil), op: "create", class: provider.ErrPermanent,
},
{
name: "create start task fails",
setup: func(fx *fixture, _ context.CancelFunc) { fx.fake.startFailsWith = "fake: start failed" },
call: createWith(nil), op: "create", class: provider.ErrTransient,
},
{
name: "get malformed id",
call: func(ctx context.Context, fx *fixture) error { _, err := fx.p.Get(ctx, "vms/abc/"+testName); return err },
op: "get", id: "vms/abc/" + testName, class: provider.ErrPermanent,
},
{name: "get missing vm", call: get, op: "get", id: "self", class: provider.ErrNotFound},
{
name: "get locked vm",
setup: func(fx *fixture, _ context.CancelFunc) { fx.addOurVM(testVMID, testName, "running").lock = "backup" },
call: get, op: "get", id: "self", class: provider.ErrTransient,
},
{
name: "get unauthorized",
setup: func(fx *fixture, _ context.CancelFunc) {
fx.fake.failWith("ClusterResources", loadAPIError(fx.t, "err_unauthorized"))
},
call: get, op: "get", id: "self", class: provider.ErrPermanent,
},
{
name: "delete malformed id",
call: func(ctx context.Context, fx *fixture) error { return fx.p.Delete(ctx, "vms/150042") },
op: "delete", id: "vms/150042", class: provider.ErrPermanent,
},
{name: "delete waiting on the name lock", setup: holdNameLock, call: del, op: "delete", id: "self", class: provider.ErrTransient},
{
name: "delete destroy task fails",
setup: func(fx *fixture, _ context.CancelFunc) {
fx.addOurVM(testVMID, testName, "stopped")
fx.fake.destroyFailsWith = "fake: destroy failed"
},
call: del, op: "delete", id: "self", class: provider.ErrTransient,
},
{
name: "delete seed forbidden",
setup: func(fx *fixture, _ context.CancelFunc) {
fx.addOurVM(testVMID, testName, "stopped")
fx.fake.failWith("DeleteVolume", loadAPIError(fx.t, "err_forbidden_volume_delete"))
},
call: del, op: "delete", id: "self", class: provider.ErrPermanent,
},
{
name: "list resources unreadable",
setup: func(fx *fixture, _ context.CancelFunc) { fx.fake.failWith("ClusterResources", broken) },
call: list, op: "list", class: provider.ErrTransient,
},
{
name: "list unauthorized",
setup: func(fx *fixture, _ context.CancelFunc) {
fx.fake.failWith("ClusterResources", loadAPIError(fx.t, "err_unauthorized"))
},
call: list, op: "list", class: provider.ErrPermanent,
},
{
name: "list ended on a candidate",
setup: func(fx *fixture, cancel context.CancelFunc) {
fx.addOurVM(testVMID, testName, "running")
fx.fake.hookOn("VMConfig", func() error {
cancel()
return broken
})
},
call: list, op: "list", class: provider.ErrTransient,
},
}
for _, tc := range tests {
t.Run(tc.name, func(t *testing.T) {
t.Parallel()
fx := newCreateFixture(t, 1)
ctx, cancel := context.WithCancel(fx.ctx)
defer cancel()
if tc.setup != nil {
tc.setup(fx, cancel)
}
var err error
within(t, 5*time.Second, func() { err = tc.call(ctx, fx) })
assertProviderError(t, err, tc.op, tc.class)
wantID := tc.id
if wantID == "self" {
wantID = fx.id()
}
var pe *provider.Error
if errors.As(err, &pe) && pe.ID != wantID {
t.Errorf("error.ID = %q, want %q", pe.ID, wantID)
}
if !strings.HasPrefix(err.Error(), "provider "+testProvider+": "+tc.op) {
t.Errorf("error = %q, want it to start with the configured name and op", err)
}
})
}
}
// logSentinel is in the CloudInit the logging lifecycle creates with; no
// log line at any level may carry it.
const logSentinel = "SENTINEL-cloud-init-7e2a"
// newLoggingFixture is an observed create fixture with a token file, so
// one lifecycle covers every operation, Check and the decorator's lines.
func newLoggingFixture(t *testing.T, verbosity int) *fixture {
t.Helper()
fx := newCreateFixture(t, verbosity)
fx.p.api = observe(fx.fake, testProvider)
fx.p.cfg.TokenFile = filepath.Join(t.TempDir(), "token")
writeFile(t, fx.p.cfg.TokenFile, testToken+"\n")
return fx
}
// sentinelReq is fx.req() with logSentinel in its CloudInit.
func (fx *fixture) sentinelReq() provider.CreateRequest {
req := fx.req()
req.CloudInit = "#cloud-config\nwrite_files:\n - path: /etc/sentinel\n content: " + logSentinel + "\n"
return req
}
// lifecycle runs Create with sentinelReq, a Get that finds the address,
// ListByTag, Delete, Delete of the gone VM, and Check.
func (fx *fixture) lifecycle() {
fx.t.Helper()
id := fx.create(fx.sentinelReq())
ifaces := capturedIfaces(fx.t)
fx.fake.mu.Lock()
fx.fake.vms[slot(0)].ifaces = ifaces
fx.fake.mu.Unlock()
if inst := fx.get(id); inst.State != provider.StateRunning {
fx.t.Fatalf("Get state = %s, want Running", inst.State)
}
if _, err := fx.p.ListByTag(fx.ctx); err != nil {
fx.t.Fatalf("ListByTag error = %v", err)
}
for range 2 {
if err := fx.p.Delete(fx.ctx, id); err != nil {
fx.t.Fatalf("Delete error = %v", err)
}
}
if ts := fx.p.Check(fx.ctx); len(ts) != 1 || !ts[0].Ready {
fx.t.Fatalf("Check = %+v, want one Ready target", ts)
}
}
// assertNeverLogged checks every line for the cloud-init text and the
// token's secret.
func assertNeverLogged(t *testing.T, lines []string) {
t.Helper()
secret := testToken[strings.Index(testToken, "=")+1:]
for _, l := range lines {
for _, never := range []string{logSentinel, "#cloud-config", secret} {
if strings.Contains(l, never) {
t.Errorf("line carries %q: %s", never, l)
}
}
}
}
func msgLine(msg string) string { return `"msg"="` + msg + `"` }
// assertEveryV1Line checks that each line is from the proxmox logger at
// V(1) with the provider, and that each line about an instance, which is
// all but the decorator's, the list's and the check's, names its vmid and
// name.
func assertEveryV1Line(t *testing.T, lines []string, vmid, name string) {
t.Helper()
notInstance := []string{msgLine("proxmox api call"), msgLine("proxmox vms listed"), msgLine("proxmox node checked")}
for _, l := range lines {
if !strings.HasPrefix(l, "proxmox ") || !strings.Contains(l, `"provider"="`+testProvider+`"`) {
t.Errorf("line lacks the proxmox logger or the provider: %s", l)
}
if !strings.Contains(l, `"level"=1`) {
t.Errorf("line is not at V(1): %s", l)
}
instance := !slices.ContainsFunc(notInstance, func(m string) bool { return strings.Contains(l, m) })
if instance && (!strings.Contains(l, vmid) || !strings.Contains(l, name)) {
t.Errorf("instance line lacks vmid or name: %s", l)
}
}
}
func TestLogging_v0SilentV1OutcomeV2Detail(t *testing.T) {
t.Parallel()
vmid := `"vmid"=` + strconv.Itoa(slot(0))
name := `"name"="` + testName + `"`
t.Run("v0 is silent", func(t *testing.T) {
t.Parallel()
fx := newLoggingFixture(t, 0)
fx.lifecycle()
if lines := fx.logs.all(); len(lines) != 0 {
t.Errorf("the lifecycle logged %d lines at V(0), want none:\n%s", len(lines), strings.Join(lines, "\n"))
}
})
t.Run("v1 has one outcome per operation", func(t *testing.T) {
t.Parallel()
fx := newLoggingFixture(t, 1)
fx.lifecycle()
all := fx.logs.all()
dump := strings.Join(all, "\n")
outcomes := map[string][]string{
"proxmox vm ensured running": {vmid, name, `"node"="pv1-1"`},
"proxmox vm fetched": {vmid, name, `"state"="Running"`, `"ip"="192.168.0.150"`, `"uid"="` + testUID + `"`},
"proxmox vms listed": {`"candidates"=1`, `"managed"=1`, `"skipped"=0`},
"proxmox node checked": {`"node"="pv1-1"`, `"ready"=true`, `"reason"="Reachable"`},
}
for msg, keys := range outcomes {
lines := fx.logs.matching(msgLine(msg))
if len(lines) != 1 {
t.Errorf("lines %q = %d, want 1:\n%s", msg, len(lines), dump)
continue
}
for _, k := range keys {
if !strings.Contains(lines[0], k) {
t.Errorf("%q line lacks %s: %s", msg, k, lines[0])
}
}
}
deleted := fx.logs.matching(msgLine("proxmox vm deleted"))
if len(deleted) != 2 {
t.Fatalf("deleted lines = %d, want 2 (destroyed, then already gone):\n%s", len(deleted), dump)
}
if l := deleted[0]; !strings.Contains(l, `"destroyed"=true`) || !strings.Contains(l, `"node"="pv1-1"`) || !strings.Contains(l, vmid) || !strings.Contains(l, name) {
t.Errorf("first deleted line = %s, want destroyed=true with node, vmid and name", l)
}
if l := deleted[1]; !strings.Contains(l, `"destroyed"=false`) || strings.Contains(l, `"node"=`) || !strings.Contains(l, vmid) || !strings.Contains(l, name) {
t.Errorf("second deleted line = %s, want destroyed=false with vmid and name and no node", l)
}
assertEveryV1Line(t, all, vmid, name)
for _, v2 := range []string{"cloudInitBytes", "proxmox create resolved", "proxmox guest agent answered", "proxmox listed instance", `"description"=`, `"config"=`, `"params"=`} {
if got := fx.logs.matching(v2); len(got) != 0 {
t.Errorf("V(1) carries the V(2) detail %s:\n%s", v2, strings.Join(got, "\n"))
}
}
assertNeverLogged(t, all)
})
t.Run("v2 has detail and never the cloud-init", func(t *testing.T) {
t.Parallel()
fx := newLoggingFixture(t, 2)
fx.lifecycle()
resolved := fx.logs.matching(msgLine("proxmox create resolved"))
if len(resolved) != 1 {
t.Fatalf("resolved lines = %d, want 1", len(resolved))
}
for _, k := range []string{name, `"cloudInitBytes"=` + strconv.Itoa(len(fx.sentinelReq().CloudInit)), `"machineType"="small"`, `"cores"=1`, `"memoryMiB"=1024`, `"templateVMID"=9001`, `"isoStorage"="proxy-seeds"`} {
if !strings.Contains(resolved[0], k) {
t.Errorf("resolved line lacks %s: %s", k, resolved[0])
}
}
details := map[string]string{
msgLine("proxmox guest agent answered"): `"ip"="192.168.0.150"`,
msgLine("proxmox listed instance"): `"uid"="` + testUID + `"`,
`"call"="Clone"`: `"description"="proxy-operator-managed=true`,
`"call"="VMConfig"`: `"config"=`,
`"call"="UploadISO"`: `"bytes"=`,
}
for line, k := range details {
got := fx.logs.matching(line)
if len(got) == 0 || !strings.Contains(got[0], k) {
t.Errorf("V(2) %s line lacks %s: %v", line, k, got)
}
}
assertNeverLogged(t, fx.logs.all())
})
}
// TestLogging_noOpLinesAtV1: spec entry 20's no-op lines of Delete and
// ListByTag are there at V(1), each naming the instance.
func TestLogging_noOpLinesAtV1(t *testing.T) {
t.Parallel()
del := func(fx *fixture) error { return fx.p.Delete(fx.ctx, fx.id()) }
tests := []struct {
name string
setup func(fx *fixture)
call func(fx *fixture) error
msg string
want int
}{
{name: "vm already gone", call: del, msg: "proxmox vm already gone", want: 1},
{
name: "foreign vm",
setup: func(fx *fixture) { fx.addOurVM(testVMID, "proxy-someoneelse00000", "running") },
call: del, msg: "proxmox vm at this vmid is not ours, left alone", want: 1,
},
{
name: "unmarked vm",
setup: func(fx *fixture) { fx.fake.addVM(testVMID, testName, "running") },
call: del, msg: "proxmox vm carries no proxy-operator mark, left alone", want: 1,
},
{
name: "seed delete skipped on each node",
setup: func(fx *fixture) {
fx.p.cfg.Nodes = []string{fakeNode, "pv1-2"}
fx.fake.failWith("DeleteVolume", fakeErr(500, "fake: storage 'proxy-seeds' is not available on this node"))
},
call: del, msg: "proxmox seed delete failed on a node, skipped", want: 2,
},
{
name: "unreadable candidate",
setup: func(fx *fixture) {
fx.addOurVM(testVMID, testName, "running")
fx.fake.failWith("VMConfig", fakeErr(596, "Broken pipe"))
},
call: func(fx *fixture) error { _, err := fx.p.ListByTag(fx.ctx); return err },
msg: "skipping unreadable proxmox vm", want: 1,
},
}
for _, tc := range tests {
t.Run(tc.name, func(t *testing.T) {
t.Parallel()
fx := newFixture(t, 1)
if tc.setup != nil {
tc.setup(fx)
}
if err := tc.call(fx); err != nil {
t.Fatalf("error = %v, want nil", err)
}
lines := fx.logs.matching(msgLine(tc.msg))
if len(lines) != tc.want {
t.Fatalf("lines %q = %d, want %d:\n%s", tc.msg, len(lines), tc.want, strings.Join(fx.logs.all(), "\n"))
}
for _, l := range lines {
for _, k := range []string{`"provider"="` + testProvider + `"`, `"vmid"=150042`, `"level"=1`} {
if !strings.Contains(l, k) {
t.Errorf("line lacks %s: %s", k, l)
}
}
}
})
}
}
// TestTracing_httpSpansUnderOperationSpan runs a Get through
// provider.WithTracing over the real client against Phase 0's captures:
// each of Get's four requests is an HTTP span under provider.get, and no
// span attribute carries the token.
func TestTracing_httpSpansUnderOperationSpan(t *testing.T) {
t.Parallel()
const (
vmid = 199999
name = "proxy-reconaaaaaaaaaaa"
)
cfx := newClientFixture(t, func(w http.ResponseWriter, r *http.Request) {
path := strings.TrimPrefix(r.URL.Path, "/api2/json/")
capture := map[string]string{
"cluster/resources": "cluster_resources_during_destroy",
"nodes/pv1-1/qemu/199999/config": "config_during_destroy",
"nodes/pv1-1/qemu/199999/status/current": "status_current_running",
"nodes/pv1-1/qemu/199999/agent/network-get-interfaces": "agent_network_get_interfaces",
}[path]
if r.Method != http.MethodGet || capture == "" {
t.Errorf("unexpected request %s %s", r.Method, path)
http.NotFound(w, r)
return
}
serveJSON(w, readCapture(t, capture).Body)
})
sr := tracetest.NewSpanRecorder()
tp := sdktrace.NewTracerProvider(sdktrace.WithSpanProcessor(sr))
t.Cleanup(func() { _ = tp.Shutdown(context.Background()) })
opt := tracing.WithTracerProvider(tp)
cfg := testConfig()
cfg.URL, cfg.TokenFile, cfg.CAFile = cfx.cfg.URL, cfx.cfg.TokenFile, cfx.cfg.CAFile
p := newProvider(testProvider, cfg, observe(cfx.api(t, opt), testProvider))
p.budgets = testBudgets()
traced := provider.WithTracing(testProvider, p, opt)
inst, err := traced.Get(context.Background(), formatProviderID(vmid, name))
if err != nil {
t.Fatalf("Get error = %v", err)
}
if inst.State != provider.StateRunning || inst.IP != "192.168.0.150" {
t.Errorf("Get = %+v, want Running at 192.168.0.150", inst)
}
var (
op sdktrace.ReadOnlySpan
requests []sdktrace.ReadOnlySpan
)
for _, s := range sr.Ended() {
if s.Name() == "provider.get" {
op = s
} else {
requests = append(requests, s)
}
}
if op == nil {
t.Fatalf("no provider.get span among %d", len(sr.Ended()))
}
if len(requests) != 4 {
t.Errorf("HTTP spans = %d, want one per request of Get (4)", len(requests))
}
for _, s := range requests {
if s.Name() != "HTTP GET" {
t.Errorf("span %q, want HTTP GET (spec entry 17)", s.Name())
}
if s.Parent().SpanID() != op.SpanContext().SpanID() || s.SpanContext().TraceID() != op.SpanContext().TraceID() {
t.Errorf("span %q is not a child of provider.get", s.Name())
}
}
secret := testToken[strings.Index(testToken, "=")+1:]
for _, s := range sr.Ended() {
for _, kv := range s.Attributes() {
if v := kv.Value.Emit(); strings.Contains(v, secret) || strings.Contains(v, "PVEAPIToken") {
t.Errorf("span %s attribute %s carries the token: %s", s.Name(), kv.Key, v)
}
}
}
}
+174
View File
@@ -0,0 +1,174 @@
// Package proxmox implements the provider contract on a Proxmox VE cluster
// through its REST API: VMs are full clones of prepared templates, seeded
// with a NoCloud ISO uploaded to a storage and addressed through the guest
// agent. The design is in docs/proxmox-provider-spec.md.
package proxmox
import (
"context"
"fmt"
"os"
"time"
"github.com/go-logr/logr"
logf "sigs.k8s.io/controller-runtime/pkg/log"
"gitea.home.hrajfrisbee.cz/kacerr/egress-proxies-operator/internal/provider"
"gitea.home.hrajfrisbee.cz/kacerr/egress-proxies-operator/internal/provider/keyedlock"
)
const (
defaultVMIDMin = 100000
defaultVMIDMax = 199999
defaultRequestTimeoutSeconds = 30
)
// budgets bounds every wait of the provider (spec entry 18); tests shrink
// them.
type budgets struct {
clone time.Duration // the clone task, and waiting for a VM's lock
upload time.Duration
resize time.Duration
start time.Duration
stop time.Duration
destroy time.Duration
seedDelete time.Duration
running time.Duration // from the start task's end to status running
poll time.Duration // task and status poll interval
}
func defaultBudgets() budgets {
return budgets{
clone: 10 * time.Minute,
upload: 2 * time.Minute,
resize: 2 * time.Minute,
start: 2 * time.Minute,
stop: 2 * time.Minute,
destroy: 5 * time.Minute,
seedDelete: time.Minute,
running: 60 * time.Second,
poll: time.Second,
}
}
// Provider is one configured proxmox entry: one cluster reached through one
// API URL with one token.
type Provider struct {
name string
cfg provider.ProxmoxConfig
api api
now func() time.Time
budgets budgets
// nameLocks serialises Create and Delete per instance name (spec entry
// 19), so the reconciler's Delete and GC's Delete never race on stop and
// destroy, and a Delete cannot interleave with a Create's steps.
nameLocks *keyedlock.Locks
}
var _ provider.Provider = (*Provider)(nil)
// New builds a Provider from pc.Proxmox. It never makes a request: the token
// file is only checked for existence, and a missing one is a warning, not an
// error, because the Secret mount is optional and may land later.
func New(ctx context.Context, pc provider.ProviderConfig) (provider.Provider, error) {
cfg := provider.ProxmoxConfig{}
if pc.Proxmox != nil {
cfg = *pc.Proxmox
}
client, err := newHTTPAPI(withDefaults(cfg))
if err != nil {
return nil, fmt.Errorf("proxmox provider %s: %w", pc.Name, err)
}
p := newProvider(pc.Name, cfg, observe(client, pc.Name))
log := p.logger(ctx)
if _, err := os.Stat(p.cfg.TokenFile); err != nil {
log.Info("WARNING: token file is not readable; every call fails until it is", "path", p.cfg.TokenFile, "reason", err.Error())
}
if p.cfg.InsecureSkipTLSVerify {
log.Info("WARNING: insecureSkipTLSVerify is set; the API's certificate is not verified")
}
return p, nil
}
// newProvider is the constructor tests use, with an in-memory api.
func newProvider(name string, cfg provider.ProxmoxConfig, a api) *Provider {
return &Provider{
name: name,
cfg: withDefaults(cfg),
api: a,
now: time.Now,
budgets: defaultBudgets(),
nameLocks: keyedlock.New(),
}
}
// withDefaults returns cfg with every optional field filled in, so the rest
// of the package never re-checks for an empty value. Slices, maps and
// pointers are copied, not shared with the caller.
func withDefaults(cfg provider.ProxmoxConfig) provider.ProxmoxConfig {
cfg.Nodes = append([]string(nil), cfg.Nodes...)
if cfg.VMIDRange == nil {
cfg.VMIDRange = &provider.ProxmoxVMIDRange{Min: defaultVMIDMin, Max: defaultVMIDMax}
} else {
r := *cfg.VMIDRange
cfg.VMIDRange = &r
}
if cfg.RequestTimeoutSeconds == 0 {
cfg.RequestTimeoutSeconds = defaultRequestTimeoutSeconds
}
install := true
if cfg.InstallGuestAgent != nil {
install = *cfg.InstallGuestAgent
}
cfg.InstallGuestAgent = &install
templates := make(map[string]provider.ProxmoxTemplate, len(cfg.Templates))
for k, v := range cfg.Templates {
templates[k] = v
}
cfg.Templates = templates
machineTypes := make(map[string]provider.ProxmoxMachineType, len(cfg.MachineTypes))
for k, v := range cfg.MachineTypes {
machineTypes[k] = v
}
cfg.MachineTypes = machineTypes
return cfg
}
func (p *Provider) logger(ctx context.Context) logr.Logger {
return logf.FromContext(ctx).WithName("proxmox").WithValues("provider", p.name)
}
// node resolves placement.zone to a configured node. An empty zone is
// allowed only when exactly one node is configured.
func (p *Provider) node(zone string) (string, error) {
if zone == "" {
if len(p.cfg.Nodes) == 1 {
return p.cfg.Nodes[0], nil
}
return "", permanentf("placement.zone is required when proxmox.nodes lists more than one node")
}
for _, n := range p.cfg.Nodes {
if n == zone {
return n, nil
}
}
return "", permanentf("placement.zone %q is not in proxmox.nodes", zone)
}
// wrapErr is the single exit for errors: it classifies err and wraps it with
// the configured provider name, op and id.
func (p *Provider) wrapErr(op, id string, err error) error {
return provider.Wrap(classify(op, err), op, p.name, id, err)
}
// lockName takes the per-name lock of spec entry 19, or gives up with ctx,
// so a caller whose ctx ends while another caller's Create waits on a clone
// returns promptly.
func (p *Provider) lockName(ctx context.Context, name string) (func(), error) {
release, err := p.nameLocks.Acquire(ctx, name)
if err != nil {
return nil, fmt.Errorf("%w: waiting for the lock on %s: %w", provider.ErrTransient, name, err)
}
return release, nil
}
@@ -0,0 +1,624 @@
//go:build integration
package proxmox
import (
"context"
"crypto/rand"
"encoding/hex"
"errors"
"fmt"
"net"
"net/http"
"net/url"
"os"
"strconv"
"strings"
"sync"
"testing"
"time"
"github.com/go-logr/logr"
"github.com/go-logr/logr/funcr"
sdktrace "go.opentelemetry.io/otel/sdk/trace"
"go.opentelemetry.io/otel/sdk/trace/tracetest"
"k8s.io/apimachinery/pkg/types"
logf "sigs.k8s.io/controller-runtime/pkg/log"
"gitea.home.hrajfrisbee.cz/kacerr/egress-proxies-operator/internal/provider"
"gitea.home.hrajfrisbee.cz/kacerr/egress-proxies-operator/internal/tracing"
)
// The real-host test of spec entry 21. It runs only with -tags=integration
// and skips itself unless PROXMOX_TEST_* names a host; see docs/testing.md
// for the variables and what one run asserts.
const (
integrationProvider = "proxmox-integration"
integrationImage = "test-template"
integrationMachine = "small"
integrationPort = 3128
// integrationDiskGiB is above the 3 GiB of the recon template, so a run
// on such a template exercises the resize task; a larger template skips
// it.
integrationDiskGiB = 4
// integrationCloudInit is the user-data the VM boots with. Nothing
// asserts on the guest running it; it exists so the sentinel check can
// prove that no span attribute carries the cloud-init text.
integrationCloudInit = "#cloud-config\nruncmd:\n - echo proxmox-integration\n"
defaultIntegrationTimeout = 20 * time.Minute
defaultIntegrationLogLvl = 1
getPollInterval = 5 * time.Second
cleanupBudget = 5 * time.Minute
)
type integrationEnv struct {
url, tokenFile, node, isoStorage string
caFile, storage, bridge string
templateVMID, vlanTag int
vmidMin, vmidMax int
insecure, keep bool
timeout time.Duration
logLevel int
}
// loadIntegrationEnv reads the PROXMOX_TEST_* variables, skipping the test
// under -short or when a required one is missing.
func loadIntegrationEnv(t *testing.T) integrationEnv {
t.Helper()
if testing.Short() {
t.Skip("real-host test skipped under -short")
}
env := integrationEnv{timeout: defaultIntegrationTimeout, logLevel: defaultIntegrationLogLvl}
var templateVMID string
required := []struct {
name string
dst *string
}{
{"PROXMOX_TEST_URL", &env.url},
{"PROXMOX_TEST_TOKEN_FILE", &env.tokenFile},
{"PROXMOX_TEST_NODE", &env.node},
{"PROXMOX_TEST_TEMPLATE_VMID", &templateVMID},
{"PROXMOX_TEST_ISO_STORAGE", &env.isoStorage},
}
for _, r := range required {
v := os.Getenv(r.name)
if v == "" {
t.Skipf("%s is not set; the proxmox integration test needs a real Proxmox VE host (see docs/testing.md)", r.name)
}
*r.dst = v
}
env.templateVMID = envInt(t, "PROXMOX_TEST_TEMPLATE_VMID", 0)
env.caFile = os.Getenv("PROXMOX_TEST_CA_FILE")
env.storage = os.Getenv("PROXMOX_TEST_STORAGE")
env.bridge = os.Getenv("PROXMOX_TEST_BRIDGE")
env.vlanTag = envInt(t, "PROXMOX_TEST_VLAN_TAG", 0)
env.vmidMin = envInt(t, "PROXMOX_TEST_VMID_MIN", defaultVMIDMin)
env.vmidMax = envInt(t, "PROXMOX_TEST_VMID_MAX", defaultVMIDMax)
if env.vmidMin > env.vmidMax {
t.Fatalf("PROXMOX_TEST_VMID_MIN=%d is above PROXMOX_TEST_VMID_MAX=%d", env.vmidMin, env.vmidMax)
}
env.keep = os.Getenv("PROXMOX_TEST_KEEP") != ""
if v := os.Getenv("PROXMOX_TEST_INSECURE"); v != "" {
b, err := strconv.ParseBool(v)
if err != nil {
t.Fatalf("PROXMOX_TEST_INSECURE=%q: %v", v, err)
}
env.insecure = b
}
if v := os.Getenv("PROXMOX_TEST_TIMEOUT"); v != "" {
d, err := time.ParseDuration(v)
if err != nil {
t.Fatalf("PROXMOX_TEST_TIMEOUT=%q: %v", v, err)
}
env.timeout = d
}
env.logLevel = envInt(t, "PROXMOX_TEST_LOG_LEVEL", defaultIntegrationLogLvl)
return env
}
// envInt reads a non-negative integer variable, def when it is unset.
func envInt(t *testing.T, name string, def int) int {
t.Helper()
v := os.Getenv(name)
if v == "" {
return def
}
n, err := strconv.Atoi(v)
if err != nil || n < 0 {
t.Fatalf("%s=%q: want a non-negative integer", name, v)
}
return n
}
// providerConfig is what the operator would load for this host: one node,
// one template, one machine type.
func (e integrationEnv) providerConfig() provider.ProviderConfig {
return provider.ProviderConfig{
Name: integrationProvider,
Type: "proxmox",
Proxmox: &provider.ProxmoxConfig{
URL: e.url,
TokenFile: e.tokenFile,
CAFile: e.caFile,
InsecureSkipTLSVerify: e.insecure,
Nodes: []string{e.node},
ISOStorage: e.isoStorage,
Storage: e.storage,
Bridge: e.bridge,
VLANTag: e.vlanTag,
VMIDRange: &provider.ProxmoxVMIDRange{Min: e.vmidMin, Max: e.vmidMax},
Templates: map[string]provider.ProxmoxTemplate{integrationImage: {VMID: e.templateVMID}},
MachineTypes: map[string]provider.ProxmoxMachineType{
integrationMachine: {Cores: 1, MemoryMiB: 1024, DiskGiB: integrationDiskGiB},
},
},
}
}
// integrationLog routes the provider's logr output into t.Log through funcr
// at the chosen verbosity, keeps every line for assertions, and strips the
// key/value pairs that are constant for the run (stated once up front) so
// each line stays short. Lines arriving after the test ended are dropped:
// t.Log panics then.
type integrationLog struct {
t *testing.T
mu sync.Mutex
drop []string
lines []string
done bool
}
func newIntegrationLog(t *testing.T, level int, drop ...string) (*integrationLog, logr.Logger) {
t.Helper()
sink := &integrationLog{t: t, drop: drop}
t.Cleanup(func() {
sink.mu.Lock()
defer sink.mu.Unlock()
sink.done = true
})
log := funcr.New(func(prefix, args string) { sink.write(prefix, args) }, funcr.Options{Verbosity: level})
return sink, log
}
// dropValue adds a rendered pair (`"key"="value"`) to strip from later lines.
func (l *integrationLog) dropValue(key, value string) {
l.mu.Lock()
defer l.mu.Unlock()
l.drop = append(l.drop, fmt.Sprintf("%q=%q", key, value))
}
func (l *integrationLog) write(prefix, args string) {
l.mu.Lock()
defer l.mu.Unlock()
l.lines = append(l.lines, args)
if l.done {
return
}
for _, d := range l.drop {
args = strings.ReplaceAll(args, " "+d, "")
}
l.t.Logf("%s %s %s", time.Now().Format("15:04:05.000"), prefix, args)
}
func (l *integrationLog) matching(substr string) []string {
l.mu.Lock()
defer l.mu.Unlock()
var out []string
for _, line := range l.lines {
if strings.Contains(line, substr) {
out = append(out, line)
}
}
return out
}
// newIntegrationProvider builds the provider the way New does, with the
// span recorder's tracer on the transport, so the HTTP spans land in the
// same recorder as provider.WithTracing's provider.* spans. The bare client
// is returned too, for the test's own read-only inspection of the host.
func newIntegrationProvider(t *testing.T, pc provider.ProviderConfig, tp *sdktrace.TracerProvider) (*Provider, *httpAPI) {
t.Helper()
cfg := withDefaults(*pc.Proxmox)
client, err := newHTTPAPI(cfg, tracing.WithTracerProvider(tp))
if err != nil {
t.Fatalf("newHTTPAPI error = %v", err)
}
return newProvider(pc.Name, cfg, observe(client, pc.Name)), client
}
// randomUID is a UUID-shaped string, so the run's VM never collides with a
// previous run's leftovers or with a real Proxy's.
func randomUID(t *testing.T) string {
t.Helper()
var b [16]byte
if _, err := rand.Read(b[:]); err != nil {
t.Fatalf("reading random bytes: %v", err)
}
h := hex.EncodeToString(b[:])
return h[0:8] + "-" + h[8:12] + "-" + h[12:16] + "-" + h[16:20] + "-" + h[20:32]
}
// stepDone logs one lifecycle step with its duration; the log is the record
// of how long a clone or a boot took on this host.
func stepDone(t *testing.T, step string, start time.Time) {
t.Helper()
t.Logf("step %s: %s", step, time.Since(start).Round(time.Millisecond))
}
// waitUntil polls cond every interval until it is true or ctx ends.
func waitUntil(ctx context.Context, t *testing.T, interval time.Duration, what string, cond func() bool) {
t.Helper()
ticker := time.NewTicker(interval)
defer ticker.Stop()
for !cond() {
select {
case <-ticker.C:
case <-ctx.Done():
t.Fatalf("gave up waiting for %s: %v", what, ctx.Err())
}
}
}
// qemuVMs lists the cluster's QEMU VMs by VMID.
func qemuVMs(ctx context.Context, t *testing.T, client *httpAPI) map[int]vmResource {
t.Helper()
rs, err := client.ClusterResources(ctx)
if err != nil {
t.Fatalf("listing cluster resources: %v", err)
}
out := make(map[int]vmResource, len(rs))
for _, r := range rs {
if r.Type == "qemu" {
out[r.VMID] = r
}
}
return out
}
// vmidByName finds the VM named name, as a crashed Create may have left it.
func vmidByName(ctx context.Context, t *testing.T, client *httpAPI, name string) (int, bool) {
t.Helper()
for vmid, r := range qemuVMs(ctx, t, client) {
if r.Name == name {
return vmid, true
}
}
return 0, false
}
// seedOnStorage reports whether the ISO storage on node holds name's seed.
// It is one extra read-only GET of /nodes/{node}/storage/{storage}/content
// (captured as storage_content_iso.json): the seam has no method to list
// storage content, and gets none for a test.
func seedOnStorage(ctx context.Context, t *testing.T, client *httpAPI, node, storage, name string) bool {
t.Helper()
var content []struct {
VolID string `json:"volid"`
}
path := apiPath("nodes", node, "storage", storage, "content")
if err := client.do(ctx, http.MethodGet, path, url.Values{"content": {"iso"}}, nil, &content); err != nil {
t.Fatalf("listing iso content of %s on %s: %v", storage, node, err)
}
want := seedVolID(storage, name)
for _, c := range content {
if c.VolID == want {
return true
}
}
return false
}
// checkNode runs the health check and requires the one node Ready, with the
// details an admin reads.
func checkNode(ctx context.Context, t *testing.T, built provider.Provider, env integrationEnv) {
t.Helper()
checker, ok := built.(provider.Checker)
if !ok {
t.Fatalf("New() returned %T, which is not a provider.Checker", built)
}
targets := checker.Check(ctx)
if len(targets) != 1 {
t.Fatalf("Check() returned %d targets, want 1: %+v", len(targets), targets)
}
ts := targets[0]
if !ts.Ready || ts.Reason != provider.ReasonReachable {
t.Fatalf("Check() = ready %v reason %q message %q, want Ready/%s (details %v)", ts.Ready, ts.Reason, ts.Message, provider.ReasonReachable, ts.Details)
}
for _, key := range []string{"pveVersion", "kernel", "tokenID", "isoStorageAvailGiB", "templates"} {
if ts.Details[key] == "" {
t.Errorf("Check() details lack %q: %v", key, ts.Details)
}
}
if ts.Details["templates"] != "1/1" {
t.Errorf("Check() templates = %q, want 1/1", ts.Details["templates"])
}
t.Logf("node %s (%s): pve %s, kernel %s, token %s, %s free on %s, templates %s, latency %s",
ts.Name, ts.Address, ts.Details["pveVersion"], ts.Details["kernel"], ts.Details["tokenID"],
ts.Details["isoStorageAvailGiB"]+" GiB", env.isoStorage, ts.Details["templates"], ts.Latency.Round(time.Millisecond))
}
func TestProxmox_lifecycle(t *testing.T) {
env := loadIntegrationEnv(t)
pc := env.providerConfig()
sink, log := newIntegrationLog(t, env.logLevel,
fmt.Sprintf("%q=%q", "provider", integrationProvider),
fmt.Sprintf("%q=%q", "node", env.node))
ctx, cancel := context.WithTimeout(logf.IntoContext(context.Background(), log), env.timeout)
defer cancel()
t.Logf("provider=%s url=%s node=%s template=%d isoStorage=%s storage=%q bridge=%q vlanTag=%d vmids %d..%d budget=%s log level %d; provider, node and name are left off every provider log line",
integrationProvider, env.url, env.node, env.templateVMID, env.isoStorage, env.storage, env.bridge, env.vlanTag,
env.vmidMin, env.vmidMax, env.timeout, env.logLevel)
if env.insecure {
t.Log("PROXMOX_TEST_INSECURE is set: the API's certificate is NOT verified for this run")
}
// 1. New returns without a request.
start := time.Now()
built, err := New(ctx, pc)
elapsed := time.Since(start)
if err != nil {
t.Fatalf("New() error = %v", err)
}
if elapsed > 100*time.Millisecond {
t.Errorf("New() took %s, want under 100ms: it must not make a request", elapsed)
}
if lines := sink.matching(`"msg"="proxmox api call"`); len(lines) > 0 {
t.Errorf("New() called the API before any operation: %q", lines)
}
stepDone(t, "1 New", start)
// 2. Check reports the node Ready with details.
start = time.Now()
checkNode(ctx, t, built, env)
stepDone(t, "2 Check", start)
// 3. Every later call goes through the tracing decorator, with the
// transport's HTTP spans recorded alongside.
sr := tracetest.NewSpanRecorder()
tp := sdktrace.NewTracerProvider(sdktrace.WithSpanProcessor(sr))
t.Cleanup(func() { _ = tp.Shutdown(context.Background()) })
p, client := newIntegrationProvider(t, pc, tp)
traced := provider.WithTracing(pc.Name, p, tracing.WithTracerProvider(tp))
// 4. Create.
uid := randomUID(t)
name := provider.NameFromUID(types.UID(uid))
sink.dropValue("name", name)
req := provider.CreateRequest{
Name: name,
UID: uid,
Namespace: "integration",
ProxyName: "proxmox-lifecycle",
Placement: provider.Placement{Zone: env.node, MachineType: integrationMachine, Image: integrationImage},
CloudInit: integrationCloudInit,
Port: integrationPort,
}
t.Logf("VM name %s, uid %s", name, uid)
deleted := false
t.Cleanup(func() {
if deleted {
return
}
cctx, ccancel := context.WithTimeout(logf.IntoContext(context.Background(), log), cleanupBudget)
defer ccancel()
vmid, found := vmidByName(cctx, t, client, name)
if env.keep {
msg := "PROXMOX_TEST_KEEP is set: the VM and its seed stay. Clean up as root on " + env.node + " with:\n"
if found {
msg += fmt.Sprintf(" qm stop %d\n qm destroy %d --purge\n", vmid, vmid)
}
t.Log(msg + " pvesm free " + seedVolID(env.isoStorage, name))
return
}
if !found {
// No VM, but a seed may be left: Delete of an absent VM still
// deletes it.
vmid = vmidFor(uid, env.vmidMin, env.vmidMax, 0)
}
if err := p.Delete(cctx, formatProviderID(vmid, name)); err != nil {
t.Errorf("cleanup Delete of VM %d error = %v; remove it and %s by hand", vmid, err, seedVolID(env.isoStorage, name))
}
})
start = time.Now()
id, err := traced.Create(ctx, req)
if err != nil {
t.Fatalf("Create() error = %v", err)
}
vmid, gotName, err := parseProviderID(id)
if err != nil || gotName != name {
t.Fatalf("Create() = %q, want vms/<vmid>/%s (parse error %v)", id, name, err)
}
if vmid < env.vmidMin || vmid > env.vmidMax {
t.Errorf("Create() VMID %d is outside %d..%d", vmid, env.vmidMin, env.vmidMax)
}
sink.dropValue("vmid", strconv.Itoa(vmid))
createReturned := time.Now()
t.Logf("VM %d, id %s", vmid, id)
stepDone(t, "4 Create", start)
// 5. Create again: same ID, nothing new in the cluster.
start = time.Now()
before := len(qemuVMs(ctx, t, client))
again, err := traced.Create(ctx, req)
if err != nil {
t.Fatalf("second Create() error = %v", err)
}
if again != id {
t.Errorf("second Create() = %q, want %q", again, id)
}
if after := len(qemuVMs(ctx, t, client)); after != before {
t.Errorf("second Create() changed the cluster's VM count from %d to %d", before, after)
}
stepDone(t, "5 Create again", start)
// 6. Get reaches Running with an IPv4, and reads back what Create wrote.
start = time.Now()
var inst *provider.Instance
waitUntil(ctx, t, getPollInterval, "the VM to be Running with an IPv4", func() bool {
var err error
inst, err = traced.Get(ctx, id)
if errors.Is(err, provider.ErrTransient) {
t.Logf("Get(%s) is Transient while the guest boots: %v", id, err)
return false
}
if err != nil {
t.Fatalf("Get(%s) error = %v", id, err)
}
return inst.State == provider.StateRunning && net.ParseIP(inst.IP).To4() != nil
})
assertMarkReadBack(ctx, t, client, env.node, vmid, inst, uid)
t.Logf("VM running at %s, %s after Create returned (createdAt %s)", inst.IP, time.Since(createReturned).Round(time.Millisecond), inst.CreatedAt.Format(time.RFC3339))
stepDone(t, "6 Get until Running", start)
// 7. ListByTag contains it.
start = time.Now()
list, err := traced.ListByTag(ctx)
if err != nil {
t.Fatalf("ListByTag() error = %v", err)
}
assertListed(t, list, inst)
stepDone(t, "7 ListByTag", start)
// 8. Reachability, logged only: this machine may not route to the VM's
// network.
start = time.Now()
addr := net.JoinHostPort(inst.IP, "22")
if conn, err := net.DialTimeout("tcp", addr, 3*time.Second); err != nil {
t.Logf("TCP connect to %s failed (informational; the test host may not route to the VM's network): %v", addr, err)
} else {
_ = conn.Close()
t.Logf("TCP connect to %s succeeded", addr)
}
stepDone(t, "8 TCP connect", start)
if env.keep {
t.Log("PROXMOX_TEST_KEEP is set: skipping Delete and the leftovers check")
} else {
// 9. Delete, Get is NotFound, Delete again.
start = time.Now()
if err := traced.Delete(ctx, id); err != nil {
t.Fatalf("Delete() error = %v", err)
}
deleted = true
if _, err := traced.Get(ctx, id); !errors.Is(err, provider.ErrNotFound) {
t.Errorf("Get() after Delete error = %v, want ErrNotFound", err)
}
if err := traced.Delete(ctx, id); err != nil {
t.Errorf("second Delete() error = %v, want nil", err)
}
stepDone(t, "9 Delete, Get, Delete", start)
// 10. No VM with the name and no seed remain.
start = time.Now()
if left, ok := vmidByName(ctx, t, client, name); ok {
t.Errorf("VM %d named %s is still in the cluster after Delete", left, name)
}
if seedOnStorage(ctx, t, client, env.node, env.isoStorage, name) {
t.Errorf("%s is still on %s after Delete", seedVolID(env.isoStorage, name), env.node)
}
stepDone(t, "10 leftovers", start)
}
// 11. Spans: HTTP spans are children of the decorator's.
assertLifecycleSpans(t, sr.Ended())
}
// assertMarkReadBack checks that Get's UID and CreatedAt are the mark
// Create wrote into the VM's description, read back from its config, and
// that CreatedAt is recent.
func assertMarkReadBack(ctx context.Context, t *testing.T, client *httpAPI, node string, vmid int, inst *provider.Instance, uid string) {
t.Helper()
cfg, err := client.VMConfig(ctx, node, vmid)
if err != nil {
t.Fatalf("reading the config of VM %d: %v", vmid, err)
}
written := parseDescription(cfg["description"])
if inst.UID != uid || written.UID != uid {
t.Errorf("Get() UID = %q, the VM's mark says %q, want %q", inst.UID, written.UID, uid)
}
if !inst.CreatedAt.Equal(written.CreatedAt) || inst.CreatedAt.IsZero() {
t.Errorf("Get() CreatedAt = %s, the VM's mark says %s; want them equal and set", inst.CreatedAt, written.CreatedAt)
} else if skew := time.Since(inst.CreatedAt).Abs(); skew > 5*time.Minute {
t.Errorf("Get() CreatedAt = %s, %s from now; want within a few minutes", inst.CreatedAt, skew)
}
}
// assertListed checks that list holds inst's ID with its UID and
// CreatedAt.
func assertListed(t *testing.T, list []provider.Instance, inst *provider.Instance) {
t.Helper()
for _, li := range list {
if li.ID != inst.ID {
continue
}
if li.UID != inst.UID || !li.CreatedAt.Equal(inst.CreatedAt) {
t.Errorf("ListByTag() entry = %+v, want UID %q and CreatedAt %s", li, inst.UID, inst.CreatedAt)
}
return
}
t.Errorf("ListByTag() = %d instances, none with ID %s", len(list), inst.ID)
}
// assertLifecycleSpans checks that at least one provider.create span has an
// HTTP child, and that no attribute of any span carries the cloud-init text.
func assertLifecycleSpans(t *testing.T, spans []sdktrace.ReadOnlySpan) {
t.Helper()
satisfied := false
for _, create := range spans {
if create.Name() != "provider.create" {
continue
}
parent := create.SpanContext().SpanID()
children := 0
for _, s := range spans {
if s.Parent().SpanID() == parent && strings.HasPrefix(s.Name(), "HTTP ") {
children++
}
}
t.Logf("provider.create span %s: %d HTTP children", parent, children)
if children > 0 {
satisfied = true
}
}
if !satisfied {
names := make([]string, len(spans))
for i, s := range spans {
names[i] = s.Name()
}
t.Errorf("no provider.create span has an HTTP child; spans: %v", names)
}
for _, s := range spans {
for _, a := range s.Attributes() {
v := a.Value.Emit()
if strings.Contains(v, integrationCloudInit) || strings.Contains(v, "#cloud-config") {
t.Errorf("span %s attribute %s carries the cloud-init text", s.Name(), a.Key)
}
}
}
}
// TestProxmoxCheck_realHost runs the health check against the host and
// creates nothing.
func TestProxmoxCheck_realHost(t *testing.T) {
env := loadIntegrationEnv(t)
pc := env.providerConfig()
_, log := newIntegrationLog(t, env.logLevel,
fmt.Sprintf("%q=%q", "provider", integrationProvider),
fmt.Sprintf("%q=%q", "node", env.node))
ctx, cancel := context.WithTimeout(logf.IntoContext(context.Background(), log), env.timeout)
defer cancel()
if env.insecure {
t.Log("PROXMOX_TEST_INSECURE is set: the API's certificate is NOT verified for this run")
}
built, err := New(ctx, pc)
if err != nil {
t.Fatalf("New() error = %v", err)
}
checkNode(ctx, t, built, env)
}
+237
View File
@@ -0,0 +1,237 @@
package proxmox
import (
"context"
"errors"
"strings"
"sync"
"testing"
"time"
"github.com/go-logr/logr/funcr"
logf "sigs.k8s.io/controller-runtime/pkg/log"
"gitea.home.hrajfrisbee.cz/kacerr/egress-proxies-operator/internal/provider"
)
// captureContext collects funcr output at V(2).
func captureContext() (context.Context, *[]string) {
lines := &[]string{}
var mu sync.Mutex
log := funcr.New(func(prefix, args string) {
mu.Lock()
defer mu.Unlock()
*lines = append(*lines, prefix+" "+args)
}, funcr.Options{Verbosity: 2})
return logf.IntoContext(context.Background(), log), lines
}
func testConfig() provider.ProxmoxConfig {
return provider.ProxmoxConfig{
URL: "https://192.0.2.8:8006",
TokenFile: "/nonexistent/proxmox/token",
Nodes: []string{"pv1-1"},
ISOStorage: "proxy-seeds",
Templates: map[string]provider.ProxmoxTemplate{"debian-13": {VMID: 9001}},
MachineTypes: map[string]provider.ProxmoxMachineType{"small": {Cores: 1, MemoryMiB: 1024}},
}
}
func TestNew_doesNotDialAndWarnsOnMissingToken(t *testing.T) {
t.Parallel()
ctx, lines := captureContext()
cfg := testConfig()
start := time.Now()
p, err := New(ctx, provider.ProviderConfig{Name: "proxmox-home", Type: "proxmox", Proxmox: &cfg})
if err != nil || p == nil {
t.Fatalf("New() = %v, %v; want a provider", p, err)
}
if elapsed := time.Since(start); elapsed > 100*time.Millisecond {
t.Errorf("New() took %v, want under 100ms (it must not make a request)", elapsed)
}
var warnings int
for _, l := range *lines {
if strings.Contains(l, "WARNING:") && strings.Contains(l, "token file") {
warnings++
}
if !strings.Contains(l, `"provider"="proxmox-home"`) {
t.Errorf("log line lacks the provider value: %s", l)
}
}
if warnings != 1 {
t.Errorf("got %d token WARNING: lines, want 1:\n%s", warnings, strings.Join(*lines, "\n"))
}
}
func TestNew_insecureSkipTLSVerifyWarns(t *testing.T) {
t.Parallel()
ctx, lines := captureContext()
cfg := testConfig()
cfg.InsecureSkipTLSVerify = true
if _, err := New(ctx, provider.ProviderConfig{Name: "proxmox-home", Type: "proxmox", Proxmox: &cfg}); err != nil {
t.Fatalf("New() error = %v, want nil", err)
}
for _, l := range *lines {
if strings.Contains(l, "WARNING:") && strings.Contains(l, "not verified") {
return
}
}
t.Errorf("no insecureSkipTLSVerify warning in:\n%s", strings.Join(*lines, "\n"))
}
func TestNew_nilBlockDoesNotPanic(t *testing.T) {
t.Parallel()
ctx, _ := captureContext()
p, err := New(ctx, provider.ProviderConfig{Name: "proxmox-empty", Type: "proxmox"})
if err != nil {
t.Fatalf("New() error = %v, want nil", err)
}
pp, ok := p.(*Provider)
if !ok {
t.Fatalf("New() = %T, want *Provider", p)
}
if pp.name != "proxmox-empty" || pp.cfg.VMIDRange == nil || pp.now == nil {
t.Errorf("provider = %+v, want its name, defaults and a clock", pp)
}
}
// TestNew_wiresObservedHTTPClient checks that production calls go through
// the logging decorator to the HTTP client, with /api2/json appended and
// the request timeout from the config.
func TestNew_wiresObservedHTTPClient(t *testing.T) {
t.Parallel()
ctx, _ := captureContext()
cfg := testConfig()
cfg.RequestTimeoutSeconds = 45
p, err := New(ctx, provider.ProviderConfig{Name: "proxmox-home", Type: "proxmox", Proxmox: &cfg})
if err != nil {
t.Fatalf("New() error = %v", err)
}
o, ok := p.(*Provider).api.(*observed)
if !ok {
t.Fatalf("api = %T, want *observed", p.(*Provider).api)
}
h, ok := o.inner.(*httpAPI)
if !ok {
t.Fatalf("observed.inner = %T, want *httpAPI", o.inner)
}
if o.provider != "proxmox-home" || h.base.String() != "https://192.0.2.8:8006/api2/json" || h.timeout != 45*time.Second || h.tokenFile != cfg.TokenFile {
t.Errorf("wiring = provider %q base %q timeout %v token file %q", o.provider, h.base, h.timeout, h.tokenFile)
}
}
func TestWithDefaults_table(t *testing.T) {
t.Parallel()
off := false
tests := []struct {
name string
in provider.ProxmoxConfig
wantRange provider.ProxmoxVMIDRange
wantTimeout int
wantAgent bool
}{
{name: "empty fills every default", wantRange: provider.ProxmoxVMIDRange{Min: 100000, Max: 199999}, wantTimeout: 30, wantAgent: true},
{
name: "explicit values kept",
in: provider.ProxmoxConfig{VMIDRange: &provider.ProxmoxVMIDRange{Min: 500, Max: 600}, RequestTimeoutSeconds: 45, InstallGuestAgent: &off},
wantRange: provider.ProxmoxVMIDRange{Min: 500, Max: 600},
wantTimeout: 45,
wantAgent: false,
},
}
for _, tc := range tests {
t.Run(tc.name, func(t *testing.T) {
t.Parallel()
got := withDefaults(tc.in)
if *got.VMIDRange != tc.wantRange || got.RequestTimeoutSeconds != tc.wantTimeout || *got.InstallGuestAgent != tc.wantAgent {
t.Errorf("withDefaults() = range %+v timeout %d agent %v; want %+v %d %v",
*got.VMIDRange, got.RequestTimeoutSeconds, *got.InstallGuestAgent, tc.wantRange, tc.wantTimeout, tc.wantAgent)
}
})
}
}
func TestWithDefaults_doesNotShareWithCaller(t *testing.T) {
t.Parallel()
in := testConfig()
in.VMIDRange = &provider.ProxmoxVMIDRange{Min: 500, Max: 600}
got := withDefaults(in)
got.Nodes[0] = "changed"
got.VMIDRange.Min = 1
got.Templates["debian-13"] = provider.ProxmoxTemplate{VMID: 1}
got.MachineTypes["small"] = provider.ProxmoxMachineType{}
if in.Nodes[0] != "pv1-1" || in.VMIDRange.Min != 500 || in.Templates["debian-13"].VMID != 9001 || in.MachineTypes["small"].Cores != 1 {
t.Errorf("mutating the result changed the caller's config: %+v", in)
}
}
func TestNode_table(t *testing.T) {
t.Parallel()
tests := []struct {
name string
nodes []string
zone string
want string
wantErr string
}{
{name: "one node and empty zone selects it", nodes: []string{"pv1-1"}, zone: "", want: "pv1-1"},
{name: "several nodes and empty zone is permanent", nodes: []string{"pv1-1", "pv1-2"}, zone: "", wantErr: "placement.zone is required"},
{name: "known node", nodes: []string{"pv1-1", "pv1-2"}, zone: "pv1-2", want: "pv1-2"},
{name: "unknown node is permanent", nodes: []string{"pv1-1"}, zone: "pv9", wantErr: `placement.zone "pv9" is not in proxmox.nodes`},
}
for _, tc := range tests {
t.Run(tc.name, func(t *testing.T) {
t.Parallel()
cfg := testConfig()
cfg.Nodes = tc.nodes
got, err := newProvider("proxmox-home", cfg, nil).node(tc.zone)
if tc.wantErr == "" {
if err != nil || got != tc.want {
t.Errorf("node(%q) = %q, %v; want %q, nil", tc.zone, got, err, tc.want)
}
return
}
if !errors.Is(err, provider.ErrPermanent) || !strings.Contains(err.Error(), tc.wantErr) {
t.Errorf("node(%q) error = %v, want ErrPermanent containing %q", tc.zone, err, tc.wantErr)
}
})
}
}
func TestWrapErr_carriesProviderNameOpAndID(t *testing.T) {
t.Parallel()
p := newProvider("proxmox-home", testConfig(), nil)
err := p.wrapErr("get", "vms/150042/proxy-x", loadAPIError(t, "err_vm_missing"))
var pe *provider.Error
if !errors.As(err, &pe) {
t.Fatalf("wrapErr() = %T, want *provider.Error", err)
}
if pe.Provider != "proxmox-home" || pe.Op != "get" || pe.ID != "vms/150042/proxy-x" || !errors.Is(err, provider.ErrNotFound) {
t.Errorf("wrapErr() = %+v, want configured name, op, id and class NotFound", pe)
}
var ae *apiError
if !errors.As(err, &ae) || ae.Status != 500 {
t.Errorf("wrapErr() lost the underlying apiError: %v", err)
}
}
func TestLockName_givesUpWithContextAsTransient(t *testing.T) {
t.Parallel()
p := newProvider("proxmox-home", testConfig(), nil)
release, err := p.lockName(context.Background(), "proxy-abc")
if err != nil {
t.Fatalf("lockName error = %v", err)
}
defer release()
other, err := p.lockName(context.Background(), "proxy-def")
if err != nil {
t.Fatalf("lockName on another name error = %v, want it free", err)
}
other()
ctx, cancel := context.WithTimeout(context.Background(), 20*time.Millisecond)
defer cancel()
within(t, 5*time.Second, func() { _, err = p.lockName(ctx, "proxy-abc") })
if !errors.Is(err, provider.ErrTransient) || !errors.Is(err, context.DeadlineExceeded) {
t.Errorf("lockName on a held name error = %v, want ErrTransient wrapping DeadlineExceeded", err)
}
}
+65
View File
@@ -0,0 +1,65 @@
package proxmox
import (
"bytes"
"context"
"errors"
"fmt"
"github.com/go-logr/logr"
"gitea.home.hrajfrisbee.cz/kacerr/egress-proxies-operator/internal/provider"
"gitea.home.hrajfrisbee.cz/kacerr/egress-proxies-operator/internal/provider/nocloud"
)
// seedISOName is the seed's file name on the ISO storage. Upload refuses a
// name without the .iso suffix (F12), and GC's blind spot is documented by
// this pattern (spec entry 11).
func seedISOName(name string) string {
return name + "-seed.iso"
}
// seedVolID is the seed's volume id, as ide2 references it and as the
// storage content delete takes it.
func seedVolID(storage, name string) string {
return storage + ":iso/" + seedISOName(name)
}
// buildSeed renders the NoCloud seed for req in memory (spec entry 5): a
// seed is tens of KiB. Nothing about its content is logged or put in an
// error. Only an empty name is Permanent; a staging failure (no /tmp, a
// full emptyDir) is left unclassified, so classify makes it Transient and
// the reconciler retries instead of latching the Proxy.
func (p *Provider) buildSeed(req provider.CreateRequest) (*bytes.Buffer, error) {
vendorData, err := nocloud.BuildVendorData(p.cfg.VendorData, *p.cfg.InstallGuestAgent)
if err != nil {
return nil, fmt.Errorf("building vendor-data for %s: %w", req.Name, err)
}
var iso bytes.Buffer
if err := nocloud.BuildSeedISO(req.Name, req.CloudInit, vendorData, &iso); err != nil {
if errors.Is(err, nocloud.ErrEmptyName) {
return nil, permanentf("building seed ISO: %v", err)
}
return nil, fmt.Errorf("building seed ISO for %s: %w", req.Name, err)
}
return &iso, nil
}
// uploadSeed puts a freshly built seed on the VM's node and waits for the
// copy (spec entry 5). An upload overwrites a file of the same name (F12),
// so what a cut-off attempt left is replaced without a delete first.
func (p *Provider) uploadSeed(ctx context.Context, log logr.Logger, node string, req provider.CreateRequest) error {
iso, err := p.buildSeed(req)
if err != nil {
return err
}
volid := seedVolID(p.cfg.ISOStorage, req.Name)
upid, err := p.api.UploadISO(ctx, node, p.cfg.ISOStorage, seedISOName(req.Name), iso, int64(iso.Len()))
if err != nil {
return fmt.Errorf("uploading seed %s: %w", volid, err)
}
if err := p.waitTask(ctx, log, upid, p.budgets.upload); err != nil {
return fmt.Errorf("uploading seed %s: %w", volid, err)
}
return nil
}
+139
View File
@@ -0,0 +1,139 @@
package proxmox
import (
"bytes"
"errors"
"io"
"strings"
"testing"
"github.com/kdomanski/iso9660"
"gitea.home.hrajfrisbee.cz/kacerr/egress-proxies-operator/internal/provider"
"gitea.home.hrajfrisbee.cz/kacerr/egress-proxies-operator/internal/provider/nocloud"
)
// seedFiles opens a built seed and returns its label and files by name.
func seedFiles(t *testing.T, iso *bytes.Buffer) (string, map[string]string) {
t.Helper()
img, err := iso9660.OpenImage(bytes.NewReader(iso.Bytes()))
if err != nil {
t.Fatalf("OpenImage: %v", err)
}
label, err := img.Label()
if err != nil {
t.Fatalf("Label: %v", err)
}
root, err := img.RootDir()
if err != nil {
t.Fatalf("RootDir: %v", err)
}
children, err := root.GetChildren()
if err != nil {
t.Fatalf("GetChildren: %v", err)
}
files := make(map[string]string, len(children))
for _, c := range children {
data, err := io.ReadAll(c.Reader())
if err != nil {
t.Fatalf("reading %s: %v", c.Name(), err)
}
files[c.Name()] = string(data)
}
return label, files
}
func TestSeedNames_table(t *testing.T) {
t.Parallel()
if got, want := seedISOName("proxy-abc"), "proxy-abc-seed.iso"; got != want {
t.Errorf("seedISOName = %q, want %q", got, want)
}
if got, want := seedVolID("proxy-seeds", "proxy-abc"), "proxy-seeds:iso/proxy-abc-seed.iso"; got != want {
t.Errorf("seedVolID = %q, want %q", got, want)
}
}
func TestBuildSeed_table(t *testing.T) {
t.Parallel()
off := false
userData := "#cloud-config\r\npackages: [squid]\r\nwrite_files:\n - content: \"caf\xc3\xa9\"\n"
configured := "#cloud-config\nntp:\n enabled: true\n"
tests := []struct {
name string
vendorData string
install *bool
want func(t *testing.T, got string)
}{
{name: "agent script by default", want: func(t *testing.T, got string) {
if got != nocloud.GuestAgentScript {
t.Errorf("vendor-data is not the guest agent script: %q", got)
}
}},
{name: "configured joined with the agent script", vendorData: configured, want: func(t *testing.T, got string) {
if !strings.HasPrefix(got, "MIME-Version: 1.0\r\n") || !strings.Contains(got, configured) || !strings.Contains(got, nocloud.GuestAgentScript) {
t.Errorf("vendor-data is not a multipart of both documents: %q", got)
}
}},
{name: "agent off passes configured through", vendorData: configured, install: &off, want: func(t *testing.T, got string) {
if got != configured {
t.Errorf("vendor-data = %q, want the configured document", got)
}
}},
{name: "agent off with nothing configured is empty", install: &off, want: func(t *testing.T, got string) {
if got != "" {
t.Errorf("vendor-data = %q, want empty", got)
}
}},
}
for _, tc := range tests {
t.Run(tc.name, func(t *testing.T) {
t.Parallel()
cfg := testConfig()
cfg.VendorData = tc.vendorData
cfg.InstallGuestAgent = tc.install
p := newProvider("proxmox-home", cfg, nil)
iso, err := p.buildSeed(provider.CreateRequest{Name: "proxy-abc", CloudInit: userData})
if err != nil {
t.Fatalf("buildSeed error = %v", err)
}
label, files := seedFiles(t, iso)
if label != nocloud.SeedLabel {
t.Errorf("label = %q, want %q", label, nocloud.SeedLabel)
}
if files["user-data"] != userData {
t.Errorf("user-data changed:\n got %q\nwant %q", files["user-data"], userData)
}
if want := "instance-id: proxy-abc\nlocal-hostname: proxy-abc\n"; files["meta-data"] != want {
t.Errorf("meta-data = %q, want %q", files["meta-data"], want)
}
tc.want(t, files["vendor-data"])
})
}
}
func TestBuildSeed_emptyNameIsPermanentWithoutPayload(t *testing.T) {
t.Parallel()
const sentinel = "SENTINEL-cloud-init-payload"
p := newProvider("proxmox-home", testConfig(), nil)
_, err := p.buildSeed(provider.CreateRequest{CloudInit: sentinel})
if !errors.Is(err, provider.ErrPermanent) {
t.Fatalf("buildSeed error = %v, want ErrPermanent", err)
}
if strings.Contains(err.Error(), sentinel) {
t.Errorf("error carries the cloud-init payload: %v", err)
}
}
// Not parallel: it points TMPDIR at a path that cannot exist for the
// process.
func TestBuildSeed_stagingFailureIsTransient(t *testing.T) {
t.Setenv("TMPDIR", t.TempDir()+"/missing")
p := newProvider("proxmox-home", testConfig(), nil)
_, err := p.buildSeed(provider.CreateRequest{Name: "proxy-abc", CloudInit: "#cloud-config\n"})
if err == nil {
t.Fatal("buildSeed with no staging dir succeeded")
}
if got := classify("create", err); got != provider.ErrTransient {
t.Errorf("classify(buildSeed error) = %v, want ErrTransient: %v", got, err)
}
}
+55
View File
@@ -0,0 +1,55 @@
package proxmox
import (
"net/netip"
"gitea.home.hrajfrisbee.cz/kacerr/egress-proxies-operator/internal/provider"
)
// mapState turns status/current's status and qmpstatus plus the discovered
// address into the contract's state (spec entry 9). A VM is Running only
// with an address; prelaunch is still starting; any other qmpstatus of a
// running VM (paused, suspended, io-error, …) and any status not recognised
// is Stopped, which the reconciler answers by replacing the VM. An empty
// qmpstatus is taken as running: the field is missing, not a state.
func mapState(status, qmpStatus, ip string) provider.InstanceState {
switch {
case guestRunning(status, qmpStatus):
if ip == "" {
return provider.StateProvisioning
}
return provider.StateRunning
case status == "running" && qmpStatus == "prelaunch":
return provider.StateProvisioning
default:
return provider.StateStopped
}
}
// guestRunning reports a VM whose guest runs, the only state in which its
// agent can answer and mapState can call it Running.
func guestRunning(status, qmpStatus string) bool {
return status == "running" && (qmpStatus == "running" || qmpStatus == "")
}
// firstIPv4 picks the first usable IPv4 from the guest agent's interfaces,
// in the agent's order: not on the loopback interface, not loopback,
// link-local or unspecified. Empty means no address yet.
func firstIPv4(ifaces []agentInterface) string {
for _, iface := range ifaces {
if iface.Name == "lo" {
continue
}
for _, a := range iface.IPAddresses {
if a.Type != "ipv4" {
continue
}
addr, err := netip.ParseAddr(a.Address)
if err != nil || !addr.Is4() || addr.IsLoopback() || addr.IsLinkLocalUnicast() || addr.IsUnspecified() {
continue
}
return addr.String()
}
}
return ""
}
+114
View File
@@ -0,0 +1,114 @@
package proxmox
import (
"testing"
"gitea.home.hrajfrisbee.cz/kacerr/egress-proxies-operator/internal/provider"
)
func TestMapState_table(t *testing.T) {
t.Parallel()
tests := []struct {
status, qmp, ip string
want provider.InstanceState
}{
{"running", "running", "192.168.4.147", provider.StateRunning},
{"running", "running", "", provider.StateProvisioning},
{"running", "", "192.168.4.147", provider.StateRunning},
{"running", "", "", provider.StateProvisioning},
{"running", "prelaunch", "", provider.StateProvisioning},
{"running", "paused", "192.168.4.147", provider.StateStopped},
{"running", "suspended", "", provider.StateStopped},
{"running", "io-error", "", provider.StateStopped},
{"running", "something-new", "", provider.StateStopped},
{"stopped", "stopped", "", provider.StateStopped},
{"stopped", "", "", provider.StateStopped},
{"unknown", "", "", provider.StateStopped},
{"", "", "", provider.StateStopped},
}
for _, tc := range tests {
t.Run(tc.status+"/"+tc.qmp+"/"+tc.ip, func(t *testing.T) {
t.Parallel()
if got := mapState(tc.status, tc.qmp, tc.ip); got != tc.want {
t.Errorf("mapState(%q, %q, %q) = %q, want %q", tc.status, tc.qmp, tc.ip, got, tc.want)
}
})
}
}
// TestMapState_fromCaptures feeds status/current as Proxmox reported it for
// a running, a stopped and a paused VM.
func TestMapState_fromCaptures(t *testing.T) {
t.Parallel()
tests := []struct {
capture string
ip string
want provider.InstanceState
}{
{"status_current_running", "192.168.0.150", provider.StateRunning},
{"status_current_running", "", provider.StateProvisioning},
{"status_current_stopped", "", provider.StateStopped},
{"status_current_paused", "192.168.0.150", provider.StateStopped},
}
for _, tc := range tests {
t.Run(tc.capture+"/"+tc.ip, func(t *testing.T) {
t.Parallel()
s := decodeData[vmStatus](t, tc.capture)
if got := mapState(s.Status, s.QMPStatus, tc.ip); got != tc.want {
t.Errorf("mapState(%q, %q, %q) = %q, want %q", s.Status, s.QMPStatus, tc.ip, got, tc.want)
}
})
}
}
func TestFirstIPv4_filters(t *testing.T) {
t.Parallel()
ip := func(addr, typ string) agentIP { return agentIP{Address: addr, Type: typ} }
tests := []struct {
name string
ifaces []agentInterface
want string
}{
{name: "none", want: ""},
{name: "loopback interface only", ifaces: []agentInterface{{Name: "lo", IPAddresses: []agentIP{ip("127.0.0.1", "ipv4")}}}, want: ""},
{name: "ipv6 only", ifaces: []agentInterface{{Name: "ens18", IPAddresses: []agentIP{ip("fe80::1", "ipv6"), ip("2001:db8::5", "ipv6")}}}, want: ""},
{name: "link-local skipped", ifaces: []agentInterface{{Name: "ens18", IPAddresses: []agentIP{ip("169.254.3.4", "ipv4"), ip("192.168.4.20", "ipv4")}}}, want: "192.168.4.20"},
{name: "unspecified skipped", ifaces: []agentInterface{{Name: "ens18", IPAddresses: []agentIP{ip("0.0.0.0", "ipv4")}}}, want: ""},
{name: "loopback address on another interface skipped", ifaces: []agentInterface{{Name: "dummy0", IPAddresses: []agentIP{ip("127.0.0.2", "ipv4")}}}, want: ""},
{name: "type and literal disagree", ifaces: []agentInterface{{Name: "ens18", IPAddresses: []agentIP{ip("fe80::1", "ipv4"), ip("10.0.0.9", "ipv6")}}}, want: ""},
{name: "unparsable skipped", ifaces: []agentInterface{{Name: "ens18", IPAddresses: []agentIP{ip("not-an-ip", "ipv4"), ip("10.0.0.9", "ipv4")}}}, want: "10.0.0.9"},
{name: "agent order decides", ifaces: []agentInterface{
{Name: "ens18", IPAddresses: []agentIP{ip("192.168.4.20", "ipv4")}},
{Name: "ens19", IPAddresses: []agentIP{ip("10.0.0.9", "ipv4")}},
}, want: "192.168.4.20"},
}
for _, tc := range tests {
t.Run(tc.name, func(t *testing.T) {
t.Parallel()
if got := firstIPv4(tc.ifaces); got != tc.want {
t.Errorf("firstIPv4() = %q, want %q", got, tc.want)
}
})
}
}
// TestFirstIPv4_fromCaptures reads the guest agent's answer from the two
// Phase 0 runs, untagged and on VLAN 4.
func TestFirstIPv4_fromCaptures(t *testing.T) {
t.Parallel()
tests := map[string]string{
"agent_network_get_interfaces": "192.168.0.150",
"agent_network_get_interfaces_vlan": "192.168.4.147",
}
for capture, want := range tests {
t.Run(capture, func(t *testing.T) {
t.Parallel()
res := decodeData[struct {
Result []agentInterface `json:"result"`
}](t, capture)
if got := firstIPv4(res.Result); got != want {
t.Errorf("firstIPv4(%s) = %q, want %q", capture, got, want)
}
})
}
}
+128
View File
@@ -0,0 +1,128 @@
package proxmox
import (
"context"
"fmt"
"regexp"
"time"
"github.com/go-logr/logr"
"gitea.home.hrajfrisbee.cz/kacerr/egress-proxies-operator/internal/provider"
)
// upidPattern is Proxmox's task id: UPID:<node>:<pid>:<pstart>:<starttime>:
// <type>:<id>:<user>: with hex pid, pstart and starttime.
var upidPattern = regexp.MustCompile(`^UPID:([a-zA-Z0-9]([a-zA-Z0-9-]*[a-zA-Z0-9])?):[0-9A-Fa-f]{8}:[0-9A-Fa-f]{8,9}:[0-9A-Fa-f]{8}:[^:\s]+:[^:\s]*:[^:\s]+:$`)
// parseUPID returns the node a task runs on, which is where its status is
// read (spec entry 10). A malformed UPID is Permanent: Proxmox never sends
// one, so retrying cannot help.
func parseUPID(upid string) (string, error) {
m := upidPattern.FindStringSubmatch(upid)
if m == nil {
return "", permanentf("malformed task id %q", upid)
}
return m[1], nil
}
// taskSucceeded reads a stopped task's exitstatus the way Proxmox does:
// "OK", or "WARNINGS: <n>" for a task that finished but logged warnings.
func taskSucceeded(exit string) bool {
return exit == "OK" || taskWarnings.MatchString(exit)
}
var taskWarnings = regexp.MustCompile(`^WARNINGS: [0-9]+$`)
// pollUntil calls check every interval until it reports done or fails. Once
// budget has passed or ctx has ended it gives up with ErrTransient naming
// what it waited for; the backend keeps working, and the next attempt picks
// the result up.
func pollUntil(ctx context.Context, interval, budget time.Duration, what string, check func() (bool, error)) error {
deadline := time.NewTimer(budget)
defer deadline.Stop()
tick := time.NewTicker(interval)
defer tick.Stop()
for {
done, err := check()
if err != nil || done {
return err
}
select {
case <-ctx.Done():
return fmt.Errorf("%w: waiting for %s: %w", provider.ErrTransient, what, ctx.Err())
case <-deadline.C:
return transientf("waiting for %s: not done after %v", what, budget)
case <-tick.C:
}
}
}
// waitTask polls a task on the node its UPID names until it stops. A
// failed task is ErrTransient carrying its exitstatus, and logged at V(1)
// (spec entries 10 and 20).
func (p *Provider) waitTask(ctx context.Context, log logr.Logger, upid string, budget time.Duration) error {
node, err := parseUPID(upid)
if err != nil {
return err
}
var exit string
err = pollUntil(ctx, p.budgets.poll, budget, "task "+upid, func() (bool, error) {
s, err := p.api.TaskStatus(ctx, node, upid)
if err != nil {
return false, fmt.Errorf("reading task %s: %w", upid, err)
}
exit = s.ExitStatus
return s.Status == "stopped", nil
})
if err != nil {
return err
}
if !taskSucceeded(exit) {
log.V(1).Info("proxmox task failed", "upid", upid, "exitstatus", exit)
return transientf("proxmox task %s failed: %s", upid, exit)
}
if exit != "OK" {
log.V(1).Info("proxmox task finished with warnings", "upid", upid, "exitstatus", exit)
}
return nil
}
// waitUnlocked polls a VM's config until no task holds its lock and returns
// that config, for Create's ownership check (spec entry 2). A suspended lock
// is returned at once: it marks a hibernated VM, not a task, and never
// clears on its own, so waiting would only burn the clone budget. A VM that
// disappears meanwhile, as a failed clone does, returns the does-not-exist
// error for the caller to judge.
func (p *Provider) waitUnlocked(ctx context.Context, node string, vmid int, budget time.Duration) (vmConfig, error) {
var cfg vmConfig
err := pollUntil(ctx, p.budgets.poll, budget, fmt.Sprintf("the lock on VM %d", vmid), func() (bool, error) {
c, err := p.api.VMConfig(ctx, node, vmid)
if err != nil {
return false, fmt.Errorf("reading config of VM %d: %w", vmid, err)
}
cfg = c
return c["lock"] == "" || c["lock"] == lockSuspended, nil
})
if err != nil {
return nil, err
}
return cfg, nil
}
// waitRunning polls status/current after the start task until QEMU runs
// the guest (spec entry 10, step 9). A VM that is stopped again is
// Transient at once: it will not come up on its own, and the next attempt
// adopts and starts it.
func (p *Provider) waitRunning(ctx context.Context, node string, vmid int) error {
return pollUntil(ctx, p.budgets.poll, p.budgets.running, fmt.Sprintf("VM %d to run", vmid), func() (bool, error) {
st, err := p.api.VMStatus(ctx, node, vmid)
if err != nil {
return false, fmt.Errorf("reading status of VM %d: %w", vmid, err)
}
if st.Status == "stopped" {
return false, transientf("VM %d stopped right after its start", vmid)
}
return st.Status == "running" && st.QMPStatus == "running", nil
})
}
+240
View File
@@ -0,0 +1,240 @@
package proxmox
import (
"context"
"errors"
"strings"
"testing"
"time"
"gitea.home.hrajfrisbee.cz/kacerr/egress-proxies-operator/internal/provider"
)
// addTask registers a fake task on node that ends with exit.
func addTask(f *fakeAPI, node, exit string) string {
f.mu.Lock()
defer f.mu.Unlock()
return f.newTask(node, "qmstart", "150042", exit, nil)
}
func TestParseUPID_table(t *testing.T) {
t.Parallel()
tests := []struct {
name string
upid string
wantNode string
}{
{name: "captured clone", upid: decodeData[string](t, "clone_response"), wantNode: "pv1-1"},
{name: "captured task status", upid: decodeData[taskStatus](t, "task_status_stopped_ok").UPID, wantNode: "pv1-1"},
{name: "captured upload", upid: decodeData[string](t, "upload_response_proxy_seeds"), wantNode: "pv1-1"},
{name: "fake task on another node", upid: addTask(newFakeAPI(), "pv1-2", "OK"), wantNode: "pv1-2"},
{name: "empty", upid: ""},
{name: "prefix only", upid: "UPID:"},
{name: "lower-case prefix", upid: "upid:pv1-1:0010F9DC:02801DA1:6AB37871:qmclone:9001:root@pam:"},
{name: "no trailing colon", upid: "UPID:pv1-1:0010F9DC:02801DA1:6AB37871:qmclone:9001:root@pam"},
{name: "slash in node", upid: "UPID:pv1/1:0010F9DC:02801DA1:6AB37871:qmclone:9001:root@pam:"},
{name: "pid not hex", upid: "UPID:pv1-1:0010F9DZ:02801DA1:6AB37871:qmclone:9001:root@pam:"},
{name: "missing user", upid: "UPID:pv1-1:0010F9DC:02801DA1:6AB37871:qmclone:9001::"},
}
for _, tc := range tests {
t.Run(tc.name, func(t *testing.T) {
t.Parallel()
node, err := parseUPID(tc.upid)
if tc.wantNode == "" {
if !errors.Is(err, provider.ErrPermanent) {
t.Errorf("parseUPID(%q) = %q, %v; want ErrPermanent", tc.upid, node, err)
}
return
}
if err != nil || node != tc.wantNode {
t.Errorf("parseUPID(%q) = %q, %v; want %q", tc.upid, node, err, tc.wantNode)
}
})
}
}
func TestWaitTask_table(t *testing.T) {
t.Parallel()
failedExit := loadTaskExit(t, "task_status_stopped_error")
tests := []struct {
name string
node string
exit string
upid string // overrides the registered task's UPID
polls int
budget time.Duration
cancelled bool
statusErr error
wantClass error // nil for success
wantCalls int
wantInErr string
wantLog string
}{
{name: "ok on the first poll", exit: "OK", wantCalls: 1},
{name: "polls until stopped", exit: "OK", polls: 3, wantCalls: 4},
{name: "polls the node the UPID names", node: "pv1-2", exit: "OK", wantCalls: 1},
{name: "warnings count as success", exit: "WARNINGS: 2", wantCalls: 1, wantLog: `"msg"="proxmox task finished with warnings"`},
{
name: "failed exitstatus is transient and logged", exit: failedExit, wantClass: provider.ErrTransient, wantCalls: 1,
wantInErr: failedExit, wantLog: `"msg"="proxmox task failed"`,
},
{name: "budget exceeded names the UPID", exit: "OK", polls: 1 << 30, budget: 20 * time.Millisecond, wantClass: provider.ErrTransient, wantCalls: -1, wantInErr: "UPID:pv1-1:"},
{name: "ended ctx is transient", exit: "OK", polls: 1 << 30, cancelled: true, wantClass: provider.ErrTransient, wantCalls: 1, wantInErr: context.Canceled.Error()},
{name: "malformed UPID makes no call", upid: "UPID:bogus", wantClass: provider.ErrPermanent, wantCalls: 0},
{name: "status read failing is returned", exit: "OK", statusErr: loadAPIError(t, "err_forbidden"), wantClass: provider.ErrPermanent, wantCalls: 1, wantInErr: "Permission check failed"},
}
for _, tc := range tests {
t.Run(tc.name, func(t *testing.T) {
t.Parallel()
fx := newFixture(t, 1)
fx.fake.taskPolls = tc.polls
if tc.statusErr != nil {
fx.fake.failWith("TaskStatus", tc.statusErr)
}
node := tc.node
if node == "" {
node = fakeNode
}
upid := addTask(fx.fake, node, tc.exit)
if tc.upid != "" {
upid = tc.upid
}
budget := tc.budget
if budget == 0 {
budget = time.Minute
}
ctx := fx.ctx
if tc.cancelled {
var cancel context.CancelFunc
ctx, cancel = context.WithCancel(ctx)
cancel()
}
start := time.Now()
var err error
within(t, 10*time.Second, func() { err = fx.p.waitTask(ctx, fx.p.logger(ctx), upid, budget) })
if elapsed := time.Since(start); elapsed > 5*time.Second {
t.Errorf("waitTask took %v", elapsed)
}
if tc.wantClass == nil {
if err != nil {
t.Fatalf("waitTask error = %v, want nil", err)
}
} else if classify("create", err) != tc.wantClass {
t.Fatalf("waitTask error = %v, want class %v", err, tc.wantClass)
}
if tc.wantInErr != "" && !strings.Contains(err.Error(), tc.wantInErr) {
t.Errorf("waitTask error = %q, want it to contain %q", err, tc.wantInErr)
}
if got := fx.fake.callCount("TaskStatus"); tc.wantCalls >= 0 && got != tc.wantCalls {
t.Errorf("TaskStatus calls = %d, want %d", got, tc.wantCalls)
}
if tc.wantLog != "" {
lines := fx.logs.matching(tc.wantLog)
if len(lines) != 1 || !strings.Contains(lines[0], `"upid"="`+upid+`"`) || !strings.Contains(lines[0], `"exitstatus"="`+tc.exit+`"`) {
t.Errorf("log lines %q = %v, want one with upid and exitstatus", tc.wantLog, lines)
}
}
})
}
}
func TestWaitTask_quietAtV0(t *testing.T) {
t.Parallel()
fx := newFixture(t, 0)
err := fx.p.waitTask(fx.ctx, fx.p.logger(fx.ctx), addTask(fx.fake, fakeNode, "boom"), time.Minute)
if err == nil {
t.Fatal("waitTask on a failed task returned nil")
}
if lines := fx.logs.all(); len(lines) != 0 {
t.Errorf("V(0) lines = %v, want none", lines)
}
}
func TestWaitUnlocked_table(t *testing.T) {
t.Parallel()
tests := []struct {
name string
lock string
// onCall runs on the n-th VMConfig call (1-based) under f.mu.
onCall func(f *fakeAPI, n int)
budget time.Duration
cancelled bool
wantErr func(error) bool
wantCalls int
wantLock string
}{
{name: "unlocked answers at once", wantCalls: 1},
{name: "suspended lock is not waited on", lock: lockSuspended, budget: time.Hour, wantCalls: 1, wantLock: lockSuspended},
{
name: "waits until the lock clears", lock: "clone", wantCalls: 3,
onCall: func(f *fakeAPI, n int) {
if n == 3 {
f.vms[testVMID].lock = ""
}
},
},
{
name: "lock that never clears is transient", lock: "clone", budget: 20 * time.Millisecond, wantCalls: -1,
wantErr: func(err error) bool {
return errors.Is(err, provider.ErrTransient) && strings.Contains(err.Error(), "the lock on VM 150042")
},
},
{
name: "VM removed by a failed clone gives does not exist", lock: "clone", wantCalls: 2,
onCall: func(f *fakeAPI, n int) {
if n == 2 {
delete(f.vms, testVMID)
}
},
wantErr: isNotExist,
},
{
name: "ended ctx is transient", lock: "clone", cancelled: true, wantCalls: 1,
wantErr: func(err error) bool {
return errors.Is(err, provider.ErrTransient) && errors.Is(err, context.Canceled)
},
},
}
for _, tc := range tests {
t.Run(tc.name, func(t *testing.T) {
t.Parallel()
fx := newFixture(t, 1)
fx.addOurVM(testVMID, testName, "stopped").lock = tc.lock
var n int
fx.fake.hookOn("VMConfig", func() error {
fx.fake.mu.Lock()
defer fx.fake.mu.Unlock()
n++
if tc.onCall != nil {
tc.onCall(fx.fake, n)
}
return nil
})
budget := tc.budget
if budget == 0 {
budget = time.Minute
}
ctx := fx.ctx
if tc.cancelled {
var cancel context.CancelFunc
ctx, cancel = context.WithCancel(ctx)
cancel()
}
var (
cfg vmConfig
err error
)
within(t, 10*time.Second, func() { cfg, err = fx.p.waitUnlocked(ctx, fakeNode, testVMID, budget) })
if tc.wantErr == nil {
if err != nil || cfg["name"] != testName || cfg["lock"] != tc.wantLock {
t.Errorf("waitUnlocked = %v, %v; want the config with lock %q", cfg, err, tc.wantLock)
}
} else if !tc.wantErr(err) || cfg != nil {
t.Errorf("waitUnlocked = %v, %v; want no config and the expected error", cfg, err)
}
if got := fx.fake.callCount("VMConfig"); tc.wantCalls >= 0 && got != tc.wantCalls {
t.Errorf("VMConfig calls = %d, want %d", got, tc.wantCalls)
}
})
}
}
@@ -0,0 +1,5 @@
{
"status": 200,
"reason": "OK",
"body": "{\"data\":{\"result\":[{\"statistics\":{\"tx-errs\":0,\"rx-dropped\":0,\"rx-bytes\":7101,\"rx-packets\":86,\"rx-errs\":0,\"tx-dropped\":0,\"tx-packets\":86,\"tx-bytes\":7101},\"name\":\"lo\",\"ip-addresses\":[{\"ip-address\":\"127.0.0.1\",\"ip-address-type\":\"ipv4\",\"prefix\":8},{\"ip-address\":\"::1\",\"ip-address-type\":\"ipv6\",\"prefix\":128}],\"hardware-address\":\"00:00:00:00:00:00\"},{\"hardware-address\":\"bc:24:11:48:11:46\",\"ip-addresses\":[{\"ip-address\":\"192.168.0.150\",\"ip-address-type\":\"ipv4\",\"prefix\":24},{\"ip-address\":\"fe80::be24:11ff:fe48:1146\",\"ip-address-type\":\"ipv6\",\"prefix\":64}],\"name\":\"ens18\",\"statistics\":{\"rx-dropped\":0,\"rx-bytes\":29688403,\"rx-packets\":6744,\"tx-errs\":0,\"tx-bytes\":221044,\"tx-dropped\":0,\"tx-packets\":2970,\"rx-errs\":0}}]}}"
}
@@ -0,0 +1,5 @@
{
"status": 200,
"reason": "OK",
"body": "{\"data\":{\"result\":[{\"ip-addresses\":[{\"prefix\":8,\"ip-address-type\":\"ipv4\",\"ip-address\":\"127.0.0.1\"},{\"prefix\":128,\"ip-address-type\":\"ipv6\",\"ip-address\":\"::1\"}],\"hardware-address\":\"00:00:00:00:00:00\",\"statistics\":{\"tx-errs\":0,\"rx-packets\":86,\"rx-bytes\":7101,\"rx-dropped\":0,\"rx-errs\":0,\"tx-dropped\":0,\"tx-packets\":86,\"tx-bytes\":7101},\"name\":\"lo\"},{\"ip-addresses\":[{\"ip-address-type\":\"ipv4\",\"ip-address\":\"192.168.4.147\",\"prefix\":24},{\"ip-address\":\"fe80::be24:11ff:fea9:b4eb\",\"ip-address-type\":\"ipv6\",\"prefix\":64}],\"hardware-address\":\"bc:24:11:a9:b4:eb\",\"statistics\":{\"rx-errs\":0,\"tx-bytes\":319460,\"tx-dropped\":0,\"tx-packets\":4544,\"tx-errs\":0,\"rx-bytes\":29576554,\"rx-dropped\":0,\"rx-packets\":6027},\"name\":\"ens18\"}]}}"
}
@@ -0,0 +1,5 @@
{
"status": 200,
"reason": "OK",
"body": "{\"data\":\"UPID:pv1-1:0010F9DC:02801DA1:6AB37871:qmclone:9001:proxy-operator@pve!operator:\"}"
}
@@ -0,0 +1,5 @@
{
"status": 200,
"reason": "OK",
"body": "{\"data\":[{\"maxmem\":2147483648,\"netin\":0,\"uptime\":0,\"disk\":0,\"netout\":0,\"mem\":0,\"maxdisk\":34359738368,\"maxcpu\":2,\"type\":\"qemu\",\"memhost\":0,\"template\":0,\"id\":\"qemu/100\",\"cpu\":0,\"name\":\"ubuntu-vm-1\",\"vmid\":100,\"status\":\"stopped\",\"diskwrite\":0,\"diskread\":0,\"node\":\"pv1-1\"},{\"node\":\"pv1-1\",\"diskread\":0,\"diskwrite\":0,\"status\":\"stopped\",\"vmid\":9000,\"cpu\":0,\"name\":\"ubuntu-2604-template\",\"template\":1,\"id\":\"qemu/9000\",\"memhost\":0,\"type\":\"qemu\",\"maxcpu\":2,\"maxdisk\":34359738368,\"mem\":0,\"disk\":0,\"netout\":0,\"uptime\":0,\"netin\":0,\"maxmem\":2147483648},{\"uptime\":0,\"netout\":0,\"disk\":0,\"netin\":0,\"maxmem\":2147483648,\"maxdisk\":3221225472,\"mem\":0,\"id\":\"qemu/9001\",\"template\":1,\"name\":\"debian-13-template\",\"cpu\":0,\"type\":\"qemu\",\"maxcpu\":2,\"memhost\":0,\"diskwrite\":0,\"diskread\":0,\"node\":\"pv1-1\",\"vmid\":9001,\"status\":\"stopped\"}]}"
}
@@ -0,0 +1,5 @@
{
"status": 200,
"reason": "OK",
"body": "{\"data\":[{\"template\":0,\"id\":\"qemu/100\",\"cpu\":0,\"name\":\"ubuntu-vm-1\",\"type\":\"qemu\",\"maxcpu\":2,\"memhost\":0,\"diskread\":0,\"diskwrite\":0,\"node\":\"pv1-1\",\"vmid\":100,\"status\":\"stopped\",\"uptime\":0,\"disk\":0,\"netout\":0,\"netin\":0,\"maxmem\":2147483648,\"maxdisk\":34359738368,\"mem\":0},{\"type\":\"qemu\",\"vmid\":199999,\"lock\":\"clone\",\"status\":\"unknown\",\"id\":\"qemu/199999\",\"node\":\"pv1-1\"},{\"diskwrite\":0,\"diskread\":0,\"node\":\"pv1-1\",\"vmid\":9000,\"status\":\"stopped\",\"id\":\"qemu/9000\",\"template\":1,\"name\":\"ubuntu-2604-template\",\"cpu\":0,\"maxcpu\":2,\"type\":\"qemu\",\"memhost\":0,\"maxdisk\":34359738368,\"mem\":0,\"uptime\":0,\"netout\":0,\"disk\":0,\"netin\":0,\"maxmem\":2147483648},{\"type\":\"qemu\",\"maxcpu\":2,\"memhost\":0,\"template\":1,\"id\":\"qemu/9001\",\"cpu\":0,\"name\":\"debian-13-template\",\"vmid\":9001,\"status\":\"stopped\",\"diskwrite\":0,\"diskread\":0,\"node\":\"pv1-1\",\"netin\":0,\"maxmem\":2147483648,\"uptime\":0,\"disk\":0,\"netout\":0,\"mem\":0,\"maxdisk\":3221225472}]}"
}
@@ -0,0 +1,5 @@
{
"status": 200,
"reason": "OK",
"body": "{\"data\":[{\"id\":\"qemu/100\",\"template\":0,\"name\":\"ubuntu-vm-1\",\"cpu\":0,\"type\":\"qemu\",\"maxcpu\":2,\"memhost\":0,\"diskwrite\":0,\"diskread\":0,\"node\":\"pv1-1\",\"vmid\":100,\"status\":\"stopped\",\"uptime\":0,\"netout\":0,\"disk\":0,\"maxmem\":2147483648,\"netin\":0,\"maxdisk\":34359738368,\"mem\":0},{\"tags\":\"proxy_operator\",\"mem\":0,\"maxdisk\":4294967296,\"netin\":0,\"maxmem\":1073741824,\"uptime\":0,\"netout\":0,\"disk\":0,\"vmid\":199999,\"status\":\"stopped\",\"diskread\":0,\"diskwrite\":0,\"node\":\"pv1-1\",\"maxcpu\":1,\"type\":\"qemu\",\"memhost\":0,\"id\":\"qemu/199999\",\"template\":0,\"name\":\"proxy-reconaaaaaaaaaaa\",\"cpu\":0},{\"uptime\":0,\"netout\":0,\"disk\":0,\"netin\":0,\"maxmem\":2147483648,\"maxdisk\":34359738368,\"mem\":0,\"id\":\"qemu/9000\",\"template\":1,\"name\":\"ubuntu-2604-template\",\"cpu\":0,\"maxcpu\":2,\"type\":\"qemu\",\"memhost\":0,\"diskwrite\":0,\"diskread\":0,\"node\":\"pv1-1\",\"vmid\":9000,\"status\":\"stopped\"},{\"mem\":0,\"maxdisk\":3221225472,\"maxmem\":2147483648,\"netin\":0,\"uptime\":0,\"disk\":0,\"netout\":0,\"vmid\":9001,\"status\":\"stopped\",\"diskwrite\":0,\"diskread\":0,\"node\":\"pv1-1\",\"maxcpu\":2,\"type\":\"qemu\",\"memhost\":0,\"template\":1,\"id\":\"qemu/9001\",\"cpu\":0,\"name\":\"debian-13-template\"}]}"
}
@@ -0,0 +1,5 @@
{
"status": 200,
"reason": "OK",
"body": "{\"data\":{\"meta\":\"creation-qemu=11.0.3,ctime=1790145676\",\"description\":\"proxy-operator-managed=true\\nproxy-operator-uid=00000000-0000-0000-0000-00000000recon\\nproxy-operator-created-at=2026-09-23T07:00:00Z\",\"ide2\":\"local:iso/proxy-reconaaaaaaaaaaa-seed.iso,media=cdrom,size=44K\",\"cores\":1,\"memory\":\"1024\",\"scsihw\":\"virtio-scsi-pci\",\"net0\":\"virtio=BC:24:11:48:11:46,bridge=vmbr0\",\"efidisk0\":\"local-lvm:vm-199999-disk-0,efitype=4m,pre-enrolled-keys=0,size=4M\",\"scsi0\":\"local-lvm:vm-199999-disk-1,size=4G\",\"boot\":\"order=scsi0;ide2\",\"vmgenid\":\"2dd339a4-d7fb-4d47-96fd-5a61a25f808d\",\"digest\":\"98a4fff988b5508a7e1f6aa77c34768812d2382c\",\"name\":\"proxy-reconaaaaaaaaaaa\",\"vga\":\"serial0\",\"serial0\":\"socket\",\"smbios1\":\"uuid=f2ade202-ffdb-4a89-9867-3d419384471d\",\"machine\":\"q35\",\"bios\":\"ovmf\",\"tags\":\"proxy_operator\",\"agent\":\"1\",\"ostype\":\"l26\"}}"
}
@@ -0,0 +1,5 @@
{
"status": 200,
"reason": "OK",
"body": "{\"data\":\"UPID:pv1-1:00111FF7:02853E62:6AB38591:imgdel:proxy-seeds:proxy-operator@pve!operator:\"}"
}
@@ -0,0 +1,5 @@
{
"status": 200,
"reason": "OK",
"body": "{\"data\":\"UPID:pv1-1:00111FF3:02853D30:6AB3858E:imgdel:proxy-seeds:proxy-operator@pve!operator:\"}"
}
@@ -0,0 +1,5 @@
{
"status": 200,
"reason": "OK",
"body": "{\"data\":\"UPID:pv1-1:0014F167:030AA126:6AB4DB09:qmdestroy:199999:proxy-operator@pve!operator:\"}"
}
@@ -0,0 +1,5 @@
{
"status": 200,
"reason": "OK",
"body": "{\"data\":\"UPID:pv1-1:00110564:0281A657:6AB37C5E:qmdestroy:199999:proxy-operator@pve!operator:\"}"
}
@@ -0,0 +1,5 @@
{
"status": 500,
"reason": "QEMU guest agent is not running",
"body": "{\"message\":\"QEMU guest agent is not running\\n\",\"data\":null}"
}
@@ -0,0 +1,5 @@
{
"status": 500,
"reason": "No QEMU guest agent configured",
"body": "{\"message\":\"No QEMU guest agent configured\\n\",\"data\":null}"
}
@@ -0,0 +1,5 @@
{
"status": 500,
"reason": "VM 199999 is not running",
"body": "{\"message\":\"VM 199999 is not running\\n\",\"data\":null}"
}
+5
View File
@@ -0,0 +1,5 @@
{
"status": 400,
"reason": "Parameter verification failed.",
"body": "{\"data\":null,\"message\":\"Parameter verification failed.\\n\",\"errors\":{\"newid\":\"type check ('integer') failed - got 'abc'\"}}"
}
@@ -0,0 +1,5 @@
{
"status": 400,
"reason": "Parameter verification failed.",
"body": "{\"errors\":{\"storage\":\"storage 'local' does not support vm images\"},\"message\":\"Parameter verification failed.\\n\",\"data\":null}"
}
@@ -0,0 +1,5 @@
{
"status": 500,
"reason": "VM is locked (clone)",
"body": "{\"data\":null,\"message\":\"VM is locked (clone)\\n\"}"
}
@@ -0,0 +1,5 @@
{
"status": 500,
"reason": "Configuration file 'nodes/pv1-1/qemu-server/199999.conf' does not exist",
"body": "{\"data\":null,\"message\":\"Configuration file 'nodes/pv1-1/qemu-server/199999.conf' does not exist\\n\"}"
}
+5
View File
@@ -0,0 +1,5 @@
{
"status": 403,
"reason": "Permission check failed (/nodes/pv1-1, Sys.Syslog)",
"body": "{\"message\":\"Permission check failed (/nodes/pv1-1, Sys.Syslog)\\n\",\"data\":null}"
}
@@ -0,0 +1,5 @@
{
"status": 403,
"reason": "Permission check failed (/storage/local, Datastore.Allocate)",
"body": "{\"data\":null,\"message\":\"Permission check failed (/storage/local, Datastore.Allocate)\\n\"}"
}

Some files were not shown because too many files have changed in this diff Show More