egress-proxies-operator/docs/plans-executions/2026-09-18-1502-proxmox-provider.md

Execution: proxmox, a Proxmox VE provider built from the provider contract

Pairs with docs/plans/2026-09-18-1502-proxmox-provider.md.

Status

• Phase 0 — Plan file, branch, host recon, fixtures
• Phase 1 — The spec (docs/proxmox-provider-spec.md)
• Phase 2 — Config block, registration, package skeleton
• Phase 3 — Extract nocloud and keyedlock from kvm2
• Phase 4 — Seam, HTTP client, decorator, fake
• Phase 5 — Get, ListByTag, task helpers
• Phase 6 — Create
• Phase 7 — Delete, Checker, CRD comment
• Phase 8 — Contract and observability sweep, integration test, testing.md
• Phase 9 — Deployment and documentation
• Phase 10 — Review, real-host validation, changelog, MR

Phase 0 — Plan file, branch, host recon, fixtures

The plan was committed alone on main on 2026-09-18 (708989c, then 2167819 retargeting it at PVE 9.2). origin/main had not moved in the meantime, so the branch was cut straight from local main:

git fetch -q origin && git rev-list --left-right --count origin/main...main   # 0 2
git switch -c feat/proxmox-provider

The host is pv1-1, a single node (no cluster peers), pve-manager/9.2.20, kernel 7.0.14-17-pve, QEMU 11.0. The user prepared the token (~/.config/proxmox-token, privsep off) and the CA (~/.config/proxmox-ca.pem) before the session.

TLS: 192.168.2.8 is not in the node certificate. The first probe failed with curl exit 60. The CA is right (openssl s_client -CAfile verifies the chain), but the leaf's SANs are 127.0.0.1, ::1, localhost, 192.168.0.8, pv1-1, pv1-1.home.lab, and neither name resolves on the workstation. 192.168.0.8 and 192.168.2.8 present the same certificate fingerprint: the node sits on vmbr0 (192.168.0.8/24, untagged) with VLAN interfaces vmbr0.2 (192.168.2.8/24) and vmbr0.4 (192.168.4.8/24). All recon kept full verification by connecting to 192.168.2.8 while verifying the certificate as pv1-1.home.lab:

curl -s --resolve pv1-1.home.lab:8006:192.168.2.8 --cacert ~/.config/proxmox-ca.pem \
  -H @<(printf 'Authorization: PVEAPIToken=%s\n' "$(tr -d '[:space:]' < ~/.config/proxmox-token)") \
  https://pv1-1.home.lab:8006/api2/json/version
# {"data":{"release":"9.2","version":"9.2.20","repoid":"49318c671b82f31e"}}

Every capture went through a scratch helper wrapping exactly that call. It passes the token through a file descriptor so it never reaches argv or output, and writes {"status","reason","body"} to internal/provider/proxmox/testdata/<name>.json, refusing to save a body that contains PVEAPIToken.

A real NoCloud seed was used instead of the plan's 1 KiB placeholder, so the agent could be installed and F21 captured from a live guest. No ISO tool is installed locally, so a throwaway module in the scratchpad reused the repository's github.com/kdomanski/iso9660 v0.4.0 from the module cache and copied kvm2's guestAgentScript verbatim:

GOFLAGS=-mod=mod GOPROXY=off GOSUMDB=off go run . proxy-reconaaaaaaaaaaa proxy-reconaaaaaaaaaaa-seed.iso
# 45056 bytes, ISO 9660 'cidata'

Write experiments ran against VMID 199999 (proxy-reconaaaaaaaaaaa), a clone of template 9001. Sequence: validation error; clone; resources, config write and start while the clone lock was held; status and agent while running bare; stop; config read; clone onto the taken VMID; tag tests; upload twice plus a bad name; attach ide2, set cores and memory; three resizes; start and agent poll; suspend and resume; a clone onto a storage without image support; stop; destroy with purge=1&destroy-unreferenced-disks=1; destroy again; ISO delete.

Facts F1 to F23 (cited as Evidence by the spec)

• F1. Errors carry the text twice: in the HTTP reason phrase and as the body's message, which ends in \n. A 400 adds errors, a map from parameter to message. A 401 has the reason phrase only and an empty body. The client should use the trimmed body message and fall back to the reason phrase. Headers are ordinary CRLF.
• F2. Missing VM, identical for GET config, GET status/current and DELETE: 500 Configuration file 'nodes/pv1-1/qemu-server/999999.conf' does not exist.
• F3. A start issued while the VM is clone-locked is accepted (200 with a UPID). The task log reads trying to acquire lock... OK and the VM starts once the clone finishes. Setting a lock through the API was not tried.
• F4. Clone onto a taken VMID, synchronous: 500 unable to create VM 199999: config file already exists.
• F5. Agent enabled but not running: 500 QEMU guest agent is not running. Agent not enabled in the config: 500 No QEMU guest agent configured. Stopped VM with agent: 1: 500 VM 199999 is not running.
• F6. Not produced. local-lvm is thin and overprovisions, so quota: absent.
• F7. Config write while clone-locked: 500 VM is locked (clone).
• F8. /cluster/resources?type=vm entries carry vmid, name, node, status, type, template, tags, maxcpu, maxmem, maxdisk, uptime and more; lock appears only when set. Status values seen: running, stopped, and unknown during a clone. Mid-clone the entry has no name: only type, vmid, id, node, lock: clone, status: unknown.
• F9. Hyphenated tags are accepted and stored as given. Uppercase is folded (Proxy_Operator → proxy_operator), the separator is ;, and characters such as a space or ! give 400 … tags: invalid format - invalid characters in tag. The datacenter has no tag-style or user-tag-access set.
• F10. description on clone is accepted and survives verbatim, newlines included. The 22-character proxy-<base32> name is accepted.
• F11. The clone's meta is copied from the template (creation-qemu=11.0.3,ctime=1790145676, identical to 9001), so it says nothing about when the clone was made. written-at-create is required.
• F12. Upload: multipart with field content=iso and the file part named filename, carrying the target file name. Returns a UPID for an imgcopy task. A name without .iso gives 400 … filename: wrong file extension. Re-uploading an existing name silently overwrites it (the task log shows a plain cp). Datastore.AllocateTemplate suffices.
• F13. Deleting a volume needs Datastore.Allocate on /storage/<id>; without it the call gives 403 Permission check failed (/storage/local, Datastore.Allocate). With it (on proxy-seeds), the delete runs as an imgdel task and returns a UPID. Deleting a volume that no longer exists is also accepted and its task still ends OK; the log reads disk image '…' does not exist, then Removed volume. Seed cleanup is idempotent with no message matching.
• F14. Stopped: status: stopped, qmpstatus: stopped. Paused: status: running, qmpstatus: paused in status/current, while /cluster/resources shows plain running.
• F15. A clone onto a storage without image support is rejected up front (400 … storage 'local' does not support vm images) and leaves no VM. A clone failing mid-task was not observed.
• F16. Task status fields: status (running, stopped), exitstatus (OK or the error text), type, id, node, user, tokenid, upid, starttime, pid, pstart. A full clone of the 3 GiB template onto local-lvm finished within about two seconds.
• F17. The planned role, granted on /, covered clone, config, resize, start, stop, suspend, agent read, upload and destroy. Missing: Datastore.Allocate for deleting the seed ISO. VM.Config.Cloudinit is not needed for the ide2 write. SDN.Use was held but never exercised, because no call rewrote net0.
• F18. HTTP/1.1 only: curl --http2 negotiates HTTP/1.1.
• F19. Template 9001 (debian-13-template): scsi0 on local-lvm, boot: order=scsi0, ide2 free, agent: 1, OVMF, q35, net0 untagged on vmbr0. Attaching ide2 makes Proxmox append it to the boot order (order=scsi0;ide2). Template 9000 (Ubuntu) is unsuitable: it has a Proxmox cloud-init drive on ide2 and VLAN tag 4 on net0.
• F20. Resize runs as a task (returns a UPID). The same size is an OK no-op; shrinking fails inside the task with exitstatus shrinking disks is not supported.
• F21. Agent response: result[] of {name, ip-addresses[{ip-address, ip-address-type}]}. The live guest reported lo and ens18 with 192.168.0.150, 50 seconds after start, via DHCP on untagged vmbr0. This proves the NoCloud-seed-on-ide2 route end to end on this host.
• F22. A wrong secret and an unknown token id both give 401 Authentication failed!; no credentials gives 401 No ticket.
• F23. Uploads are staged in /var/tmp/pveupload-* and then copied to /var/lib/vz/template/iso/, so the tmpfs /tmp is not involved.

Corrections to the plan that Phase 1 must fold into the spec

1. Check lock before the name. A mid-clone VM has no name in /cluster/resources (F8), so name-first logic turns a clone in progress into NotFound. Get, Delete and Create's lookup must treat a locked or nameless entry at our VMID as in-progress (Transient or wait), never absent.
2. Inspect a taken VMID before probing forward. After a crash mid-clone, the retry's name lookup misses the nameless VM, the clone onto the deterministic VMID fails with F4, and forward probing would clone a duplicate. On F4, read that VMID: lock: clone means wait for it, and our UID in the description means adopt it. Probe forward only for a VM that is demonstrably someone else's.
3. A start queues behind a lock instead of failing (F3).
4. Resize is asynchronous (F20); wait on its task.
5. Drop delete-before-upload for the seed: upload overwrites (F12). Seed deletion is a task (F13): wait on it. A repeat delete succeeds, so Delete needs no absent-volume matcher.
6. Get reads status/current, not only /cluster/resources, to tell paused from running (F14).
7. Tag rationale: hyphens are accepted on 9.2.20 (F9). proxy_operator stays, since it is valid everywhere, but the claim that hyphens are rejected is removed.
8. The boot-disk logic must skip ide2, which Proxmox adds to the boot order (F19). The plan's rule already excludes it; add a test with order=scsi0;ide2.
9. Three agent messages, not one (F5): QEMU guest agent is not running (booting, or the install failed), No QEMU guest agent configured (our agent=1 was removed), and VM <vmid> is not running (stopped between the status read and the agent call). The spec decides each class; none may become NotFound.
10. Internet on the proxy network is a hard prerequisite, twice over: the vendor-data agent install needs a package mirror, and the proxy itself needs egress. The guide says so, and recommends baking qemu-guest-agent into the template with installGuestAgent: false to drop the mirror dependency from IP discovery.

Decisions made with the user in Phase 0

• Seeds go to a dedicated directory storage, proxy-seeds, with ISO content only. The token gets Datastore.Allocate there and nowhere else, so it can delete seeds but never backups or other ISOs on local. The sample config's isoStorage becomes proxy-seeds. The guide must say that an ACL on a more specific path replaces the inherited one for that user, so the role granted on /storage/proxy-seeds must repeat Datastore.AllocateTemplate, AllocateSpace and Audit alongside Allocate.
• Proxies land on a VLAN set in config. ProxmoxConfig gains an optional vlanTag (1 to 4094), applied when Create rewrites net0 together with the optional bridge override, keeping the MAC. Unset leaves net0 as cloned. This is the first call that exercises SDN.Use (F17).
• The API URL is https://192.168.0.8:8006, the address in the node certificate, so caFile verifies with no provider change. No tlsServerName option. The operator's cluster must be able to reach 192.168.0.x.

Seed storage and VLAN 4

The user created proxy-seeds (dir, ISO content only), granted the token Datastore.Allocate, AllocateSpace, AllocateTemplate and Audit on /storage/proxy-seeds through a role ProxyOperatorSeeds, and freed the leftover recon ISO on local. An upload to proxy-seeds and two deletes of a throwaway ISO there produced content_delete_response.json, content_delete_again_response.json and task_status_volume_missing.json.

The user gave the VLAN as "vlan 4, bridge: vmbr0.4@vmbr0". That is the host's own VLAN interface name as ip link prints it; a VM NIC cannot attach to it because it is not a bridge. On the VLAN-aware vmbr0 the Proxmox form is bridge=vmbr0,tag=4, which is what template 9000 uses, so the provider's vlanTag: 4 appends tag=4 to the cloned net0.

A second clone of 9001 at VMID 199999 got its seed from proxy-seeds and net0 rewritten from virtio=BC:24:11:EE:48:83,bridge=vmbr0 to virtio=BC:24:11:EE:48:83,bridge=vmbr0,tag=4. The write succeeded, so the token's SDN.Use on / covers a VLAN tag on vmbr0 (set_config_vlan_response.json).

The first VLAN 4 attempt never produced an agent in seven minutes of polling (every 13 seconds, 33 polls). The only signal was the agent call, so the cause could not be seen from the API. A debug seed added a recon user (user-data only, with a new instance id so cloud-init re-ran), and the user logged in on the console: DHCP worked, but VLAN 4 had no route to the internet because of a routing misconfiguration on the Proxmox host, so the vendor-data apt-get install qemu-guest-agent could not run. The user fixed the routing.

Rerun from scratch after the fix, same sequence the provider will use (destroy and seed delete first, then a fresh clone and seed, instance-id equal to the name):

Step	Elapsed
clone task done	4 s
seed upload task done	4 s
config (ide2, net0 with tag=4, tag)	4 s
start task done	5 s
agent reports 192.168.4.147	47 s after start

The guest was reachable from the workstation (ping, TCP 22), as the contract requires of a proxy address. Stopping it gave the F5 stopped-VM fixture; destroy and seed delete then left the host with only its original VMs 100, 9000 and 9001, no ISOs on local or proxy-seeds, and no local-lvm volumes of 199999. The capture is agent_network_get_interfaces_vlan.json.

Worth noting

• The token file is mode 664; 600 would be tighter.
• Fixtures: 48 files, all valid JSON, none containing PVEAPIToken. Several names differ from the plan's list because what they captured differed from what was expected: start_while_clone_locked_response.json (the start succeeded), task_status_stopped_error.json (from the shrink, not a start), upload_overwrite_response.json (success, not an error), resize_same_size_response.json and resize_shrink_response.json (UPIDs), err_forbidden_volume_delete.json (the 403 from F13). err_no_space.json was not produced (F6).
• The host is back to its original state; see the table above.
• The scratch seed builder (seedgen in the session scratchpad) took the user-data file and instance id as arguments by the end, for the debug login. It is not part of the repository; Phase 3's nocloud package replaces it.
• Time to IP was about 50 s on both networks, with the agent installed at first boot. Budgets in the plan (running wait 60 s for the VM state, the agent discovered by Get polling) are consistent with this.

Phase 1 — The spec

Wrote docs/proxmox-provider-spec.md, all 24 entries of the template, from the plan, the contract and the Phase 0 facts. The ten Phase 0 corrections are folded in, and the plan now opens with a note that the spec wins where the two disagree.

Judgment calls made here, each recorded in its entry:

• Create's allocation loop (entry 2) runs an ownership check on any taken VMID: wait out a lock or a nameless entry, then adopt when the name and UID are ours, skip when the name differs, refuse as Permanent when the name is ours but the mark is not. This is the fix for the duplicate-VM bug.
• Get's check order (entry 9): lock, then name and mark from the config, then status/current, then the agent. lock: suspended alone maps to Stopped, since a hibernated VM never resumes by itself.
• Agent messages (entry 7): QEMU guest agent is not running is Provisioning, VM <vmid> is not running is Transient, and No QEMU guest agent configured is Stopped, so the reconciler replaces a VM that can never report an address instead of latching the Proxy.
• Templates are checked before cloning (entry 4): not a template, or a Proxmox cloud-init drive in any slot, is Permanent. This makes the "no cloud-init drive" prerequisite enforceable instead of a silent failure.
• isoStorage is required, with no default (entry 15), because the token must hold Datastore.Allocate on it.
• TLS verification failures are Permanent, with a Checker reason CertificateInvalid (entries 13, 16), since the fix is the URL or CA file.
• The CA is loaded lazily and cached; the token is read per request (entry 14).

Checks, as the plan's Verify clause asks:

grep -c '^## [0-9]*\. ' docs/proxmox-provider-spec.md     # 24
grep -n '^\*\*Answer:\*\*$' docs/proxmox-provider-spec.md # nothing
grep -c '—' docs/proxmox-provider-spec.md                 # 0

plus a script comparing every `<axis>`: `<profile>` pair against the contract's §4.20 table (no mismatches, every axis answered) and a list of the F numbers cited (all 23).

Worth noting: the first pass stated two things as fact that recon could not show and was corrected before commit. That an unreachable node's guests stay listed as unknown is Proxmox's documented design, unverifiable on a single node; entry 13 now says so. And the matcher for VM <vmid> is not running is a whole-message regexp, because is not running alone also matches QEMU guest agent is not running.

Phase 2 — Config block, registration, package skeleton

Added the proxmox: block (internal/provider/config_proxmox.go), wired it into config.go (the field, setBlocks, the allow-list and the dispatch: three "proxmox" sites) and registered "proxmox": proxmox.New in cmd/main.go, so the two type lists changed in one commit. The package internal/provider/proxmox now holds the seam interface and its data types (api.go), id.go, mark.go, state.go, errors.go and the constructor (proxmox.go). The four contract methods are stubs that answer ErrTransient, so a Proxy on a configured proxmox entry retries instead of latching until Phases 5 to 7 land. New passes a nil api until Phase 4 builds the HTTP client.

Every test that depends on Proxmox's wording or JSON shape reads the Phase 0 captures through fixtures_test.go (readCapture, loadAPIError, decodeData), so no message string is repeated in a test. The one exception is TestClassify_table's synthetic rows (a 404, 502, 595, 596, a storage message), for statuses recon never produced.

Deviations from the plan's Phase 2 list, both making it smaller:

• No temporary locks.go. Only Create and Delete take the per-name lock, and they arrive after Phase 3 extracts internal/provider/keyedlock, so copying kvm2's lock now would only be deleted next phase.
• No provider.Checker assertion yet. cmd/main.go type-asserts the Checker and would call a stub; the assertion comes with the real Check in Phase 7.

Added beyond the plan: api_test.go decodes the captured responses into the seam's types, so a misspelled JSON tag fails now rather than in Phase 5, and TestParseDescription_fromClone reads the mark back from the description of the VM cloned in Phase 0.

A mutation check confirmed the suite can fail. Matching VM <vmid> is not running as a plain suffix (the first spec draft's mistake) failed TestAPIError_matchers on the agent-not-running capture; treating a paused VM as running failed TestMapState_table and TestMapState_fromCaptures. Both mutations were reverted and the files compared identical to backups.

gofmt -w cmd/main.go            # realigned the constructor map for the longer key
grep -c '"proxmox"' internal/provider/config.go                     # 3
go build ./... && go vet ./...
go test -race -count=1 ./internal/provider/... ./cmd/...            # all ok
go test -short -count=1 ./...                                       # all ok
go test -count=1 -v ./internal/provider/proxmox/ | grep -c '^=== RUN' # 152

Worth noting: the validProxmoxYAML constant is the shape Phase 9's commented sample entry should take, with the user's values (192.168.0.8, pv1-1, proxy-seeds, vlanTag: 4, template 9001).

Phase 3 — Extract nocloud and keyedlock from kvm2

kvm2's seed.go and seed_test.go moved to internal/provider/nocloud with git mv, so history follows them. The package exports BuildSeedISO, BuildVendorData, GuestAgentScript, SeedLabel and a new ErrEmptyName sentinel in place of kvm2's permanentf. kvm2's keyedLocks became internal/provider/keyedlock (Locks, New, Acquire), and its three TestKeyedLocks_* tests moved with it under their old names. kvm2 now imports both packages, and its create.go still wraps every seed-build error with permanentf, so kvm2's behaviour is unchanged. proxmox gained the nameLocks field and lockName.

The test counts match the plan's Verify clause: kvm2 dropped from 465 to 447 === RUN entries, which is the 12 moved tests plus the 6 subtests of TestSeedISO_userDataByteForByte. nocloud (15) and keyedlock (3) carry exactly those 18.

git mv internal/provider/kvm2/seed.go internal/provider/nocloud/seed.go
git mv internal/provider/kvm2/seed_test.go internal/provider/nocloud/seed_test.go
go test -count=1 -v ./internal/provider/kvm2/ | grep -c '^=== RUN'   # 465 before, 447 after
grep -rn 'iso9660' internal/provider/kvm2/                          # nothing
go build ./... && go vet ./... && go vet -tags=integration ./...
go test -race -count=1 ./internal/provider/...                      # all ok, run three times
go test -short -count=1 ./...                                       # all ok

Pulled forward from Phase 6 so proxmox imports nocloud now, as the "Done when" line requires: seed.go with seedISOName, seedVolID and buildSeed, the pure part of the seed step. uploadSeed still belongs to Phase 6, which calls buildSeed. The new tests are TestSeedNames_table, TestBuildSeed_table (4 rows), TestBuildSeed_emptyNameIsPermanentWithoutPayload, TestBuildSeed_stagingFailureIsTransient and TestLockName_givesUpWithContextAsTransient. proxmox went from 152 to 161 test entries.

proxmox does not copy kvm2's open review finding S5.1 (docs/reviews/2026-09-17-kvm2-provider-review/sweep.md). In kvm2, a staging failure such as a missing /tmp or a full emptyDir becomes Permanent, and that latches the Proxy. buildSeed makes only ErrEmptyName Permanent and wraps any other error with %w, so classify reads it as Transient. The sentinel makes the same three-line fix possible in kvm2, but this phase leaves kvm2's behaviour unchanged.

Worth noting:

• The older kvm provider has its own seed.go with a different signature (buildSeedISO returns []byte). The plan scopes the extraction to kvm2, so kvm is untouched.
• GuestAgentScript now says the operator reads the address "from the hypervisor" instead of "through libvirt", in both the Go comment and the script's comment line inside the guest. The script's commands are unchanged.
• The docs now name the new packages: kvm2-provider.md, kvm2-provider-spec.md entry 19, the kvm2 columns and section of providers-existing.md, plus a "Shared provider helpers" row in the architecture.md component table. docs/reviews/ keeps the old names, because it records the code as it was reviewed.
• TestCheck_hostsCheckedConcurrently in kvm2 failed once under the full -race suite (latency 199.96 ms against a 200 ms floor). It passed 20 runs out of 20 in isolation, with and without this phase's changes. The test starts its 200 ms release timers before Check starts its clock, so under load the measured latency can come in just under 200 ms. The flake predates this phase, and the test is not touched here.

Phase 4 — Seam, HTTP client, decorator, fake

client.go holds httpAPI, the thin net/http adapter for all sixteen seam methods. It reads the token file on every request, turns any non-2xx answer into *apiError from the body's message or the reason phrase (F1), and wraps its transport in tracing.RestConfigWrapper. observe.go is the logging decorator: one V(1) proxmox api call line per call, detail at V(2), and no span, because the transport already traces each request. New now builds observe(newHTTPAPI(...)) instead of passing a nil api, still without touching the network or the token file. fake_test.go holds fakeAPI, which Phases 5 to 7 build on.

Where this phase departs from the plan's Phase 4 text:

• The http.Client is built on first use, not in newHTTPAPI. Spec entry 14 wins over the plan: the CA is loaded lazily, a missing CA file is Transient and retried until the file lands, one without a certificate is Permanent, and the pool is cached only once it loads. A transport's RootCAs cannot change after first use, so the whole client waits for the CA (TestClient_caFileLoadedLazily).
• The upload is not an io.Pipe with a goroutine. The multipart framing (the content=iso field, the filename part header and the closing boundary) is rendered into two small buffers, and the body is io.MultiReader(head, io.LimitReader(r, size), tail) with an exact Content-Length. The seed is still streamed and never held by the client. There is also no goroutine to leak, and the request carries a length, as Phase 0's curl -F upload did. A chunked upload to pveproxy was never tried. TestClient_uploadStreamsMultipart proves the streaming with a reader that stalls after its first 256 KiB until the server has seen file bytes.
• The fake's DeleteVolume tolerates a missing volume, as spec correction 5 and F13 say, instead of the plan's error.
• Added: secret redaction in apiError (contract §7), because otherwise a server that echoes the header would put the secret into Proxy status; an errCredentials sentinel on token-file errors, so Phase 7's Checker can report CredentialsUnavailable; and the V(1) identifiers newid and name on a clone, filename on an upload and volid on a volume delete. Spec entry 20 now lists them.

The volid's / travels as %2F. Neither the plan nor the spec says how {volid} is encoded in the content-delete path. The Phase 0 session's transcript shows every delete went through a helper encoding the whole volid:

enc() { python3 -c "import urllib.parse,sys; print(urllib.parse.quote(sys.argv[1], safe=''))" "$1"; }
curl ... -X DELETE ".../storage/proxy-seeds/content/$(enc 'proxy-seeds:iso/proxy-reconaaaaaaaaaaa-seed.iso')"

So apiPath runs url.PathEscape over each segment (TestClient_volIDSlashIsEscaped).

net/http cannot send a custom reason phrase, so the client tests replay every capture byte for byte through a hijacked connection (serveCapture). TestClient_apiErrorFromStatusReasonAndBody globs testdata/err_*.json and skips err_tag_hyphen.json, which is a 200 despite its name: hyphenated tags were accepted (F9).

Tests beyond the plan's list: TestClient_methodPathAndParams (the method, path, query and form of every seam method, with a reflection check that no method lacks a row), TestClient_cloneOmitsEmptyStorageAndTarget, TestClient_volIDSlashIsEscaped, TestClient_caFileLoadedLazily, TestClient_x509FailureIsPermanent (from spec entry 21), TestDecodeConfig_scalarsAsStrings, TestReadToken_trimsTrailingNewline, TestObserve_startsNoSpan, TestObserve_identifiersAtV1, TestNew_wiresObservedHTTPClient, and six TestFakeAPI_* tests. TestFakeAPI_errorsMatchCaptures compares the fake's error texts with the captures, so every test that goes through the fake exercises the real wording. Texts that recon never produced start with fake:.

A mutation check confirmed the new tests can fail: dropping redact, buffering the upload with io.ReadAll, and logging V(2) detail at V(1) each failed their test. Removing only the empty TLSNextProto map did not fail TestClient_http1Only. A transport with a custom TLSClientConfig and DialContext never offers h2 unless ForceAttemptHTTP2 is set, so the map is a second guard. With both guards removed the test failed with client spoke HTTP/2. Every mutation was reverted, and the files were compared identical to backups.

go test -race -count=2 ./internal/provider/proxmox/          # ok, run three times
grep -n 'ReadAll' internal/provider/proxmox/client.go        # only the 64 KiB error-body read
grep -n 'context.Background()' internal/provider/proxmox/*.go | grep -v _test.go   # nothing
go build ./... && go vet ./... && go vet -tags=integration ./...
go test -short -count=1 ./...                                # all ok
go test -count=1 -v ./internal/provider/proxmox/ | grep -c '^=== RUN'   # 161 before, 289 after

Worth noting:

• bin/golangci-lint run ./internal/provider/proxmox/... panics inside staticcheck's IR builder (unexpected expr: *ast.KeyValueExpr in the Go 1.27 standard library's poll package). This is a toolchain problem, not this code. With --disable staticcheck the new files' findings are goconst on form keys (kvm2 carries 50 of the same) and two unparam hits on the fake's addTemplate and resourceByID, which Phases 5 and 6 will call with other VMIDs. The mapsloop and gocyclo findings in proxmox.go and api_test.go date from Phase 2.
• Nothing in this phase talked to the host. The client has only met httptest servers replaying the captures. The first real request is Phase 10's integration run, which is where the Content-Length upload and the %2F volid get proven against pveproxy.
• The fake refuses to destroy a running VM with an uncaptured fake: VM <vmid> is running - destroy failed, so Phase 7's stop-before-destroy order is enforced by the fake even though the real text is unknown.

Phase 5 — Get, ListByTag, task helpers

task.go holds parseUPID, a shared pollUntil loop, waitTask and waitUnlocked. get.go holds Get, the guest-agent read and resolve (one resources call, QEMU entries only, which Delete will reuse). list.go holds ListByTag and its candidate rule. The Get and ListByTag stubs are gone, and TestStubs_answerTransient now covers only Create and Delete.

The spec overrides the plan in the places the Phase 4 notes listed. Get checks in entry 9's order, so a lock or status: unknown in the resources entry is Transient before the name is ever read, and a running Get makes four calls (TestGet_runningMakesFourCalls). The three agent messages follow entry 7. The poll interval is budgets.poll, and a failed task logs a V(1) proxmox task failed line with upid and exitstatus.

Judgment calls, most of them spec gaps:

• WARNINGS: <n> counts as a successful task. Proxmox reports a task that finished but logged warnings with that exitstatus and treats it as finished, not failed. Under the spec's old "anything but OK is Transient" rule, a start that logged a warning would make Create retry forever. Recon never produced one, so this row relies on how Proxmox is known to behave, not on a capture. The spec's entry 10 text and entry 13 table now say so.
• A suspended lock still goes through the name and mark checks. The entry 9 table puts "lock suspended is Stopped" before the config rows. Taken literally, a hibernated VM that reused our VMID would be reported as ours and Stopped, and Delete would then skip it as foreign in a loop. Get therefore reads the config, which it needs anyway for UID and CreatedAt, applies the name and mark checks, and only then answers Stopped. It skips status/current and the agent. Entry 9 now says this.
• TestGet_twoVMsWithOurNameIsPermanent became TestGet_otherVMWithOurNameIsIgnored. Spec entries 2 and 9 address Get by VMID, and only Create refuses a duplicate name. Get answering Permanent would latch a working Proxy as Failed because of a VM someone else created.
• TestGet_unmanagedDescriptionHasEmptyUID tests a mark with no uid line. The plan's name read literally ("unmanaged") would contradict "no mark is NotFound".
• Agent errors outside the entry 7 table are forced to Transient, even a 403 or a does not exist. Otherwise classify would make them Permanent or NotFound, and the spec says "anything else: Transient".
• waitUnlocked returns the unlocked config, which saves Create's ownership check one read. When the VM disappears while it waits (a failed clone), it returns the does-not-exist error and lets Create decide.

Pulled forward and left behind:

• provider_test.go now exists with newFixture, testBudgets, addOurVM and assertProviderError, because every Get and ListByTag test needs them. The plan created this file in Phase 8. Phase 8 adds the contract sweep to it.
• waitRunning is left to Phase 6. The plan's package layout puts it in task.go, but Phase 5 lists no test for it and only Create uses it. It should be built on pollUntil.
• The fake gained two things. fakeVM.kind makes an entry listed as lxc invisible to the qemu endpoints. hookOn is a general per-method hook, and failWith is now built on it. Tests use hookOn (through mutateOn in get_test.go) to change a VM between two of Get's calls.

A mutation check confirmed the new tests can fail. Each of these failed a test: classifying agent errors instead of forcing Transient; dropping the config lock check; dropping the suspended special case; ignoring an ended ctx in ListByTag; treating WARNINGS as failure; and not mapping No QEMU guest agent configured to Stopped. Removing the resources-entry lock and unknown checks together failed only TestGet_unknownStatusIsTransient. The fake reports a lock in the config as well, so the config check catches every lock row, including a mid-clone VM. The resources check stays because the spec orders it first and because it spares a config read. Every mutation was reverted, and the files were compared identical to backups.

go test -race -count=2 ./internal/provider/proxmox/                     # ok
grep -n 'not implemented' internal/provider/proxmox/*.go | grep -v _test # Create and Delete only
go build ./... && go vet ./... && go vet -tags=integration ./...
go test -race -count=1 ./internal/provider/... ./cmd/...                 # all ok
go test -short -count=1 ./...                                            # all ok
go test -count=1 -v ./internal/provider/proxmox/ | grep -c '^=== RUN'   # 289 before, 393 after

Lint ran with staticcheck disabled, because staticcheck panics on Go 1.27:

bin/golangci-lint run --disable staticcheck ./internal/provider/proxmox/...

Worth noting:

• The lint findings in the new files are goconst on test strings and logcheck on get, agentAddress and waitTask taking both a ctx and a logger. That shape copies kvm2, which has five of the same findings. The errcheck hits on assertGetFails were fixed by returning the error text. The two unparam hits and the mapsloop hits predate this phase.
• waitUnlocked waits on any lock, suspended included. Phase 6's ownership check should decide whether a hibernated VM at our VMID is worth waiting out the clone budget for, or should be refused at once.
• Nothing in this phase talked to the host. Every Proxmox message a test relies on comes from testdata/ or from fake errors that TestFakeAPI_errorsMatchCaptures ties to the captures.

Phase 6 — Create

create.go holds Create and its steps: placement, the by-name lookup, the template check, the VMID probe with its ownership check (inspect), the clone, and provision (seed, config, resize, start). configParams, rewriteNet0, resizeTarget, bootDisk and diskBytes are pure helpers with their own table tests. seed.go gains uploadSeed, which uploads over any existing file with no delete first (F12). task.go gains waitRunning, built on pollUntil with budgets.running. It finishes only at running/running, and a stopped status after the start is Transient at once. The Create stub is gone, and TestStubs_answerTransient now covers only Delete.

The spec won over the plan wherever the Phase 6 notes said it would. The plan's allocation loop is replaced by entry 2: a taken VMID and a clone answering F4 both go through the ownership check before any probing, and a locked or nameless entry is waited on with waitUnlocked and the clone budget. The seed is uploaded without a delete. The resize is a task. net0 changes only when bridge or vlanTag is set, and tag= is replaced or appended in place.

Judgment calls, with spec entries 2, 4 and 10 updated to match:

• The Phase 5 open question: a suspended lock is not waited on. waitUnlocked now returns at once when it sees suspended, because a hibernated VM is not a task and would otherwise burn the 10-minute clone budget on every attempt. The ownership check then judges the VM the way Get does. A foreign hibernated VM is probed past. Ours is Permanent ("resume or destroy it"), following kvm2's precedent for a paused or pmsuspended domain in Create. Transient was the alternative: it would self-heal once an admin acts, but it would retry indefinitely on something only an admin can fix. TestWaitUnlocked_table has a new row, and TestCreate_hibernatedVMIsNotWaitedOn covers both cases.
• The template is checked only when a clone is needed. Spec entry 10 read the template's config in step 2, before the by-name lookup. A retry for a VM that already exists would then fail Permanent if the template had been deleted since, which entry 4 explicitly allows. The template check now runs after the by-name lookup misses, so a repeat Create costs three calls: resources, config and status (TestCreate_adoptionDoesNotNeedTheTemplate).
• The template's shape is checked before the clone. No net0 while bridge or vlanTag is set, or no boot disk (or none with a readable size=) while diskGiB is set, is Permanent. Found after the clone, the VM would already carry the bad config, so fixing the template could not help that Proxy's next attempt. template() runs the same configParams and resizeTarget over the template's config that provision later runs over the clone's.
• An empty req.Name is Permanent at placement. A nameless request would match nameless mid-clone entries in the by-name lookup.
• Adopted VMs get one status/current read rather than trusting the listing's status, which may be stale after a lock wait. A running VM, whatever its qmpstatus, returns its ID at once, as entry 10 says. Get then reports prelaunch as Provisioning and paused as Stopped.
• After the clone task, Create reads the config through waitUnlocked. That read supplies the net0 and boot disk the later steps need, and it turns a clone that ended but kept its lock (F15) into a Transient lock wait rather than a refused config write.
• An F4 whose VMID then resolves to no QEMU VM probes on. Either the racer's clone failed and removed it, or a container took it.
• Errors carry no ID (wrapErr("create", "", …)), since the VMID is known only after allocation. The messages name the VMID wherever one exists.

Test names: every plan and entry 21 name exists except TestCreate_resizeOnlyWhenLarger. It is covered by the spec's pair: TestCreate_resizeWaitsForTask for the larger case and TestCreate_resizeSkippedWhenNotLarger for equal, smaller and unset. Tests beyond both lists: TestCreate_vmThatStopsAfterStartIsTransientAtOnce, TestCreate_hibernatedVMIsNotWaitedOn, TestCreate_templateShapeRefusedBeforeClone, TestCreate_adoptionDoesNotNeedTheTemplate, TestCreate_cloneTargetAndStorage (read from the decorator's V(2) clone line), TestCreate_adoptedVMStaysOnItsNode, TestCreate_outcomeLine, TestCreate_quietAtV0, TestRewriteNet0_table, TestBootDisk_capturedClone and TestDiskBytes_table. The only new test helpers are the small ones in create_test.go (slot, newCreateFixture, req/create/createFails, assertRunning, callsBefore, clearLockOnVMConfig). Everything else reuses the Phase 5 fixture and fake. Two tests check config-write contents without a recording wrapper. TestCreate_markWrittenInCloneCall reads the VM at the start of UploadISO, which is after the clone and before the config write. TestCreate_noBridgeOverrideLeavesNet0 changes net0 at the start of SetVMConfig, so a write that carried net0 would put the old value back.

A mutation check applied 18 mutations, and each failed at least one test. They were: no lock wait in the ownership check; taken VMIDs never inspected; the F4 race skipping the ownership check; no cloud-init drive check; upload or resize task not waited on; resize at equal size; a CD-ROM counted as a boot disk; prelaunch counted as running; the suspended lock waited on; net0 always written; bridge= appended instead of replaced; duplicate names ignored; a running VM re-provisioned; target always set; a foreign UID adopted; the tag dropped; probes not capped by the range. The last one was first missed. A 4-VMID range with 32 probes wraps, re-inspects each VM eight times and still ends Transient, so TestCreate_vmidRangeExhaustedIsTransient now pins five config reads. Five mutations did not compile on the first try and were redone. Every mutation was reverted, and the files were compared identical to backups.

go test -race -count=2 ./internal/provider/proxmox/                          # ok
grep -n 'CloudInit' internal/provider/proxmox/create.go internal/provider/proxmox/seed.go
#   seed.go: the pass-through into nocloud.BuildSeedISO; create.go: len(req.CloudInit)
grep -n 'not implemented' internal/provider/proxmox/*.go | grep -v _test      # Delete only
grep -n 'context.Background()' internal/provider/proxmox/*.go | grep -v _test.go   # nothing
go build ./... && go vet ./... && go vet -tags=integration ./...
go test -short -count=1 ./...                                                 # all ok but the parked kvm2 flake, below
go test -count=1 -v ./internal/provider/proxmox/ | grep -c '^=== RUN'         # 393 before, 519 after
bin/golangci-lint run --disable staticcheck ./internal/provider/proxmox/...

Worth noting:

• go test -short ./... failed once on the parked kvm2 flake TestCheck_hostsCheckedConcurrently (measured 199.98 ms against a 200 ms floor). Three reruns passed. This phase does not touch kvm2.
• The new lint findings follow the baseline's shape. There are logcheck hits on create, allocate and clone (ctx plus logger, as in kvm2), and goconst on test strings. One more unparam than the notes list: (*fakeAPI).iso's storage always receives fakeISOStorage, because this phase is its first caller. addTemplate's hit changed from a constant VMID to an unused result.
• The config write sets agent=1 and tags=proxy_operator as values. A template's agent options (such as fstrim_cloned_disks=1) and its own tags are replaced, not merged. That is what entry 10 specifies, and the guide should say so.
• In the fake, a cross-node clone puts the config on the target at once. Real Proxmox may move it at the end of the task, in which case a lock wait on the target node would see does not exist and the next attempt would find the VM. This stays untested, like every cross-node path (entry 24).
• Nothing in this phase talked to the host.

Phase 7 — Delete, Checker, CRD comment

delete.go holds Delete: parse, the per-name lock, destroy (resolve, the lock and status checks, config, ownership, stop, destroy) and the seed delete. check.go holds Check, which fans out one checkNode per node with wg.Go and returns the verdicts in config order, plus checkReason and the four provider reasons. The Delete stub and TestStubs_answerTransient are gone, and grep -rn 'not implemented' internal/provider/proxmox/ is empty. All four contract methods and Check are real. The status.type comment in api/v1alpha1/providerhealth_types.go now ends kvm2 or proxmox, and make manifests changed only that description line in the ProviderHealth CRD.

The spec won over the plan as the Phase 7 notes said. An absent VM still gets its seed deleted, on every configured node. A task waits on the seed delete, and nothing matches F13's absent-volume message. A paused VM gets the stop too. The Checker's reasons include CertificateInvalid and CredentialsUnavailable, and pveproxy's 595 and 596 are Unreachable.

Open questions decided (spec entries 11 and 16 updated):

• Our hibernated VM (lock: suspended) is Permanent in Delete: "resume or destroy it in Proxmox", the same answer Create's ownership check gives. Proxmox's destroy refuses a locked VM (the fake answers with F7's VM is locked (…) text; the suspended variant was never captured), and that lock never clears on its own. Transient would retry forever. A Permanent from Delete in reconcileManaged's Stopped branch goes through providerFailure and latches the Proxy with the admin action in its status. On the finalizer path deletionFailure keeps retrying, and the retries can be seen. Nothing captured shows what a hibernated VM does on a stop or start through the API, so there was no evidence for a self-healing path (for example, start to resume, then stop and destroy). A hibernated VM that is not ours is left alone. The resources-entry lock check skips suspended, so ownership is judged from the config, as in Get.
• The Checker reads p.cfg.TokenFile itself, through the client's readToken, before its first request, and keeps only the part before =. A seam method would break the rule of one method per endpoint, and both TestClient_methodPathAndParams and TestObserve_everySeamMethodLogsOneLine enumerate the interface. An unusable file answers CredentialsUnavailable without a request. readToken's errors never carry the content. Every row of TestCheck_reasonsTable runs at V(2) over the decorator with a sentinel secret and asserts that the secret is in no message, detail or log line.

Other judgment calls:

• The seed is deleted on the VM's node only when Delete destroyed a VM of ours; otherwise on every node. Entry 11 said "skip to step 7" for a VM that is foreign, unmarked, or gone by the time the config is read, without naming a node. In each of those cases our VM's node is unknown: the node in the listing belongs to someone else's VM, or to one that just vanished. A container at the VMID takes the same path.
• Stop's does not exist means the VM is gone, the same as Destroy's. The seed is then deleted on its node, and Destroy is not called.
• errCAFile joins errCredentials in client.go. A missing CA file or one without a certificate is CertificateInvalid ("fix caFile"). Without the sentinel it would be APIError. Its class is unchanged (Transient or Permanent, as entry 14 says).
• Any 500 from the storage status call is StorageNotFound, carrying Proxmox's text. The plan matched a 500 "does not exist", but no response for a missing storage was captured. Matching an invented string would go against the fixture discipline, and every 500 there (missing, not enabled on that node, offline) is fixed on the storage anyway. A 401, 403, 595 or 596 keeps its shared reason.
• checkReason puts *net.OpError before the deadline check, so a dial that runs out of its own 10 s budget is Unreachable, not Timeout. A request timing out on its own while the round still runs is Timeout. The round's ctx ending is Timeout whatever the error says, and that guard is in checkNode, as in kvm2.
• The outcome line proxmox vm deleted carries destroyed, which is false when there was no VM of ours left to destroy. It has no node in that case. Entry 20 lists the new V(1) no-op lines.
• templates counts catalogue entries, not unique VMIDs.

Test names. These plan names changed with the behaviour: TestDelete_absentIsNoOp → TestDelete_absentStillDeletesSeedOnEveryNode, TestDelete_foreignNameIsNoOp → TestDelete_foreignNameLeavesVMAlone, TestDelete_unmarkedIsNoOp → TestDelete_unmarkedLeavesVMAlone, and TestDelete_toleratesMissingISO merged into entry 21's TestDelete_seedDeleteIsTaskAndIdempotent, which runs with the seed both present and absent and ties the absent case to task_status_volume_missing.json. Every other plan and entry 21 name exists. TestDelete_usesPurgeAndDestroyUnreferencedDisks runs Delete over the real httpAPI against the Phase 0 destroy captures (cluster_resources_during_destroy, config_during_destroy, destroy_response, content_delete_response, task_status_volume_missing). TestGet_duringDeleteStillReturnsInstance holds the destroy task in its first poll and first checks that the mid-destroy captures show the VM listed, stopped and without a lock. Tests beyond both lists: TestDelete_containerAtVMIDIsLeftAlone, TestDelete_hibernatedVMIsPermanentIfOurs, TestDelete_seedDeleteFailureOnVMNodeFails, TestDelete_seedDeleteNodeFailureSkippedWhenNodeUnknown, TestDelete_goneMidwayIsSuccess, TestDelete_resourcesFailureIsTransient, TestDelete_waitsOnTheNameLock (a held name lock means no call at all, which is what makes a Delete racing a Create wait), TestDelete_cancelledMidWaitReturnsPromptly, TestDelete_outcomeLine and TestCheck_reasonsOverHTTPClient, which takes the transport reasons from real net/http and crypto/tls errors: unknown CA, hostname, missing CA file, a replayed 401, 596, a closed server and a cut response. The fake gained destroyFailsWith. New test helpers are only the small ones in delete_test.go (putSeed, seedPresent, delete/deleteFails, seedDeleteNodes read from the decorator's lines, twoNodeFixture) and check_test.go (newCheckFixture, assertNoSecret, setStorage). TestCheck_nodesCheckedConcurrently uses a barrier, not a timer: every node's Version waits until all have arrived, so checking the nodes one at a time fails the verdicts deterministically and does not measure elapsed time. That avoids the kind of flake kvm2's 200 ms test has.

The mutation check applied 34 mutations: 17 in delete.go, 16 in check.go and 1 in client.go. Every one compiled and every one failed at least one test. Three did not compile on the first try (an unused upid or tokenID) and were redone. The first run of "never stop" did fail tests, but it also hung TestDelete_concurrentSameNameSafe, whose wait for the first Stop call could not end. The wait is now bounded by the first Delete returning, and the rerun fails fast. One reasons row ("dial cut short by the round's deadline") was added before the run, because no existing row would have caught dropping checkNode's ctx guard. Every mutation was reverted by a script that compared each file's hash with the original.

go test -race -count=2 ./internal/provider/proxmox/ ./internal/providerhealth/   # ok
make manifests && git diff --stat config/    # crawl.example.com_providerhealths.yaml | 2 +-
grep -rn 'not implemented' internal/provider/proxmox/                            # nothing
go build ./... && go vet ./... && go vet -tags=integration ./...
go test -short -count=1 ./...                                                     # all ok
go test -count=1 -v ./internal/provider/proxmox/ | grep -c '^=== RUN'            # 519 before, 649 after
bin/golangci-lint run --disable staticcheck --max-same-issues 0 --max-issues-per-linter 0 ./internal/provider/proxmox/...

Worth noting:

• Lint defaults hide repeats: with --max-same-issues 0 the logcheck hits are 4 in create.go, 4 in the new delete.go, and 2, 1 and 1 in get.go, seed.go and task.go, all the ctx-plus-logger shape. Nothing else is new except goconst on test strings. A prealloc hit in delete_test.go was fixed.
• TestCreate_concurrentSameNameSerialised has the same unbounded for callCount("Clone") == 0 wait that hung the Delete test under a mutation. It is left for the Phase 8 sweep, because this phase does not touch Create.
• Never seen on the host, for Phase 10: what the storage status call answers for a missing or disabled storage, what destroy says for a hibernated VM, and a stop on a paused VM. The try-every-node seed delete and the multi-node Checker have only met the fake (entry 24).
• Nothing in this phase talked to the host.

Phase 8 — Contract and observability sweep, integration test, testing.md

The sweep went into the existing provider_test.go and reuses the earlier phases' helpers. The new helpers are cancelWhileBlocked, within, newLoggingFixture with lifecycle/sentinelReq, assertNeverLogged, assertEveryV1Line and msgLine.

• TestProvider_cancelledContextReturnsPromptly blocks Create, Get, Delete and ListByTag on ClusterResources, and Check on Version, with blockOn. It asserts that the blocked call was the first in the call log, then cancels. Each must return within 1 s, as a Transient that still carries context.Canceled (Check: one target, Timeout). Two more rows block mid-operation: Create on UploadISO and Delete on Stop.
• TestProvider_everyErrorIsWrappedWithConfiguredName has 20 rows, one or more behind every wrapErr call site: placement, the name lock with an ended ctx, resources, a 403 on the clone, a failed start task, the malformed, missing, locked and 401 Get, the hibernated VM, a failed destroy task, a 403 on the seed delete, and the list with a ctx that ends on a candidate. Each row also checks Error.ID (empty for Create) and the provider proxmox-home: <op> prefix.
• TestLogging_v0SilentV1OutcomeV2Detail runs one whole lifecycle over the observed fake with a token file: Create with a sentinel in CloudInit, a Get that finds the captured address, ListByTag, Delete, Delete of the gone VM, and Check. It runs at V(0), V(1) and V(2). V(0) logs nothing. V(1) has each outcome line once, with its keys, and two proxmox vm deleted lines: destroyed=true with node, then destroyed=false without. Every line has the proxmox logger, the provider and level 1, and no V(2) key. V(2) has the resolved line with cloudInitBytes, agent addresses, the listed instance, and the clone's description. The sentinel, #cloud-config and the token secret appear at no level.
• TestLogging_noOpLinesAtV1 (a name beyond the plan) covers entry 20's no-op lines. Each carries provider and vmid: VM already gone, foreign, unmarked, seed delete skipped on each of two nodes, unreadable candidate.
• TestTracing_httpSpansUnderOperationSpan routes captures by path, as TestDelete_usesPurgeAndDestroyUnreferencedDisks does. The captures are cluster_resources_during_destroy, config_during_destroy, status_current_running and agent_network_get_interfaces. A Get runs over the real httpAPI, wrapped in observe and provider.WithTracing, with the recorder's tracer passed through newHTTPAPI's opts. It asserts Running at 192.168.0.150, and exactly four spans named HTTP GET (otelhttp v0.70's defaultTransportFormatter), each a child of provider.get in its trace. No attribute on any span carries the secret or PVEAPIToken.

Judgment calls:

• "Instance lines carry vmid and name" is checked on the provider's lines, not the decorator's. Entry 20 gives proxmox api call the fields call, node, vmid, storage, upid and duration, with name only on a clone. So assertEveryV1Line exempts that line and the two lines that are not about one instance (vms listed, node checked), and requires vmid and name on all the others.
• The integration test compares CreatedAt with the mark Create wrote. kvm2 only checks that it is non-zero. Entry 21 says "equal what Create wrote", so the test reads the VM's config through the bare client and parses its description.
• Get's poll tolerates Transient. kvm2's poll fails on any error. Entry 7 makes every agent answer the provider cannot place Transient, and a booting guest gives some. Those answers are logged and polled through; any other class fails.
• The integration machine type has diskGiB: 4. The 3 GiB recon template is then grown by the resize task on the real host. A larger template skips the resize.
• The seed check after Delete is one extra read-only GET in the test: client.do on /nodes/{node}/storage/{storage}/content?content=iso, decoded the way storage_content_iso.json is shaped. No seam method was added. The VM count and the by-name lookup use the existing ClusterResources.
• The cleanup finds the VM by name, so a Create that failed halfway is also removed. With PROXMOX_TEST_KEEP it prints qm stop <vmid> before qm destroy <vmid> --purge, because destroy refuses a running VM. It then prints pvesm free <isoStorage>:iso/<name>-seed.iso.

Unbounded waits. TestCreate_concurrentSameNameSerialised now uses the first channel shape of TestDelete_concurrentSameNameSafe. The grep found no other sleep loop; client_test.go's for { reads multipart parts until EOF. The mutation check found a different shape of the same hang: tests that call a blocking operation directly on the test goroutine, where the lock or blockOn is released only after the call returns. Under "lockName ignores ctx", "Check ignores ctx" and "pollUntil ignores ctx" the package ran into its 60 s timeout on TestDelete_waitsOnTheNameLock, TestLockName_givesUpWithContextAsTransient, TestCheck_expiredContextReturnsPromptly, two TestCheck_reasonsTable rows, TestCreate_cancelledMidWaitReturnsPromptly, TestDelete_cancelledMidWaitReturnsPromptly, and the ended-ctx rows of TestWaitTask_table and TestWaitUnlocked_table. Each call now runs in within(t, 5s or 10s, …), which fails the test by name. After the fix the same mutations fail those tests in seconds.

The mutation check ran 14 mutations on production code. The script copied each file, applied the edit (asserting that it matched exactly once), and ran go test -c before counting a mutation. It then ran go test -count=1 -timeout=60s, restored the file from the copy, and asserted the sha256 matched. Afterwards sha256sum -c over the eight files hashed up front was OK. task.go, added for M14, was covered by the script's own assertion. git status showed no production file changed. Results:

• Caught by the new tests:
  • Get drops the span context before its requests → TestTracing_httpSpansUnderOperationSpan.
  • Cloud-init logged at V(2) → TestLogging_… and TestCreate_cloudInitSentinel….
  • Delete's destroy path skips wrapErr → the wrap test and 6 Delete tests.
  • ListByTag's ended-ctx path skips wrapErr → the wrap test's list ended on a candidate row.
  • Delete's resolve with context.Background() → the cancel test's delete row.
  • lockName with context.Background() → the wrap test's two lock rows.
  • Check's probe with context.Background() → the cancel test's check row.
  • Delete's logger without the name, destroyed always true, vms listed at V(0), the clone description at V(1), and the list skip line without vmid → TestLogging_*.
• Caught only by the older tests: pollUntil without its ctx.Done() case → the four waits tests above. The first attempt deleted the case and did not compile (the provider import became unused). The rerun replaced it with a nil channel.
• Survived, and equivalent: newHTTPAPI without the tracer-provider option. The transport starts a span only under a valid parent. With no WithTracerProvider, otelhttp v0.70 takes the tracer from trace.SpanFromContext(r.Context()).TracerProvider() (transport.go:99), so the child always lands in the parent's provider. No test can tell the two apart.

Makefile: test-integration-proxmox next to the kvm2 target, with a ## help line naming the five required variables. docs/testing.md gained:

• rows for nocloud, keyedlock and five for proxmox (both packages were missing);
• a "Live Proxmox, by default" bullet under "deliberately do NOT cover", naming the cross-node paths and the three answers never captured;
• a "The proxmox integration test" section: variables, the eleven assertions in order, the KEEP cleanup, the cost, and that it has not run yet.

In spec entry 21 every §11 box is ticked. The list's intro says the two integration tests compile but have not run against a host. The cancel and sentinel items name the new coverage.

# the names in entry 21, each checked before ticking: 42 of 42 exist
awk '/^## 21\. Tests/,/^## 22\./' docs/proxmox-provider-spec.md | grep -o '`Test[A-Za-z0-9_]*`' | tr -d '`' | sort -u \
  | while read n; do grep -rqE "^func $n\(" internal/provider/proxmox/ && echo "ok $n" || echo "MISSING $n"; done
go test -race -count=2 ./internal/provider/proxmox/                        # ok
go vet -tags=integration ./... && go vet -tags=e2e ./...                    # ok
make -n test-integration-proxmox
# go test -race -tags=integration -count=1 -v -timeout 30m -run TestProxmox ./internal/provider/proxmox/
go test -tags=integration -count=1 -v -run TestProxmox ./internal/provider/proxmox/   # both SKIP: PROXMOX_TEST_URL is not set
go build ./... && go test -short -count=1 ./...                             # all ok
go test -count=1 -v ./internal/provider/proxmox/ | grep -c '^=== RUN'      # 649 before, 688 after
bin/golangci-lint run --disable staticcheck --max-same-issues 0 --max-issues-per-linter 0 ./internal/provider/proxmox/...
bin/golangci-lint run --build-tags integration --disable staticcheck --max-same-issues 0 --max-issues-per-linter 0 ./internal/provider/proxmox/...

Worth noting:

• Lint is at the baseline:
  • the logcheck hits (4/4/2/1/1), mapsloop ×2 in proxmox.go, unparam ×3 in fake_test.go, gocyclo in api_test.go, and goconst on test strings.
  • Two new gocyclo hits were removed, not accepted. TestLogging_v0SilentV1OutcomeV2Detail was at 32; the per-line loop became assertEveryV1Line. TestProxmox_lifecycle was at 40; steps 6 and 7 became assertMarkReadBack and assertListed. kvm2's lifecycle test is still at 39.
• The integration tests only compile so far. The first real run is Phase 10's. That run meets the Datastore.Allocate point too: a token without it on the ISO storage fails step 9 as Permanent. Phase 0 granted it on proxy-seeds, and it is already a Phase 9 guide item.
• The parked items did not bite: kvm2's S5.1 and its 200 ms TestCheck_hostsCheckedConcurrently passed in every go test -short ./... run of this phase.
• .claude/settings.json stays out of the commit. Nothing in this phase talked to the host.

Phase 9 — Deployment and documentation

Deployment: the optional proxmox-token Secret (keys token and ca.pem) is mounted read-only at /etc/proxy-operator/proxmox in config/manager/manager.yaml, next to kvm-ssh and commented the same way. The Makefile gains proxmox-token-secret. Both providers configs gain a commented proxmox-home entry after kvm2's. config/samples/proxy_proxmox.yaml is listed in the samples kustomization. There is no RBAC change and no hack/providers-dev.yaml entry. Docs: docs/proxmox-provider.md follows kvm2's guide heading for heading and has no —. The rows and sections named below went into providers-existing.md, architecture.md, the README, providers.md and CLAUDE.md. docs/testing.md and the integration Makefile target were done in Phase 8 and were not touched; the guide's Testing section links to them.

Where the spec and the plan's Phase 9 text disagree, the spec won. The URL is https://192.168.0.8:8006, not 192.168.2.8, which is not in the node certificate. The Secret has two keys, and the target takes PROXMOX_CA_FILE as well as PROXMOX_TOKEN_FILE.

Judgment calls:

• The target also checks both files' content. The token file must be exactly one non-blank line matching <user>@<realm>!<tokenid>=<secret>. That is the shape readToken accepts: TrimSpace, then an anchored regexp. The check uses grep -c and grep -qx, so the secret is never printed. The CA check mirrors kvm-ssh-secret's PRIVATE KEY check: the file must contain BEGIN CERTIFICATE. With insecureSkipTLSVerify the guide says to create the Secret by hand with the token key only.
• The sample-entry test covers both files. commentedEntry finds the # - name: proxmox-home line and takes its indent. It keeps the lines that continue at <indent># and strips # at that indent. The same helper then works for the ConfigMap, whose entry sits six spaces deep inside a | block. So TestLoadConfig_proxmoxSampleEntry has a sample subtest and a configmap subtest. It checks every field the spec names, and that the optional fields stay commented out.
• A second test goes beyond the plan: TestLoadConfig_proxmoxSampleProxyMatchesSampleEntry. The sample Proxy's provider, zone, machine type and image must resolve in the sample entry. kvm has exactly this bug as a known gap (proxy_kvm.yaml places on nas1).
• The sample entry also has medium: {cores: 2, memoryMiB: 2048, diskGiB: 10}, next to the small the spec asks for, so diskGiB appears somewhere. The optional fields are commented out with a short note each, because no live proxmox entry exists to explain them. The kvm2 entry can rely on kvm's.
• In the ConfigMap the proxmox entry is commented out, although kvm2's is live there. It needs a host and a token.
• providers-existing.md needed more than the column and the section. The intro now says five providers and links the guide. In the shared gaps, six sentences would otherwise be false, and each now names proxmox where it applies: adoption verified in two of the five, Create blocking in all three VM providers, no Terminated, the list policy, the collector gap, and the "kvm2 does this" in proposed features. The new section is §8, so the shared gaps and proposed features move to §9 and §10. No doc links those sections by anchor (grep).
• architecture.md §11 now says "All five providers do". The sentence counts Checkers, and proxmox has one. The four Decisions bullets are entry 23's: the VMID from the UID with the ownership check, the mark in the description, the seed uploaded through the API, and the node resolved per call.
• The guide lists the ten design limits first, then four more. They are the hibernated-VM latch, the one-request seed upload, the open-proxy sample (kvm2's guide lists it too), and "not yet proven end to end" with the three answers never captured. The Phase 6 to 8 guide items are each in the section where a reader looks for them. The config write replacing agent and tags is in What lands on Proxmox, and so are the seed cleanup, qm unlock and the 403 that makes Delete Permanent. The certificate rule is in Cluster-side setup. CA caching versus token rotation is in both.
• The guide resumes a hibernated VM with qm start, not qm resume. A VM suspended to disk resumes from its state file on its next start; qm resume is for a VM paused in RAM.
• The guide's pvesm add dir proxy-seeds --path /var/lib/proxy-seeds --content iso is the standard form, not a recorded command. Phase 0 says only that the user created proxy-seeds as a dir storage with ISO content; the path is illustrative.

The token-file check, run without touching a cluster: KUBECTL=echo prints only the apply stage, and KUBECTL=true runs nothing. Scratch files covered one good line, two lines, a bare secret, blank lines around a good line, a non-PEM CA and a missing file. The real files on this workstation pass:

make -n proxmox-token-secret
make -s proxmox-token-secret PROXMOX_TOKEN_FILE=$D/<case> PROXMOX_CA_FILE=$D/ca KUBECTL=echo
# good, ws: "apply -f -"; two, bad: "... is not one line of user@realm!tokenid=secret"; noca: "... holds no PEM certificate"
make -s proxmox-token-secret KUBECTL=true && echo "real files pass"   # real files pass

The mutation check for the sample tests edited one line per run, asserting a single match. It ran the two tests, restored the file from a copy, and at the end sha256sum -c confirmed the three files. All five mutations failed the tests: vlanTag: 5000 and isoStorag: in the sample, 192.168.2.8 and type: kvm2 in the ConfigMap, and image: debian-12 in the sample Proxy. type: kvm2 was rerun after lint removed the test's own type comparison. It still fails, because LoadConfig refuses a proxmox block under another type.

The link sweep is a short script. It takes the docs as arguments, so it can be rerun after any later edit. It skips fenced code and inline code spans. For each relative link it checks that the file exists, and that the #anchor matches a GitHub-style slug of a heading in the target file (or the same file). Before the real run it was checked on a scratch copy of the guide with five links added. It reported exactly the three bad ones: a missing file, a missing anchor in testing.md, and a missing same-file anchor. It checked 78 relative links in the six edited docs, plus this log's, and found 0 broken:

cat > doclinks.py <<'PY'
import os, re, sys
def slug(h, seen):
    s = re.sub(r"[^\w\- ]", "", h.strip().lower()).replace(" ", "-")
    n = seen.get(s, 0); seen[s] = n + 1
    return s if n == 0 else f"{s}-{n}"
def prose(path):
    out, fence = [], False
    for line in open(path, encoding="utf-8"):
        if line.lstrip().startswith("```"): fence = not fence; continue
        if not fence: out.append(line)
    return out
def anchors(path):
    seen = {}
    return {slug(re.sub(r"`", "", l.lstrip("#").strip()), seen) for l in prose(path) if re.match(r"#{1,6} ", l)}
bad = 0
for doc in sys.argv[1:]:
    for n, line in enumerate(prose(doc), 1):
        for target in re.findall(r"\]\(([^)\s]+)\)", re.sub(r"`[^`]*`", "", line)):
            if re.match(r"[a-z]+:", target): continue
            f, _, a = target.partition("#")
            p = os.path.normpath(os.path.join(os.path.dirname(doc), f)) if f else doc
            if not os.path.exists(p): print(f"{doc}: missing file {target}"); bad += 1
            elif a and p.endswith(".md") and a not in anchors(p): print(f"{doc}: missing anchor {target}"); bad += 1
print(f"{bad} broken"); sys.exit(1 if bad else 0)
PY
python3 doclinks.py docs/proxmox-provider.md docs/providers-existing.md docs/architecture.md \
  README.md docs/providers.md CLAUDE.md docs/plans-executions/2026-09-18-1502-proxmox-provider.md   # 0 broken

Verification:

bin/kustomize build config/default >/dev/null && bin/kustomize build config/samples >/dev/null   # ok
go test -count=1 -run 'TestLoadConfig_proxmoxSample' -v ./internal/provider/                     # 3 PASS
grep -c '—' docs/proxmox-provider.md                                                             # 0
go build ./... && go vet ./... && go vet -tags=integration ./... && go test -short -count=1 ./...   # all ok
bin/golangci-lint run --disable staticcheck --max-same-issues 0 --max-issues-per-linter 0 ./internal/provider/
git status --short config/   # the four edited files named above, plus proxy_proxmox.yaml

Worth noting:

• Lint on internal/provider is at its baseline of three goconst hits. The first version of the new test pushed it to five: its p.Type != "proxmox" made "proxmox" a three-time literal, flagged in both config.go and the test. That comparison added nothing, so it went. internal/provider/proxmox was not touched.
• The tmp volume comment in manager.yaml still says only the kvm provider stages seeds there. kvm2 and proxmox do too. It was already stale for kvm2, so it was left out of this phase's diff.
• The architecture §5 block's right-hand bracket was aligned by character, not byte (→ is three bytes), so every line puts │ in column 83.
• Nothing in this phase talked to the Proxmox host or a cluster. .claude/settings.json stays out of the commit. The parked kvm2 items (S5.1, and the 200 ms TestCheck_hostsCheckedConcurrently) did not bite: go test -short ./... passed.
• Next is Phase 10: the first real run of TestProxmox_lifecycle and TestProxmoxCheck_realHost on pv1-1, with the env block in docs/testing.md. The three answers never captured go with it, and the guide's "Not yet proven end to end" gap and the README's matching clause come out once they pass. Then the CHANGELOG entry after the user confirms, and the MR.

Phase 10 — Review, real-host validation, changelog, MR

Where the plan's Phase 10 text and the spec disagreed, the spec won, as in every phase since Phase 1. The host is pv1-1 at https://192.168.0.8:8006, not 192.168.2.8, the template is 9001 and the ISO storage proxy-seeds.

Review. Two read-only passes over git diff main...HEAD: the go-operator-reviewer agent, briefed with the spec and the contract (P1 to P6, S1 to S3), then /code-review high on the branch (CR1 to CR10). Of the 19 findings (two found by both passes), 9 were fixed, 3 accepted as documented limits and 7 rejected with a reason, all recorded in docs/reviews/2026-09-23-proxmox-provider-review/REPORT.md. The one that mattered was CR1/P1. Delete destroyed the VM and then deleted the seed, so a failed seed delete left a seed nothing would ever remove: the next Get is NotFound, the finalizer goes, and GC lists only VMs. Delete now removes the seed between the stop and the destroy, so the VM Get looks at goes last. A 403 on the every-node path now fails instead of being skipped. P2 (CR3) was the other code change: a probe VMID whose listed name is someone else's is skipped at once, instead of waiting up to the clone budget on its lock on every retry.

Lint. make lint still panics inside staticcheck on Go 1.27, so, as in Phases 8 and 9:

bin/golangci-lint run --disable staticcheck --max-same-issues 0 --max-issues-per-linter 0 ./internal/provider/proxmox/... ./internal/provider/
bin/golangci-lint run --build-tags integration --disable staticcheck --max-same-issues 0 --max-issues-per-linter 0 ./internal/provider/proxmox/... ./internal/provider/

Each run was diffed against the pre-fix baseline by file and message, with line numbers and occurrence counts stripped. Three new goconst hits came in along the way, and all three were fixed rather than accepted. The P2 test's lock value made "backup", then "migrate", a repeated literal; it now uses "snapshot-delete". The fake's missing-storage error repeated "Parameter verification failed."; it is now built from the raw captured body. Final counts: goconst 224 (231 with the tag), logcheck 12, modernize 2, unparam 3, gocyclo 1.

Real host. The user ran the host checklist on pv1-1 and then:

export PROXMOX_TEST_URL=https://192.168.0.8:8006
export PROXMOX_TEST_TOKEN_FILE=$HOME/.config/proxmox-token
export PROXMOX_TEST_CA_FILE=$HOME/.config/proxmox-ca.pem
export PROXMOX_TEST_NODE=pv1-1
export PROXMOX_TEST_TEMPLATE_VMID=9001
export PROXMOX_TEST_ISO_STORAGE=proxy-seeds
export PROXMOX_TEST_VLAN_TAG=4
make test-integration-proxmox
# --- PASS: TestProxmox_lifecycle (59.68s)
# --- PASS: TestProxmoxCheck_realHost (0.13s)

It ran the review fixes, uncommitted, on the first try. Clone task about 3 s, Create 6.2 s, address 50.2 s after Create returned, Delete plus Get plus a second Delete 3.1 s. The Delete deleted the seed while the stopped VM's ide2 still referenced it (imgdel OK), then destroyed the VM, and step 10 found neither left. That settles the one assumption behind the reorder that the fake could not prove. The run confirmed P6 as well: the first Get, about 6 s after the clone, met status: unknown in the listing and answered Transient, blaming the node, which was up. The message now names both causes.

The three answers Phase 0 never captured. Captured with a scratch helper that passes the token to curl through a process substitution and writes the Phase 0 fixture shape. Its first offline test caught a bug: Python's text mode folds \r\n, so the header split found nothing, and the file is now opened with newline="". The calls were:

pvecap storage_status_missing GET /nodes/pv1-1/storage/proxy-missing/status
pve POST /nodes/pv1-1/qemu/9001/clone -d newid=199999 -d name=proxy-reconaaaaaaaaaaa -d full=1
pve POST /nodes/pv1-1/qemu/199999/status/start
pve POST /nodes/pv1-1/qemu/199999/status/suspend                 # pause
pvecap stop_paused_response POST /nodes/pv1-1/qemu/199999/status/stop
pvecap task_status_stop_paused GET /nodes/pv1-1/tasks/<upid>/status
pvecap status_current_after_stop_paused GET /nodes/pv1-1/qemu/199999/status/current
pve POST /nodes/pv1-1/qemu/199999/status/start
pve POST /nodes/pv1-1/qemu/199999/status/suspend -d todisk=1     # hibernate: stopped, lock suspended
pvecap err_destroy_hibernated DELETE '/nodes/pv1-1/qemu/199999?purge=1&destroy-unreferenced-disks=1'
pvecap task_status_destroy_hibernated GET /nodes/pv1-1/tasks/<upid>/status

Two of the three contradicted the code:

• A missing storage is a 400 with the field error storage: No such storage., not a 500, so the Checker reported APIError. isNoSuchStorage matches that exact field error, and a 500 is still taken as missing (not captured). The matcher matrix caught a first version that fired on any storage field error, which a clone onto a storage without image support also has.
• The destroy accepts a hibernated VM. Its task ended OK, removing both disks and the vm-199999-state-suspend-… volume. Delete refused such a VM of ours as Permanent, believing the destroy would, so it now destroys it like a stopped one. Create's adoption still refuses one it finds before it was ever recorded. The capture was renamed destroy_hibernated_response.json, because the client test globs err_*.json as errors.
• A stop on a paused VM ends OK with the VM stopped, as assumed.

The destroy also cleaned up the scratch VM, and a listing of local-lvm content for VMID 199999 came back empty.

Kind pass. The user built and loaded the image, deployed, applied a scratch providers config with only kubernetes and proxmox-home, ran the sample Proxy to Ready, ran a curl -x through it, deleted it and checked pv1-1 for leftovers. Everything passed. The first attempt at the override went to a ConfigMap named providers-config. config/default's namePrefix makes the mounted one egress-proxies-operator-providers-config, so the pod kept the shipped config: no proxmox-home ProviderHealth, and kvm-home and kvm2-home failing on the missing SSH key. The corrected command replaced the prefixed ConfigMap and deleted the stray one. The Secret keeps its plain name (proxmox-token), since kustomize renames only resources it builds. make deploy's image edit to config/manager/kustomization.yaml was reverted, not committed.

Worth noting:

• Behaviour changes from Phase 9. Delete's order is now stop, seed, destroy. A hibernated VM of ours is destroyed rather than latched. The Checker reports a missing ISO storage as StorageNotFound. A foreign-named VM on a probe VMID is skipped without a config read.
• Accepted, not fixed (guide Known gaps, spec entry 24):
  • P3: Create adopts a leftover stopped VM whatever the current placement says.
  • P4: a retry can clone a second VM in a narrow crash window.
• Still unproven:
  • A storage that exists but cannot be used on a node.
  • Every cross-node path. The host has one node.
• Flakes. kvm's TestKeepalive_keepsHealthyClientOpen failed once under -race ("unexpected packet in response to channel open") and passed 5 of 5 on rerun. This branch does not touch internal/provider/kvm. The parked kvm2 items (S5.1, the 200 ms TestCheck_hostsCheckedConcurrently) did not bite, though the reviewer agent saw the latter fail once.
• Docs. The "not yet run" wording is gone from the README, the guide, testing.md, spec entry 21 and providers-existing.md. testing.md gained a timings table and the PROXMOX_TEST_VLAN_TAG export line. The link sweep script from the Phase 9 entry found 0 broken links over the seven edited docs, and the guide has no —.
• .claude/settings.json stays out of the commit.