egress-proxies-operator/docs/plans-executions/2026-09-18-1502-proxmox-provider.md

Execution: proxmox, a Proxmox VE provider built from the provider contract

Pairs with docs/plans/2026-09-18-1502-proxmox-provider.md.

Status

• Phase 0 — Plan file, branch, host recon, fixtures
• Phase 1 — The spec (docs/proxmox-provider-spec.md)
• Phase 2 — Config block, registration, package skeleton
• Phase 3 — Extract nocloud and keyedlock from kvm2
• Phase 4 — Seam, HTTP client, decorator, fake
• Phase 5 — Get, ListByTag, task helpers
• Phase 6 — Create
• Phase 7 — Delete, Checker, CRD comment
• Phase 8 — Contract and observability sweep, integration test, testing.md
• Phase 9 — Deployment and documentation
• Phase 10 — Review, real-host validation, changelog, MR

Phase 0 — Plan file, branch, host recon, fixtures

The plan was committed alone on main on 2026-09-18 (708989c, then 2167819 retargeting it at PVE 9.2). origin/main had not moved in the meantime, so the branch was cut straight from local main:

git fetch -q origin && git rev-list --left-right --count origin/main...main   # 0 2
git switch -c feat/proxmox-provider

The host is pv1-1, a single node (no cluster peers), pve-manager/9.2.20, kernel 7.0.14-17-pve, QEMU 11.0. The user prepared the token (~/.config/proxmox-token, privsep off) and the CA (~/.config/proxmox-ca.pem) before the session.

TLS: 192.168.2.8 is not in the node certificate. The first probe failed with curl exit 60. The CA is right (openssl s_client -CAfile verifies the chain), but the leaf's SANs are 127.0.0.1, ::1, localhost, 192.168.0.8, pv1-1, pv1-1.home.lab, and neither name resolves on the workstation. 192.168.0.8 and 192.168.2.8 present the same certificate fingerprint: the node sits on vmbr0 (192.168.0.8/24, untagged) with VLAN interfaces vmbr0.2 (192.168.2.8/24) and vmbr0.4 (192.168.4.8/24). All recon kept full verification by connecting to 192.168.2.8 while verifying the certificate as pv1-1.home.lab:

curl -s --resolve pv1-1.home.lab:8006:192.168.2.8 --cacert ~/.config/proxmox-ca.pem \
  -H @<(printf 'Authorization: PVEAPIToken=%s\n' "$(tr -d '[:space:]' < ~/.config/proxmox-token)") \
  https://pv1-1.home.lab:8006/api2/json/version
# {"data":{"release":"9.2","version":"9.2.20","repoid":"49318c671b82f31e"}}

Every capture went through a scratch helper wrapping exactly that call. It passes the token through a file descriptor so it never reaches argv or output, and writes {"status","reason","body"} to internal/provider/proxmox/testdata/<name>.json, refusing to save a body that contains PVEAPIToken.

A real NoCloud seed was used instead of the plan's 1 KiB placeholder, so the agent could be installed and F21 captured from a live guest. No ISO tool is installed locally, so a throwaway module in the scratchpad reused the repository's github.com/kdomanski/iso9660 v0.4.0 from the module cache and copied kvm2's guestAgentScript verbatim:

GOFLAGS=-mod=mod GOPROXY=off GOSUMDB=off go run . proxy-reconaaaaaaaaaaa proxy-reconaaaaaaaaaaa-seed.iso
# 45056 bytes, ISO 9660 'cidata'

Write experiments ran against VMID 199999 (proxy-reconaaaaaaaaaaa), a clone of template 9001. Sequence: validation error; clone; resources, config write and start while the clone lock was held; status and agent while running bare; stop; config read; clone onto the taken VMID; tag tests; upload twice plus a bad name; attach ide2, set cores and memory; three resizes; start and agent poll; suspend and resume; a clone onto a storage without image support; stop; destroy with purge=1&destroy-unreferenced-disks=1; destroy again; ISO delete.

Facts F1 to F23 (cited as Evidence by the spec)

• F1. Errors carry the text twice: in the HTTP reason phrase and as the body's message, which ends in \n. A 400 adds errors, a map from parameter to message. A 401 has the reason phrase only and an empty body. The client should use the trimmed body message and fall back to the reason phrase. Headers are ordinary CRLF.
• F2. Missing VM, identical for GET config, GET status/current and DELETE: 500 Configuration file 'nodes/pv1-1/qemu-server/999999.conf' does not exist.
• F3. A start issued while the VM is clone-locked is accepted (200 with a UPID). The task log reads trying to acquire lock... OK and the VM starts once the clone finishes. Setting a lock through the API was not tried.
• F4. Clone onto a taken VMID, synchronous: 500 unable to create VM 199999: config file already exists.
• F5. Agent enabled but not running: 500 QEMU guest agent is not running. Agent not enabled in the config: 500 No QEMU guest agent configured. Stopped VM with agent: 1: 500 VM 199999 is not running.
• F6. Not produced. local-lvm is thin and overprovisions, so quota: absent.
• F7. Config write while clone-locked: 500 VM is locked (clone).
• F8. /cluster/resources?type=vm entries carry vmid, name, node, status, type, template, tags, maxcpu, maxmem, maxdisk, uptime and more; lock appears only when set. Status values seen: running, stopped, and unknown during a clone. Mid-clone the entry has no name: only type, vmid, id, node, lock: clone, status: unknown.
• F9. Hyphenated tags are accepted and stored as given. Uppercase is folded (Proxy_Operator → proxy_operator), the separator is ;, and characters such as a space or ! give 400 … tags: invalid format - invalid characters in tag. The datacenter has no tag-style or user-tag-access set.
• F10. description on clone is accepted and survives verbatim, newlines included. The 22-character proxy-<base32> name is accepted.
• F11. The clone's meta is copied from the template (creation-qemu=11.0.3,ctime=1790145676, identical to 9001), so it says nothing about when the clone was made. written-at-create is required.
• F12. Upload: multipart with field content=iso and the file part named filename, carrying the target file name. Returns a UPID for an imgcopy task. A name without .iso gives 400 … filename: wrong file extension. Re-uploading an existing name silently overwrites it (the task log shows a plain cp). Datastore.AllocateTemplate suffices.
• F13. Deleting a volume needs Datastore.Allocate on /storage/<id>; without it the call gives 403 Permission check failed (/storage/local, Datastore.Allocate). With it (on proxy-seeds), the delete runs as an imgdel task and returns a UPID. Deleting a volume that no longer exists is also accepted and its task still ends OK; the log reads disk image '…' does not exist, then Removed volume. Seed cleanup is idempotent with no message matching.
• F14. Stopped: status: stopped, qmpstatus: stopped. Paused: status: running, qmpstatus: paused in status/current, while /cluster/resources shows plain running.
• F15. A clone onto a storage without image support is rejected up front (400 … storage 'local' does not support vm images) and leaves no VM. A clone failing mid-task was not observed.
• F16. Task status fields: status (running, stopped), exitstatus (OK or the error text), type, id, node, user, tokenid, upid, starttime, pid, pstart. A full clone of the 3 GiB template onto local-lvm finished within about two seconds.
• F17. The planned role, granted on /, covered clone, config, resize, start, stop, suspend, agent read, upload and destroy. Missing: Datastore.Allocate for deleting the seed ISO. VM.Config.Cloudinit is not needed for the ide2 write. SDN.Use was held but never exercised, because no call rewrote net0.
• F18. HTTP/1.1 only: curl --http2 negotiates HTTP/1.1.
• F19. Template 9001 (debian-13-template): scsi0 on local-lvm, boot: order=scsi0, ide2 free, agent: 1, OVMF, q35, net0 untagged on vmbr0. Attaching ide2 makes Proxmox append it to the boot order (order=scsi0;ide2). Template 9000 (Ubuntu) is unsuitable: it has a Proxmox cloud-init drive on ide2 and VLAN tag 4 on net0.
• F20. Resize runs as a task (returns a UPID). The same size is an OK no-op; shrinking fails inside the task with exitstatus shrinking disks is not supported.
• F21. Agent response: result[] of {name, ip-addresses[{ip-address, ip-address-type}]}. The live guest reported lo and ens18 with 192.168.0.150, 50 seconds after start, via DHCP on untagged vmbr0. This proves the NoCloud-seed-on-ide2 route end to end on this host.
• F22. A wrong secret and an unknown token id both give 401 Authentication failed!; no credentials gives 401 No ticket.
• F23. Uploads are staged in /var/tmp/pveupload-* and then copied to /var/lib/vz/template/iso/, so the tmpfs /tmp is not involved.

Corrections to the plan that Phase 1 must fold into the spec

1. Check lock before the name. A mid-clone VM has no name in /cluster/resources (F8), so name-first logic turns a clone in progress into NotFound. Get, Delete and Create's lookup must treat a locked or nameless entry at our VMID as in-progress (Transient or wait), never absent.
2. Inspect a taken VMID before probing forward. After a crash mid-clone, the retry's name lookup misses the nameless VM, the clone onto the deterministic VMID fails with F4, and forward probing would clone a duplicate. On F4, read that VMID: lock: clone means wait for it, and our UID in the description means adopt it. Probe forward only for a VM that is demonstrably someone else's.
3. A start queues behind a lock instead of failing (F3).
4. Resize is asynchronous (F20); wait on its task.
5. Drop delete-before-upload for the seed: upload overwrites (F12). Seed deletion is a task (F13): wait on it. A repeat delete succeeds, so Delete needs no absent-volume matcher.
6. Get reads status/current, not only /cluster/resources, to tell paused from running (F14).
7. Tag rationale: hyphens are accepted on 9.2.20 (F9). proxy_operator stays, since it is valid everywhere, but the claim that hyphens are rejected is removed.
8. The boot-disk logic must skip ide2, which Proxmox adds to the boot order (F19). The plan's rule already excludes it; add a test with order=scsi0;ide2.
9. Three agent messages, not one (F5): QEMU guest agent is not running (booting, or the install failed), No QEMU guest agent configured (our agent=1 was removed), and VM <vmid> is not running (stopped between the status read and the agent call). The spec decides each class; none may become NotFound.
10. Internet on the proxy network is a hard prerequisite, twice over: the vendor-data agent install needs a package mirror, and the proxy itself needs egress. The guide says so, and recommends baking qemu-guest-agent into the template with installGuestAgent: false to drop the mirror dependency from IP discovery.

Decisions made with the user in Phase 0

• Seeds go to a dedicated directory storage, proxy-seeds, with ISO content only. The token gets Datastore.Allocate there and nowhere else, so it can delete seeds but never backups or other ISOs on local. The sample config's isoStorage becomes proxy-seeds. The guide must say that an ACL on a more specific path replaces the inherited one for that user, so the role granted on /storage/proxy-seeds must repeat Datastore.AllocateTemplate, AllocateSpace and Audit alongside Allocate.
• Proxies land on a VLAN set in config. ProxmoxConfig gains an optional vlanTag (1 to 4094), applied when Create rewrites net0 together with the optional bridge override, keeping the MAC. Unset leaves net0 as cloned. This is the first call that exercises SDN.Use (F17).
• The API URL is https://192.168.0.8:8006, the address in the node certificate, so caFile verifies with no provider change. No tlsServerName option. The operator's cluster must be able to reach 192.168.0.x.

Seed storage and VLAN 4

The user created proxy-seeds (dir, ISO content only), granted the token Datastore.Allocate, AllocateSpace, AllocateTemplate and Audit on /storage/proxy-seeds through a role ProxyOperatorSeeds, and freed the leftover recon ISO on local. An upload to proxy-seeds and two deletes of a throwaway ISO there produced content_delete_response.json, content_delete_again_response.json and task_status_volume_missing.json.

The user gave the VLAN as "vlan 4, bridge: vmbr0.4@vmbr0". That is the host's own VLAN interface name as ip link prints it; a VM NIC cannot attach to it because it is not a bridge. On the VLAN-aware vmbr0 the Proxmox form is bridge=vmbr0,tag=4, which is what template 9000 uses, so the provider's vlanTag: 4 appends tag=4 to the cloned net0.

A second clone of 9001 at VMID 199999 got its seed from proxy-seeds and net0 rewritten from virtio=BC:24:11:EE:48:83,bridge=vmbr0 to virtio=BC:24:11:EE:48:83,bridge=vmbr0,tag=4. The write succeeded, so the token's SDN.Use on / covers a VLAN tag on vmbr0 (set_config_vlan_response.json).

The first VLAN 4 attempt never produced an agent in seven minutes of polling (every 13 seconds, 33 polls). The only signal was the agent call, so the cause could not be seen from the API. A debug seed added a recon user (user-data only, with a new instance id so cloud-init re-ran), and the user logged in on the console: DHCP worked, but VLAN 4 had no route to the internet because of a routing misconfiguration on the Proxmox host, so the vendor-data apt-get install qemu-guest-agent could not run. The user fixed the routing.

Rerun from scratch after the fix, same sequence the provider will use (destroy and seed delete first, then a fresh clone and seed, instance-id equal to the name):

Step	Elapsed
clone task done	4 s
seed upload task done	4 s
config (ide2, net0 with tag=4, tag)	4 s
start task done	5 s
agent reports 192.168.4.147	47 s after start

The guest was reachable from the workstation (ping, TCP 22), as the contract requires of a proxy address. Stopping it gave the F5 stopped-VM fixture; destroy and seed delete then left the host with only its original VMs 100, 9000 and 9001, no ISOs on local or proxy-seeds, and no local-lvm volumes of 199999. The capture is agent_network_get_interfaces_vlan.json.

Worth noting

• The token file is mode 664; 600 would be tighter.
• Fixtures: 48 files, all valid JSON, none containing PVEAPIToken. Several names differ from the plan's list because what they captured differed from what was expected: start_while_clone_locked_response.json (the start succeeded), task_status_stopped_error.json (from the shrink, not a start), upload_overwrite_response.json (success, not an error), resize_same_size_response.json and resize_shrink_response.json (UPIDs), err_forbidden_volume_delete.json (the 403 from F13). err_no_space.json was not produced (F6).
• The host is back to its original state; see the table above.
• The scratch seed builder (seedgen in the session scratchpad) took the user-data file and instance id as arguments by the end, for the debug login. It is not part of the repository; Phase 3's nocloud package replaces it.
• Time to IP was about 50 s on both networks, with the agent installed at first boot. Budgets in the plan (running wait 60 s for the VM state, the agent discovered by Get polling) are consistent with this.

Phase 1 — The spec

Wrote docs/proxmox-provider-spec.md, all 24 entries of the template, from the plan, the contract and the Phase 0 facts. The ten Phase 0 corrections are folded in, and the plan now opens with a note that the spec wins where the two disagree.

Judgment calls made here, each recorded in its entry:

• Create's allocation loop (entry 2) runs an ownership check on any taken VMID: wait out a lock or a nameless entry, then adopt when the name and UID are ours, skip when the name differs, refuse as Permanent when the name is ours but the mark is not. This is the fix for the duplicate-VM bug.
• Get's check order (entry 9): lock, then name and mark from the config, then status/current, then the agent. lock: suspended alone maps to Stopped, since a hibernated VM never resumes by itself.
• Agent messages (entry 7): QEMU guest agent is not running is Provisioning, VM <vmid> is not running is Transient, and No QEMU guest agent configured is Stopped, so the reconciler replaces a VM that can never report an address instead of latching the Proxy.
• Templates are checked before cloning (entry 4): not a template, or a Proxmox cloud-init drive in any slot, is Permanent. This makes the "no cloud-init drive" prerequisite enforceable instead of a silent failure.
• isoStorage is required, with no default (entry 15), because the token must hold Datastore.Allocate on it.
• TLS verification failures are Permanent, with a Checker reason CertificateInvalid (entries 13, 16), since the fix is the URL or CA file.
• The CA is loaded lazily and cached; the token is read per request (entry 14).

Checks, as the plan's Verify clause asks:

grep -c '^## [0-9]*\. ' docs/proxmox-provider-spec.md     # 24
grep -n '^\*\*Answer:\*\*$' docs/proxmox-provider-spec.md # nothing
grep -c '—' docs/proxmox-provider-spec.md                 # 0

plus a script comparing every `<axis>`: `<profile>` pair against the contract's §4.20 table (no mismatches, every axis answered) and a list of the F numbers cited (all 23).

Worth noting: the first pass stated two things as fact that recon could not show and was corrected before commit. That an unreachable node's guests stay listed as unknown is Proxmox's documented design, unverifiable on a single node; entry 13 now says so. And the matcher for VM <vmid> is not running is a whole-message regexp, because is not running alone also matches QEMU guest agent is not running.

Phase 2 — Config block, registration, package skeleton

Added the proxmox: block (internal/provider/config_proxmox.go), wired it into config.go (the field, setBlocks, the allow-list and the dispatch: three "proxmox" sites) and registered "proxmox": proxmox.New in cmd/main.go, so the two type lists changed in one commit. The package internal/provider/proxmox now holds the seam interface and its data types (api.go), id.go, mark.go, state.go, errors.go and the constructor (proxmox.go). The four contract methods are stubs that answer ErrTransient, so a Proxy on a configured proxmox entry retries instead of latching until Phases 5 to 7 land. New passes a nil api until Phase 4 builds the HTTP client.

Every test that depends on Proxmox's wording or JSON shape reads the Phase 0 captures through fixtures_test.go (readCapture, loadAPIError, decodeData), so no message string is repeated in a test. The one exception is TestClassify_table's synthetic rows (a 404, 502, 595, 596, a storage message), for statuses recon never produced.

Deviations from the plan's Phase 2 list, both making it smaller:

• No temporary locks.go. Only Create and Delete take the per-name lock, and they arrive after Phase 3 extracts internal/provider/keyedlock, so copying kvm2's lock now would only be deleted next phase.
• No provider.Checker assertion yet. cmd/main.go type-asserts the Checker and would call a stub; the assertion comes with the real Check in Phase 7.

Added beyond the plan: api_test.go decodes the captured responses into the seam's types, so a misspelled JSON tag fails now rather than in Phase 5, and TestParseDescription_fromClone reads the mark back from the description of the VM cloned in Phase 0.

A mutation check confirmed the suite can fail. Matching VM <vmid> is not running as a plain suffix (the first spec draft's mistake) failed TestAPIError_matchers on the agent-not-running capture; treating a paused VM as running failed TestMapState_table and TestMapState_fromCaptures. Both mutations were reverted and the files compared identical to backups.

gofmt -w cmd/main.go            # realigned the constructor map for the longer key
grep -c '"proxmox"' internal/provider/config.go                     # 3
go build ./... && go vet ./...
go test -race -count=1 ./internal/provider/... ./cmd/...            # all ok
go test -short -count=1 ./...                                       # all ok
go test -count=1 -v ./internal/provider/proxmox/ | grep -c '^=== RUN' # 152

Worth noting: the validProxmoxYAML constant is the shape Phase 9's commented sample entry should take, with the user's values (192.168.0.8, pv1-1, proxy-seeds, vlanTag: 4, template 9001).

Phase 3 — Extract nocloud and keyedlock from kvm2

kvm2's seed.go and seed_test.go moved to internal/provider/nocloud with git mv, so history follows them. The package exports BuildSeedISO, BuildVendorData, GuestAgentScript, SeedLabel and a new ErrEmptyName sentinel in place of kvm2's permanentf. kvm2's keyedLocks became internal/provider/keyedlock (Locks, New, Acquire), and its three TestKeyedLocks_* tests moved with it under their old names. kvm2 now imports both packages, and its create.go still wraps every seed-build error with permanentf, so kvm2's behaviour is unchanged. proxmox gained the nameLocks field and lockName.

The test counts match the plan's Verify clause: kvm2 dropped from 465 to 447 === RUN entries, which is the 12 moved tests plus the 6 subtests of TestSeedISO_userDataByteForByte. nocloud (15) and keyedlock (3) carry exactly those 18.

git mv internal/provider/kvm2/seed.go internal/provider/nocloud/seed.go
git mv internal/provider/kvm2/seed_test.go internal/provider/nocloud/seed_test.go
go test -count=1 -v ./internal/provider/kvm2/ | grep -c '^=== RUN'   # 465 before, 447 after
grep -rn 'iso9660' internal/provider/kvm2/                          # nothing
go build ./... && go vet ./... && go vet -tags=integration ./...
go test -race -count=1 ./internal/provider/...                      # all ok, run three times
go test -short -count=1 ./...                                       # all ok

Pulled forward from Phase 6 so proxmox imports nocloud now, as the "Done when" line requires: seed.go with seedISOName, seedVolID and buildSeed, the pure part of the seed step. uploadSeed still belongs to Phase 6, which calls buildSeed. The new tests are TestSeedNames_table, TestBuildSeed_table (4 rows), TestBuildSeed_emptyNameIsPermanentWithoutPayload, TestBuildSeed_stagingFailureIsTransient and TestLockName_givesUpWithContextAsTransient. proxmox went from 152 to 161 test entries.

proxmox does not copy kvm2's open review finding S5.1 (docs/reviews/2026-09-17-kvm2-provider-review/sweep.md). In kvm2, a staging failure such as a missing /tmp or a full emptyDir becomes Permanent, and that latches the Proxy. buildSeed makes only ErrEmptyName Permanent and wraps any other error with %w, so classify reads it as Transient. The sentinel makes the same three-line fix possible in kvm2, but this phase leaves kvm2's behaviour unchanged.

Worth noting:

• The older kvm provider has its own seed.go with a different signature (buildSeedISO returns []byte). The plan scopes the extraction to kvm2, so kvm is untouched.
• GuestAgentScript now says the operator reads the address "from the hypervisor" instead of "through libvirt", in both the Go comment and the script's comment line inside the guest. The script's commands are unchanged.
• The docs now name the new packages: kvm2-provider.md, kvm2-provider-spec.md entry 19, the kvm2 columns and section of providers-existing.md, plus a "Shared provider helpers" row in the architecture.md component table. docs/reviews/ keeps the old names, because it records the code as it was reviewed.
• TestCheck_hostsCheckedConcurrently in kvm2 failed once under the full -race suite (latency 199.96 ms against a 200 ms floor). It passed 20 runs out of 20 in isolation, with and without this phase's changes. The test starts its 200 ms release timers before Check starts its clock, so under load the measured latency can come in just under 200 ms. The flake predates this phase, and the test is not touched here.

Phase 4 — Seam, HTTP client, decorator, fake

client.go holds httpAPI, the thin net/http adapter for all sixteen seam methods. It reads the token file on every request, turns any non-2xx answer into *apiError from the body's message or the reason phrase (F1), and wraps its transport in tracing.RestConfigWrapper. observe.go is the logging decorator: one V(1) proxmox api call line per call, detail at V(2), and no span, because the transport already traces each request. New now builds observe(newHTTPAPI(...)) instead of passing a nil api, still without touching the network or the token file. fake_test.go holds fakeAPI, which Phases 5 to 7 build on.

Where this phase departs from the plan's Phase 4 text:

• The http.Client is built on first use, not in newHTTPAPI. Spec entry 14 wins over the plan: the CA is loaded lazily, a missing CA file is Transient and retried until the file lands, one without a certificate is Permanent, and the pool is cached only once it loads. A transport's RootCAs cannot change after first use, so the whole client waits for the CA (TestClient_caFileLoadedLazily).
• The upload is not an io.Pipe with a goroutine. The multipart framing (the content=iso field, the filename part header and the closing boundary) is rendered into two small buffers, and the body is io.MultiReader(head, io.LimitReader(r, size), tail) with an exact Content-Length. The seed is still streamed and never held by the client. There is also no goroutine to leak, and the request carries a length, as Phase 0's curl -F upload did. A chunked upload to pveproxy was never tried. TestClient_uploadStreamsMultipart proves the streaming with a reader that stalls after its first 256 KiB until the server has seen file bytes.
• The fake's DeleteVolume tolerates a missing volume, as spec correction 5 and F13 say, instead of the plan's error.
• Added: secret redaction in apiError (contract §7), because otherwise a server that echoes the header would put the secret into Proxy status; an errCredentials sentinel on token-file errors, so Phase 7's Checker can report CredentialsUnavailable; and the V(1) identifiers newid and name on a clone, filename on an upload and volid on a volume delete. Spec entry 20 now lists them.

The volid's / travels as %2F. Neither the plan nor the spec says how {volid} is encoded in the content-delete path. The Phase 0 session's transcript shows every delete went through a helper encoding the whole volid:

enc() { python3 -c "import urllib.parse,sys; print(urllib.parse.quote(sys.argv[1], safe=''))" "$1"; }
curl ... -X DELETE ".../storage/proxy-seeds/content/$(enc 'proxy-seeds:iso/proxy-reconaaaaaaaaaaa-seed.iso')"

So apiPath runs url.PathEscape over each segment (TestClient_volIDSlashIsEscaped).

net/http cannot send a custom reason phrase, so the client tests replay every capture byte for byte through a hijacked connection (serveCapture). TestClient_apiErrorFromStatusReasonAndBody globs testdata/err_*.json and skips err_tag_hyphen.json, which is a 200 despite its name: hyphenated tags were accepted (F9).

Tests beyond the plan's list: TestClient_methodPathAndParams (the method, path, query and form of every seam method, with a reflection check that no method lacks a row), TestClient_cloneOmitsEmptyStorageAndTarget, TestClient_volIDSlashIsEscaped, TestClient_caFileLoadedLazily, TestClient_x509FailureIsPermanent (from spec entry 21), TestDecodeConfig_scalarsAsStrings, TestReadToken_trimsTrailingNewline, TestObserve_startsNoSpan, TestObserve_identifiersAtV1, TestNew_wiresObservedHTTPClient, and six TestFakeAPI_* tests. TestFakeAPI_errorsMatchCaptures compares the fake's error texts with the captures, so every test that goes through the fake exercises the real wording. Texts that recon never produced start with fake:.

A mutation check confirmed the new tests can fail: dropping redact, buffering the upload with io.ReadAll, and logging V(2) detail at V(1) each failed their test. Removing only the empty TLSNextProto map did not fail TestClient_http1Only. A transport with a custom TLSClientConfig and DialContext never offers h2 unless ForceAttemptHTTP2 is set, so the map is a second guard. With both guards removed the test failed with client spoke HTTP/2. Every mutation was reverted, and the files were compared identical to backups.

go test -race -count=2 ./internal/provider/proxmox/          # ok, run three times
grep -n 'ReadAll' internal/provider/proxmox/client.go        # only the 64 KiB error-body read
grep -n 'context.Background()' internal/provider/proxmox/*.go | grep -v _test.go   # nothing
go build ./... && go vet ./... && go vet -tags=integration ./...
go test -short -count=1 ./...                                # all ok
go test -count=1 -v ./internal/provider/proxmox/ | grep -c '^=== RUN'   # 161 before, 289 after

Worth noting:

• bin/golangci-lint run ./internal/provider/proxmox/... panics inside staticcheck's IR builder (unexpected expr: *ast.KeyValueExpr in the Go 1.27 standard library's poll package). This is a toolchain problem, not this code. With --disable staticcheck the new files' findings are goconst on form keys (kvm2 carries 50 of the same) and two unparam hits on the fake's addTemplate and resourceByID, which Phases 5 and 6 will call with other VMIDs. The mapsloop and gocyclo findings in proxmox.go and api_test.go date from Phase 2.
• Nothing in this phase talked to the host. The client has only met httptest servers replaying the captures. The first real request is Phase 10's integration run, which is where the Content-Length upload and the %2F volid get proven against pveproxy.
• The fake refuses to destroy a running VM with an uncaptured fake: VM <vmid> is running - destroy failed, so Phase 7's stop-before-destroy order is enforced by the fake even though the real text is unknown.