diff --git a/docs/plans/2026-09-18-1502-proxmox-provider.md b/docs/plans/2026-09-18-1502-proxmox-provider.md new file mode 100644 index 0000000..9dfed1c --- /dev/null +++ b/docs/plans/2026-09-18-1502-proxmox-provider.md @@ -0,0 +1,844 @@ +# Plan: proxmox, a Proxmox VE provider built from the provider contract + +**Created:** 2026-09-18 15:02 + +## Context + +The provider contract (`docs/provider-contract.md`), the spec template +(`docs/provider-spec-template.md`) and the kvm2 build proved the workflow: +spec first, then implementation against the contract, one commit per step. +This plan adds a third VM-style backend, `proxmox`, on the Proxmox VE 8.x +REST API at `https://:8006/api2/json`. The user has a real host at +192.168.2.8 and prepares the API token and a cloud-init-capable template +before Phase 0 (checklist below). + +The implementation runs in sessions whose context gets compacted, so every +phase is self-contained: it names what to read first, what it creates or +edits, the test names it must add, a `Verify:` clause, a "Done when" line +and a commit subject. A fresh session reading only this plan, the spec +(`docs/proxmox-provider-spec.md`, from Phase 1 on) and the execution log's +checklist (`docs/plans-executions/2026-09-18-1502-proxmox-provider.md`) must be able to +pick up where the last one stopped. + +**Resumption protocol, run at the start of every phase.** Read this plan's +Context and the phase entry; read the spec; read the execution log's +checklist; run `git branch --show-current` (must be `feat/proxmox-provider` +from Phase 0 on), `git status --short` and `git log --oneline -8`. The first +unchecked box is the phase to do. If its files already exist partially, its +"Done when" line says whether it counts as done; finish it, never redo it. A +phase is finished only when its "Done when" holds and its execution-log entry +(a few sentences, exact commands, "worth noting", the box ticked) is +committed together with the code. Follow `CLAUDE.md` for the log format. + +## Before Phase 0: what the user prepares on 192.168.2.8 + +Run as root on the Proxmox node. Drop any privilege `pveum` rejects on the +installed version (`VM.GuestAgent.Audit` exists from 8.2; older releases use +`VM.Monitor` for the agent read) and note it for F17. + +```bash +pveum user add proxy-operator@pve --comment "egress-proxies-operator" +pveum role add ProxyOperator -privs "VM.Allocate VM.Clone VM.Audit VM.Config.CDROM VM.Config.CPU VM.Config.Memory VM.Config.Disk VM.Config.Network VM.Config.Options VM.PowerMgmt VM.Monitor VM.GuestAgent.Audit Datastore.Audit Datastore.AllocateSpace Datastore.AllocateTemplate Sys.Audit SDN.Use" +pveum aclmod / -user proxy-operator@pve -role ProxyOperator +pveum user token add proxy-operator@pve operator --privsep 0 +# The secret is printed once. Save it on the workstation as one line: +# proxy-operator@pve!operator= +# in ~/.config/proxmox-token (mode 0600). The same file becomes the +# Kubernetes Secret later. +cp /etc/pve/pve-root-ca.pem /tmp/ # copy to the workstation as ~/.config/proxmox-ca.pem, or use insecureSkipTLSVerify +``` + +A cloud-init template without a Proxmox cloud-init drive (the provider puts +its own NoCloud ISO on `ide2`). Debian 13's BIOS boot path did not boot +under kvm2, so the template boots UEFI: + +```bash +wget https://cloud.debian.org/images/cloud/trixie/latest/debian-13-generic-amd64.qcow2 +qm create 9000 --name debian-13-template --memory 2048 --cores 2 --ostype l26 \ + --machine q35 --bios ovmf --efidisk0 local-lvm:1,efitype=4m,pre-enrolled-keys=0 \ + --scsihw virtio-scsi-pci --net0 virtio,bridge=vmbr0 \ + --serial0 socket --vga serial0 --agent 1 +qm set 9000 --scsi0 local-lvm:0,import-from=/root/debian-13-generic-amd64.qcow2 +qm set 9000 --boot order=scsi0 +qm template 9000 +``` + +Facts the user hands over at Phase 0: the token file path, the CA file path +(or "insecure"), the node name, the template VMID, the ISO-content storage +id (usually `local`), the VM disk storage (`local-lvm`), the bridge, and +whether the operator's cluster can route to that bridge's LAN. Everything +else Phase 0 reads from the API. + +## Design decisions made with the user (not re-opened) + +- **Backend and client.** Proxmox VE 8.x REST API, JSON, HTTP/1.1 only. + Thin client on `net/http` (`client-kind: thin-client`), no SDK, no new + dependency. Auth header + `Authorization: PVEAPIToken=@!=`, read from a + mounted token file on every request (rotation free). TLS: `caFile` or + `insecureSkipTLSVerify` (warned at startup). Child spans from a parent-gated + otelhttp transport: `tracing.RestConfigWrapper()(base)` returns a plain + `http.RoundTripper` and works as is (`child-spans: transport-wrapper`). +- **Image source.** Clone a prepared template. Config catalogue + `templates: {: {vmid: 9000}}`; `Placement.Image` is a key. + `Placement.Zone` is a node name from `nodes: [..]` (optional with one node). + `Placement.MachineType` is a key of `machineTypes: {: {cores, memoryMiB, + diskGiB}}`; `diskGiB` (optional) grows the boot disk via `PUT .../resize`. + `Placement.Region` is ignored (documented). +- **Identity.** `primary-key: allocated-id`. VMID derived deterministically + from the Proxy UID (FNV-1a 64 into `vmidRange: {min, max}`, default + 100000 to 199999) with forward probing on collision; a clone whose `newid` + is taken fails with a 500, which is the allocate-then-verify step. Create + first looks the name up in `GET /cluster/resources?type=vm`; an existing VM + with that name must carry our mark and a matching or empty UID, else + Permanent. providerID is location-free: `vms//` (VMs migrate; + the node is resolved from `/cluster/resources` on every call). Get and + Delete verify the VM at that VMID has that name and our mark; a different + name means our VM is gone and the VMID was reused: Get returns NotFound, + Delete returns nil without touching it. +- **Cloud-init.** NoCloud seed ISO built in-process by reusing kvm2's seed + code, extracted into a shared package `internal/provider/nocloud` (its own + phase). Uploaded to `isoStorage` (default `local`) on the target node via + `POST /nodes/{node}/storage/{storage}/upload` (multipart, `content=iso`, + `filename=-seed.iso`, streamed through `io.Pipe`, returns a UPID), + attached as `ide2: :iso/-seed.iso,media=cdrom`. The template + must have no Proxmox cloudinit drive on ide2 (documented prerequisite). + Vendor-data installs qemu-guest-agent like kvm2 (`installGuestAgent`, + `vendorData`). +- **GC mark.** `gc-mark: written-metadata` in the VM `description`, which the + clone call accepts (`tags` is not a clone parameter, so tags would be a + mark window). Content, one per line: `proxy-operator-managed=true`, + `proxy-operator-uid=`, `proxy-operator-created-at=`. + `created-at: written-at-create`. The config step after the clone also sets + tag `proxy-operator` for UI filtering only. ListByTag: one + `GET /cluster/resources?type=vm`; candidates are entries with tag + `proxy-operator` or name prefix `proxy-`; a `GET /nodes/{node}/qemu/{vmid}/config` + per candidate for the description; unmarked skipped; one unreadable VM + skipped with a V(1) line unless ctx ended; the resources call failing fails + the whole list (`list-partial: fail-whole`). +- **Create** (`create-return: blocking`, `provisioning-state: create-blocks`, + `async-ops: task-handle` and `backend-lock`), every step idempotent, in + this order: validate placement (Permanent before any call); resources + lookup by name (adopt or allocate); `POST /nodes/{tmplnode}/qemu/{tmplvmid}/clone` + `{newid, name, description, full=1, target?, storage?}`; wait the task + (`exitstatus != OK` is Transient with the message; a VM with `lock=clone` + on retry is waited on); upload the seed (always rebuilt, delete then + upload; before the config step because `ide2` may refuse a volume that + does not exist yet); config (`cores`, `memory`, `net0` bridge override if + configured, `agent=1`, `tags`, `ide2` seed); optional resize; + `POST .../status/start` and wait; poll `status/current` until `running` + (capped); return `vms//`. Retry: name found running returns + the ID; found stopped with our mark continues from the seed step; found + locked waits. +- **Get.** Resources lookup by vmid: absent is NotFound; name mismatch or no + mark is NotFound (logged); `lock` set is Transient; `running` reads + `GET /nodes/{node}/qemu/{vmid}/agent/network-get-interfaces`, first + routable IPv4 excluding loopback and link-local gives Running, else + Provisioning (the agent not running is a 500 "QEMU guest agent is not + running", which is Provisioning, not an error); `stopped` is Stopped; + `unknown` is Transient; `qmpstatus=paused` is Stopped; anything else is + Stopped. UID and CreatedAt come from the description on every call, so a + Get is three calls: resources, config, agent. +- **Delete** (`delete-return: waits`, `secondary-resources: explicit`): + resolve by vmid via resources; absent or foreign is nil (logged); running + gets `POST status/stop` and a task wait; then + `DELETE /nodes/{node}/qemu/{vmid}?purge=1&destroy-unreferenced-disks=1` and + a task wait; then delete the seed ISO + `DELETE /nodes/{node}/storage/{storage}/content/{storage}:iso/-seed.iso` + tolerating absence; a locked VM is Transient. Called again mid-delete it + repeats the same tolerant steps. GC blind spot: seed ISOs without a VM + (documented; the ISO name carries the instance name). +- **Errors**, one `classify(op, err)` over `*apiError{Status, Message, Errors}`: + 401, 403, 400 and 404 are Permanent (404 means the path never exists; + NotFound comes only from the list-and-miss on resources and from the 500 + "does not exist" message in Get and Delete, matched against captured + fixtures); 500 "does not exist" is NotFound (Transient inside Create); 500 + "VM is locked" is Transient; 500 "already exists" is handled by the + allocator, otherwise Transient; 500 "not enough free space" or "no space + left" is QuotaExceeded once a real message is captured, else `quota: + absent`; other 5xx, 595, 596, network and ctx errors are Transient; + anything not an apiError is Transient. Error text: Proxmox puts most of it + in the HTTP status reason phrase (`HTTP/1.1 500 Configuration file '...' + does not exist`) and newer releases also in the body's `message`; the + client reads both, body first. +- **Checker.** One target per configured node, in config order, concurrently, + on the same http client: `GET /version` (Details `pveVersion`), + `GET /nodes/{node}/status` (Details `kernel`), + `GET /nodes/{node}/storage/{isoStorage}/status` (Details `isoStorage`, + `isoStorageAvailGiB`), template existence from `GET /cluster/resources?type=vm` + (the `template` flag, any node; Details `templates` as `/`). + Reasons: the shared ones plus `TemplateNotFound` and `StorageNotFound`. + The `status.type` comment in `api/v1alpha1/providerhealth_types.go` gains + `proxmox`; `make manifests`. +- **Timeouts** (`timeouts: per-call`): request timeout 30s + (`requestTimeoutSeconds`); task-wait caps clone 10m, upload 2m, start and + stop 2m, destroy 5m, running wait 60s; all in a `budgets` struct + overridable in tests. A cancelled ctx cancels that request only; the + connection pool is unaffected. Per-name keyed lock for Create and Delete + (`internal/provider/keyedlock`, extracted from kvm2); Get and ListByTag + lock-free. +- **Logging** (`logging: tiered`): `logf.FromContext(ctx).WithName("proxmox").WithValues("provider", name)`, + `vmid` and `name` on instance lines; V(1) outcomes and one + `proxmox api call` line per seam call from the `observed` decorator; V(2) + request params and response summaries; never the token, never cloud-init, + never the multipart body, never `description` at V(1); no V(5). The + decorator logs, the transport traces: no `proxmox.` spans, so + nothing is double-counted. +- **Config writes** use `PUT .../config` (synchronous; the VM is stopped at + that point). `resize` is skipped when the boot disk's current `size=` is + at or above the target. Boot disk: first disk key in `boot: order=...` + that is not `net*` and not the ide2 cdrom, else `bootdisk`, else the first + of `scsi0`, `virtio0`, `sata0`, `ide0`; none is Permanent. +- **A UPID names its node** (`UPID::...`); task polling uses the node + parsed from the UPID. +- **Two VMs with our name** (Proxmox does not enforce unique names) make + Create and Get return Permanent naming both VMIDs; an admin resolves it. +- **Seam.** `api` interface with one method per endpoint (table below), + `httpAPI` production adapter, `fakeAPI` in `fake_test.go` keeping VMs, tasks + and ISOs in memory with lock simulation, task polls, per-method errors and + block channels, and a call log. +- **Integration test** `proxmox_integration_test.go` behind + `//go:build integration`, `PROXMOX_TEST_URL`, `PROXMOX_TEST_TOKEN_FILE`, + `PROXMOX_TEST_NODE`, `PROXMOX_TEST_TEMPLATE_VMID`, `PROXMOX_TEST_ISO_STORAGE`, + optional `_CA_FILE`, `_INSECURE`, `_BRIDGE`, `_STORAGE`, `_VMID_MIN`, + `_VMID_MAX`, `_TIMEOUT`, `_KEEP`, `_LOG_LEVEL`; `make test-integration-proxmox`. +- **Deployment.** Secret `proxmox-token` mounted optional at + `/etc/proxy-operator/proxmox/token`, `make proxmox-token-secret`, commented + entries in `config/manager/providers_config.yaml` and + `config/samples/providers-config.yaml`, `config/samples/proxy_proxmox.yaml`, + `docs/proxmox-provider.md`, `providers-existing.md` column and section, + `architecture.md` rows, README row, `testing.md` section, CHANGELOG after + confirmation. Branch `feat/proxmox-provider`, MR via the Gitea API (no `tea`). + +## Backend facts Phase 0 must verify on 192.168.2.8 (numbered for citation) + +- F1. Where error text lives: status reason phrase vs. JSON `message`; the + `errors` map shape on a 400. +- F2. Exact 500 message for a missing VM (`GET /nodes/{node}/qemu/999999/config`). +- F3. Exact 500 message for a locked VM (`POST status/start` on a VM with + `lock` set), and whether `PUT config` with `lock=backup` sets the lock and + `delete=lock` clears it through the API, or `qm unlock` is needed. +- F4. Exact 500 message when `newid` is in use on clone. +- F5. Exact 500 message for the agent not running, and for the agent call + on a stopped VM. +- F6. Exact message for no space, if it can be produced cheaply; otherwise + `quota: absent`. +- F7. Whether `PUT .../config` is refused while `lock=clone`, and with which + message. +- F8. `/cluster/resources?type=vm` fields present: `vmid`, `name`, `node`, + `status` values (`running`, `stopped`, `unknown`), `type` (`qemu`, `lxc`), + `lock`, `tags` (separator), `template`; whether `lock` appears there during + a clone. +- F9. Tag rules: allowed characters, case, length, `tags` separator on + `PUT config` and in `/cluster/resources`. +- F10. `description` on `POST .../clone`: accepted, survives verbatim + (newlines), max length; whether `name` accepts the 22-character + `proxy-` form. +- F11. `meta` in the cloned config: is `ctime` the clone time (evidence only; + `written-at-create` stays the profile). +- F12. Upload endpoint: field names (`content`, `filename`, file part name + `filename`), the filename rule (`.iso` suffix, no slashes), the response + (UPID), the privilege it needs, and whether re-uploading an existing name + is refused (then delete-then-upload) or overwrites. +- F13. `DELETE /nodes/{node}/storage/{storage}/content/{volid}`: response + shape (UPID or null) and the message for a missing volume. +- F14. `status/current` on a stopped VM: is `qmpstatus` present; on a paused + VM: `status=running`, `qmpstatus=paused`. +- F15. `POST .../clone`: whether a failed clone destroys the partial VM by + itself, and what a VM interrupted mid-clone looks like (config present, + `lock=clone`). +- F16. Task status shape: `status` (`running`, `stopped`), `exitstatus` + (`OK` or the error string), and how long a full clone of the template + takes. +- F17. Privileges: which of the prepared list the token really needs, in + particular `SDN.Use` on `/sdn/zones/localnetwork/` for the `net0` + write and `VM.GuestAgent.Audit` or `VM.Monitor` for the agent read. +- F18. HTTP/2 is not offered (`curl --http2 -I` falls back or fails), so the + client disables it. +- F19. The template's disk key (`scsi0` or `virtio0`), its `boot` line, that + ide2 is free and no `cloudinit` drive exists. +- F20. `resize` on a stopped VM: request shape, the message when the target + is smaller or equal, and that it is synchronous. +- F21. Agent response shape: `result[].name`, `result[]["ip-addresses"][]` + with `ip-address`, `ip-address-type`. + +## Proxmox endpoint map for the seam + +| `api` method | HTTP | +|---|---| +| `Version(ctx)` | `GET /version` | +| `NodeStatus(ctx, node)` | `GET /nodes/{node}/status` | +| `StorageStatus(ctx, node, storage)` | `GET /nodes/{node}/storage/{storage}/status` | +| `ClusterResources(ctx)` | `GET /cluster/resources?type=vm` | +| `VMConfig(ctx, node, vmid)` | `GET /nodes/{node}/qemu/{vmid}/config` | +| `SetVMConfig(ctx, node, vmid, params)` | `PUT /nodes/{node}/qemu/{vmid}/config` | +| `Clone(ctx, node, vmid, cloneParams)` | `POST /nodes/{node}/qemu/{vmid}/clone` | +| `Resize(ctx, node, vmid, disk, size)` | `PUT /nodes/{node}/qemu/{vmid}/resize` | +| `Start(ctx, node, vmid)` | `POST /nodes/{node}/qemu/{vmid}/status/start` | +| `Stop(ctx, node, vmid)` | `POST /nodes/{node}/qemu/{vmid}/status/stop` | +| `VMStatus(ctx, node, vmid)` | `GET /nodes/{node}/qemu/{vmid}/status/current` | +| `Destroy(ctx, node, vmid)` | `DELETE /nodes/{node}/qemu/{vmid}?purge=1&destroy-unreferenced-disks=1` | +| `TaskStatus(ctx, node, upid)` | `GET /nodes/{node}/tasks/{upid}/status` | +| `AgentNetworkInterfaces(ctx, node, vmid)` | `GET /nodes/{node}/qemu/{vmid}/agent/network-get-interfaces` | +| `UploadISO(ctx, node, storage, filename, r, size)` | `POST /nodes/{node}/storage/{storage}/upload` (multipart) | +| `DeleteVolume(ctx, node, storage, volid)` | `DELETE /nodes/{node}/storage/{storage}/content/{volid}` | + +## Package layout (`internal/provider/proxmox/`) + +`proxmox.go` (`Provider`, `New`, `newProvider`, `withDefaults`, `logger`, +`node`, `wrapErr`, `lockName`, `providerOption`, `budgets`), `api.go` (the +`api` interface and DTOs `vmResource`, `vmConfig`, `vmStatus`, `taskStatus`, +`agentInterface`, `storageStatus`, `nodeStatus`, `versionInfo`, +`cloneParams`), `client.go` (`httpAPI`, `newHTTPAPI`, `do`, `readToken`, +`decodeConfig`), `errors.go` (`apiError`, `classify`, `permanentf`, +`transientf`, `isNotExist`, `isLocked`, `isAlreadyExists`, +`isAgentNotRunning`, `isNoSpace`), `id.go` (`formatProviderID`, +`parseProviderID`, `vmidFor`), `mark.go` (`instanceMark`, +`formatDescription`, `parseDescription`), `state.go` (`mapState`, +`firstIPv4`), `observe.go` (`observed`, `observe`), `task.go` (`parseUPID`, +`waitTask`, `waitUnlocked`, `waitRunning`), `seed.go` (`seedISOName`, +`seedVolID`, `uploadSeed`), `create.go`, `get.go`, `delete.go`, `list.go`, +`check.go`, plus `fake_test.go`, `provider_test.go`, +`proxmox_integration_test.go`, `testdata/*.json`. Shared: +`internal/provider/config_proxmox.go`, `internal/provider/nocloud/`, +`internal/provider/keyedlock/`. + +Reused as-is: `provider.NameFromUID`, `provider.Wrap`, the `Err*` sentinels, +`provider.Label*` constants, `provider.Reason*` constants, +`tracing.RestConfigWrapper`, `tracing.WithTracerProvider`, kvm2's test +idioms (`fixture`, `assertProviderError`, `logCapture`). + +## Steps (one commit per phase, on `feat/proxmox-provider` off `main`) + +- **Phase 0 (orchestrator with the user): plan file, branch, host recon, + fixtures.** Read first: contract §5, §4.9, §7; + `docs/plans-executions/2026-09-16-1333-kvm2-provider.md` Step 0 entry. + Do, in this order: `git switch main && git pull --ff-only`; copy this plan + to `docs/plans/2026-09-18-1502-proxmox-provider.md` with the real timestamp + (`date "+%Y-%m-%d-%H%M"`), commit it alone on `main` + (`docs: plan the proxmox provider`); create `feat/proxmox-provider`. Ask + the user for the hand-over facts listed above. Then, with the user's + go-ahead (each `curl` is a permission prompt; read-only ones first, one at + a time), capture F1 to F21 with + `curl -s --cacert "$CA" -D /tmp/h.txt -o /tmp/b.json -w '%{http_code}\n' -H "Authorization: PVEAPIToken=$(cat "$TOKEN_FILE")" https://192.168.2.8:8006/api2/json/`, + storing each response as `internal/provider/proxmox/testdata/.json` + in the shape `{"status": , "reason": "", "body": ""}`: `version.json`, `node_status.json`, + `storage_status.json`, `cluster_resources.json`, + `vm_config_template.json`, `vm_config_clone.json`, + `status_current_running.json`, `status_current_stopped.json`, + `agent_network_get_interfaces.json`, `task_status_running.json`, + `task_status_stopped_ok.json`, `task_status_stopped_error.json`, + `upload_response.json`, `content_delete_response.json`, + `err_vm_missing.json`, `err_locked.json`, `err_newid_in_use.json`, + `err_agent_not_running.json`, `err_agent_vm_stopped.json`, + `err_unauthorized.json` (a wrong secret), `err_forbidden.json` (a path + the token lacks), `err_bad_param.json` (`newid=abc`), `err_no_space.json` + (only if F6 is cheap), `err_volume_missing.json` (F13). The write + experiments: one clone of the template into the top of the VMID range + (`newid=199999`, a multi-line `description`, a `name` of the `proxy-` + form), a config read of the clone (F10, F11, F19), a `PUT config` while + it is locked if timing allows (F7), an upload of a 1 KiB + `recon-seed.iso` (F12) and its delete (F13), a resize attempt (F20), a + start, an agent read (F5, F21), a stop, then destroy with + `purge=1&destroy-unreferenced-disks=1`; a final resources read shows + nothing remains. Redact nothing but the token. Write the answers to F1 to + F21 and the privilege outcome into the execution log's Phase 0 entry, one + line per fact; Phase 1 cites them as Evidence. Create the execution log + `docs/plans-executions/2026-09-18-1502-proxmox-provider.md` with the checklist (Phase + 0 to Phase 10) and this entry. + Verify: `ls internal/provider/proxmox/testdata | wc -l` is at least 22; + `grep -rl 'PVEAPIToken' internal/provider/proxmox/testdata` is empty; + every fixture parses as JSON; the host shows no VM in the range and no + `recon-seed.iso`. + Done when: the plan commit is on `main`, the branch exists, the fixtures + and the execution log with F1 to F21 are committed. Commit: `proxmox: + captured API fixtures and host facts from 192.168.2.8`. + +- **Phase 1 (orchestrator): the spec.** Read first: contract §2, §4, §5, + §6, §11, §12; `docs/provider-spec-template.md` end to end; + `docs/kvm2-provider-spec.md` for the level of detail; this plan's + decisions; the Phase 0 log entry. Create `docs/proxmox-provider-spec.md` + answering all 24 entries: 1 `vms//`, strict parsing, the + node-name and storage-id regexps; 2 `allocated-id`, `vmidFor(uid, rng, + attempt)`, the two-VMs-with-our-name rule, the adoption guard; 3 the + placement table (Region ignored); 4 the template catalogue and the "add a + key, never edit" warning; 5 the seed ISO via `nocloud` and the ide2 + prerequisite; 6 `machineTypes` with `diskGiB`; 7 `provisioning-window`, + `discovered` via the guest agent, the IPv4 filter, `Port` unused; 8 + `written-metadata` in `description` plus the tag, `written-at-create`, F9 + to F11 as Evidence; 9 the state table, `create-blocks`; 10 the ordered + Create steps with what a retry finds at each, `task-handle` plus + `backend-lock`, the budgets; 11 the Delete order, the ISO blind spot, + node-local ISO storage after a migration; 12 `fail-whole`, the candidate + rule, one config read per candidate; 13 the classify table with the F1 to + F6 messages, `not-found: error-code` plus list-and-miss on resources + (allowed because offline nodes' guests appear as `unknown`, not absent); + 14 `mounted-files` (Secret `proxmox-token`, key `token`, path + `/etc/proxy-operator/proxmox/token`), `ca-file` or + `insecureSkipTLSVerify`, the privilege list from F17, rotation free; 15 + the config table (Phase 2's struct); 16 the Checker; 17 `thin-client`, + `transport-wrapper`, the seam's method list; 18 the budgets table; 19 the + keyed lock per name; 20 the log keys; 21 the §11 list with the test names + from Phases 2 to 8 (copy from this plan); 22 the deployment list; 23 the + docs list; 24 the accepted limits (blocking Create up to the clone budget, + ISO blind spot, VMID range shared with humans, node-local ISO storage, no + adoption of foreign VMs, two-installs GC caveat, `unknown` status while a + node is down). + Verify: `grep -c '^\*\*Answer:\*\*$' docs/proxmox-provider-spec.md` is 0; + every axis code used appears in contract §4.20; Evidence lines cite F + numbers; `grep -c '—' docs/proxmox-provider-spec.md` is 0. + Done when: the spec is committed with all 24 entries and the Phase 1 box + ticked. Commit: `docs: proxmox provider spec`. + +- **Phase 2 (implementer): config block, registration, package skeleton.** + Read first: spec entries 1, 2, 3, 6, 8, 9, 13, 15; contract §10, §5, §6; + `internal/provider/config.go`, `config_kvm.go`, `config_test.go`, + `config_kvm_test.go`; `internal/provider/kvm2/{kvm2.go,id.go,metadata.go,state.go,errors.go}` + and their tests as the pattern; `cmd/main.go`; `testdata/err_*.json`. + Create `internal/provider/config_proxmox.go`: `ProxmoxConfig{URL, + TokenFile, CAFile string; InsecureSkipTLSVerify bool; Nodes []string; + ISOStorage, Storage, Bridge string; VMIDRange *ProxmoxVMIDRange; Templates + map[string]ProxmoxTemplate; MachineTypes map[string]ProxmoxMachineType; + RequestTimeoutSeconds int; VendorData string; InstallGuestAgent *bool}`, + `ProxmoxVMIDRange{Min, Max int}`, `ProxmoxTemplate{VMID int}`, + `ProxmoxMachineType{Cores int32; MemoryMiB int64; DiskGiB int64}`, + `validateProxmox(prefix string, c *ProxmoxConfig) error` (URL http(s) + with a host; `tokenFile` non-empty; at least one node, unique, matching + `proxmoxNodeName = ^[a-z0-9][a-z0-9-]*$`; `isoStorage` and `storage` + matching `proxmoxStorageID = ^[A-Za-z][A-Za-z0-9_.-]*$`; `vmidRange` min + at least 100, max at most 999999999, min below max; at least one template + with `vmid` at least 100; at least one machine type with cores and + memoryMiB above zero and diskGiB not negative; `requestTimeoutSeconds` + not negative). Edit `config.go`: `Proxmox *ProxmoxConfig` with tag + `proxmox,omitempty`, `setBlocks`, the allow-list, the dispatch. Edit + `cmd/main.go`: `"proxmox": proxmox.New`. Create in + `internal/provider/proxmox/`: `proxmox.go` (`Provider{name, cfg, api, + now, budgets, nameLocks, tracing}`; until Phase 3 a local `keyedLocks` + copy in `locks.go` that Phase 3 deletes; `New` never dials: builds the + client, `os.Stat`s the token file and warns, warns on + `insecureSkipTLSVerify`; `withDefaults`: `isoStorage` `local`, `vmidRange` + 100000 to 199999, `requestTimeoutSeconds` 30, `installGuestAgent` true; + `node(zone)` mirrors kvm2's `host`; the four methods stubbed with + `transientf("not implemented")`), `api.go` (the interface and DTOs; + `vmConfig` is `map[string]string` with scalars rendered as strings by + `decodeConfig`), `id.go`, `mark.go` (keys from `provider.LabelManaged`, + `provider.LabelManagedYes`, `provider.LabelUID` and + `proxy-operator-created-at`; `parseDescription` tolerates extra lines + and CRLF), `state.go`, `errors.go` (`apiError{Status int; Message + string; Errors map[string]string}` whose `Error()` never includes headers + or URLs; `classify` per spec entry 13; matcher helpers keyed on the exact + substrings captured in F2 to F6). Tests: `config_proxmox_test.go` + `TestValidateProxmox_table` (missing url, non-http scheme, missing + tokenFile, no nodes, duplicate node, node with a slash, bad isoStorage + id, vmidRange min below 100, min above max, no templates, template vmid + below 100, no machineTypes, cores zero, negative diskGiB, negative + timeout, valid full, valid minimal) with a `validProxmoxYAML` const; + `config_test.go` rows `proxmox type with kvm block`, `kvm type with + proxmox block`, `proxmox minimal entry loads`. `id_test.go`: + `TestFormatProviderID_roundTrip`, `TestParseProviderID_malformed`, + `TestVMIDFor_deterministicAndInRange`, `TestVMIDFor_probesForwardAndWraps`. + `mark_test.go`: `TestFormatDescription_keys`, `TestParseDescription_table`. + `state_test.go`: `TestMapState_table`, `TestFirstIPv4_filters`. + `errors_test.go`: `TestClassify_table` (every class, every matched + message loaded from `testdata/err_*.json` through `loadFixture(t, name) + *apiError`, `op == "create"` turning NotFound into Transient, + `context.Canceled`, a `*net.OpError`, `errors.New("x")`, + already-classified errors keep their class), `TestAPIError_matchers`, + `TestPermanentfAndTransientf`. `proxmox_test.go`: + `TestNew_doesNotDialAndWarnsOnMissingToken`, + `TestNew_insecureSkipTLSVerifyWarns`, `TestNew_nilBlockDoesNotPanic`, + `TestWithDefaults_table`, `TestNode_table`, + `TestWrapErr_carriesProviderNameOpAndID`. + Verify: `grep -c '"proxmox"' internal/provider/config.go` is 3; + `go build ./... && go vet ./... && go test -race -count=1 ./internal/provider/... ./cmd/... && go test -short ./...`; + the fixture loader fails loudly on a missing file. + Done when: `go test -race ./internal/provider/proxmox/` passes with the + stubs in place and the constructor is registered. Commit: `proxmox: + config block, IDs, marks, state mapping, classify, registration`. + +- **Phase 3 (implementer): extract `nocloud` and `keyedlock` from kvm2.** + Read first: `internal/provider/kvm2/{seed.go,seed_test.go,kvm2.go,kvm2_test.go,create.go}`; + contract §1.4, §2.1; spec entries 5 and 19; + `grep -n 'seed.go\|keyedLocks\|buildSeedISO' docs/*.md`. Create + `internal/provider/nocloud/seed.go` exporting `BuildSeedISO(name, + userData, vendorData string, w io.Writer) error` (plain errors, an + `ErrEmptyName` sentinel instead of kvm2's `permanentf`), + `BuildVendorData(configured string, installAgent bool) (string, error)`, + `GuestAgentScript`, `SeedLabel`; move the seed tests to + `nocloud/seed_test.go` under their existing names. Create + `internal/provider/keyedlock/keyedlock.go` exporting `Locks`, `New() + *Locks`, `(*Locks).Acquire(ctx, key) (release func(), err error)`; move + the `TestKeyedLocks_*` tests. Edit kvm2: delete `seed.go`, `seed_test.go`, + the `keyedLocks` type and its tests; `create.go` calls + `nocloud.BuildVendorData` and `nocloud.BuildSeedISO` and wraps their + errors with `permanentf`; `kvm2.go` uses `*keyedlock.Locks`. Edit + `internal/provider/proxmox/proxmox.go` to use `keyedlock.Locks` and delete + `locks.go`. Fix the doc mentions so they name the new packages. + Verify: `go test -race -count=1 ./internal/provider/...`; kvm2's test + count drops by exactly the moved tests and the two new packages carry + them; `grep -rn 'iso9660' internal/provider/kvm2/` is empty; + `go vet -tags=integration ./...` still compiles. + Done when: kvm2 and proxmox import both packages, all tests pass, the docs + name the new packages. Commit: `provider: extract nocloud seed builder + and keyedlock from kvm2`. + +- **Phase 4 (implementer): seam, HTTP client, decorator, fake.** Read first: + spec entries 14, 17, 18, 20; contract §4.15, §7, §9; + `internal/tracing/transport.go`; `internal/provider/kvm2/{observe.go,observe_test.go,fake_test.go}`; + `internal/provider/proxmox/{api.go,errors.go}`; `testdata/*.json`. Create + `client.go`: `httpAPI{base *url.URL; http *http.Client; tokenFile + string}`; `newHTTPAPI(cfg provider.ProxmoxConfig, opts ...tracing.Option) + (*httpAPI, error)` builds a transport with `DialContext` timeout 10s, + `TLSHandshakeTimeout` 10s, `ResponseHeaderTimeout` from the request + timeout, `IdleConnTimeout` 90s, `MaxIdleConnsPerHost` 4, + `ForceAttemptHTTP2: false`, an empty `TLSNextProto` map (F18), `RootCAs` + from `caFile` or `InsecureSkipVerify` from the flag, wrapped with + `tracing.RestConfigWrapper(opts...)(transport)`; `http.Client.Timeout` is + the request timeout; `do(ctx, method, path string, query, form + url.Values, out any) error` reads the token file on every call (missing + file `transientf`, a file without `!` and `=` `permanentf`), sets the + header, decodes `{"data": ...}` into `out`, turns any non-2xx into + `*apiError` per F1 with the error body read capped at 64 KiB; + `UploadISO` streams a multipart body through `io.Pipe` with a goroutine + writing `content=iso`, `filename` and the file part. Create `observe.go`: + `observed{inner api; provider string}` writing one V(1) `proxmox api + call` line per seam call with `call`, `node`, `vmid`, `storage`, `upid`, + `duration`, `httpStatus` and `error` on failure, V(2) adding request + params and a response summary; `observe(inner api, providerName string) + api`; `New` wraps `httpAPI` in it. Create `fake_test.go`: `fakeAPI{mu; vms + map[int]*fakeVM; tasks map[string]*fakeTask; isos + map[string]map[string][]byte; hooks; block; calls; log; taskPolls int; + cloneFailsWith, startFailsWith string; agentNotRunning, cloneLeavesLock + bool}`, `fakeVM{node, name, status, qmpstatus, lock string; template + bool; config map[string]string; ifaces []agentInterface}`, + `fakeTask{node string; polls int; exit string; onDone func()}`; `Clone` + creates the VM with `lock=clone` and a task that clears the lock or, per + `cloneFailsWith`, removes the VM (F15); `Start` and `Stop` flip status + through a task; `Destroy` removes the VM through a task; `UploadISO` + records bytes; `DeleteVolume` errors with the F13 message when absent; + `enter(ctx, method)` for blocks and hooks like kvm2; helpers `addVM`, + `addTemplate`, `failWith`, `blockOn`, `callCount`, `callLog`, + `fakeErr(status int, msg string) *apiError`. Tests: `client_test.go` over + `httptest.NewTLSServer`: `TestClient_sendsTokenHeaderFromFileOnEveryRequest` + (rotate the file between two calls), `TestClient_missingTokenFileIsTransient`, + `TestClient_unparsableTokenFileIsPermanent`, + `TestClient_apiErrorFromStatusReasonAndBody` (serve each + `testdata/err_*.json` verbatim), `TestClient_decodesDataEnvelope`, + `TestClient_http1Only`, `TestClient_requestTimeoutIsTransient`, + `TestClient_cancelledContextReturnsPromptly`, + `TestClient_childSpanUnderParent`, `TestClient_noSpanWithoutParent`, + `TestClient_tokenNeverInErrorsOrLogs` (sentinel secret echoed by the + server in a 401 body), `TestClient_caFileTrustsPrivateCA`, + `TestClient_insecureSkipsVerification`, `TestClient_uploadStreamsMultipart` + (the server parses the multipart; the client never `io.ReadAll`s the + reader), `TestClient_destroyQueryFlags`. `observe_test.go`: + `TestObserve_everySeamMethodLogsOneLine` (reflection over `api`), + `TestObserve_detailOnlyAtV2`, `TestObserve_uploadBodyNeverLogged`, + `TestObserve_errorFieldsOnFailure`, `TestObserve_v0Silent`. + Verify: `go test -race -count=2 ./internal/provider/proxmox/`; + `grep -n 'ReadAll' internal/provider/proxmox/client.go` shows only the + bounded error-body read; `grep -n 'context.Background()' internal/provider/proxmox/*.go` + is empty outside tests. + Done when: the client passes its tests against the fixtures and the fake + compiles against the full `api` interface. Commit: `proxmox: thin HTTP + client, seam, log decorator and in-memory fake`. + +- **Phase 5 (implementer): Get, ListByTag, task helpers.** Read first: spec + entries 7, 8, 9, 12, 13; contract §2.2, §2.4, §4.7, §4.12; + `internal/provider/kvm2/{get.go,get_test.go,list.go,list_test.go}`; the + resources, config, status, agent and task fixtures. Create `task.go`: + `parseUPID(upid) (node string, err error)` (Permanent when malformed), + `waitTask(ctx, log, api, upid, budget) error` polling `TaskStatus` every + `budgets.taskPoll` (2s; tests shrink it) until `status == "stopped"`, + `exitstatus != "OK"` giving `transientf` with the message, the budget + giving `transientf`, ctx giving Transient; `waitUnlocked(ctx, api, node, + vmid, budget)` polling `VMConfig` until no `lock`. Create `get.go`: + `resolve(ctx, api, vmid) (vmResource, bool, error)` (one resources call, + `Type == "qemu"` and the vmid), the name check, `VMConfig` for the mark, + `lock` Transient, `unknown` Transient, `running` reads the agent + (`isAgentNotRunning` gives no IP, other errors Transient), `mapState`. + Create `list.go` per spec entry 12; `Instance` carries ID, UID, CreatedAt + only. Replace the two stubs. Tests: `task_test.go` `TestParseUPID_table`, + `TestWaitTask_table` (ok, error exitstatus, budget exceeded, ctx + cancelled, poll count), `TestWaitUnlocked_table`. `get_test.go`: + `TestGet_missingVMIsNotFound`, `TestGet_nameMismatchIsNotFound`, + `TestGet_unmarkedIsNotFound`, `TestGet_runningWithoutIPIsProvisioning`, + `TestGet_runningWithIPIsRunning`, `TestGet_agentNotRunningIsProvisioning`, + `TestGet_agentOtherErrorIsTransient`, `TestGet_ipFilterSkipsLoopbackAndLinkLocal`, + `TestGet_stoppedIsStopped`, `TestGet_pausedIsStopped`, + `TestGet_unknownStatusIsTransient`, `TestGet_lockedIsTransientNeverNotFound`, + `TestGet_unknownStateIsStopped`, `TestGet_readsUIDAndCreatedAtFromBackend`, + `TestGet_unmanagedDescriptionHasEmptyUID`, `TestGet_malformedIDIsPermanent`, + `TestGet_resourcesFailureIsTransient`, `TestGet_twoVMsWithOurNameIsPermanent`. + `list_test.go`: `TestListByTag_excludesUnmarkedVMs`, + `TestListByTag_emptyUIDStillListed`, `TestListByTag_candidatesByTagOrNamePrefix`, + `TestListByTag_skipsLXCAndTemplates`, `TestListByTag_resourcesFailureFailsWhole`, + `TestListByTag_unreadableVMSkipped`, `TestListByTag_unreadableVMWithEndedContextFails`, + `TestListByTag_unknownStatusStillAttempted`, `TestListByTag_doesNotReadStatusOrAgent`, + `TestListByTag_idsAcceptedByParse`. + Verify: `go test -race -count=2 ./internal/provider/proxmox/`; + `grep -n 'not implemented' internal/provider/proxmox/*.go` lists only + Create and Delete. + Done when: Get and ListByTag are real and every listed test passes. + Commit: `proxmox: Get, ListByTag and task polling`. + +- **Phase 6 (implementer): Create.** Read first: spec entries 2, 3, 4, 5, 6, + 10, 19; contract §2.1, §4.2, §4.10, §4.11, §5; + `internal/provider/kvm2/{create.go,create_test.go}`; + `internal/provider/nocloud`; F4, F7, F10, F12, F15, F16, F19, F20 in the + Phase 0 log. Create `seed.go`: `seedISOName(name)` is `-seed.iso`, + `seedVolID(storage, name)` is `:iso/-seed.iso`, + `uploadSeed(ctx, log, api, node, storage, req, cfg)` builds vendor-data + and the ISO into a `bytes.Buffer` (tens of KiB), deletes an existing + volume tolerating absence, uploads with the buffer's length, waits the + task with `budgets.upload`, logs `cloudInitBytes` only. Create `create.go` + implementing the ordered steps: placement validation (`node`, + `MachineTypes`, `Templates`, all Permanent naming field and value), the + per-name lock, `ClusterResources` once for both the name lookup (adoption + guard, then `waitUnlocked` if locked, then continue from the seed step if + stopped, return if running) and the template's node (absent template is + Permanent); allocation loop `for attempt := 0; attempt < maxProbes (32)` + calling `Clone` with `vmidFor(req.UID, rng, attempt)`, `isAlreadyExists` + moving to the next attempt, exhaustion Transient; `waitTask` with + `budgets.clone`; `uploadSeed`; `SetVMConfig` with `cores`, `memory`, + `agent=1`, `tags=proxy-operator`, `ide2=,media=cdrom`, and `net0` + rewritten from the cloned config with `bridge=` only when + configured (MAC kept); `bootDisk(cfg)` and `Resize` when `diskGiB` is + above the current `size=`; `Start` and `waitTask` with `budgets.start`; + `waitRunning` on `VMStatus` with `budgets.running`; return the ID. + `isNotExist` inside Create is Transient (classify with `op == "create"`). + Tests `create_test.go`: `TestCreate_repeatReturnsSameIDAndCreatesNothing`, + `TestCreate_resumesAfterEachPartialStep` (after clone task, after seed + upload, after config, after resize, after start, after running), + `TestCreate_returnsOnlyWhenRunning`, `TestCreate_runningWaitCapIsTransient`, + `TestCreate_rejectsUnknownPlacement`, `TestCreate_zoneOptionalWithOneNode`, + `TestCreate_regionIgnored`, `TestCreate_adoptionWithForeignUIDIsPermanent`, + `TestCreate_adoptsOnlyMarkedVMs`, `TestCreate_duplicateNameIsPermanent`, + `TestCreate_vmidCollisionProbesForward`, `TestCreate_vmidRangeExhaustedIsTransient`, + `TestCreate_waitsForCloneLockOnRetry`, `TestCreate_cloneTaskFailureIsTransient`, + `TestCreate_cloneTaskTimeoutIsTransient`, `TestCreate_markWrittenInCloneCall`, + `TestCreate_tagSetInConfigStep`, `TestCreate_seedAlwaysRebuilt`, + `TestCreate_seedUploadedBeforeConfig`, `TestCreate_configSetsCoresMemoryAgentAndSeed`, + `TestCreate_bridgeOverrideKeepsMAC`, `TestCreate_noBridgeOverrideLeavesNet0`, + `TestCreate_resizeOnlyWhenLarger`, `TestBootDisk_table`, + `TestCreate_startTaskFailureIsTransient`, `TestCreate_templateMissingIsPermanent`, + `TestCreate_notExistInsideCreateIsTransientNotNotFound`, + `TestCreate_cloudInitSentinelNeverLoggedOrInErrors` (V(2) capture, every + error path), `TestCreate_concurrentSameNameSerialised`, + `TestCreate_cancelledMidWaitReturnsPromptly`. + Verify: `grep -n 'CloudInit' internal/provider/proxmox/create.go internal/provider/proxmox/seed.go` + shows only `len(req.CloudInit)` and the pass-through into + `nocloud.BuildSeedISO`; `go test -race -count=2 ./internal/provider/proxmox/`; + `grep -n 'not implemented'` lists only Delete. + Done when: Create is real and every listed test passes. Commit: + `proxmox: Create by cloning a template, seed upload, config, start`. + +- **Phase 7 (implementer): Delete, Checker, CRD comment.** Read first: spec + entries 11, 16; contract §2.3, §4.3, §4.13, §8; + `internal/provider/kvm2/{delete.go,delete_test.go,check.go,check_test.go}`; + `internal/provider/check.go`; `api/v1alpha1/providerhealth_types.go`; F13, + F17 in the Phase 0 log. Create `delete.go`: parse the ID (Permanent), the + per-name lock, `resolve`; absent is a logged no-op; name mismatch or + unmarked is a logged no-op; `lock` Transient; `unknown` Transient; + `running` gets `Stop` and `waitTask` (`budgets.stop`); `Destroy` and + `waitTask` (`budgets.destroy`), `isNotExist` from `Destroy` being + success; `DeleteVolume` of the seed on the VM's node tolerating the F13 + absent message; nil. Create `check.go`: `var _ provider.Checker = + (*Provider)(nil)`; `reasonTemplateNotFound = "TemplateNotFound"`, + `reasonStorageNotFound = "StorageNotFound"`; `Check` fans out one + goroutine per node in config order (`wg.Go`), each `checkNode` running + `Version`, `NodeStatus`, `StorageStatus` (a 500 "does not exist" is + `StorageNotFound`; a storage whose `content` lacks `iso` is + `StorageNotFound` with a message saying so), `ClusterResources` for the + templates; `checkReason(err)`: ctx ended `Timeout`, 401 `Unauthorized`, + 403 `Forbidden`, other `*apiError` `APIError`, network `Unreachable`; + Details `pveVersion`, `kernel`, `isoStorage`, `isoStorageAvailGiB`, + `templates`; `Name` is the node, `Address` the base URL. Edit + `api/v1alpha1/providerhealth_types.go` (`kubernetes, gcp, kvm, kvm2 or + proxmox`) and run `make manifests`. Tests `delete_test.go`: + `TestDelete_malformedIDIsPermanent`, `TestDelete_twiceSucceeds`, + `TestDelete_absentIsNoOp`, `TestDelete_foreignNameIsNoOp`, + `TestDelete_unmarkedIsNoOp`, `TestDelete_runningVMStoppedFirst`, + `TestDelete_stoppedVMSkipsStop`, `TestDelete_lockedIsTransient`, + `TestDelete_unknownStatusIsTransient`, `TestDelete_removesSeedISO`, + `TestDelete_toleratesMissingISO`, `TestDelete_destroyTaskFailureIsTransient`, + `TestDelete_usesPurgeAndDestroyUnreferencedDisks`, + `TestDelete_concurrentSameNameSafe`, `TestDelete_noOpLinesAreSilentAtV0`; + in `get_test.go`: `TestGet_afterDeleteReturnsNotFound`, + `TestGet_duringDeleteStillReturnsInstance` (block the fake's `Destroy`). + `check_test.go`: `TestCheck_oneEntryPerNodeInConfigOrder`, + `TestCheck_nodesCheckedConcurrently`, `TestCheck_expiredContextReturnsPromptly`, + `TestCheck_reasonsTable`, `TestCheck_detailsKeys`, `TestCheck_logsVerdictPerNode`, + `TestCheckReason_table`. + Verify: `go test -race -count=2 ./internal/provider/proxmox/ ./internal/providerhealth/`; + `git diff --stat config/` shows only the CRD description line; + `grep -rn 'not implemented' internal/provider/proxmox/` is empty. + Done when: all four contract methods and `Check` are real. Commit: + `proxmox: Delete with seed cleanup, health Checker, CRD type comment`. + +- **Phase 8 (implementer): contract sweep, observability sweep, integration + test, Makefile, testing.md.** Read first: contract §11 and §9; spec + entries 17, 18, 20, 21; `internal/provider/kvm2/{provider_test.go,kvm2_integration_test.go}`; + `internal/provider/tracing_test.go`; `Makefile` integration targets; + `docs/testing.md` kvm2 rows and section. Create `provider_test.go`: the + shared `fixture` (fake, log capture, budgets shrunk), `assertProviderError`, + `TestProvider_cancelledContextReturnsPromptly` (each method blocked on + its first seam call), `TestProvider_everyErrorIsWrappedWithConfiguredName`, + `TestLogging_v0SilentV1OutcomeV2Detail` (V(1) lines `proxmox vm ensured + running`, `proxmox vm fetched`, `proxmox vm deleted`, `proxmox vms + listed`, `proxmox node checked`, every line with `provider`, instance + lines with `vmid` and `name`; V(2) has `cloudInitBytes` and never the + text), `TestTracing_httpSpansUnderOperationSpan` (a real `httpAPI` + against an `httptest` server playing the fixtures for a Get, wrapped in + `provider.WithTracing`, with a span recorder: every HTTP span is a child + of `provider.get` and no attribute carries the token). Create + `proxmox_integration_test.go` (`//go:build integration`): + `loadIntegrationEnv` with the `PROXMOX_TEST_*` variables, `providerConfig()`, + the `integrationLog` pattern from kvm2; `TestProxmox_lifecycle` asserting + in order: `New` returns within 100 ms without a request; `Check` Ready + with `pveVersion`, `isoStorageAvailGiB` and `templates`; `Create` returns + `vms//` with the vmid in range; a second `Create` returns the + same ID and the resources count is unchanged; `Get` reaches Running with + an IPv4 within the budget, UID and CreatedAt as written; `ListByTag` + contains it; a TCP connect to the guest's port is logged only; `Delete` + nil; `Get` NotFound; `Delete` again nil; the ISO storage has no + `-seed.iso` and resources has no VM with the name; an HTTP span is + a child of `provider.create` and no attribute carries the cloud-init + text; `PROXMOX_TEST_KEEP` skips the teardown and prints the `qm destroy` + and ISO delete commands. `TestProxmoxCheck_realHost` creates nothing. Edit + `Makefile`: `test-integration-proxmox` (`go test -race -tags=integration + -count=1 -v -timeout 30m -run TestProxmox ./internal/provider/proxmox/`). + Edit `docs/testing.md`: rows for the package in the suites table and a + "The proxmox integration test" section. Update spec entry 21's tick list + with the final test names. + Verify: `go test -race -count=2 ./internal/provider/proxmox/`; + `go vet -tags=integration ./... && go vet -tags=e2e ./...`; + `make -n test-integration-proxmox`; every §11 item has a test name in + spec entry 21 that exists. + Done when: the package covers every §11 item by name, the integration + test compiles, the target exists. Commit: `proxmox: contract and + observability tests, integration test, testing docs`. + +- **Phase 9 (implementer): deployment and documentation.** Read first: + contract §12 items 11 to 13; spec entries 22, 23, 24; + `config/manager/manager.yaml`, `config/manager/providers_config.yaml`, + `config/samples/{providers-config.yaml,proxy_kvm2.yaml,kustomization.yaml}`; + `Makefile` `kvm-ssh-secret`; `docs/kvm2-provider.md` headings; + `docs/providers-existing.md`; `docs/architecture.md` lines mentioning + kvm2; `README.md`; `docs/providers.md`; `CLAUDE.md` Gotchas. Edit + `config/manager/manager.yaml`: an optional Secret mount `proxmox-token` at + `/etc/proxy-operator/proxmox` (key `token`). Edit `Makefile`: + `PROXMOX_TOKEN_FILE ?= $(HOME)/.config/proxmox-token`, + `proxmox-token-secret` creating `proxmox-token` from `--from-file=token=`. + Edit both providers-config files: a commented `proxmox-home` entry (`url: + https://192.168.2.8:8006`, `tokenFile`, `caFile` or + `insecureSkipTLSVerify`, `nodes`, `isoStorage`, `storage`, `bridge`, + `vmidRange`, `templates`, `machineTypes`). Create + `config/samples/proxy_proxmox.yaml` (provider `proxmox-home`, zone the + node, machine type `small`, image the template key, the Squid cloud-init + from `proxy_kvm2.yaml`) and list it in `config/samples/kustomization.yaml`. + Create `docs/proxmox-provider.md` with sections: Proxmox prerequisites + (the template commands from this plan, no cloudinit drive on ide2, the + guest agent or outbound access, the bridge, the privilege list from F17, + reserving the VMID range), Cluster-side setup (the Secret, the config + entry), Config reference (spec entry 15's table), Placement (Region + ignored, catalogue-edit warning), What lands on Proxmox (VMID scheme, + description mark, tag, seed ISO, config keys, `qm unlock` for a stuck + clone), Logging, Tracing, Timeouts, Health check, Error classes, Known + gaps (spec entry 24), Testing. Edit `docs/providers-existing.md`: a fifth + column in the three tables and a proxmox section mirroring kvm2's. Edit + `docs/architecture.md`: components row, a proxmox block in §5, the span + note in §10, a §11 row (and the "All four providers" count), Decisions + bullets. Edit `README.md`, `docs/providers.md`, `CLAUDE.md` Gotchas + (`make test-integration-proxmox`). + Verify: `bin/kustomize build config/default >/dev/null && bin/kustomize build config/samples >/dev/null`; + a test `TestLoadConfig_proxmoxSampleEntry` loads the uncommented block + from `config/samples/providers-config.yaml` through `provider.LoadConfig`; + a link sweep over the edited docs finds no missing target; + `grep -c '—' docs/proxmox-provider.md` is 0. + Done when: manifests build, the sample loads, docs are linked and + link-clean. Commit: `proxmox: deployment files, sample, user guide and + doc rows`. + +- **Phase 10 (orchestrator with the user): review, real-host validation, + changelog, MR.** Read first: `docs/plans-executions/2026-09-17-1515-kvm2-review-fixes.md`; + `.claude/agents/go-operator-reviewer.md`; the latest report under + `docs/reviews/` for the shape; the spec; this plan's Risks. Do: run + `go-operator-reviewer` on the branch with the spec and the contract as + inputs, then `/code-review high`, `make lint` (if the local toolchain + panics in staticcheck as it did during kvm2, record it and run + `bin/golangci-lint run ./internal/provider/proxmox/...` with staticcheck + disabled), `make test`. Fix accepted findings with regression tests in + one commit; record rejected ones under `docs/reviews/-proxmox-provider-review/`. + Hand the user the host checklist (template ready, no leftover VMs in the + range, ISO storage has space, the token's privileges per F17) and the + `PROXMOX_TEST_*` export lines; with the go-ahead run + `make test-integration-proxmox` and record the clone duration and time to + IP. Optionally the in-cluster pass on kind: build and load the image, + `make proxmox-token-secret`, a scratch ConfigMap with the `proxmox-home` + entry, apply `proxy_proxmox.yaml`, wait for `Ready`, `kubectl get + providerhealth proxmox-home -o yaml`, a `curl -x` through the proxy, + delete and confirm the VM and ISO are gone; scratch overrides are not + committed. After the user confirms: CHANGELOG entry + (`date "+%Y-%m-%d %H:%M %Z"`), push with `-u` using the token helper from + memory (`~/.config/gitea-token`), open the MR with the Gitea API + (`POST /api/v1/repos/kacerr/egress-proxies-operator/pulls` with `head: + feat/proxmox-provider`, `base: main`, the body ending with the + attribution line; `pr.json` in the scratchpad, never committed). Never + merge. + Verify: the integration run's ordered assertions pass on 192.168.2.8; + `make test` green; the review report lists every finding as accepted or + rejected. + Done when: the MR URL is printed and the execution log's last box is + ticked. Commit: `proxmox: review fixes, real-host validation record, + changelog`. + +## Verification + +- Per phase: the `Verify:` clause, plus `go build ./... && go test -short ./...` + before every commit and `git diff --stat` reviewed for files outside the + phase's list. +- Contract mapping: after Phase 8 every rule of contract §2 and §8 has a + named test in the package, and spec entry 21 maps the §11 list 1:1 to + existing test names. +- Fixture discipline: every message `classify` or a matcher helper compares + against is loaded from `testdata/` in `TestClassify_table` or + `TestAPIError_matchers`; a string literal in `errors.go` without a fixture + is a review finding. +- Secrets: `TestClient_tokenNeverInErrorsOrLogs` and + `TestCreate_cloudInitSentinelNeverLoggedOrInErrors` run at V(2), and the + integration test asserts no span attribute carries the cloud-init text. +- End to end: the integration run (Phase 8's assertions) on 192.168.2.8 and + the optional kind pass in Phase 10. + +## Risks + +- **Error text location and exact wording (F1 to F6)** decide the whole + NotFound path; if messages vary by storage type or PVE minor version, the + matchers get one fixture per variant and the guide names the tested + version. +- **Clone cleanup (F15).** A failed or interrupted clone may leave a VM with + `lock=clone` that never clears; Create then waits out `budgets.clone` and + returns Transient every time. The guide documents `qm unlock`; the + reconciler keeps retrying rather than latching. +- **`SDN.Use` (F17).** PVE 8 refuses `net0` writes to non-root users without + it; a missing privilege is a 403 Permanent that latches the Proxy until + its spec changes. The guide's privilege list is the mitigation. +- **The ide2 seed and a template with a cloudinit drive** both claim + cloud-init; the Proxy's user-data would be ignored. Documented + prerequisite; the Checker cannot see it. +- **VMID range shared with humans.** A human VM inside the range costs a + probe; a human VM carrying our name is Permanent. The guide says to + reserve the range. +- **Node-local ISO storage after a migration** leaves the seed on the + original node; Delete tolerates its absence and the ISO is a documented + blind spot. Shared ISO storage avoids it. +- **Blocking Create** holds a reconcile worker for up to the clone budget; + the guide sets expectations. +- **The 30s request timeout on the seed upload** is fine for tens of KiB; a + multi-megabyte `vendorData` would trip it. Documented. +- **Two operator installs** on one Proxmox cluster delete each other's VMs + (contract §13); documented in entry 24. +- **kvm2 edits in Phase 3** touch a merged provider; the moved tests are the + guard, and `go vet -tags=integration` keeps its integration test compiling.