kacerr/egress-proxies-operator
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

docs: plan the proxmox provider

Browse Source 
Eleven self-contained phases (host recon and fixtures, spec, config and
skeleton, nocloud/keyedlock extraction from kvm2, HTTP client and fake,
Get/List, Create, Delete/Checker, test sweep and integration test,
deployment and docs, review and MR), with the Proxmox host setup
checklist and the backend facts Phase 0 must verify on 192.168.2.8.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
This commit is contained in:
kacerr
2026-09-18 15:02:53 +00:00
co-authored by Claude Fable 5.1
parent 8fa9e8aad9
commit 708989c40f
1 changed files with 844 additions and 0 deletions
Download Patch File Download Diff File Expand all files Collapse all files
docs/plans/2026-09-18-1502-proxmox-provider.md
+844
View File
@@ -0,0 +1,844 @@
# Plan: proxmox, a Proxmox VE provider built from the provider contract
**Created:** 2026-09-18 15:02
## Context
The provider contract (`docs/provider-contract.md`), the spec template
(`docs/provider-spec-template.md`) and the kvm2 build proved the workflow:
spec first, then implementation against the contract, one commit per step.
This plan adds a third VM-style backend, `proxmox`, on the Proxmox VE 8.x
REST API at `https://<host>:8006/api2/json`. The user has a real host at
192.168.2.8 and prepares the API token and a cloud-init-capable template
before Phase 0 (checklist below).
The implementation runs in sessions whose context gets compacted, so every
phase is self-contained: it names what to read first, what it creates or
edits, the test names it must add, a `Verify:` clause, a "Done when" line
and a commit subject. A fresh session reading only this plan, the spec
(`docs/proxmox-provider-spec.md`, from Phase 1 on) and the execution log's
checklist (`docs/plans-executions/2026-09-18-1502-proxmox-provider.md`) must be able to
pick up where the last one stopped.
**Resumption protocol, run at the start of every phase.** Read this plan's
Context and the phase entry; read the spec; read the execution log's
checklist; run `git branch --show-current` (must be `feat/proxmox-provider`
from Phase 0 on), `git status --short` and `git log --oneline -8`. The first
unchecked box is the phase to do. If its files already exist partially, its
"Done when" line says whether it counts as done; finish it, never redo it. A
phase is finished only when its "Done when" holds and its execution-log entry
(a few sentences, exact commands, "worth noting", the box ticked) is
committed together with the code. Follow `CLAUDE.md` for the log format.
## Before Phase 0: what the user prepares on 192.168.2.8
Run as root on the Proxmox node. Drop any privilege `pveum` rejects on the
installed version (`VM.GuestAgent.Audit` exists from 8.2; older releases use
`VM.Monitor` for the agent read) and note it for F17.
```bash
pveum user add proxy-operator@pve --comment "egress-proxies-operator"
pveum role add ProxyOperator -privs "VM.Allocate VM.Clone VM.Audit VM.Config.CDROM VM.Config.CPU VM.Config.Memory VM.Config.Disk VM.Config.Network VM.Config.Options VM.PowerMgmt VM.Monitor VM.GuestAgent.Audit Datastore.Audit Datastore.AllocateSpace Datastore.AllocateTemplate Sys.Audit SDN.Use"
pveum aclmod / -user proxy-operator@pve -role ProxyOperator
pveum user token add proxy-operator@pve operator --privsep 0
# The secret is printed once. Save it on the workstation as one line:
# proxy-operator@pve!operator=<secret>
# in ~/.config/proxmox-token (mode 0600). The same file becomes the
# Kubernetes Secret later.
cp /etc/pve/pve-root-ca.pem /tmp/ # copy to the workstation as ~/.config/proxmox-ca.pem, or use insecureSkipTLSVerify
```
A cloud-init template without a Proxmox cloud-init drive (the provider puts
its own NoCloud ISO on `ide2`). Debian 13's BIOS boot path did not boot
under kvm2, so the template boots UEFI:
```bash
wget https://cloud.debian.org/images/cloud/trixie/latest/debian-13-generic-amd64.qcow2
qm create 9000 --name debian-13-template --memory 2048 --cores 2 --ostype l26 \
--machine q35 --bios ovmf --efidisk0 local-lvm:1,efitype=4m,pre-enrolled-keys=0 \
--scsihw virtio-scsi-pci --net0 virtio,bridge=vmbr0 \
--serial0 socket --vga serial0 --agent 1
qm set 9000 --scsi0 local-lvm:0,import-from=/root/debian-13-generic-amd64.qcow2
qm set 9000 --boot order=scsi0
qm template 9000
```
Facts the user hands over at Phase 0: the token file path, the CA file path
(or "insecure"), the node name, the template VMID, the ISO-content storage
id (usually `local`), the VM disk storage (`local-lvm`), the bridge, and
whether the operator's cluster can route to that bridge's LAN. Everything
else Phase 0 reads from the API.
## Design decisions made with the user (not re-opened)
- **Backend and client.** Proxmox VE 8.x REST API, JSON, HTTP/1.1 only.
Thin client on `net/http` (`client-kind: thin-client`), no SDK, no new
dependency. Auth header
`Authorization: PVEAPIToken=<user>@<realm>!<tokenid>=<secret>`, read from a
mounted token file on every request (rotation free). TLS: `caFile` or
`insecureSkipTLSVerify` (warned at startup). Child spans from a parent-gated
otelhttp transport: `tracing.RestConfigWrapper()(base)` returns a plain
`http.RoundTripper` and works as is (`child-spans: transport-wrapper`).
- **Image source.** Clone a prepared template. Config catalogue
`templates: {<key>: {vmid: 9000}}`; `Placement.Image` is a key.
`Placement.Zone` is a node name from `nodes: [..]` (optional with one node).
`Placement.MachineType` is a key of `machineTypes: {<key>: {cores, memoryMiB,
diskGiB}}`; `diskGiB` (optional) grows the boot disk via `PUT .../resize`.
`Placement.Region` is ignored (documented).
- **Identity.** `primary-key: allocated-id`. VMID derived deterministically
from the Proxy UID (FNV-1a 64 into `vmidRange: {min, max}`, default
100000 to 199999) with forward probing on collision; a clone whose `newid`
is taken fails with a 500, which is the allocate-then-verify step. Create
first looks the name up in `GET /cluster/resources?type=vm`; an existing VM
with that name must carry our mark and a matching or empty UID, else
Permanent. providerID is location-free: `vms/<vmid>/<name>` (VMs migrate;
the node is resolved from `/cluster/resources` on every call). Get and
Delete verify the VM at that VMID has that name and our mark; a different
name means our VM is gone and the VMID was reused: Get returns NotFound,
Delete returns nil without touching it.
- **Cloud-init.** NoCloud seed ISO built in-process by reusing kvm2's seed
code, extracted into a shared package `internal/provider/nocloud` (its own
phase). Uploaded to `isoStorage` (default `local`) on the target node via
`POST /nodes/{node}/storage/{storage}/upload` (multipart, `content=iso`,
`filename=<name>-seed.iso`, streamed through `io.Pipe`, returns a UPID),
attached as `ide2: <storage>:iso/<name>-seed.iso,media=cdrom`. The template
must have no Proxmox cloudinit drive on ide2 (documented prerequisite).
Vendor-data installs qemu-guest-agent like kvm2 (`installGuestAgent`,
`vendorData`).
- **GC mark.** `gc-mark: written-metadata` in the VM `description`, which the
clone call accepts (`tags` is not a clone parameter, so tags would be a
mark window). Content, one per line: `proxy-operator-managed=true`,
`proxy-operator-uid=<uid>`, `proxy-operator-created-at=<RFC3339>`.
`created-at: written-at-create`. The config step after the clone also sets
tag `proxy-operator` for UI filtering only. ListByTag: one
`GET /cluster/resources?type=vm`; candidates are entries with tag
`proxy-operator` or name prefix `proxy-`; a `GET /nodes/{node}/qemu/{vmid}/config`
per candidate for the description; unmarked skipped; one unreadable VM
skipped with a V(1) line unless ctx ended; the resources call failing fails
the whole list (`list-partial: fail-whole`).
- **Create** (`create-return: blocking`, `provisioning-state: create-blocks`,
`async-ops: task-handle` and `backend-lock`), every step idempotent, in
this order: validate placement (Permanent before any call); resources
lookup by name (adopt or allocate); `POST /nodes/{tmplnode}/qemu/{tmplvmid}/clone`
`{newid, name, description, full=1, target?, storage?}`; wait the task
(`exitstatus != OK` is Transient with the message; a VM with `lock=clone`
on retry is waited on); upload the seed (always rebuilt, delete then
upload; before the config step because `ide2` may refuse a volume that
does not exist yet); config (`cores`, `memory`, `net0` bridge override if
configured, `agent=1`, `tags`, `ide2` seed); optional resize;
`POST .../status/start` and wait; poll `status/current` until `running`
(capped); return `vms/<vmid>/<name>`. Retry: name found running returns
the ID; found stopped with our mark continues from the seed step; found
locked waits.
- **Get.** Resources lookup by vmid: absent is NotFound; name mismatch or no
mark is NotFound (logged); `lock` set is Transient; `running` reads
`GET /nodes/{node}/qemu/{vmid}/agent/network-get-interfaces`, first
routable IPv4 excluding loopback and link-local gives Running, else
Provisioning (the agent not running is a 500 "QEMU guest agent is not
running", which is Provisioning, not an error); `stopped` is Stopped;
`unknown` is Transient; `qmpstatus=paused` is Stopped; anything else is
Stopped. UID and CreatedAt come from the description on every call, so a
Get is three calls: resources, config, agent.
- **Delete** (`delete-return: waits`, `secondary-resources: explicit`):
resolve by vmid via resources; absent or foreign is nil (logged); running
gets `POST status/stop` and a task wait; then
`DELETE /nodes/{node}/qemu/{vmid}?purge=1&destroy-unreferenced-disks=1` and
a task wait; then delete the seed ISO
`DELETE /nodes/{node}/storage/{storage}/content/{storage}:iso/<name>-seed.iso`
tolerating absence; a locked VM is Transient. Called again mid-delete it
repeats the same tolerant steps. GC blind spot: seed ISOs without a VM
(documented; the ISO name carries the instance name).
- **Errors**, one `classify(op, err)` over `*apiError{Status, Message, Errors}`:
401, 403, 400 and 404 are Permanent (404 means the path never exists;
NotFound comes only from the list-and-miss on resources and from the 500
"does not exist" message in Get and Delete, matched against captured
fixtures); 500 "does not exist" is NotFound (Transient inside Create); 500
"VM is locked" is Transient; 500 "already exists" is handled by the
allocator, otherwise Transient; 500 "not enough free space" or "no space
left" is QuotaExceeded once a real message is captured, else `quota:
absent`; other 5xx, 595, 596, network and ctx errors are Transient;
anything not an apiError is Transient. Error text: Proxmox puts most of it
in the HTTP status reason phrase (`HTTP/1.1 500 Configuration file '...'
does not exist`) and newer releases also in the body's `message`; the
client reads both, body first.
- **Checker.** One target per configured node, in config order, concurrently,
on the same http client: `GET /version` (Details `pveVersion`),
`GET /nodes/{node}/status` (Details `kernel`),
`GET /nodes/{node}/storage/{isoStorage}/status` (Details `isoStorage`,
`isoStorageAvailGiB`), template existence from `GET /cluster/resources?type=vm`
(the `template` flag, any node; Details `templates` as `<found>/<configured>`).
Reasons: the shared ones plus `TemplateNotFound` and `StorageNotFound`.
The `status.type` comment in `api/v1alpha1/providerhealth_types.go` gains
`proxmox`; `make manifests`.
- **Timeouts** (`timeouts: per-call`): request timeout 30s
(`requestTimeoutSeconds`); task-wait caps clone 10m, upload 2m, start and
stop 2m, destroy 5m, running wait 60s; all in a `budgets` struct
overridable in tests. A cancelled ctx cancels that request only; the
connection pool is unaffected. Per-name keyed lock for Create and Delete
(`internal/provider/keyedlock`, extracted from kvm2); Get and ListByTag
lock-free.
- **Logging** (`logging: tiered`): `logf.FromContext(ctx).WithName("proxmox").WithValues("provider", name)`,
`vmid` and `name` on instance lines; V(1) outcomes and one
`proxmox api call` line per seam call from the `observed` decorator; V(2)
request params and response summaries; never the token, never cloud-init,
never the multipart body, never `description` at V(1); no V(5). The
decorator logs, the transport traces: no `proxmox.<Method>` spans, so
nothing is double-counted.
- **Config writes** use `PUT .../config` (synchronous; the VM is stopped at
that point). `resize` is skipped when the boot disk's current `size=` is
at or above the target. Boot disk: first disk key in `boot: order=...`
that is not `net*` and not the ide2 cdrom, else `bootdisk`, else the first
of `scsi0`, `virtio0`, `sata0`, `ide0`; none is Permanent.
- **A UPID names its node** (`UPID:<node>:...`); task polling uses the node
parsed from the UPID.
- **Two VMs with our name** (Proxmox does not enforce unique names) make
Create and Get return Permanent naming both VMIDs; an admin resolves it.
- **Seam.** `api` interface with one method per endpoint (table below),
`httpAPI` production adapter, `fakeAPI` in `fake_test.go` keeping VMs, tasks
and ISOs in memory with lock simulation, task polls, per-method errors and
block channels, and a call log.
- **Integration test** `proxmox_integration_test.go` behind
`//go:build integration`, `PROXMOX_TEST_URL`, `PROXMOX_TEST_TOKEN_FILE`,
`PROXMOX_TEST_NODE`, `PROXMOX_TEST_TEMPLATE_VMID`, `PROXMOX_TEST_ISO_STORAGE`,
optional `_CA_FILE`, `_INSECURE`, `_BRIDGE`, `_STORAGE`, `_VMID_MIN`,
`_VMID_MAX`, `_TIMEOUT`, `_KEEP`, `_LOG_LEVEL`; `make test-integration-proxmox`.
- **Deployment.** Secret `proxmox-token` mounted optional at
`/etc/proxy-operator/proxmox/token`, `make proxmox-token-secret`, commented
entries in `config/manager/providers_config.yaml` and
`config/samples/providers-config.yaml`, `config/samples/proxy_proxmox.yaml`,
`docs/proxmox-provider.md`, `providers-existing.md` column and section,
`architecture.md` rows, README row, `testing.md` section, CHANGELOG after
confirmation. Branch `feat/proxmox-provider`, MR via the Gitea API (no `tea`).
## Backend facts Phase 0 must verify on 192.168.2.8 (numbered for citation)
- F1. Where error text lives: status reason phrase vs. JSON `message`; the
`errors` map shape on a 400.
- F2. Exact 500 message for a missing VM (`GET /nodes/{node}/qemu/999999/config`).
- F3. Exact 500 message for a locked VM (`POST status/start` on a VM with
`lock` set), and whether `PUT config` with `lock=backup` sets the lock and
`delete=lock` clears it through the API, or `qm unlock` is needed.
- F4. Exact 500 message when `newid` is in use on clone.
- F5. Exact 500 message for the agent not running, and for the agent call
on a stopped VM.
- F6. Exact message for no space, if it can be produced cheaply; otherwise
`quota: absent`.
- F7. Whether `PUT .../config` is refused while `lock=clone`, and with which
message.
- F8. `/cluster/resources?type=vm` fields present: `vmid`, `name`, `node`,
`status` values (`running`, `stopped`, `unknown`), `type` (`qemu`, `lxc`),
`lock`, `tags` (separator), `template`; whether `lock` appears there during
a clone.
- F9. Tag rules: allowed characters, case, length, `tags` separator on
`PUT config` and in `/cluster/resources`.
- F10. `description` on `POST .../clone`: accepted, survives verbatim
(newlines), max length; whether `name` accepts the 22-character
`proxy-<base32>` form.
- F11. `meta` in the cloned config: is `ctime` the clone time (evidence only;
`written-at-create` stays the profile).
- F12. Upload endpoint: field names (`content`, `filename`, file part name
`filename`), the filename rule (`.iso` suffix, no slashes), the response
(UPID), the privilege it needs, and whether re-uploading an existing name
is refused (then delete-then-upload) or overwrites.
- F13. `DELETE /nodes/{node}/storage/{storage}/content/{volid}`: response
shape (UPID or null) and the message for a missing volume.
- F14. `status/current` on a stopped VM: is `qmpstatus` present; on a paused
VM: `status=running`, `qmpstatus=paused`.
- F15. `POST .../clone`: whether a failed clone destroys the partial VM by
itself, and what a VM interrupted mid-clone looks like (config present,
`lock=clone`).
- F16. Task status shape: `status` (`running`, `stopped`), `exitstatus`
(`OK` or the error string), and how long a full clone of the template
takes.
- F17. Privileges: which of the prepared list the token really needs, in
particular `SDN.Use` on `/sdn/zones/localnetwork/<bridge>` for the `net0`
write and `VM.GuestAgent.Audit` or `VM.Monitor` for the agent read.
- F18. HTTP/2 is not offered (`curl --http2 -I` falls back or fails), so the
client disables it.
- F19. The template's disk key (`scsi0` or `virtio0`), its `boot` line, that
ide2 is free and no `cloudinit` drive exists.
- F20. `resize` on a stopped VM: request shape, the message when the target
is smaller or equal, and that it is synchronous.
- F21. Agent response shape: `result[].name`, `result[]["ip-addresses"][]`
with `ip-address`, `ip-address-type`.
## Proxmox endpoint map for the seam
| `api` method | HTTP |
|---|---|
| `Version(ctx)` | `GET /version` |
| `NodeStatus(ctx, node)` | `GET /nodes/{node}/status` |
| `StorageStatus(ctx, node, storage)` | `GET /nodes/{node}/storage/{storage}/status` |
| `ClusterResources(ctx)` | `GET /cluster/resources?type=vm` |
| `VMConfig(ctx, node, vmid)` | `GET /nodes/{node}/qemu/{vmid}/config` |
| `SetVMConfig(ctx, node, vmid, params)` | `PUT /nodes/{node}/qemu/{vmid}/config` |
| `Clone(ctx, node, vmid, cloneParams)` | `POST /nodes/{node}/qemu/{vmid}/clone` |
| `Resize(ctx, node, vmid, disk, size)` | `PUT /nodes/{node}/qemu/{vmid}/resize` |
| `Start(ctx, node, vmid)` | `POST /nodes/{node}/qemu/{vmid}/status/start` |
| `Stop(ctx, node, vmid)` | `POST /nodes/{node}/qemu/{vmid}/status/stop` |
| `VMStatus(ctx, node, vmid)` | `GET /nodes/{node}/qemu/{vmid}/status/current` |
| `Destroy(ctx, node, vmid)` | `DELETE /nodes/{node}/qemu/{vmid}?purge=1&destroy-unreferenced-disks=1` |
| `TaskStatus(ctx, node, upid)` | `GET /nodes/{node}/tasks/{upid}/status` |
| `AgentNetworkInterfaces(ctx, node, vmid)` | `GET /nodes/{node}/qemu/{vmid}/agent/network-get-interfaces` |
| `UploadISO(ctx, node, storage, filename, r, size)` | `POST /nodes/{node}/storage/{storage}/upload` (multipart) |
| `DeleteVolume(ctx, node, storage, volid)` | `DELETE /nodes/{node}/storage/{storage}/content/{volid}` |
## Package layout (`internal/provider/proxmox/`)
`proxmox.go` (`Provider`, `New`, `newProvider`, `withDefaults`, `logger`,
`node`, `wrapErr`, `lockName`, `providerOption`, `budgets`), `api.go` (the
`api` interface and DTOs `vmResource`, `vmConfig`, `vmStatus`, `taskStatus`,
`agentInterface`, `storageStatus`, `nodeStatus`, `versionInfo`,
`cloneParams`), `client.go` (`httpAPI`, `newHTTPAPI`, `do`, `readToken`,
`decodeConfig`), `errors.go` (`apiError`, `classify`, `permanentf`,
`transientf`, `isNotExist`, `isLocked`, `isAlreadyExists`,
`isAgentNotRunning`, `isNoSpace`), `id.go` (`formatProviderID`,
`parseProviderID`, `vmidFor`), `mark.go` (`instanceMark`,
`formatDescription`, `parseDescription`), `state.go` (`mapState`,
`firstIPv4`), `observe.go` (`observed`, `observe`), `task.go` (`parseUPID`,
`waitTask`, `waitUnlocked`, `waitRunning`), `seed.go` (`seedISOName`,
`seedVolID`, `uploadSeed`), `create.go`, `get.go`, `delete.go`, `list.go`,
`check.go`, plus `fake_test.go`, `provider_test.go`,
`proxmox_integration_test.go`, `testdata/*.json`. Shared:
`internal/provider/config_proxmox.go`, `internal/provider/nocloud/`,
`internal/provider/keyedlock/`.
Reused as-is: `provider.NameFromUID`, `provider.Wrap`, the `Err*` sentinels,
`provider.Label*` constants, `provider.Reason*` constants,
`tracing.RestConfigWrapper`, `tracing.WithTracerProvider`, kvm2's test
idioms (`fixture`, `assertProviderError`, `logCapture`).
## Steps (one commit per phase, on `feat/proxmox-provider` off `main`)
- **Phase 0 (orchestrator with the user): plan file, branch, host recon,
fixtures.** Read first: contract §5, §4.9, §7;
`docs/plans-executions/2026-09-16-1333-kvm2-provider.md` Step 0 entry.
Do, in this order: `git switch main && git pull --ff-only`; copy this plan
to `docs/plans/2026-09-18-1502-proxmox-provider.md` with the real timestamp
(`date "+%Y-%m-%d-%H%M"`), commit it alone on `main`
(`docs: plan the proxmox provider`); create `feat/proxmox-provider`. Ask
the user for the hand-over facts listed above. Then, with the user's
go-ahead (each `curl` is a permission prompt; read-only ones first, one at
a time), capture F1 to F21 with
`curl -s --cacert "$CA" -D /tmp/h.txt -o /tmp/b.json -w '%{http_code}\n' -H "Authorization: PVEAPIToken=$(cat "$TOKEN_FILE")" https://192.168.2.8:8006/api2/json/<path>`,
storing each response as `internal/provider/proxmox/testdata/<name>.json`
in the shape `{"status": <code>, "reason": "<status line after the
code>", "body": "<raw body>"}`: `version.json`, `node_status.json`,
`storage_status.json`, `cluster_resources.json`,
`vm_config_template.json`, `vm_config_clone.json`,
`status_current_running.json`, `status_current_stopped.json`,
`agent_network_get_interfaces.json`, `task_status_running.json`,
`task_status_stopped_ok.json`, `task_status_stopped_error.json`,
`upload_response.json`, `content_delete_response.json`,
`err_vm_missing.json`, `err_locked.json`, `err_newid_in_use.json`,
`err_agent_not_running.json`, `err_agent_vm_stopped.json`,
`err_unauthorized.json` (a wrong secret), `err_forbidden.json` (a path
the token lacks), `err_bad_param.json` (`newid=abc`), `err_no_space.json`
(only if F6 is cheap), `err_volume_missing.json` (F13). The write
experiments: one clone of the template into the top of the VMID range
(`newid=199999`, a multi-line `description`, a `name` of the `proxy-`
form), a config read of the clone (F10, F11, F19), a `PUT config` while
it is locked if timing allows (F7), an upload of a 1 KiB
`recon-seed.iso` (F12) and its delete (F13), a resize attempt (F20), a
start, an agent read (F5, F21), a stop, then destroy with
`purge=1&destroy-unreferenced-disks=1`; a final resources read shows
nothing remains. Redact nothing but the token. Write the answers to F1 to
F21 and the privilege outcome into the execution log's Phase 0 entry, one
line per fact; Phase 1 cites them as Evidence. Create the execution log
`docs/plans-executions/2026-09-18-1502-proxmox-provider.md` with the checklist (Phase
0 to Phase 10) and this entry.
Verify: `ls internal/provider/proxmox/testdata | wc -l` is at least 22;
`grep -rl 'PVEAPIToken' internal/provider/proxmox/testdata` is empty;
every fixture parses as JSON; the host shows no VM in the range and no
`recon-seed.iso`.
Done when: the plan commit is on `main`, the branch exists, the fixtures
and the execution log with F1 to F21 are committed. Commit: `proxmox:
captured API fixtures and host facts from 192.168.2.8`.
- **Phase 1 (orchestrator): the spec.** Read first: contract §2, §4, §5,
§6, §11, §12; `docs/provider-spec-template.md` end to end;
`docs/kvm2-provider-spec.md` for the level of detail; this plan's
decisions; the Phase 0 log entry. Create `docs/proxmox-provider-spec.md`
answering all 24 entries: 1 `vms/<vmid>/<name>`, strict parsing, the
node-name and storage-id regexps; 2 `allocated-id`, `vmidFor(uid, rng,
attempt)`, the two-VMs-with-our-name rule, the adoption guard; 3 the
placement table (Region ignored); 4 the template catalogue and the "add a
key, never edit" warning; 5 the seed ISO via `nocloud` and the ide2
prerequisite; 6 `machineTypes` with `diskGiB`; 7 `provisioning-window`,
`discovered` via the guest agent, the IPv4 filter, `Port` unused; 8
`written-metadata` in `description` plus the tag, `written-at-create`, F9
to F11 as Evidence; 9 the state table, `create-blocks`; 10 the ordered
Create steps with what a retry finds at each, `task-handle` plus
`backend-lock`, the budgets; 11 the Delete order, the ISO blind spot,
node-local ISO storage after a migration; 12 `fail-whole`, the candidate
rule, one config read per candidate; 13 the classify table with the F1 to
F6 messages, `not-found: error-code` plus list-and-miss on resources
(allowed because offline nodes' guests appear as `unknown`, not absent);
14 `mounted-files` (Secret `proxmox-token`, key `token`, path
`/etc/proxy-operator/proxmox/token`), `ca-file` or
`insecureSkipTLSVerify`, the privilege list from F17, rotation free; 15
the config table (Phase 2's struct); 16 the Checker; 17 `thin-client`,
`transport-wrapper`, the seam's method list; 18 the budgets table; 19 the
keyed lock per name; 20 the log keys; 21 the §11 list with the test names
from Phases 2 to 8 (copy from this plan); 22 the deployment list; 23 the
docs list; 24 the accepted limits (blocking Create up to the clone budget,
ISO blind spot, VMID range shared with humans, node-local ISO storage, no
adoption of foreign VMs, two-installs GC caveat, `unknown` status while a
node is down).
Verify: `grep -c '^\*\*Answer:\*\*$' docs/proxmox-provider-spec.md` is 0;
every axis code used appears in contract §4.20; Evidence lines cite F
numbers; `grep -c '—' docs/proxmox-provider-spec.md` is 0.
Done when: the spec is committed with all 24 entries and the Phase 1 box
ticked. Commit: `docs: proxmox provider spec`.
- **Phase 2 (implementer): config block, registration, package skeleton.**
Read first: spec entries 1, 2, 3, 6, 8, 9, 13, 15; contract §10, §5, §6;
`internal/provider/config.go`, `config_kvm.go`, `config_test.go`,
`config_kvm_test.go`; `internal/provider/kvm2/{kvm2.go,id.go,metadata.go,state.go,errors.go}`
and their tests as the pattern; `cmd/main.go`; `testdata/err_*.json`.
Create `internal/provider/config_proxmox.go`: `ProxmoxConfig{URL,
TokenFile, CAFile string; InsecureSkipTLSVerify bool; Nodes []string;
ISOStorage, Storage, Bridge string; VMIDRange *ProxmoxVMIDRange; Templates
map[string]ProxmoxTemplate; MachineTypes map[string]ProxmoxMachineType;
RequestTimeoutSeconds int; VendorData string; InstallGuestAgent *bool}`,
`ProxmoxVMIDRange{Min, Max int}`, `ProxmoxTemplate{VMID int}`,
`ProxmoxMachineType{Cores int32; MemoryMiB int64; DiskGiB int64}`,
`validateProxmox(prefix string, c *ProxmoxConfig) error` (URL http(s)
with a host; `tokenFile` non-empty; at least one node, unique, matching
`proxmoxNodeName = ^[a-z0-9][a-z0-9-]*$`; `isoStorage` and `storage`
matching `proxmoxStorageID = ^[A-Za-z][A-Za-z0-9_.-]*$`; `vmidRange` min
at least 100, max at most 999999999, min below max; at least one template
with `vmid` at least 100; at least one machine type with cores and
memoryMiB above zero and diskGiB not negative; `requestTimeoutSeconds`
not negative). Edit `config.go`: `Proxmox *ProxmoxConfig` with tag
`proxmox,omitempty`, `setBlocks`, the allow-list, the dispatch. Edit
`cmd/main.go`: `"proxmox": proxmox.New`. Create in
`internal/provider/proxmox/`: `proxmox.go` (`Provider{name, cfg, api,
now, budgets, nameLocks, tracing}`; until Phase 3 a local `keyedLocks`
copy in `locks.go` that Phase 3 deletes; `New` never dials: builds the
client, `os.Stat`s the token file and warns, warns on
`insecureSkipTLSVerify`; `withDefaults`: `isoStorage` `local`, `vmidRange`
100000 to 199999, `requestTimeoutSeconds` 30, `installGuestAgent` true;
`node(zone)` mirrors kvm2's `host`; the four methods stubbed with
`transientf("not implemented")`), `api.go` (the interface and DTOs;
`vmConfig` is `map[string]string` with scalars rendered as strings by
`decodeConfig`), `id.go`, `mark.go` (keys from `provider.LabelManaged`,
`provider.LabelManagedYes`, `provider.LabelUID` and
`proxy-operator-created-at`; `parseDescription` tolerates extra lines
and CRLF), `state.go`, `errors.go` (`apiError{Status int; Message
string; Errors map[string]string}` whose `Error()` never includes headers
or URLs; `classify` per spec entry 13; matcher helpers keyed on the exact
substrings captured in F2 to F6). Tests: `config_proxmox_test.go`
`TestValidateProxmox_table` (missing url, non-http scheme, missing
tokenFile, no nodes, duplicate node, node with a slash, bad isoStorage
id, vmidRange min below 100, min above max, no templates, template vmid
below 100, no machineTypes, cores zero, negative diskGiB, negative
timeout, valid full, valid minimal) with a `validProxmoxYAML` const;
`config_test.go` rows `proxmox type with kvm block`, `kvm type with
proxmox block`, `proxmox minimal entry loads`. `id_test.go`:
`TestFormatProviderID_roundTrip`, `TestParseProviderID_malformed`,
`TestVMIDFor_deterministicAndInRange`, `TestVMIDFor_probesForwardAndWraps`.
`mark_test.go`: `TestFormatDescription_keys`, `TestParseDescription_table`.
`state_test.go`: `TestMapState_table`, `TestFirstIPv4_filters`.
`errors_test.go`: `TestClassify_table` (every class, every matched
message loaded from `testdata/err_*.json` through `loadFixture(t, name)
*apiError`, `op == "create"` turning NotFound into Transient,
`context.Canceled`, a `*net.OpError`, `errors.New("x")`,
already-classified errors keep their class), `TestAPIError_matchers`,
`TestPermanentfAndTransientf`. `proxmox_test.go`:
`TestNew_doesNotDialAndWarnsOnMissingToken`,
`TestNew_insecureSkipTLSVerifyWarns`, `TestNew_nilBlockDoesNotPanic`,
`TestWithDefaults_table`, `TestNode_table`,
`TestWrapErr_carriesProviderNameOpAndID`.
Verify: `grep -c '"proxmox"' internal/provider/config.go` is 3;
`go build ./... && go vet ./... && go test -race -count=1 ./internal/provider/... ./cmd/... && go test -short ./...`;
the fixture loader fails loudly on a missing file.
Done when: `go test -race ./internal/provider/proxmox/` passes with the
stubs in place and the constructor is registered. Commit: `proxmox:
config block, IDs, marks, state mapping, classify, registration`.
- **Phase 3 (implementer): extract `nocloud` and `keyedlock` from kvm2.**
Read first: `internal/provider/kvm2/{seed.go,seed_test.go,kvm2.go,kvm2_test.go,create.go}`;
contract §1.4, §2.1; spec entries 5 and 19;
`grep -n 'seed.go\|keyedLocks\|buildSeedISO' docs/*.md`. Create
`internal/provider/nocloud/seed.go` exporting `BuildSeedISO(name,
userData, vendorData string, w io.Writer) error` (plain errors, an
`ErrEmptyName` sentinel instead of kvm2's `permanentf`),
`BuildVendorData(configured string, installAgent bool) (string, error)`,
`GuestAgentScript`, `SeedLabel`; move the seed tests to
`nocloud/seed_test.go` under their existing names. Create
`internal/provider/keyedlock/keyedlock.go` exporting `Locks`, `New()
*Locks`, `(*Locks).Acquire(ctx, key) (release func(), err error)`; move
the `TestKeyedLocks_*` tests. Edit kvm2: delete `seed.go`, `seed_test.go`,
the `keyedLocks` type and its tests; `create.go` calls
`nocloud.BuildVendorData` and `nocloud.BuildSeedISO` and wraps their
errors with `permanentf`; `kvm2.go` uses `*keyedlock.Locks`. Edit
`internal/provider/proxmox/proxmox.go` to use `keyedlock.Locks` and delete
`locks.go`. Fix the doc mentions so they name the new packages.
Verify: `go test -race -count=1 ./internal/provider/...`; kvm2's test
count drops by exactly the moved tests and the two new packages carry
them; `grep -rn 'iso9660' internal/provider/kvm2/` is empty;
`go vet -tags=integration ./...` still compiles.
Done when: kvm2 and proxmox import both packages, all tests pass, the docs
name the new packages. Commit: `provider: extract nocloud seed builder
and keyedlock from kvm2`.
- **Phase 4 (implementer): seam, HTTP client, decorator, fake.** Read first:
spec entries 14, 17, 18, 20; contract §4.15, §7, §9;
`internal/tracing/transport.go`; `internal/provider/kvm2/{observe.go,observe_test.go,fake_test.go}`;
`internal/provider/proxmox/{api.go,errors.go}`; `testdata/*.json`. Create
`client.go`: `httpAPI{base *url.URL; http *http.Client; tokenFile
string}`; `newHTTPAPI(cfg provider.ProxmoxConfig, opts ...tracing.Option)
(*httpAPI, error)` builds a transport with `DialContext` timeout 10s,
`TLSHandshakeTimeout` 10s, `ResponseHeaderTimeout` from the request
timeout, `IdleConnTimeout` 90s, `MaxIdleConnsPerHost` 4,
`ForceAttemptHTTP2: false`, an empty `TLSNextProto` map (F18), `RootCAs`
from `caFile` or `InsecureSkipVerify` from the flag, wrapped with
`tracing.RestConfigWrapper(opts...)(transport)`; `http.Client.Timeout` is
the request timeout; `do(ctx, method, path string, query, form
url.Values, out any) error` reads the token file on every call (missing
file `transientf`, a file without `!` and `=` `permanentf`), sets the
header, decodes `{"data": ...}` into `out`, turns any non-2xx into
`*apiError` per F1 with the error body read capped at 64 KiB;
`UploadISO` streams a multipart body through `io.Pipe` with a goroutine
writing `content=iso`, `filename` and the file part. Create `observe.go`:
`observed{inner api; provider string}` writing one V(1) `proxmox api
call` line per seam call with `call`, `node`, `vmid`, `storage`, `upid`,
`duration`, `httpStatus` and `error` on failure, V(2) adding request
params and a response summary; `observe(inner api, providerName string)
api`; `New` wraps `httpAPI` in it. Create `fake_test.go`: `fakeAPI{mu; vms
map[int]*fakeVM; tasks map[string]*fakeTask; isos
map[string]map[string][]byte; hooks; block; calls; log; taskPolls int;
cloneFailsWith, startFailsWith string; agentNotRunning, cloneLeavesLock
bool}`, `fakeVM{node, name, status, qmpstatus, lock string; template
bool; config map[string]string; ifaces []agentInterface}`,
`fakeTask{node string; polls int; exit string; onDone func()}`; `Clone`
creates the VM with `lock=clone` and a task that clears the lock or, per
`cloneFailsWith`, removes the VM (F15); `Start` and `Stop` flip status
through a task; `Destroy` removes the VM through a task; `UploadISO`
records bytes; `DeleteVolume` errors with the F13 message when absent;
`enter(ctx, method)` for blocks and hooks like kvm2; helpers `addVM`,
`addTemplate`, `failWith`, `blockOn`, `callCount`, `callLog`,
`fakeErr(status int, msg string) *apiError`. Tests: `client_test.go` over
`httptest.NewTLSServer`: `TestClient_sendsTokenHeaderFromFileOnEveryRequest`
(rotate the file between two calls), `TestClient_missingTokenFileIsTransient`,
`TestClient_unparsableTokenFileIsPermanent`,
`TestClient_apiErrorFromStatusReasonAndBody` (serve each
`testdata/err_*.json` verbatim), `TestClient_decodesDataEnvelope`,
`TestClient_http1Only`, `TestClient_requestTimeoutIsTransient`,
`TestClient_cancelledContextReturnsPromptly`,
`TestClient_childSpanUnderParent`, `TestClient_noSpanWithoutParent`,
`TestClient_tokenNeverInErrorsOrLogs` (sentinel secret echoed by the
server in a 401 body), `TestClient_caFileTrustsPrivateCA`,
`TestClient_insecureSkipsVerification`, `TestClient_uploadStreamsMultipart`
(the server parses the multipart; the client never `io.ReadAll`s the
reader), `TestClient_destroyQueryFlags`. `observe_test.go`:
`TestObserve_everySeamMethodLogsOneLine` (reflection over `api`),
`TestObserve_detailOnlyAtV2`, `TestObserve_uploadBodyNeverLogged`,
`TestObserve_errorFieldsOnFailure`, `TestObserve_v0Silent`.
Verify: `go test -race -count=2 ./internal/provider/proxmox/`;
`grep -n 'ReadAll' internal/provider/proxmox/client.go` shows only the
bounded error-body read; `grep -n 'context.Background()' internal/provider/proxmox/*.go`
is empty outside tests.
Done when: the client passes its tests against the fixtures and the fake
compiles against the full `api` interface. Commit: `proxmox: thin HTTP
client, seam, log decorator and in-memory fake`.
- **Phase 5 (implementer): Get, ListByTag, task helpers.** Read first: spec
entries 7, 8, 9, 12, 13; contract §2.2, §2.4, §4.7, §4.12;
`internal/provider/kvm2/{get.go,get_test.go,list.go,list_test.go}`; the
resources, config, status, agent and task fixtures. Create `task.go`:
`parseUPID(upid) (node string, err error)` (Permanent when malformed),
`waitTask(ctx, log, api, upid, budget) error` polling `TaskStatus` every
`budgets.taskPoll` (2s; tests shrink it) until `status == "stopped"`,
`exitstatus != "OK"` giving `transientf` with the message, the budget
giving `transientf`, ctx giving Transient; `waitUnlocked(ctx, api, node,
vmid, budget)` polling `VMConfig` until no `lock`. Create `get.go`:
`resolve(ctx, api, vmid) (vmResource, bool, error)` (one resources call,
`Type == "qemu"` and the vmid), the name check, `VMConfig` for the mark,
`lock` Transient, `unknown` Transient, `running` reads the agent
(`isAgentNotRunning` gives no IP, other errors Transient), `mapState`.
Create `list.go` per spec entry 12; `Instance` carries ID, UID, CreatedAt
only. Replace the two stubs. Tests: `task_test.go` `TestParseUPID_table`,
`TestWaitTask_table` (ok, error exitstatus, budget exceeded, ctx
cancelled, poll count), `TestWaitUnlocked_table`. `get_test.go`:
`TestGet_missingVMIsNotFound`, `TestGet_nameMismatchIsNotFound`,
`TestGet_unmarkedIsNotFound`, `TestGet_runningWithoutIPIsProvisioning`,
`TestGet_runningWithIPIsRunning`, `TestGet_agentNotRunningIsProvisioning`,
`TestGet_agentOtherErrorIsTransient`, `TestGet_ipFilterSkipsLoopbackAndLinkLocal`,
`TestGet_stoppedIsStopped`, `TestGet_pausedIsStopped`,
`TestGet_unknownStatusIsTransient`, `TestGet_lockedIsTransientNeverNotFound`,
`TestGet_unknownStateIsStopped`, `TestGet_readsUIDAndCreatedAtFromBackend`,
`TestGet_unmanagedDescriptionHasEmptyUID`, `TestGet_malformedIDIsPermanent`,
`TestGet_resourcesFailureIsTransient`, `TestGet_twoVMsWithOurNameIsPermanent`.
`list_test.go`: `TestListByTag_excludesUnmarkedVMs`,
`TestListByTag_emptyUIDStillListed`, `TestListByTag_candidatesByTagOrNamePrefix`,
`TestListByTag_skipsLXCAndTemplates`, `TestListByTag_resourcesFailureFailsWhole`,
`TestListByTag_unreadableVMSkipped`, `TestListByTag_unreadableVMWithEndedContextFails`,
`TestListByTag_unknownStatusStillAttempted`, `TestListByTag_doesNotReadStatusOrAgent`,
`TestListByTag_idsAcceptedByParse`.
Verify: `go test -race -count=2 ./internal/provider/proxmox/`;
`grep -n 'not implemented' internal/provider/proxmox/*.go` lists only
Create and Delete.
Done when: Get and ListByTag are real and every listed test passes.
Commit: `proxmox: Get, ListByTag and task polling`.
- **Phase 6 (implementer): Create.** Read first: spec entries 2, 3, 4, 5, 6,
10, 19; contract §2.1, §4.2, §4.10, §4.11, §5;
`internal/provider/kvm2/{create.go,create_test.go}`;
`internal/provider/nocloud`; F4, F7, F10, F12, F15, F16, F19, F20 in the
Phase 0 log. Create `seed.go`: `seedISOName(name)` is `<name>-seed.iso`,
`seedVolID(storage, name)` is `<storage>:iso/<name>-seed.iso`,
`uploadSeed(ctx, log, api, node, storage, req, cfg)` builds vendor-data
and the ISO into a `bytes.Buffer` (tens of KiB), deletes an existing
volume tolerating absence, uploads with the buffer's length, waits the
task with `budgets.upload`, logs `cloudInitBytes` only. Create `create.go`
implementing the ordered steps: placement validation (`node`,
`MachineTypes`, `Templates`, all Permanent naming field and value), the
per-name lock, `ClusterResources` once for both the name lookup (adoption
guard, then `waitUnlocked` if locked, then continue from the seed step if
stopped, return if running) and the template's node (absent template is
Permanent); allocation loop `for attempt := 0; attempt < maxProbes (32)`
calling `Clone` with `vmidFor(req.UID, rng, attempt)`, `isAlreadyExists`
moving to the next attempt, exhaustion Transient; `waitTask` with
`budgets.clone`; `uploadSeed`; `SetVMConfig` with `cores`, `memory`,
`agent=1`, `tags=proxy-operator`, `ide2=<volid>,media=cdrom`, and `net0`
rewritten from the cloned config with `bridge=<cfg.Bridge>` only when
configured (MAC kept); `bootDisk(cfg)` and `Resize` when `diskGiB` is
above the current `size=`; `Start` and `waitTask` with `budgets.start`;
`waitRunning` on `VMStatus` with `budgets.running`; return the ID.
`isNotExist` inside Create is Transient (classify with `op == "create"`).
Tests `create_test.go`: `TestCreate_repeatReturnsSameIDAndCreatesNothing`,
`TestCreate_resumesAfterEachPartialStep` (after clone task, after seed
upload, after config, after resize, after start, after running),
`TestCreate_returnsOnlyWhenRunning`, `TestCreate_runningWaitCapIsTransient`,
`TestCreate_rejectsUnknownPlacement`, `TestCreate_zoneOptionalWithOneNode`,
`TestCreate_regionIgnored`, `TestCreate_adoptionWithForeignUIDIsPermanent`,
`TestCreate_adoptsOnlyMarkedVMs`, `TestCreate_duplicateNameIsPermanent`,
`TestCreate_vmidCollisionProbesForward`, `TestCreate_vmidRangeExhaustedIsTransient`,
`TestCreate_waitsForCloneLockOnRetry`, `TestCreate_cloneTaskFailureIsTransient`,
`TestCreate_cloneTaskTimeoutIsTransient`, `TestCreate_markWrittenInCloneCall`,
`TestCreate_tagSetInConfigStep`, `TestCreate_seedAlwaysRebuilt`,
`TestCreate_seedUploadedBeforeConfig`, `TestCreate_configSetsCoresMemoryAgentAndSeed`,
`TestCreate_bridgeOverrideKeepsMAC`, `TestCreate_noBridgeOverrideLeavesNet0`,
`TestCreate_resizeOnlyWhenLarger`, `TestBootDisk_table`,
`TestCreate_startTaskFailureIsTransient`, `TestCreate_templateMissingIsPermanent`,
`TestCreate_notExistInsideCreateIsTransientNotNotFound`,
`TestCreate_cloudInitSentinelNeverLoggedOrInErrors` (V(2) capture, every
error path), `TestCreate_concurrentSameNameSerialised`,
`TestCreate_cancelledMidWaitReturnsPromptly`.
Verify: `grep -n 'CloudInit' internal/provider/proxmox/create.go internal/provider/proxmox/seed.go`
shows only `len(req.CloudInit)` and the pass-through into
`nocloud.BuildSeedISO`; `go test -race -count=2 ./internal/provider/proxmox/`;
`grep -n 'not implemented'` lists only Delete.
Done when: Create is real and every listed test passes. Commit:
`proxmox: Create by cloning a template, seed upload, config, start`.
- **Phase 7 (implementer): Delete, Checker, CRD comment.** Read first: spec
entries 11, 16; contract §2.3, §4.3, §4.13, §8;
`internal/provider/kvm2/{delete.go,delete_test.go,check.go,check_test.go}`;
`internal/provider/check.go`; `api/v1alpha1/providerhealth_types.go`; F13,
F17 in the Phase 0 log. Create `delete.go`: parse the ID (Permanent), the
per-name lock, `resolve`; absent is a logged no-op; name mismatch or
unmarked is a logged no-op; `lock` Transient; `unknown` Transient;
`running` gets `Stop` and `waitTask` (`budgets.stop`); `Destroy` and
`waitTask` (`budgets.destroy`), `isNotExist` from `Destroy` being
success; `DeleteVolume` of the seed on the VM's node tolerating the F13
absent message; nil. Create `check.go`: `var _ provider.Checker =
(*Provider)(nil)`; `reasonTemplateNotFound = "TemplateNotFound"`,
`reasonStorageNotFound = "StorageNotFound"`; `Check` fans out one
goroutine per node in config order (`wg.Go`), each `checkNode` running
`Version`, `NodeStatus`, `StorageStatus` (a 500 "does not exist" is
`StorageNotFound`; a storage whose `content` lacks `iso` is
`StorageNotFound` with a message saying so), `ClusterResources` for the
templates; `checkReason(err)`: ctx ended `Timeout`, 401 `Unauthorized`,
403 `Forbidden`, other `*apiError` `APIError`, network `Unreachable`;
Details `pveVersion`, `kernel`, `isoStorage`, `isoStorageAvailGiB`,
`templates`; `Name` is the node, `Address` the base URL. Edit
`api/v1alpha1/providerhealth_types.go` (`kubernetes, gcp, kvm, kvm2 or
proxmox`) and run `make manifests`. Tests `delete_test.go`:
`TestDelete_malformedIDIsPermanent`, `TestDelete_twiceSucceeds`,
`TestDelete_absentIsNoOp`, `TestDelete_foreignNameIsNoOp`,
`TestDelete_unmarkedIsNoOp`, `TestDelete_runningVMStoppedFirst`,
`TestDelete_stoppedVMSkipsStop`, `TestDelete_lockedIsTransient`,
`TestDelete_unknownStatusIsTransient`, `TestDelete_removesSeedISO`,
`TestDelete_toleratesMissingISO`, `TestDelete_destroyTaskFailureIsTransient`,
`TestDelete_usesPurgeAndDestroyUnreferencedDisks`,
`TestDelete_concurrentSameNameSafe`, `TestDelete_noOpLinesAreSilentAtV0`;
in `get_test.go`: `TestGet_afterDeleteReturnsNotFound`,
`TestGet_duringDeleteStillReturnsInstance` (block the fake's `Destroy`).
`check_test.go`: `TestCheck_oneEntryPerNodeInConfigOrder`,
`TestCheck_nodesCheckedConcurrently`, `TestCheck_expiredContextReturnsPromptly`,
`TestCheck_reasonsTable`, `TestCheck_detailsKeys`, `TestCheck_logsVerdictPerNode`,
`TestCheckReason_table`.
Verify: `go test -race -count=2 ./internal/provider/proxmox/ ./internal/providerhealth/`;
`git diff --stat config/` shows only the CRD description line;
`grep -rn 'not implemented' internal/provider/proxmox/` is empty.
Done when: all four contract methods and `Check` are real. Commit:
`proxmox: Delete with seed cleanup, health Checker, CRD type comment`.
- **Phase 8 (implementer): contract sweep, observability sweep, integration
test, Makefile, testing.md.** Read first: contract §11 and §9; spec
entries 17, 18, 20, 21; `internal/provider/kvm2/{provider_test.go,kvm2_integration_test.go}`;
`internal/provider/tracing_test.go`; `Makefile` integration targets;
`docs/testing.md` kvm2 rows and section. Create `provider_test.go`: the
shared `fixture` (fake, log capture, budgets shrunk), `assertProviderError`,
`TestProvider_cancelledContextReturnsPromptly` (each method blocked on
its first seam call), `TestProvider_everyErrorIsWrappedWithConfiguredName`,
`TestLogging_v0SilentV1OutcomeV2Detail` (V(1) lines `proxmox vm ensured
running`, `proxmox vm fetched`, `proxmox vm deleted`, `proxmox vms
listed`, `proxmox node checked`, every line with `provider`, instance
lines with `vmid` and `name`; V(2) has `cloudInitBytes` and never the
text), `TestTracing_httpSpansUnderOperationSpan` (a real `httpAPI`
against an `httptest` server playing the fixtures for a Get, wrapped in
`provider.WithTracing`, with a span recorder: every HTTP span is a child
of `provider.get` and no attribute carries the token). Create
`proxmox_integration_test.go` (`//go:build integration`):
`loadIntegrationEnv` with the `PROXMOX_TEST_*` variables, `providerConfig()`,
the `integrationLog` pattern from kvm2; `TestProxmox_lifecycle` asserting
in order: `New` returns within 100 ms without a request; `Check` Ready
with `pveVersion`, `isoStorageAvailGiB` and `templates`; `Create` returns
`vms/<vmid>/<name>` with the vmid in range; a second `Create` returns the
same ID and the resources count is unchanged; `Get` reaches Running with
an IPv4 within the budget, UID and CreatedAt as written; `ListByTag`
contains it; a TCP connect to the guest's port is logged only; `Delete`
nil; `Get` NotFound; `Delete` again nil; the ISO storage has no
`<name>-seed.iso` and resources has no VM with the name; an HTTP span is
a child of `provider.create` and no attribute carries the cloud-init
text; `PROXMOX_TEST_KEEP` skips the teardown and prints the `qm destroy`
and ISO delete commands. `TestProxmoxCheck_realHost` creates nothing. Edit
`Makefile`: `test-integration-proxmox` (`go test -race -tags=integration
-count=1 -v -timeout 30m -run TestProxmox ./internal/provider/proxmox/`).
Edit `docs/testing.md`: rows for the package in the suites table and a
"The proxmox integration test" section. Update spec entry 21's tick list
with the final test names.
Verify: `go test -race -count=2 ./internal/provider/proxmox/`;
`go vet -tags=integration ./... && go vet -tags=e2e ./...`;
`make -n test-integration-proxmox`; every §11 item has a test name in
spec entry 21 that exists.
Done when: the package covers every §11 item by name, the integration
test compiles, the target exists. Commit: `proxmox: contract and
observability tests, integration test, testing docs`.
- **Phase 9 (implementer): deployment and documentation.** Read first:
contract §12 items 11 to 13; spec entries 22, 23, 24;
`config/manager/manager.yaml`, `config/manager/providers_config.yaml`,
`config/samples/{providers-config.yaml,proxy_kvm2.yaml,kustomization.yaml}`;
`Makefile` `kvm-ssh-secret`; `docs/kvm2-provider.md` headings;
`docs/providers-existing.md`; `docs/architecture.md` lines mentioning
kvm2; `README.md`; `docs/providers.md`; `CLAUDE.md` Gotchas. Edit
`config/manager/manager.yaml`: an optional Secret mount `proxmox-token` at
`/etc/proxy-operator/proxmox` (key `token`). Edit `Makefile`:
`PROXMOX_TOKEN_FILE ?= $(HOME)/.config/proxmox-token`,
`proxmox-token-secret` creating `proxmox-token` from `--from-file=token=`.
Edit both providers-config files: a commented `proxmox-home` entry (`url:
https://192.168.2.8:8006`, `tokenFile`, `caFile` or
`insecureSkipTLSVerify`, `nodes`, `isoStorage`, `storage`, `bridge`,
`vmidRange`, `templates`, `machineTypes`). Create
`config/samples/proxy_proxmox.yaml` (provider `proxmox-home`, zone the
node, machine type `small`, image the template key, the Squid cloud-init
from `proxy_kvm2.yaml`) and list it in `config/samples/kustomization.yaml`.
Create `docs/proxmox-provider.md` with sections: Proxmox prerequisites
(the template commands from this plan, no cloudinit drive on ide2, the
guest agent or outbound access, the bridge, the privilege list from F17,
reserving the VMID range), Cluster-side setup (the Secret, the config
entry), Config reference (spec entry 15's table), Placement (Region
ignored, catalogue-edit warning), What lands on Proxmox (VMID scheme,
description mark, tag, seed ISO, config keys, `qm unlock` for a stuck
clone), Logging, Tracing, Timeouts, Health check, Error classes, Known
gaps (spec entry 24), Testing. Edit `docs/providers-existing.md`: a fifth
column in the three tables and a proxmox section mirroring kvm2's. Edit
`docs/architecture.md`: components row, a proxmox block in §5, the span
note in §10, a §11 row (and the "All four providers" count), Decisions
bullets. Edit `README.md`, `docs/providers.md`, `CLAUDE.md` Gotchas
(`make test-integration-proxmox`).
Verify: `bin/kustomize build config/default >/dev/null && bin/kustomize build config/samples >/dev/null`;
a test `TestLoadConfig_proxmoxSampleEntry` loads the uncommented block
from `config/samples/providers-config.yaml` through `provider.LoadConfig`;
a link sweep over the edited docs finds no missing target;
`grep -c '—' docs/proxmox-provider.md` is 0.
Done when: manifests build, the sample loads, docs are linked and
link-clean. Commit: `proxmox: deployment files, sample, user guide and
doc rows`.
- **Phase 10 (orchestrator with the user): review, real-host validation,
changelog, MR.** Read first: `docs/plans-executions/2026-09-17-1515-kvm2-review-fixes.md`;
`.claude/agents/go-operator-reviewer.md`; the latest report under
`docs/reviews/` for the shape; the spec; this plan's Risks. Do: run
`go-operator-reviewer` on the branch with the spec and the contract as
inputs, then `/code-review high`, `make lint` (if the local toolchain
panics in staticcheck as it did during kvm2, record it and run
`bin/golangci-lint run ./internal/provider/proxmox/...` with staticcheck
disabled), `make test`. Fix accepted findings with regression tests in
one commit; record rejected ones under `docs/reviews/<date>-proxmox-provider-review/`.
Hand the user the host checklist (template ready, no leftover VMs in the
range, ISO storage has space, the token's privileges per F17) and the
`PROXMOX_TEST_*` export lines; with the go-ahead run
`make test-integration-proxmox` and record the clone duration and time to
IP. Optionally the in-cluster pass on kind: build and load the image,
`make proxmox-token-secret`, a scratch ConfigMap with the `proxmox-home`
entry, apply `proxy_proxmox.yaml`, wait for `Ready`, `kubectl get
providerhealth proxmox-home -o yaml`, a `curl -x` through the proxy,
delete and confirm the VM and ISO are gone; scratch overrides are not
committed. After the user confirms: CHANGELOG entry
(`date "+%Y-%m-%d %H:%M %Z"`), push with `-u` using the token helper from
memory (`~/.config/gitea-token`), open the MR with the Gitea API
(`POST /api/v1/repos/kacerr/egress-proxies-operator/pulls` with `head:
feat/proxmox-provider`, `base: main`, the body ending with the
attribution line; `pr.json` in the scratchpad, never committed). Never
merge.
Verify: the integration run's ordered assertions pass on 192.168.2.8;
`make test` green; the review report lists every finding as accepted or
rejected.
Done when: the MR URL is printed and the execution log's last box is
ticked. Commit: `proxmox: review fixes, real-host validation record,
changelog`.
## Verification
- Per phase: the `Verify:` clause, plus `go build ./... && go test -short ./...`
before every commit and `git diff --stat` reviewed for files outside the
phase's list.
- Contract mapping: after Phase 8 every rule of contract §2 and §8 has a
named test in the package, and spec entry 21 maps the §11 list 1:1 to
existing test names.
- Fixture discipline: every message `classify` or a matcher helper compares
against is loaded from `testdata/` in `TestClassify_table` or
`TestAPIError_matchers`; a string literal in `errors.go` without a fixture
is a review finding.
- Secrets: `TestClient_tokenNeverInErrorsOrLogs` and
`TestCreate_cloudInitSentinelNeverLoggedOrInErrors` run at V(2), and the
integration test asserts no span attribute carries the cloud-init text.
- End to end: the integration run (Phase 8's assertions) on 192.168.2.8 and
the optional kind pass in Phase 10.
## Risks
- **Error text location and exact wording (F1 to F6)** decide the whole
NotFound path; if messages vary by storage type or PVE minor version, the
matchers get one fixture per variant and the guide names the tested
version.
- **Clone cleanup (F15).** A failed or interrupted clone may leave a VM with
`lock=clone` that never clears; Create then waits out `budgets.clone` and
returns Transient every time. The guide documents `qm unlock`; the
reconciler keeps retrying rather than latching.
- **`SDN.Use` (F17).** PVE 8 refuses `net0` writes to non-root users without
it; a missing privilege is a 403 Permanent that latches the Proxy until
its spec changes. The guide's privilege list is the mitigation.
- **The ide2 seed and a template with a cloudinit drive** both claim
cloud-init; the Proxy's user-data would be ignored. Documented
prerequisite; the Checker cannot see it.
- **VMID range shared with humans.** A human VM inside the range costs a
probe; a human VM carrying our name is Permanent. The guide says to
reserve the range.
- **Node-local ISO storage after a migration** leaves the seed on the
original node; Delete tolerates its absence and the ISO is a documented
blind spot. Shared ISO storage avoids it.
- **Blocking Create** holds a reconcile worker for up to the clone budget;
the guide sets expectations.
- **The 30s request timeout on the seed upload** is fine for tens of KiB; a
multi-megabyte `vendorData` would trip it. Documented.
- **Two operator installs** on one Proxmox cluster delete each other's VMs
(contract §13); documented in entry 24.
- **kvm2 edits in Phase 3** touch a merged provider; the moved tests are the
guard, and `go vet -tags=integration` keeps its integration test compiling.