egress-proxies-operator/docs/proxmox-provider-spec.md

Provider spec: proxmox

Filled from provider-spec-template.md against provider-contract.md. This is the design the proxmox package is built from. Where an entry names an axis in backticks, the answer is one of that axis's profiles in contract §4.

Backend facts come from the Phase 0 recon on the user's host, recorded in the execution log as facts F1 to F23, with the raw responses under internal/provider/proxmox/testdata/. An Evidence line citing F<n> points there. Where this spec and the plan (docs/plans/2026-09-18-1502-proxmox-provider.md) disagree, this spec wins: it folds in the ten corrections Phase 0 found.

Header

• Type: proxmox
• Example configured name: proxmox-home
• Backend and API version this spec was written against: Proxmox VE 9.2.20 (pve-manager/9.2.20/49318c671b82f31e, kernel 7.0.14-17-pve, QEMU 11.0.3) on the single node pv1-1, API at https://192.168.0.8:8006/api2/json.
• Client: thin-client: net/http and encoding/json from the standard library, with the transport wrapped by the already-required go.opentelemetry.io/contrib/instrumentation/net/http/otelhttp. No new dependency. The seed ISO uses github.com/kdomanski/iso9660, already required by kvm2.
• Author and date: Claude (orchestrator session), 2026-09-23.

1. Identity and providerID

Answer: vms/<vmid>/<name>, where <vmid> is the VM's numeric id and <name> is req.Name. The ID is location-free: the node is looked up in GET /cluster/resources?type=vm on every call, so a VM migrated between nodes stays addressable. Parsing is strict: exactly three segments, the literal vms, a decimal VMID without leading zeros between 100 and 999999999, and a non-empty name without /. Anything else is ErrPermanent from Get and Delete. The name is carried so that Get and Delete can tell our VM from another VM that later received the same VMID (entry 2). No config value is pasted into the ID, so no ID-safety regexp is needed; node names and storage ids are still validated at load (entry 15) because they end up in request paths.

Evidence: contract §6 and §4.10 (the ID must not depend on the location when instances can migrate); F8 (node is a field of every resource entry).

2. Primary key (primary-key)

Answer: allocated-id. Proxmox addresses VMs by VMID; name is only a label and is not unique. The VMID is derived from the Proxy UID so a retry lands on the same number: vmidFor(uid, attempt) is min + (fnv1a64(uid) + attempt) mod (max - min + 1) over vmidRange (entry 15). Create, under the per-name lock (entry 19):

1. Reads /cluster/resources once. Entries with type: qemu, template: 0 and name == req.Name are the by-name candidates. Two or more is ErrPermanent naming every VMID: an admin must remove the duplicate. Exactly one goes to the ownership check below and is adopted or refused.
2. With none, probes attempt = 0, 1, … up to 32:
   • VMID free in the listing: clone onto it. If the clone fails with unable to create VM <vmid>: config file already exists (F4), someone took it between the listing and the clone; run the ownership check on it.
   • VMID taken: run the ownership check.
   • Exhausted: ErrTransient naming the range.
3. The ownership check on a VMID: if its resource entry carries a lock or has no name (a clone in progress lists neither, F8), wait until it is unlocked (entry 10, clone budget). Then read its config. It is ours when name == req.Name and the description carries the mark with uid equal to req.UID or empty: adopt it and resume Create at the seed step (entry 10). It is foreign when the name differs: move to the next attempt. The same name with no mark, or with a different non-empty uid, is ErrPermanent naming the VMID and both UIDs; the operator never takes over a VM it did not create. A suspended lock (a VM hibernated to disk) is not waited on: it is not a task and never clears by itself. Ownership is judged at once, as Get does (entry 9): a foreign hibernated VM is probed past, and ours is ErrPermanent (resume or destroy it), like kvm2's paused domain. Container and template entries at a VMID are foreign without a config read, and so is a VM whose listed name is not req.Name, whatever its lock: only a clone in progress is nameless, and waiting on another VM's backup lock would hold Create for the whole clone budget on every retry (review finding P2).

The wait in step 3 is what closes the duplicate-VM bug Phase 0 found: after a crash mid-clone, the retry's by-name lookup misses the nameless VM, but attempt 0 lands on the same VMID, waits for the clone to finish, and adopts it instead of probing past it.

Get and Delete check the name and mark at the VMID in the ID. A VM there with another name means ours is gone and the VMID was reused: Get answers ErrNotFound, Delete succeeds without touching it.

Evidence: F4 (taken VMID fails synchronously); F8 (a mid-clone entry has no name); contract §4.10 (allocate-then-verify, deterministic from the UID) and §13 (adoption not verified: closed here).

3. Placement mapping

Answer: three fields are catalogue keys checked before any backend call, one is ignored:

Field	Meaning	Required	Unknown value
Zone	a node from nodes	only when more than one node is configured; with one node an empty zone selects it	ErrPermanent: placement.zone "x" is not in proxmox.nodes
MachineType	a key of machineTypes (cores, memoryMiB, optional diskGiB)	yes	ErrPermanent: placement.machineType "x" is not in proxmox.machineTypes
Image	a key of templates (a template VMID)	yes	ErrPermanent: placement.image "x" is not in proxmox.templates
Region	ignored		still part of the spec hash: changing it replaces the VM

The guide documents the ignored field and that catalogue edits affect only new VMs.

Evidence: contract §2.1 (reject early, MAY ignore with documentation).

4. Image source

Answer: a template VM prepared by the admin, referenced by a catalogue key: templates.<key>.vmid. Create makes a full clone (full=1), so a VM owns copies of its disks and keeps nothing linked to the template: the template may be edited, replaced or deleted without affecting running VMs. Before cloning, Create reads the template's config and refuses it with ErrPermanent when it is not a template (template is not 1) or when any disk slot holds a Proxmox cloud-init drive (a value containing cloudinit), which would compete with the provider's seed (entry 5). Because the clone copies the template's config, what the config or resize step would refuse on the clone is refused on the template too, before any VM exists: no net0 while bridge or vlanTag is set, and no boot disk, or one without a readable size=, while diskGiB is set. A Permanent error after the clone would leave a VM that fixing the template cannot help. The template must boot from its first disk, run cloud-init with the NoCloud datasource, and either ship qemu-guest-agent or be able to install it (entry 7).

Clone parameters: newid, name, description (the mark, entry 8), full=1, storage when configured (empty clones onto the template's own disk storage), and target when the chosen node differs from the template's node. A cross-node clone needs the template on shared storage; this host has one node, so that path is untested (entry 24).

Catalogue edit warning for the guide: changing a key's vmid changes only VMs created afterwards. Add a new key rather than repointing one in use, so the Proxies that name it keep a meaning you can reason about.

Evidence: F19 (template 9001: scsi0, boot: order=scsi0, ide2 free, agent: 1; template 9000 carries a cloud-init drive on ide2); F15 (a clone onto a storage without image support is refused up front); F16 (a full clone of 9001 took about two seconds).

5. Cloud-init delivery

Answer: a NoCloud seed ISO built in-process by the shared internal/provider/nocloud package (extracted from kvm2 in Phase 3): volume label cidata, files user-data (the Proxy's CloudInit, byte for byte, never parsed), meta-data (instance-id: <name> and local-hostname: <name>) and vendor-data (the guest agent script, joined with the configured vendorData as a MIME multipart when both are set; identical to kvm2's rules). The builder stages under os.TempDir() (/tmp, the manager's emptyDir) and writes into a bytes.Buffer; a seed is tens of KiB, so buffering is within contract §1.4.

The ISO is uploaded with POST /nodes/{node}/storage/{isoStorage}/upload as multipart (content=iso, file part filename named <name>-seed.iso) and the returned imgcopy task is waited on. An upload overwrites an existing file of the same name, so no delete precedes it; the seed is rebuilt and re-uploaded on every Create attempt, since a previous attempt may have been cut off. It is attached as ide2: <isoStorage>:iso/<name>-seed.iso,media=cdrom; Proxmox then appends ide2 to the boot order on its own, which is harmless because the boot disk stays first.

Only the Proxmox API is used: the storage upload endpoint accepts iso, vztmpl or import, never snippets, so cicustom would need shell access to the node. Size: the request timeout (entry 18) bounds the upload; a seed with a multi-megabyte vendorData would need a larger requestTimeoutSeconds.

Evidence: F12 (upload fields, UPID, overwrite, .iso suffix required); F19 (ide2 appended to the boot order); F21 (the seed on ide2 installed the agent on both the untagged network and VLAN 4); F23 (the upload is staged in /var/tmp on the node, not the tmpfs /tmp); contract §2.1 and §11 (byte for byte, sentinel test).

6. Machine type mapping

Answer: a catalogue: machineTypes.<key> is {cores, memoryMiB, diskGiB}. cores and memoryMiB are written in the config step (entry 10). diskGiB is optional: when set and larger than the cloned boot disk, the disk is grown with PUT /nodes/{node}/qemu/{vmid}/resize to the absolute size <diskGiB>G, and the returned task is waited on; when unset or not larger, the disk keeps the template's size and no call is made. The provider never shrinks a disk. Cloud-init's growpart expands the root filesystem at first boot.

The boot disk is the first entry of the config's boot: order=… that is a disk key (scsiN, virtioN, sataN, ideN) and not a CD-ROM, else the bootdisk key, else the first present of scsi0, virtio0, sata0, ide0. Its current size comes from the size= option of that disk's value. No boot disk, when diskGiB is set, is ErrPermanent.

Evidence: F20 (resize is a task; the same size is an OK no-op; a shrink fails in the task with shrinking disks is not supported); F19 (boot: order=scsi0, becoming order=scsi0;ide2 once the seed is attached).

7. Network and IP discovery (readiness, ip-source)

Answer: readiness: provisioning-window. ip-source: discovered, from the QEMU guest agent only (GET /nodes/{node}/qemu/{vmid}/agent/network-get-interfaces); the API offers no ARP or DHCP-lease source. The address reported is the first IPv4 of the first non-loopback interface, in the agent's order, that is not loopback, link-local or unspecified.

The NIC is the template's net0, rewritten in the config step only when the config sets bridge or vlanTag: the value's comma-separated options are kept in order with the model and MAC untouched, bridge= and tag= are replaced or appended. With neither set, net0 is left as cloned.

The agent answers:

Agent response	Meaning	Get reports
200 with a routable IPv4	booted	Running with that IP
200 without one	network still coming up	Provisioning
500 QEMU guest agent is not running	booting, or the agent install has not finished or failed	Provisioning
500 VM <vmid> is not running	the VM stopped between the status read and this call	ErrTransient
500 No QEMU guest agent configured	agent: 1 was removed from the config; this VM can never report an address	Stopped, so the reconciler replaces it with a VM that has the agent on
anything else	cannot tell	ErrTransient

The guest needs the agent. The vendor-data script installs it at first boot unless installGuestAgent: false, and that install needs a package mirror: the proxy network must have a route to the internet, which an egress proxy needs anyway. Baking qemu-guest-agent into the template and setting installGuestAgent: false removes the mirror from IP discovery. Port is not used by the provider. Reachability is a prerequisite in the guide: the operator's pod network and the clients must route to the proxy network (on the user's host, VLAN 4, 192.168.4.0/24).

Evidence: F5 (all three agent messages); F21 (agent response shape; an address about 50 seconds after start on the untagged network and on VLAN 4); the Phase 0 log's VLAN 4 section (no internet route meant no agent); contract §1.3 and §2.2 (Running only with an IP).

8. GC marks and CreatedAt (gc-mark, created-at)

Answer: gc-mark: written-metadata. created-at: written-at-create. The mark is three lines in the VM's description, passed in the clone call itself, so there is no mark window:

proxy-operator-managed=true
proxy-operator-uid=<req.UID>
proxy-operator-created-at=<RFC3339 UTC>

The keys are provider.LabelManaged, provider.LabelUID and proxy-operator-created-at; the managed value is provider.LabelManagedYes. The parser reads key=value lines, tolerates CRLF, blank lines and any other lines an admin adds, and treats the VM as managed only when the managed value is exactly true. A missing uid line gives an empty UID. A created-at that does not parse gives the zero time, which GC treats as old; that is accepted. The clone's own meta: …,ctime=… is not usable: Proxmox copies it from the template.

The config step also sets the tag proxy_operator, for filtering in the Proxmox UI and as a ListByTag candidate hint (entry 12). It is not the mark: tags cannot be passed to the clone call. Hyphens are accepted in tags on 9.2.20, but the underscore form is kept because it is valid under every tag rule Proxmox has had. Uppercase is folded to lowercase by Proxmox.

The seed ISO carries no mark; its name <name>-seed.iso identifies it (entry 11).

Evidence: F10 (the description survives the clone verbatim, newlines included); F11 (the clone's meta is the template's); F9 (tag rules); contract §2.1 (mark in the create call) and §4.5.

9. State mapping (provisioning-state)

Answer: provisioning-state: create-blocks. Create returns only once status/current reports the VM running (entry 10), so a VM the provider cloned but never started is only ever seen by a Create retry, which adopts and starts it; the reconciler has no ID for it to Get.

Get checks in this order; the first match decides:

Check	Source	Result
VMID absent	/cluster/resources	ErrNotFound
lock set, other than suspended	resources entry (a clone in progress also has no name)	ErrTransient
lock: suspended (hibernated to disk)	resources entry	Stopped
status: unknown (node not reachable, or a VM seconds old the listing has not reported yet: the first Get after Create met this on pv1-1)	resources entry	ErrTransient
config does not exist	GET …/config	ErrNotFound (deleted between the two calls)
config lock set	config	ErrTransient
name differs from the ID's name	config	ErrNotFound, logged: the VMID was reused
no mark in the description	config	ErrNotFound, logged: not ours
status: stopped	status/current	Stopped
status: running, qmpstatus: running	status/current	the agent decides (entry 7)
status: running, qmpstatus: prelaunch	status/current	Provisioning
status: running, any other qmpstatus (paused, suspended, io-error, …)	status/current	Stopped
any other status	status/current	Stopped

A suspended lock decides the state but not ownership: the config is still read, and a foreign name or a missing mark is still ErrNotFound, so a hibernated VM that reused our VMID is never reported as ours. Only status/current and the agent are skipped for it.

status/current is read, not only the resources entry, because a paused VM shows as plain running in /cluster/resources. UID and CreatedAt come from the description on every call. A running Get makes four calls: resources, config, status and agent.

Evidence: F8 (resources fields; mid-clone entries lack a name, carry lock: clone and status: unknown); F14 (stopped and paused as seen by status/current and by resources); F2 (the does not exist message); contract §2.2 and §4.11.

10. Create: blocking and task tracking (create-return, async-ops)

Answer: create-return: blocking. async-ops: task-handle and backend-lock. Clone, upload, resize, start, stop, destroy and volume delete each return a UPID; the provider polls GET /nodes/{node}/tasks/{upid}/status every second until status is stopped, then reads exitstatus: OK is success, and so is WARNINGS: <n>, which Proxmox itself counts as a finished task (logged at V(1)); anything else is ErrTransient carrying that text. The node comes from the UPID (UPID:<node>:…). The handle is held within the call only: Create blocks until the VM runs, so no handle outlives it, and a failure surfaces as Create's own error with the task's message. Locks: a VM locked by a task is waited on in Create (the ownership check) and reported as ErrTransient by Get and Delete. A start issued while a clone holds the lock queues behind it rather than failing; the provider never relies on that, because it waits for the clone task first.

Steps, each idempotent, in order, under the per-name lock (entry 19):

1. Validate placement (entry 3). No backend call yet.
2. GET /cluster/resources once, then look the name up (entry 2).
3. When the name finds nothing: find the template's node (a missing template is ErrPermanent), read the template's config and refuse it as entry 4 says, then allocate the VMID (entry 2). The template is checked only here, because a VM that already exists no longer depends on it (entry 4): a template deleted after its clone must not fail that clone's adoption. An adopted VM, found by name or by the probe, gets one status/current read. Running, it returns its ID at once. Stopped, it continues at step 5.
4. Clone and wait (clone budget).
5. Build the seed and upload it to the VM's node, overwriting; wait (upload budget).
6. PUT /nodes/{node}/qemu/{vmid}/config (synchronous) with cores, memory, agent=1, tags=proxy_operator, ide2=<isoStorage>:iso/<name>-seed.iso,media=cdrom and, when configured, the rewritten net0 (entry 7).
7. Resize when diskGiB is larger than the boot disk (entry 6); wait (resize budget).
8. POST …/status/start; wait (start budget).
9. Poll status/current every second until status: running and qmpstatus: running, up to the running budget; expiry is ErrTransient, and so is a stopped status after the start.
10. Return vms/<vmid>/<name>.

What a retry finds:

Previous attempt stopped after	The retry
nothing	allocates, clones
submitting the clone	finds the VMID locked and nameless, waits, adopts, continues at 5
the clone	adopts the stopped VM, continues at 5
any of steps 5 to 7	adopts, repeats 5 to 7 (upload overwrites, config is a replace, resize is skipped once large enough)
the start	adopts the running VM, returns its ID

A Proxmox does not exist inside Create (a VM or volume vanishing between steps) is ErrTransient, never ErrNotFound. The log line for a Create records cloudInitBytes, never the content.

Evidence: F3 (a start queues behind the clone lock); F7 (a config write under the clone lock is refused with VM is locked (clone)); F16 (task status fields); F20 (resize is a task); the Phase 0 log's VLAN 4 rerun (clone to started VM in about five seconds); contract §2.1, §4.2, §4.12 and §5 ("do not return NotFound from Create").

11. Delete and secondary resources (delete-return, secondary-resources)

Answer: delete-return: waits. secondary-resources: explicit for the seed ISO; the VM's disks are removed by the destroy call itself, with purge=1&destroy-unreferenced-disks=1. Under the per-name lock:

1. Parse the ID (ErrPermanent if malformed).
2. /cluster/resources. VMID absent, or held by a container: skip to step 6, trying every configured node because the VM's node is no longer known.
3. lock set, other than suspended, or status: unknown: ErrTransient. A clone or another task still running finishes first; the reconciler calls Delete again after DeletionPoll.
4. Read the config. does not exist: skip to step 6 on every node. A different name, or no mark: the VM is not ours; log it at V(1) and skip to step 6 on every node without touching it. Our VM with a lock other than suspended (taken since the listing): ErrTransient. Our VM with lock: suspended (hibernated) is destroyed like a stopped one: the listing shows it stopped, so step 5 is skipped, and the destroy accepts it and removes its suspend state volume with the disks (destroy_hibernated_response.json, task_status_destroy_hibernated.json, captured in Phase 10). Until then Delete refused it as ErrPermanent on the belief that the destroy would. A hibernated VM that is not ours is left alone like any other.
5. Status (from the listing) other than stopped: POST …/status/stop, wait (stop budget). Stop works on a paused VM too, which the listing shows as running: the task ends OK and the VM is stopped (stop_paused_response.json, task_status_stop_paused.json, status_current_after_stop_paused.json, captured in Phase 10). does not exist means the VM is gone.
6. DELETE /nodes/{node}/storage/{isoStorage}/content/<isoStorage>:iso/<name>-seed.iso, wait (seed-delete budget), on the VM's node when step 5 was reached on a VM of ours, and on every configured node in config order otherwise (then step 7 is skipped). A missing volume is still a successful task, so no message is matched. The seed goes before the destroy: once the VM is gone Get is ErrNotFound, the reconciler releases the finalizer and GC no longer lists it, so a seed left by a failed step after the destroy would never be deleted (review finding CR1). On the VM's node a failure fails the call and leaves the VM stopped for the next Delete. On the every-node path a node whose request or task fails is logged at V(1) and skipped, unless ctx ended or the failure is ErrPermanent (a 403 for Datastore.Allocate, which every node would give), which fail the call, so the missing privilege is visible on both paths (review finding P1).
7. DELETE /nodes/{node}/qemu/{vmid}?purge=1&destroy-unreferenced-disks=1, wait (destroy budget). does not exist is success. The destroy leaves the seed alone (Phase 0 deleted it after the destroy), which is why step 6 exists. That Proxmox deletes an ISO a stopped VM's ide2 still references was not in the Phase 0 captures; the Phase 10 real-host run proved it (imgdel OK on the stopped VM's seed, then the destroy).
8. Return nil.

Everything is gone when Delete returns nil, so Get afterwards is ErrNotFound. A concurrent Get during a Delete keeps seeing the VM (as Stopped once it is stopped) until the destroy task removes its config, because destroying sets no lock and leaves the resource entry in place until the end. A second Delete waits on the lock, then finds nothing.

GC blind spot: a VM destroyed outside the operator leaves its seed ISO, which ListByTag cannot see. The guide says seeds are named proxy-*-seed.iso on isoStorage and any without a matching VM may be removed.

Evidence: F2 (a second destroy gives does not exist); F13 (volume delete is an imgdel task, needs Datastore.Allocate, and succeeds on a missing volume); the Phase 0 capture of resources and config mid-destroy (cluster_resources_during_destroy.json, config_during_destroy.json); contract §2.3.

12. ListByTag scope and partial failure (list-partial)

Answer: fail-whole for the one listing, with single unreadable VMs skipped. Scope: every VM the cluster knows, from one GET /cluster/resources?type=vm, which covers every node, including unreachable ones, whose guests Proxmox lists with status: unknown (entry 13). Candidates are entries with type: qemu, template: 0, and either the tag proxy_operator (tags split on ;, , or space) or a name starting with proxy-. A nameless entry (a clone in progress) is not a candidate: its Create is live and it is younger than MinAge. For each candidate the config is read and the description parsed; managed VMs become an Instance with ID, UID and CreatedAt only, which is all GC reads (contract §2.4); no status or agent call is made. A candidate whose config cannot be read (a node down, a VM destroyed mid-sweep) is skipped with a V(1) line, unless ctx ended, which fails the call. The resources call failing fails the whole call. Filtering by the mark is client-side; the tag narrows the candidates. Cost: one resources call plus one config read per candidate per sweep.

Evidence: F8 (resources fields, tags separated by ;, nameless mid-clone entries); F9 (tags case-folded); contract §2.4 and §4.7.

13. Error classification (not-found, quota)

Answer: not-found: error-code, the 500 message Configuration file 'nodes/<node>/qemu-server/<vmid>.conf' does not exist, plus list-and-miss on /cluster/resources. The listing can miss a VM only when it is gone: it is built from the cluster filesystem, which holds every guest's config cluster-wide, and Proxmox lists the guests of an unreachable node with status: unknown rather than dropping them. That last point is Proxmox's documented design, not something the single-node host could show; Phase 10 or a second node verifies it. quota: absent. The thin local-lvm storage overprovisions, so no out-of-space response could be produced; capacity failures are ErrTransient, and the Checker reports free space in Details instead (entry 16).

The client carries each failure as apiError{Status, Message, Errors}: Message is the body's message with the trailing newline trimmed, or the reason phrase when the body has none (a 401); Errors is the 400 body's per-parameter map. One classify(op string, err error):

Backend response	Class	Why
401 Authentication failed!, 401 No ticket	Permanent	the token or its secret is wrong; fixed in the Secret
403 Permission check failed (<path>, <privilege>)	Permanent	the token lacks a privilege; the message names it
400 Parameter verification failed. with errors	Permanent	the request is wrong; the message includes the field errors
400 as above with storage: No such storage.	Permanent from a call; the Checker's StorageNotFound (entry 16)	the storage does not exist in the cluster
other 4xx	Permanent	a request Proxmox will never accept
500 Configuration file '…' does not exist	NotFound (Transient when op == "create")	the VM is gone
500 VM is locked (<lock>)	Transient	a task holds the VM
500 unable to create VM <vmid>: config file already exists	handled by the allocator (entry 2); Transient if it escapes	the VMID is taken
500 QEMU guest agent is not running, No QEMU guest agent configured, VM <vmid> is not running	handled by Get (entry 7); Transient if they escape	agent state
any other 5xx, including pveproxy's 595 and 596	Transient	unknown means retry
task exitstatus other than OK or WARNINGS: <n>	Transient, with the text	the task failed; the next attempt retries
TLS verification failure (x509.UnknownAuthorityError, x509.HostnameError, x509.CertificateInvalidError)	Permanent	wrong URL or CA file; the admin must act
token file missing (fs.ErrNotExist)	Transient	the Secret has not landed yet
token file present but not <user>@<realm>!<id>=<secret>	Permanent	wrong content in the Secret
context.Canceled, context.DeadlineExceeded, net.OpError, io.EOF, io.ErrUnexpectedEOF	Transient	network or cancellation
an error already wrapped with a class	that class	validation and builders classify at the source
anything else	Transient	contract default

Every matched string is backed by a captured response under testdata/; the table test loads them rather than repeating the text. Matching is on fixed strings anchored to the 500 status: does not exist, config file already exists, and the whole messages QEMU guest agent is not running and No QEMU guest agent configured, plus, on a 400, the storage field error No such storage. (whole; a clone onto a storage without image support has another storage error). VM is locked and the stopped VM's VM <vmid> is not running are not matched by the provider: both are Transient through the default branch. The tests keep a matcher for each, with the whole stopped-VM message against ^VM [0-9]+ is not running$ (its tail is a suffix of the agent-not-running answer), only to hold the fake's wording to the captures (review finding CR5).

Evidence: F1 (error text in the reason phrase and the body; 401 has no body); F2, F4, F5, F7 (the 500 messages); F6 (no out-of-space response could be produced on thin storage); F22 (both 401 texts); fixtures err_forbidden.json, err_bad_param.json, err_tag_invalid.json, err_upload_bad_name.json, and from Phase 10 storage_status_missing.json (a missing storage is a 400, not the 500 the spec had assumed); the Phase 0 TLS finding (192.168.2.8 not in the node certificate); contract §5.

14. Credentials and TLS (credentials)

Answer: mounted-files. The config names tokenFile, one line <user>@<realm>!<tokenid>=<secret>, sent as Authorization: PVEAPIToken=<that line>. It is expected in the optional Secret proxmox-token, key token, mounted at /etc/proxy-operator/proxmox. The file is read on every request, so rotating the Secret takes effect without a restart. New stats it and logs a warning when it is missing, but never fails on it and never dials.

TLS: ca-file. caFile names a PEM bundle, typically the node's /etc/pve/pve-root-ca.pem, carried in the same Secret as key ca.pem. It is loaded on first use and cached, so rotating the CA needs a restart; a missing CA file is ErrTransient until it lands, one that does not parse is ErrPermanent. Without caFile the system roots are used. The URL's host must be a name or address in the node certificate's SANs: on the user's host that is 192.168.0.8 or pv1-1.home.lab, not 192.168.2.8. insecureSkipTLSVerify: true disables verification and is logged as a warning by New.

Minimum privileges, as verified on 9.2.20, for proxy-operator@pve with a token without privilege separation:

• a role on / with VM.Allocate VM.Clone VM.Audit VM.Config.CDROM VM.Config.CPU VM.Config.Memory VM.Config.Disk VM.Config.Network VM.Config.Options VM.PowerMgmt VM.GuestAgent.Audit Datastore.Audit Datastore.AllocateSpace Datastore.AllocateTemplate Sys.Audit SDN.Use;
• a role on /storage/<isoStorage> with Datastore.Allocate Datastore.AllocateTemplate Datastore.AllocateSpace Datastore.Audit, because deleting a volume needs Datastore.Allocate and a grant on a more specific path replaces the inherited one. A dedicated ISO-only storage keeps that privilege away from backups.

VM.Monitor no longer exists in PVE 9 and is not needed. VM.Config.Cloudinit is not needed. SDN.Use on / covers a VLAN tag on vmbr0.

Hygiene: the token never appears in logs, span attributes or error strings. apiError.Error() carries status and message only, never headers or URLs. Tested with a sentinel secret that a test server echoes in a 401 body.

Evidence: F17 (privileges exercised and the one missing); the Phase 0 log's seed storage section; F22 (auth failure texts); contract §7.

15. Config block

Answer: a new block proxmox: with the Go type provider.ProxmoxConfig in internal/provider/config_proxmox.go:

Field	Required	Default	Validation	Ends up in
url	yes		https URL with a host and no path other than /	request base, …/api2/json appended
tokenFile	yes		non-empty	Authorization header
caFile		system roots	non-empty when set	TLS roots
insecureSkipTLSVerify		false		TLS, warning at New
nodes	yes, at least one		unique, ^[a-z0-9]([a-z0-9-]*[a-z0-9])?$	request paths, clone target
isoStorage	yes		Proxmox storage id ^[a-zA-Z][a-zA-Z0-9_.-]*$	upload and delete paths, ide2
storage		the template's disk storage	storage id regexp when set	clone storage
bridge		as cloned	non-empty when set	net0 bridge=
vlanTag		as cloned	1 to 4094 when set	net0 tag=
vmidRange.min, vmidRange.max		100000, 199999	min at least 100, max at most 999999999, min below max	vmidFor
templates.<key>.vmid	at least one template		at least 100	clone source
machineTypes.<key>	at least one		cores and memoryMiB above zero, diskGiB not negative	config step, resize
requestTimeoutSeconds		30	not negative	HTTP client timeout
vendorData				seed vendor-data
installGuestAgent		true		seed vendor-data

isoStorage has no default on purpose: the token must be allowed to delete volumes there (entry 14), so the choice must be explicit. nodes holds node names as Proxmox knows them (pv1-1), not addresses; the one URL reaches the whole cluster.

Wiring: ProviderConfig.Proxmox *ProxmoxConfig with tag proxmox,omitempty; setBlocks reports proxmox; the allow-list in Config.validate and the constructor map in cmd/main.go both gain proxmox in the same commit; validateProxmox(prefix, c) in config_proxmox.go with config_proxmox_test.go.

Evidence: contract §10; the Phase 0 decisions (dedicated seed storage, vlanTag, URL at 192.168.0.8); F13 (why isoStorage needs Datastore.Allocate).

16. Checker

Answer: one target per nodes entry, Name the node, Address the URL, checked concurrently, returned in config order. The check shares the provider's HTTP client; each request is independent, so a check that runs out of time cancels only its own requests. Per node:

1. GET /version (pveVersion).
2. GET /nodes/{node}/status (the node answers; kernel).
3. GET /nodes/{node}/storage/{isoStorage}/status: the storage exists, is active, and lists iso in its content (isoStorageAvailGiB).
4. GET /cluster/resources?type=vm: every configured template exists and is a template (templates as <found>/<configured>).

Reasons, shared first: Reachable; Timeout (the round's ctx ended, whichever request was running); Unreachable (TCP connect or TLS handshake failed for a network reason, or pveproxy answered 595 or 596); Unauthorized (401); Forbidden (403, the message names the privilege); APIError (any other failure). Provider-specific, each fixed differently by an admin:

• CertificateInvalid: TLS verification failed, or caFile is missing or holds no certificate. Fix url or caFile, not the host.
• CredentialsUnavailable: the token file is missing or malformed. Fix the Secret, not Proxmox.
• StorageNotFound: isoStorage is missing, inactive, or has no iso content. Fix the storage on the node. A storage the cluster does not know answers 400 Parameter verification failed. with the field error storage: No such storage. (storage_status_missing.json, captured in Phase 10); that exact field error is matched, and the message carries it. Any 500 from the storage status call counts as missing too, whatever its wording: Proxmox knows the storage but cannot use it on that node (not enabled there, or offline). That answer was not captured, so no text is matched. Any other 4xx, and a 401, 403, 595 or 596, keeps its shared reason.
• TemplateNotFound: a configured template VMID does not exist or is not a template. Fix the template or the catalogue.

Details: pveVersion, kernel, tokenID (the part before =, never the secret), isoStorage, isoStorageAvailGiB, templates, filled as far as the check got; isoStorage is always there. templates counts catalogue entries. The Checker reads tokenFile itself, before the first request, with the client's readToken: the seam stays one method per endpoint, and a missing or malformed file answers CredentialsUnavailable without a request. Only the part before = is kept.

Timeout also covers a request that runs out of its own timeout while the round is still running. A dial or a connection cut mid-response is Unreachable, even when the dial ran out of its own 10 s budget, because the node did not answer. Cost: four requests per node per round. The status.type comment in api/v1alpha1/providerhealth_types.go gains proxmox and the CRD is regenerated.

Evidence: contract §8; fixtures version.json, node_status.json, storage_status.json, cluster_resources.json.

17. Client and child spans (client-kind, child-spans)

Answer: client-kind: thin-client. child-spans: transport-wrapper.

The seam is the interface api, one method per endpoint, ctx first:

Method	HTTP	Returns
Version	GET /version	version info
NodeStatus	GET /nodes/{node}/status	node status
StorageStatus	GET /nodes/{node}/storage/{storage}/status	storage status
ClusterResources	GET /cluster/resources?type=vm	entries
VMConfig	GET /nodes/{node}/qemu/{vmid}/config	config as map[string]string
SetVMConfig	PUT /nodes/{node}/qemu/{vmid}/config	nothing (synchronous)
Clone	POST /nodes/{node}/qemu/{vmid}/clone	UPID
Resize	PUT /nodes/{node}/qemu/{vmid}/resize	UPID
Start	POST /nodes/{node}/qemu/{vmid}/status/start	UPID
Stop	POST /nodes/{node}/qemu/{vmid}/status/stop	UPID
VMStatus	GET /nodes/{node}/qemu/{vmid}/status/current	status and qmpstatus
Destroy	DELETE /nodes/{node}/qemu/{vmid}?purge=1&destroy-unreferenced-disks=1	UPID
TaskStatus	GET /nodes/{node}/tasks/{upid}/status	status and exitstatus
AgentNetworkInterfaces	GET /nodes/{node}/qemu/{vmid}/agent/network-get-interfaces	interfaces
UploadISO	POST /nodes/{node}/storage/{storage}/upload (multipart)	UPID
DeleteVolume	DELETE /nodes/{node}/storage/{storage}/content/{volid}	UPID

httpAPI is the production adapter; fakeAPI in fake_test.go keeps VMs, tasks and ISOs in memory, simulates locks (a clone leaves a nameless locked entry until its task completes), task polls, per-method errors and blocking, and records a call log. Scalars in a config are rendered as strings by the adapter (cores arrives as a JSON number, memory as a string).

The transport is tracing.RestConfigWrapper(opts...)(base), a parent-gated otelhttp transport, so each request is an HTTP <METHOD> span under provider.create, provider.get, provider.delete, provider.list or the monitor's providerhealth.check, and background traffic makes no root spans. The observed decorator over api logs one line per call and starts no span, so nothing is counted twice. Span attributes are otelhttp's standard ones; the token is in a header, which otelhttp does not record. Tests: a span recorder asserts the HTTP spans are children of the operation span and carry no token; a reflection test asserts every api method logs exactly one line.

Evidence: contract §4.15, §4.18, §9; RestConfigWrapper and parentGatedTransport in internal/tracing/transport.go; F18 (HTTP/1.1 only).

18. Timeouts (timeouts)

Answer: per-call.

Bound	Budget
TCP dial	10 s
TLS handshake	10 s
one request, response included (http.Client.Timeout)	requestTimeoutSeconds, default 30 s
clone task, and the wait for a VM's lock during the ownership check	10 m
upload task	2 m
resize task	2 m
start task	2 m
stop task	2 m
destroy task	5 m
seed delete task	1 m
wait for running after the start	60 s
task and status poll interval	1 s

The budgets live in one budgets struct that tests shrink. A task wait that runs out is ErrTransient naming the UPID, and the task keeps running on Proxmox; the next attempt finds its result through the ownership check or the lock. A cancelled ctx cancels only that request: the transport keeps its idle connections (MaxIdleConnsPerHost 4, IdleConnTimeout 90 s), and no other call is disturbed. HTTP/2 is disabled explicitly (ForceAttemptHTTP2: false, an empty TLSNextProto).

Evidence: contract §1.5 and §4.16; F16 (a full clone of the 3 GiB template took about two seconds; tasks finish within a second or two); F21 (about 50 s from start to an address, which the reconciler's polling covers, not the running wait).

19. Concurrency

Answer: shared state is the HTTP client (safe for concurrent use), the lazily loaded CA pool (loaded once under a sync.Once-style guard), and a lock per instance name from the shared internal/provider/keyedlock package (extracted from kvm2 in Phase 3): a one-slot semaphore per key, waitable with ctx, removed from the map once unused. Create and Delete for one name run under that name's lock, so the reconciler's Delete and GC's Delete never race on stop and destroy, and a Delete cannot interleave with a Create's steps. Get and ListByTag take no lock. Two Creates for different Proxies that land on the same VMID are safe: one clone wins, the other gets the config file already exists answer, waits in the ownership check for the winner's clone to finish, sees a different name, and probes on. Three reconciles, a GC sweep and a check round at once are safe.

Evidence: contract §1.2 (concurrency); F4; entry 2.

20. Logging (logging)

Answer: tiered. Logger from logf.FromContext(ctx) on every call, named proxmox, with provider on every line and vmid, name and node on every line about an instance.

• V(1): one line per operation outcome (proxmox vm ensured running, proxmox vm fetched with state, ip and uid, proxmox vm deleted with destroyed (false when no VM of ours was left to destroy), proxmox vms listed with counts, proxmox node checked with the reason), one proxmox api call line per seam call from the decorator (call, node, vmid, storage, upid, duration, plus newid and name on a clone, filename on an upload and volid on a volume delete, and on failure httpStatus and error), a proxmox task failed line with the UPID and exitstatus, and the no-op lines of Delete and ListByTag (VM already gone, foreign or unmarked VM left alone, seed delete skipped on a node, unreadable candidate skipped).
• V(2): request parameters (without description's content at V(1) and never the upload body), config summaries, agent addresses, the resolved Create parameters with cloudInitBytes.
• No V(5) wire tier.
• Never logged at any level: CloudInit, the token, the CA content, the upload body.

Evidence: contract §4.17 and §9.

21. Tests (backend-test)

Answer: backend-test: integration-tag. Unit tests run over fakeAPI behind the seam, the HTTP client over httptest.NewTLSServer serving the captured fixtures, and classify over the fixtures directly. The integration test is proxmox_integration_test.go behind //go:build integration, skipped unless PROXMOX_TEST_URL, PROXMOX_TEST_TOKEN_FILE, PROXMOX_TEST_NODE, PROXMOX_TEST_TEMPLATE_VMID and PROXMOX_TEST_ISO_STORAGE are set; optional PROXMOX_TEST_CA_FILE, PROXMOX_TEST_INSECURE, PROXMOX_TEST_STORAGE, PROXMOX_TEST_BRIDGE, PROXMOX_TEST_VLAN_TAG, PROXMOX_TEST_VMID_MIN, PROXMOX_TEST_VMID_MAX, PROXMOX_TEST_TIMEOUT (default 20m), PROXMOX_TEST_KEEP, PROXMOX_TEST_LOG_LEVEL; run with make test-integration-proxmox. It asserts, in order: New returns without a request; Check reports the node Ready with details; Create returns vms/<vmid>/<name> with the VMID in range; a second Create returns the same ID and the VM count is unchanged; Get reaches Running with an IPv4 within the budget and its UID and CreatedAt equal what Create wrote; ListByTag contains it; Delete returns nil; Get is ErrNotFound; Delete again returns nil; no VM with the name and no <name>-seed.iso remain; an HTTP span is a child of provider.create and no span attribute carries the cloud-init text.

Contract §11 list, with the test for each item. Every name below exists (Phase 8 checked each with a grep for func <name>(). The two integration tests passed on pv1-1 on 2026-09-24 (timings in testing.md).

• repeat Create: TestCreate_repeatReturnsSameIDAndCreatesNothing
• Get after Delete: TestGet_afterDeleteReturnsNotFound
• Delete twice: TestDelete_twiceSucceeds
• malformed ID: TestParseProviderID_malformed, TestGet_malformedIDIsPermanent, TestDelete_malformedIDIsPermanent
• unknown state maps to StateStopped: TestMapState_table, TestGet_unknownStateIsStopped
• StateRunning requires an IP: TestGet_runningWithoutIPIsProvisioning, TestGet_runningWithIPIsRunning
• UID and CreatedAt read back: TestGet_readsUIDAndCreatedAtFromBackend
• ListByTag excludes unmarked instances: TestListByTag_excludesUnmarkedVMs, TestListByTag_emptyUIDStillListed
• Get during an in-progress delete: TestGet_duringDeleteStillReturnsInstance
• Get during an asynchronous operation or lock: TestGet_lockedIsTransientNeverNotFound, TestGet_midCloneNamelessEntryIsTransient
• never started is never StateStopped: TestCreate_returnsOnlyWhenRunning
• cancelled ctx returns promptly: TestProvider_cancelledContextReturnsPromptly (every method and Check, blocked on the first seam call, plus Create and Delete blocked mid-operation), TestClient_cancelledContextReturnsPromptly
• classify table with captured responses: TestClassify_table
• Checker reasons, expired ctx, stable order: TestCheck_reasonsTable, TestCheck_expiredContextReturnsPromptly, TestCheck_oneEntryPerNodeInConfigOrder
• sentinel tests for CloudInit and the credential: TestCreate_cloudInitSentinelNeverLoggedOrInErrors, TestClient_tokenNeverInErrorsOrLogs, and TestLogging_v0SilentV1OutcomeV2Detail over a whole lifecycle at every level
• child spans under the operation span: TestTracing_httpSpansUnderOperationSpan, TestClient_childSpanUnderParent
• real-backend lifecycle test: TestProxmox_lifecycle and TestProxmoxCheck_realHost in proxmox_integration_test.go, passed on pv1-1

Tests that cover the Phase 0 corrections specifically, in addition to the plan's per-phase lists:

• TestCreate_resumesMidCloneVMAtDeterministicVMID: a crash after the clone was submitted; the retry waits for the lock and adopts, creating nothing.
• TestCreate_takenVMIDInspectedBeforeProbing: a foreign VM at attempt 0 is skipped; ours at attempt 0 is adopted.
• TestCreate_cloneAlreadyExistsRaceRunsOwnershipCheck.
• TestCreate_templateWithCloudInitDriveIsPermanent.
• TestCreate_seedUploadOverwritesWithoutDelete.
• TestCreate_resizeWaitsForTask, TestCreate_resizeSkippedWhenNotLarger.
• TestCreate_vlanTagAppendedKeepsMACAndOptions.
• TestBootDisk_table, with a row for order=scsi0;ide2.
• TestGet_pausedIsStoppedFromStatusCurrent.
• TestGet_agentMessagesTable (the three messages and a 200 without IPv4).
• TestDelete_seedDeleteIsTaskAndIdempotent.
• TestClient_x509FailureIsPermanent.

Evidence: contract §11; docs/testing.md conventions.

22. Deployment

Answer: a new optional Secret and its mount, two Makefile targets, and samples:

• config/manager/manager.yaml: an optional Secret proxmox-token mounted read-only at /etc/proxy-operator/proxmox, keys token and ca.pem. /tmp is already an emptyDir.
• Makefile: proxmox-token-secret, creating or updating proxmox-token from PROXMOX_TOKEN_FILE (default $(HOME)/.config/proxmox-token) and PROXMOX_CA_FILE (default $(HOME)/.config/proxmox-ca.pem), shaped like kvm-ssh-secret; and test-integration-proxmox (go test -race -tags=integration -count=1 -v -timeout 30m -run TestProxmox ./internal/provider/proxmox/).
• config/manager/providers_config.yaml and config/samples/providers-config.yaml: a commented proxmox-home entry for the user's host: url: https://192.168.0.8:8006, the two file paths, nodes: [pv1-1], isoStorage: proxy-seeds, vlanTag: 4, templates: {debian-13: {vmid: 9001}}, a small machine type.
• config/samples/proxy_proxmox.yaml, listed in config/samples/kustomization.yaml: provider proxmox-home, zone pv1-1, machine type small, image debian-13, the Squid cloud-init from proxy_kvm2.yaml.
• RBAC: none; the provider does not talk to the cluster's API.
• hack/providers-dev.yaml: no entry (needs credentials).

Evidence: contract §12.

23. Docs

Answer: docs/proxmox-provider.md with sections: Proxmox prerequisites (the template recipe and what disqualifies a template, the dedicated seed storage, the user, both roles and the token, the proxy network with DHCP and internet, reserving the VMID range), Cluster-side setup (the Secret, the config entry, the URL-must-match-the-certificate rule), Config reference (entry 15), Placement (Region ignored, catalogue-edit warning), What lands on Proxmox (the VMID scheme, the description mark, the tag, the seed ISO, the config keys written, qm unlock for a stuck clone), Logging, Tracing, Timeouts, Health check (entry 16), Error classes (entry 13), Known gaps (entry 24), Testing. A fifth column in the three tables of docs/providers-existing.md and a proxmox section. docs/architecture.md: components row, a proxmox block in §5, the span note in §10, a §11 row and the provider count, Decisions bullets (VMID from the UID hash with the ownership check, the mark in the description, the seed by API upload, the node resolved per call). A README row. docs/providers.md link. docs/testing.md rows and a proxmox integration section. CLAUDE.md Gotchas: make test-integration-proxmox. A CHANGELOG.md entry after the user confirms.

Evidence: contract §12.

24. Known limits accepted at design time

Answer: ten limits, each with its consequence and workaround:

• Create blocks until the VM runs, holding one of three reconcile workers. On the user's host that is about five seconds plus the clone; large templates on slow storage take longer.
• A clone lock that never clears (a node crash mid-clone) makes every Create for that Proxy wait out the clone budget and return ErrTransient. Clear it with qm unlock <vmid> or destroy the VM.
• Seed ISOs of VMs destroyed outside the operator are invisible to GC. Remove proxy-*-seed.iso files without a matching VM by hand.
• The VMID range is shared with humans. A VM created by hand inside it costs a probe; a hand-made VM named like a proxy is refused as ErrPermanent. Reserve the range.
• Cross-node clones need shared storage and are untested, because the host has one node.
• A VM whose agent was switched off is replaced, not repaired (entry 7).
• The proxy network needs internet for the agent install, unless the template ships the agent.
• The CA is cached after first use; rotating it needs a restart. The token rotates freely.
• A node that stops answering makes Get and Delete for its VMs ErrTransient until it returns, so deleting such a Proxy waits.
• Contract §13 limits that apply unchanged: config read once per process, no installation identity in GC (two operators sharing a cluster and a VMID range delete each other's VMs), an unconfigured provider latches the Proxy.

Accepted after the Phase 10 review (docs/reviews/2026-09-23-proxmox-provider-review/, findings P3 and P4), and listed in the guide's Known gaps:

• A Create that failed permanently after its clone keeps its VM, and the next Create adopts it by name even when placement has changed. Destroy the VM before changing placement on such a Proxy.
• A retry can clone a second VM when a Create is cut off while cloning onto a later probe VMID, the VM on the earlier one is removed, and the retry comes before the clone ends. The next Create names both VMIDs as ErrPermanent.

Changed by the Phase 10 captures of the three answers Phase 0 never saw:

• A hibernated VM of ours latches only Create's adoption (ErrPermanent, resume or destroy it; entry 2). Get reports it Stopped, so the reconciler replaces it, and Delete destroys it, since the destroy takes a hibernated VM (entry 11). Before the capture, Delete refused it as well.
• A missing isoStorage is a 400 with storage: No such storage., which the Checker reports as StorageNotFound (entry 16); a missing storage at upload time is Permanent like any 4xx.
• A paused VM stops like a running one (entry 11, step 5), as assumed.

Evidence: contract §13; entries 2, 7, 10, 11, 14 above.