# Provider spec: proxmox Filled from [provider-spec-template.md](provider-spec-template.md) against [provider-contract.md](provider-contract.md). This is the design the `proxmox` package is built from. Where an entry names an axis in backticks, the answer is one of that axis's profiles in contract §4. Backend facts come from the Phase 0 recon on the user's host, recorded in [the execution log](plans-executions/2026-09-18-1502-proxmox-provider.md) as facts F1 to F23, with the raw responses under `internal/provider/proxmox/testdata/`. An Evidence line citing `F` points there. Where this spec and the plan ([docs/plans/2026-09-18-1502-proxmox-provider.md](plans/2026-09-18-1502-proxmox-provider.md)) disagree, this spec wins: it folds in the ten corrections Phase 0 found. ## Header - **Type**: `proxmox` - **Example configured name**: `proxmox-home` - **Backend and API version this spec was written against**: Proxmox VE 9.2.20 (`pve-manager/9.2.20/49318c671b82f31e`, kernel `7.0.14-17-pve`, QEMU 11.0.3) on the single node `pv1-1`, API at `https://192.168.0.8:8006/api2/json`. - **Client**: `thin-client`: `net/http` and `encoding/json` from the standard library, with the transport wrapped by the already-required `go.opentelemetry.io/contrib/instrumentation/net/http/otelhttp`. No new dependency. The seed ISO uses `github.com/kdomanski/iso9660`, already required by kvm2. - **Author and date**: Claude (orchestrator session), 2026-09-23. ## 1. Identity and providerID **Answer:** `vms//`, where `` is the VM's numeric id and `` is `req.Name`. The ID is location-free: the node is looked up in `GET /cluster/resources?type=vm` on every call, so a VM migrated between nodes stays addressable. Parsing is strict: exactly three segments, the literal `vms`, a decimal VMID without leading zeros between 100 and 999999999, and a non-empty name without `/`. Anything else is `ErrPermanent` from `Get` and `Delete`. The name is carried so that `Get` and `Delete` can tell our VM from another VM that later received the same VMID (entry 2). No config value is pasted into the ID, so no ID-safety regexp is needed; node names and storage ids are still validated at load (entry 15) because they end up in request paths. **Evidence:** contract §6 and §4.10 (the ID must not depend on the location when instances can migrate); F8 (`node` is a field of every resource entry). ## 2. Primary key (`primary-key`) **Answer:** `allocated-id`. Proxmox addresses VMs by VMID; `name` is only a label and is not unique. The VMID is derived from the Proxy UID so a retry lands on the same number: `vmidFor(uid, attempt)` is `min + (fnv1a64(uid) + attempt) mod (max - min + 1)` over `vmidRange` (entry 15). Create, under the per-name lock (entry 19): 1. Reads `/cluster/resources` once. Entries with `type: qemu`, `template: 0` and `name == req.Name` are the by-name candidates. Two or more is `ErrPermanent` naming every VMID: an admin must remove the duplicate. Exactly one goes to the ownership check below and is adopted or refused. 2. With none, probes `attempt = 0, 1, …` up to 32: - VMID free in the listing: clone onto it. If the clone fails with `unable to create VM : config file already exists` (F4), someone took it between the listing and the clone; run the ownership check on it. - VMID taken: run the ownership check. - Exhausted: `ErrTransient` naming the range. 3. The **ownership check** on a VMID: if its resource entry carries a `lock` or has no `name` (a clone in progress lists neither, F8), wait until it is unlocked (entry 10, clone budget). Then read its config. It is **ours** when `name == req.Name` and the description carries the mark with `uid` equal to `req.UID` or empty: adopt it and resume Create at the seed step (entry 10). It is **foreign** when the name differs: move to the next attempt. The same name with no mark, or with a different non-empty `uid`, is `ErrPermanent` naming the VMID and both UIDs; the operator never takes over a VM it did not create. A `suspended` lock (a VM hibernated to disk) is not waited on: it is not a task and never clears by itself. Ownership is judged at once, as `Get` does (entry 9): a foreign hibernated VM is probed past, and ours is `ErrPermanent` (resume or destroy it), like kvm2's paused domain. Container and template entries at a VMID are foreign without a config read. The wait in step 3 is what closes the duplicate-VM bug Phase 0 found: after a crash mid-clone, the retry's by-name lookup misses the nameless VM, but attempt 0 lands on the same VMID, waits for the clone to finish, and adopts it instead of probing past it. `Get` and `Delete` check the name and mark at the VMID in the ID. A VM there with another name means ours is gone and the VMID was reused: `Get` answers `ErrNotFound`, `Delete` succeeds without touching it. **Evidence:** F4 (taken VMID fails synchronously); F8 (a mid-clone entry has no name); contract §4.10 (allocate-then-verify, deterministic from the UID) and §13 (adoption not verified: closed here). ## 3. Placement mapping **Answer:** three fields are catalogue keys checked before any backend call, one is ignored: | Field | Meaning | Required | Unknown value | |---|---|---|---| | `Zone` | a node from `nodes` | only when more than one node is configured; with one node an empty zone selects it | `ErrPermanent`: `placement.zone "x" is not in proxmox.nodes` | | `MachineType` | a key of `machineTypes` (`cores`, `memoryMiB`, optional `diskGiB`) | yes | `ErrPermanent`: `placement.machineType "x" is not in proxmox.machineTypes` | | `Image` | a key of `templates` (a template VMID) | yes | `ErrPermanent`: `placement.image "x" is not in proxmox.templates` | | `Region` | ignored | | still part of the spec hash: changing it replaces the VM | The guide documents the ignored field and that catalogue edits affect only new VMs. **Evidence:** contract §2.1 (reject early, MAY ignore with documentation). ## 4. Image source **Answer:** a template VM prepared by the admin, referenced by a catalogue key: `templates..vmid`. Create makes a **full clone** (`full=1`), so a VM owns copies of its disks and keeps nothing linked to the template: the template may be edited, replaced or deleted without affecting running VMs. Before cloning, Create reads the template's config and refuses it with `ErrPermanent` when it is not a template (`template` is not 1) or when any disk slot holds a Proxmox cloud-init drive (a value containing `cloudinit`), which would compete with the provider's seed (entry 5). Because the clone copies the template's config, what the config or resize step would refuse on the clone is refused on the template too, before any VM exists: no `net0` while `bridge` or `vlanTag` is set, and no boot disk, or one without a readable `size=`, while `diskGiB` is set. A Permanent error after the clone would leave a VM that fixing the template cannot help. The template must boot from its first disk, run cloud-init with the NoCloud datasource, and either ship `qemu-guest-agent` or be able to install it (entry 7). Clone parameters: `newid`, `name`, `description` (the mark, entry 8), `full=1`, `storage` when configured (empty clones onto the template's own disk storage), and `target` when the chosen node differs from the template's node. A cross-node clone needs the template on shared storage; this host has one node, so that path is untested (entry 24). Catalogue edit warning for the guide: changing a key's `vmid` changes only VMs created afterwards. Add a new key rather than repointing one in use, so the Proxies that name it keep a meaning you can reason about. **Evidence:** F19 (template 9001: `scsi0`, `boot: order=scsi0`, `ide2` free, `agent: 1`; template 9000 carries a cloud-init drive on `ide2`); F15 (a clone onto a storage without image support is refused up front); F16 (a full clone of 9001 took about two seconds). ## 5. Cloud-init delivery **Answer:** a NoCloud seed ISO built in-process by the shared `internal/provider/nocloud` package (extracted from kvm2 in Phase 3): volume label `cidata`, files `user-data` (the Proxy's `CloudInit`, byte for byte, never parsed), `meta-data` (`instance-id: ` and `local-hostname: `) and `vendor-data` (the guest agent script, joined with the configured `vendorData` as a MIME multipart when both are set; identical to kvm2's rules). The builder stages under `os.TempDir()` (`/tmp`, the manager's emptyDir) and writes into a `bytes.Buffer`; a seed is tens of KiB, so buffering is within contract §1.4. The ISO is uploaded with `POST /nodes/{node}/storage/{isoStorage}/upload` as multipart (`content=iso`, file part `filename` named `-seed.iso`) and the returned `imgcopy` task is waited on. An upload **overwrites** an existing file of the same name, so no delete precedes it; the seed is rebuilt and re-uploaded on every Create attempt, since a previous attempt may have been cut off. It is attached as `ide2: :iso/-seed.iso,media=cdrom`; Proxmox then appends `ide2` to the boot order on its own, which is harmless because the boot disk stays first. Only the Proxmox API is used: the storage upload endpoint accepts `iso`, `vztmpl` or `import`, never `snippets`, so `cicustom` would need shell access to the node. Size: the request timeout (entry 18) bounds the upload; a seed with a multi-megabyte `vendorData` would need a larger `requestTimeoutSeconds`. **Evidence:** F12 (upload fields, UPID, overwrite, `.iso` suffix required); F19 (`ide2` appended to the boot order); F21 (the seed on `ide2` installed the agent on both the untagged network and VLAN 4); F23 (the upload is staged in `/var/tmp` on the node, not the tmpfs `/tmp`); contract §2.1 and §11 (byte for byte, sentinel test). ## 6. Machine type mapping **Answer:** a catalogue: `machineTypes.` is `{cores, memoryMiB, diskGiB}`. `cores` and `memoryMiB` are written in the config step (entry 10). `diskGiB` is optional: when set and larger than the cloned boot disk, the disk is grown with `PUT /nodes/{node}/qemu/{vmid}/resize` to the absolute size `G`, and the returned task is waited on; when unset or not larger, the disk keeps the template's size and no call is made. The provider never shrinks a disk. Cloud-init's growpart expands the root filesystem at first boot. The boot disk is the first entry of the config's `boot: order=…` that is a disk key (`scsiN`, `virtioN`, `sataN`, `ideN`) and not a CD-ROM, else the `bootdisk` key, else the first present of `scsi0`, `virtio0`, `sata0`, `ide0`. Its current size comes from the `size=` option of that disk's value. No boot disk, when `diskGiB` is set, is `ErrPermanent`. **Evidence:** F20 (resize is a task; the same size is an `OK` no-op; a shrink fails in the task with `shrinking disks is not supported`); F19 (`boot: order=scsi0`, becoming `order=scsi0;ide2` once the seed is attached). ## 7. Network and IP discovery (`readiness`, `ip-source`) **Answer:** `readiness`: `provisioning-window`. `ip-source`: `discovered`, from the QEMU guest agent only (`GET /nodes/{node}/qemu/{vmid}/agent/network-get-interfaces`); the API offers no ARP or DHCP-lease source. The address reported is the first IPv4 of the first non-loopback interface, in the agent's order, that is not loopback, link-local or unspecified. The NIC is the template's `net0`, rewritten in the config step only when the config sets `bridge` or `vlanTag`: the value's comma-separated options are kept in order with the model and MAC untouched, `bridge=` and `tag=` are replaced or appended. With neither set, `net0` is left as cloned. The agent answers: | Agent response | Meaning | Get reports | |---|---|---| | 200 with a routable IPv4 | booted | Running with that IP | | 200 without one | network still coming up | Provisioning | | 500 `QEMU guest agent is not running` | booting, or the agent install has not finished or failed | Provisioning | | 500 `VM is not running` | the VM stopped between the status read and this call | `ErrTransient` | | 500 `No QEMU guest agent configured` | `agent: 1` was removed from the config; this VM can never report an address | Stopped, so the reconciler replaces it with a VM that has the agent on | | anything else | cannot tell | `ErrTransient` | The guest needs the agent. The vendor-data script installs it at first boot unless `installGuestAgent: false`, and that install needs a package mirror: **the proxy network must have a route to the internet**, which an egress proxy needs anyway. Baking `qemu-guest-agent` into the template and setting `installGuestAgent: false` removes the mirror from IP discovery. `Port` is not used by the provider. Reachability is a prerequisite in the guide: the operator's pod network and the clients must route to the proxy network (on the user's host, VLAN 4, `192.168.4.0/24`). **Evidence:** F5 (all three agent messages); F21 (agent response shape; an address about 50 seconds after start on the untagged network and on VLAN 4); the Phase 0 log's VLAN 4 section (no internet route meant no agent); contract §1.3 and §2.2 (Running only with an IP). ## 8. GC marks and CreatedAt (`gc-mark`, `created-at`) **Answer:** `gc-mark`: `written-metadata`. `created-at`: `written-at-create`. The mark is three lines in the VM's `description`, passed in the clone call itself, so there is no mark window: ``` proxy-operator-managed=true proxy-operator-uid= proxy-operator-created-at= ``` The keys are `provider.LabelManaged`, `provider.LabelUID` and `proxy-operator-created-at`; the managed value is `provider.LabelManagedYes`. The parser reads `key=value` lines, tolerates CRLF, blank lines and any other lines an admin adds, and treats the VM as managed only when the managed value is exactly `true`. A missing `uid` line gives an empty UID. A `created-at` that does not parse gives the zero time, which GC treats as old; that is accepted. The clone's own `meta: …,ctime=…` is not usable: Proxmox copies it from the template. The config step also sets the tag `proxy_operator`, for filtering in the Proxmox UI and as a ListByTag candidate hint (entry 12). It is not the mark: tags cannot be passed to the clone call. Hyphens are accepted in tags on 9.2.20, but the underscore form is kept because it is valid under every tag rule Proxmox has had. Uppercase is folded to lowercase by Proxmox. The seed ISO carries no mark; its name `-seed.iso` identifies it (entry 11). **Evidence:** F10 (the description survives the clone verbatim, newlines included); F11 (the clone's `meta` is the template's); F9 (tag rules); contract §2.1 (mark in the create call) and §4.5. ## 9. State mapping (`provisioning-state`) **Answer:** `provisioning-state`: `create-blocks`. Create returns only once `status/current` reports the VM running (entry 10), so a VM the provider cloned but never started is only ever seen by a Create retry, which adopts and starts it; the reconciler has no ID for it to `Get`. `Get` checks in this order; the first match decides: | Check | Source | Result | |---|---|---| | VMID absent | `/cluster/resources` | `ErrNotFound` | | `lock` set, other than `suspended` | resources entry (a clone in progress also has no name) | `ErrTransient` | | `lock: suspended` (hibernated to disk) | resources entry | Stopped | | `status: unknown` (node not reachable) | resources entry | `ErrTransient` | | config `does not exist` | `GET …/config` | `ErrNotFound` (deleted between the two calls) | | config `lock` set | config | `ErrTransient` | | `name` differs from the ID's name | config | `ErrNotFound`, logged: the VMID was reused | | no mark in the description | config | `ErrNotFound`, logged: not ours | | `status: stopped` | `status/current` | Stopped | | `status: running`, `qmpstatus: running` | `status/current` | the agent decides (entry 7) | | `status: running`, `qmpstatus: prelaunch` | `status/current` | Provisioning | | `status: running`, any other `qmpstatus` (`paused`, `suspended`, `io-error`, …) | `status/current` | Stopped | | any other `status` | `status/current` | Stopped | A `suspended` lock decides the state but not ownership: the config is still read, and a foreign name or a missing mark is still `ErrNotFound`, so a hibernated VM that reused our VMID is never reported as ours. Only `status/current` and the agent are skipped for it. `status/current` is read, not only the resources entry, because a paused VM shows as plain `running` in `/cluster/resources`. `UID` and `CreatedAt` come from the description on every call. A running `Get` makes four calls: resources, config, status and agent. **Evidence:** F8 (resources fields; mid-clone entries lack a name, carry `lock: clone` and `status: unknown`); F14 (stopped and paused as seen by `status/current` and by resources); F2 (the `does not exist` message); contract §2.2 and §4.11. ## 10. Create: blocking and task tracking (`create-return`, `async-ops`) **Answer:** `create-return`: `blocking`. `async-ops`: `task-handle` and `backend-lock`. Clone, upload, resize, start, stop, destroy and volume delete each return a UPID; the provider polls `GET /nodes/{node}/tasks/{upid}/status` every second until `status` is `stopped`, then reads `exitstatus`: `OK` is success, and so is `WARNINGS: `, which Proxmox itself counts as a finished task (logged at V(1)); anything else is `ErrTransient` carrying that text. The node comes from the UPID (`UPID::…`). The handle is held within the call only: Create blocks until the VM runs, so no handle outlives it, and a failure surfaces as Create's own error with the task's message. Locks: a VM locked by a task is waited on in Create (the ownership check) and reported as `ErrTransient` by `Get` and `Delete`. A start issued while a clone holds the lock queues behind it rather than failing; the provider never relies on that, because it waits for the clone task first. Steps, each idempotent, in order, under the per-name lock (entry 19): 1. Validate placement (entry 3). No backend call yet. 2. `GET /cluster/resources` once, then look the name up (entry 2). 3. When the name finds nothing: find the template's node (a missing template is `ErrPermanent`), read the template's config and refuse it as entry 4 says, then allocate the VMID (entry 2). The template is checked only here, because a VM that already exists no longer depends on it (entry 4): a template deleted after its clone must not fail that clone's adoption. An adopted VM, found by name or by the probe, gets one `status/current` read. Running, it returns its ID at once. Stopped, it continues at step 5. 4. Clone and wait (clone budget). 5. Build the seed and upload it to the VM's node, overwriting; wait (upload budget). 6. `PUT /nodes/{node}/qemu/{vmid}/config` (synchronous) with `cores`, `memory`, `agent=1`, `tags=proxy_operator`, `ide2=:iso/-seed.iso,media=cdrom` and, when configured, the rewritten `net0` (entry 7). 7. Resize when `diskGiB` is larger than the boot disk (entry 6); wait (resize budget). 8. `POST …/status/start`; wait (start budget). 9. Poll `status/current` every second until `status: running` and `qmpstatus: running`, up to the running budget; expiry is `ErrTransient`, and so is a `stopped` status after the start. 10. Return `vms//`. What a retry finds: | Previous attempt stopped after | The retry | |---|---| | nothing | allocates, clones | | submitting the clone | finds the VMID locked and nameless, waits, adopts, continues at 5 | | the clone | adopts the stopped VM, continues at 5 | | any of steps 5 to 7 | adopts, repeats 5 to 7 (upload overwrites, config is a replace, resize is skipped once large enough) | | the start | adopts the running VM, returns its ID | A Proxmox `does not exist` inside Create (a VM or volume vanishing between steps) is `ErrTransient`, never `ErrNotFound`. The log line for a Create records `cloudInitBytes`, never the content. **Evidence:** F3 (a start queues behind the clone lock); F7 (a config write under the clone lock is refused with `VM is locked (clone)`); F16 (task status fields); F20 (resize is a task); the Phase 0 log's VLAN 4 rerun (clone to started VM in about five seconds); contract §2.1, §4.2, §4.12 and §5 ("do not return NotFound from Create"). ## 11. Delete and secondary resources (`delete-return`, `secondary-resources`) **Answer:** `delete-return`: `waits`. `secondary-resources`: `explicit` for the seed ISO; the VM's disks are removed by the destroy call itself, with `purge=1&destroy-unreferenced-disks=1`. Under the per-name lock: 1. Parse the ID (`ErrPermanent` if malformed). 2. `/cluster/resources`. VMID absent, or held by a container: skip to step 7, trying every configured node because the VM's node is no longer known. 3. `lock` set, other than `suspended`, or `status: unknown`: `ErrTransient`. A clone or another task still running finishes first; the reconciler calls Delete again after `DeletionPoll`. 4. Read the config. `does not exist`: skip to step 7 on every node. A different name, or no mark: the VM is not ours; log it at V(1) and skip to step 7 on every node without touching it. Our VM with a `lock` other than `suspended` (taken since the listing): `ErrTransient`. Our VM with `lock: suspended` (hibernated): `ErrPermanent` asking the admin to resume or destroy it, the same answer Create's ownership check gives. The destroy refuses a locked VM, and that lock never clears on its own, so Transient would retry forever; Permanent puts the admin action in the Proxy's status, where `reconcileManaged` latches it and the finalizer path keeps retrying it visibly. A hibernated VM that is not ours is left alone like any other. 5. Status (from the listing) other than `stopped`: `POST …/status/stop`, wait (stop budget). Stop works on a paused VM too, which the listing shows as `running`. `does not exist` means the VM is gone. 6. `DELETE /nodes/{node}/qemu/{vmid}?purge=1&destroy-unreferenced-disks=1`, wait (destroy budget). `does not exist` is success. 7. `DELETE /nodes/{node}/storage/{isoStorage}/content/:iso/-seed.iso`, wait (seed-delete budget), on the VM's node when steps 5 and 6 ran on a VM of ours, and on every configured node in config order otherwise. A missing volume is still a successful task, so no message is matched. On the every-node path a node whose request or task fails is logged at V(1) and skipped, unless ctx ended, which fails the call; on the VM's node a failure fails the call (a 403 for `Datastore.Allocate` is Permanent). 8. Return nil. Everything is gone when Delete returns nil, so `Get` afterwards is `ErrNotFound`. A concurrent `Get` during a Delete keeps seeing the VM (as Stopped once it is stopped) until the destroy task removes its config, because destroying sets no lock and leaves the resource entry in place until the end. A second Delete waits on the lock, then finds nothing. GC blind spot: a VM destroyed outside the operator leaves its seed ISO, which ListByTag cannot see. The guide says seeds are named `proxy-*-seed.iso` on `isoStorage` and any without a matching VM may be removed. **Evidence:** F2 (a second destroy gives `does not exist`); F13 (volume delete is an `imgdel` task, needs `Datastore.Allocate`, and succeeds on a missing volume); the Phase 0 capture of resources and config mid-destroy (`cluster_resources_during_destroy.json`, `config_during_destroy.json`); contract §2.3. ## 12. ListByTag scope and partial failure (`list-partial`) **Answer:** `fail-whole` for the one listing, with single unreadable VMs skipped. Scope: every VM the cluster knows, from one `GET /cluster/resources?type=vm`, which covers every node, including unreachable ones, whose guests Proxmox lists with `status: unknown` (entry 13). Candidates are entries with `type: qemu`, `template: 0`, and either the tag `proxy_operator` (tags split on `;`, `,` or space) or a name starting with `proxy-`. A nameless entry (a clone in progress) is not a candidate: its Create is live and it is younger than `MinAge`. For each candidate the config is read and the description parsed; managed VMs become an `Instance` with `ID`, `UID` and `CreatedAt` only, which is all GC reads (contract §2.4); no status or agent call is made. A candidate whose config cannot be read (a node down, a VM destroyed mid-sweep) is skipped with a V(1) line, unless ctx ended, which fails the call. The resources call failing fails the whole call. Filtering by the mark is client-side; the tag narrows the candidates. Cost: one resources call plus one config read per candidate per sweep. **Evidence:** F8 (resources fields, tags separated by `;`, nameless mid-clone entries); F9 (tags case-folded); contract §2.4 and §4.7. ## 13. Error classification (`not-found`, `quota`) **Answer:** `not-found`: `error-code`, the 500 message `Configuration file 'nodes//qemu-server/.conf' does not exist`, plus `list-and-miss` on `/cluster/resources`. The listing can miss a VM only when it is gone: it is built from the cluster filesystem, which holds every guest's config cluster-wide, and Proxmox lists the guests of an unreachable node with `status: unknown` rather than dropping them. That last point is Proxmox's documented design, not something the single-node host could show; Phase 10 or a second node verifies it. `quota`: `absent`. The thin `local-lvm` storage overprovisions, so no out-of-space response could be produced; capacity failures are `ErrTransient`, and the Checker reports free space in `Details` instead (entry 16). The client carries each failure as `apiError{Status, Message, Errors}`: `Message` is the body's `message` with the trailing newline trimmed, or the reason phrase when the body has none (a 401); `Errors` is the 400 body's per-parameter map. One `classify(op string, err error)`: | Backend response | Class | Why | |---|---|---| | 401 `Authentication failed!`, 401 `No ticket` | Permanent | the token or its secret is wrong; fixed in the Secret | | 403 `Permission check failed (, )` | Permanent | the token lacks a privilege; the message names it | | 400 `Parameter verification failed.` with `errors` | Permanent | the request is wrong; the message includes the field errors | | other 4xx | Permanent | a request Proxmox will never accept | | 500 `Configuration file '…' does not exist` | NotFound (Transient when `op == "create"`) | the VM is gone | | 500 `VM is locked ()` | Transient | a task holds the VM | | 500 `unable to create VM : config file already exists` | handled by the allocator (entry 2); Transient if it escapes | the VMID is taken | | 500 `QEMU guest agent is not running`, `No QEMU guest agent configured`, `VM is not running` | handled by `Get` (entry 7); Transient if they escape | agent state | | any other 5xx, including pveproxy's 595 and 596 | Transient | unknown means retry | | task `exitstatus` other than `OK` or `WARNINGS: ` | Transient, with the text | the task failed; the next attempt retries | | TLS verification failure (`x509.UnknownAuthorityError`, `x509.HostnameError`, `x509.CertificateInvalidError`) | Permanent | wrong URL or CA file; the admin must act | | token file missing (`fs.ErrNotExist`) | Transient | the Secret has not landed yet | | token file present but not `@!=` | Permanent | wrong content in the Secret | | `context.Canceled`, `context.DeadlineExceeded`, `net.OpError`, `io.EOF`, `io.ErrUnexpectedEOF` | Transient | network or cancellation | | an error already wrapped with a class | that class | validation and builders classify at the source | | anything else | Transient | contract default | Every matched string is backed by a captured response under `testdata/`; the table test loads them rather than repeating the text. Matching is on fixed substrings anchored to the 500 status: `does not exist`, `VM is locked`, `config file already exists`, `QEMU guest agent is not running`, `No QEMU guest agent configured`, and for the stopped VM the whole message against `^VM [0-9]+ is not running$`. The agent messages are tested in that order, because the last is a suffix of the first. **Evidence:** F1 (error text in the reason phrase and the body; 401 has no body); F2, F4, F5, F7 (the 500 messages); F6 (no out-of-space response could be produced on thin storage); F22 (both 401 texts); fixtures `err_forbidden.json`, `err_bad_param.json`, `err_tag_invalid.json`, `err_upload_bad_name.json`; the Phase 0 TLS finding (`192.168.2.8` not in the node certificate); contract §5. ## 14. Credentials and TLS (`credentials`) **Answer:** `mounted-files`. The config names `tokenFile`, one line `@!=`, sent as `Authorization: PVEAPIToken=`. It is expected in the optional Secret `proxmox-token`, key `token`, mounted at `/etc/proxy-operator/proxmox`. The file is read on **every request**, so rotating the Secret takes effect without a restart. `New` stats it and logs a warning when it is missing, but never fails on it and never dials. TLS: `ca-file`. `caFile` names a PEM bundle, typically the node's `/etc/pve/pve-root-ca.pem`, carried in the same Secret as key `ca.pem`. It is loaded on first use and cached, so rotating the CA needs a restart; a missing CA file is `ErrTransient` until it lands, one that does not parse is `ErrPermanent`. Without `caFile` the system roots are used. The URL's host must be a name or address in the node certificate's SANs: on the user's host that is `192.168.0.8` or `pv1-1.home.lab`, not `192.168.2.8`. `insecureSkipTLSVerify: true` disables verification and is logged as a warning by `New`. Minimum privileges, as verified on 9.2.20, for `proxy-operator@pve` with a token without privilege separation: - a role on `/` with `VM.Allocate VM.Clone VM.Audit VM.Config.CDROM VM.Config.CPU VM.Config.Memory VM.Config.Disk VM.Config.Network VM.Config.Options VM.PowerMgmt VM.GuestAgent.Audit Datastore.Audit Datastore.AllocateSpace Datastore.AllocateTemplate Sys.Audit SDN.Use`; - a role on `/storage/` with `Datastore.Allocate Datastore.AllocateTemplate Datastore.AllocateSpace Datastore.Audit`, because deleting a volume needs `Datastore.Allocate` and a grant on a more specific path replaces the inherited one. A dedicated ISO-only storage keeps that privilege away from backups. `VM.Monitor` no longer exists in PVE 9 and is not needed. `VM.Config.Cloudinit` is not needed. `SDN.Use` on `/` covers a VLAN tag on `vmbr0`. Hygiene: the token never appears in logs, span attributes or error strings. `apiError.Error()` carries status and message only, never headers or URLs. Tested with a sentinel secret that a test server echoes in a 401 body. **Evidence:** F17 (privileges exercised and the one missing); the Phase 0 log's seed storage section; F22 (auth failure texts); contract §7. ## 15. Config block **Answer:** a new block `proxmox:` with the Go type `provider.ProxmoxConfig` in `internal/provider/config_proxmox.go`: | Field | Required | Default | Validation | Ends up in | |---|---|---|---|---| | `url` | yes | | `https` URL with a host and no path other than `/` | request base, `…/api2/json` appended | | `tokenFile` | yes | | non-empty | `Authorization` header | | `caFile` | | system roots | non-empty when set | TLS roots | | `insecureSkipTLSVerify` | | false | | TLS, warning at `New` | | `nodes` | yes, at least one | | unique, `^[a-z0-9]([a-z0-9-]*[a-z0-9])?$` | request paths, clone `target` | | `isoStorage` | yes | | Proxmox storage id `^[a-zA-Z][a-zA-Z0-9_.-]*$` | upload and delete paths, `ide2` | | `storage` | | the template's disk storage | storage id regexp when set | clone `storage` | | `bridge` | | as cloned | non-empty when set | `net0` `bridge=` | | `vlanTag` | | as cloned | 1 to 4094 when set | `net0` `tag=` | | `vmidRange.min`, `vmidRange.max` | | 100000, 199999 | min at least 100, max at most 999999999, min below max | `vmidFor` | | `templates..vmid` | at least one template | | at least 100 | clone source | | `machineTypes.` | at least one | | `cores` and `memoryMiB` above zero, `diskGiB` not negative | config step, resize | | `requestTimeoutSeconds` | | 30 | not negative | HTTP client timeout | | `vendorData` | | | | seed `vendor-data` | | `installGuestAgent` | | true | | seed `vendor-data` | `isoStorage` has no default on purpose: the token must be allowed to delete volumes there (entry 14), so the choice must be explicit. `nodes` holds node names as Proxmox knows them (`pv1-1`), not addresses; the one URL reaches the whole cluster. Wiring: `ProviderConfig.Proxmox *ProxmoxConfig` with tag `proxmox,omitempty`; `setBlocks` reports `proxmox`; the allow-list in `Config.validate` and the constructor map in `cmd/main.go` both gain `proxmox` in the same commit; `validateProxmox(prefix, c)` in `config_proxmox.go` with `config_proxmox_test.go`. **Evidence:** contract §10; the Phase 0 decisions (dedicated seed storage, `vlanTag`, URL at `192.168.0.8`); F13 (why `isoStorage` needs `Datastore.Allocate`). ## 16. Checker **Answer:** one target per `nodes` entry, `Name` the node, `Address` the URL, checked concurrently, returned in config order. The check shares the provider's HTTP client; each request is independent, so a check that runs out of time cancels only its own requests. Per node: 1. `GET /version` (`pveVersion`). 2. `GET /nodes/{node}/status` (the node answers; `kernel`). 3. `GET /nodes/{node}/storage/{isoStorage}/status`: the storage exists, is active, and lists `iso` in its content (`isoStorageAvailGiB`). 4. `GET /cluster/resources?type=vm`: every configured template exists and is a template (`templates` as `/`). Reasons, shared first: `Reachable`; `Timeout` (the round's ctx ended, whichever request was running); `Unreachable` (TCP connect or TLS handshake failed for a network reason, or pveproxy answered 595 or 596); `Unauthorized` (401); `Forbidden` (403, the message names the privilege); `APIError` (any other failure). Provider-specific, each fixed differently by an admin: - `CertificateInvalid`: TLS verification failed, or `caFile` is missing or holds no certificate. Fix `url` or `caFile`, not the host. - `CredentialsUnavailable`: the token file is missing or malformed. Fix the Secret, not Proxmox. - `StorageNotFound`: `isoStorage` is missing, inactive, or has no `iso` content. Fix the storage on the node. Any 500 from the storage status call counts as missing, whatever its wording (it does not exist, is not enabled on that node, or is offline; no missing-storage response was captured, so no text is matched), and the message carries Proxmox's text. A 401, 403, 595 or 596 there keeps its shared reason. - `TemplateNotFound`: a configured template VMID does not exist or is not a template. Fix the template or the catalogue. Details: `pveVersion`, `kernel`, `tokenID` (the part before `=`, never the secret), `isoStorage`, `isoStorageAvailGiB`, `templates`, filled as far as the check got; `isoStorage` is always there. `templates` counts catalogue entries. The Checker reads `tokenFile` itself, before the first request, with the client's `readToken`: the seam stays one method per endpoint, and a missing or malformed file answers `CredentialsUnavailable` without a request. Only the part before `=` is kept. `Timeout` also covers a request that runs out of its own timeout while the round is still running. A dial or a connection cut mid-response is `Unreachable`, even when the dial ran out of its own 10 s budget, because the node did not answer. Cost: four requests per node per round. The `status.type` comment in `api/v1alpha1/providerhealth_types.go` gains `proxmox` and the CRD is regenerated. **Evidence:** contract §8; fixtures `version.json`, `node_status.json`, `storage_status.json`, `cluster_resources.json`. ## 17. Client and child spans (`client-kind`, `child-spans`) **Answer:** `client-kind`: `thin-client`. `child-spans`: `transport-wrapper`. The seam is the interface `api`, one method per endpoint, `ctx` first: | Method | HTTP | Returns | |---|---|---| | `Version` | `GET /version` | version info | | `NodeStatus` | `GET /nodes/{node}/status` | node status | | `StorageStatus` | `GET /nodes/{node}/storage/{storage}/status` | storage status | | `ClusterResources` | `GET /cluster/resources?type=vm` | entries | | `VMConfig` | `GET /nodes/{node}/qemu/{vmid}/config` | config as `map[string]string` | | `SetVMConfig` | `PUT /nodes/{node}/qemu/{vmid}/config` | nothing (synchronous) | | `Clone` | `POST /nodes/{node}/qemu/{vmid}/clone` | UPID | | `Resize` | `PUT /nodes/{node}/qemu/{vmid}/resize` | UPID | | `Start` | `POST /nodes/{node}/qemu/{vmid}/status/start` | UPID | | `Stop` | `POST /nodes/{node}/qemu/{vmid}/status/stop` | UPID | | `VMStatus` | `GET /nodes/{node}/qemu/{vmid}/status/current` | status and qmpstatus | | `Destroy` | `DELETE /nodes/{node}/qemu/{vmid}?purge=1&destroy-unreferenced-disks=1` | UPID | | `TaskStatus` | `GET /nodes/{node}/tasks/{upid}/status` | status and exitstatus | | `AgentNetworkInterfaces` | `GET /nodes/{node}/qemu/{vmid}/agent/network-get-interfaces` | interfaces | | `UploadISO` | `POST /nodes/{node}/storage/{storage}/upload` (multipart) | UPID | | `DeleteVolume` | `DELETE /nodes/{node}/storage/{storage}/content/{volid}` | UPID | `httpAPI` is the production adapter; `fakeAPI` in `fake_test.go` keeps VMs, tasks and ISOs in memory, simulates locks (a clone leaves a nameless locked entry until its task completes), task polls, per-method errors and blocking, and records a call log. Scalars in a config are rendered as strings by the adapter (`cores` arrives as a JSON number, `memory` as a string). The transport is `tracing.RestConfigWrapper(opts...)(base)`, a parent-gated `otelhttp` transport, so each request is an `HTTP ` span under `provider.create`, `provider.get`, `provider.delete`, `provider.list` or the monitor's `providerhealth.check`, and background traffic makes no root spans. The `observed` decorator over `api` logs one line per call and starts no span, so nothing is counted twice. Span attributes are otelhttp's standard ones; the token is in a header, which otelhttp does not record. Tests: a span recorder asserts the HTTP spans are children of the operation span and carry no token; a reflection test asserts every `api` method logs exactly one line. **Evidence:** contract §4.15, §4.18, §9; `RestConfigWrapper` and `parentGatedTransport` in `internal/tracing/transport.go`; F18 (HTTP/1.1 only). ## 18. Timeouts (`timeouts`) **Answer:** `per-call`. | Bound | Budget | |---|---| | TCP dial | 10 s | | TLS handshake | 10 s | | one request, response included (`http.Client.Timeout`) | `requestTimeoutSeconds`, default 30 s | | clone task, and the wait for a VM's lock during the ownership check | 10 m | | upload task | 2 m | | resize task | 2 m | | start task | 2 m | | stop task | 2 m | | destroy task | 5 m | | seed delete task | 1 m | | wait for `running` after the start | 60 s | | task and status poll interval | 1 s | The budgets live in one `budgets` struct that tests shrink. A task wait that runs out is `ErrTransient` naming the UPID, and the task keeps running on Proxmox; the next attempt finds its result through the ownership check or the lock. A cancelled `ctx` cancels only that request: the transport keeps its idle connections (`MaxIdleConnsPerHost` 4, `IdleConnTimeout` 90 s), and no other call is disturbed. HTTP/2 is disabled explicitly (`ForceAttemptHTTP2: false`, an empty `TLSNextProto`). **Evidence:** contract §1.5 and §4.16; F16 (a full clone of the 3 GiB template took about two seconds; tasks finish within a second or two); F21 (about 50 s from start to an address, which the reconciler's polling covers, not the running wait). ## 19. Concurrency **Answer:** shared state is the HTTP client (safe for concurrent use), the lazily loaded CA pool (loaded once under a `sync.Once`-style guard), and a lock per instance name from the shared `internal/provider/keyedlock` package (extracted from kvm2 in Phase 3): a one-slot semaphore per key, waitable with ctx, removed from the map once unused. Create and Delete for one name run under that name's lock, so the reconciler's Delete and GC's Delete never race on stop and destroy, and a Delete cannot interleave with a Create's steps. Get and ListByTag take no lock. Two Creates for different Proxies that land on the same VMID are safe: one clone wins, the other gets the `config file already exists` answer, waits in the ownership check for the winner's clone to finish, sees a different name, and probes on. Three reconciles, a GC sweep and a check round at once are safe. **Evidence:** contract §1.2 (concurrency); F4; entry 2. ## 20. Logging (`logging`) **Answer:** `tiered`. Logger from `logf.FromContext(ctx)` on every call, named `proxmox`, with `provider` on every line and `vmid`, `name` and `node` on every line about an instance. - V(1): one line per operation outcome (`proxmox vm ensured running`, `proxmox vm fetched` with state, ip and uid, `proxmox vm deleted` with `destroyed` (false when no VM of ours was left to destroy), `proxmox vms listed` with counts, `proxmox node checked` with the reason), one `proxmox api call` line per seam call from the decorator (`call`, `node`, `vmid`, `storage`, `upid`, `duration`, plus `newid` and `name` on a clone, `filename` on an upload and `volid` on a volume delete, and on failure `httpStatus` and `error`), a `proxmox task failed` line with the UPID and `exitstatus`, and the no-op lines of Delete and ListByTag (VM already gone, foreign or unmarked VM left alone, seed delete skipped on a node, unreadable candidate skipped). - V(2): request parameters (without `description`'s content at V(1) and never the upload body), config summaries, agent addresses, the resolved Create parameters with `cloudInitBytes`. - No V(5) wire tier. - Never logged at any level: `CloudInit`, the token, the CA content, the upload body. **Evidence:** contract §4.17 and §9. ## 21. Tests (`backend-test`) **Answer:** `backend-test`: `integration-tag`. Unit tests run over `fakeAPI` behind the seam, the HTTP client over `httptest.NewTLSServer` serving the captured fixtures, and `classify` over the fixtures directly. The integration test is `proxmox_integration_test.go` behind `//go:build integration`, skipped unless `PROXMOX_TEST_URL`, `PROXMOX_TEST_TOKEN_FILE`, `PROXMOX_TEST_NODE`, `PROXMOX_TEST_TEMPLATE_VMID` and `PROXMOX_TEST_ISO_STORAGE` are set; optional `PROXMOX_TEST_CA_FILE`, `PROXMOX_TEST_INSECURE`, `PROXMOX_TEST_STORAGE`, `PROXMOX_TEST_BRIDGE`, `PROXMOX_TEST_VLAN_TAG`, `PROXMOX_TEST_VMID_MIN`, `PROXMOX_TEST_VMID_MAX`, `PROXMOX_TEST_TIMEOUT` (default 20m), `PROXMOX_TEST_KEEP`, `PROXMOX_TEST_LOG_LEVEL`; run with `make test-integration-proxmox`. It asserts, in order: `New` returns without a request; `Check` reports the node Ready with details; `Create` returns `vms//` with the VMID in range; a second `Create` returns the same ID and the VM count is unchanged; `Get` reaches Running with an IPv4 within the budget and its `UID` and `CreatedAt` equal what Create wrote; `ListByTag` contains it; `Delete` returns nil; `Get` is `ErrNotFound`; `Delete` again returns nil; no VM with the name and no `-seed.iso` remain; an HTTP span is a child of `provider.create` and no span attribute carries the cloud-init text. Contract §11 list, with the planned test for each item (Phase 8 ticks them once they exist): - [ ] repeat `Create`: `TestCreate_repeatReturnsSameIDAndCreatesNothing` - [ ] `Get` after `Delete`: `TestGet_afterDeleteReturnsNotFound` - [ ] `Delete` twice: `TestDelete_twiceSucceeds` - [ ] malformed ID: `TestParseProviderID_malformed`, `TestGet_malformedIDIsPermanent`, `TestDelete_malformedIDIsPermanent` - [ ] unknown state maps to `StateStopped`: `TestMapState_table`, `TestGet_unknownStateIsStopped` - [ ] `StateRunning` requires an IP: `TestGet_runningWithoutIPIsProvisioning`, `TestGet_runningWithIPIsRunning` - [ ] `UID` and `CreatedAt` read back: `TestGet_readsUIDAndCreatedAtFromBackend` - [ ] `ListByTag` excludes unmarked instances: `TestListByTag_excludesUnmarkedVMs`, `TestListByTag_emptyUIDStillListed` - [ ] `Get` during an in-progress delete: `TestGet_duringDeleteStillReturnsInstance` - [ ] `Get` during an asynchronous operation or lock: `TestGet_lockedIsTransientNeverNotFound`, `TestGet_midCloneNamelessEntryIsTransient` - [ ] never started is never `StateStopped`: `TestCreate_returnsOnlyWhenRunning` - [ ] cancelled `ctx` returns promptly: `TestProvider_cancelledContextReturnsPromptly`, `TestClient_cancelledContextReturnsPromptly` - [ ] `classify` table with captured responses: `TestClassify_table` - [ ] Checker reasons, expired `ctx`, stable order: `TestCheck_reasonsTable`, `TestCheck_expiredContextReturnsPromptly`, `TestCheck_oneEntryPerNodeInConfigOrder` - [ ] sentinel tests for `CloudInit` and the credential: `TestCreate_cloudInitSentinelNeverLoggedOrInErrors`, `TestClient_tokenNeverInErrorsOrLogs` - [ ] child spans under the operation span: `TestTracing_httpSpansUnderOperationSpan`, `TestClient_childSpanUnderParent` - [ ] real-backend lifecycle test: `TestProxmox_lifecycle` and `TestProxmoxCheck_realHost` in `proxmox_integration_test.go` Tests that cover the Phase 0 corrections specifically, in addition to the plan's per-phase lists: - `TestCreate_resumesMidCloneVMAtDeterministicVMID`: a crash after the clone was submitted; the retry waits for the lock and adopts, creating nothing. - `TestCreate_takenVMIDInspectedBeforeProbing`: a foreign VM at attempt 0 is skipped; ours at attempt 0 is adopted. - `TestCreate_cloneAlreadyExistsRaceRunsOwnershipCheck`. - `TestCreate_templateWithCloudInitDriveIsPermanent`. - `TestCreate_seedUploadOverwritesWithoutDelete`. - `TestCreate_resizeWaitsForTask`, `TestCreate_resizeSkippedWhenNotLarger`. - `TestCreate_vlanTagAppendedKeepsMACAndOptions`. - `TestBootDisk_table`, with a row for `order=scsi0;ide2`. - `TestGet_pausedIsStoppedFromStatusCurrent`. - `TestGet_agentMessagesTable` (the three messages and a 200 without IPv4). - `TestDelete_seedDeleteIsTaskAndIdempotent`. - `TestClient_x509FailureIsPermanent`. **Evidence:** contract §11; docs/testing.md conventions. ## 22. Deployment **Answer:** a new optional Secret and its mount, two Makefile targets, and samples: - `config/manager/manager.yaml`: an optional Secret `proxmox-token` mounted read-only at `/etc/proxy-operator/proxmox`, keys `token` and `ca.pem`. `/tmp` is already an emptyDir. - Makefile: `proxmox-token-secret`, creating or updating `proxmox-token` from `PROXMOX_TOKEN_FILE` (default `$(HOME)/.config/proxmox-token`) and `PROXMOX_CA_FILE` (default `$(HOME)/.config/proxmox-ca.pem`), shaped like `kvm-ssh-secret`; and `test-integration-proxmox` (`go test -race -tags=integration -count=1 -v -timeout 30m -run TestProxmox ./internal/provider/proxmox/`). - `config/manager/providers_config.yaml` and `config/samples/providers-config.yaml`: a commented `proxmox-home` entry for the user's host: `url: https://192.168.0.8:8006`, the two file paths, `nodes: [pv1-1]`, `isoStorage: proxy-seeds`, `vlanTag: 4`, `templates: {debian-13: {vmid: 9001}}`, a `small` machine type. - `config/samples/proxy_proxmox.yaml`, listed in `config/samples/kustomization.yaml`: provider `proxmox-home`, zone `pv1-1`, machine type `small`, image `debian-13`, the Squid cloud-init from `proxy_kvm2.yaml`. - RBAC: none; the provider does not talk to the cluster's API. - `hack/providers-dev.yaml`: no entry (needs credentials). **Evidence:** contract §12. ## 23. Docs **Answer:** `docs/proxmox-provider.md` with sections: Proxmox prerequisites (the template recipe and what disqualifies a template, the dedicated seed storage, the user, both roles and the token, the proxy network with DHCP and internet, reserving the VMID range), Cluster-side setup (the Secret, the config entry, the URL-must-match-the-certificate rule), Config reference (entry 15), Placement (Region ignored, catalogue-edit warning), What lands on Proxmox (the VMID scheme, the description mark, the tag, the seed ISO, the config keys written, `qm unlock` for a stuck clone), Logging, Tracing, Timeouts, Health check (entry 16), Error classes (entry 13), Known gaps (entry 24), Testing. A fifth column in the three tables of `docs/providers-existing.md` and a proxmox section. `docs/architecture.md`: components row, a `proxmox` block in §5, the span note in §10, a §11 row and the provider count, Decisions bullets (VMID from the UID hash with the ownership check, the mark in the description, the seed by API upload, the node resolved per call). A README row. `docs/providers.md` link. `docs/testing.md` rows and a proxmox integration section. `CLAUDE.md` Gotchas: `make test-integration-proxmox`. A `CHANGELOG.md` entry after the user confirms. **Evidence:** contract §12. ## 24. Known limits accepted at design time **Answer:** ten limits, each with its consequence and workaround: - **Create blocks** until the VM runs, holding one of three reconcile workers. On the user's host that is about five seconds plus the clone; large templates on slow storage take longer. - **A clone lock that never clears** (a node crash mid-clone) makes every Create for that Proxy wait out the clone budget and return `ErrTransient`. Clear it with `qm unlock ` or destroy the VM. - **Seed ISOs of VMs destroyed outside the operator** are invisible to GC. Remove `proxy-*-seed.iso` files without a matching VM by hand. - **The VMID range is shared with humans.** A VM created by hand inside it costs a probe; a hand-made VM named like a proxy is refused as `ErrPermanent`. Reserve the range. - **Cross-node clones need shared storage** and are untested, because the host has one node. - **A VM whose agent was switched off is replaced**, not repaired (entry 7). - **The proxy network needs internet** for the agent install, unless the template ships the agent. - **The CA is cached** after first use; rotating it needs a restart. The token rotates freely. - **A node that stops answering** makes `Get` and `Delete` for its VMs `ErrTransient` until it returns, so deleting such a Proxy waits. - Contract §13 limits that apply unchanged: config read once per process, no installation identity in GC (two operators sharing a cluster and a VMID range delete each other's VMs), an unconfigured provider latches the Proxy. **Evidence:** contract §13; entries 2, 7, 10, 11, 14 above.