diff --git a/docs/plans-executions/2026-09-18-1502-proxmox-provider.md b/docs/plans-executions/2026-09-18-1502-proxmox-provider.md index 058a1c6..ec9cc10 100644 --- a/docs/plans-executions/2026-09-18-1502-proxmox-provider.md +++ b/docs/plans-executions/2026-09-18-1502-proxmox-provider.md @@ -5,7 +5,7 @@ Pairs with [docs/plans/2026-09-18-1502-proxmox-provider.md](../plans/2026-09-18- ## Status - [x] Phase 0 — Plan file, branch, host recon, fixtures -- [ ] Phase 1 — The spec (`docs/proxmox-provider-spec.md`) +- [x] Phase 1 — The spec (`docs/proxmox-provider-spec.md`) - [ ] Phase 2 — Config block, registration, package skeleton - [ ] Phase 3 — Extract `nocloud` and `keyedlock` from kvm2 - [ ] Phase 4 — Seam, HTTP client, decorator, fake @@ -280,3 +280,52 @@ volumes of 199999. The capture is `agent_network_get_interfaces_vlan.json`. - Time to IP was about 50 s on both networks, with the agent installed at first boot. Budgets in the plan (running wait 60 s for the VM state, the agent discovered by `Get` polling) are consistent with this. + +## Phase 1 — The spec + +Wrote [docs/proxmox-provider-spec.md](../proxmox-provider-spec.md), all 24 +entries of the template, from the plan, the contract and the Phase 0 facts. +The ten Phase 0 corrections are folded in, and the plan now opens with a +note that the spec wins where the two disagree. + +Judgment calls made here, each recorded in its entry: + +- **Create's allocation loop** (entry 2) runs an ownership check on any + taken VMID: wait out a lock or a nameless entry, then adopt when the name + and UID are ours, skip when the name differs, refuse as Permanent when the + name is ours but the mark is not. This is the fix for the duplicate-VM bug. +- **Get's check order** (entry 9): lock, then name and mark from the config, + then `status/current`, then the agent. `lock: suspended` alone maps to + Stopped, since a hibernated VM never resumes by itself. +- **Agent messages** (entry 7): `QEMU guest agent is not running` is + Provisioning, `VM is not running` is Transient, and + `No QEMU guest agent configured` is **Stopped**, so the reconciler replaces + a VM that can never report an address instead of latching the Proxy. +- **Templates are checked before cloning** (entry 4): not a template, or a + Proxmox cloud-init drive in any slot, is Permanent. This makes the "no + cloud-init drive" prerequisite enforceable instead of a silent failure. +- **`isoStorage` is required**, with no default (entry 15), because the token + must hold `Datastore.Allocate` on it. +- **TLS verification failures are Permanent**, with a Checker reason + `CertificateInvalid` (entries 13, 16), since the fix is the URL or CA file. +- **The CA is loaded lazily and cached**; the token is read per request + (entry 14). + +Checks, as the plan's Verify clause asks: + +```bash +grep -c '^## [0-9]*\. ' docs/proxmox-provider-spec.md # 24 +grep -n '^\*\*Answer:\*\*$' docs/proxmox-provider-spec.md # nothing +grep -c '—' docs/proxmox-provider-spec.md # 0 +``` + +plus a script comparing every `` ``: `` `` pair against the +contract's §4.20 table (no mismatches, every axis answered) and a list of +the F numbers cited (all 23). + +Worth noting: the first pass stated two things as fact that recon could not +show and was corrected before commit. That an unreachable node's guests stay +listed as `unknown` is Proxmox's documented design, unverifiable on a single +node; entry 13 now says so. And the matcher for `VM is not running` +is a whole-message regexp, because ` is not running` alone also matches +`QEMU guest agent is not running`. diff --git a/docs/plans/2026-09-18-1502-proxmox-provider.md b/docs/plans/2026-09-18-1502-proxmox-provider.md index d7c42bd..b0e9704 100644 --- a/docs/plans/2026-09-18-1502-proxmox-provider.md +++ b/docs/plans/2026-09-18-1502-proxmox-provider.md @@ -4,6 +4,15 @@ ## Context +> **Superseded in places by the spec.** Phase 0 recon found ten corrections +> to the design below (listed in the execution log), and Phase 1 folded them +> into [docs/proxmox-provider-spec.md](../proxmox-provider-spec.md). Where a +> phase step here disagrees with the spec (the Create allocation loop, Get's +> check order, resize and seed delete as tasks, no delete before upload, +> `isoStorage` required, `vlanTag`, the agent messages), **the spec wins**. +> Test names in the phases below remain the minimum; spec entry 21 adds the +> ones for the corrections. + The provider contract (`docs/provider-contract.md`), the spec template (`docs/provider-spec-template.md`) and the kvm2 build proved the workflow: spec first, then implementation against the contract, one commit per step. diff --git a/docs/proxmox-provider-spec.md b/docs/proxmox-provider-spec.md new file mode 100644 index 0000000..8b2bd2f --- /dev/null +++ b/docs/proxmox-provider-spec.md @@ -0,0 +1,856 @@ +# Provider spec: proxmox + +Filled from [provider-spec-template.md](provider-spec-template.md) against +[provider-contract.md](provider-contract.md). This is the design the +`proxmox` package is built from. Where an entry names an axis in backticks, +the answer is one of that axis's profiles in contract §4. + +Backend facts come from the Phase 0 recon on the user's host, recorded in +[the execution log](plans-executions/2026-09-18-1502-proxmox-provider.md) as +facts F1 to F23, with the raw responses under +`internal/provider/proxmox/testdata/`. An Evidence line citing `F` points +there. Where this spec and the plan +([docs/plans/2026-09-18-1502-proxmox-provider.md](plans/2026-09-18-1502-proxmox-provider.md)) +disagree, this spec wins: it folds in the ten corrections Phase 0 found. + +## Header + +- **Type**: `proxmox` +- **Example configured name**: `proxmox-home` +- **Backend and API version this spec was written against**: Proxmox VE + 9.2.20 (`pve-manager/9.2.20/49318c671b82f31e`, kernel `7.0.14-17-pve`, + QEMU 11.0.3) on the single node `pv1-1`, API at + `https://192.168.0.8:8006/api2/json`. +- **Client**: `thin-client`: `net/http` and `encoding/json` from the standard + library, with the transport wrapped by the already-required + `go.opentelemetry.io/contrib/instrumentation/net/http/otelhttp`. No new + dependency. The seed ISO uses `github.com/kdomanski/iso9660`, already + required by kvm2. +- **Author and date**: Claude (orchestrator session), 2026-09-23. + +## 1. Identity and providerID + +**Answer:** `vms//`, where `` is the VM's numeric id and +`` is `req.Name`. The ID is location-free: the node is looked up in +`GET /cluster/resources?type=vm` on every call, so a VM migrated between +nodes stays addressable. Parsing is strict: exactly three segments, the +literal `vms`, a decimal VMID without leading zeros between 100 and +999999999, and a non-empty name without `/`. Anything else is `ErrPermanent` +from `Get` and `Delete`. The name is carried so that `Get` and `Delete` can +tell our VM from another VM that later received the same VMID (entry 2). No +config value is pasted into the ID, so no ID-safety regexp is needed; node +names and storage ids are still validated at load (entry 15) because they +end up in request paths. + +**Evidence:** contract §6 and §4.10 (the ID must not depend on the location +when instances can migrate); F8 (`node` is a field of every resource entry). + +## 2. Primary key (`primary-key`) + +**Answer:** `allocated-id`. Proxmox addresses VMs by VMID; `name` is only a +label and is not unique. The VMID is derived from the Proxy UID so a retry +lands on the same number: `vmidFor(uid, attempt)` is +`min + (fnv1a64(uid) + attempt) mod (max - min + 1)` over `vmidRange` +(entry 15). Create, under the per-name lock (entry 19): + +1. Reads `/cluster/resources` once. Entries with `type: qemu`, `template: 0` + and `name == req.Name` are the by-name candidates. Two or more is + `ErrPermanent` naming every VMID: an admin must remove the duplicate. + Exactly one goes to the ownership check below and is adopted or refused. +2. With none, probes `attempt = 0, 1, …` up to 32: + - VMID free in the listing: clone onto it. If the clone fails with + `unable to create VM : config file already exists` (F4), someone + took it between the listing and the clone; run the ownership check on + it. + - VMID taken: run the ownership check. + - Exhausted: `ErrTransient` naming the range. +3. The **ownership check** on a VMID: if its resource entry carries a + `lock` or has no `name` (a clone in progress lists neither, F8), wait + until it is unlocked (entry 10, clone budget). Then read its config. It + is **ours** when `name == req.Name` and the description carries the mark + with `uid` equal to `req.UID` or empty: adopt it and resume Create at the + seed step (entry 10). It is **foreign** when the name differs: move to + the next attempt. The same name with no mark, or with a different + non-empty `uid`, is `ErrPermanent` naming the VMID and both UIDs; the + operator never takes over a VM it did not create. + +The wait in step 3 is what closes the duplicate-VM bug Phase 0 found: after +a crash mid-clone, the retry's by-name lookup misses the nameless VM, but +attempt 0 lands on the same VMID, waits for the clone to finish, and adopts +it instead of probing past it. + +`Get` and `Delete` check the name and mark at the VMID in the ID. A VM there +with another name means ours is gone and the VMID was reused: `Get` answers +`ErrNotFound`, `Delete` succeeds without touching it. + +**Evidence:** F4 (taken VMID fails synchronously); F8 (a mid-clone entry has +no name); contract §4.10 (allocate-then-verify, deterministic from the UID) +and §13 (adoption not verified: closed here). + +## 3. Placement mapping + +**Answer:** three fields are catalogue keys checked before any backend call, +one is ignored: + +| Field | Meaning | Required | Unknown value | +|---|---|---|---| +| `Zone` | a node from `nodes` | only when more than one node is configured; with one node an empty zone selects it | `ErrPermanent`: `placement.zone "x" is not in proxmox.nodes` | +| `MachineType` | a key of `machineTypes` (`cores`, `memoryMiB`, optional `diskGiB`) | yes | `ErrPermanent`: `placement.machineType "x" is not in proxmox.machineTypes` | +| `Image` | a key of `templates` (a template VMID) | yes | `ErrPermanent`: `placement.image "x" is not in proxmox.templates` | +| `Region` | ignored | | still part of the spec hash: changing it replaces the VM | + +The guide documents the ignored field and that catalogue edits affect only +new VMs. + +**Evidence:** contract §2.1 (reject early, MAY ignore with documentation). + +## 4. Image source + +**Answer:** a template VM prepared by the admin, referenced by a catalogue +key: `templates..vmid`. Create makes a **full clone** (`full=1`), so a +VM owns copies of its disks and keeps nothing linked to the template: the +template may be edited, replaced or deleted without affecting running VMs. +Before cloning, Create reads the template's config and refuses it with +`ErrPermanent` when it is not a template (`template` is not 1) or when any +disk slot holds a Proxmox cloud-init drive (a value containing `cloudinit`), +which would compete with the provider's seed (entry 5). The template must +boot from its first disk, run cloud-init with the NoCloud datasource, and +either ship `qemu-guest-agent` or be able to install it (entry 7). + +Clone parameters: `newid`, `name`, `description` (the mark, entry 8), +`full=1`, `storage` when configured (empty clones onto the template's own +disk storage), and `target` when the chosen node differs from the template's +node. A cross-node clone needs the template on shared storage; this host has +one node, so that path is untested (entry 24). + +Catalogue edit warning for the guide: changing a key's `vmid` changes only +VMs created afterwards. Add a new key rather than repointing one in use, so +the Proxies that name it keep a meaning you can reason about. + +**Evidence:** F19 (template 9001: `scsi0`, `boot: order=scsi0`, `ide2` +free, `agent: 1`; template 9000 carries a cloud-init drive on `ide2`); F15 +(a clone onto a storage without image support is refused up front); F16 (a +full clone of 9001 took about two seconds). + +## 5. Cloud-init delivery + +**Answer:** a NoCloud seed ISO built in-process by the shared +`internal/provider/nocloud` package (extracted from kvm2 in Phase 3): +volume label `cidata`, files `user-data` (the Proxy's `CloudInit`, byte for +byte, never parsed), `meta-data` (`instance-id: ` and +`local-hostname: `) and `vendor-data` (the guest agent script, joined +with the configured `vendorData` as a MIME multipart when both are set; +identical to kvm2's rules). The builder stages under `os.TempDir()` (`/tmp`, +the manager's emptyDir) and writes into a `bytes.Buffer`; a seed is tens of +KiB, so buffering is within contract §1.4. + +The ISO is uploaded with `POST /nodes/{node}/storage/{isoStorage}/upload` +as multipart (`content=iso`, file part `filename` named +`-seed.iso`) and the returned `imgcopy` task is waited on. An upload +**overwrites** an existing file of the same name, so no delete precedes it; +the seed is rebuilt and re-uploaded on every Create attempt, since a +previous attempt may have been cut off. It is attached as +`ide2: :iso/-seed.iso,media=cdrom`; Proxmox then appends +`ide2` to the boot order on its own, which is harmless because the boot +disk stays first. + +Only the Proxmox API is used: the storage upload endpoint accepts `iso`, +`vztmpl` or `import`, never `snippets`, so `cicustom` would need shell +access to the node. Size: the request timeout (entry 18) bounds the upload; +a seed with a multi-megabyte `vendorData` would need a larger +`requestTimeoutSeconds`. + +**Evidence:** F12 (upload fields, UPID, overwrite, `.iso` suffix required); +F19 (`ide2` appended to the boot order); F21 (the seed on `ide2` installed +the agent on both the untagged network and VLAN 4); F23 (the upload is +staged in `/var/tmp` on the node, not the tmpfs `/tmp`); contract §2.1 and +§11 (byte for byte, sentinel test). + +## 6. Machine type mapping + +**Answer:** a catalogue: `machineTypes.` is `{cores, memoryMiB, +diskGiB}`. `cores` and `memoryMiB` are written in the config step (entry +10). `diskGiB` is optional: when set and larger than the cloned boot disk, +the disk is grown with `PUT /nodes/{node}/qemu/{vmid}/resize` to the +absolute size `G`, and the returned task is waited on; when unset +or not larger, the disk keeps the template's size and no call is made. The +provider never shrinks a disk. Cloud-init's growpart expands the root +filesystem at first boot. + +The boot disk is the first entry of the config's `boot: order=…` that is a +disk key (`scsiN`, `virtioN`, `sataN`, `ideN`) and not a CD-ROM, else the +`bootdisk` key, else the first present of `scsi0`, `virtio0`, `sata0`, +`ide0`. Its current size comes from the `size=` option of that disk's value. +No boot disk, when `diskGiB` is set, is `ErrPermanent`. + +**Evidence:** F20 (resize is a task; the same size is an `OK` no-op; a +shrink fails in the task with `shrinking disks is not supported`); F19 +(`boot: order=scsi0`, becoming `order=scsi0;ide2` once the seed is +attached). + +## 7. Network and IP discovery (`readiness`, `ip-source`) + +**Answer:** `readiness`: `provisioning-window`. `ip-source`: `discovered`, +from the QEMU guest agent only (`GET /nodes/{node}/qemu/{vmid}/agent/network-get-interfaces`); +the API offers no ARP or DHCP-lease source. The address reported is the first +IPv4 of the first non-loopback interface, in the agent's order, that is not +loopback, link-local or unspecified. + +The NIC is the template's `net0`, rewritten in the config step only when the +config sets `bridge` or `vlanTag`: the value's comma-separated options are +kept in order with the model and MAC untouched, `bridge=` and `tag=` are +replaced or appended. With neither set, `net0` is left as cloned. + +The agent answers: + +| Agent response | Meaning | Get reports | +|---|---|---| +| 200 with a routable IPv4 | booted | Running with that IP | +| 200 without one | network still coming up | Provisioning | +| 500 `QEMU guest agent is not running` | booting, or the agent install has not finished or failed | Provisioning | +| 500 `VM is not running` | the VM stopped between the status read and this call | `ErrTransient` | +| 500 `No QEMU guest agent configured` | `agent: 1` was removed from the config; this VM can never report an address | Stopped, so the reconciler replaces it with a VM that has the agent on | +| anything else | cannot tell | `ErrTransient` | + +The guest needs the agent. The vendor-data script installs it at first boot +unless `installGuestAgent: false`, and that install needs a package mirror: +**the proxy network must have a route to the internet**, which an egress +proxy needs anyway. Baking `qemu-guest-agent` into the template and setting +`installGuestAgent: false` removes the mirror from IP discovery. `Port` is +not used by the provider. Reachability is a prerequisite in the guide: the +operator's pod network and the clients must route to the proxy network (on +the user's host, VLAN 4, `192.168.4.0/24`). + +**Evidence:** F5 (all three agent messages); F21 (agent response shape; an +address about 50 seconds after start on the untagged network and on VLAN 4); +the Phase 0 log's VLAN 4 section (no internet route meant no agent); +contract §1.3 and §2.2 (Running only with an IP). + +## 8. GC marks and CreatedAt (`gc-mark`, `created-at`) + +**Answer:** `gc-mark`: `written-metadata`. `created-at`: `written-at-create`. +The mark is three lines in the VM's `description`, passed in the clone call +itself, so there is no mark window: + +``` +proxy-operator-managed=true +proxy-operator-uid= +proxy-operator-created-at= +``` + +The keys are `provider.LabelManaged`, `provider.LabelUID` and +`proxy-operator-created-at`; the managed value is `provider.LabelManagedYes`. +The parser reads `key=value` lines, tolerates CRLF, blank lines and any +other lines an admin adds, and treats the VM as managed only when the +managed value is exactly `true`. A missing `uid` line gives an empty UID. A +`created-at` that does not parse gives the zero time, which GC treats as +old; that is accepted. The clone's own `meta: …,ctime=…` is not usable: +Proxmox copies it from the template. + +The config step also sets the tag `proxy_operator`, for filtering in the +Proxmox UI and as a ListByTag candidate hint (entry 12). It is not the mark: +tags cannot be passed to the clone call. Hyphens are accepted in tags on +9.2.20, but the underscore form is kept because it is valid under every +tag rule Proxmox has had. Uppercase is folded to lowercase by Proxmox. + +The seed ISO carries no mark; its name `-seed.iso` identifies it +(entry 11). + +**Evidence:** F10 (the description survives the clone verbatim, newlines +included); F11 (the clone's `meta` is the template's); F9 (tag rules); +contract §2.1 (mark in the create call) and §4.5. + +## 9. State mapping (`provisioning-state`) + +**Answer:** `provisioning-state`: `create-blocks`. Create returns only once +`status/current` reports the VM running (entry 10), so a VM the provider +cloned but never started is only ever seen by a Create retry, which adopts +and starts it; the reconciler has no ID for it to `Get`. + +`Get` checks in this order; the first match decides: + +| Check | Source | Result | +|---|---|---| +| VMID absent | `/cluster/resources` | `ErrNotFound` | +| `lock` set, other than `suspended` | resources entry (a clone in progress also has no name) | `ErrTransient` | +| `lock: suspended` (hibernated to disk) | resources entry | Stopped | +| `status: unknown` (node not reachable) | resources entry | `ErrTransient` | +| config `does not exist` | `GET …/config` | `ErrNotFound` (deleted between the two calls) | +| config `lock` set | config | `ErrTransient` | +| `name` differs from the ID's name | config | `ErrNotFound`, logged: the VMID was reused | +| no mark in the description | config | `ErrNotFound`, logged: not ours | +| `status: stopped` | `status/current` | Stopped | +| `status: running`, `qmpstatus: running` | `status/current` | the agent decides (entry 7) | +| `status: running`, `qmpstatus: prelaunch` | `status/current` | Provisioning | +| `status: running`, any other `qmpstatus` (`paused`, `suspended`, `io-error`, …) | `status/current` | Stopped | +| any other `status` | `status/current` | Stopped | + +`status/current` is read, not only the resources entry, because a paused VM +shows as plain `running` in `/cluster/resources`. `UID` and `CreatedAt` come +from the description on every call. A running `Get` makes four calls: +resources, config, status and agent. + +**Evidence:** F8 (resources fields; mid-clone entries lack a name, carry +`lock: clone` and `status: unknown`); F14 (stopped and paused as seen by +`status/current` and by resources); F2 (the `does not exist` message); +contract §2.2 and §4.11. + +## 10. Create: blocking and task tracking (`create-return`, `async-ops`) + +**Answer:** `create-return`: `blocking`. `async-ops`: `task-handle` and +`backend-lock`. Clone, upload, resize, start, stop, destroy and volume +delete each return a UPID; the provider polls +`GET /nodes/{node}/tasks/{upid}/status` every second until `status` is +`stopped`, then reads `exitstatus`: `OK` is success, anything else is +`ErrTransient` carrying that text. The node comes from the UPID +(`UPID::…`). The handle is held within the call only: Create blocks +until the VM runs, so no handle outlives it, and a failure surfaces as +Create's own error with the task's message. Locks: a VM locked by a task is +waited on in Create (the ownership check) and reported as `ErrTransient` by +`Get` and `Delete`. A start issued while a clone holds the lock queues +behind it rather than failing; the provider never relies on that, because it +waits for the clone task first. + +Steps, each idempotent, in order, under the per-name lock (entry 19): + +1. Validate placement (entry 3). No backend call yet. +2. `GET /cluster/resources`. Find the template's node; a missing template + is `ErrPermanent`. Read the template's config and refuse it as entry 4 + says. +3. Find or allocate the VMID (entry 2). An adopted VM that is running + returns its ID at once. An adopted VM that is stopped continues at step 5. +4. Clone and wait (clone budget). +5. Build the seed and upload it to the VM's node, overwriting; wait + (upload budget). +6. `PUT /nodes/{node}/qemu/{vmid}/config` (synchronous) with `cores`, + `memory`, `agent=1`, `tags=proxy_operator`, + `ide2=:iso/-seed.iso,media=cdrom` and, when configured, + the rewritten `net0` (entry 7). +7. Resize when `diskGiB` is larger than the boot disk (entry 6); wait + (resize budget). +8. `POST …/status/start`; wait (start budget). +9. Poll `status/current` every second until `status: running` and + `qmpstatus: running`, up to the running budget; expiry is + `ErrTransient`, and so is a `stopped` status after the start. +10. Return `vms//`. + +What a retry finds: + +| Previous attempt stopped after | The retry | +|---|---| +| nothing | allocates, clones | +| submitting the clone | finds the VMID locked and nameless, waits, adopts, continues at 5 | +| the clone | adopts the stopped VM, continues at 5 | +| any of steps 5 to 7 | adopts, repeats 5 to 7 (upload overwrites, config is a replace, resize is skipped once large enough) | +| the start | adopts the running VM, returns its ID | + +A Proxmox `does not exist` inside Create (a VM or volume vanishing between +steps) is `ErrTransient`, never `ErrNotFound`. The log line for a Create +records `cloudInitBytes`, never the content. + +**Evidence:** F3 (a start queues behind the clone lock); F7 (a config write +under the clone lock is refused with `VM is locked (clone)`); F16 (task +status fields); F20 (resize is a task); the Phase 0 log's VLAN 4 rerun +(clone to started VM in about five seconds); contract §2.1, §4.2, §4.12 and +§5 ("do not return NotFound from Create"). + +## 11. Delete and secondary resources (`delete-return`, `secondary-resources`) + +**Answer:** `delete-return`: `waits`. `secondary-resources`: `explicit` for +the seed ISO; the VM's disks are removed by the destroy call itself, with +`purge=1&destroy-unreferenced-disks=1`. Under the per-name lock: + +1. Parse the ID (`ErrPermanent` if malformed). +2. `/cluster/resources`. VMID absent: skip to step 7, trying every configured + node because the VM's node is no longer known. +3. `lock` set or `status: unknown`: `ErrTransient`. A clone or another task + still running finishes first; the reconciler calls Delete again after + `DeletionPoll`. +4. Read the config. `does not exist`: skip to step 7. A different name, or + no mark: the VM is not ours; log it and skip to step 7 without touching + it. +5. Status other than `stopped`: `POST …/status/stop`, wait (stop budget). + Stop works on a paused VM too. +6. `DELETE /nodes/{node}/qemu/{vmid}?purge=1&destroy-unreferenced-disks=1`, + wait (destroy budget). `does not exist` is success. +7. `DELETE /nodes/{node}/storage/{isoStorage}/content/:iso/-seed.iso`, + wait (seed-delete budget). A missing volume is still a successful task, + so no message is matched. On the try-every-node path a node whose + request fails is logged and skipped. +8. Return nil. + +Everything is gone when Delete returns nil, so `Get` afterwards is +`ErrNotFound`. A concurrent `Get` during a Delete keeps seeing the VM (as +Stopped once it is stopped) until the destroy task removes its config, +because destroying sets no lock and leaves the resource entry in place until +the end. A second Delete waits on the lock, then finds nothing. + +GC blind spot: a VM destroyed outside the operator leaves its seed ISO, which +ListByTag cannot see. The guide says seeds are named `proxy-*-seed.iso` on +`isoStorage` and any without a matching VM may be removed. + +**Evidence:** F2 (a second destroy gives `does not exist`); F13 (volume +delete is an `imgdel` task, needs `Datastore.Allocate`, and succeeds on a +missing volume); the Phase 0 capture of resources and config mid-destroy +(`cluster_resources_during_destroy.json`, `config_during_destroy.json`); +contract §2.3. + +## 12. ListByTag scope and partial failure (`list-partial`) + +**Answer:** `fail-whole` for the one listing, with single unreadable VMs +skipped. Scope: every VM the cluster knows, from one +`GET /cluster/resources?type=vm`, which covers every node, including +unreachable ones, whose guests Proxmox lists with `status: unknown` (entry +13). Candidates are entries +with `type: qemu`, `template: 0`, and either the tag `proxy_operator` (tags +split on `;`, `,` or space) or a name starting with `proxy-`. A nameless +entry (a clone in progress) is not a candidate: its Create is live and it is +younger than `MinAge`. For each candidate the config is read and the +description parsed; managed VMs become an `Instance` with `ID`, `UID` and +`CreatedAt` only, which is all GC reads (contract §2.4); no status or agent +call is made. A candidate whose config cannot be read (a node down, a VM +destroyed mid-sweep) is skipped with a V(1) line, unless ctx ended, which +fails the call. The resources call failing fails the whole call. Filtering +by the mark is client-side; the tag narrows the candidates. Cost: one +resources call plus one config read per candidate per sweep. + +**Evidence:** F8 (resources fields, tags separated by `;`, nameless mid-clone +entries); F9 (tags case-folded); contract §2.4 and §4.7. + +## 13. Error classification (`not-found`, `quota`) + +**Answer:** `not-found`: `error-code`, the 500 message +`Configuration file 'nodes//qemu-server/.conf' does not exist`, +plus `list-and-miss` on `/cluster/resources`. The listing can miss a VM only +when it is gone: it is built from the cluster filesystem, which holds every +guest's config cluster-wide, and Proxmox lists the guests of an unreachable +node with `status: unknown` rather than dropping them. That last point is +Proxmox's documented design, not something the single-node host could show; +Phase 10 or a second node verifies it. `quota`: `absent`. The thin `local-lvm` storage +overprovisions, so no out-of-space response could be produced; capacity +failures are `ErrTransient`, and the Checker reports free space in +`Details` instead (entry 16). + +The client carries each failure as `apiError{Status, Message, Errors}`: +`Message` is the body's `message` with the trailing newline trimmed, or the +reason phrase when the body has none (a 401); `Errors` is the 400 body's +per-parameter map. One `classify(op string, err error)`: + +| Backend response | Class | Why | +|---|---|---| +| 401 `Authentication failed!`, 401 `No ticket` | Permanent | the token or its secret is wrong; fixed in the Secret | +| 403 `Permission check failed (, )` | Permanent | the token lacks a privilege; the message names it | +| 400 `Parameter verification failed.` with `errors` | Permanent | the request is wrong; the message includes the field errors | +| other 4xx | Permanent | a request Proxmox will never accept | +| 500 `Configuration file '…' does not exist` | NotFound (Transient when `op == "create"`) | the VM is gone | +| 500 `VM is locked ()` | Transient | a task holds the VM | +| 500 `unable to create VM : config file already exists` | handled by the allocator (entry 2); Transient if it escapes | the VMID is taken | +| 500 `QEMU guest agent is not running`, `No QEMU guest agent configured`, `VM is not running` | handled by `Get` (entry 7); Transient if they escape | agent state | +| any other 5xx, including pveproxy's 595 and 596 | Transient | unknown means retry | +| task `exitstatus` other than `OK` | Transient, with the text | the task failed; the next attempt retries | +| TLS verification failure (`x509.UnknownAuthorityError`, `x509.HostnameError`, `x509.CertificateInvalidError`) | Permanent | wrong URL or CA file; the admin must act | +| token file missing (`fs.ErrNotExist`) | Transient | the Secret has not landed yet | +| token file present but not `@!=` | Permanent | wrong content in the Secret | +| `context.Canceled`, `context.DeadlineExceeded`, `net.OpError`, `io.EOF`, `io.ErrUnexpectedEOF` | Transient | network or cancellation | +| an error already wrapped with a class | that class | validation and builders classify at the source | +| anything else | Transient | contract default | + +Every matched string is backed by a captured response under `testdata/`; +the table test loads them rather than repeating the text. Matching is on +fixed substrings anchored to the 500 status: `does not exist`, +`VM is locked`, `config file already exists`, `QEMU guest agent is not +running`, `No QEMU guest agent configured`, and for the stopped VM the whole +message against `^VM [0-9]+ is not running$`. The agent messages are tested +in that order, because the last is a suffix of the first. + +**Evidence:** F1 (error text in the reason phrase and the body; 401 has no +body); F2, F4, F5, F7 (the 500 messages); F6 (no out-of-space response +could be produced on thin storage); F22 (both 401 texts); fixtures +`err_forbidden.json`, `err_bad_param.json`, `err_tag_invalid.json`, +`err_upload_bad_name.json`; the Phase 0 TLS finding (`192.168.2.8` not in +the node certificate); contract §5. + +## 14. Credentials and TLS (`credentials`) + +**Answer:** `mounted-files`. The config names `tokenFile`, one line +`@!=`, sent as +`Authorization: PVEAPIToken=`. It is expected in the optional +Secret `proxmox-token`, key `token`, mounted at +`/etc/proxy-operator/proxmox`. The file is read on **every request**, so +rotating the Secret takes effect without a restart. `New` stats it and logs +a warning when it is missing, but never fails on it and never dials. + +TLS: `ca-file`. `caFile` names a PEM bundle, typically the node's +`/etc/pve/pve-root-ca.pem`, carried in the same Secret as key `ca.pem`. It +is loaded on first use and cached, so rotating the CA needs a restart; a +missing CA file is `ErrTransient` until it lands, one that does not parse is +`ErrPermanent`. Without `caFile` the system roots are used. The URL's host +must be a name or address in the node certificate's SANs: on the user's +host that is `192.168.0.8` or `pv1-1.home.lab`, not `192.168.2.8`. +`insecureSkipTLSVerify: true` disables verification and is logged as a +warning by `New`. + +Minimum privileges, as verified on 9.2.20, for `proxy-operator@pve` with a +token without privilege separation: + +- a role on `/` with `VM.Allocate VM.Clone VM.Audit VM.Config.CDROM + VM.Config.CPU VM.Config.Memory VM.Config.Disk VM.Config.Network + VM.Config.Options VM.PowerMgmt VM.GuestAgent.Audit Datastore.Audit + Datastore.AllocateSpace Datastore.AllocateTemplate Sys.Audit SDN.Use`; +- a role on `/storage/` with `Datastore.Allocate + Datastore.AllocateTemplate Datastore.AllocateSpace Datastore.Audit`, + because deleting a volume needs `Datastore.Allocate` and a grant on a more + specific path replaces the inherited one. A dedicated ISO-only storage + keeps that privilege away from backups. + +`VM.Monitor` no longer exists in PVE 9 and is not needed. +`VM.Config.Cloudinit` is not needed. `SDN.Use` on `/` covers a VLAN tag on +`vmbr0`. + +Hygiene: the token never appears in logs, span attributes or error strings. +`apiError.Error()` carries status and message only, never headers or URLs. +Tested with a sentinel secret that a test server echoes in a 401 body. + +**Evidence:** F17 (privileges exercised and the one missing); the Phase 0 +log's seed storage section; F22 (auth failure texts); contract §7. + +## 15. Config block + +**Answer:** a new block `proxmox:` with the Go type `provider.ProxmoxConfig` +in `internal/provider/config_proxmox.go`: + +| Field | Required | Default | Validation | Ends up in | +|---|---|---|---|---| +| `url` | yes | | `https` URL with a host and no path other than `/` | request base, `…/api2/json` appended | +| `tokenFile` | yes | | non-empty | `Authorization` header | +| `caFile` | | system roots | non-empty when set | TLS roots | +| `insecureSkipTLSVerify` | | false | | TLS, warning at `New` | +| `nodes` | yes, at least one | | unique, `^[a-z0-9]([a-z0-9-]*[a-z0-9])?$` | request paths, clone `target` | +| `isoStorage` | yes | | Proxmox storage id `^[a-zA-Z][a-zA-Z0-9_.-]*$` | upload and delete paths, `ide2` | +| `storage` | | the template's disk storage | storage id regexp when set | clone `storage` | +| `bridge` | | as cloned | non-empty when set | `net0` `bridge=` | +| `vlanTag` | | as cloned | 1 to 4094 when set | `net0` `tag=` | +| `vmidRange.min`, `vmidRange.max` | | 100000, 199999 | min at least 100, max at most 999999999, min below max | `vmidFor` | +| `templates..vmid` | at least one template | | at least 100 | clone source | +| `machineTypes.` | at least one | | `cores` and `memoryMiB` above zero, `diskGiB` not negative | config step, resize | +| `requestTimeoutSeconds` | | 30 | not negative | HTTP client timeout | +| `vendorData` | | | | seed `vendor-data` | +| `installGuestAgent` | | true | | seed `vendor-data` | + +`isoStorage` has no default on purpose: the token must be allowed to delete +volumes there (entry 14), so the choice must be explicit. `nodes` holds node +names as Proxmox knows them (`pv1-1`), not addresses; the one URL reaches the +whole cluster. + +Wiring: `ProviderConfig.Proxmox *ProxmoxConfig` with tag +`proxmox,omitempty`; `setBlocks` reports `proxmox`; the allow-list in +`Config.validate` and the constructor map in `cmd/main.go` both gain +`proxmox` in the same commit; `validateProxmox(prefix, c)` in +`config_proxmox.go` with `config_proxmox_test.go`. + +**Evidence:** contract §10; the Phase 0 decisions (dedicated seed storage, +`vlanTag`, URL at `192.168.0.8`); F13 (why `isoStorage` needs +`Datastore.Allocate`). + +## 16. Checker + +**Answer:** one target per `nodes` entry, `Name` the node, `Address` the +URL, checked concurrently, returned in config order. The check shares the +provider's HTTP client; each request is independent, so a check that runs +out of time cancels only its own requests. Per node: + +1. `GET /version` (`pveVersion`). +2. `GET /nodes/{node}/status` (the node answers; `kernel`). +3. `GET /nodes/{node}/storage/{isoStorage}/status`: the storage exists, is + active, and lists `iso` in its content (`isoStorageAvailGiB`). +4. `GET /cluster/resources?type=vm`: every configured template exists and + is a template (`templates` as `/`). + +Reasons, shared first: `Reachable`; `Timeout` (the round's ctx ended, +whichever request was running); `Unreachable` (TCP connect or TLS handshake +failed for a network reason, or pveproxy answered 595 or 596); +`Unauthorized` (401); `Forbidden` (403, the message names the privilege); +`APIError` (any other failure). Provider-specific, each fixed differently by +an admin: + +- `CertificateInvalid`: TLS verification failed. Fix `url` or `caFile`, not + the host. +- `CredentialsUnavailable`: the token file is missing or malformed. Fix the + Secret, not Proxmox. +- `StorageNotFound`: `isoStorage` is missing, inactive, or has no `iso` + content. Fix the storage on the node. +- `TemplateNotFound`: a configured template VMID does not exist or is not a + template. Fix the template or the catalogue. + +Details: `pveVersion`, `kernel`, `tokenID` (the part before `=`, never the +secret), `isoStorage`, `isoStorageAvailGiB`, `templates`, filled as far as +the check got. Cost: four requests per node per round. The `status.type` +comment in `api/v1alpha1/providerhealth_types.go` gains `proxmox` and the +CRD is regenerated. + +**Evidence:** contract §8; fixtures `version.json`, `node_status.json`, +`storage_status.json`, `cluster_resources.json`. + +## 17. Client and child spans (`client-kind`, `child-spans`) + +**Answer:** `client-kind`: `thin-client`. `child-spans`: `transport-wrapper`. + +The seam is the interface `api`, one method per endpoint, `ctx` first: + +| Method | HTTP | Returns | +|---|---|---| +| `Version` | `GET /version` | version info | +| `NodeStatus` | `GET /nodes/{node}/status` | node status | +| `StorageStatus` | `GET /nodes/{node}/storage/{storage}/status` | storage status | +| `ClusterResources` | `GET /cluster/resources?type=vm` | entries | +| `VMConfig` | `GET /nodes/{node}/qemu/{vmid}/config` | config as `map[string]string` | +| `SetVMConfig` | `PUT /nodes/{node}/qemu/{vmid}/config` | nothing (synchronous) | +| `Clone` | `POST /nodes/{node}/qemu/{vmid}/clone` | UPID | +| `Resize` | `PUT /nodes/{node}/qemu/{vmid}/resize` | UPID | +| `Start` | `POST /nodes/{node}/qemu/{vmid}/status/start` | UPID | +| `Stop` | `POST /nodes/{node}/qemu/{vmid}/status/stop` | UPID | +| `VMStatus` | `GET /nodes/{node}/qemu/{vmid}/status/current` | status and qmpstatus | +| `Destroy` | `DELETE /nodes/{node}/qemu/{vmid}?purge=1&destroy-unreferenced-disks=1` | UPID | +| `TaskStatus` | `GET /nodes/{node}/tasks/{upid}/status` | status and exitstatus | +| `AgentNetworkInterfaces` | `GET /nodes/{node}/qemu/{vmid}/agent/network-get-interfaces` | interfaces | +| `UploadISO` | `POST /nodes/{node}/storage/{storage}/upload` (multipart) | UPID | +| `DeleteVolume` | `DELETE /nodes/{node}/storage/{storage}/content/{volid}` | UPID | + +`httpAPI` is the production adapter; `fakeAPI` in `fake_test.go` keeps VMs, +tasks and ISOs in memory, simulates locks (a clone leaves a nameless locked +entry until its task completes), task polls, per-method errors and blocking, +and records a call log. Scalars in a config are rendered as strings by the +adapter (`cores` arrives as a JSON number, `memory` as a string). + +The transport is `tracing.RestConfigWrapper(opts...)(base)`, a parent-gated +`otelhttp` transport, so each request is an `HTTP ` span under +`provider.create`, `provider.get`, `provider.delete`, `provider.list` or the +monitor's `providerhealth.check`, and background traffic makes no root +spans. The `observed` decorator over `api` logs one line per call and starts +no span, so nothing is counted twice. Span attributes are otelhttp's +standard ones; the token is in a header, which otelhttp does not record. +Tests: a span recorder asserts the HTTP spans are children of the operation +span and carry no token; a reflection test asserts every `api` method logs +exactly one line. + +**Evidence:** contract §4.15, §4.18, §9; `RestConfigWrapper` and +`parentGatedTransport` in `internal/tracing/transport.go`; F18 (HTTP/1.1 +only). + +## 18. Timeouts (`timeouts`) + +**Answer:** `per-call`. + +| Bound | Budget | +|---|---| +| TCP dial | 10 s | +| TLS handshake | 10 s | +| one request, response included (`http.Client.Timeout`) | `requestTimeoutSeconds`, default 30 s | +| clone task, and the wait for a VM's lock during the ownership check | 10 m | +| upload task | 2 m | +| resize task | 2 m | +| start task | 2 m | +| stop task | 2 m | +| destroy task | 5 m | +| seed delete task | 1 m | +| wait for `running` after the start | 60 s | +| task and status poll interval | 1 s | + +The budgets live in one `budgets` struct that tests shrink. A task wait that +runs out is `ErrTransient` naming the UPID, and the task keeps running on +Proxmox; the next attempt finds its result through the ownership check or +the lock. A cancelled `ctx` cancels only that request: the transport keeps +its idle connections (`MaxIdleConnsPerHost` 4, `IdleConnTimeout` 90 s), and +no other call is disturbed. HTTP/2 is disabled explicitly +(`ForceAttemptHTTP2: false`, an empty `TLSNextProto`). + +**Evidence:** contract §1.5 and §4.16; F16 (a full clone of the 3 GiB +template took about two seconds; tasks finish within a second or two); F21 +(about 50 s from start to an address, which the reconciler's polling covers, +not the running wait). + +## 19. Concurrency + +**Answer:** shared state is the HTTP client (safe for concurrent use), the +lazily loaded CA pool (loaded once under a `sync.Once`-style guard), and a +lock per instance name from the shared `internal/provider/keyedlock` package +(extracted from kvm2 in Phase 3): a one-slot semaphore per key, waitable +with ctx, removed from the map once unused. Create and Delete for one name +run under that name's lock, so the reconciler's Delete and GC's Delete never +race on stop and destroy, and a Delete cannot interleave with a Create's +steps. Get and ListByTag take no lock. Two Creates for different Proxies +that land on the same VMID are safe: one clone wins, the other gets the +`config file already exists` answer, waits in the ownership check for the +winner's clone to finish, sees a different name, and probes on. Three +reconciles, a GC sweep and a check round at once are safe. + +**Evidence:** contract §1.2 (concurrency); F4; entry 2. + +## 20. Logging (`logging`) + +**Answer:** `tiered`. Logger from `logf.FromContext(ctx)` on every call, +named `proxmox`, with `provider` on every line and `vmid`, `name` and +`node` on every line about an instance. + +- V(1): one line per operation outcome (`proxmox vm ensured running`, + `proxmox vm fetched` with state, ip and uid, `proxmox vm deleted`, + `proxmox vms listed` with counts, `proxmox node checked` with the reason), + one `proxmox api call` line per seam call from the decorator (`call`, + `node`, `vmid`, `storage`, `upid`, `duration`, and on failure + `httpStatus` and `error`), a `proxmox task failed` line with the UPID and + `exitstatus`, and the no-op lines of Delete and ListByTag (foreign VM + left alone, unreadable candidate skipped). +- V(2): request parameters (without `description`'s content at V(1) and + never the upload body), config summaries, agent addresses, the resolved + Create parameters with `cloudInitBytes`. +- No V(5) wire tier. +- Never logged at any level: `CloudInit`, the token, the CA content, the + upload body. + +**Evidence:** contract §4.17 and §9. + +## 21. Tests (`backend-test`) + +**Answer:** `backend-test`: `integration-tag`. Unit tests run over `fakeAPI` +behind the seam, the HTTP client over `httptest.NewTLSServer` serving the +captured fixtures, and `classify` over the fixtures directly. The +integration test is `proxmox_integration_test.go` behind +`//go:build integration`, skipped unless `PROXMOX_TEST_URL`, +`PROXMOX_TEST_TOKEN_FILE`, `PROXMOX_TEST_NODE`, +`PROXMOX_TEST_TEMPLATE_VMID` and `PROXMOX_TEST_ISO_STORAGE` are set; +optional `PROXMOX_TEST_CA_FILE`, `PROXMOX_TEST_INSECURE`, +`PROXMOX_TEST_STORAGE`, `PROXMOX_TEST_BRIDGE`, `PROXMOX_TEST_VLAN_TAG`, +`PROXMOX_TEST_VMID_MIN`, `PROXMOX_TEST_VMID_MAX`, `PROXMOX_TEST_TIMEOUT` +(default 20m), `PROXMOX_TEST_KEEP`, `PROXMOX_TEST_LOG_LEVEL`; run with +`make test-integration-proxmox`. It asserts, in order: `New` returns without +a request; `Check` reports the node Ready with details; `Create` returns +`vms//` with the VMID in range; a second `Create` returns the +same ID and the VM count is unchanged; `Get` reaches Running with an IPv4 +within the budget and its `UID` and `CreatedAt` equal what Create wrote; +`ListByTag` contains it; `Delete` returns nil; `Get` is `ErrNotFound`; +`Delete` again returns nil; no VM with the name and no `-seed.iso` +remain; an HTTP span is a child of `provider.create` and no span attribute +carries the cloud-init text. + +Contract §11 list, with the planned test for each item (Phase 8 ticks them +once they exist): + +- [ ] repeat `Create`: `TestCreate_repeatReturnsSameIDAndCreatesNothing` +- [ ] `Get` after `Delete`: `TestGet_afterDeleteReturnsNotFound` +- [ ] `Delete` twice: `TestDelete_twiceSucceeds` +- [ ] malformed ID: `TestParseProviderID_malformed`, `TestGet_malformedIDIsPermanent`, `TestDelete_malformedIDIsPermanent` +- [ ] unknown state maps to `StateStopped`: `TestMapState_table`, `TestGet_unknownStateIsStopped` +- [ ] `StateRunning` requires an IP: `TestGet_runningWithoutIPIsProvisioning`, `TestGet_runningWithIPIsRunning` +- [ ] `UID` and `CreatedAt` read back: `TestGet_readsUIDAndCreatedAtFromBackend` +- [ ] `ListByTag` excludes unmarked instances: `TestListByTag_excludesUnmarkedVMs`, `TestListByTag_emptyUIDStillListed` +- [ ] `Get` during an in-progress delete: `TestGet_duringDeleteStillReturnsInstance` +- [ ] `Get` during an asynchronous operation or lock: `TestGet_lockedIsTransientNeverNotFound`, `TestGet_midCloneNamelessEntryIsTransient` +- [ ] never started is never `StateStopped`: `TestCreate_returnsOnlyWhenRunning` +- [ ] cancelled `ctx` returns promptly: `TestProvider_cancelledContextReturnsPromptly`, `TestClient_cancelledContextReturnsPromptly` +- [ ] `classify` table with captured responses: `TestClassify_table` +- [ ] Checker reasons, expired `ctx`, stable order: `TestCheck_reasonsTable`, `TestCheck_expiredContextReturnsPromptly`, `TestCheck_oneEntryPerNodeInConfigOrder` +- [ ] sentinel tests for `CloudInit` and the credential: `TestCreate_cloudInitSentinelNeverLoggedOrInErrors`, `TestClient_tokenNeverInErrorsOrLogs` +- [ ] child spans under the operation span: `TestTracing_httpSpansUnderOperationSpan`, `TestClient_childSpanUnderParent` +- [ ] real-backend lifecycle test: `TestProxmox_lifecycle` and `TestProxmoxCheck_realHost` in `proxmox_integration_test.go` + +Tests that cover the Phase 0 corrections specifically, in addition to the +plan's per-phase lists: + +- `TestCreate_resumesMidCloneVMAtDeterministicVMID`: a crash after the clone + was submitted; the retry waits for the lock and adopts, creating nothing. +- `TestCreate_takenVMIDInspectedBeforeProbing`: a foreign VM at attempt 0 is + skipped; ours at attempt 0 is adopted. +- `TestCreate_cloneAlreadyExistsRaceRunsOwnershipCheck`. +- `TestCreate_templateWithCloudInitDriveIsPermanent`. +- `TestCreate_seedUploadOverwritesWithoutDelete`. +- `TestCreate_resizeWaitsForTask`, `TestCreate_resizeSkippedWhenNotLarger`. +- `TestCreate_vlanTagAppendedKeepsMACAndOptions`. +- `TestBootDisk_table`, with a row for `order=scsi0;ide2`. +- `TestGet_pausedIsStoppedFromStatusCurrent`. +- `TestGet_agentMessagesTable` (the three messages and a 200 without IPv4). +- `TestDelete_seedDeleteIsTaskAndIdempotent`. +- `TestClient_x509FailureIsPermanent`. + +**Evidence:** contract §11; docs/testing.md conventions. + +## 22. Deployment + +**Answer:** a new optional Secret and its mount, two Makefile targets, and +samples: + +- `config/manager/manager.yaml`: an optional Secret `proxmox-token` mounted + read-only at `/etc/proxy-operator/proxmox`, keys `token` and `ca.pem`. + `/tmp` is already an emptyDir. +- Makefile: `proxmox-token-secret`, creating or updating `proxmox-token` + from `PROXMOX_TOKEN_FILE` (default `$(HOME)/.config/proxmox-token`) and + `PROXMOX_CA_FILE` (default `$(HOME)/.config/proxmox-ca.pem`), shaped like + `kvm-ssh-secret`; and `test-integration-proxmox` (`go test -race + -tags=integration -count=1 -v -timeout 30m -run TestProxmox + ./internal/provider/proxmox/`). +- `config/manager/providers_config.yaml` and + `config/samples/providers-config.yaml`: a commented `proxmox-home` entry + for the user's host: `url: https://192.168.0.8:8006`, the two file paths, + `nodes: [pv1-1]`, `isoStorage: proxy-seeds`, `vlanTag: 4`, + `templates: {debian-13: {vmid: 9001}}`, a `small` machine type. +- `config/samples/proxy_proxmox.yaml`, listed in + `config/samples/kustomization.yaml`: provider `proxmox-home`, zone + `pv1-1`, machine type `small`, image `debian-13`, the Squid cloud-init + from `proxy_kvm2.yaml`. +- RBAC: none; the provider does not talk to the cluster's API. +- `hack/providers-dev.yaml`: no entry (needs credentials). + +**Evidence:** contract §12. + +## 23. Docs + +**Answer:** `docs/proxmox-provider.md` with sections: Proxmox prerequisites +(the template recipe and what disqualifies a template, the dedicated seed +storage, the user, both roles and the token, the proxy network with DHCP and +internet, reserving the VMID range), Cluster-side setup (the Secret, the +config entry, the URL-must-match-the-certificate rule), Config reference +(entry 15), Placement (Region ignored, catalogue-edit warning), What lands on +Proxmox (the VMID scheme, the description mark, the tag, the seed ISO, the +config keys written, `qm unlock` for a stuck clone), Logging, Tracing, +Timeouts, Health check (entry 16), Error classes (entry 13), Known gaps +(entry 24), Testing. A fifth column in the three tables of +`docs/providers-existing.md` and a proxmox section. `docs/architecture.md`: +components row, a `proxmox` block in §5, the span note in §10, a §11 row and +the provider count, Decisions bullets (VMID from the UID hash with the +ownership check, the mark in the description, the seed by API upload, the +node resolved per call). A README row. `docs/providers.md` link. +`docs/testing.md` rows and a proxmox integration section. `CLAUDE.md` +Gotchas: `make test-integration-proxmox`. A `CHANGELOG.md` entry after the +user confirms. + +**Evidence:** contract §12. + +## 24. Known limits accepted at design time + +**Answer:** ten limits, each with its consequence and workaround: + +- **Create blocks** until the VM runs, holding one of three reconcile + workers. On the user's host that is about five seconds plus the clone; + large templates on slow storage take longer. +- **A clone lock that never clears** (a node crash mid-clone) makes every + Create for that Proxy wait out the clone budget and return + `ErrTransient`. Clear it with `qm unlock ` or destroy the VM. +- **Seed ISOs of VMs destroyed outside the operator** are invisible to GC. + Remove `proxy-*-seed.iso` files without a matching VM by hand. +- **The VMID range is shared with humans.** A VM created by hand inside it + costs a probe; a hand-made VM named like a proxy is refused as + `ErrPermanent`. Reserve the range. +- **Cross-node clones need shared storage** and are untested, because the + host has one node. +- **A VM whose agent was switched off is replaced**, not repaired (entry 7). +- **The proxy network needs internet** for the agent install, unless the + template ships the agent. +- **The CA is cached** after first use; rotating it needs a restart. The + token rotates freely. +- **A node that stops answering** makes `Get` and `Delete` for its VMs + `ErrTransient` until it returns, so deleting such a Proxy waits. +- Contract §13 limits that apply unchanged: config read once per process, + no installation identity in GC (two operators sharing a cluster and a + VMID range delete each other's VMs), an unconfigured provider latches the + Proxy. + +**Evidence:** contract §13; entries 2, 7, 10, 11, 14 above.