docs: proxmox provider spec

Phase 1 of the proxmox provider plan: docs/proxmox-provider-spec.md answers all 24 entries of the provider spec template against the contract, citing the Phase 0 facts from pv1-1 as evidence. It folds in the ten corrections recon found. The two that matter most: Create inspects a taken VMID, waiting out a clone lock, before probing past it, so a crash-retry adopts its own half-cloned VM instead of cloning a duplicate; and Get checks the lock before the name, because a VM being cloned has no name yet. The plan now opens with a note that the spec wins where they disagree. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-09-23 09:18:29 +00:00
parent ff94fe70bb
commit 4e974f30f6
3 changed files with 915 additions and 1 deletions
@@ -5,7 +5,7 @@ Pairs with [docs/plans/2026-09-18-1502-proxmox-provider.md](../plans/2026-09-18-
 ## Status
 - [x] Phase 0 — Plan file, branch, host recon, fixtures
- [ ] Phase 1 — The spec (`docs/proxmox-provider-spec.md`)
+- [x] Phase 1 — The spec (`docs/proxmox-provider-spec.md`)
 - [ ] Phase 2 — Config block, registration, package skeleton
 - [ ] Phase 3 — Extract `nocloud` and `keyedlock` from kvm2
 - [ ] Phase 4 — Seam, HTTP client, decorator, fake
@@ -280,3 +280,52 @@ volumes of 199999. The capture is `agent_network_get_interfaces_vlan.json`.
 - Time to IP was about 50 s on both networks, with the agent installed at
  first boot. Budgets in the plan (running wait 60 s for the VM state, the
  agent discovered by `Get` polling) are consistent with this.
 ## Phase 1 — The spec
 Wrote [docs/proxmox-provider-spec.md](../proxmox-provider-spec.md), all 24
 entries of the template, from the plan, the contract and the Phase 0 facts.
 The ten Phase 0 corrections are folded in, and the plan now opens with a
 note that the spec wins where the two disagree.
 Judgment calls made here, each recorded in its entry:
 - **Create's allocation loop** (entry 2) runs an ownership check on any
  taken VMID: wait out a lock or a nameless entry, then adopt when the name
  and UID are ours, skip when the name differs, refuse as Permanent when the
  name is ours but the mark is not. This is the fix for the duplicate-VM bug.
 - **Get's check order** (entry 9): lock, then name and mark from the config,
  then `status/current`, then the agent. `lock: suspended` alone maps to
  Stopped, since a hibernated VM never resumes by itself.
 - **Agent messages** (entry 7): `QEMU guest agent is not running` is
  Provisioning, `VM <vmid> is not running` is Transient, and
  `No QEMU guest agent configured` is **Stopped**, so the reconciler replaces
  a VM that can never report an address instead of latching the Proxy.
 - **Templates are checked before cloning** (entry 4): not a template, or a
  Proxmox cloud-init drive in any slot, is Permanent. This makes the "no
  cloud-init drive" prerequisite enforceable instead of a silent failure.
 - **`isoStorage` is required**, with no default (entry 15), because the token
  must hold `Datastore.Allocate` on it.
 - **TLS verification failures are Permanent**, with a Checker reason
  `CertificateInvalid` (entries 13, 16), since the fix is the URL or CA file.
 - **The CA is loaded lazily and cached**; the token is read per request
  (entry 14).
 Checks, as the plan's Verify clause asks:
 ```bash
 grep -c '^## [0-9]*\. ' docs/proxmox-provider-spec.md     # 24
 grep -n '^\*\*Answer:\*\*$' docs/proxmox-provider-spec.md # nothing
 grep -c '—' docs/proxmox-provider-spec.md                 # 0
 ```
 plus a script comparing every `` `<axis>`: `<profile>` `` pair against the
 contract's §4.20 table (no mismatches, every axis answered) and a list of
 the F numbers cited (all 23).
 Worth noting: the first pass stated two things as fact that recon could not
 show and was corrected before commit. That an unreachable node's guests stay
 listed as `unknown` is Proxmox's documented design, unverifiable on a single
 node; entry 13 now says so. And the matcher for `VM <vmid> is not running`
 is a whole-message regexp, because ` is not running` alone also matches
 `QEMU guest agent is not running`.
@@ -4,6 +4,15 @@
 ## Context
 > **Superseded in places by the spec.** Phase 0 recon found ten corrections
 > to the design below (listed in the execution log), and Phase 1 folded them
 > into [docs/proxmox-provider-spec.md](../proxmox-provider-spec.md). Where a
 > phase step here disagrees with the spec (the Create allocation loop, Get's
 > check order, resize and seed delete as tasks, no delete before upload,
 > `isoStorage` required, `vlanTag`, the agent messages), **the spec wins**.
 > Test names in the phases below remain the minimum; spec entry 21 adds the
 > ones for the corrections.
 The provider contract (`docs/provider-contract.md`), the spec template
 (`docs/provider-spec-template.md`) and the kvm2 build proved the workflow:
 spec first, then implementation against the contract, one commit per step.
@@ -0,0 +1,856 @@
 # Provider spec: proxmox
 Filled from [provider-spec-template.md](provider-spec-template.md) against
 [provider-contract.md](provider-contract.md). This is the design the
 `proxmox` package is built from. Where an entry names an axis in backticks,
 the answer is one of that axis's profiles in contract §4.
 Backend facts come from the Phase 0 recon on the user's host, recorded in
 [the execution log](plans-executions/2026-09-18-1502-proxmox-provider.md) as
 facts F1 to F23, with the raw responses under
 `internal/provider/proxmox/testdata/`. An Evidence line citing `F<n>` points
 there. Where this spec and the plan
 ([docs/plans/2026-09-18-1502-proxmox-provider.md](plans/2026-09-18-1502-proxmox-provider.md))
 disagree, this spec wins: it folds in the ten corrections Phase 0 found.
 ## Header
 - **Type**: `proxmox`
 - **Example configured name**: `proxmox-home`
 - **Backend and API version this spec was written against**: Proxmox VE
  9.2.20 (`pve-manager/9.2.20/49318c671b82f31e`, kernel `7.0.14-17-pve`,
  QEMU 11.0.3) on the single node `pv1-1`, API at
  `https://192.168.0.8:8006/api2/json`.
 - **Client**: `thin-client`: `net/http` and `encoding/json` from the standard
  library, with the transport wrapped by the already-required
  `go.opentelemetry.io/contrib/instrumentation/net/http/otelhttp`. No new
  dependency. The seed ISO uses `github.com/kdomanski/iso9660`, already
  required by kvm2.
 - **Author and date**: Claude (orchestrator session), 2026-09-23.
 ## 1. Identity and providerID
 **Answer:** `vms/<vmid>/<name>`, where `<vmid>` is the VM's numeric id and
 `<name>` is `req.Name`. The ID is location-free: the node is looked up in
 `GET /cluster/resources?type=vm` on every call, so a VM migrated between
 nodes stays addressable. Parsing is strict: exactly three segments, the
 literal `vms`, a decimal VMID without leading zeros between 100 and
 999999999, and a non-empty name without `/`. Anything else is `ErrPermanent`
 from `Get` and `Delete`. The name is carried so that `Get` and `Delete` can
 tell our VM from another VM that later received the same VMID (entry 2). No
 config value is pasted into the ID, so no ID-safety regexp is needed; node
 names and storage ids are still validated at load (entry 15) because they
 end up in request paths.
 **Evidence:** contract §6 and §4.10 (the ID must not depend on the location
 when instances can migrate); F8 (`node` is a field of every resource entry).
 ## 2. Primary key (`primary-key`)
 **Answer:** `allocated-id`. Proxmox addresses VMs by VMID; `name` is only a
 label and is not unique. The VMID is derived from the Proxy UID so a retry
 lands on the same number: `vmidFor(uid, attempt)` is
 `min + (fnv1a64(uid) + attempt) mod (max - min + 1)` over `vmidRange`
 (entry 15). Create, under the per-name lock (entry 19):
 1. Reads `/cluster/resources` once. Entries with `type: qemu`, `template: 0`
   and `name == req.Name` are the by-name candidates. Two or more is
   `ErrPermanent` naming every VMID: an admin must remove the duplicate.
   Exactly one goes to the ownership check below and is adopted or refused.
 2. With none, probes `attempt = 0, 1, …` up to 32:
   - VMID free in the listing: clone onto it. If the clone fails with
     `unable to create VM <vmid>: config file already exists` (F4), someone
     took it between the listing and the clone; run the ownership check on
     it.
   - VMID taken: run the ownership check.
   - Exhausted: `ErrTransient` naming the range.
 3. The **ownership check** on a VMID: if its resource entry carries a
   `lock` or has no `name` (a clone in progress lists neither, F8), wait
   until it is unlocked (entry 10, clone budget). Then read its config. It
   is **ours** when `name == req.Name` and the description carries the mark
   with `uid` equal to `req.UID` or empty: adopt it and resume Create at the
   seed step (entry 10). It is **foreign** when the name differs: move to
   the next attempt. The same name with no mark, or with a different
   non-empty `uid`, is `ErrPermanent` naming the VMID and both UIDs; the
   operator never takes over a VM it did not create.
 The wait in step 3 is what closes the duplicate-VM bug Phase 0 found: after
 a crash mid-clone, the retry's by-name lookup misses the nameless VM, but
 attempt 0 lands on the same VMID, waits for the clone to finish, and adopts
 it instead of probing past it.
 `Get` and `Delete` check the name and mark at the VMID in the ID. A VM there
 with another name means ours is gone and the VMID was reused: `Get` answers
 `ErrNotFound`, `Delete` succeeds without touching it.
 **Evidence:** F4 (taken VMID fails synchronously); F8 (a mid-clone entry has
 no name); contract §4.10 (allocate-then-verify, deterministic from the UID)
 and §13 (adoption not verified: closed here).
 ## 3. Placement mapping
 **Answer:** three fields are catalogue keys checked before any backend call,
 one is ignored:
 | Field | Meaning | Required | Unknown value |
 |---|---|---|---|
 | `Zone` | a node from `nodes` | only when more than one node is configured; with one node an empty zone selects it | `ErrPermanent`: `placement.zone "x" is not in proxmox.nodes` |
 | `MachineType` | a key of `machineTypes` (`cores`, `memoryMiB`, optional `diskGiB`) | yes | `ErrPermanent`: `placement.machineType "x" is not in proxmox.machineTypes` |
 | `Image` | a key of `templates` (a template VMID) | yes | `ErrPermanent`: `placement.image "x" is not in proxmox.templates` |
 | `Region` | ignored | | still part of the spec hash: changing it replaces the VM |
 The guide documents the ignored field and that catalogue edits affect only
 new VMs.
 **Evidence:** contract §2.1 (reject early, MAY ignore with documentation).
 ## 4. Image source
 **Answer:** a template VM prepared by the admin, referenced by a catalogue
 key: `templates.<key>.vmid`. Create makes a **full clone** (`full=1`), so a
 VM owns copies of its disks and keeps nothing linked to the template: the
 template may be edited, replaced or deleted without affecting running VMs.
 Before cloning, Create reads the template's config and refuses it with
 `ErrPermanent` when it is not a template (`template` is not 1) or when any
 disk slot holds a Proxmox cloud-init drive (a value containing `cloudinit`),
 which would compete with the provider's seed (entry 5). The template must
 boot from its first disk, run cloud-init with the NoCloud datasource, and
 either ship `qemu-guest-agent` or be able to install it (entry 7).
 Clone parameters: `newid`, `name`, `description` (the mark, entry 8),
 `full=1`, `storage` when configured (empty clones onto the template's own
 disk storage), and `target` when the chosen node differs from the template's
 node. A cross-node clone needs the template on shared storage; this host has
 one node, so that path is untested (entry 24).
 Catalogue edit warning for the guide: changing a key's `vmid` changes only
 VMs created afterwards. Add a new key rather than repointing one in use, so
 the Proxies that name it keep a meaning you can reason about.
 **Evidence:** F19 (template 9001: `scsi0`, `boot: order=scsi0`, `ide2`
 free, `agent: 1`; template 9000 carries a cloud-init drive on `ide2`); F15
 (a clone onto a storage without image support is refused up front); F16 (a
 full clone of 9001 took about two seconds).
 ## 5. Cloud-init delivery
 **Answer:** a NoCloud seed ISO built in-process by the shared
 `internal/provider/nocloud` package (extracted from kvm2 in Phase 3):
 volume label `cidata`, files `user-data` (the Proxy's `CloudInit`, byte for
 byte, never parsed), `meta-data` (`instance-id: <name>` and
 `local-hostname: <name>`) and `vendor-data` (the guest agent script, joined
 with the configured `vendorData` as a MIME multipart when both are set;
 identical to kvm2's rules). The builder stages under `os.TempDir()` (`/tmp`,
 the manager's emptyDir) and writes into a `bytes.Buffer`; a seed is tens of
 KiB, so buffering is within contract §1.4.
 The ISO is uploaded with `POST /nodes/{node}/storage/{isoStorage}/upload`
 as multipart (`content=iso`, file part `filename` named
 `<name>-seed.iso`) and the returned `imgcopy` task is waited on. An upload
 **overwrites** an existing file of the same name, so no delete precedes it;
 the seed is rebuilt and re-uploaded on every Create attempt, since a
 previous attempt may have been cut off. It is attached as
 `ide2: <isoStorage>:iso/<name>-seed.iso,media=cdrom`; Proxmox then appends
 `ide2` to the boot order on its own, which is harmless because the boot
 disk stays first.
 Only the Proxmox API is used: the storage upload endpoint accepts `iso`,
 `vztmpl` or `import`, never `snippets`, so `cicustom` would need shell
 access to the node. Size: the request timeout (entry 18) bounds the upload;
 a seed with a multi-megabyte `vendorData` would need a larger
 `requestTimeoutSeconds`.
 **Evidence:** F12 (upload fields, UPID, overwrite, `.iso` suffix required);
 F19 (`ide2` appended to the boot order); F21 (the seed on `ide2` installed
 the agent on both the untagged network and VLAN 4); F23 (the upload is
 staged in `/var/tmp` on the node, not the tmpfs `/tmp`); contract §2.1 and
 §11 (byte for byte, sentinel test).
 ## 6. Machine type mapping
 **Answer:** a catalogue: `machineTypes.<key>` is `{cores, memoryMiB,
 diskGiB}`. `cores` and `memoryMiB` are written in the config step (entry
 10). `diskGiB` is optional: when set and larger than the cloned boot disk,
 the disk is grown with `PUT /nodes/{node}/qemu/{vmid}/resize` to the
 absolute size `<diskGiB>G`, and the returned task is waited on; when unset
 or not larger, the disk keeps the template's size and no call is made. The
 provider never shrinks a disk. Cloud-init's growpart expands the root
 filesystem at first boot.
 The boot disk is the first entry of the config's `boot: order=…` that is a
 disk key (`scsiN`, `virtioN`, `sataN`, `ideN`) and not a CD-ROM, else the
 `bootdisk` key, else the first present of `scsi0`, `virtio0`, `sata0`,
 `ide0`. Its current size comes from the `size=` option of that disk's value.
 No boot disk, when `diskGiB` is set, is `ErrPermanent`.
 **Evidence:** F20 (resize is a task; the same size is an `OK` no-op; a
 shrink fails in the task with `shrinking disks is not supported`); F19
 (`boot: order=scsi0`, becoming `order=scsi0;ide2` once the seed is
 attached).
 ## 7. Network and IP discovery (`readiness`, `ip-source`)
 **Answer:** `readiness`: `provisioning-window`. `ip-source`: `discovered`,
 from the QEMU guest agent only (`GET /nodes/{node}/qemu/{vmid}/agent/network-get-interfaces`);
 the API offers no ARP or DHCP-lease source. The address reported is the first
 IPv4 of the first non-loopback interface, in the agent's order, that is not
 loopback, link-local or unspecified.
 The NIC is the template's `net0`, rewritten in the config step only when the
 config sets `bridge` or `vlanTag`: the value's comma-separated options are
 kept in order with the model and MAC untouched, `bridge=` and `tag=` are
 replaced or appended. With neither set, `net0` is left as cloned.
 The agent answers:
 | Agent response | Meaning | Get reports |
 |---|---|---|
 | 200 with a routable IPv4 | booted | Running with that IP |
 | 200 without one | network still coming up | Provisioning |
 | 500 `QEMU guest agent is not running` | booting, or the agent install has not finished or failed | Provisioning |
 | 500 `VM <vmid> is not running` | the VM stopped between the status read and this call | `ErrTransient` |
 | 500 `No QEMU guest agent configured` | `agent: 1` was removed from the config; this VM can never report an address | Stopped, so the reconciler replaces it with a VM that has the agent on |
 | anything else | cannot tell | `ErrTransient` |
 The guest needs the agent. The vendor-data script installs it at first boot
 unless `installGuestAgent: false`, and that install needs a package mirror:
 **the proxy network must have a route to the internet**, which an egress
 proxy needs anyway. Baking `qemu-guest-agent` into the template and setting
 `installGuestAgent: false` removes the mirror from IP discovery. `Port` is
 not used by the provider. Reachability is a prerequisite in the guide: the
 operator's pod network and the clients must route to the proxy network (on
 the user's host, VLAN 4, `192.168.4.0/24`).
 **Evidence:** F5 (all three agent messages); F21 (agent response shape; an
 address about 50 seconds after start on the untagged network and on VLAN 4);
 the Phase 0 log's VLAN 4 section (no internet route meant no agent);
 contract §1.3 and §2.2 (Running only with an IP).
 ## 8. GC marks and CreatedAt (`gc-mark`, `created-at`)
 **Answer:** `gc-mark`: `written-metadata`. `created-at`: `written-at-create`.
 The mark is three lines in the VM's `description`, passed in the clone call
 itself, so there is no mark window:
 ```
 proxy-operator-managed=true
 proxy-operator-uid=<req.UID>
 proxy-operator-created-at=<RFC3339 UTC>
 ```
 The keys are `provider.LabelManaged`, `provider.LabelUID` and
 `proxy-operator-created-at`; the managed value is `provider.LabelManagedYes`.
 The parser reads `key=value` lines, tolerates CRLF, blank lines and any
 other lines an admin adds, and treats the VM as managed only when the
 managed value is exactly `true`. A missing `uid` line gives an empty UID. A
 `created-at` that does not parse gives the zero time, which GC treats as
 old; that is accepted. The clone's own `meta: …,ctime=…` is not usable:
 Proxmox copies it from the template.
 The config step also sets the tag `proxy_operator`, for filtering in the
 Proxmox UI and as a ListByTag candidate hint (entry 12). It is not the mark:
 tags cannot be passed to the clone call. Hyphens are accepted in tags on
 9.2.20, but the underscore form is kept because it is valid under every
 tag rule Proxmox has had. Uppercase is folded to lowercase by Proxmox.
 The seed ISO carries no mark; its name `<name>-seed.iso` identifies it
 (entry 11).
 **Evidence:** F10 (the description survives the clone verbatim, newlines
 included); F11 (the clone's `meta` is the template's); F9 (tag rules);
 contract §2.1 (mark in the create call) and §4.5.
 ## 9. State mapping (`provisioning-state`)
 **Answer:** `provisioning-state`: `create-blocks`. Create returns only once
 `status/current` reports the VM running (entry 10), so a VM the provider
 cloned but never started is only ever seen by a Create retry, which adopts
 and starts it; the reconciler has no ID for it to `Get`.
 `Get` checks in this order; the first match decides:
 | Check | Source | Result |
 |---|---|---|
 | VMID absent | `/cluster/resources` | `ErrNotFound` |
 | `lock` set, other than `suspended` | resources entry (a clone in progress also has no name) | `ErrTransient` |
 | `lock: suspended` (hibernated to disk) | resources entry | Stopped |
 | `status: unknown` (node not reachable) | resources entry | `ErrTransient` |
 | config `does not exist` | `GET …/config` | `ErrNotFound` (deleted between the two calls) |
 | config `lock` set | config | `ErrTransient` |
 | `name` differs from the ID's name | config | `ErrNotFound`, logged: the VMID was reused |
 | no mark in the description | config | `ErrNotFound`, logged: not ours |
 | `status: stopped` | `status/current` | Stopped |
 | `status: running`, `qmpstatus: running` | `status/current` | the agent decides (entry 7) |
 | `status: running`, `qmpstatus: prelaunch` | `status/current` | Provisioning |
 | `status: running`, any other `qmpstatus` (`paused`, `suspended`, `io-error`, …) | `status/current` | Stopped |
 | any other `status` | `status/current` | Stopped |
 `status/current` is read, not only the resources entry, because a paused VM
 shows as plain `running` in `/cluster/resources`. `UID` and `CreatedAt` come
 from the description on every call. A running `Get` makes four calls:
 resources, config, status and agent.
 **Evidence:** F8 (resources fields; mid-clone entries lack a name, carry
 `lock: clone` and `status: unknown`); F14 (stopped and paused as seen by
 `status/current` and by resources); F2 (the `does not exist` message);
 contract §2.2 and §4.11.
 ## 10. Create: blocking and task tracking (`create-return`, `async-ops`)
 **Answer:** `create-return`: `blocking`. `async-ops`: `task-handle` and
 `backend-lock`. Clone, upload, resize, start, stop, destroy and volume
 delete each return a UPID; the provider polls
 `GET /nodes/{node}/tasks/{upid}/status` every second until `status` is
 `stopped`, then reads `exitstatus`: `OK` is success, anything else is
 `ErrTransient` carrying that text. The node comes from the UPID
 (`UPID:<node>:…`). The handle is held within the call only: Create blocks
 until the VM runs, so no handle outlives it, and a failure surfaces as
 Create's own error with the task's message. Locks: a VM locked by a task is
 waited on in Create (the ownership check) and reported as `ErrTransient` by
 `Get` and `Delete`. A start issued while a clone holds the lock queues
 behind it rather than failing; the provider never relies on that, because it
 waits for the clone task first.
 Steps, each idempotent, in order, under the per-name lock (entry 19):
 1. Validate placement (entry 3). No backend call yet.
 2. `GET /cluster/resources`. Find the template's node; a missing template
   is `ErrPermanent`. Read the template's config and refuse it as entry 4
   says.
 3. Find or allocate the VMID (entry 2). An adopted VM that is running
   returns its ID at once. An adopted VM that is stopped continues at step 5.
 4. Clone and wait (clone budget).
 5. Build the seed and upload it to the VM's node, overwriting; wait
   (upload budget).
 6. `PUT /nodes/{node}/qemu/{vmid}/config` (synchronous) with `cores`,
   `memory`, `agent=1`, `tags=proxy_operator`,
   `ide2=<isoStorage>:iso/<name>-seed.iso,media=cdrom` and, when configured,
   the rewritten `net0` (entry 7).
 7. Resize when `diskGiB` is larger than the boot disk (entry 6); wait
   (resize budget).
 8. `POST …/status/start`; wait (start budget).
 9. Poll `status/current` every second until `status: running` and
   `qmpstatus: running`, up to the running budget; expiry is
   `ErrTransient`, and so is a `stopped` status after the start.
 10. Return `vms/<vmid>/<name>`.
 What a retry finds:
 | Previous attempt stopped after | The retry |
 |---|---|
 | nothing | allocates, clones |
 | submitting the clone | finds the VMID locked and nameless, waits, adopts, continues at 5 |
 | the clone | adopts the stopped VM, continues at 5 |
 | any of steps 5 to 7 | adopts, repeats 5 to 7 (upload overwrites, config is a replace, resize is skipped once large enough) |
 | the start | adopts the running VM, returns its ID |
 A Proxmox `does not exist` inside Create (a VM or volume vanishing between
 steps) is `ErrTransient`, never `ErrNotFound`. The log line for a Create
 records `cloudInitBytes`, never the content.
 **Evidence:** F3 (a start queues behind the clone lock); F7 (a config write
 under the clone lock is refused with `VM is locked (clone)`); F16 (task
 status fields); F20 (resize is a task); the Phase 0 log's VLAN 4 rerun
 (clone to started VM in about five seconds); contract §2.1, §4.2, §4.12 and
 §5 ("do not return NotFound from Create").
 ## 11. Delete and secondary resources (`delete-return`, `secondary-resources`)
 **Answer:** `delete-return`: `waits`. `secondary-resources`: `explicit` for
 the seed ISO; the VM's disks are removed by the destroy call itself, with
 `purge=1&destroy-unreferenced-disks=1`. Under the per-name lock:
 1. Parse the ID (`ErrPermanent` if malformed).
 2. `/cluster/resources`. VMID absent: skip to step 7, trying every configured
   node because the VM's node is no longer known.
 3. `lock` set or `status: unknown`: `ErrTransient`. A clone or another task
   still running finishes first; the reconciler calls Delete again after
   `DeletionPoll`.
 4. Read the config. `does not exist`: skip to step 7. A different name, or
   no mark: the VM is not ours; log it and skip to step 7 without touching
   it.
 5. Status other than `stopped`: `POST …/status/stop`, wait (stop budget).
   Stop works on a paused VM too.
 6. `DELETE /nodes/{node}/qemu/{vmid}?purge=1&destroy-unreferenced-disks=1`,
   wait (destroy budget). `does not exist` is success.
 7. `DELETE /nodes/{node}/storage/{isoStorage}/content/<isoStorage>:iso/<name>-seed.iso`,
   wait (seed-delete budget). A missing volume is still a successful task,
   so no message is matched. On the try-every-node path a node whose
   request fails is logged and skipped.
 8. Return nil.
 Everything is gone when Delete returns nil, so `Get` afterwards is
 `ErrNotFound`. A concurrent `Get` during a Delete keeps seeing the VM (as
 Stopped once it is stopped) until the destroy task removes its config,
 because destroying sets no lock and leaves the resource entry in place until
 the end. A second Delete waits on the lock, then finds nothing.
 GC blind spot: a VM destroyed outside the operator leaves its seed ISO, which
 ListByTag cannot see. The guide says seeds are named `proxy-*-seed.iso` on
 `isoStorage` and any without a matching VM may be removed.
 **Evidence:** F2 (a second destroy gives `does not exist`); F13 (volume
 delete is an `imgdel` task, needs `Datastore.Allocate`, and succeeds on a
 missing volume); the Phase 0 capture of resources and config mid-destroy
 (`cluster_resources_during_destroy.json`, `config_during_destroy.json`);
 contract §2.3.
 ## 12. ListByTag scope and partial failure (`list-partial`)
 **Answer:** `fail-whole` for the one listing, with single unreadable VMs
 skipped. Scope: every VM the cluster knows, from one
 `GET /cluster/resources?type=vm`, which covers every node, including
 unreachable ones, whose guests Proxmox lists with `status: unknown` (entry
 13). Candidates are entries
 with `type: qemu`, `template: 0`, and either the tag `proxy_operator` (tags
 split on `;`, `,` or space) or a name starting with `proxy-`. A nameless
 entry (a clone in progress) is not a candidate: its Create is live and it is
 younger than `MinAge`. For each candidate the config is read and the
 description parsed; managed VMs become an `Instance` with `ID`, `UID` and
 `CreatedAt` only, which is all GC reads (contract §2.4); no status or agent
 call is made. A candidate whose config cannot be read (a node down, a VM
 destroyed mid-sweep) is skipped with a V(1) line, unless ctx ended, which
 fails the call. The resources call failing fails the whole call. Filtering
 by the mark is client-side; the tag narrows the candidates. Cost: one
 resources call plus one config read per candidate per sweep.
 **Evidence:** F8 (resources fields, tags separated by `;`, nameless mid-clone
 entries); F9 (tags case-folded); contract §2.4 and §4.7.
 ## 13. Error classification (`not-found`, `quota`)
 **Answer:** `not-found`: `error-code`, the 500 message
 `Configuration file 'nodes/<node>/qemu-server/<vmid>.conf' does not exist`,
 plus `list-and-miss` on `/cluster/resources`. The listing can miss a VM only
 when it is gone: it is built from the cluster filesystem, which holds every
 guest's config cluster-wide, and Proxmox lists the guests of an unreachable
 node with `status: unknown` rather than dropping them. That last point is
 Proxmox's documented design, not something the single-node host could show;
 Phase 10 or a second node verifies it. `quota`: `absent`. The thin `local-lvm` storage
 overprovisions, so no out-of-space response could be produced; capacity
 failures are `ErrTransient`, and the Checker reports free space in
 `Details` instead (entry 16).
 The client carries each failure as `apiError{Status, Message, Errors}`:
 `Message` is the body's `message` with the trailing newline trimmed, or the
 reason phrase when the body has none (a 401); `Errors` is the 400 body's
 per-parameter map. One `classify(op string, err error)`:
 | Backend response | Class | Why |
 |---|---|---|
 | 401 `Authentication failed!`, 401 `No ticket` | Permanent | the token or its secret is wrong; fixed in the Secret |
 | 403 `Permission check failed (<path>, <privilege>)` | Permanent | the token lacks a privilege; the message names it |
 | 400 `Parameter verification failed.` with `errors` | Permanent | the request is wrong; the message includes the field errors |
 | other 4xx | Permanent | a request Proxmox will never accept |
 | 500 `Configuration file '…' does not exist` | NotFound (Transient when `op == "create"`) | the VM is gone |
 | 500 `VM is locked (<lock>)` | Transient | a task holds the VM |
 | 500 `unable to create VM <vmid>: config file already exists` | handled by the allocator (entry 2); Transient if it escapes | the VMID is taken |
 | 500 `QEMU guest agent is not running`, `No QEMU guest agent configured`, `VM <vmid> is not running` | handled by `Get` (entry 7); Transient if they escape | agent state |
 | any other 5xx, including pveproxy's 595 and 596 | Transient | unknown means retry |
 | task `exitstatus` other than `OK` | Transient, with the text | the task failed; the next attempt retries |
 | TLS verification failure (`x509.UnknownAuthorityError`, `x509.HostnameError`, `x509.CertificateInvalidError`) | Permanent | wrong URL or CA file; the admin must act |
 | token file missing (`fs.ErrNotExist`) | Transient | the Secret has not landed yet |
 | token file present but not `<user>@<realm>!<id>=<secret>` | Permanent | wrong content in the Secret |
 | `context.Canceled`, `context.DeadlineExceeded`, `net.OpError`, `io.EOF`, `io.ErrUnexpectedEOF` | Transient | network or cancellation |
 | an error already wrapped with a class | that class | validation and builders classify at the source |
 | anything else | Transient | contract default |
 Every matched string is backed by a captured response under `testdata/`;
 the table test loads them rather than repeating the text. Matching is on
 fixed substrings anchored to the 500 status: `does not exist`,
 `VM is locked`, `config file already exists`, `QEMU guest agent is not
 running`, `No QEMU guest agent configured`, and for the stopped VM the whole
 message against `^VM [0-9]+ is not running$`. The agent messages are tested
 in that order, because the last is a suffix of the first.
 **Evidence:** F1 (error text in the reason phrase and the body; 401 has no
 body); F2, F4, F5, F7 (the 500 messages); F6 (no out-of-space response
 could be produced on thin storage); F22 (both 401 texts); fixtures
 `err_forbidden.json`, `err_bad_param.json`, `err_tag_invalid.json`,
 `err_upload_bad_name.json`; the Phase 0 TLS finding (`192.168.2.8` not in
 the node certificate); contract §5.
 ## 14. Credentials and TLS (`credentials`)
 **Answer:** `mounted-files`. The config names `tokenFile`, one line
 `<user>@<realm>!<tokenid>=<secret>`, sent as
 `Authorization: PVEAPIToken=<that line>`. It is expected in the optional
 Secret `proxmox-token`, key `token`, mounted at
 `/etc/proxy-operator/proxmox`. The file is read on **every request**, so
 rotating the Secret takes effect without a restart. `New` stats it and logs
 a warning when it is missing, but never fails on it and never dials.
 TLS: `ca-file`. `caFile` names a PEM bundle, typically the node's
 `/etc/pve/pve-root-ca.pem`, carried in the same Secret as key `ca.pem`. It
 is loaded on first use and cached, so rotating the CA needs a restart; a
 missing CA file is `ErrTransient` until it lands, one that does not parse is
 `ErrPermanent`. Without `caFile` the system roots are used. The URL's host
 must be a name or address in the node certificate's SANs: on the user's
 host that is `192.168.0.8` or `pv1-1.home.lab`, not `192.168.2.8`.
 `insecureSkipTLSVerify: true` disables verification and is logged as a
 warning by `New`.
 Minimum privileges, as verified on 9.2.20, for `proxy-operator@pve` with a
 token without privilege separation:
 - a role on `/` with `VM.Allocate VM.Clone VM.Audit VM.Config.CDROM
  VM.Config.CPU VM.Config.Memory VM.Config.Disk VM.Config.Network
  VM.Config.Options VM.PowerMgmt VM.GuestAgent.Audit Datastore.Audit
  Datastore.AllocateSpace Datastore.AllocateTemplate Sys.Audit SDN.Use`;
 - a role on `/storage/<isoStorage>` with `Datastore.Allocate
  Datastore.AllocateTemplate Datastore.AllocateSpace Datastore.Audit`,
  because deleting a volume needs `Datastore.Allocate` and a grant on a more
  specific path replaces the inherited one. A dedicated ISO-only storage
  keeps that privilege away from backups.
 `VM.Monitor` no longer exists in PVE 9 and is not needed.
 `VM.Config.Cloudinit` is not needed. `SDN.Use` on `/` covers a VLAN tag on
 `vmbr0`.
 Hygiene: the token never appears in logs, span attributes or error strings.
 `apiError.Error()` carries status and message only, never headers or URLs.
 Tested with a sentinel secret that a test server echoes in a 401 body.
 **Evidence:** F17 (privileges exercised and the one missing); the Phase 0
 log's seed storage section; F22 (auth failure texts); contract §7.
 ## 15. Config block
 **Answer:** a new block `proxmox:` with the Go type `provider.ProxmoxConfig`
 in `internal/provider/config_proxmox.go`:
 | Field | Required | Default | Validation | Ends up in |
 |---|---|---|---|---|
 | `url` | yes | | `https` URL with a host and no path other than `/` | request base, `…/api2/json` appended |
 | `tokenFile` | yes | | non-empty | `Authorization` header |
 | `caFile` | | system roots | non-empty when set | TLS roots |
 | `insecureSkipTLSVerify` | | false | | TLS, warning at `New` |
 | `nodes` | yes, at least one | | unique, `^[a-z0-9]([a-z0-9-]*[a-z0-9])?$` | request paths, clone `target` |
 | `isoStorage` | yes | | Proxmox storage id `^[a-zA-Z][a-zA-Z0-9_.-]*$` | upload and delete paths, `ide2` |
 | `storage` | | the template's disk storage | storage id regexp when set | clone `storage` |
 | `bridge` | | as cloned | non-empty when set | `net0` `bridge=` |
 | `vlanTag` | | as cloned | 1 to 4094 when set | `net0` `tag=` |
 | `vmidRange.min`, `vmidRange.max` | | 100000, 199999 | min at least 100, max at most 999999999, min below max | `vmidFor` |
 | `templates.<key>.vmid` | at least one template | | at least 100 | clone source |
 | `machineTypes.<key>` | at least one | | `cores` and `memoryMiB` above zero, `diskGiB` not negative | config step, resize |
 | `requestTimeoutSeconds` | | 30 | not negative | HTTP client timeout |
 | `vendorData` | | | | seed `vendor-data` |
 | `installGuestAgent` | | true | | seed `vendor-data` |
 `isoStorage` has no default on purpose: the token must be allowed to delete
 volumes there (entry 14), so the choice must be explicit. `nodes` holds node
 names as Proxmox knows them (`pv1-1`), not addresses; the one URL reaches the
 whole cluster.
 Wiring: `ProviderConfig.Proxmox *ProxmoxConfig` with tag
 `proxmox,omitempty`; `setBlocks` reports `proxmox`; the allow-list in
 `Config.validate` and the constructor map in `cmd/main.go` both gain
 `proxmox` in the same commit; `validateProxmox(prefix, c)` in
 `config_proxmox.go` with `config_proxmox_test.go`.
 **Evidence:** contract §10; the Phase 0 decisions (dedicated seed storage,
 `vlanTag`, URL at `192.168.0.8`); F13 (why `isoStorage` needs
 `Datastore.Allocate`).
 ## 16. Checker
 **Answer:** one target per `nodes` entry, `Name` the node, `Address` the
 URL, checked concurrently, returned in config order. The check shares the
 provider's HTTP client; each request is independent, so a check that runs
 out of time cancels only its own requests. Per node:
 1. `GET /version` (`pveVersion`).
 2. `GET /nodes/{node}/status` (the node answers; `kernel`).
 3. `GET /nodes/{node}/storage/{isoStorage}/status`: the storage exists, is
   active, and lists `iso` in its content (`isoStorageAvailGiB`).
 4. `GET /cluster/resources?type=vm`: every configured template exists and
   is a template (`templates` as `<found>/<configured>`).
 Reasons, shared first: `Reachable`; `Timeout` (the round's ctx ended,
 whichever request was running); `Unreachable` (TCP connect or TLS handshake
 failed for a network reason, or pveproxy answered 595 or 596);
 `Unauthorized` (401); `Forbidden` (403, the message names the privilege);
 `APIError` (any other failure). Provider-specific, each fixed differently by
 an admin:
 - `CertificateInvalid`: TLS verification failed. Fix `url` or `caFile`, not
  the host.
 - `CredentialsUnavailable`: the token file is missing or malformed. Fix the
  Secret, not Proxmox.
 - `StorageNotFound`: `isoStorage` is missing, inactive, or has no `iso`
  content. Fix the storage on the node.
 - `TemplateNotFound`: a configured template VMID does not exist or is not a
  template. Fix the template or the catalogue.
 Details: `pveVersion`, `kernel`, `tokenID` (the part before `=`, never the
 secret), `isoStorage`, `isoStorageAvailGiB`, `templates`, filled as far as
 the check got. Cost: four requests per node per round. The `status.type`
 comment in `api/v1alpha1/providerhealth_types.go` gains `proxmox` and the
 CRD is regenerated.
 **Evidence:** contract §8; fixtures `version.json`, `node_status.json`,
 `storage_status.json`, `cluster_resources.json`.
 ## 17. Client and child spans (`client-kind`, `child-spans`)
 **Answer:** `client-kind`: `thin-client`. `child-spans`: `transport-wrapper`.
 The seam is the interface `api`, one method per endpoint, `ctx` first:
 | Method | HTTP | Returns |
 |---|---|---|
 | `Version` | `GET /version` | version info |
 | `NodeStatus` | `GET /nodes/{node}/status` | node status |
 | `StorageStatus` | `GET /nodes/{node}/storage/{storage}/status` | storage status |
 | `ClusterResources` | `GET /cluster/resources?type=vm` | entries |
 | `VMConfig` | `GET /nodes/{node}/qemu/{vmid}/config` | config as `map[string]string` |
 | `SetVMConfig` | `PUT /nodes/{node}/qemu/{vmid}/config` | nothing (synchronous) |
 | `Clone` | `POST /nodes/{node}/qemu/{vmid}/clone` | UPID |
 | `Resize` | `PUT /nodes/{node}/qemu/{vmid}/resize` | UPID |
 | `Start` | `POST /nodes/{node}/qemu/{vmid}/status/start` | UPID |
 | `Stop` | `POST /nodes/{node}/qemu/{vmid}/status/stop` | UPID |
 | `VMStatus` | `GET /nodes/{node}/qemu/{vmid}/status/current` | status and qmpstatus |
 | `Destroy` | `DELETE /nodes/{node}/qemu/{vmid}?purge=1&destroy-unreferenced-disks=1` | UPID |
 | `TaskStatus` | `GET /nodes/{node}/tasks/{upid}/status` | status and exitstatus |
 | `AgentNetworkInterfaces` | `GET /nodes/{node}/qemu/{vmid}/agent/network-get-interfaces` | interfaces |
 | `UploadISO` | `POST /nodes/{node}/storage/{storage}/upload` (multipart) | UPID |
 | `DeleteVolume` | `DELETE /nodes/{node}/storage/{storage}/content/{volid}` | UPID |
 `httpAPI` is the production adapter; `fakeAPI` in `fake_test.go` keeps VMs,
 tasks and ISOs in memory, simulates locks (a clone leaves a nameless locked
 entry until its task completes), task polls, per-method errors and blocking,
 and records a call log. Scalars in a config are rendered as strings by the
 adapter (`cores` arrives as a JSON number, `memory` as a string).
 The transport is `tracing.RestConfigWrapper(opts...)(base)`, a parent-gated
 `otelhttp` transport, so each request is an `HTTP <METHOD>` span under
 `provider.create`, `provider.get`, `provider.delete`, `provider.list` or the
 monitor's `providerhealth.check`, and background traffic makes no root
 spans. The `observed` decorator over `api` logs one line per call and starts
 no span, so nothing is counted twice. Span attributes are otelhttp's
 standard ones; the token is in a header, which otelhttp does not record.
 Tests: a span recorder asserts the HTTP spans are children of the operation
 span and carry no token; a reflection test asserts every `api` method logs
 exactly one line.
 **Evidence:** contract §4.15, §4.18, §9; `RestConfigWrapper` and
 `parentGatedTransport` in `internal/tracing/transport.go`; F18 (HTTP/1.1
 only).
 ## 18. Timeouts (`timeouts`)
 **Answer:** `per-call`.
 | Bound | Budget |
 |---|---|
 | TCP dial | 10 s |
 | TLS handshake | 10 s |
 | one request, response included (`http.Client.Timeout`) | `requestTimeoutSeconds`, default 30 s |
 | clone task, and the wait for a VM's lock during the ownership check | 10 m |
 | upload task | 2 m |
 | resize task | 2 m |
 | start task | 2 m |
 | stop task | 2 m |
 | destroy task | 5 m |
 | seed delete task | 1 m |
 | wait for `running` after the start | 60 s |
 | task and status poll interval | 1 s |
 The budgets live in one `budgets` struct that tests shrink. A task wait that
 runs out is `ErrTransient` naming the UPID, and the task keeps running on
 Proxmox; the next attempt finds its result through the ownership check or
 the lock. A cancelled `ctx` cancels only that request: the transport keeps
 its idle connections (`MaxIdleConnsPerHost` 4, `IdleConnTimeout` 90 s), and
 no other call is disturbed. HTTP/2 is disabled explicitly
 (`ForceAttemptHTTP2: false`, an empty `TLSNextProto`).
 **Evidence:** contract §1.5 and §4.16; F16 (a full clone of the 3 GiB
 template took about two seconds; tasks finish within a second or two); F21
 (about 50 s from start to an address, which the reconciler's polling covers,
 not the running wait).
 ## 19. Concurrency
 **Answer:** shared state is the HTTP client (safe for concurrent use), the
 lazily loaded CA pool (loaded once under a `sync.Once`-style guard), and a
 lock per instance name from the shared `internal/provider/keyedlock` package
 (extracted from kvm2 in Phase 3): a one-slot semaphore per key, waitable
 with ctx, removed from the map once unused. Create and Delete for one name
 run under that name's lock, so the reconciler's Delete and GC's Delete never
 race on stop and destroy, and a Delete cannot interleave with a Create's
 steps. Get and ListByTag take no lock. Two Creates for different Proxies
 that land on the same VMID are safe: one clone wins, the other gets the
 `config file already exists` answer, waits in the ownership check for the
 winner's clone to finish, sees a different name, and probes on. Three
 reconciles, a GC sweep and a check round at once are safe.
 **Evidence:** contract §1.2 (concurrency); F4; entry 2.
 ## 20. Logging (`logging`)
 **Answer:** `tiered`. Logger from `logf.FromContext(ctx)` on every call,
 named `proxmox`, with `provider` on every line and `vmid`, `name` and
 `node` on every line about an instance.
 - V(1): one line per operation outcome (`proxmox vm ensured running`,
  `proxmox vm fetched` with state, ip and uid, `proxmox vm deleted`,
  `proxmox vms listed` with counts, `proxmox node checked` with the reason),
  one `proxmox api call` line per seam call from the decorator (`call`,
  `node`, `vmid`, `storage`, `upid`, `duration`, and on failure
  `httpStatus` and `error`), a `proxmox task failed` line with the UPID and
  `exitstatus`, and the no-op lines of Delete and ListByTag (foreign VM
  left alone, unreadable candidate skipped).
 - V(2): request parameters (without `description`'s content at V(1) and
  never the upload body), config summaries, agent addresses, the resolved
  Create parameters with `cloudInitBytes`.
 - No V(5) wire tier.
 - Never logged at any level: `CloudInit`, the token, the CA content, the
  upload body.
 **Evidence:** contract §4.17 and §9.
 ## 21. Tests (`backend-test`)
 **Answer:** `backend-test`: `integration-tag`. Unit tests run over `fakeAPI`
 behind the seam, the HTTP client over `httptest.NewTLSServer` serving the
 captured fixtures, and `classify` over the fixtures directly. The
 integration test is `proxmox_integration_test.go` behind
 `//go:build integration`, skipped unless `PROXMOX_TEST_URL`,
 `PROXMOX_TEST_TOKEN_FILE`, `PROXMOX_TEST_NODE`,
 `PROXMOX_TEST_TEMPLATE_VMID` and `PROXMOX_TEST_ISO_STORAGE` are set;
 optional `PROXMOX_TEST_CA_FILE`, `PROXMOX_TEST_INSECURE`,
 `PROXMOX_TEST_STORAGE`, `PROXMOX_TEST_BRIDGE`, `PROXMOX_TEST_VLAN_TAG`,
 `PROXMOX_TEST_VMID_MIN`, `PROXMOX_TEST_VMID_MAX`, `PROXMOX_TEST_TIMEOUT`
 (default 20m), `PROXMOX_TEST_KEEP`, `PROXMOX_TEST_LOG_LEVEL`; run with
 `make test-integration-proxmox`. It asserts, in order: `New` returns without
 a request; `Check` reports the node Ready with details; `Create` returns
 `vms/<vmid>/<name>` with the VMID in range; a second `Create` returns the
 same ID and the VM count is unchanged; `Get` reaches Running with an IPv4
 within the budget and its `UID` and `CreatedAt` equal what Create wrote;
 `ListByTag` contains it; `Delete` returns nil; `Get` is `ErrNotFound`;
 `Delete` again returns nil; no VM with the name and no `<name>-seed.iso`
 remain; an HTTP span is a child of `provider.create` and no span attribute
 carries the cloud-init text.
 Contract §11 list, with the planned test for each item (Phase 8 ticks them
 once they exist):
 - [ ] repeat `Create`: `TestCreate_repeatReturnsSameIDAndCreatesNothing`
 - [ ] `Get` after `Delete`: `TestGet_afterDeleteReturnsNotFound`
 - [ ] `Delete` twice: `TestDelete_twiceSucceeds`
 - [ ] malformed ID: `TestParseProviderID_malformed`, `TestGet_malformedIDIsPermanent`, `TestDelete_malformedIDIsPermanent`
 - [ ] unknown state maps to `StateStopped`: `TestMapState_table`, `TestGet_unknownStateIsStopped`
 - [ ] `StateRunning` requires an IP: `TestGet_runningWithoutIPIsProvisioning`, `TestGet_runningWithIPIsRunning`
 - [ ] `UID` and `CreatedAt` read back: `TestGet_readsUIDAndCreatedAtFromBackend`
 - [ ] `ListByTag` excludes unmarked instances: `TestListByTag_excludesUnmarkedVMs`, `TestListByTag_emptyUIDStillListed`
 - [ ] `Get` during an in-progress delete: `TestGet_duringDeleteStillReturnsInstance`
 - [ ] `Get` during an asynchronous operation or lock: `TestGet_lockedIsTransientNeverNotFound`, `TestGet_midCloneNamelessEntryIsTransient`
 - [ ] never started is never `StateStopped`: `TestCreate_returnsOnlyWhenRunning`
 - [ ] cancelled `ctx` returns promptly: `TestProvider_cancelledContextReturnsPromptly`, `TestClient_cancelledContextReturnsPromptly`
 - [ ] `classify` table with captured responses: `TestClassify_table`
 - [ ] Checker reasons, expired `ctx`, stable order: `TestCheck_reasonsTable`, `TestCheck_expiredContextReturnsPromptly`, `TestCheck_oneEntryPerNodeInConfigOrder`
 - [ ] sentinel tests for `CloudInit` and the credential: `TestCreate_cloudInitSentinelNeverLoggedOrInErrors`, `TestClient_tokenNeverInErrorsOrLogs`
 - [ ] child spans under the operation span: `TestTracing_httpSpansUnderOperationSpan`, `TestClient_childSpanUnderParent`
 - [ ] real-backend lifecycle test: `TestProxmox_lifecycle` and `TestProxmoxCheck_realHost` in `proxmox_integration_test.go`
 Tests that cover the Phase 0 corrections specifically, in addition to the
 plan's per-phase lists:
 - `TestCreate_resumesMidCloneVMAtDeterministicVMID`: a crash after the clone
  was submitted; the retry waits for the lock and adopts, creating nothing.
 - `TestCreate_takenVMIDInspectedBeforeProbing`: a foreign VM at attempt 0 is
  skipped; ours at attempt 0 is adopted.
 - `TestCreate_cloneAlreadyExistsRaceRunsOwnershipCheck`.
 - `TestCreate_templateWithCloudInitDriveIsPermanent`.
 - `TestCreate_seedUploadOverwritesWithoutDelete`.
 - `TestCreate_resizeWaitsForTask`, `TestCreate_resizeSkippedWhenNotLarger`.
 - `TestCreate_vlanTagAppendedKeepsMACAndOptions`.
 - `TestBootDisk_table`, with a row for `order=scsi0;ide2`.
 - `TestGet_pausedIsStoppedFromStatusCurrent`.
 - `TestGet_agentMessagesTable` (the three messages and a 200 without IPv4).
 - `TestDelete_seedDeleteIsTaskAndIdempotent`.
 - `TestClient_x509FailureIsPermanent`.
 **Evidence:** contract §11; docs/testing.md conventions.
 ## 22. Deployment
 **Answer:** a new optional Secret and its mount, two Makefile targets, and
 samples:
 - `config/manager/manager.yaml`: an optional Secret `proxmox-token` mounted
  read-only at `/etc/proxy-operator/proxmox`, keys `token` and `ca.pem`.
  `/tmp` is already an emptyDir.
 - Makefile: `proxmox-token-secret`, creating or updating `proxmox-token`
  from `PROXMOX_TOKEN_FILE` (default `$(HOME)/.config/proxmox-token`) and
  `PROXMOX_CA_FILE` (default `$(HOME)/.config/proxmox-ca.pem`), shaped like
  `kvm-ssh-secret`; and `test-integration-proxmox` (`go test -race
  -tags=integration -count=1 -v -timeout 30m -run TestProxmox
  ./internal/provider/proxmox/`).
 - `config/manager/providers_config.yaml` and
  `config/samples/providers-config.yaml`: a commented `proxmox-home` entry
  for the user's host: `url: https://192.168.0.8:8006`, the two file paths,
  `nodes: [pv1-1]`, `isoStorage: proxy-seeds`, `vlanTag: 4`,
  `templates: {debian-13: {vmid: 9001}}`, a `small` machine type.
 - `config/samples/proxy_proxmox.yaml`, listed in
  `config/samples/kustomization.yaml`: provider `proxmox-home`, zone
  `pv1-1`, machine type `small`, image `debian-13`, the Squid cloud-init
  from `proxy_kvm2.yaml`.
 - RBAC: none; the provider does not talk to the cluster's API.
 - `hack/providers-dev.yaml`: no entry (needs credentials).
 **Evidence:** contract §12.
 ## 23. Docs
 **Answer:** `docs/proxmox-provider.md` with sections: Proxmox prerequisites
 (the template recipe and what disqualifies a template, the dedicated seed
 storage, the user, both roles and the token, the proxy network with DHCP and
 internet, reserving the VMID range), Cluster-side setup (the Secret, the
 config entry, the URL-must-match-the-certificate rule), Config reference
 (entry 15), Placement (Region ignored, catalogue-edit warning), What lands on
 Proxmox (the VMID scheme, the description mark, the tag, the seed ISO, the
 config keys written, `qm unlock` for a stuck clone), Logging, Tracing,
 Timeouts, Health check (entry 16), Error classes (entry 13), Known gaps
 (entry 24), Testing. A fifth column in the three tables of
 `docs/providers-existing.md` and a proxmox section. `docs/architecture.md`:
 components row, a `proxmox` block in §5, the span note in §10, a §11 row and
 the provider count, Decisions bullets (VMID from the UID hash with the
 ownership check, the mark in the description, the seed by API upload, the
 node resolved per call). A README row. `docs/providers.md` link.
 `docs/testing.md` rows and a proxmox integration section. `CLAUDE.md`
 Gotchas: `make test-integration-proxmox`. A `CHANGELOG.md` entry after the
 user confirms.
 **Evidence:** contract §12.
 ## 24. Known limits accepted at design time
 **Answer:** ten limits, each with its consequence and workaround:
 - **Create blocks** until the VM runs, holding one of three reconcile
  workers. On the user's host that is about five seconds plus the clone;
  large templates on slow storage take longer.
 - **A clone lock that never clears** (a node crash mid-clone) makes every
  Create for that Proxy wait out the clone budget and return
  `ErrTransient`. Clear it with `qm unlock <vmid>` or destroy the VM.
 - **Seed ISOs of VMs destroyed outside the operator** are invisible to GC.
  Remove `proxy-*-seed.iso` files without a matching VM by hand.
 - **The VMID range is shared with humans.** A VM created by hand inside it
  costs a probe; a hand-made VM named like a proxy is refused as
  `ErrPermanent`. Reserve the range.
 - **Cross-node clones need shared storage** and are untested, because the
  host has one node.
 - **A VM whose agent was switched off is replaced**, not repaired (entry 7).
 - **The proxy network needs internet** for the agent install, unless the
  template ships the agent.
 - **The CA is cached** after first use; rotating it needs a restart. The
  token rotates freely.
 - **A node that stops answering** makes `Get` and `Delete` for its VMs
  `ErrTransient` until it returns, so deleting such a Proxy waits.
 - Contract §13 limits that apply unchanged: config read once per process,
  no installation identity in GC (two operators sharing a cluster and a
  VMID range delete each other's VMs), an unconfigured provider latches the
  Proxy.
 **Evidence:** contract §13; entries 2, 7, 10, 11, 14 above.