egress-proxies-operator/docs/proxmox-provider-spec.md

# Provider spec: proxmox

Filled from [provider-spec-template.md](provider-spec-template.md) against
[provider-contract.md](provider-contract.md). This is the design the
`proxmox` package is built from. Where an entry names an axis in backticks,
the answer is one of that axis's profiles in contract §4.

Backend facts come from the Phase 0 recon on the user's host, recorded in
[the execution log](plans-executions/2026-09-18-1502-proxmox-provider.md) as
facts F1 to F23, with the raw responses under
`internal/provider/proxmox/testdata/`. An Evidence line citing `F<n>` points
there. Where this spec and the plan
([docs/plans/2026-09-18-1502-proxmox-provider.md](plans/2026-09-18-1502-proxmox-provider.md))
disagree, this spec wins: it folds in the ten corrections Phase 0 found.

## Header

- **Type**: `proxmox`
- **Example configured name**: `proxmox-home`
- **Backend and API version this spec was written against**: Proxmox VE
  9.2.20 (`pve-manager/9.2.20/49318c671b82f31e`, kernel `7.0.14-17-pve`,
  QEMU 11.0.3) on the single node `pv1-1`, API at
  `https://192.168.0.8:8006/api2/json`.
- **Client**: `thin-client`: `net/http` and `encoding/json` from the standard
  library, with the transport wrapped by the already-required
  `go.opentelemetry.io/contrib/instrumentation/net/http/otelhttp`. No new
  dependency. The seed ISO uses `github.com/kdomanski/iso9660`, already
  required by kvm2.
- **Author and date**: Claude (orchestrator session), 2026-09-23.

## 1. Identity and providerID

**Answer:** `vms/<vmid>/<name>`, where `<vmid>` is the VM's numeric id and
`<name>` is `req.Name`. The ID is location-free: the node is looked up in
`GET /cluster/resources?type=vm` on every call, so a VM migrated between
nodes stays addressable. Parsing is strict: exactly three segments, the
literal `vms`, a decimal VMID without leading zeros between 100 and
999999999, and a non-empty name without `/`. Anything else is `ErrPermanent`
from `Get` and `Delete`. The name is carried so that `Get` and `Delete` can
tell our VM from another VM that later received the same VMID (entry 2). No
config value is pasted into the ID, so no ID-safety regexp is needed; node
names and storage ids are still validated at load (entry 15) because they
end up in request paths.

**Evidence:** contract §6 and §4.10 (the ID must not depend on the location
when instances can migrate); F8 (`node` is a field of every resource entry).

## 2. Primary key (`primary-key`)

**Answer:** `allocated-id`. Proxmox addresses VMs by VMID; `name` is only a
label and is not unique. The VMID is derived from the Proxy UID so a retry
lands on the same number: `vmidFor(uid, attempt)` is
`min + (fnv1a64(uid) + attempt) mod (max - min + 1)` over `vmidRange`
(entry 15). Create, under the per-name lock (entry 19):

1. Reads `/cluster/resources` once. Entries with `type: qemu`, `template: 0`
   and `name == req.Name` are the by-name candidates. Two or more is
   `ErrPermanent` naming every VMID: an admin must remove the duplicate.
   Exactly one goes to the ownership check below and is adopted or refused.
2. With none, probes `attempt = 0, 1, …` up to 32:
   - VMID free in the listing: clone onto it. If the clone fails with
     `unable to create VM <vmid>: config file already exists` (F4), someone
     took it between the listing and the clone; run the ownership check on
     it.
   - VMID taken: run the ownership check.
   - Exhausted: `ErrTransient` naming the range.
3. The **ownership check** on a VMID: if its resource entry carries a
   `lock` or has no `name` (a clone in progress lists neither, F8), wait
   until it is unlocked (entry 10, clone budget). Then read its config. It
   is **ours** when `name == req.Name` and the description carries the mark
   with `uid` equal to `req.UID` or empty: adopt it and resume Create at the
   seed step (entry 10). It is **foreign** when the name differs: move to
   the next attempt. The same name with no mark, or with a different
   non-empty `uid`, is `ErrPermanent` naming the VMID and both UIDs; the
   operator never takes over a VM it did not create.

The wait in step 3 is what closes the duplicate-VM bug Phase 0 found: after
a crash mid-clone, the retry's by-name lookup misses the nameless VM, but
attempt 0 lands on the same VMID, waits for the clone to finish, and adopts
it instead of probing past it.

`Get` and `Delete` check the name and mark at the VMID in the ID. A VM there
with another name means ours is gone and the VMID was reused: `Get` answers
`ErrNotFound`, `Delete` succeeds without touching it.

**Evidence:** F4 (taken VMID fails synchronously); F8 (a mid-clone entry has
no name); contract §4.10 (allocate-then-verify, deterministic from the UID)
and §13 (adoption not verified: closed here).

## 3. Placement mapping

**Answer:** three fields are catalogue keys checked before any backend call,
one is ignored:

| Field | Meaning | Required | Unknown value |
|---|---|---|---|
| `Zone` | a node from `nodes` | only when more than one node is configured; with one node an empty zone selects it | `ErrPermanent`: `placement.zone "x" is not in proxmox.nodes` |
| `MachineType` | a key of `machineTypes` (`cores`, `memoryMiB`, optional `diskGiB`) | yes | `ErrPermanent`: `placement.machineType "x" is not in proxmox.machineTypes` |
| `Image` | a key of `templates` (a template VMID) | yes | `ErrPermanent`: `placement.image "x" is not in proxmox.templates` |
| `Region` | ignored | | still part of the spec hash: changing it replaces the VM |

The guide documents the ignored field and that catalogue edits affect only
new VMs.

**Evidence:** contract §2.1 (reject early, MAY ignore with documentation).

## 4. Image source

**Answer:** a template VM prepared by the admin, referenced by a catalogue
key: `templates.<key>.vmid`. Create makes a **full clone** (`full=1`), so a
VM owns copies of its disks and keeps nothing linked to the template: the
template may be edited, replaced or deleted without affecting running VMs.
Before cloning, Create reads the template's config and refuses it with
`ErrPermanent` when it is not a template (`template` is not 1) or when any
disk slot holds a Proxmox cloud-init drive (a value containing `cloudinit`),
which would compete with the provider's seed (entry 5). The template must
boot from its first disk, run cloud-init with the NoCloud datasource, and
either ship `qemu-guest-agent` or be able to install it (entry 7).

Clone parameters: `newid`, `name`, `description` (the mark, entry 8),
`full=1`, `storage` when configured (empty clones onto the template's own
disk storage), and `target` when the chosen node differs from the template's
node. A cross-node clone needs the template on shared storage; this host has
one node, so that path is untested (entry 24).

Catalogue edit warning for the guide: changing a key's `vmid` changes only
VMs created afterwards. Add a new key rather than repointing one in use, so
the Proxies that name it keep a meaning you can reason about.

**Evidence:** F19 (template 9001: `scsi0`, `boot: order=scsi0`, `ide2`
free, `agent: 1`; template 9000 carries a cloud-init drive on `ide2`); F15
(a clone onto a storage without image support is refused up front); F16 (a
full clone of 9001 took about two seconds).

## 5. Cloud-init delivery

**Answer:** a NoCloud seed ISO built in-process by the shared
`internal/provider/nocloud` package (extracted from kvm2 in Phase 3):
volume label `cidata`, files `user-data` (the Proxy's `CloudInit`, byte for
byte, never parsed), `meta-data` (`instance-id: <name>` and
`local-hostname: <name>`) and `vendor-data` (the guest agent script, joined
with the configured `vendorData` as a MIME multipart when both are set;
identical to kvm2's rules). The builder stages under `os.TempDir()` (`/tmp`,
the manager's emptyDir) and writes into a `bytes.Buffer`; a seed is tens of
KiB, so buffering is within contract §1.4.

The ISO is uploaded with `POST /nodes/{node}/storage/{isoStorage}/upload`
as multipart (`content=iso`, file part `filename` named
`<name>-seed.iso`) and the returned `imgcopy` task is waited on. An upload
**overwrites** an existing file of the same name, so no delete precedes it;
the seed is rebuilt and re-uploaded on every Create attempt, since a
previous attempt may have been cut off. It is attached as
`ide2: <isoStorage>:iso/<name>-seed.iso,media=cdrom`; Proxmox then appends
`ide2` to the boot order on its own, which is harmless because the boot
disk stays first.

Only the Proxmox API is used: the storage upload endpoint accepts `iso`,
`vztmpl` or `import`, never `snippets`, so `cicustom` would need shell
access to the node. Size: the request timeout (entry 18) bounds the upload;
a seed with a multi-megabyte `vendorData` would need a larger
`requestTimeoutSeconds`.

**Evidence:** F12 (upload fields, UPID, overwrite, `.iso` suffix required);
F19 (`ide2` appended to the boot order); F21 (the seed on `ide2` installed
the agent on both the untagged network and VLAN 4); F23 (the upload is
staged in `/var/tmp` on the node, not the tmpfs `/tmp`); contract §2.1 and
§11 (byte for byte, sentinel test).

## 6. Machine type mapping

**Answer:** a catalogue: `machineTypes.<key>` is `{cores, memoryMiB,
diskGiB}`. `cores` and `memoryMiB` are written in the config step (entry
10). `diskGiB` is optional: when set and larger than the cloned boot disk,
the disk is grown with `PUT /nodes/{node}/qemu/{vmid}/resize` to the
absolute size `<diskGiB>G`, and the returned task is waited on; when unset
or not larger, the disk keeps the template's size and no call is made. The
provider never shrinks a disk. Cloud-init's growpart expands the root
filesystem at first boot.

The boot disk is the first entry of the config's `boot: order=…` that is a
disk key (`scsiN`, `virtioN`, `sataN`, `ideN`) and not a CD-ROM, else the
`bootdisk` key, else the first present of `scsi0`, `virtio0`, `sata0`,
`ide0`. Its current size comes from the `size=` option of that disk's value.
No boot disk, when `diskGiB` is set, is `ErrPermanent`.

**Evidence:** F20 (resize is a task; the same size is an `OK` no-op; a
shrink fails in the task with `shrinking disks is not supported`); F19
(`boot: order=scsi0`, becoming `order=scsi0;ide2` once the seed is
attached).

## 7. Network and IP discovery (`readiness`, `ip-source`)

**Answer:** `readiness`: `provisioning-window`. `ip-source`: `discovered`,
from the QEMU guest agent only (`GET /nodes/{node}/qemu/{vmid}/agent/network-get-interfaces`);
the API offers no ARP or DHCP-lease source. The address reported is the first
IPv4 of the first non-loopback interface, in the agent's order, that is not
loopback, link-local or unspecified.

The NIC is the template's `net0`, rewritten in the config step only when the
config sets `bridge` or `vlanTag`: the value's comma-separated options are
kept in order with the model and MAC untouched, `bridge=` and `tag=` are
replaced or appended. With neither set, `net0` is left as cloned.

The agent answers:

| Agent response | Meaning | Get reports |
|---|---|---|
| 200 with a routable IPv4 | booted | Running with that IP |
| 200 without one | network still coming up | Provisioning |
| 500 `QEMU guest agent is not running` | booting, or the agent install has not finished or failed | Provisioning |
| 500 `VM <vmid> is not running` | the VM stopped between the status read and this call | `ErrTransient` |
| 500 `No QEMU guest agent configured` | `agent: 1` was removed from the config; this VM can never report an address | Stopped, so the reconciler replaces it with a VM that has the agent on |
| anything else | cannot tell | `ErrTransient` |

The guest needs the agent. The vendor-data script installs it at first boot
unless `installGuestAgent: false`, and that install needs a package mirror:
**the proxy network must have a route to the internet**, which an egress
proxy needs anyway. Baking `qemu-guest-agent` into the template and setting
`installGuestAgent: false` removes the mirror from IP discovery. `Port` is
not used by the provider. Reachability is a prerequisite in the guide: the
operator's pod network and the clients must route to the proxy network (on
the user's host, VLAN 4, `192.168.4.0/24`).

**Evidence:** F5 (all three agent messages); F21 (agent response shape; an
address about 50 seconds after start on the untagged network and on VLAN 4);
the Phase 0 log's VLAN 4 section (no internet route meant no agent);
contract §1.3 and §2.2 (Running only with an IP).

## 8. GC marks and CreatedAt (`gc-mark`, `created-at`)

**Answer:** `gc-mark`: `written-metadata`. `created-at`: `written-at-create`.
The mark is three lines in the VM's `description`, passed in the clone call
itself, so there is no mark window:

```
proxy-operator-managed=true
proxy-operator-uid=<req.UID>
proxy-operator-created-at=<RFC3339 UTC>
```

The keys are `provider.LabelManaged`, `provider.LabelUID` and
`proxy-operator-created-at`; the managed value is `provider.LabelManagedYes`.
The parser reads `key=value` lines, tolerates CRLF, blank lines and any
other lines an admin adds, and treats the VM as managed only when the
managed value is exactly `true`. A missing `uid` line gives an empty UID. A
`created-at` that does not parse gives the zero time, which GC treats as
old; that is accepted. The clone's own `meta: …,ctime=…` is not usable:
Proxmox copies it from the template.

The config step also sets the tag `proxy_operator`, for filtering in the
Proxmox UI and as a ListByTag candidate hint (entry 12). It is not the mark:
tags cannot be passed to the clone call. Hyphens are accepted in tags on
9.2.20, but the underscore form is kept because it is valid under every
tag rule Proxmox has had. Uppercase is folded to lowercase by Proxmox.

The seed ISO carries no mark; its name `<name>-seed.iso` identifies it
(entry 11).

**Evidence:** F10 (the description survives the clone verbatim, newlines
included); F11 (the clone's `meta` is the template's); F9 (tag rules);
contract §2.1 (mark in the create call) and §4.5.

## 9. State mapping (`provisioning-state`)

**Answer:** `provisioning-state`: `create-blocks`. Create returns only once
`status/current` reports the VM running (entry 10), so a VM the provider
cloned but never started is only ever seen by a Create retry, which adopts
and starts it; the reconciler has no ID for it to `Get`.

`Get` checks in this order; the first match decides:

| Check | Source | Result |
|---|---|---|
| VMID absent | `/cluster/resources` | `ErrNotFound` |
| `lock` set, other than `suspended` | resources entry (a clone in progress also has no name) | `ErrTransient` |
| `lock: suspended` (hibernated to disk) | resources entry | Stopped |
| `status: unknown` (node not reachable) | resources entry | `ErrTransient` |
| config `does not exist` | `GET …/config` | `ErrNotFound` (deleted between the two calls) |
| config `lock` set | config | `ErrTransient` |
| `name` differs from the ID's name | config | `ErrNotFound`, logged: the VMID was reused |
| no mark in the description | config | `ErrNotFound`, logged: not ours |
| `status: stopped` | `status/current` | Stopped |
| `status: running`, `qmpstatus: running` | `status/current` | the agent decides (entry 7) |
| `status: running`, `qmpstatus: prelaunch` | `status/current` | Provisioning |
| `status: running`, any other `qmpstatus` (`paused`, `suspended`, `io-error`, …) | `status/current` | Stopped |
| any other `status` | `status/current` | Stopped |

`status/current` is read, not only the resources entry, because a paused VM
shows as plain `running` in `/cluster/resources`. `UID` and `CreatedAt` come
from the description on every call. A running `Get` makes four calls:
resources, config, status and agent.

**Evidence:** F8 (resources fields; mid-clone entries lack a name, carry
`lock: clone` and `status: unknown`); F14 (stopped and paused as seen by
`status/current` and by resources); F2 (the `does not exist` message);
contract §2.2 and §4.11.

## 10. Create: blocking and task tracking (`create-return`, `async-ops`)

**Answer:** `create-return`: `blocking`. `async-ops`: `task-handle` and
`backend-lock`. Clone, upload, resize, start, stop, destroy and volume
delete each return a UPID; the provider polls
`GET /nodes/{node}/tasks/{upid}/status` every second until `status` is
`stopped`, then reads `exitstatus`: `OK` is success, anything else is
`ErrTransient` carrying that text. The node comes from the UPID
(`UPID:<node>:…`). The handle is held within the call only: Create blocks
until the VM runs, so no handle outlives it, and a failure surfaces as
Create's own error with the task's message. Locks: a VM locked by a task is
waited on in Create (the ownership check) and reported as `ErrTransient` by
`Get` and `Delete`. A start issued while a clone holds the lock queues
behind it rather than failing; the provider never relies on that, because it
waits for the clone task first.

Steps, each idempotent, in order, under the per-name lock (entry 19):

1. Validate placement (entry 3). No backend call yet.
2. `GET /cluster/resources`. Find the template's node; a missing template
   is `ErrPermanent`. Read the template's config and refuse it as entry 4
   says.
3. Find or allocate the VMID (entry 2). An adopted VM that is running
   returns its ID at once. An adopted VM that is stopped continues at step 5.
4. Clone and wait (clone budget).
5. Build the seed and upload it to the VM's node, overwriting; wait
   (upload budget).
6. `PUT /nodes/{node}/qemu/{vmid}/config` (synchronous) with `cores`,
   `memory`, `agent=1`, `tags=proxy_operator`,
   `ide2=<isoStorage>:iso/<name>-seed.iso,media=cdrom` and, when configured,
   the rewritten `net0` (entry 7).
7. Resize when `diskGiB` is larger than the boot disk (entry 6); wait
   (resize budget).
8. `POST …/status/start`; wait (start budget).
9. Poll `status/current` every second until `status: running` and
   `qmpstatus: running`, up to the running budget; expiry is
   `ErrTransient`, and so is a `stopped` status after the start.
10. Return `vms/<vmid>/<name>`.

What a retry finds:

| Previous attempt stopped after | The retry |
|---|---|
| nothing | allocates, clones |
| submitting the clone | finds the VMID locked and nameless, waits, adopts, continues at 5 |
| the clone | adopts the stopped VM, continues at 5 |
| any of steps 5 to 7 | adopts, repeats 5 to 7 (upload overwrites, config is a replace, resize is skipped once large enough) |
| the start | adopts the running VM, returns its ID |

A Proxmox `does not exist` inside Create (a VM or volume vanishing between
steps) is `ErrTransient`, never `ErrNotFound`. The log line for a Create
records `cloudInitBytes`, never the content.

**Evidence:** F3 (a start queues behind the clone lock); F7 (a config write
under the clone lock is refused with `VM is locked (clone)`); F16 (task
status fields); F20 (resize is a task); the Phase 0 log's VLAN 4 rerun
(clone to started VM in about five seconds); contract §2.1, §4.2, §4.12 and
§5 ("do not return NotFound from Create").

## 11. Delete and secondary resources (`delete-return`, `secondary-resources`)

**Answer:** `delete-return`: `waits`. `secondary-resources`: `explicit` for
the seed ISO; the VM's disks are removed by the destroy call itself, with
`purge=1&destroy-unreferenced-disks=1`. Under the per-name lock:

1. Parse the ID (`ErrPermanent` if malformed).
2. `/cluster/resources`. VMID absent: skip to step 7, trying every configured
   node because the VM's node is no longer known.
3. `lock` set or `status: unknown`: `ErrTransient`. A clone or another task
   still running finishes first; the reconciler calls Delete again after
   `DeletionPoll`.
4. Read the config. `does not exist`: skip to step 7. A different name, or
   no mark: the VM is not ours; log it and skip to step 7 without touching
   it.
5. Status other than `stopped`: `POST …/status/stop`, wait (stop budget).
   Stop works on a paused VM too.
6. `DELETE /nodes/{node}/qemu/{vmid}?purge=1&destroy-unreferenced-disks=1`,
   wait (destroy budget). `does not exist` is success.
7. `DELETE /nodes/{node}/storage/{isoStorage}/content/<isoStorage>:iso/<name>-seed.iso`,
   wait (seed-delete budget). A missing volume is still a successful task,
   so no message is matched. On the try-every-node path a node whose
   request fails is logged and skipped.
8. Return nil.

Everything is gone when Delete returns nil, so `Get` afterwards is
`ErrNotFound`. A concurrent `Get` during a Delete keeps seeing the VM (as
Stopped once it is stopped) until the destroy task removes its config,
because destroying sets no lock and leaves the resource entry in place until
the end. A second Delete waits on the lock, then finds nothing.

GC blind spot: a VM destroyed outside the operator leaves its seed ISO, which
ListByTag cannot see. The guide says seeds are named `proxy-*-seed.iso` on
`isoStorage` and any without a matching VM may be removed.

**Evidence:** F2 (a second destroy gives `does not exist`); F13 (volume
delete is an `imgdel` task, needs `Datastore.Allocate`, and succeeds on a
missing volume); the Phase 0 capture of resources and config mid-destroy
(`cluster_resources_during_destroy.json`, `config_during_destroy.json`);
contract §2.3.

## 12. ListByTag scope and partial failure (`list-partial`)

**Answer:** `fail-whole` for the one listing, with single unreadable VMs
skipped. Scope: every VM the cluster knows, from one
`GET /cluster/resources?type=vm`, which covers every node, including
unreachable ones, whose guests Proxmox lists with `status: unknown` (entry
13). Candidates are entries
with `type: qemu`, `template: 0`, and either the tag `proxy_operator` (tags
split on `;`, `,` or space) or a name starting with `proxy-`. A nameless
entry (a clone in progress) is not a candidate: its Create is live and it is
younger than `MinAge`. For each candidate the config is read and the
description parsed; managed VMs become an `Instance` with `ID`, `UID` and
`CreatedAt` only, which is all GC reads (contract §2.4); no status or agent
call is made. A candidate whose config cannot be read (a node down, a VM
destroyed mid-sweep) is skipped with a V(1) line, unless ctx ended, which
fails the call. The resources call failing fails the whole call. Filtering
by the mark is client-side; the tag narrows the candidates. Cost: one
resources call plus one config read per candidate per sweep.

**Evidence:** F8 (resources fields, tags separated by `;`, nameless mid-clone
entries); F9 (tags case-folded); contract §2.4 and §4.7.

## 13. Error classification (`not-found`, `quota`)

**Answer:** `not-found`: `error-code`, the 500 message
`Configuration file 'nodes/<node>/qemu-server/<vmid>.conf' does not exist`,
plus `list-and-miss` on `/cluster/resources`. The listing can miss a VM only
when it is gone: it is built from the cluster filesystem, which holds every
guest's config cluster-wide, and Proxmox lists the guests of an unreachable
node with `status: unknown` rather than dropping them. That last point is
Proxmox's documented design, not something the single-node host could show;
Phase 10 or a second node verifies it. `quota`: `absent`. The thin `local-lvm` storage
overprovisions, so no out-of-space response could be produced; capacity
failures are `ErrTransient`, and the Checker reports free space in
`Details` instead (entry 16).

The client carries each failure as `apiError{Status, Message, Errors}`:
`Message` is the body's `message` with the trailing newline trimmed, or the
reason phrase when the body has none (a 401); `Errors` is the 400 body's
per-parameter map. One `classify(op string, err error)`:

| Backend response | Class | Why |
|---|---|---|
| 401 `Authentication failed!`, 401 `No ticket` | Permanent | the token or its secret is wrong; fixed in the Secret |
| 403 `Permission check failed (<path>, <privilege>)` | Permanent | the token lacks a privilege; the message names it |
| 400 `Parameter verification failed.` with `errors` | Permanent | the request is wrong; the message includes the field errors |
| other 4xx | Permanent | a request Proxmox will never accept |
| 500 `Configuration file '…' does not exist` | NotFound (Transient when `op == "create"`) | the VM is gone |
| 500 `VM is locked (<lock>)` | Transient | a task holds the VM |
| 500 `unable to create VM <vmid>: config file already exists` | handled by the allocator (entry 2); Transient if it escapes | the VMID is taken |
| 500 `QEMU guest agent is not running`, `No QEMU guest agent configured`, `VM <vmid> is not running` | handled by `Get` (entry 7); Transient if they escape | agent state |
| any other 5xx, including pveproxy's 595 and 596 | Transient | unknown means retry |
| task `exitstatus` other than `OK` | Transient, with the text | the task failed; the next attempt retries |
| TLS verification failure (`x509.UnknownAuthorityError`, `x509.HostnameError`, `x509.CertificateInvalidError`) | Permanent | wrong URL or CA file; the admin must act |
| token file missing (`fs.ErrNotExist`) | Transient | the Secret has not landed yet |
| token file present but not `<user>@<realm>!<id>=<secret>` | Permanent | wrong content in the Secret |
| `context.Canceled`, `context.DeadlineExceeded`, `net.OpError`, `io.EOF`, `io.ErrUnexpectedEOF` | Transient | network or cancellation |
| an error already wrapped with a class | that class | validation and builders classify at the source |
| anything else | Transient | contract default |

Every matched string is backed by a captured response under `testdata/`;
the table test loads them rather than repeating the text. Matching is on
fixed substrings anchored to the 500 status: `does not exist`,
`VM is locked`, `config file already exists`, `QEMU guest agent is not
running`, `No QEMU guest agent configured`, and for the stopped VM the whole
message against `^VM [0-9]+ is not running$`. The agent messages are tested
in that order, because the last is a suffix of the first.

**Evidence:** F1 (error text in the reason phrase and the body; 401 has no
body); F2, F4, F5, F7 (the 500 messages); F6 (no out-of-space response
could be produced on thin storage); F22 (both 401 texts); fixtures
`err_forbidden.json`, `err_bad_param.json`, `err_tag_invalid.json`,
`err_upload_bad_name.json`; the Phase 0 TLS finding (`192.168.2.8` not in
the node certificate); contract §5.

## 14. Credentials and TLS (`credentials`)

**Answer:** `mounted-files`. The config names `tokenFile`, one line
`<user>@<realm>!<tokenid>=<secret>`, sent as
`Authorization: PVEAPIToken=<that line>`. It is expected in the optional
Secret `proxmox-token`, key `token`, mounted at
`/etc/proxy-operator/proxmox`. The file is read on **every request**, so
rotating the Secret takes effect without a restart. `New` stats it and logs
a warning when it is missing, but never fails on it and never dials.

TLS: `ca-file`. `caFile` names a PEM bundle, typically the node's
`/etc/pve/pve-root-ca.pem`, carried in the same Secret as key `ca.pem`. It
is loaded on first use and cached, so rotating the CA needs a restart; a
missing CA file is `ErrTransient` until it lands, one that does not parse is
`ErrPermanent`. Without `caFile` the system roots are used. The URL's host
must be a name or address in the node certificate's SANs: on the user's
host that is `192.168.0.8` or `pv1-1.home.lab`, not `192.168.2.8`.
`insecureSkipTLSVerify: true` disables verification and is logged as a
warning by `New`.

Minimum privileges, as verified on 9.2.20, for `proxy-operator@pve` with a
token without privilege separation:

- a role on `/` with `VM.Allocate VM.Clone VM.Audit VM.Config.CDROM
  VM.Config.CPU VM.Config.Memory VM.Config.Disk VM.Config.Network
  VM.Config.Options VM.PowerMgmt VM.GuestAgent.Audit Datastore.Audit
  Datastore.AllocateSpace Datastore.AllocateTemplate Sys.Audit SDN.Use`;
- a role on `/storage/<isoStorage>` with `Datastore.Allocate
  Datastore.AllocateTemplate Datastore.AllocateSpace Datastore.Audit`,
  because deleting a volume needs `Datastore.Allocate` and a grant on a more
  specific path replaces the inherited one. A dedicated ISO-only storage
  keeps that privilege away from backups.

`VM.Monitor` no longer exists in PVE 9 and is not needed.
`VM.Config.Cloudinit` is not needed. `SDN.Use` on `/` covers a VLAN tag on
`vmbr0`.

Hygiene: the token never appears in logs, span attributes or error strings.
`apiError.Error()` carries status and message only, never headers or URLs.
Tested with a sentinel secret that a test server echoes in a 401 body.

**Evidence:** F17 (privileges exercised and the one missing); the Phase 0
log's seed storage section; F22 (auth failure texts); contract §7.

## 15. Config block

**Answer:** a new block `proxmox:` with the Go type `provider.ProxmoxConfig`
in `internal/provider/config_proxmox.go`:

| Field | Required | Default | Validation | Ends up in |
|---|---|---|---|---|
| `url` | yes | | `https` URL with a host and no path other than `/` | request base, `…/api2/json` appended |
| `tokenFile` | yes | | non-empty | `Authorization` header |
| `caFile` | | system roots | non-empty when set | TLS roots |
| `insecureSkipTLSVerify` | | false | | TLS, warning at `New` |
| `nodes` | yes, at least one | | unique, `^[a-z0-9]([a-z0-9-]*[a-z0-9])?$` | request paths, clone `target` |
| `isoStorage` | yes | | Proxmox storage id `^[a-zA-Z][a-zA-Z0-9_.-]*$` | upload and delete paths, `ide2` |
| `storage` | | the template's disk storage | storage id regexp when set | clone `storage` |
| `bridge` | | as cloned | non-empty when set | `net0` `bridge=` |
| `vlanTag` | | as cloned | 1 to 4094 when set | `net0` `tag=` |
| `vmidRange.min`, `vmidRange.max` | | 100000, 199999 | min at least 100, max at most 999999999, min below max | `vmidFor` |
| `templates.<key>.vmid` | at least one template | | at least 100 | clone source |
| `machineTypes.<key>` | at least one | | `cores` and `memoryMiB` above zero, `diskGiB` not negative | config step, resize |
| `requestTimeoutSeconds` | | 30 | not negative | HTTP client timeout |
| `vendorData` | | | | seed `vendor-data` |
| `installGuestAgent` | | true | | seed `vendor-data` |

`isoStorage` has no default on purpose: the token must be allowed to delete
volumes there (entry 14), so the choice must be explicit. `nodes` holds node
names as Proxmox knows them (`pv1-1`), not addresses; the one URL reaches the
whole cluster.

Wiring: `ProviderConfig.Proxmox *ProxmoxConfig` with tag
`proxmox,omitempty`; `setBlocks` reports `proxmox`; the allow-list in
`Config.validate` and the constructor map in `cmd/main.go` both gain
`proxmox` in the same commit; `validateProxmox(prefix, c)` in
`config_proxmox.go` with `config_proxmox_test.go`.

**Evidence:** contract §10; the Phase 0 decisions (dedicated seed storage,
`vlanTag`, URL at `192.168.0.8`); F13 (why `isoStorage` needs
`Datastore.Allocate`).

## 16. Checker

**Answer:** one target per `nodes` entry, `Name` the node, `Address` the
URL, checked concurrently, returned in config order. The check shares the
provider's HTTP client; each request is independent, so a check that runs
out of time cancels only its own requests. Per node:

1. `GET /version` (`pveVersion`).
2. `GET /nodes/{node}/status` (the node answers; `kernel`).
3. `GET /nodes/{node}/storage/{isoStorage}/status`: the storage exists, is
   active, and lists `iso` in its content (`isoStorageAvailGiB`).
4. `GET /cluster/resources?type=vm`: every configured template exists and
   is a template (`templates` as `<found>/<configured>`).

Reasons, shared first: `Reachable`; `Timeout` (the round's ctx ended,
whichever request was running); `Unreachable` (TCP connect or TLS handshake
failed for a network reason, or pveproxy answered 595 or 596);
`Unauthorized` (401); `Forbidden` (403, the message names the privilege);
`APIError` (any other failure). Provider-specific, each fixed differently by
an admin:

- `CertificateInvalid`: TLS verification failed. Fix `url` or `caFile`, not
  the host.
- `CredentialsUnavailable`: the token file is missing or malformed. Fix the
  Secret, not Proxmox.
- `StorageNotFound`: `isoStorage` is missing, inactive, or has no `iso`
  content. Fix the storage on the node.
- `TemplateNotFound`: a configured template VMID does not exist or is not a
  template. Fix the template or the catalogue.

Details: `pveVersion`, `kernel`, `tokenID` (the part before `=`, never the
secret), `isoStorage`, `isoStorageAvailGiB`, `templates`, filled as far as
the check got. Cost: four requests per node per round. The `status.type`
comment in `api/v1alpha1/providerhealth_types.go` gains `proxmox` and the
CRD is regenerated.

**Evidence:** contract §8; fixtures `version.json`, `node_status.json`,
`storage_status.json`, `cluster_resources.json`.

## 17. Client and child spans (`client-kind`, `child-spans`)

**Answer:** `client-kind`: `thin-client`. `child-spans`: `transport-wrapper`.

The seam is the interface `api`, one method per endpoint, `ctx` first:

| Method | HTTP | Returns |
|---|---|---|
| `Version` | `GET /version` | version info |
| `NodeStatus` | `GET /nodes/{node}/status` | node status |
| `StorageStatus` | `GET /nodes/{node}/storage/{storage}/status` | storage status |
| `ClusterResources` | `GET /cluster/resources?type=vm` | entries |
| `VMConfig` | `GET /nodes/{node}/qemu/{vmid}/config` | config as `map[string]string` |
| `SetVMConfig` | `PUT /nodes/{node}/qemu/{vmid}/config` | nothing (synchronous) |
| `Clone` | `POST /nodes/{node}/qemu/{vmid}/clone` | UPID |
| `Resize` | `PUT /nodes/{node}/qemu/{vmid}/resize` | UPID |
| `Start` | `POST /nodes/{node}/qemu/{vmid}/status/start` | UPID |
| `Stop` | `POST /nodes/{node}/qemu/{vmid}/status/stop` | UPID |
| `VMStatus` | `GET /nodes/{node}/qemu/{vmid}/status/current` | status and qmpstatus |
| `Destroy` | `DELETE /nodes/{node}/qemu/{vmid}?purge=1&destroy-unreferenced-disks=1` | UPID |
| `TaskStatus` | `GET /nodes/{node}/tasks/{upid}/status` | status and exitstatus |
| `AgentNetworkInterfaces` | `GET /nodes/{node}/qemu/{vmid}/agent/network-get-interfaces` | interfaces |
| `UploadISO` | `POST /nodes/{node}/storage/{storage}/upload` (multipart) | UPID |
| `DeleteVolume` | `DELETE /nodes/{node}/storage/{storage}/content/{volid}` | UPID |

`httpAPI` is the production adapter; `fakeAPI` in `fake_test.go` keeps VMs,
tasks and ISOs in memory, simulates locks (a clone leaves a nameless locked
entry until its task completes), task polls, per-method errors and blocking,
and records a call log. Scalars in a config are rendered as strings by the
adapter (`cores` arrives as a JSON number, `memory` as a string).

The transport is `tracing.RestConfigWrapper(opts...)(base)`, a parent-gated
`otelhttp` transport, so each request is an `HTTP <METHOD>` span under
`provider.create`, `provider.get`, `provider.delete`, `provider.list` or the
monitor's `providerhealth.check`, and background traffic makes no root
spans. The `observed` decorator over `api` logs one line per call and starts
no span, so nothing is counted twice. Span attributes are otelhttp's
standard ones; the token is in a header, which otelhttp does not record.
Tests: a span recorder asserts the HTTP spans are children of the operation
span and carry no token; a reflection test asserts every `api` method logs
exactly one line.

**Evidence:** contract §4.15, §4.18, §9; `RestConfigWrapper` and
`parentGatedTransport` in `internal/tracing/transport.go`; F18 (HTTP/1.1
only).

## 18. Timeouts (`timeouts`)

**Answer:** `per-call`.

| Bound | Budget |
|---|---|
| TCP dial | 10 s |
| TLS handshake | 10 s |
| one request, response included (`http.Client.Timeout`) | `requestTimeoutSeconds`, default 30 s |
| clone task, and the wait for a VM's lock during the ownership check | 10 m |
| upload task | 2 m |
| resize task | 2 m |
| start task | 2 m |
| stop task | 2 m |
| destroy task | 5 m |
| seed delete task | 1 m |
| wait for `running` after the start | 60 s |
| task and status poll interval | 1 s |

The budgets live in one `budgets` struct that tests shrink. A task wait that
runs out is `ErrTransient` naming the UPID, and the task keeps running on
Proxmox; the next attempt finds its result through the ownership check or
the lock. A cancelled `ctx` cancels only that request: the transport keeps
its idle connections (`MaxIdleConnsPerHost` 4, `IdleConnTimeout` 90 s), and
no other call is disturbed. HTTP/2 is disabled explicitly
(`ForceAttemptHTTP2: false`, an empty `TLSNextProto`).

**Evidence:** contract §1.5 and §4.16; F16 (a full clone of the 3 GiB
template took about two seconds; tasks finish within a second or two); F21
(about 50 s from start to an address, which the reconciler's polling covers,
not the running wait).

## 19. Concurrency

**Answer:** shared state is the HTTP client (safe for concurrent use), the
lazily loaded CA pool (loaded once under a `sync.Once`-style guard), and a
lock per instance name from the shared `internal/provider/keyedlock` package
(extracted from kvm2 in Phase 3): a one-slot semaphore per key, waitable
with ctx, removed from the map once unused. Create and Delete for one name
run under that name's lock, so the reconciler's Delete and GC's Delete never
race on stop and destroy, and a Delete cannot interleave with a Create's
steps. Get and ListByTag take no lock. Two Creates for different Proxies
that land on the same VMID are safe: one clone wins, the other gets the
`config file already exists` answer, waits in the ownership check for the
winner's clone to finish, sees a different name, and probes on. Three
reconciles, a GC sweep and a check round at once are safe.

**Evidence:** contract §1.2 (concurrency); F4; entry 2.

## 20. Logging (`logging`)

**Answer:** `tiered`. Logger from `logf.FromContext(ctx)` on every call,
named `proxmox`, with `provider` on every line and `vmid`, `name` and
`node` on every line about an instance.

- V(1): one line per operation outcome (`proxmox vm ensured running`,
  `proxmox vm fetched` with state, ip and uid, `proxmox vm deleted`,
  `proxmox vms listed` with counts, `proxmox node checked` with the reason),
  one `proxmox api call` line per seam call from the decorator (`call`,
  `node`, `vmid`, `storage`, `upid`, `duration`, and on failure
  `httpStatus` and `error`), a `proxmox task failed` line with the UPID and
  `exitstatus`, and the no-op lines of Delete and ListByTag (foreign VM
  left alone, unreadable candidate skipped).
- V(2): request parameters (without `description`'s content at V(1) and
  never the upload body), config summaries, agent addresses, the resolved
  Create parameters with `cloudInitBytes`.
- No V(5) wire tier.
- Never logged at any level: `CloudInit`, the token, the CA content, the
  upload body.

**Evidence:** contract §4.17 and §9.

## 21. Tests (`backend-test`)

**Answer:** `backend-test`: `integration-tag`. Unit tests run over `fakeAPI`
behind the seam, the HTTP client over `httptest.NewTLSServer` serving the
captured fixtures, and `classify` over the fixtures directly. The
integration test is `proxmox_integration_test.go` behind
`//go:build integration`, skipped unless `PROXMOX_TEST_URL`,
`PROXMOX_TEST_TOKEN_FILE`, `PROXMOX_TEST_NODE`,
`PROXMOX_TEST_TEMPLATE_VMID` and `PROXMOX_TEST_ISO_STORAGE` are set;
optional `PROXMOX_TEST_CA_FILE`, `PROXMOX_TEST_INSECURE`,
`PROXMOX_TEST_STORAGE`, `PROXMOX_TEST_BRIDGE`, `PROXMOX_TEST_VLAN_TAG`,
`PROXMOX_TEST_VMID_MIN`, `PROXMOX_TEST_VMID_MAX`, `PROXMOX_TEST_TIMEOUT`
(default 20m), `PROXMOX_TEST_KEEP`, `PROXMOX_TEST_LOG_LEVEL`; run with
`make test-integration-proxmox`. It asserts, in order: `New` returns without
a request; `Check` reports the node Ready with details; `Create` returns
`vms/<vmid>/<name>` with the VMID in range; a second `Create` returns the
same ID and the VM count is unchanged; `Get` reaches Running with an IPv4
within the budget and its `UID` and `CreatedAt` equal what Create wrote;
`ListByTag` contains it; `Delete` returns nil; `Get` is `ErrNotFound`;
`Delete` again returns nil; no VM with the name and no `<name>-seed.iso`
remain; an HTTP span is a child of `provider.create` and no span attribute
carries the cloud-init text.

Contract §11 list, with the planned test for each item (Phase 8 ticks them
once they exist):

- [ ] repeat `Create`: `TestCreate_repeatReturnsSameIDAndCreatesNothing`
- [ ] `Get` after `Delete`: `TestGet_afterDeleteReturnsNotFound`
- [ ] `Delete` twice: `TestDelete_twiceSucceeds`
- [ ] malformed ID: `TestParseProviderID_malformed`, `TestGet_malformedIDIsPermanent`, `TestDelete_malformedIDIsPermanent`
- [ ] unknown state maps to `StateStopped`: `TestMapState_table`, `TestGet_unknownStateIsStopped`
- [ ] `StateRunning` requires an IP: `TestGet_runningWithoutIPIsProvisioning`, `TestGet_runningWithIPIsRunning`
- [ ] `UID` and `CreatedAt` read back: `TestGet_readsUIDAndCreatedAtFromBackend`
- [ ] `ListByTag` excludes unmarked instances: `TestListByTag_excludesUnmarkedVMs`, `TestListByTag_emptyUIDStillListed`
- [ ] `Get` during an in-progress delete: `TestGet_duringDeleteStillReturnsInstance`
- [ ] `Get` during an asynchronous operation or lock: `TestGet_lockedIsTransientNeverNotFound`, `TestGet_midCloneNamelessEntryIsTransient`
- [ ] never started is never `StateStopped`: `TestCreate_returnsOnlyWhenRunning`
- [ ] cancelled `ctx` returns promptly: `TestProvider_cancelledContextReturnsPromptly`, `TestClient_cancelledContextReturnsPromptly`
- [ ] `classify` table with captured responses: `TestClassify_table`
- [ ] Checker reasons, expired `ctx`, stable order: `TestCheck_reasonsTable`, `TestCheck_expiredContextReturnsPromptly`, `TestCheck_oneEntryPerNodeInConfigOrder`
- [ ] sentinel tests for `CloudInit` and the credential: `TestCreate_cloudInitSentinelNeverLoggedOrInErrors`, `TestClient_tokenNeverInErrorsOrLogs`
- [ ] child spans under the operation span: `TestTracing_httpSpansUnderOperationSpan`, `TestClient_childSpanUnderParent`
- [ ] real-backend lifecycle test: `TestProxmox_lifecycle` and `TestProxmoxCheck_realHost` in `proxmox_integration_test.go`

Tests that cover the Phase 0 corrections specifically, in addition to the
plan's per-phase lists:

- `TestCreate_resumesMidCloneVMAtDeterministicVMID`: a crash after the clone
  was submitted; the retry waits for the lock and adopts, creating nothing.
- `TestCreate_takenVMIDInspectedBeforeProbing`: a foreign VM at attempt 0 is
  skipped; ours at attempt 0 is adopted.
- `TestCreate_cloneAlreadyExistsRaceRunsOwnershipCheck`.
- `TestCreate_templateWithCloudInitDriveIsPermanent`.
- `TestCreate_seedUploadOverwritesWithoutDelete`.
- `TestCreate_resizeWaitsForTask`, `TestCreate_resizeSkippedWhenNotLarger`.
- `TestCreate_vlanTagAppendedKeepsMACAndOptions`.
- `TestBootDisk_table`, with a row for `order=scsi0;ide2`.
- `TestGet_pausedIsStoppedFromStatusCurrent`.
- `TestGet_agentMessagesTable` (the three messages and a 200 without IPv4).
- `TestDelete_seedDeleteIsTaskAndIdempotent`.
- `TestClient_x509FailureIsPermanent`.

**Evidence:** contract §11; docs/testing.md conventions.

## 22. Deployment

**Answer:** a new optional Secret and its mount, two Makefile targets, and
samples:

- `config/manager/manager.yaml`: an optional Secret `proxmox-token` mounted
  read-only at `/etc/proxy-operator/proxmox`, keys `token` and `ca.pem`.
  `/tmp` is already an emptyDir.
- Makefile: `proxmox-token-secret`, creating or updating `proxmox-token`
  from `PROXMOX_TOKEN_FILE` (default `$(HOME)/.config/proxmox-token`) and
  `PROXMOX_CA_FILE` (default `$(HOME)/.config/proxmox-ca.pem`), shaped like
  `kvm-ssh-secret`; and `test-integration-proxmox` (`go test -race
  -tags=integration -count=1 -v -timeout 30m -run TestProxmox
  ./internal/provider/proxmox/`).
- `config/manager/providers_config.yaml` and
  `config/samples/providers-config.yaml`: a commented `proxmox-home` entry
  for the user's host: `url: https://192.168.0.8:8006`, the two file paths,
  `nodes: [pv1-1]`, `isoStorage: proxy-seeds`, `vlanTag: 4`,
  `templates: {debian-13: {vmid: 9001}}`, a `small` machine type.
- `config/samples/proxy_proxmox.yaml`, listed in
  `config/samples/kustomization.yaml`: provider `proxmox-home`, zone
  `pv1-1`, machine type `small`, image `debian-13`, the Squid cloud-init
  from `proxy_kvm2.yaml`.
- RBAC: none; the provider does not talk to the cluster's API.
- `hack/providers-dev.yaml`: no entry (needs credentials).

**Evidence:** contract §12.

## 23. Docs

**Answer:** `docs/proxmox-provider.md` with sections: Proxmox prerequisites
(the template recipe and what disqualifies a template, the dedicated seed
storage, the user, both roles and the token, the proxy network with DHCP and
internet, reserving the VMID range), Cluster-side setup (the Secret, the
config entry, the URL-must-match-the-certificate rule), Config reference
(entry 15), Placement (Region ignored, catalogue-edit warning), What lands on
Proxmox (the VMID scheme, the description mark, the tag, the seed ISO, the
config keys written, `qm unlock` for a stuck clone), Logging, Tracing,
Timeouts, Health check (entry 16), Error classes (entry 13), Known gaps
(entry 24), Testing. A fifth column in the three tables of
`docs/providers-existing.md` and a proxmox section. `docs/architecture.md`:
components row, a `proxmox` block in §5, the span note in §10, a §11 row and
the provider count, Decisions bullets (VMID from the UID hash with the
ownership check, the mark in the description, the seed by API upload, the
node resolved per call). A README row. `docs/providers.md` link.
`docs/testing.md` rows and a proxmox integration section. `CLAUDE.md`
Gotchas: `make test-integration-proxmox`. A `CHANGELOG.md` entry after the
user confirms.

**Evidence:** contract §12.

## 24. Known limits accepted at design time

**Answer:** ten limits, each with its consequence and workaround:

- **Create blocks** until the VM runs, holding one of three reconcile
  workers. On the user's host that is about five seconds plus the clone;
  large templates on slow storage take longer.
- **A clone lock that never clears** (a node crash mid-clone) makes every
  Create for that Proxy wait out the clone budget and return
  `ErrTransient`. Clear it with `qm unlock <vmid>` or destroy the VM.
- **Seed ISOs of VMs destroyed outside the operator** are invisible to GC.
  Remove `proxy-*-seed.iso` files without a matching VM by hand.
- **The VMID range is shared with humans.** A VM created by hand inside it
  costs a probe; a hand-made VM named like a proxy is refused as
  `ErrPermanent`. Reserve the range.
- **Cross-node clones need shared storage** and are untested, because the
  host has one node.
- **A VM whose agent was switched off is replaced**, not repaired (entry 7).
- **The proxy network needs internet** for the agent install, unless the
  template ships the agent.
- **The CA is cached** after first use; rotating it needs a restart. The
  token rotates freely.
- **A node that stops answering** makes `Get` and `Delete` for its VMs
  `ErrTransient` until it returns, so deleting such a Proxy waits.
- Contract §13 limits that apply unchanged: config read once per process,
  no installation identity in GC (two operators sharing a cluster and a
  VMID range delete each other's VMs), an unconfigured provider latches the
  Proxy.

**Evidence:** contract §13; entries 2, 7, 10, 11, 14 above.