# Execution: proxmox, a Proxmox VE provider built from the provider contract

Pairs with [docs/plans/2026-09-18-1502-proxmox-provider.md](../plans/2026-09-18-1502-proxmox-provider.md).

## Status

- [x] Phase 0 — Plan file, branch, host recon, fixtures
- [ ] Phase 1 — The spec (`docs/proxmox-provider-spec.md`)
- [ ] Phase 2 — Config block, registration, package skeleton
- [ ] Phase 3 — Extract `nocloud` and `keyedlock` from kvm2
- [ ] Phase 4 — Seam, HTTP client, decorator, fake
- [ ] Phase 5 — Get, ListByTag, task helpers
- [ ] Phase 6 — Create
- [ ] Phase 7 — Delete, Checker, CRD comment
- [ ] Phase 8 — Contract and observability sweep, integration test, testing.md
- [ ] Phase 9 — Deployment and documentation
- [ ] Phase 10 — Review, real-host validation, changelog, MR

## Phase 0 — Plan file, branch, host recon, fixtures

The plan was committed alone on `main` on 2026-09-18 (`708989c`, then
`2167819` retargeting it at PVE 9.2). `origin/main` had not moved in the
meantime, so the branch was cut straight from local `main`:

```bash
git fetch -q origin && git rev-list --left-right --count origin/main...main   # 0 2
git switch -c feat/proxmox-provider
```

The host is **pv1-1**, a single node (no cluster peers), `pve-manager/9.2.20`,
kernel `7.0.14-17-pve`, QEMU 11.0. The user prepared the token
(`~/.config/proxmox-token`, privsep off) and the CA (`~/.config/proxmox-ca.pem`)
before the session.

**TLS: 192.168.2.8 is not in the node certificate.** The first probe failed
with curl exit 60. The CA is right (`openssl s_client -CAfile` verifies the
chain), but the leaf's SANs are `127.0.0.1, ::1, localhost, 192.168.0.8,
pv1-1, pv1-1.home.lab`, and neither name resolves on the workstation.
192.168.0.8 and 192.168.2.8 present the same certificate fingerprint: the
node sits on `vmbr0` (192.168.0.8/24, untagged) with VLAN interfaces
`vmbr0.2` (192.168.2.8/24) and `vmbr0.4` (192.168.4.8/24). All recon kept
full verification by connecting to 192.168.2.8 while verifying the
certificate as `pv1-1.home.lab`:

```bash
curl -s --resolve pv1-1.home.lab:8006:192.168.2.8 --cacert ~/.config/proxmox-ca.pem \
  -H @<(printf 'Authorization: PVEAPIToken=%s\n' "$(tr -d '[:space:]' < ~/.config/proxmox-token)") \
  https://pv1-1.home.lab:8006/api2/json/version
# {"data":{"release":"9.2","version":"9.2.20","repoid":"49318c671b82f31e"}}
```

Every capture went through a scratch helper wrapping exactly that call. It
passes the token through a file descriptor so it never reaches argv or
output, and writes `{"status","reason","body"}` to
`internal/provider/proxmox/testdata/<name>.json`, refusing to save a body
that contains `PVEAPIToken`.

A real NoCloud seed was used instead of the plan's 1 KiB placeholder, so the
agent could be installed and F21 captured from a live guest. No ISO tool is
installed locally, so a throwaway module in the scratchpad reused the
repository's `github.com/kdomanski/iso9660 v0.4.0` from the module cache and
copied kvm2's `guestAgentScript` verbatim:

```bash
GOFLAGS=-mod=mod GOPROXY=off GOSUMDB=off go run . proxy-reconaaaaaaaaaaa proxy-reconaaaaaaaaaaa-seed.iso
# 45056 bytes, ISO 9660 'cidata'
```

Write experiments ran against VMID 199999 (`proxy-reconaaaaaaaaaaa`), a
clone of template 9001. Sequence: validation error; clone; resources, config
write and start while the clone lock was held; status and agent while
running bare; stop; config read; clone onto the taken VMID; tag tests;
upload twice plus a bad name; attach `ide2`, set cores and memory; three
resizes; start and agent poll; suspend and resume; a clone onto a storage
without image support; stop; destroy with `purge=1&destroy-unreferenced-disks=1`;
destroy again; ISO delete.

### Facts F1 to F23 (cited as Evidence by the spec)

- **F1.** Errors carry the text twice: in the HTTP reason phrase and as the
  body's `message`, which ends in `\n`. A 400 adds `errors`, a map from
  parameter to message. A 401 has the reason phrase only and an empty body.
  The client should use the trimmed body `message` and fall back to the
  reason phrase. Headers are ordinary CRLF.
- **F2.** Missing VM, identical for `GET config`, `GET status/current` and
  `DELETE`: `500 Configuration file 'nodes/pv1-1/qemu-server/999999.conf' does not exist`.
- **F3.** A start issued while the VM is clone-locked is **accepted** (200
  with a UPID). The task log reads `trying to acquire lock... OK` and the VM
  starts once the clone finishes. Setting a lock through the API was not
  tried.
- **F4.** Clone onto a taken VMID, synchronous:
  `500 unable to create VM 199999: config file already exists`.
- **F5.** Agent enabled but not running: `500 QEMU guest agent is not running`.
  Agent not enabled in the config: `500 No QEMU guest agent configured`.
  Stopped VM with `agent: 1`: `500 VM 199999 is not running`.
- **F6.** Not produced. `local-lvm` is thin and overprovisions, so
  `quota: absent`.
- **F7.** Config write while clone-locked: `500 VM is locked (clone)`.
- **F8.** `/cluster/resources?type=vm` entries carry `vmid, name, node,
  status, type, template, tags, maxcpu, maxmem, maxdisk, uptime` and more;
  `lock` appears only when set. Status values seen: `running`, `stopped`,
  and `unknown` during a clone. **Mid-clone the entry has no `name`**: only
  `type, vmid, id, node, lock: clone, status: unknown`.
- **F9.** Hyphenated tags are **accepted** and stored as given. Uppercase is
  folded (`Proxy_Operator` → `proxy_operator`), the separator is `;`, and
  characters such as a space or `!` give
  `400 … tags: invalid format - invalid characters in tag`. The datacenter
  has no `tag-style` or `user-tag-access` set.
- **F10.** `description` on clone is accepted and survives verbatim,
  newlines included. The 22-character `proxy-<base32>` name is accepted.
- **F11.** The clone's `meta` is copied from the template
  (`creation-qemu=11.0.3,ctime=1790145676`, identical to 9001), so it says
  nothing about when the clone was made. `written-at-create` is required.
- **F12.** Upload: multipart with field `content=iso` and the file part named
  `filename`, carrying the target file name. Returns a UPID for an
  `imgcopy` task. A name without `.iso` gives
  `400 … filename: wrong file extension`. **Re-uploading an existing name
  silently overwrites it** (the task log shows a plain `cp`).
  `Datastore.AllocateTemplate` suffices.
- **F13.** Deleting a volume needs `Datastore.Allocate` on
  `/storage/<id>`; without it the call gives
  `403 Permission check failed (/storage/local, Datastore.Allocate)`. With it
  (on `proxy-seeds`), the delete **runs as an `imgdel` task** and returns a
  UPID. Deleting a volume that no longer exists is also accepted and its
  task still ends `OK`; the log reads `disk image '…' does not exist`, then
  `Removed volume`. Seed cleanup is idempotent with no message matching.
- **F14.** Stopped: `status: stopped, qmpstatus: stopped`. Paused:
  `status: running, qmpstatus: paused` in `status/current`, while
  `/cluster/resources` shows plain `running`.
- **F15.** A clone onto a storage without image support is rejected up
  front (`400 … storage 'local' does not support vm images`) and leaves no
  VM. A clone failing mid-task was not observed.
- **F16.** Task status fields: `status` (`running`, `stopped`), `exitstatus`
  (`OK` or the error text), `type, id, node, user, tokenid, upid, starttime,
  pid, pstart`. A full clone of the 3 GiB template onto `local-lvm` finished
  within about two seconds.
- **F17.** The planned role, granted on `/`, covered clone, config, resize,
  start, stop, suspend, agent read, upload and destroy. **Missing:
  `Datastore.Allocate` for deleting the seed ISO.** `VM.Config.Cloudinit` is
  not needed for the `ide2` write. `SDN.Use` was held but never exercised,
  because no call rewrote `net0`.
- **F18.** HTTP/1.1 only: `curl --http2` negotiates `HTTP/1.1`.
- **F19.** Template 9001 (`debian-13-template`): `scsi0` on `local-lvm`,
  `boot: order=scsi0`, `ide2` free, `agent: 1`, OVMF, q35, `net0` untagged on
  `vmbr0`. **Attaching `ide2` makes Proxmox append it to the boot order**
  (`order=scsi0;ide2`). Template 9000 (Ubuntu) is unsuitable: it has a
  Proxmox cloud-init drive on `ide2` and VLAN tag 4 on `net0`.
- **F20.** Resize **runs as a task** (returns a UPID). The same size is an
  `OK` no-op; shrinking fails inside the task with `exitstatus`
  `shrinking disks is not supported`.
- **F21.** Agent response: `result[]` of `{name, ip-addresses[{ip-address,
  ip-address-type}]}`. The live guest reported `lo` and `ens18` with
  `192.168.0.150`, 50 seconds after start, via DHCP on untagged `vmbr0`.
  This proves the NoCloud-seed-on-`ide2` route end to end on this host.
- **F22.** A wrong secret and an unknown token id both give
  `401 Authentication failed!`; no credentials gives `401 No ticket`.
- **F23.** Uploads are staged in `/var/tmp/pveupload-*` and then copied to
  `/var/lib/vz/template/iso/`, so the tmpfs `/tmp` is not involved.

### Corrections to the plan that Phase 1 must fold into the spec

1. **Check `lock` before the name.** A mid-clone VM has no name in
   `/cluster/resources` (F8), so name-first logic turns a clone in progress
   into NotFound. Get, Delete and Create's lookup must treat a locked or
   nameless entry at our VMID as in-progress (Transient or wait), never
   absent.
2. **Inspect a taken VMID before probing forward.** After a crash mid-clone,
   the retry's name lookup misses the nameless VM, the clone onto the
   deterministic VMID fails with F4, and forward probing would clone a
   duplicate. On F4, read that VMID: `lock: clone` means wait for it, and our
   UID in the description means adopt it. Probe forward only for a VM that
   is demonstrably someone else's.
3. **A start queues behind a lock** instead of failing (F3).
4. **Resize is asynchronous** (F20); wait on its task.
5. **Drop delete-before-upload** for the seed: upload overwrites (F12).
   **Seed deletion is a task** (F13): wait on it. A repeat delete succeeds,
   so Delete needs no absent-volume matcher.
6. **Get reads `status/current`**, not only `/cluster/resources`, to tell
   paused from running (F14).
7. **Tag rationale**: hyphens are accepted on 9.2.20 (F9). `proxy_operator`
   stays, since it is valid everywhere, but the claim that hyphens are
   rejected is removed.
8. **The boot-disk logic must skip `ide2`**, which Proxmox adds to the boot
   order (F19). The plan's rule already excludes it; add a test with
   `order=scsi0;ide2`.
9. **Three agent messages, not one** (F5): `QEMU guest agent is not running`
   (booting, or the install failed), `No QEMU guest agent configured` (our
   `agent=1` was removed), and `VM <vmid> is not running` (stopped between
   the status read and the agent call). The spec decides each class; none
   may become NotFound.
10. **Internet on the proxy network is a hard prerequisite**, twice over: the
    vendor-data agent install needs a package mirror, and the proxy itself
    needs egress. The guide says so, and recommends baking
    `qemu-guest-agent` into the template with `installGuestAgent: false` to
    drop the mirror dependency from IP discovery.

### Decisions made with the user in Phase 0

- **Seeds go to a dedicated directory storage**, `proxy-seeds`, with ISO
  content only. The token gets `Datastore.Allocate` there and nowhere else,
  so it can delete seeds but never backups or other ISOs on `local`. The
  sample config's `isoStorage` becomes `proxy-seeds`. The guide must say that
  an ACL on a more specific path replaces the inherited one for that user,
  so the role granted on `/storage/proxy-seeds` must repeat
  `Datastore.AllocateTemplate`, `AllocateSpace` and `Audit` alongside
  `Allocate`.
- **Proxies land on a VLAN** set in config. `ProxmoxConfig` gains an
  optional `vlanTag` (1 to 4094), applied when Create rewrites `net0`
  together with the optional bridge override, keeping the MAC. Unset leaves
  `net0` as cloned. This is the first call that exercises `SDN.Use` (F17).
- **The API URL is `https://192.168.0.8:8006`**, the address in the node
  certificate, so `caFile` verifies with no provider change. No
  `tlsServerName` option. The operator's cluster must be able to reach
  192.168.0.x.

### Seed storage and VLAN 4

The user created `proxy-seeds` (`dir`, ISO content only), granted the token
`Datastore.Allocate`, `AllocateSpace`, `AllocateTemplate` and `Audit` on
`/storage/proxy-seeds` through a role `ProxyOperatorSeeds`, and freed the
leftover recon ISO on `local`. An upload to `proxy-seeds` and two deletes of
a throwaway ISO there produced `content_delete_response.json`,
`content_delete_again_response.json` and `task_status_volume_missing.json`.

The user gave the VLAN as "vlan 4, bridge: vmbr0.4@vmbr0". That is the
host's own VLAN interface name as `ip link` prints it; a VM NIC cannot
attach to it because it is not a bridge. On the VLAN-aware `vmbr0` the
Proxmox form is `bridge=vmbr0,tag=4`, which is what template 9000 uses, so
the provider's `vlanTag: 4` appends `tag=4` to the cloned `net0`.

A second clone of 9001 at VMID 199999 got its seed from `proxy-seeds` and
`net0` rewritten from `virtio=BC:24:11:EE:48:83,bridge=vmbr0` to
`virtio=BC:24:11:EE:48:83,bridge=vmbr0,tag=4`. **The write succeeded, so the
token's `SDN.Use` on `/` covers a VLAN tag on `vmbr0`**
(`set_config_vlan_response.json`).

**The first VLAN 4 attempt never produced an agent** in seven minutes of
polling (every 13 seconds, 33 polls). The only signal was the agent call, so
the cause could not be seen from the API. A debug seed added a `recon` user
(user-data only, with a new instance id so cloud-init re-ran), and the user
logged in on the console: DHCP worked, but VLAN 4 had no route to the
internet because of a routing misconfiguration on the Proxmox host, so the
vendor-data `apt-get install qemu-guest-agent` could not run. The user fixed
the routing.

**Rerun from scratch after the fix**, same sequence the provider will use
(destroy and seed delete first, then a fresh clone and seed, `instance-id`
equal to the name):

| Step | Elapsed |
|---|---|
| clone task done | 4 s |
| seed upload task done | 4 s |
| config (`ide2`, `net0` with `tag=4`, tag) | 4 s |
| start task done | 5 s |
| agent reports `192.168.4.147` | 47 s after start |

The guest was reachable from the workstation (ping, TCP 22), as the contract
requires of a proxy address. Stopping it gave the F5 stopped-VM fixture;
destroy and seed delete then left the host with only its original VMs 100,
9000 and 9001, no ISOs on `local` or `proxy-seeds`, and no `local-lvm`
volumes of 199999. The capture is `agent_network_get_interfaces_vlan.json`.

### Worth noting

- The token file is mode 664; 600 would be tighter.
- Fixtures: 48 files, all valid JSON, none containing `PVEAPIToken`.
  Several names differ from the plan's list because what they captured
  differed from what was expected: `start_while_clone_locked_response.json`
  (the start succeeded), `task_status_stopped_error.json` (from the shrink,
  not a start), `upload_overwrite_response.json` (success, not an error),
  `resize_same_size_response.json` and `resize_shrink_response.json`
  (UPIDs), `err_forbidden_volume_delete.json` (the 403 from F13).
  `err_no_space.json` was not produced (F6).
- The host is back to its original state; see the table above.
- The scratch seed builder (`seedgen` in the session scratchpad) took the
  user-data file and instance id as arguments by the end, for the debug
  login. It is not part of the repository; Phase 3's `nocloud` package
  replaces it.
- Time to IP was about 50 s on both networks, with the agent installed at
  first boot. Budgets in the plan (running wait 60 s for the VM state, the
  agent discovered by `Get` polling) are consistent with this.