egress-proxies-operator/docs/plans-executions/2026-09-18-1502-proxmox-provider.md

# Execution: proxmox, a Proxmox VE provider built from the provider contract

Pairs with [docs/plans/2026-09-18-1502-proxmox-provider.md](../plans/2026-09-18-1502-proxmox-provider.md).

## Status

- [x] Phase 0 — Plan file, branch, host recon, fixtures
- [x] Phase 1 — The spec (`docs/proxmox-provider-spec.md`)
- [ ] Phase 2 — Config block, registration, package skeleton
- [ ] Phase 3 — Extract `nocloud` and `keyedlock` from kvm2
- [ ] Phase 4 — Seam, HTTP client, decorator, fake
- [ ] Phase 5 — Get, ListByTag, task helpers
- [ ] Phase 6 — Create
- [ ] Phase 7 — Delete, Checker, CRD comment
- [ ] Phase 8 — Contract and observability sweep, integration test, testing.md
- [ ] Phase 9 — Deployment and documentation
- [ ] Phase 10 — Review, real-host validation, changelog, MR

## Phase 0 — Plan file, branch, host recon, fixtures

The plan was committed alone on `main` on 2026-09-18 (`708989c`, then
`2167819` retargeting it at PVE 9.2). `origin/main` had not moved in the
meantime, so the branch was cut straight from local `main`:

```bash
git fetch -q origin && git rev-list --left-right --count origin/main...main   # 0 2
git switch -c feat/proxmox-provider
```

The host is **pv1-1**, a single node (no cluster peers), `pve-manager/9.2.20`,
kernel `7.0.14-17-pve`, QEMU 11.0. The user prepared the token
(`~/.config/proxmox-token`, privsep off) and the CA (`~/.config/proxmox-ca.pem`)
before the session.

**TLS: 192.168.2.8 is not in the node certificate.** The first probe failed
with curl exit 60. The CA is right (`openssl s_client -CAfile` verifies the
chain), but the leaf's SANs are `127.0.0.1, ::1, localhost, 192.168.0.8,
pv1-1, pv1-1.home.lab`, and neither name resolves on the workstation.
192.168.0.8 and 192.168.2.8 present the same certificate fingerprint: the
node sits on `vmbr0` (192.168.0.8/24, untagged) with VLAN interfaces
`vmbr0.2` (192.168.2.8/24) and `vmbr0.4` (192.168.4.8/24). All recon kept
full verification by connecting to 192.168.2.8 while verifying the
certificate as `pv1-1.home.lab`:

```bash
curl -s --resolve pv1-1.home.lab:8006:192.168.2.8 --cacert ~/.config/proxmox-ca.pem \
  -H @<(printf 'Authorization: PVEAPIToken=%s\n' "$(tr -d '[:space:]' < ~/.config/proxmox-token)") \
  https://pv1-1.home.lab:8006/api2/json/version
# {"data":{"release":"9.2","version":"9.2.20","repoid":"49318c671b82f31e"}}
```

Every capture went through a scratch helper wrapping exactly that call. It
passes the token through a file descriptor so it never reaches argv or
output, and writes `{"status","reason","body"}` to
`internal/provider/proxmox/testdata/<name>.json`, refusing to save a body
that contains `PVEAPIToken`.

A real NoCloud seed was used instead of the plan's 1 KiB placeholder, so the
agent could be installed and F21 captured from a live guest. No ISO tool is
installed locally, so a throwaway module in the scratchpad reused the
repository's `github.com/kdomanski/iso9660 v0.4.0` from the module cache and
copied kvm2's `guestAgentScript` verbatim:

```bash
GOFLAGS=-mod=mod GOPROXY=off GOSUMDB=off go run . proxy-reconaaaaaaaaaaa proxy-reconaaaaaaaaaaa-seed.iso
# 45056 bytes, ISO 9660 'cidata'
```

Write experiments ran against VMID 199999 (`proxy-reconaaaaaaaaaaa`), a
clone of template 9001. Sequence: validation error; clone; resources, config
write and start while the clone lock was held; status and agent while
running bare; stop; config read; clone onto the taken VMID; tag tests;
upload twice plus a bad name; attach `ide2`, set cores and memory; three
resizes; start and agent poll; suspend and resume; a clone onto a storage
without image support; stop; destroy with `purge=1&destroy-unreferenced-disks=1`;
destroy again; ISO delete.

### Facts F1 to F23 (cited as Evidence by the spec)

- **F1.** Errors carry the text twice: in the HTTP reason phrase and as the
  body's `message`, which ends in `\n`. A 400 adds `errors`, a map from
  parameter to message. A 401 has the reason phrase only and an empty body.
  The client should use the trimmed body `message` and fall back to the
  reason phrase. Headers are ordinary CRLF.
- **F2.** Missing VM, identical for `GET config`, `GET status/current` and
  `DELETE`: `500 Configuration file 'nodes/pv1-1/qemu-server/999999.conf' does not exist`.
- **F3.** A start issued while the VM is clone-locked is **accepted** (200
  with a UPID). The task log reads `trying to acquire lock... OK` and the VM
  starts once the clone finishes. Setting a lock through the API was not
  tried.
- **F4.** Clone onto a taken VMID, synchronous:
  `500 unable to create VM 199999: config file already exists`.
- **F5.** Agent enabled but not running: `500 QEMU guest agent is not running`.
  Agent not enabled in the config: `500 No QEMU guest agent configured`.
  Stopped VM with `agent: 1`: `500 VM 199999 is not running`.
- **F6.** Not produced. `local-lvm` is thin and overprovisions, so
  `quota: absent`.
- **F7.** Config write while clone-locked: `500 VM is locked (clone)`.
- **F8.** `/cluster/resources?type=vm` entries carry `vmid, name, node,
  status, type, template, tags, maxcpu, maxmem, maxdisk, uptime` and more;
  `lock` appears only when set. Status values seen: `running`, `stopped`,
  and `unknown` during a clone. **Mid-clone the entry has no `name`**: only
  `type, vmid, id, node, lock: clone, status: unknown`.
- **F9.** Hyphenated tags are **accepted** and stored as given. Uppercase is
  folded (`Proxy_Operator` → `proxy_operator`), the separator is `;`, and
  characters such as a space or `!` give
  `400 … tags: invalid format - invalid characters in tag`. The datacenter
  has no `tag-style` or `user-tag-access` set.
- **F10.** `description` on clone is accepted and survives verbatim,
  newlines included. The 22-character `proxy-<base32>` name is accepted.
- **F11.** The clone's `meta` is copied from the template
  (`creation-qemu=11.0.3,ctime=1790145676`, identical to 9001), so it says
  nothing about when the clone was made. `written-at-create` is required.
- **F12.** Upload: multipart with field `content=iso` and the file part named
  `filename`, carrying the target file name. Returns a UPID for an
  `imgcopy` task. A name without `.iso` gives
  `400 … filename: wrong file extension`. **Re-uploading an existing name
  silently overwrites it** (the task log shows a plain `cp`).
  `Datastore.AllocateTemplate` suffices.
- **F13.** Deleting a volume needs `Datastore.Allocate` on
  `/storage/<id>`; without it the call gives
  `403 Permission check failed (/storage/local, Datastore.Allocate)`. With it
  (on `proxy-seeds`), the delete **runs as an `imgdel` task** and returns a
  UPID. Deleting a volume that no longer exists is also accepted and its
  task still ends `OK`; the log reads `disk image '…' does not exist`, then
  `Removed volume`. Seed cleanup is idempotent with no message matching.
- **F14.** Stopped: `status: stopped, qmpstatus: stopped`. Paused:
  `status: running, qmpstatus: paused` in `status/current`, while
  `/cluster/resources` shows plain `running`.
- **F15.** A clone onto a storage without image support is rejected up
  front (`400 … storage 'local' does not support vm images`) and leaves no
  VM. A clone failing mid-task was not observed.
- **F16.** Task status fields: `status` (`running`, `stopped`), `exitstatus`
  (`OK` or the error text), `type, id, node, user, tokenid, upid, starttime,
  pid, pstart`. A full clone of the 3 GiB template onto `local-lvm` finished
  within about two seconds.
- **F17.** The planned role, granted on `/`, covered clone, config, resize,
  start, stop, suspend, agent read, upload and destroy. **Missing:
  `Datastore.Allocate` for deleting the seed ISO.** `VM.Config.Cloudinit` is
  not needed for the `ide2` write. `SDN.Use` was held but never exercised,
  because no call rewrote `net0`.
- **F18.** HTTP/1.1 only: `curl --http2` negotiates `HTTP/1.1`.
- **F19.** Template 9001 (`debian-13-template`): `scsi0` on `local-lvm`,
  `boot: order=scsi0`, `ide2` free, `agent: 1`, OVMF, q35, `net0` untagged on
  `vmbr0`. **Attaching `ide2` makes Proxmox append it to the boot order**
  (`order=scsi0;ide2`). Template 9000 (Ubuntu) is unsuitable: it has a
  Proxmox cloud-init drive on `ide2` and VLAN tag 4 on `net0`.
- **F20.** Resize **runs as a task** (returns a UPID). The same size is an
  `OK` no-op; shrinking fails inside the task with `exitstatus`
  `shrinking disks is not supported`.
- **F21.** Agent response: `result[]` of `{name, ip-addresses[{ip-address,
  ip-address-type}]}`. The live guest reported `lo` and `ens18` with
  `192.168.0.150`, 50 seconds after start, via DHCP on untagged `vmbr0`.
  This proves the NoCloud-seed-on-`ide2` route end to end on this host.
- **F22.** A wrong secret and an unknown token id both give
  `401 Authentication failed!`; no credentials gives `401 No ticket`.
- **F23.** Uploads are staged in `/var/tmp/pveupload-*` and then copied to
  `/var/lib/vz/template/iso/`, so the tmpfs `/tmp` is not involved.

### Corrections to the plan that Phase 1 must fold into the spec

1. **Check `lock` before the name.** A mid-clone VM has no name in
   `/cluster/resources` (F8), so name-first logic turns a clone in progress
   into NotFound. Get, Delete and Create's lookup must treat a locked or
   nameless entry at our VMID as in-progress (Transient or wait), never
   absent.
2. **Inspect a taken VMID before probing forward.** After a crash mid-clone,
   the retry's name lookup misses the nameless VM, the clone onto the
   deterministic VMID fails with F4, and forward probing would clone a
   duplicate. On F4, read that VMID: `lock: clone` means wait for it, and our
   UID in the description means adopt it. Probe forward only for a VM that
   is demonstrably someone else's.
3. **A start queues behind a lock** instead of failing (F3).
4. **Resize is asynchronous** (F20); wait on its task.
5. **Drop delete-before-upload** for the seed: upload overwrites (F12).
   **Seed deletion is a task** (F13): wait on it. A repeat delete succeeds,
   so Delete needs no absent-volume matcher.
6. **Get reads `status/current`**, not only `/cluster/resources`, to tell
   paused from running (F14).
7. **Tag rationale**: hyphens are accepted on 9.2.20 (F9). `proxy_operator`
   stays, since it is valid everywhere, but the claim that hyphens are
   rejected is removed.
8. **The boot-disk logic must skip `ide2`**, which Proxmox adds to the boot
   order (F19). The plan's rule already excludes it; add a test with
   `order=scsi0;ide2`.
9. **Three agent messages, not one** (F5): `QEMU guest agent is not running`
   (booting, or the install failed), `No QEMU guest agent configured` (our
   `agent=1` was removed), and `VM <vmid> is not running` (stopped between
   the status read and the agent call). The spec decides each class; none
   may become NotFound.
10. **Internet on the proxy network is a hard prerequisite**, twice over: the
    vendor-data agent install needs a package mirror, and the proxy itself
    needs egress. The guide says so, and recommends baking
    `qemu-guest-agent` into the template with `installGuestAgent: false` to
    drop the mirror dependency from IP discovery.

### Decisions made with the user in Phase 0

- **Seeds go to a dedicated directory storage**, `proxy-seeds`, with ISO
  content only. The token gets `Datastore.Allocate` there and nowhere else,
  so it can delete seeds but never backups or other ISOs on `local`. The
  sample config's `isoStorage` becomes `proxy-seeds`. The guide must say that
  an ACL on a more specific path replaces the inherited one for that user,
  so the role granted on `/storage/proxy-seeds` must repeat
  `Datastore.AllocateTemplate`, `AllocateSpace` and `Audit` alongside
  `Allocate`.
- **Proxies land on a VLAN** set in config. `ProxmoxConfig` gains an
  optional `vlanTag` (1 to 4094), applied when Create rewrites `net0`
  together with the optional bridge override, keeping the MAC. Unset leaves
  `net0` as cloned. This is the first call that exercises `SDN.Use` (F17).
- **The API URL is `https://192.168.0.8:8006`**, the address in the node
  certificate, so `caFile` verifies with no provider change. No
  `tlsServerName` option. The operator's cluster must be able to reach
  192.168.0.x.

### Seed storage and VLAN 4

The user created `proxy-seeds` (`dir`, ISO content only), granted the token
`Datastore.Allocate`, `AllocateSpace`, `AllocateTemplate` and `Audit` on
`/storage/proxy-seeds` through a role `ProxyOperatorSeeds`, and freed the
leftover recon ISO on `local`. An upload to `proxy-seeds` and two deletes of
a throwaway ISO there produced `content_delete_response.json`,
`content_delete_again_response.json` and `task_status_volume_missing.json`.

The user gave the VLAN as "vlan 4, bridge: vmbr0.4@vmbr0". That is the
host's own VLAN interface name as `ip link` prints it; a VM NIC cannot
attach to it because it is not a bridge. On the VLAN-aware `vmbr0` the
Proxmox form is `bridge=vmbr0,tag=4`, which is what template 9000 uses, so
the provider's `vlanTag: 4` appends `tag=4` to the cloned `net0`.

A second clone of 9001 at VMID 199999 got its seed from `proxy-seeds` and
`net0` rewritten from `virtio=BC:24:11:EE:48:83,bridge=vmbr0` to
`virtio=BC:24:11:EE:48:83,bridge=vmbr0,tag=4`. **The write succeeded, so the
token's `SDN.Use` on `/` covers a VLAN tag on `vmbr0`**
(`set_config_vlan_response.json`).

**The first VLAN 4 attempt never produced an agent** in seven minutes of
polling (every 13 seconds, 33 polls). The only signal was the agent call, so
the cause could not be seen from the API. A debug seed added a `recon` user
(user-data only, with a new instance id so cloud-init re-ran), and the user
logged in on the console: DHCP worked, but VLAN 4 had no route to the
internet because of a routing misconfiguration on the Proxmox host, so the
vendor-data `apt-get install qemu-guest-agent` could not run. The user fixed
the routing.

**Rerun from scratch after the fix**, same sequence the provider will use
(destroy and seed delete first, then a fresh clone and seed, `instance-id`
equal to the name):

| Step | Elapsed |
|---|---|
| clone task done | 4 s |
| seed upload task done | 4 s |
| config (`ide2`, `net0` with `tag=4`, tag) | 4 s |
| start task done | 5 s |
| agent reports `192.168.4.147` | 47 s after start |

The guest was reachable from the workstation (ping, TCP 22), as the contract
requires of a proxy address. Stopping it gave the F5 stopped-VM fixture;
destroy and seed delete then left the host with only its original VMs 100,
9000 and 9001, no ISOs on `local` or `proxy-seeds`, and no `local-lvm`
volumes of 199999. The capture is `agent_network_get_interfaces_vlan.json`.

### Worth noting

- The token file is mode 664; 600 would be tighter.
- Fixtures: 48 files, all valid JSON, none containing `PVEAPIToken`.
  Several names differ from the plan's list because what they captured
  differed from what was expected: `start_while_clone_locked_response.json`
  (the start succeeded), `task_status_stopped_error.json` (from the shrink,
  not a start), `upload_overwrite_response.json` (success, not an error),
  `resize_same_size_response.json` and `resize_shrink_response.json`
  (UPIDs), `err_forbidden_volume_delete.json` (the 403 from F13).
  `err_no_space.json` was not produced (F6).
- The host is back to its original state; see the table above.
- The scratch seed builder (`seedgen` in the session scratchpad) took the
  user-data file and instance id as arguments by the end, for the debug
  login. It is not part of the repository; Phase 3's `nocloud` package
  replaces it.
- Time to IP was about 50 s on both networks, with the agent installed at
  first boot. Budgets in the plan (running wait 60 s for the VM state, the
  agent discovered by `Get` polling) are consistent with this.

## Phase 1 — The spec

Wrote [docs/proxmox-provider-spec.md](../proxmox-provider-spec.md), all 24
entries of the template, from the plan, the contract and the Phase 0 facts.
The ten Phase 0 corrections are folded in, and the plan now opens with a
note that the spec wins where the two disagree.

Judgment calls made here, each recorded in its entry:

- **Create's allocation loop** (entry 2) runs an ownership check on any
  taken VMID: wait out a lock or a nameless entry, then adopt when the name
  and UID are ours, skip when the name differs, refuse as Permanent when the
  name is ours but the mark is not. This is the fix for the duplicate-VM bug.
- **Get's check order** (entry 9): lock, then name and mark from the config,
  then `status/current`, then the agent. `lock: suspended` alone maps to
  Stopped, since a hibernated VM never resumes by itself.
- **Agent messages** (entry 7): `QEMU guest agent is not running` is
  Provisioning, `VM <vmid> is not running` is Transient, and
  `No QEMU guest agent configured` is **Stopped**, so the reconciler replaces
  a VM that can never report an address instead of latching the Proxy.
- **Templates are checked before cloning** (entry 4): not a template, or a
  Proxmox cloud-init drive in any slot, is Permanent. This makes the "no
  cloud-init drive" prerequisite enforceable instead of a silent failure.
- **`isoStorage` is required**, with no default (entry 15), because the token
  must hold `Datastore.Allocate` on it.
- **TLS verification failures are Permanent**, with a Checker reason
  `CertificateInvalid` (entries 13, 16), since the fix is the URL or CA file.
- **The CA is loaded lazily and cached**; the token is read per request
  (entry 14).

Checks, as the plan's Verify clause asks:

```bash
grep -c '^## [0-9]*\. ' docs/proxmox-provider-spec.md     # 24
grep -n '^\*\*Answer:\*\*$' docs/proxmox-provider-spec.md # nothing
grep -c '—' docs/proxmox-provider-spec.md                 # 0
```

plus a script comparing every `` `<axis>`: `<profile>` `` pair against the
contract's §4.20 table (no mismatches, every axis answered) and a list of
the F numbers cited (all 23).

Worth noting: the first pass stated two things as fact that recon could not
show and was corrected before commit. That an unreachable node's guests stay
listed as `unknown` is Proxmox's documented design, unverifiable on a single
node; entry 13 now says so. And the matcher for `VM <vmid> is not running`
is a whole-message regexp, because ` is not running` alone also matches
`QEMU guest agent is not running`.