egress-proxies-operator/docs/proxmox-provider.md

# The proxmox provider

Runs Managed proxies as VMs on a Proxmox VE cluster. It is the third
VM-style provider and the second built from
[provider-contract.md](provider-contract.md), from the design in
[proxmox-provider-spec.md](proxmox-provider-spec.md). The operator talks to
the cluster's REST API only (`https://<host>:8006/api2/json`) with an API
token. It needs no SSH, no shell on the nodes, and no library beyond Go's
standard `net/http`.

Each VM is a **full clone of a template** you prepare once. The operator
builds the Proxy's cloud-init into a NoCloud seed ISO in-process, uploads
it to a storage on the VM's node, attaches it as a CD-ROM, sets the size,
starts the VM and learns its address from the QEMU guest agent. Everything
here was proven against **Proxmox VE 9.2.20** (QEMU 11.0) on the single
node `pv1-1`; every error text the provider matches was captured from that
host.

## Proxmox prerequisites

Once per cluster, as root on a node:

1. **Proxmox VE 9.x with the API reachable** from the operator's pods on
   port 8006. The operator's pods and the crawler clients must also route to
   the network the proxies land on (see step 5).
2. **A template without a Proxmox cloud-init drive.** The provider brings
   its own seed on `ide2`; a Proxmox cloud-init drive would compete with it
   and the Proxy's user-data would be ignored. Debian 13's generic cloud
   image boots UEFI (under BIOS it hangs in GRUB on a headless VM), so the
   template does too:

   ```sh
   wget https://cloud.debian.org/images/cloud/trixie/latest/debian-13-generic-amd64.qcow2
   qm create 9001 --name debian-13-template --memory 2048 --cores 2 --ostype l26 \
     --machine q35 --bios ovmf --efidisk0 local-lvm:1,efitype=4m,pre-enrolled-keys=0 \
     --scsihw virtio-scsi-pci --net0 virtio,bridge=vmbr0 \
     --serial0 socket --vga serial0 --agent 1
   qm set 9001 --scsi0 local-lvm:0,import-from=/root/debian-13-generic-amd64.qcow2
   qm set 9001 --boot order=scsi0
   qm template 9001
   ```

   The template must boot from its first disk, run cloud-init with the
   NoCloud datasource (the Debian and Ubuntu cloud images do), and either
   ship `qemu-guest-agent` or be able to install it at first boot. Before
   the first clone for a Proxy, `Create` reads the template's config and
   refuses it with a permanent error when:

   - it is not a template (`qm template` was not run);
   - any disk slot holds a Proxmox cloud-init drive (a value containing
     `cloudinit`), such as the one `qm set <vmid> --ide2 local-lvm:cloudinit`
     adds for Proxmox's own cloud-init support;
   - it has no `net0` while the config sets `bridge` or `vlanTag`;
   - the machine type sets `diskGiB` and the template has no boot disk, or
     one without a readable `size=`.

   These are checked on the template because the clone copies its config:
   a VM cloned from a bad template could not be fixed by fixing the
   template. The template is checked only when a VM is about to be cloned;
   a VM that already exists no longer depends on it. The health check sees
   only whether each template exists and is a template.
3. **A storage for the seed ISOs**, holding `iso` content. A dedicated
   directory storage is best, because the token needs `Datastore.Allocate`
   on it to delete seeds, and on a dedicated storage that privilege cannot
   touch backups or other ISOs:

   ```sh
   pvesm add dir proxy-seeds --path /var/lib/proxy-seeds --content iso
   ```

   A seed is tens of KiB. Proxmox stages uploads under `/var/tmp`, not the
   node's tmpfs `/tmp`. On a cluster, a storage that is not shared keeps
   each seed on the node the VM was created on (see *Known gaps*).
4. **A user, two roles and a token.** `VM.Monitor` no longer exists in PVE
   9 and `pveum` rejects it; the agent read needs `VM.GuestAgent.Audit`
   alone. `VM.Config.Cloudinit` is not needed for the seed on `ide2`.
   `SDN.Use` on `/` is what lets the config write put a VLAN tag on
   `vmbr0`:

   ```sh
   pveum user add proxy-operator@pve --comment "egress-proxies-operator"
   pveum role add ProxyOperator -privs "VM.Allocate VM.Clone VM.Audit VM.Config.CDROM VM.Config.CPU VM.Config.Memory VM.Config.Disk VM.Config.Network VM.Config.Options VM.PowerMgmt VM.GuestAgent.Audit Datastore.Audit Datastore.AllocateSpace Datastore.AllocateTemplate Sys.Audit SDN.Use"
   pveum aclmod / -user proxy-operator@pve -role ProxyOperator
   pveum role add ProxyOperatorSeeds -privs "Datastore.Allocate Datastore.AllocateTemplate Datastore.AllocateSpace Datastore.Audit"
   pveum aclmod /storage/proxy-seeds -user proxy-operator@pve -role ProxyOperatorSeeds
   pveum user token add proxy-operator@pve operator --privsep 0
   ```

   The second role repeats three privileges of the first on purpose: an
   ACL on a more specific path **replaces** what the user inherits from
   `/` for that path, so a role with `Datastore.Allocate` alone would take
   the upload's `Datastore.AllocateTemplate` away. The role on `/` covers
   clone, config, resize, start, stop, the agent read, the upload and the
   destroy; what it lacks is `Datastore.Allocate`, without which deleting
   a seed is a 403 (`Permission check failed (/storage/<id>,
   Datastore.Allocate)`). With `--privsep 0` the token carries the user's
   privileges.

   The token's secret is printed once. Save it on your workstation as one
   line, `proxy-operator@pve!operator=<secret>`, in
   `~/.config/proxmox-token` with mode 0600. Copy the node's CA,
   `/etc/pve/pve-root-ca.pem`, to `~/.config/proxmox-ca.pem`.
5. **A proxy network with DHCP and a route to the internet.** Each VM
   keeps the template's one NIC, optionally moved to another bridge or
   VLAN by the config, and takes its address from that network's DHCP. The
   network needs internet twice over: the vendor-data script installs
   `qemu-guest-agent` from the distro mirrors at first boot, and the agent
   is the only way the operator learns the VM's address; and an egress
   proxy needs egress anyway. Baking `qemu-guest-agent` into the template
   and setting `installGuestAgent: false` removes the mirror from IP
   discovery. The operator's pods and the clients must route to that
   network: the health engine probes `http://<ip>:<port>` through the
   proxy, and clients get the same address from the discovery API. The
   provider does not use `spec.port` and configures no Proxmox firewall.
6. **A reserved VMID range.** VMIDs are allocated from `vmidRange`
   (default 100000 to 199999). Keep hand-made VMs out of it: one there
   costs a probe, and one named like a proxy is refused (see *What lands
   on Proxmox*).

Check from the operator's point of view before wiring it in. This sends
the token through a file descriptor, so it never reaches the command line:

```sh
pve() {
  curl -s --cacert ~/.config/proxmox-ca.pem \
    -H @<(printf 'Authorization: PVEAPIToken=%s\n' "$(tr -d '[:space:]' < ~/.config/proxmox-token)") \
    "https://192.168.0.8:8006/api2/json/$1"; echo
}
pve version                                  # the token works, TLS verifies
pve nodes/pv1-1/storage/proxy-seeds/status   # active, content includes iso
pve 'cluster/resources?type=vm'              # the template is listed with "template":1
```

## Cluster-side setup

1. Create the Secret. It holds two keys, `token` (the one-line token file)
   and `ca.pem` (the node's CA):

   ```sh
   make proxmox-token-secret
   # or by hand:
   kubectl -n egress-proxies-operator-system create secret generic proxmox-token \
     --from-file=token=$HOME/.config/proxmox-token \
     --from-file=ca.pem=$HOME/.config/proxmox-ca.pem
   ```

   The make target runs that command as an apply, so it also updates an
   existing Secret. Override the paths with
   `PROXMOX_TOKEN_FILE=... PROXMOX_CA_FILE=...`. Before it applies, it
   checks that both files exist, that the token file is exactly one line
   of `<user>@<realm>!<tokenid>=<secret>` (without printing it), and that
   the CA file holds a PEM certificate. With `insecureSkipTLSVerify`
   instead of a CA, create the Secret by hand with the `token` key only.

   `config/manager/manager.yaml` mounts the Secret (optional, read-only)
   at `/etc/proxy-operator/proxmox`, so the files are
   `/etc/proxy-operator/proxmox/token` and
   `/etc/proxy-operator/proxmox/ca.pem`. The deployment starts fine
   without it, even with a proxmox provider configured: `New` makes no
   request, only checks that the token file exists, and logs a `WARNING:`
   line when it does not. The provider's proxies then fail transiently and
   recover once the Secret lands; the health check reports
   `CredentialsUnavailable` meanwhile.

   **Rotating the token needs no restart.** The token file is read on
   every request, so the next request after the kubelet refreshes the
   mount uses the new token. **Rotating the CA does**: it is loaded on
   first use and cached for the life of the process. A missing CA file is
   retried until it lands; one without a certificate is a permanent error.

2. Add a provider entry to the providers config
   (`config/manager/providers_config.yaml` in-cluster; the full sample,
   commented out, is in `config/samples/providers-config.yaml`):

   ```yaml
   - name: proxmox-home
     type: proxmox
     proxmox:
       url: https://192.168.0.8:8006
       tokenFile: /etc/proxy-operator/proxmox/token
       caFile: /etc/proxy-operator/proxmox/ca.pem
       nodes: [pv1-1]
       isoStorage: proxy-seeds
       vlanTag: 4
       templates:
         debian-13: {vmid: 9001}
       machineTypes:
         small: {cores: 1, memoryMiB: 1024}
   ```

   **The URL's host must be in the node certificate.** The node's
   self-signed certificate lists its own addresses and names, and TLS
   verification checks the URL's host against them. On `pv1-1` it lists
   `192.168.0.8`, `pv1-1` and `pv1-1.home.lab`, so
   `https://192.168.2.8:8006` reaches the same node and still fails
   verification (`CertificateInvalid` in the health check, a permanent
   error on every call). See what yours lists with:

   ```sh
   openssl s_client -connect 192.168.0.8:8006 </dev/null 2>/dev/null | openssl x509 -noout -ext subjectAltName
   ```

   If the address the operator must use is not there, either give the
   node a certificate that names it (a custom certificate in the node's
   Certificates panel) and put that certificate's CA in `caFile`, or set
   `insecureSkipTLSVerify: true`, which `New` logs as a `WARNING:` line.

   `nodes` holds node names as Proxmox knows them, not addresses: the one
   URL reaches the whole cluster. The providers config is read once per
   process, so adding a node, a template or a machine type needs a manager
   restart.

3. Apply a Proxy (`config/samples/proxy_proxmox.yaml`) and watch it:

   ```sh
   kubectl get px -w
   # NAME                   MODE      PROVIDER       PHASE          IP              HEALTHY
   # proxy-proxmox-sample   Managed   proxmox-home   Provisioning
   # proxy-proxmox-sample   Managed   proxmox-home   Running        192.168.4.147
   # proxy-proxmox-sample   Managed   proxmox-home   Ready          192.168.4.147   True
   ```

   (Illustrative.) `Create` returns once the VM runs, a few seconds after
   the clone; the address follows when the guest agent is up, about 50
   seconds after the start on `pv1-1` with the agent installed at first
   boot.

## Config reference

The block is `proxmox:` and its Go type is `ProxmoxConfig` in
`internal/provider/config_proxmox.go`, validated at load by
`validateProxmox`; unknown keys are rejected. Defaults are applied by
`withDefaults` in `internal/provider/proxmox/proxmox.go`.

| Field | Required | Default | Meaning |
|---|---|---|---|
| `url` | yes | | `https://<host>:8006`, with no path (the client appends `/api2/json`). The host must be in the node certificate |
| `tokenFile` | yes | | file holding one line, `<user>@<realm>!<tokenid>=<secret>`, read on every request and sent as `Authorization: PVEAPIToken=<line>` |
| `caFile` | | system roots | PEM bundle trusted for the API's certificate, typically the node's `/etc/pve/pve-root-ca.pem`. Loaded on first use and cached |
| `insecureSkipTLSVerify` | | `false` | skip certificate verification; `New` logs a `WARNING:` line when it is on |
| `nodes` | yes, at least one | | node names `placement.zone` selects from; unique, lowercase letters, digits and dashes. They end up in request paths and as the clone's `target` |
| `isoStorage` | yes | | storage id the seeds are uploaded to and deleted from. Must hold `iso` content; the token needs `Datastore.Allocate` on it. No default on purpose |
| `storage` | | the template's disk storage | storage id the full clone's disks go to |
| `bridge` | | as cloned | replaces `bridge=` in the cloned `net0` |
| `vlanTag` | | as cloned | 1 to 4094; replaces or appends `tag=` in the cloned `net0` |
| `vmidRange.min`, `vmidRange.max` | | `100000`, `199999` | where VMIDs are allocated; at least 100, at most 999999999, min below max |
| `templates.<key>.vmid` | at least one template | | the key names the template in `placement.image`; the VMID is at least 100 |
| `machineTypes.<key>` | at least one | | `{cores, memoryMiB, diskGiB}` size profile. `cores` and `memoryMiB` above zero. `diskGiB` is optional: when larger than the cloned boot disk the disk is grown to it; it is never shrunk |
| `requestTimeoutSeconds` | | `30` | bound on one API request, response included. The seed upload is one request, so a multi-megabyte `vendorData` needs more |
| `vendorData` | | | extra cloud-init vendor-data placed on every seed next to the Proxy's user-data; a MIME multipart part beside the agent script when `installGuestAgent` is on. Cloud-init drops any top-level key of a vendor-data cloud-config that the user-data also defines |
| `installGuestAgent` | | `true` | add the vendor-data script that installs and starts `qemu-guest-agent`. Turn it off for templates that ship the agent |

## Placement

All three catalogue fields are checked before any request. An unknown
value is `ErrPermanent`, which the reconciler latches as `Failed` with the
message below until the spec changes.

| Proxy field | Meaning for proxmox | Error when unknown |
|---|---|---|
| `placement.zone` | a node from `nodes`. May be omitted when exactly one node is configured; otherwise required | `placement.zone "x" is not in proxmox.nodes`; with several nodes and no zone, `placement.zone is required when proxmox.nodes lists more than one node` |
| `placement.machineType` | a key in `machineTypes` | `placement.machineType "x" is not in proxmox.machineTypes` |
| `placement.image` | a key in `templates` | `placement.image "x" is not in proxmox.templates` |
| `placement.region` | ignored. It is still part of the spec hash, so changing it replaces the VM and does nothing else | |

The messages come from `placement` in `internal/provider/proxmox/create.go`
and `node` in `proxmox.go`. A key that exists but names a VMID that is
missing or not a template fails later, before any clone:
`placement.image "x": template VM 9001 does not exist` or `... is not a
template`.

**Catalogue edits affect only new VMs.** The spec hash covers the
placement values, not what the config maps them to, so repointing
`templates.debian-13.vmid` at another template changes the VMs created
afterwards and leaves the running ones alone. Add a new key rather than
repointing one in use, so the Proxies that name a key keep a meaning you
can reason about. A template may be edited, replaced or deleted without
affecting existing VMs: they are full clones and share nothing with it.

## What lands on Proxmox

For a Proxy whose deterministic name is `proxy-abc…`:

| Object | Name | Lifetime |
|---|---|---|
| VM | name `proxy-abc…`, a VMID from `vmidRange` | with the Proxy |
| its disks | the template's disks, fully copied onto `storage` (or the template's storage) | destroyed with the VM (`purge=1&destroy-unreferenced-disks=1`) |
| seed ISO | `<isoStorage>:iso/proxy-abc…-seed.iso` on the VM's node | with the Proxy |

**The VMID** is derived from the Proxy's UID, so a retried Create lands on
the same number: an FNV-1a hash of the UID placed in `vmidRange`, moving
forward one at a time on collision, up to 32 probes (`vmidFor` in
`id.go`). Before allocating, `Create` looks the name up in the cluster's
resources. A VMID that is taken, or a VM that already has the name, goes
through an ownership check (`inspect` in `create.go`): a VM with a lock
(a clone still running, which also has no name yet) is waited on; then

- another name: someone else's VM, so the next VMID is probed;
- our name and our mark with our UID or an empty one: **adopted**, and
  Create carries on from where the previous attempt stopped;
- our name without the mark, or with another UID: **permanent error**
  naming the VMID and both UIDs. The operator never takes over a VM it did
  not create;
- our name, our mark, hibernated (`lock: suspended`): **permanent error**,
  `resume or destroy it in Proxmox` (see *Known gaps*).

Two VMs with the proxy's name are a permanent error naming every VMID;
remove all but one.

**The providerID** is `vms/<vmid>/<name>`. It names no node, because VMs
migrate: the node is looked up in `GET /cluster/resources` on every call.
`Get` and `Delete` check that the VM at that VMID still has that name and
the mark. Another name means ours is gone and the VMID was reused: `Get`
answers NotFound, `Delete` succeeds without touching it.

**The mark** is three lines in the VM's description (Notes in the UI),
passed in the clone call itself, so a VM is never unmarked:

```
proxy-operator-managed=true
proxy-operator-uid=0f3a…
proxy-operator-created-at=2026-09-23T17:42:11Z
```

That is how orphan GC recognises its own VMs and how a repeat `Create`
knows whether a VM of the same name belongs to this Proxy. Lines an admin
adds to the description are kept and ignored. The clone's own
`meta: ctime=` is useless for this: Proxmox copies it from the template.

**The config write.** After the clone and the seed upload, one
`PUT /nodes/{node}/qemu/{vmid}/config` sets:

| Key | Value |
|---|---|
| `cores`, `memory` | from the machine type |
| `agent` | `1` |
| `tags` | `proxy_operator` |
| `ide2` | `<isoStorage>:iso/<name>-seed.iso,media=cdrom` |
| `net0` | only when `bridge` or `vlanTag` is set: the cloned value with `bridge=` and `tag=` replaced or appended, the model, MAC and every other option kept |

The write **replaces the template's `agent` option and tags**: a
template's `agent: 1,fstrim_cloned_disks=1` becomes `agent: 1`, and its
tags give way to `proxy_operator`. The tag is for filtering in the
Proxmox UI and narrows GC's candidates; the mark decides. Proxmox appends
`ide2` to the boot order on its own (`order=scsi0;ide2`), which is
harmless because the boot disk stays first. When the machine type's
`diskGiB` is larger than the boot disk (the first disk in `boot: order=`
that is not a CD-ROM), a resize task grows it; cloud-init's growpart
expands the root filesystem at first boot.

**The seed** (`nocloud.BuildSeedISO`, shared with kvm2 in
`internal/provider/nocloud`) is an ISO 9660 image labelled `cidata`.
`user-data` is the Proxy's `cloudInit` **byte for byte**; the operator
never parses or re-renders it. `meta-data` holds `instance-id: <name>` and
`local-hostname: <name>`. `vendor-data` carries the guest-agent script
(a shell script, not a cloud-config, so a Proxy's own `packages:` cannot
shadow it), joined with `vendorData` as a MIME multipart when both are
set. It is staged under `/tmp` (the manager's emptyDir), built in memory
and uploaded with `POST /nodes/{node}/storage/{isoStorage}/upload`. An
upload overwrites a file of the same name, so it is rebuilt and
re-uploaded on every Create attempt with no delete first: a previous
attempt may have been cut off. Only the API is used; the storage upload
endpoint does not accept `snippets`, so Proxmox's own `cicustom` would
need shell access to the node.

**What a retried Create finds.** Every step is safe to repeat. After a
crash mid-clone, the retry lands on the same VMID, waits for the clone's
lock, and adopts the VM. A stopped VM of ours gets the seed, the config
and the resize again (the upload overwrites, the config is a replace, the
resize is skipped once large enough), then the start. A running one is
returned at once.

**A clone lock that never clears.** A node that crashes mid-clone can
leave the VM locked with `lock: clone` for good. Every Create for that
Proxy then waits out the 10 minute clone budget and returns a transient
error, and the reconciler keeps retrying rather than latching. Clear it
on the node:

```sh
qm unlock <vmid>      # then let the next Create adopt and finish it
qm destroy <vmid> --purge   # or remove it and let Create clone again
```

**Seeds without a VM.** A VM destroyed outside the operator leaves its
seed ISO, which GC cannot see (it lists VMs). Seeds are named
`proxy-*-seed.iso` on `isoStorage`; any without a matching VM may be
removed by hand:

```sh
pvesm list proxy-seeds --content iso
pvesm free proxy-seeds:iso/proxy-abc…-seed.iso
```

**Delete** runs under a per-name lock shared with Create: stop (a paused
VM too), destroy with `purge=1&destroy-unreferenced-disks=1`, then delete
the seed on the VM's node. Each is a task that is waited on. A VM that is
already gone, not ours, or unmarked is left alone, and the seed is then
looked for on every configured node, where a node that fails is logged
and skipped. On the VM's own node a failed seed delete fails the call:
**without `Datastore.Allocate` on `isoStorage` the 403 makes Delete
permanent**, and the Proxy's deletion stays stuck with that message until
the privilege is granted. When Delete returns nil, nothing of the Proxy
is left.

## Logging

The provider logs through controller-runtime's logger, named `proxmox`,
with `provider` (the configured name) on every line, and `vmid`, `name`
and `node` on every line about an instance. Every line inside a traced
operation also carries `traceID` and `spanID`.

- At startup (always on): `WARNING: token file is not readable; every
  call fails until it is` with `path` and `reason`, and `WARNING:
  insecureSkipTLSVerify is set`.
- `--zap-log-level=debug` (logr V(1)):
  - one line per operation outcome: `proxmox vm ensured running`,
    `proxmox vm adopted` with `status` and `qmpstatus`,
    `proxmox vm fetched` with `state`, `ip`, `uid`, `status` and
    `qmpstatus`, `proxmox vm not found` with `reason`,
    `proxmox vm deleted` with `destroyed` (false when no VM of ours was
    left to destroy), `proxmox vms listed` with `vms`, `candidates`,
    `managed` and `skipped` counts, and `proxmox node checked` with
    `ready`, `reason` and `latency`;
  - **one `proxmox api call` line per API request**, written by the
    decorator in `internal/provider/proxmox/observe.go`: `call` (the seam
    method, for example `Clone`), the identifiers it was given (`node`,
    `vmid`, `storage`), `upid` for a task, `newid` and `name` on a clone,
    `filename` on an upload, `volid` on a volume delete, `duration`, and
    on failure `httpStatus` and `error`. A running `Get` is four calls; a
    fresh Create is a dozen or more, depending on how many polls each
    task takes;
  - `proxmox task failed` with `upid` and `exitstatus`, and
    `proxmox task finished with warnings`;
  - the no-op lines: `proxmox vm already gone`,
    `proxmox vm at this vmid is not ours, left alone` with `vmName`,
    `proxmox vm carries no proxy-operator mark, left alone`,
    `proxmox seed delete failed on a node, skipped`,
    `skipping unreadable proxmox vm`, and the probe's
    `proxmox vmid taken by another vm, probing on`.
- `--zap-log-level=2`: `proxmox create resolved` with the resolved
  parameters (node, machine type, cores, memory, disk, template VMID,
  storages, bridge, VLAN tag) and `cloudInitBytes`; the `proxmox api
  call` lines also carry the config write's parameters, a summary of each
  config read, the clone's description, storage and target, the resize
  size, VM and task status, the agent's addresses and the upload's byte
  count; `proxmox guest agent answered`; `proxmox listed instance` per
  listed VM.
- Never logged at any level: the cloud-init content, the token, the CA
  content and the upload body. There is no raw wire tier.

## Tracing

With an OTLP exporter configured ([architecture.md
§10](architecture.md#10-tracing-internaltracing)), every API request is a
child span of the operation span the decorators open (`provider.create`,
`provider.get`, `provider.delete`, `provider.list`) or of the monitor's
`providerhealth.check`:

| Span | Around | Attributes |
|---|---|---|
| `HTTP GET`, `HTTP POST`, `HTTP PUT`, `HTTP DELETE` | one API request, from `otelhttp` on the client's transport | otelhttp's standard HTTP client attributes: method, URL, server address and port, response status |

The transport is `tracing.RestConfigWrapper()`, the same parent-gated
wrapper the kubernetes provider uses, so a request without a parent span
starts none. The logging decorator starts no span of its own, so nothing
is counted twice; its `proxmox api call` line carries the operation
span's `spanID`, not the HTTP span's. The token travels in a header,
which otelhttp does not record, and no attribute carries cloud-init.

## Timeouts

No caller puts a deadline on the context, so every request and every wait
carries its own budget (`defaultBudgets` in
`internal/provider/proxmox/proxmox.go`, and the transport in `client.go`):

| Bound | Budget |
|---|---|
| TCP connect | 10 s |
| TLS handshake | 10 s |
| one request, response included | `requestTimeoutSeconds`, default 30 s |
| clone task, and the wait for a VM's lock in the ownership check | 10 m |
| upload task, resize task, start task, stop task | 2 m each |
| destroy task | 5 m |
| seed delete task | 1 m |
| wait for `running` after the start | 60 s |
| task and status poll interval | 1 s |

A task wait that runs out is a transient error naming the UPID; the task
keeps running on Proxmox, and the next attempt finds its result through
the ownership check or the lock. A task that ends with any `exitstatus`
other than `OK` or `WARNINGS: <n>` is transient too, with Proxmox's text.

Requests share one HTTP client and its idle connections (4 per host,
closed after 90 s idle). A cancelled or overlong request ends only
itself: nothing else in flight is disturbed, and there is no connection
to kill as there is for kvm2. HTTP/2 is disabled, because pveproxy speaks
HTTP/1.1 only. Create and Delete for the same name are serialised on a
per-name lock, so the reconciler's Delete and GC's Delete never race on
stop and destroy; Get and ListByTag take no lock.

## Health check

The operator checks every configured node every
`--provider-check-interval` (default 1m) and publishes the result in the
`ProviderHealth` object named after the provider
(`kubectl get ph proxmox-home -o yaml`) and in the
`proxy_operator_provider_target_up{provider,target}` gauge. One target per
`nodes` entry, named after the node with the URL as its address, in
config order, all checked concurrently.

Each check (`checkNode` in `internal/provider/proxmox/check.go`) first
reads the token file for its id, then makes four requests on the shared
client: `GET /version`, `GET /nodes/{node}/status`,
`GET /nodes/{node}/storage/{isoStorage}/status` (the storage must exist,
be active and hold `iso` content), and `GET /cluster/resources?type=vm`
(every configured template must exist and be a template). A check that
runs out of time cancels only its own requests.

Every target's `details` carry `isoStorage`. As facts arrive they gain
`tokenID` (the part of the token before `=`, never the secret),
`pveVersion`, `kernel`, `isoStorageAvailGiB` and `templates`
(`<found>/<configured>`), so a target that fails late still says what the
node answered before. There is no quota signal; the storage's free space
is here instead.

| Reason | Meaning | Look at |
|---|---|---|
| `Reachable` | ready | |
| `CredentialsUnavailable` | the token file is missing or not `<user>@<realm>!<tokenid>=<secret>`; no request was made | the `proxmox-token` Secret; fixed in the cluster, not in Proxmox |
| `CertificateInvalid` | TLS verification failed, or `caFile` is missing or holds no certificate | `url` (its host must be in the node certificate) or `caFile`; not the node |
| `Unreachable` | the TCP connect or TLS handshake failed for a network reason, the connection was cut mid-response, or pveproxy answered 595 or 596 (it cannot reach the node behind it) | routing and firewalls between the cluster and `url`; `pveproxy` running; the node up |
| `Unauthorized` | 401: the token id or secret is wrong | the token file; `pveum user token list proxy-operator@pve` |
| `Forbidden` | 403; the message names the path and the privilege | the roles in *Proxmox prerequisites* |
| `Timeout` | the monitor's `--provider-check-timeout` (default 15s) ran out, whichever request was running, or a request ran out of `requestTimeoutSeconds` while the round still had time | a slow or hung pveproxy, a saturated link |
| `StorageNotFound` | `isoStorage` answered with a 500 on that node (it does not exist, is not enabled there, or is offline), is not active, or holds no `iso` content; the message carries Proxmox's text | `pvesm status`; the storage's nodes and content types |
| `TemplateNotFound` | a configured template VMID does not exist or is not a template | `qm list`; the `templates` catalogue |
| `APIError` | any other failure | the message carries Proxmox's own text |

The check does not look inside the templates: a cloud-init drive, a
missing `net0` or an unreadable disk size surface only when a Create is
about to clone.

## Error classes

Every error is classified once, in `classify` in
`internal/provider/proxmox/errors.go`, with errors that carry a class from
their source (validation, the token and CA files, the ownership check)
keeping it. The reconciler latches `ErrPermanent` as `Failed` until the
spec changes, recreates on `ErrNotFound` from `Get`, and retries
everything else with backoff. The provider never returns
`ErrQuotaExceeded`: the thin `local-lvm` storage overprovisions, so no
out-of-space answer could be produced to match, and capacity failures are
transient.

| Class | What lands there |
|---|---|
| `ErrPermanent` | placement values not in the catalogue; a template VMID that is missing or not a template, has a cloud-init drive, or lacks `net0` or a sized boot disk when the config needs them; a malformed providerID; any 4xx (401 `Authentication failed!` or `No ticket`: the token; 403 `Permission check failed (<path>, <privilege>)`: a missing privilege, including `Datastore.Allocate` for the seed delete; 400 `Parameter verification failed.` with the field errors); a TLS verification failure; a token file that exists but is not `<user>@<realm>!<tokenid>=<secret>`; a CA file with no certificate; a VM with the proxy's name that carries no mark or another UID; two VMs with the proxy's name; our VM hibernated (`lock: suspended`), in both Create and Delete |
| `ErrNotFound` | from `Get`: the VMID absent from the cluster's resources, the 500 `Configuration file 'nodes/<node>/qemu-server/<vmid>.conf' does not exist`, the VM at the VMID now named differently or unmarked. Inside `Create` the `does not exist` answer is `ErrTransient`, because the contract forbids NotFound from Create; `Delete` treats all of these as done |
| `ErrTransient` | everything else: a missing token or CA file (the Secret has not landed yet); network errors and cancellation; 500 `VM is locked (<lock>)` and any lock seen in the resources or config; a VM whose node does not answer (`status: unknown`); the agent's answers that `Get` cannot place, such as `VM <vmid> is not running`; any other 5xx, including pveproxy's 595 and 596; a task that fails or runs out of its budget; no free VMID after 32 probes; the wait for running expiring, or a VM that stops right after its start |

`Get` turns two agent answers into states rather than errors:
`QEMU guest agent is not running` is Provisioning (booting, or the agent
install has not finished), and `No QEMU guest agent configured` is
Stopped (someone removed `agent: 1`; the reconciler replaces the VM).

Error strings name VMIDs, nodes, storages, UPIDs and Proxmox's own text.
They never contain the token, the CA content or cloud-init.

## Known gaps

- **Create blocks** until the VM runs, holding one of the three reconcile
  workers for its duration. On `pv1-1` that is about five seconds plus the
  clone; large templates on slow storage take longer. Other Proxies keep
  reconciling on the remaining workers.
- **A clone lock that never clears** (a node crash mid-clone) makes every
  Create for that Proxy wait out the 10 minute clone budget and return a
  transient error. Clear it with `qm unlock <vmid>` or destroy the VM (see
  *What lands on Proxmox*).
- **Seed ISOs of VMs destroyed outside the operator are invisible to
  GC.** Remove `proxy-*-seed.iso` files without a matching VM by hand.
  On a storage that is not shared, a VM migrated to another node leaves
  its seed on the original node; Delete then looks on the VM's current
  node, finds nothing, and the seed stays. Shared ISO storage avoids it.
- **The VMID range is shared with humans.** A VM created by hand inside
  it costs a probe; a hand-made VM named like a proxy is refused as
  `ErrPermanent`. Reserve the range.
- **Cross-node clones need shared storage and are untested.** When the
  zone's node differs from the template's, the clone is sent with
  `target`, which Proxmox allows only when the template's disks are on
  shared storage. The host this was built on has one node.
- **A VM whose agent was switched off is replaced**, not repaired: with
  `agent: 1` removed it can never report an address, so `Get` reports it
  Stopped.
- **The proxy network needs internet** for the agent install, unless the
  template ships the agent and `installGuestAgent` is `false`. Without it
  the VM boots, gets an address, and stays `Provisioning` forever,
  because the agent is the only address source. Check
  `systemctl status qemu-guest-agent` and
  `/var/log/cloud-init-output.log` on the guest (`qm terminal <vmid>` on
  the node, or the console in the UI).
- **The CA is cached** after first use; rotating it needs a manager
  restart. The token rotates freely.
- **A node that stops answering** makes `Get` and `Delete` for its VMs
  transient until it returns, so deleting such a Proxy waits. Proxmox
  lists an unreachable node's guests with `status: unknown` rather than
  dropping them, so they are never mistaken for gone. That is Proxmox's
  documented behaviour; a single-node host could not show it.
- **Contract limits that apply unchanged** ([provider-contract.md
  §13](provider-contract.md)): the providers config is read once per
  process; GC has no installation identity, so two operators sharing a
  Proxmox cluster and a VMID range delete each other's VMs; and a Proxy
  naming an unconfigured provider latches until its spec changes.

Beyond the ten limits accepted in the design:

- **A hibernated VM latches its Proxy.** A VM of ours with
  `lock: suspended` (hibernated to disk) makes both Create and Delete fail
  permanently with `resume or destroy it in Proxmox`: the lock never
  clears on its own, and the destroy refuses a locked VM. `Get` reports it
  Stopped. Resume it by starting it (`qm start <vmid>` restores the
  hibernated state) or destroy it on the node, then touch the Proxy's
  spec, or let the pending Delete retry.
- **The seed upload is one request**, bounded by `requestTimeoutSeconds`.
  A seed is tens of KiB, but a multi-megabyte `vendorData` would need a
  larger timeout.
- **The sample cloud-init is an open proxy.**
  `config/samples/proxy_proxmox.yaml` installs Squid with
  `http_access allow all`; restrict it before exposing the proxy network
  to anything but the cluster.
- **Not yet proven end to end.** The unit tests run over captured
  responses from `pv1-1` and an in-memory fake; the integration tests
  below compile but have not run against a host yet. Three answers were
  never captured: the storage status for a missing storage, a destroy on
  a hibernated VM, and a stop on a paused VM.

## Testing

Unit tests run over an in-memory fake of the API seam, and the HTTP
client over a TLS test server replaying the responses captured from
`pv1-1` (`internal/provider/proxmox/testdata/`). The optional integration
test against a real host is described in
[testing.md](testing.md#the-proxmox-integration-test-real-proxmox-ve-host),
which also lists the variables it needs; run it with
`make test-integration-proxmox`. `TestProxmoxCheck_realHost` runs the
health check against the same host and creates nothing:
`go test -tags=integration -count=1 -v -run TestProxmoxCheck ./internal/provider/proxmox/`.