egress-proxies-operator/docs/proxmox-provider.md

The proxmox provider

Runs Managed proxies as VMs on a Proxmox VE cluster. It is the third VM-style provider and the second built from provider-contract.md, from the design in proxmox-provider-spec.md. The operator talks to the cluster's REST API only (https://<host>:8006/api2/json) with an API token. It needs no SSH, no shell on the nodes, and no library beyond Go's standard net/http.

Each VM is a full clone of a template you prepare once. The operator builds the Proxy's cloud-init into a NoCloud seed ISO in-process, uploads it to a storage on the VM's node, attaches it as a CD-ROM, sets the size, starts the VM and learns its address from the QEMU guest agent. Everything here was proven against Proxmox VE 9.2.20 (QEMU 11.0) on the single node pv1-1; every error text the provider matches was captured from that host.

Proxmox prerequisites

Once per cluster, as root on a node:

1. Proxmox VE 9.x with the API reachable from the operator's pods on port 8006. The operator's pods and the crawler clients must also route to the network the proxies land on (see step 5).

2. A template without a Proxmox cloud-init drive. The provider brings its own seed on ide2; a Proxmox cloud-init drive would compete with it and the Proxy's user-data would be ignored. Debian 13's generic cloud image boots UEFI (under BIOS it hangs in GRUB on a headless VM), so the template does too:

   wget https://cloud.debian.org/images/cloud/trixie/latest/debian-13-generic-amd64.qcow2
   qm create 9001 --name debian-13-template --memory 2048 --cores 2 --ostype l26 \
     --machine q35 --bios ovmf --efidisk0 local-lvm:1,efitype=4m,pre-enrolled-keys=0 \
     --scsihw virtio-scsi-pci --net0 virtio,bridge=vmbr0 \
     --serial0 socket --vga serial0 --agent 1
   qm set 9001 --scsi0 local-lvm:0,import-from=/root/debian-13-generic-amd64.qcow2
   qm set 9001 --boot order=scsi0
   qm template 9001
   

   The template must boot from its first disk, run cloud-init with the NoCloud datasource (the Debian and Ubuntu cloud images do), and either ship qemu-guest-agent or be able to install it at first boot. Before the first clone for a Proxy, Create reads the template's config and refuses it with a permanent error when:

   • it is not a template (qm template was not run);
   • any disk slot holds a Proxmox cloud-init drive (a value containing cloudinit), such as the one qm set <vmid> --ide2 local-lvm:cloudinit adds for Proxmox's own cloud-init support;
   • it has no net0 while the config sets bridge or vlanTag;
   • the machine type sets diskGiB and the template has no boot disk, or one without a readable size=.

   These are checked on the template because the clone copies its config: a VM cloned from a bad template could not be fixed by fixing the template. The template is checked only when a VM is about to be cloned; a VM that already exists no longer depends on it. The health check sees only whether each template exists and is a template.

3. A storage for the seed ISOs, holding iso content. A dedicated directory storage is best, because the token needs Datastore.Allocate on it to delete seeds, and on a dedicated storage that privilege cannot touch backups or other ISOs:

   pvesm add dir proxy-seeds --path /var/lib/proxy-seeds --content iso
   

   A seed is tens of KiB. Proxmox stages uploads under /var/tmp, not the node's tmpfs /tmp. On a cluster, a storage that is not shared keeps each seed on the node the VM was created on (see Known gaps).

4. A user, two roles and a token. VM.Monitor no longer exists in PVE 9 and pveum rejects it; the agent read needs VM.GuestAgent.Audit alone. VM.Config.Cloudinit is not needed for the seed on ide2. SDN.Use on / is what lets the config write put a VLAN tag on vmbr0:

   pveum user add proxy-operator@pve --comment "egress-proxies-operator"
   pveum role add ProxyOperator -privs "VM.Allocate VM.Clone VM.Audit VM.Config.CDROM VM.Config.CPU VM.Config.Memory VM.Config.Disk VM.Config.Network VM.Config.Options VM.PowerMgmt VM.GuestAgent.Audit Datastore.Audit Datastore.AllocateSpace Datastore.AllocateTemplate Sys.Audit SDN.Use"
   pveum aclmod / -user proxy-operator@pve -role ProxyOperator
   pveum role add ProxyOperatorSeeds -privs "Datastore.Allocate Datastore.AllocateTemplate Datastore.AllocateSpace Datastore.Audit"
   pveum aclmod /storage/proxy-seeds -user proxy-operator@pve -role ProxyOperatorSeeds
   pveum user token add proxy-operator@pve operator --privsep 0
   

   The second role repeats three privileges of the first on purpose: an ACL on a more specific path replaces what the user inherits from / for that path, so a role with Datastore.Allocate alone would take the upload's Datastore.AllocateTemplate away. The role on / covers clone, config, resize, start, stop, the agent read, the upload and the destroy; what it lacks is Datastore.Allocate, without which deleting a seed is a 403 (Permission check failed (/storage/<id>, Datastore.Allocate)). With --privsep 0 the token carries the user's privileges.

   The token's secret is printed once. Save it on your workstation as one line, proxy-operator@pve!operator=<secret>, in ~/.config/proxmox-token with mode 0600. Copy the node's CA, /etc/pve/pve-root-ca.pem, to ~/.config/proxmox-ca.pem.

5. A proxy network with DHCP and a route to the internet. Each VM keeps the template's one NIC, optionally moved to another bridge or VLAN by the config, and takes its address from that network's DHCP. The network needs internet twice over: the vendor-data script installs qemu-guest-agent from the distro mirrors at first boot, and the agent is the only way the operator learns the VM's address; and an egress proxy needs egress anyway. Baking qemu-guest-agent into the template and setting installGuestAgent: false removes the mirror from IP discovery. The operator's pods and the clients must route to that network: the health engine probes http://<ip>:<port> through the proxy, and clients get the same address from the discovery API. The provider does not use spec.port and configures no Proxmox firewall.

6. A reserved VMID range. VMIDs are allocated from vmidRange (default 100000 to 199999). Keep hand-made VMs out of it: one there costs a probe, and one named like a proxy is refused (see What lands on Proxmox).

Check from the operator's point of view before wiring it in. This sends the token through a file descriptor, so it never reaches the command line:

pve() {
  curl -s --cacert ~/.config/proxmox-ca.pem \
    -H @<(printf 'Authorization: PVEAPIToken=%s\n' "$(tr -d '[:space:]' < ~/.config/proxmox-token)") \
    "https://192.168.0.8:8006/api2/json/$1"; echo
}
pve version                                  # the token works, TLS verifies
pve nodes/pv1-1/storage/proxy-seeds/status   # active, content includes iso
pve 'cluster/resources?type=vm'              # the template is listed with "template":1

Cluster-side setup

1. Create the Secret. It holds two keys, token (the one-line token file) and ca.pem (the node's CA):

   make proxmox-token-secret
   # or by hand:
   kubectl -n egress-proxies-operator-system create secret generic proxmox-token \
     --from-file=token=$HOME/.config/proxmox-token \
     --from-file=ca.pem=$HOME/.config/proxmox-ca.pem
   

   The make target runs that command as an apply, so it also updates an existing Secret. Override the paths with PROXMOX_TOKEN_FILE=... PROXMOX_CA_FILE=.... Before it applies, it checks that both files exist, that the token file is exactly one line of <user>@<realm>!<tokenid>=<secret> (without printing it), and that the CA file holds a PEM certificate. With insecureSkipTLSVerify instead of a CA, create the Secret by hand with the token key only.

   config/manager/manager.yaml mounts the Secret (optional, read-only) at /etc/proxy-operator/proxmox, so the files are /etc/proxy-operator/proxmox/token and /etc/proxy-operator/proxmox/ca.pem. The deployment starts fine without it, even with a proxmox provider configured: New makes no request, only checks that the token file exists, and logs a WARNING: line when it does not. The provider's proxies then fail transiently and recover once the Secret lands; the health check reports CredentialsUnavailable meanwhile.

   Rotating the token needs no restart. The token file is read on every request, so the next request after the kubelet refreshes the mount uses the new token. Keep the old token valid until then: a 401 is a permanent error, so a Proxy whose request lands in the gap latches until its spec changes. Add the new token (pveum user token add), update the Secret, wait a couple of minutes for the mount to refresh, and only then remove the old token. A token created with --expire needs the same rotation before it expires. Rotating the CA does need a restart: it is loaded on first use and cached for the life of the process. A missing CA file is retried until it lands; one without a certificate is a permanent error.

2. Add a provider entry to the providers config (config/manager/providers_config.yaml in-cluster; the full sample, commented out, is in config/samples/providers-config.yaml):

   - name: proxmox-home
     type: proxmox
     proxmox:
       url: https://192.168.0.8:8006
       tokenFile: /etc/proxy-operator/proxmox/token
       caFile: /etc/proxy-operator/proxmox/ca.pem
       nodes: [pv1-1]
       isoStorage: proxy-seeds
       vlanTag: 4
       templates:
         debian-13: {vmid: 9001}
       machineTypes:
         small: {cores: 1, memoryMiB: 1024}
   

   The URL's host must be in the node certificate. The node's self-signed certificate lists its own addresses and names, and TLS verification checks the URL's host against them. On pv1-1 it lists 192.168.0.8, pv1-1 and pv1-1.home.lab, so https://192.168.2.8:8006 reaches the same node and still fails verification (CertificateInvalid in the health check, a permanent error on every call). See what yours lists with:

   openssl s_client -connect 192.168.0.8:8006 </dev/null 2>/dev/null | openssl x509 -noout -ext subjectAltName
   

   If the address the operator must use is not there, either give the node a certificate that names it (a custom certificate in the node's Certificates panel) and put that certificate's CA in caFile, or set insecureSkipTLSVerify: true, which New logs as a WARNING: line.

   nodes holds node names as Proxmox knows them, not addresses: the one URL reaches the whole cluster. The providers config is read once per process, so adding a node, a template or a machine type needs a manager restart.

3. Apply a Proxy (config/samples/proxy_proxmox.yaml) and watch it:

   kubectl get px -w
   # NAME                   MODE      PROVIDER       PHASE          IP              HEALTHY
   # proxy-proxmox-sample   Managed   proxmox-home   Provisioning
   # proxy-proxmox-sample   Managed   proxmox-home   Running        192.168.4.147
   # proxy-proxmox-sample   Managed   proxmox-home   Ready          192.168.4.147   True
   

   (Illustrative.) Create returns once the VM runs, a few seconds after the clone; the address follows when the guest agent is up, about 50 seconds after the start on pv1-1 with the agent installed at first boot.

Config reference

The block is proxmox: and its Go type is ProxmoxConfig in internal/provider/config_proxmox.go, validated at load by validateProxmox; unknown keys are rejected. Defaults are applied by withDefaults in internal/provider/proxmox/proxmox.go.

Field	Required	Default	Meaning
url	yes		https://<host>:8006, with no path (the client appends /api2/json). The host must be in the node certificate
tokenFile	yes		file holding one line, <user>@<realm>!<tokenid>=<secret>, read on every request and sent as Authorization: PVEAPIToken=<line>
caFile		system roots	PEM bundle trusted for the API's certificate, typically the node's /etc/pve/pve-root-ca.pem. Loaded on first use and cached
insecureSkipTLSVerify		false	skip certificate verification; New logs a WARNING: line when it is on
nodes	yes, at least one		node names placement.zone selects from; unique, lowercase letters, digits and dashes. They end up in request paths and as the clone's target
isoStorage	yes		storage id the seeds are uploaded to and deleted from. Must hold iso content; the token needs Datastore.Allocate on it. No default on purpose
storage		the template's disk storage	storage id the full clone's disks go to
bridge		as cloned	replaces bridge= in the cloned net0
vlanTag		as cloned	1 to 4094; replaces or appends tag= in the cloned net0
vmidRange.min, vmidRange.max		100000, 199999	where VMIDs are allocated; at least 100, at most 999999999, min below max
templates.<key>.vmid	at least one template		the key names the template in placement.image; the VMID is at least 100
machineTypes.<key>	at least one		{cores, memoryMiB, diskGiB} size profile. cores and memoryMiB above zero. diskGiB is optional: when larger than the cloned boot disk the disk is grown to it; it is never shrunk
requestTimeoutSeconds		30	bound on one API request, response included. The seed upload is one request, so a multi-megabyte vendorData needs more
vendorData			extra cloud-init vendor-data placed on every seed next to the Proxy's user-data; a MIME multipart part beside the agent script when installGuestAgent is on. Cloud-init drops any top-level key of a vendor-data cloud-config that the user-data also defines
installGuestAgent		true	add the vendor-data script that installs and starts qemu-guest-agent. Turn it off for templates that ship the agent

Placement

All three catalogue fields are checked before any request. An unknown value is ErrPermanent, which the reconciler latches as Failed with the message below until the spec changes.

Proxy field	Meaning for proxmox	Error when unknown
placement.zone	a node from nodes. May be omitted when exactly one node is configured; otherwise required	placement.zone "x" is not in proxmox.nodes; with several nodes and no zone, placement.zone is required when proxmox.nodes lists more than one node
placement.machineType	a key in machineTypes	placement.machineType "x" is not in proxmox.machineTypes
placement.image	a key in templates	placement.image "x" is not in proxmox.templates
placement.region	ignored. It is still part of the spec hash, so changing it replaces the VM and does nothing else

The messages come from placement in internal/provider/proxmox/create.go and node in proxmox.go. A key that exists but names a VMID that is missing or not a template fails later, before any clone: placement.image "x": template VM 9001 does not exist or ... is not a template.

Catalogue edits affect only new VMs. The spec hash covers the placement values, not what the config maps them to, so repointing templates.debian-13.vmid at another template changes the VMs created afterwards and leaves the running ones alone. Add a new key rather than repointing one in use, so the Proxies that name a key keep a meaning you can reason about. A template may be edited, replaced or deleted without affecting existing VMs: they are full clones and share nothing with it.

What lands on Proxmox

For a Proxy whose deterministic name is proxy-abc…:

Object	Name	Lifetime
VM	name proxy-abc…, a VMID from vmidRange	with the Proxy
its disks	the template's disks, fully copied onto storage (or the template's storage)	destroyed with the VM (purge=1&destroy-unreferenced-disks=1)
seed ISO	<isoStorage>:iso/proxy-abc…-seed.iso on the VM's node	with the Proxy

The VMID is derived from the Proxy's UID, so a retried Create lands on the same number: an FNV-1a hash of the UID placed in vmidRange, moving forward one at a time on collision, up to 32 probes (vmidFor in id.go). Before allocating, Create looks the name up in the cluster's resources. A VMID that is taken, or a VM that already has the name, goes through an ownership check (inspect in create.go): a VM with a lock (a clone still running, which also has no name yet) is waited on; then

• another name: someone else's VM, so the next VMID is probed;
• our name and our mark with our UID or an empty one: adopted, and Create carries on from where the previous attempt stopped;
• our name without the mark, or with another UID: permanent error naming the VMID and both UIDs. The operator never takes over a VM it did not create;
• our name, our mark, hibernated (lock: suspended): permanent error, resume or destroy it in Proxmox (see Known gaps). This only happens when Create has not recorded the VM yet; a recorded VM that gets hibernated is reported Stopped and replaced.

Two VMs with the proxy's name are a permanent error naming every VMID; remove all but one.

The providerID is vms/<vmid>/<name>. It names no node, because VMs migrate: the node is looked up in GET /cluster/resources on every call. Get and Delete check that the VM at that VMID still has that name and the mark. Another name means ours is gone and the VMID was reused: Get answers NotFound, Delete succeeds without touching it.

The mark is three lines in the VM's description (Notes in the UI), passed in the clone call itself, so a VM is never unmarked:

proxy-operator-managed=true
proxy-operator-uid=0f3a…
proxy-operator-created-at=2026-09-23T17:42:11Z

That is how orphan GC recognises its own VMs and how a repeat Create knows whether a VM of the same name belongs to this Proxy. Lines an admin adds to the description are kept and ignored. The clone's own meta: ctime= is useless for this: Proxmox copies it from the template.

The config write. After the clone and the seed upload, one PUT /nodes/{node}/qemu/{vmid}/config sets:

Key	Value
cores, memory	from the machine type
agent	1
tags	proxy_operator
ide2	<isoStorage>:iso/<name>-seed.iso,media=cdrom
net0	only when bridge or vlanTag is set: the cloned value with bridge= and tag= replaced or appended, the model, MAC and every other option kept

The write replaces the template's agent option and tags: a template's agent: 1,fstrim_cloned_disks=1 becomes agent: 1, and its tags give way to proxy_operator. The tag is for filtering in the Proxmox UI and narrows GC's candidates; the mark decides. Proxmox appends ide2 to the boot order on its own (order=scsi0;ide2), which is harmless because the boot disk stays first. When the machine type's diskGiB is larger than the boot disk (the first disk in boot: order= that is not a CD-ROM), a resize task grows it; cloud-init's growpart expands the root filesystem at first boot.

The seed (nocloud.BuildSeedISO, shared with kvm2 in internal/provider/nocloud) is an ISO 9660 image labelled cidata. user-data is the Proxy's cloudInit byte for byte; the operator never parses or re-renders it. meta-data holds instance-id: <name> and local-hostname: <name>. vendor-data carries the guest-agent script (a shell script, not a cloud-config, so a Proxy's own packages: cannot shadow it), joined with vendorData as a MIME multipart when both are set. It is staged under /tmp (the manager's emptyDir), built in memory and uploaded with POST /nodes/{node}/storage/{isoStorage}/upload. An upload overwrites a file of the same name, so it is rebuilt and re-uploaded on every Create attempt with no delete first: a previous attempt may have been cut off. Only the API is used; the storage upload endpoint does not accept snippets, so Proxmox's own cicustom would need shell access to the node.

What a retried Create finds. Every step is safe to repeat. After a crash mid-clone, the retry lands on the same VMID, waits for the clone's lock, and adopts the VM. A stopped VM of ours gets the seed, the config and the resize again (the upload overwrites, the config is a replace, the resize is skipped once large enough), then the start. A running one is returned at once.

A clone lock that never clears. A node that crashes mid-clone can leave the VM locked with lock: clone for good. Every Create for that Proxy then waits out the 10 minute clone budget and returns a transient error, and the reconciler keeps retrying rather than latching. Clear it on the node:

qm unlock <vmid>      # then let the next Create adopt and finish it
qm destroy <vmid> --purge   # or remove it and let Create clone again

Seeds without a VM. A VM destroyed outside the operator leaves its seed ISO, which GC cannot see (it lists VMs). Seeds are named proxy-*-seed.iso on isoStorage; any without a matching VM may be removed by hand:

pvesm list proxy-seeds --content iso
pvesm free proxy-seeds:iso/proxy-abc…-seed.iso

Delete runs under a per-name lock shared with Create: stop (a paused VM too), delete the seed on the VM's node, then destroy with purge=1&destroy-unreferenced-disks=1. Each is a task that is waited on. The seed goes before the VM because once the VM is gone nothing calls Delete again; a failed seed delete leaves the VM stopped, and the next Delete finishes both. A VM that is already gone, not ours, or unmarked is left alone, and the seed is then looked for on every configured node, where a node that fails is logged and skipped. A refusal every node would give fails the call instead: without Datastore.Allocate on isoStorage the 403 makes Delete permanent on either path, and the Proxy's deletion stays stuck with that message (a replaced Proxy stays failed, its VM stopped) until the privilege is granted. When Delete returns nil, nothing of the Proxy is left.

Logging

The provider logs through controller-runtime's logger, named proxmox, with provider (the configured name) on every line, and vmid, name and node on every line about an instance. Every line inside a traced operation also carries traceID and spanID.

• At startup (always on): WARNING: token file is not readable; every call fails until it is with path and reason, and WARNING: insecureSkipTLSVerify is set.
• --zap-log-level=debug (logr V(1)):
  • one line per operation outcome: proxmox vm ensured running, proxmox vm adopted with status and qmpstatus, proxmox vm fetched with state, ip, uid, status and qmpstatus, proxmox vm not found with reason, proxmox vm deleted with destroyed (false when no VM of ours was left to destroy), proxmox vms listed with vms, candidates, managed and skipped counts, and proxmox node checked with ready, reason and latency;
  • one proxmox api call line per API request, written by the decorator in internal/provider/proxmox/observe.go: call (the seam method, for example Clone), the identifiers it was given (node, vmid, storage), upid for a task, newid and name on a clone, filename on an upload, volid on a volume delete, duration, and on failure httpStatus and error. A running Get is four calls; a fresh Create is a dozen or more, depending on how many polls each task takes;
  • proxmox task failed with upid and exitstatus, and proxmox task finished with warnings;
  • the no-op lines: proxmox vm already gone, proxmox vm at this vmid is not ours, left alone with vmName, proxmox vm carries no proxy-operator mark, left alone, proxmox seed delete failed on a node, skipped, skipping unreadable proxmox vm, and the probe's proxmox vmid taken by another vm, probing on.
• --zap-log-level=2: proxmox create resolved with the resolved parameters (node, machine type, cores, memory, disk, template VMID, storages, bridge, VLAN tag) and cloudInitBytes; the proxmox api call lines also carry the config write's parameters, a summary of each config read, the clone's description, storage and target, the resize size, VM and task status, the agent's addresses and the upload's byte count; proxmox guest agent answered; proxmox listed instance per listed VM.
• Never logged at any level: the cloud-init content, the token, the CA content and the upload body. There is no raw wire tier.

Tracing

With an OTLP exporter configured (architecture.md §10), every API request is a child span of the operation span the decorators open (provider.create, provider.get, provider.delete, provider.list) or of the monitor's providerhealth.check:

Span	Around	Attributes
HTTP GET, HTTP POST, HTTP PUT, HTTP DELETE	one API request, from otelhttp on the client's transport	otelhttp's standard HTTP client attributes: method, URL, server address and port, response status

The transport is tracing.RestConfigWrapper(), the same parent-gated wrapper the kubernetes provider uses, so a request without a parent span starts none. The logging decorator starts no span of its own, so nothing is counted twice; its proxmox api call line carries the operation span's spanID, not the HTTP span's. The token travels in a header, which otelhttp does not record, and no attribute carries cloud-init.

Timeouts

No caller puts a deadline on the context, so every request and every wait carries its own budget (defaultBudgets in internal/provider/proxmox/proxmox.go, and the transport in client.go):

Bound	Budget
TCP connect	10 s
TLS handshake	10 s
one request, response included	requestTimeoutSeconds, default 30 s
clone task, and the wait for a VM's lock in the ownership check	10 m
upload task, resize task, start task, stop task	2 m each
destroy task	5 m
seed delete task	1 m
wait for running after the start	60 s
task and status poll interval	1 s

A task wait that runs out is a transient error naming the UPID; the task keeps running on Proxmox, and the next attempt finds its result through the ownership check or the lock. A task that ends with any exitstatus other than OK or WARNINGS: <n> is transient too, with Proxmox's text.

Requests share one HTTP client and its idle connections (4 per host, closed after 90 s idle). A cancelled or overlong request ends only itself: nothing else in flight is disturbed, and there is no connection to kill as there is for kvm2. HTTP/2 is disabled, because pveproxy speaks HTTP/1.1 only. Create and Delete for the same name are serialised on a per-name lock, so the reconciler's Delete and GC's Delete never race on stop and destroy; Get and ListByTag take no lock.

Health check

The operator checks every configured node every --provider-check-interval (default 1m) and publishes the result in the ProviderHealth object named after the provider (kubectl get ph proxmox-home -o yaml) and in the proxy_operator_provider_target_up{provider,target} gauge. One target per nodes entry, named after the node with the URL as its address, in config order, all checked concurrently.

Each check (checkNode in internal/provider/proxmox/check.go) first reads the token file for its id, then makes four requests on the shared client: GET /version, GET /nodes/{node}/status, GET /nodes/{node}/storage/{isoStorage}/status (the storage must exist, be active and hold iso content), and GET /cluster/resources?type=vm (every configured template must exist and be a template). A check that runs out of time cancels only its own requests.

Every target's details carry isoStorage. As facts arrive they gain tokenID (the part of the token before =, never the secret), pveVersion, kernel, isoStorageAvailGiB and templates (<found>/<configured>), so a target that fails late still says what the node answered before. There is no quota signal; the storage's free space is here instead.

Reason	Meaning	Look at
Reachable	ready
CredentialsUnavailable	the token file is missing or not <user>@<realm>!<tokenid>=<secret>; no request was made	the proxmox-token Secret; fixed in the cluster, not in Proxmox
CertificateInvalid	TLS verification failed, or caFile is missing or holds no certificate	url (its host must be in the node certificate) or caFile; not the node
Unreachable	the TCP connect or TLS handshake failed for a network reason, the connection was cut mid-response, or pveproxy answered 595 or 596 (it cannot reach the node behind it)	routing and firewalls between the cluster and url; pveproxy running; the node up
Unauthorized	401: the token id or secret is wrong	the token file; pveum user token list proxy-operator@pve
Forbidden	403; the message names the path and the privilege	the roles in Proxmox prerequisites
Timeout	the monitor's --provider-check-timeout (default 15s) ran out, whichever request was running, or a request ran out of requestTimeoutSeconds while the round still had time	a slow or hung pveproxy, a saturated link
StorageNotFound	isoStorage answered with a 500 on that node (it does not exist, is not enabled there, or is offline), is not active, or holds no iso content; the message carries Proxmox's text	pvesm status; the storage's nodes and content types
TemplateNotFound	a configured template VMID does not exist or is not a template	qm list; the templates catalogue
APIError	any other failure	the message carries Proxmox's own text

The check does not look inside the templates: a cloud-init drive, a missing net0 or an unreadable disk size surface only when a Create is about to clone.

Error classes

Every error is classified once, in classify in internal/provider/proxmox/errors.go, with errors that carry a class from their source (validation, the token and CA files, the ownership check) keeping it. The reconciler latches ErrPermanent as Failed until the spec changes, recreates on ErrNotFound from Get, and retries everything else with backoff. The provider never returns ErrQuotaExceeded: the thin local-lvm storage overprovisions, so no out-of-space answer could be produced to match, and capacity failures are transient.

Class	What lands there
ErrPermanent	placement values not in the catalogue; a template VMID that is missing or not a template, has a cloud-init drive, or lacks net0 or a sized boot disk when the config needs them; a malformed providerID; any 4xx (401 Authentication failed! or No ticket: the token; 403 Permission check failed (<path>, <privilege>): a missing privilege, including Datastore.Allocate for the seed delete; 400 Parameter verification failed. with the field errors); a TLS verification failure; a token file that exists but is not <user>@<realm>!<tokenid>=<secret>; a CA file with no certificate; a VM with the proxy's name that carries no mark or another UID; two VMs with the proxy's name; our VM hibernated (lock: suspended) found by Create's adoption
ErrNotFound	from Get: the VMID absent from the cluster's resources, the 500 Configuration file 'nodes/<node>/qemu-server/<vmid>.conf' does not exist, the VM at the VMID now named differently or unmarked. Inside Create the does not exist answer is ErrTransient, because the contract forbids NotFound from Create; Delete treats all of these as done
ErrTransient	everything else: a missing token or CA file (the Secret has not landed yet); network errors and cancellation; 500 VM is locked (<lock>) and any lock seen in the resources or config; a VM whose node does not answer, or one a few seconds old the listing has not reported yet (status: unknown); the agent's answers that Get cannot place, such as VM <vmid> is not running; any other 5xx, including pveproxy's 595 and 596; a task that fails or runs out of its budget; no free VMID after 32 probes; the wait for running expiring, or a VM that stops right after its start

Get turns two agent answers into states rather than errors: QEMU guest agent is not running is Provisioning (booting, or the agent install has not finished), and No QEMU guest agent configured is Stopped (someone removed agent: 1; the reconciler replaces the VM).

Error strings name VMIDs, nodes, storages, UPIDs and Proxmox's own text. They never contain the token, the CA content or cloud-init.

Known gaps

• Create blocks until the VM runs, holding one of the three reconcile workers for its duration. On pv1-1 that is about five seconds plus the clone; large templates on slow storage take longer. Other Proxies keep reconciling on the remaining workers.
• A clone lock that never clears (a node crash mid-clone) makes every Create for that Proxy wait out the 10 minute clone budget and return a transient error. Clear it with qm unlock <vmid> or destroy the VM (see What lands on Proxmox).
• Seed ISOs of VMs destroyed outside the operator are invisible to GC. Remove proxy-*-seed.iso files without a matching VM by hand. On a storage that is not shared, a VM migrated to another node leaves its seed on the original node; Delete then looks on the VM's current node, finds nothing, and the seed stays. Shared ISO storage avoids it.
• The VMID range is shared with humans. A VM created by hand inside it costs a probe; a hand-made VM named like a proxy is refused as ErrPermanent. Reserve the range.
• Cross-node clones need shared storage and are untested. When the zone's node differs from the template's, the clone is sent with target, which Proxmox allows only when the template's disks are on shared storage. The host this was built on has one node.
• A VM whose agent was switched off is replaced, not repaired: with agent: 1 removed it can never report an address, so Get reports it Stopped.
• The proxy network needs internet for the agent install, unless the template ships the agent and installGuestAgent is false. Without it the VM boots, gets an address, and stays Provisioning forever, because the agent is the only address source. Check systemctl status qemu-guest-agent and /var/log/cloud-init-output.log on the guest (qm terminal <vmid> on the node, or the console in the UI).
• The CA is cached after first use; rotating it needs a manager restart. The token rotates freely.
• A node that stops answering makes Get and Delete for its VMs transient until it returns, so deleting such a Proxy waits. Proxmox lists an unreachable node's guests with status: unknown rather than dropping them, so they are never mistaken for gone. That is Proxmox's documented behaviour; a single-node host could not show it.
• Contract limits that apply unchanged (provider-contract.md §13): the providers config is read once per process; GC has no installation identity, so two operators sharing a Proxmox cluster and a VMID range delete each other's VMs; and a Proxy naming an unconfigured provider latches until its spec changes.

Beyond the ten limits accepted in the design:

• A hibernated VM is replaced, or latches a Create. Get reports a VM of ours with lock: suspended (hibernated to disk) as Stopped, so the reconciler deletes it (the destroy removes its suspend state volume too) and creates a new one. Only a Create that finds such a VM before it was ever recorded, by name, fails permanently with resume or destroy it in Proxmox: resume it by starting it (qm start <vmid> restores the hibernated state) or destroy it on the node, then touch the Proxy's spec.
• A Create that failed permanently after its clone keeps its VM. The stopped VM is adopted by name on the next Create, even when the Proxy's placement has changed since: the Proxy then runs the old template on the old node. After changing placement on such a Proxy, destroy the VM on the node (qm destroy <vmid> --purge) first, or delete and recreate the Proxy.
• A retry can clone a second VM in one rare sequence: a Create is cut off while cloning onto a later probe VMID (the first was taken by someone else's VM), that other VM is removed, and the retry comes while the clone is still running, so its VM is still nameless. The retry clones onto the now free first VMID. The next Create then fails permanently with VMs … are all named …; destroy either VM.
• The seed upload is one request, bounded by requestTimeoutSeconds. A seed is tens of KiB, but a multi-megabyte vendorData would need a larger timeout.
• The sample cloud-init is an open proxy. config/samples/proxy_proxmox.yaml installs Squid with http_access allow all; restrict it before exposing the proxy network to anything but the cluster.

Testing

Unit tests run over an in-memory fake of the API seam, and the HTTP client over a TLS test server replaying the responses captured from pv1-1 (internal/provider/proxmox/testdata/). The optional integration test against a real host is described in testing.md, which also lists the variables it needs; run it with make test-integration-proxmox. TestProxmoxCheck_realHost runs the health check against the same host and creates nothing: go test -tags=integration -count=1 -v -run TestProxmoxCheck ./internal/provider/proxmox/. Both passed on pv1-1 (PVE 9.2.20) on 2026-09-24: Create took 6.2 s, and the guest reported an address 50 s after Create returned.