kacerr/egress-proxies-operator
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

proxmox: review fixes, real-host validation record, changelog

Browse Source 
Phase 10. Two review passes (go-operator-reviewer, /code-review high)
over the branch, recorded in docs/reviews/2026-09-23-proxmox-provider-review/:
9 findings fixed, 3 accepted as documented limits, 7 rejected with a
reason.

- Delete removes the seed between the stop and the destroy, so a failed
  seed delete leaves the VM for the next Delete instead of leaking the
  ISO; a Permanent refusal on the every-node path fails the call.
- Create skips a probe VMID whose listed name is someone else's without
  waiting on its lock.
- The three answers Phase 0 never captured are fixtures now: a missing
  storage is a 400 (the Checker reports StorageNotFound), a hibernated
  VM is destroyable (Delete destroys ours), a paused VM stops.
- TestProxmox_lifecycle and TestProxmoxCheck_realHost passed on pv1-1
  (PVE 9.2.20); the docs record the run and drop "not yet run".

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
This commit is contained in:
kacerr
2026-09-24 10:28:28 +00:00
co-authored by Claude Opus 5.5
parent 7f2073f94b
commit 3d72507894
28 changed files with 882 additions and 201 deletions
Download Patch File Download Diff File Expand all files Collapse all files
docs/proxmox-provider.md
+47 -26
View File
@@ -164,8 +164,14 @@ pve 'cluster/resources?type=vm' # the template is listed with "temp
**Rotating the token needs no restart.** The token file is read on
every request, so the next request after the kubelet refreshes the
mount uses the new token. **Rotating the CA does**: it is loaded on
first use and cached for the life of the process. A missing CA file is
mount uses the new token. Keep the old token valid until then: a 401
is a permanent error, so a Proxy whose request lands in the gap
latches until its spec changes. Add the new token
(`pveum user token add`), update the Secret, wait a couple of minutes
for the mount to refresh, and only then remove the old token. A token
created with `--expire` needs the same rotation before it expires.
**Rotating the CA does** need a restart: it is loaded on first use and
cached for the life of the process. A missing CA file is
retried until it lands; one without a certificate is a permanent error.
2. Add a provider entry to the providers config
@@ -302,7 +308,9 @@ through an ownership check (`inspect` in `create.go`): a VM with a lock
naming the VMID and both UIDs. The operator never takes over a VM it did
not create;
- our name, our mark, hibernated (`lock: suspended`): **permanent error**,
`resume or destroy it in Proxmox` (see *Known gaps*).
`resume or destroy it in Proxmox` (see *Known gaps*). This only happens
when Create has not recorded the VM yet; a recorded VM that gets
hibernated is reported Stopped and replaced.
Two VMs with the proxy's name are a permanent error naming every VMID;
remove all but one.
@@ -392,15 +400,18 @@ pvesm free proxy-seeds:iso/proxy-abc…-seed.iso
```
**Delete** runs under a per-name lock shared with Create: stop (a paused
VM too), destroy with `purge=1&destroy-unreferenced-disks=1`, then delete
the seed on the VM's node. Each is a task that is waited on. A VM that is
already gone, not ours, or unmarked is left alone, and the seed is then
looked for on every configured node, where a node that fails is logged
and skipped. On the VM's own node a failed seed delete fails the call:
**without `Datastore.Allocate` on `isoStorage` the 403 makes Delete
permanent**, and the Proxy's deletion stays stuck with that message until
the privilege is granted. When Delete returns nil, nothing of the Proxy
is left.
VM too), delete the seed on the VM's node, then destroy with
`purge=1&destroy-unreferenced-disks=1`. Each is a task that is waited on.
The seed goes before the VM because once the VM is gone nothing calls
Delete again; a failed seed delete leaves the VM stopped, and the next
Delete finishes both. A VM that is already gone, not ours, or unmarked is
left alone, and the seed is then looked for on every configured node,
where a node that fails is logged and skipped. A refusal every node would
give fails the call instead: **without `Datastore.Allocate` on
`isoStorage` the 403 makes Delete permanent** on either path, and the
Proxy's deletion stays stuck with that message (a replaced Proxy stays
failed, its VM stopped) until the privilege is granted. When Delete
returns nil, nothing of the Proxy is left.
## Logging
@@ -554,9 +565,9 @@ transient.
| Class | What lands there |
|---|---|
| `ErrPermanent` | placement values not in the catalogue; a template VMID that is missing or not a template, has a cloud-init drive, or lacks `net0` or a sized boot disk when the config needs them; a malformed providerID; any 4xx (401 `Authentication failed!` or `No ticket`: the token; 403 `Permission check failed (<path>, <privilege>)`: a missing privilege, including `Datastore.Allocate` for the seed delete; 400 `Parameter verification failed.` with the field errors); a TLS verification failure; a token file that exists but is not `<user>@<realm>!<tokenid>=<secret>`; a CA file with no certificate; a VM with the proxy's name that carries no mark or another UID; two VMs with the proxy's name; our VM hibernated (`lock: suspended`), in both Create and Delete |
| `ErrPermanent` | placement values not in the catalogue; a template VMID that is missing or not a template, has a cloud-init drive, or lacks `net0` or a sized boot disk when the config needs them; a malformed providerID; any 4xx (401 `Authentication failed!` or `No ticket`: the token; 403 `Permission check failed (<path>, <privilege>)`: a missing privilege, including `Datastore.Allocate` for the seed delete; 400 `Parameter verification failed.` with the field errors); a TLS verification failure; a token file that exists but is not `<user>@<realm>!<tokenid>=<secret>`; a CA file with no certificate; a VM with the proxy's name that carries no mark or another UID; two VMs with the proxy's name; our VM hibernated (`lock: suspended`) found by Create's adoption |
| `ErrNotFound` | from `Get`: the VMID absent from the cluster's resources, the 500 `Configuration file 'nodes/<node>/qemu-server/<vmid>.conf' does not exist`, the VM at the VMID now named differently or unmarked. Inside `Create` the `does not exist` answer is `ErrTransient`, because the contract forbids NotFound from Create; `Delete` treats all of these as done |
| `ErrTransient` | everything else: a missing token or CA file (the Secret has not landed yet); network errors and cancellation; 500 `VM is locked (<lock>)` and any lock seen in the resources or config; a VM whose node does not answer (`status: unknown`); the agent's answers that `Get` cannot place, such as `VM <vmid> is not running`; any other 5xx, including pveproxy's 595 and 596; a task that fails or runs out of its budget; no free VMID after 32 probes; the wait for running expiring, or a VM that stops right after its start |
| `ErrTransient` | everything else: a missing token or CA file (the Secret has not landed yet); network errors and cancellation; 500 `VM is locked (<lock>)` and any lock seen in the resources or config; a VM whose node does not answer, or one a few seconds old the listing has not reported yet (`status: unknown`); the agent's answers that `Get` cannot place, such as `VM <vmid> is not running`; any other 5xx, including pveproxy's 595 and 596; a task that fails or runs out of its budget; no free VMID after 32 probes; the wait for running expiring, or a VM that stops right after its start |
`Get` turns two agent answers into states rather than errors:
`QEMU guest agent is not running` is Provisioning (booting, or the agent
@@ -613,13 +624,26 @@ They never contain the token, the CA content or cloud-init.
Beyond the ten limits accepted in the design:
- **A hibernated VM latches its Proxy.** A VM of ours with
`lock: suspended` (hibernated to disk) makes both Create and Delete fail
permanently with `resume or destroy it in Proxmox`: the lock never
clears on its own, and the destroy refuses a locked VM. `Get` reports it
Stopped. Resume it by starting it (`qm start <vmid>` restores the
hibernated state) or destroy it on the node, then touch the Proxy's
spec, or let the pending Delete retry.
- **A hibernated VM is replaced, or latches a Create.** `Get` reports a
VM of ours with `lock: suspended` (hibernated to disk) as Stopped, so
the reconciler deletes it (the destroy removes its suspend state volume
too) and creates a new one. Only a Create that finds such a VM before it
was ever recorded, by name, fails permanently with
`resume or destroy it in Proxmox`: resume it by starting it
(`qm start <vmid>` restores the hibernated state) or destroy it on the
node, then touch the Proxy's spec.
- **A Create that failed permanently after its clone keeps its VM.**
The stopped VM is adopted by name on the next Create, even when the
Proxy's `placement` has changed since: the Proxy then runs the old
template on the old node. After changing `placement` on such a Proxy,
destroy the VM on the node (`qm destroy <vmid> --purge`) first, or
delete and recreate the Proxy.
- **A retry can clone a second VM** in one rare sequence: a Create is
cut off while cloning onto a later probe VMID (the first was taken by
someone else's VM), that other VM is removed, and the retry comes
while the clone is still running, so its VM is still nameless. The
retry clones onto the now free first VMID. The next Create then fails
permanently with `VMs … are all named …`; destroy either VM.
- **The seed upload is one request**, bounded by `requestTimeoutSeconds`.
A seed is tens of KiB, but a multi-megabyte `vendorData` would need a
larger timeout.
@@ -627,11 +651,6 @@ Beyond the ten limits accepted in the design:
`config/samples/proxy_proxmox.yaml` installs Squid with
`http_access allow all`; restrict it before exposing the proxy network
to anything but the cluster.
- **Not yet proven end to end.** The unit tests run over captured
responses from `pv1-1` and an in-memory fake; the integration tests
below compile but have not run against a host yet. Three answers were
never captured: the storage status for a missing storage, a destroy on
a hibernated VM, and a stop on a paused VM.
## Testing
@@ -644,3 +663,5 @@ which also lists the variables it needs; run it with
`make test-integration-proxmox`. `TestProxmoxCheck_realHost` runs the
health check against the same host and creates nothing:
`go test -tags=integration -count=1 -v -run TestProxmoxCheck ./internal/provider/proxmox/`.
Both passed on `pv1-1` (PVE 9.2.20) on 2026-09-24: Create took 6.2 s,
and the guest reported an address 50 s after Create returned.