From 708989c40fee43fff844dd886a29e7b0649ee8ef Mon Sep 17 00:00:00 2001 From: kacerr Date: Fri, 18 Sep 2026 15:02:53 +0000 Subject: [PATCH 01/13] docs: plan the proxmox provider Eleven self-contained phases (host recon and fixtures, spec, config and skeleton, nocloud/keyedlock extraction from kvm2, HTTP client and fake, Get/List, Create, Delete/Checker, test sweep and integration test, deployment and docs, review and MR), with the Proxmox host setup checklist and the backend facts Phase 0 must verify on 192.168.2.8. Co-Authored-By: Claude Fable 5.1 --- .../plans/2026-09-18-1502-proxmox-provider.md | 844 ++++++++++++++++++ 1 file changed, 844 insertions(+) create mode 100644 docs/plans/2026-09-18-1502-proxmox-provider.md diff --git a/docs/plans/2026-09-18-1502-proxmox-provider.md b/docs/plans/2026-09-18-1502-proxmox-provider.md new file mode 100644 index 0000000..9dfed1c --- /dev/null +++ b/docs/plans/2026-09-18-1502-proxmox-provider.md @@ -0,0 +1,844 @@ +# Plan: proxmox, a Proxmox VE provider built from the provider contract + +**Created:** 2026-09-18 15:02 + +## Context + +The provider contract (`docs/provider-contract.md`), the spec template +(`docs/provider-spec-template.md`) and the kvm2 build proved the workflow: +spec first, then implementation against the contract, one commit per step. +This plan adds a third VM-style backend, `proxmox`, on the Proxmox VE 8.x +REST API at `https://:8006/api2/json`. The user has a real host at +192.168.2.8 and prepares the API token and a cloud-init-capable template +before Phase 0 (checklist below). + +The implementation runs in sessions whose context gets compacted, so every +phase is self-contained: it names what to read first, what it creates or +edits, the test names it must add, a `Verify:` clause, a "Done when" line +and a commit subject. A fresh session reading only this plan, the spec +(`docs/proxmox-provider-spec.md`, from Phase 1 on) and the execution log's +checklist (`docs/plans-executions/2026-09-18-1502-proxmox-provider.md`) must be able to +pick up where the last one stopped. + +**Resumption protocol, run at the start of every phase.** Read this plan's +Context and the phase entry; read the spec; read the execution log's +checklist; run `git branch --show-current` (must be `feat/proxmox-provider` +from Phase 0 on), `git status --short` and `git log --oneline -8`. The first +unchecked box is the phase to do. If its files already exist partially, its +"Done when" line says whether it counts as done; finish it, never redo it. A +phase is finished only when its "Done when" holds and its execution-log entry +(a few sentences, exact commands, "worth noting", the box ticked) is +committed together with the code. Follow `CLAUDE.md` for the log format. + +## Before Phase 0: what the user prepares on 192.168.2.8 + +Run as root on the Proxmox node. Drop any privilege `pveum` rejects on the +installed version (`VM.GuestAgent.Audit` exists from 8.2; older releases use +`VM.Monitor` for the agent read) and note it for F17. + +```bash +pveum user add proxy-operator@pve --comment "egress-proxies-operator" +pveum role add ProxyOperator -privs "VM.Allocate VM.Clone VM.Audit VM.Config.CDROM VM.Config.CPU VM.Config.Memory VM.Config.Disk VM.Config.Network VM.Config.Options VM.PowerMgmt VM.Monitor VM.GuestAgent.Audit Datastore.Audit Datastore.AllocateSpace Datastore.AllocateTemplate Sys.Audit SDN.Use" +pveum aclmod / -user proxy-operator@pve -role ProxyOperator +pveum user token add proxy-operator@pve operator --privsep 0 +# The secret is printed once. Save it on the workstation as one line: +# proxy-operator@pve!operator= +# in ~/.config/proxmox-token (mode 0600). The same file becomes the +# Kubernetes Secret later. +cp /etc/pve/pve-root-ca.pem /tmp/ # copy to the workstation as ~/.config/proxmox-ca.pem, or use insecureSkipTLSVerify +``` + +A cloud-init template without a Proxmox cloud-init drive (the provider puts +its own NoCloud ISO on `ide2`). Debian 13's BIOS boot path did not boot +under kvm2, so the template boots UEFI: + +```bash +wget https://cloud.debian.org/images/cloud/trixie/latest/debian-13-generic-amd64.qcow2 +qm create 9000 --name debian-13-template --memory 2048 --cores 2 --ostype l26 \ + --machine q35 --bios ovmf --efidisk0 local-lvm:1,efitype=4m,pre-enrolled-keys=0 \ + --scsihw virtio-scsi-pci --net0 virtio,bridge=vmbr0 \ + --serial0 socket --vga serial0 --agent 1 +qm set 9000 --scsi0 local-lvm:0,import-from=/root/debian-13-generic-amd64.qcow2 +qm set 9000 --boot order=scsi0 +qm template 9000 +``` + +Facts the user hands over at Phase 0: the token file path, the CA file path +(or "insecure"), the node name, the template VMID, the ISO-content storage +id (usually `local`), the VM disk storage (`local-lvm`), the bridge, and +whether the operator's cluster can route to that bridge's LAN. Everything +else Phase 0 reads from the API. + +## Design decisions made with the user (not re-opened) + +- **Backend and client.** Proxmox VE 8.x REST API, JSON, HTTP/1.1 only. + Thin client on `net/http` (`client-kind: thin-client`), no SDK, no new + dependency. Auth header + `Authorization: PVEAPIToken=@!=`, read from a + mounted token file on every request (rotation free). TLS: `caFile` or + `insecureSkipTLSVerify` (warned at startup). Child spans from a parent-gated + otelhttp transport: `tracing.RestConfigWrapper()(base)` returns a plain + `http.RoundTripper` and works as is (`child-spans: transport-wrapper`). +- **Image source.** Clone a prepared template. Config catalogue + `templates: {: {vmid: 9000}}`; `Placement.Image` is a key. + `Placement.Zone` is a node name from `nodes: [..]` (optional with one node). + `Placement.MachineType` is a key of `machineTypes: {: {cores, memoryMiB, + diskGiB}}`; `diskGiB` (optional) grows the boot disk via `PUT .../resize`. + `Placement.Region` is ignored (documented). +- **Identity.** `primary-key: allocated-id`. VMID derived deterministically + from the Proxy UID (FNV-1a 64 into `vmidRange: {min, max}`, default + 100000 to 199999) with forward probing on collision; a clone whose `newid` + is taken fails with a 500, which is the allocate-then-verify step. Create + first looks the name up in `GET /cluster/resources?type=vm`; an existing VM + with that name must carry our mark and a matching or empty UID, else + Permanent. providerID is location-free: `vms//` (VMs migrate; + the node is resolved from `/cluster/resources` on every call). Get and + Delete verify the VM at that VMID has that name and our mark; a different + name means our VM is gone and the VMID was reused: Get returns NotFound, + Delete returns nil without touching it. +- **Cloud-init.** NoCloud seed ISO built in-process by reusing kvm2's seed + code, extracted into a shared package `internal/provider/nocloud` (its own + phase). Uploaded to `isoStorage` (default `local`) on the target node via + `POST /nodes/{node}/storage/{storage}/upload` (multipart, `content=iso`, + `filename=-seed.iso`, streamed through `io.Pipe`, returns a UPID), + attached as `ide2: :iso/-seed.iso,media=cdrom`. The template + must have no Proxmox cloudinit drive on ide2 (documented prerequisite). + Vendor-data installs qemu-guest-agent like kvm2 (`installGuestAgent`, + `vendorData`). +- **GC mark.** `gc-mark: written-metadata` in the VM `description`, which the + clone call accepts (`tags` is not a clone parameter, so tags would be a + mark window). Content, one per line: `proxy-operator-managed=true`, + `proxy-operator-uid=`, `proxy-operator-created-at=`. + `created-at: written-at-create`. The config step after the clone also sets + tag `proxy-operator` for UI filtering only. ListByTag: one + `GET /cluster/resources?type=vm`; candidates are entries with tag + `proxy-operator` or name prefix `proxy-`; a `GET /nodes/{node}/qemu/{vmid}/config` + per candidate for the description; unmarked skipped; one unreadable VM + skipped with a V(1) line unless ctx ended; the resources call failing fails + the whole list (`list-partial: fail-whole`). +- **Create** (`create-return: blocking`, `provisioning-state: create-blocks`, + `async-ops: task-handle` and `backend-lock`), every step idempotent, in + this order: validate placement (Permanent before any call); resources + lookup by name (adopt or allocate); `POST /nodes/{tmplnode}/qemu/{tmplvmid}/clone` + `{newid, name, description, full=1, target?, storage?}`; wait the task + (`exitstatus != OK` is Transient with the message; a VM with `lock=clone` + on retry is waited on); upload the seed (always rebuilt, delete then + upload; before the config step because `ide2` may refuse a volume that + does not exist yet); config (`cores`, `memory`, `net0` bridge override if + configured, `agent=1`, `tags`, `ide2` seed); optional resize; + `POST .../status/start` and wait; poll `status/current` until `running` + (capped); return `vms//`. Retry: name found running returns + the ID; found stopped with our mark continues from the seed step; found + locked waits. +- **Get.** Resources lookup by vmid: absent is NotFound; name mismatch or no + mark is NotFound (logged); `lock` set is Transient; `running` reads + `GET /nodes/{node}/qemu/{vmid}/agent/network-get-interfaces`, first + routable IPv4 excluding loopback and link-local gives Running, else + Provisioning (the agent not running is a 500 "QEMU guest agent is not + running", which is Provisioning, not an error); `stopped` is Stopped; + `unknown` is Transient; `qmpstatus=paused` is Stopped; anything else is + Stopped. UID and CreatedAt come from the description on every call, so a + Get is three calls: resources, config, agent. +- **Delete** (`delete-return: waits`, `secondary-resources: explicit`): + resolve by vmid via resources; absent or foreign is nil (logged); running + gets `POST status/stop` and a task wait; then + `DELETE /nodes/{node}/qemu/{vmid}?purge=1&destroy-unreferenced-disks=1` and + a task wait; then delete the seed ISO + `DELETE /nodes/{node}/storage/{storage}/content/{storage}:iso/-seed.iso` + tolerating absence; a locked VM is Transient. Called again mid-delete it + repeats the same tolerant steps. GC blind spot: seed ISOs without a VM + (documented; the ISO name carries the instance name). +- **Errors**, one `classify(op, err)` over `*apiError{Status, Message, Errors}`: + 401, 403, 400 and 404 are Permanent (404 means the path never exists; + NotFound comes only from the list-and-miss on resources and from the 500 + "does not exist" message in Get and Delete, matched against captured + fixtures); 500 "does not exist" is NotFound (Transient inside Create); 500 + "VM is locked" is Transient; 500 "already exists" is handled by the + allocator, otherwise Transient; 500 "not enough free space" or "no space + left" is QuotaExceeded once a real message is captured, else `quota: + absent`; other 5xx, 595, 596, network and ctx errors are Transient; + anything not an apiError is Transient. Error text: Proxmox puts most of it + in the HTTP status reason phrase (`HTTP/1.1 500 Configuration file '...' + does not exist`) and newer releases also in the body's `message`; the + client reads both, body first. +- **Checker.** One target per configured node, in config order, concurrently, + on the same http client: `GET /version` (Details `pveVersion`), + `GET /nodes/{node}/status` (Details `kernel`), + `GET /nodes/{node}/storage/{isoStorage}/status` (Details `isoStorage`, + `isoStorageAvailGiB`), template existence from `GET /cluster/resources?type=vm` + (the `template` flag, any node; Details `templates` as `/`). + Reasons: the shared ones plus `TemplateNotFound` and `StorageNotFound`. + The `status.type` comment in `api/v1alpha1/providerhealth_types.go` gains + `proxmox`; `make manifests`. +- **Timeouts** (`timeouts: per-call`): request timeout 30s + (`requestTimeoutSeconds`); task-wait caps clone 10m, upload 2m, start and + stop 2m, destroy 5m, running wait 60s; all in a `budgets` struct + overridable in tests. A cancelled ctx cancels that request only; the + connection pool is unaffected. Per-name keyed lock for Create and Delete + (`internal/provider/keyedlock`, extracted from kvm2); Get and ListByTag + lock-free. +- **Logging** (`logging: tiered`): `logf.FromContext(ctx).WithName("proxmox").WithValues("provider", name)`, + `vmid` and `name` on instance lines; V(1) outcomes and one + `proxmox api call` line per seam call from the `observed` decorator; V(2) + request params and response summaries; never the token, never cloud-init, + never the multipart body, never `description` at V(1); no V(5). The + decorator logs, the transport traces: no `proxmox.` spans, so + nothing is double-counted. +- **Config writes** use `PUT .../config` (synchronous; the VM is stopped at + that point). `resize` is skipped when the boot disk's current `size=` is + at or above the target. Boot disk: first disk key in `boot: order=...` + that is not `net*` and not the ide2 cdrom, else `bootdisk`, else the first + of `scsi0`, `virtio0`, `sata0`, `ide0`; none is Permanent. +- **A UPID names its node** (`UPID::...`); task polling uses the node + parsed from the UPID. +- **Two VMs with our name** (Proxmox does not enforce unique names) make + Create and Get return Permanent naming both VMIDs; an admin resolves it. +- **Seam.** `api` interface with one method per endpoint (table below), + `httpAPI` production adapter, `fakeAPI` in `fake_test.go` keeping VMs, tasks + and ISOs in memory with lock simulation, task polls, per-method errors and + block channels, and a call log. +- **Integration test** `proxmox_integration_test.go` behind + `//go:build integration`, `PROXMOX_TEST_URL`, `PROXMOX_TEST_TOKEN_FILE`, + `PROXMOX_TEST_NODE`, `PROXMOX_TEST_TEMPLATE_VMID`, `PROXMOX_TEST_ISO_STORAGE`, + optional `_CA_FILE`, `_INSECURE`, `_BRIDGE`, `_STORAGE`, `_VMID_MIN`, + `_VMID_MAX`, `_TIMEOUT`, `_KEEP`, `_LOG_LEVEL`; `make test-integration-proxmox`. +- **Deployment.** Secret `proxmox-token` mounted optional at + `/etc/proxy-operator/proxmox/token`, `make proxmox-token-secret`, commented + entries in `config/manager/providers_config.yaml` and + `config/samples/providers-config.yaml`, `config/samples/proxy_proxmox.yaml`, + `docs/proxmox-provider.md`, `providers-existing.md` column and section, + `architecture.md` rows, README row, `testing.md` section, CHANGELOG after + confirmation. Branch `feat/proxmox-provider`, MR via the Gitea API (no `tea`). + +## Backend facts Phase 0 must verify on 192.168.2.8 (numbered for citation) + +- F1. Where error text lives: status reason phrase vs. JSON `message`; the + `errors` map shape on a 400. +- F2. Exact 500 message for a missing VM (`GET /nodes/{node}/qemu/999999/config`). +- F3. Exact 500 message for a locked VM (`POST status/start` on a VM with + `lock` set), and whether `PUT config` with `lock=backup` sets the lock and + `delete=lock` clears it through the API, or `qm unlock` is needed. +- F4. Exact 500 message when `newid` is in use on clone. +- F5. Exact 500 message for the agent not running, and for the agent call + on a stopped VM. +- F6. Exact message for no space, if it can be produced cheaply; otherwise + `quota: absent`. +- F7. Whether `PUT .../config` is refused while `lock=clone`, and with which + message. +- F8. `/cluster/resources?type=vm` fields present: `vmid`, `name`, `node`, + `status` values (`running`, `stopped`, `unknown`), `type` (`qemu`, `lxc`), + `lock`, `tags` (separator), `template`; whether `lock` appears there during + a clone. +- F9. Tag rules: allowed characters, case, length, `tags` separator on + `PUT config` and in `/cluster/resources`. +- F10. `description` on `POST .../clone`: accepted, survives verbatim + (newlines), max length; whether `name` accepts the 22-character + `proxy-` form. +- F11. `meta` in the cloned config: is `ctime` the clone time (evidence only; + `written-at-create` stays the profile). +- F12. Upload endpoint: field names (`content`, `filename`, file part name + `filename`), the filename rule (`.iso` suffix, no slashes), the response + (UPID), the privilege it needs, and whether re-uploading an existing name + is refused (then delete-then-upload) or overwrites. +- F13. `DELETE /nodes/{node}/storage/{storage}/content/{volid}`: response + shape (UPID or null) and the message for a missing volume. +- F14. `status/current` on a stopped VM: is `qmpstatus` present; on a paused + VM: `status=running`, `qmpstatus=paused`. +- F15. `POST .../clone`: whether a failed clone destroys the partial VM by + itself, and what a VM interrupted mid-clone looks like (config present, + `lock=clone`). +- F16. Task status shape: `status` (`running`, `stopped`), `exitstatus` + (`OK` or the error string), and how long a full clone of the template + takes. +- F17. Privileges: which of the prepared list the token really needs, in + particular `SDN.Use` on `/sdn/zones/localnetwork/` for the `net0` + write and `VM.GuestAgent.Audit` or `VM.Monitor` for the agent read. +- F18. HTTP/2 is not offered (`curl --http2 -I` falls back or fails), so the + client disables it. +- F19. The template's disk key (`scsi0` or `virtio0`), its `boot` line, that + ide2 is free and no `cloudinit` drive exists. +- F20. `resize` on a stopped VM: request shape, the message when the target + is smaller or equal, and that it is synchronous. +- F21. Agent response shape: `result[].name`, `result[]["ip-addresses"][]` + with `ip-address`, `ip-address-type`. + +## Proxmox endpoint map for the seam + +| `api` method | HTTP | +|---|---| +| `Version(ctx)` | `GET /version` | +| `NodeStatus(ctx, node)` | `GET /nodes/{node}/status` | +| `StorageStatus(ctx, node, storage)` | `GET /nodes/{node}/storage/{storage}/status` | +| `ClusterResources(ctx)` | `GET /cluster/resources?type=vm` | +| `VMConfig(ctx, node, vmid)` | `GET /nodes/{node}/qemu/{vmid}/config` | +| `SetVMConfig(ctx, node, vmid, params)` | `PUT /nodes/{node}/qemu/{vmid}/config` | +| `Clone(ctx, node, vmid, cloneParams)` | `POST /nodes/{node}/qemu/{vmid}/clone` | +| `Resize(ctx, node, vmid, disk, size)` | `PUT /nodes/{node}/qemu/{vmid}/resize` | +| `Start(ctx, node, vmid)` | `POST /nodes/{node}/qemu/{vmid}/status/start` | +| `Stop(ctx, node, vmid)` | `POST /nodes/{node}/qemu/{vmid}/status/stop` | +| `VMStatus(ctx, node, vmid)` | `GET /nodes/{node}/qemu/{vmid}/status/current` | +| `Destroy(ctx, node, vmid)` | `DELETE /nodes/{node}/qemu/{vmid}?purge=1&destroy-unreferenced-disks=1` | +| `TaskStatus(ctx, node, upid)` | `GET /nodes/{node}/tasks/{upid}/status` | +| `AgentNetworkInterfaces(ctx, node, vmid)` | `GET /nodes/{node}/qemu/{vmid}/agent/network-get-interfaces` | +| `UploadISO(ctx, node, storage, filename, r, size)` | `POST /nodes/{node}/storage/{storage}/upload` (multipart) | +| `DeleteVolume(ctx, node, storage, volid)` | `DELETE /nodes/{node}/storage/{storage}/content/{volid}` | + +## Package layout (`internal/provider/proxmox/`) + +`proxmox.go` (`Provider`, `New`, `newProvider`, `withDefaults`, `logger`, +`node`, `wrapErr`, `lockName`, `providerOption`, `budgets`), `api.go` (the +`api` interface and DTOs `vmResource`, `vmConfig`, `vmStatus`, `taskStatus`, +`agentInterface`, `storageStatus`, `nodeStatus`, `versionInfo`, +`cloneParams`), `client.go` (`httpAPI`, `newHTTPAPI`, `do`, `readToken`, +`decodeConfig`), `errors.go` (`apiError`, `classify`, `permanentf`, +`transientf`, `isNotExist`, `isLocked`, `isAlreadyExists`, +`isAgentNotRunning`, `isNoSpace`), `id.go` (`formatProviderID`, +`parseProviderID`, `vmidFor`), `mark.go` (`instanceMark`, +`formatDescription`, `parseDescription`), `state.go` (`mapState`, +`firstIPv4`), `observe.go` (`observed`, `observe`), `task.go` (`parseUPID`, +`waitTask`, `waitUnlocked`, `waitRunning`), `seed.go` (`seedISOName`, +`seedVolID`, `uploadSeed`), `create.go`, `get.go`, `delete.go`, `list.go`, +`check.go`, plus `fake_test.go`, `provider_test.go`, +`proxmox_integration_test.go`, `testdata/*.json`. Shared: +`internal/provider/config_proxmox.go`, `internal/provider/nocloud/`, +`internal/provider/keyedlock/`. + +Reused as-is: `provider.NameFromUID`, `provider.Wrap`, the `Err*` sentinels, +`provider.Label*` constants, `provider.Reason*` constants, +`tracing.RestConfigWrapper`, `tracing.WithTracerProvider`, kvm2's test +idioms (`fixture`, `assertProviderError`, `logCapture`). + +## Steps (one commit per phase, on `feat/proxmox-provider` off `main`) + +- **Phase 0 (orchestrator with the user): plan file, branch, host recon, + fixtures.** Read first: contract §5, §4.9, §7; + `docs/plans-executions/2026-09-16-1333-kvm2-provider.md` Step 0 entry. + Do, in this order: `git switch main && git pull --ff-only`; copy this plan + to `docs/plans/2026-09-18-1502-proxmox-provider.md` with the real timestamp + (`date "+%Y-%m-%d-%H%M"`), commit it alone on `main` + (`docs: plan the proxmox provider`); create `feat/proxmox-provider`. Ask + the user for the hand-over facts listed above. Then, with the user's + go-ahead (each `curl` is a permission prompt; read-only ones first, one at + a time), capture F1 to F21 with + `curl -s --cacert "$CA" -D /tmp/h.txt -o /tmp/b.json -w '%{http_code}\n' -H "Authorization: PVEAPIToken=$(cat "$TOKEN_FILE")" https://192.168.2.8:8006/api2/json/`, + storing each response as `internal/provider/proxmox/testdata/.json` + in the shape `{"status": , "reason": "", "body": ""}`: `version.json`, `node_status.json`, + `storage_status.json`, `cluster_resources.json`, + `vm_config_template.json`, `vm_config_clone.json`, + `status_current_running.json`, `status_current_stopped.json`, + `agent_network_get_interfaces.json`, `task_status_running.json`, + `task_status_stopped_ok.json`, `task_status_stopped_error.json`, + `upload_response.json`, `content_delete_response.json`, + `err_vm_missing.json`, `err_locked.json`, `err_newid_in_use.json`, + `err_agent_not_running.json`, `err_agent_vm_stopped.json`, + `err_unauthorized.json` (a wrong secret), `err_forbidden.json` (a path + the token lacks), `err_bad_param.json` (`newid=abc`), `err_no_space.json` + (only if F6 is cheap), `err_volume_missing.json` (F13). The write + experiments: one clone of the template into the top of the VMID range + (`newid=199999`, a multi-line `description`, a `name` of the `proxy-` + form), a config read of the clone (F10, F11, F19), a `PUT config` while + it is locked if timing allows (F7), an upload of a 1 KiB + `recon-seed.iso` (F12) and its delete (F13), a resize attempt (F20), a + start, an agent read (F5, F21), a stop, then destroy with + `purge=1&destroy-unreferenced-disks=1`; a final resources read shows + nothing remains. Redact nothing but the token. Write the answers to F1 to + F21 and the privilege outcome into the execution log's Phase 0 entry, one + line per fact; Phase 1 cites them as Evidence. Create the execution log + `docs/plans-executions/2026-09-18-1502-proxmox-provider.md` with the checklist (Phase + 0 to Phase 10) and this entry. + Verify: `ls internal/provider/proxmox/testdata | wc -l` is at least 22; + `grep -rl 'PVEAPIToken' internal/provider/proxmox/testdata` is empty; + every fixture parses as JSON; the host shows no VM in the range and no + `recon-seed.iso`. + Done when: the plan commit is on `main`, the branch exists, the fixtures + and the execution log with F1 to F21 are committed. Commit: `proxmox: + captured API fixtures and host facts from 192.168.2.8`. + +- **Phase 1 (orchestrator): the spec.** Read first: contract §2, §4, §5, + §6, §11, §12; `docs/provider-spec-template.md` end to end; + `docs/kvm2-provider-spec.md` for the level of detail; this plan's + decisions; the Phase 0 log entry. Create `docs/proxmox-provider-spec.md` + answering all 24 entries: 1 `vms//`, strict parsing, the + node-name and storage-id regexps; 2 `allocated-id`, `vmidFor(uid, rng, + attempt)`, the two-VMs-with-our-name rule, the adoption guard; 3 the + placement table (Region ignored); 4 the template catalogue and the "add a + key, never edit" warning; 5 the seed ISO via `nocloud` and the ide2 + prerequisite; 6 `machineTypes` with `diskGiB`; 7 `provisioning-window`, + `discovered` via the guest agent, the IPv4 filter, `Port` unused; 8 + `written-metadata` in `description` plus the tag, `written-at-create`, F9 + to F11 as Evidence; 9 the state table, `create-blocks`; 10 the ordered + Create steps with what a retry finds at each, `task-handle` plus + `backend-lock`, the budgets; 11 the Delete order, the ISO blind spot, + node-local ISO storage after a migration; 12 `fail-whole`, the candidate + rule, one config read per candidate; 13 the classify table with the F1 to + F6 messages, `not-found: error-code` plus list-and-miss on resources + (allowed because offline nodes' guests appear as `unknown`, not absent); + 14 `mounted-files` (Secret `proxmox-token`, key `token`, path + `/etc/proxy-operator/proxmox/token`), `ca-file` or + `insecureSkipTLSVerify`, the privilege list from F17, rotation free; 15 + the config table (Phase 2's struct); 16 the Checker; 17 `thin-client`, + `transport-wrapper`, the seam's method list; 18 the budgets table; 19 the + keyed lock per name; 20 the log keys; 21 the §11 list with the test names + from Phases 2 to 8 (copy from this plan); 22 the deployment list; 23 the + docs list; 24 the accepted limits (blocking Create up to the clone budget, + ISO blind spot, VMID range shared with humans, node-local ISO storage, no + adoption of foreign VMs, two-installs GC caveat, `unknown` status while a + node is down). + Verify: `grep -c '^\*\*Answer:\*\*$' docs/proxmox-provider-spec.md` is 0; + every axis code used appears in contract §4.20; Evidence lines cite F + numbers; `grep -c '—' docs/proxmox-provider-spec.md` is 0. + Done when: the spec is committed with all 24 entries and the Phase 1 box + ticked. Commit: `docs: proxmox provider spec`. + +- **Phase 2 (implementer): config block, registration, package skeleton.** + Read first: spec entries 1, 2, 3, 6, 8, 9, 13, 15; contract §10, §5, §6; + `internal/provider/config.go`, `config_kvm.go`, `config_test.go`, + `config_kvm_test.go`; `internal/provider/kvm2/{kvm2.go,id.go,metadata.go,state.go,errors.go}` + and their tests as the pattern; `cmd/main.go`; `testdata/err_*.json`. + Create `internal/provider/config_proxmox.go`: `ProxmoxConfig{URL, + TokenFile, CAFile string; InsecureSkipTLSVerify bool; Nodes []string; + ISOStorage, Storage, Bridge string; VMIDRange *ProxmoxVMIDRange; Templates + map[string]ProxmoxTemplate; MachineTypes map[string]ProxmoxMachineType; + RequestTimeoutSeconds int; VendorData string; InstallGuestAgent *bool}`, + `ProxmoxVMIDRange{Min, Max int}`, `ProxmoxTemplate{VMID int}`, + `ProxmoxMachineType{Cores int32; MemoryMiB int64; DiskGiB int64}`, + `validateProxmox(prefix string, c *ProxmoxConfig) error` (URL http(s) + with a host; `tokenFile` non-empty; at least one node, unique, matching + `proxmoxNodeName = ^[a-z0-9][a-z0-9-]*$`; `isoStorage` and `storage` + matching `proxmoxStorageID = ^[A-Za-z][A-Za-z0-9_.-]*$`; `vmidRange` min + at least 100, max at most 999999999, min below max; at least one template + with `vmid` at least 100; at least one machine type with cores and + memoryMiB above zero and diskGiB not negative; `requestTimeoutSeconds` + not negative). Edit `config.go`: `Proxmox *ProxmoxConfig` with tag + `proxmox,omitempty`, `setBlocks`, the allow-list, the dispatch. Edit + `cmd/main.go`: `"proxmox": proxmox.New`. Create in + `internal/provider/proxmox/`: `proxmox.go` (`Provider{name, cfg, api, + now, budgets, nameLocks, tracing}`; until Phase 3 a local `keyedLocks` + copy in `locks.go` that Phase 3 deletes; `New` never dials: builds the + client, `os.Stat`s the token file and warns, warns on + `insecureSkipTLSVerify`; `withDefaults`: `isoStorage` `local`, `vmidRange` + 100000 to 199999, `requestTimeoutSeconds` 30, `installGuestAgent` true; + `node(zone)` mirrors kvm2's `host`; the four methods stubbed with + `transientf("not implemented")`), `api.go` (the interface and DTOs; + `vmConfig` is `map[string]string` with scalars rendered as strings by + `decodeConfig`), `id.go`, `mark.go` (keys from `provider.LabelManaged`, + `provider.LabelManagedYes`, `provider.LabelUID` and + `proxy-operator-created-at`; `parseDescription` tolerates extra lines + and CRLF), `state.go`, `errors.go` (`apiError{Status int; Message + string; Errors map[string]string}` whose `Error()` never includes headers + or URLs; `classify` per spec entry 13; matcher helpers keyed on the exact + substrings captured in F2 to F6). Tests: `config_proxmox_test.go` + `TestValidateProxmox_table` (missing url, non-http scheme, missing + tokenFile, no nodes, duplicate node, node with a slash, bad isoStorage + id, vmidRange min below 100, min above max, no templates, template vmid + below 100, no machineTypes, cores zero, negative diskGiB, negative + timeout, valid full, valid minimal) with a `validProxmoxYAML` const; + `config_test.go` rows `proxmox type with kvm block`, `kvm type with + proxmox block`, `proxmox minimal entry loads`. `id_test.go`: + `TestFormatProviderID_roundTrip`, `TestParseProviderID_malformed`, + `TestVMIDFor_deterministicAndInRange`, `TestVMIDFor_probesForwardAndWraps`. + `mark_test.go`: `TestFormatDescription_keys`, `TestParseDescription_table`. + `state_test.go`: `TestMapState_table`, `TestFirstIPv4_filters`. + `errors_test.go`: `TestClassify_table` (every class, every matched + message loaded from `testdata/err_*.json` through `loadFixture(t, name) + *apiError`, `op == "create"` turning NotFound into Transient, + `context.Canceled`, a `*net.OpError`, `errors.New("x")`, + already-classified errors keep their class), `TestAPIError_matchers`, + `TestPermanentfAndTransientf`. `proxmox_test.go`: + `TestNew_doesNotDialAndWarnsOnMissingToken`, + `TestNew_insecureSkipTLSVerifyWarns`, `TestNew_nilBlockDoesNotPanic`, + `TestWithDefaults_table`, `TestNode_table`, + `TestWrapErr_carriesProviderNameOpAndID`. + Verify: `grep -c '"proxmox"' internal/provider/config.go` is 3; + `go build ./... && go vet ./... && go test -race -count=1 ./internal/provider/... ./cmd/... && go test -short ./...`; + the fixture loader fails loudly on a missing file. + Done when: `go test -race ./internal/provider/proxmox/` passes with the + stubs in place and the constructor is registered. Commit: `proxmox: + config block, IDs, marks, state mapping, classify, registration`. + +- **Phase 3 (implementer): extract `nocloud` and `keyedlock` from kvm2.** + Read first: `internal/provider/kvm2/{seed.go,seed_test.go,kvm2.go,kvm2_test.go,create.go}`; + contract §1.4, §2.1; spec entries 5 and 19; + `grep -n 'seed.go\|keyedLocks\|buildSeedISO' docs/*.md`. Create + `internal/provider/nocloud/seed.go` exporting `BuildSeedISO(name, + userData, vendorData string, w io.Writer) error` (plain errors, an + `ErrEmptyName` sentinel instead of kvm2's `permanentf`), + `BuildVendorData(configured string, installAgent bool) (string, error)`, + `GuestAgentScript`, `SeedLabel`; move the seed tests to + `nocloud/seed_test.go` under their existing names. Create + `internal/provider/keyedlock/keyedlock.go` exporting `Locks`, `New() + *Locks`, `(*Locks).Acquire(ctx, key) (release func(), err error)`; move + the `TestKeyedLocks_*` tests. Edit kvm2: delete `seed.go`, `seed_test.go`, + the `keyedLocks` type and its tests; `create.go` calls + `nocloud.BuildVendorData` and `nocloud.BuildSeedISO` and wraps their + errors with `permanentf`; `kvm2.go` uses `*keyedlock.Locks`. Edit + `internal/provider/proxmox/proxmox.go` to use `keyedlock.Locks` and delete + `locks.go`. Fix the doc mentions so they name the new packages. + Verify: `go test -race -count=1 ./internal/provider/...`; kvm2's test + count drops by exactly the moved tests and the two new packages carry + them; `grep -rn 'iso9660' internal/provider/kvm2/` is empty; + `go vet -tags=integration ./...` still compiles. + Done when: kvm2 and proxmox import both packages, all tests pass, the docs + name the new packages. Commit: `provider: extract nocloud seed builder + and keyedlock from kvm2`. + +- **Phase 4 (implementer): seam, HTTP client, decorator, fake.** Read first: + spec entries 14, 17, 18, 20; contract §4.15, §7, §9; + `internal/tracing/transport.go`; `internal/provider/kvm2/{observe.go,observe_test.go,fake_test.go}`; + `internal/provider/proxmox/{api.go,errors.go}`; `testdata/*.json`. Create + `client.go`: `httpAPI{base *url.URL; http *http.Client; tokenFile + string}`; `newHTTPAPI(cfg provider.ProxmoxConfig, opts ...tracing.Option) + (*httpAPI, error)` builds a transport with `DialContext` timeout 10s, + `TLSHandshakeTimeout` 10s, `ResponseHeaderTimeout` from the request + timeout, `IdleConnTimeout` 90s, `MaxIdleConnsPerHost` 4, + `ForceAttemptHTTP2: false`, an empty `TLSNextProto` map (F18), `RootCAs` + from `caFile` or `InsecureSkipVerify` from the flag, wrapped with + `tracing.RestConfigWrapper(opts...)(transport)`; `http.Client.Timeout` is + the request timeout; `do(ctx, method, path string, query, form + url.Values, out any) error` reads the token file on every call (missing + file `transientf`, a file without `!` and `=` `permanentf`), sets the + header, decodes `{"data": ...}` into `out`, turns any non-2xx into + `*apiError` per F1 with the error body read capped at 64 KiB; + `UploadISO` streams a multipart body through `io.Pipe` with a goroutine + writing `content=iso`, `filename` and the file part. Create `observe.go`: + `observed{inner api; provider string}` writing one V(1) `proxmox api + call` line per seam call with `call`, `node`, `vmid`, `storage`, `upid`, + `duration`, `httpStatus` and `error` on failure, V(2) adding request + params and a response summary; `observe(inner api, providerName string) + api`; `New` wraps `httpAPI` in it. Create `fake_test.go`: `fakeAPI{mu; vms + map[int]*fakeVM; tasks map[string]*fakeTask; isos + map[string]map[string][]byte; hooks; block; calls; log; taskPolls int; + cloneFailsWith, startFailsWith string; agentNotRunning, cloneLeavesLock + bool}`, `fakeVM{node, name, status, qmpstatus, lock string; template + bool; config map[string]string; ifaces []agentInterface}`, + `fakeTask{node string; polls int; exit string; onDone func()}`; `Clone` + creates the VM with `lock=clone` and a task that clears the lock or, per + `cloneFailsWith`, removes the VM (F15); `Start` and `Stop` flip status + through a task; `Destroy` removes the VM through a task; `UploadISO` + records bytes; `DeleteVolume` errors with the F13 message when absent; + `enter(ctx, method)` for blocks and hooks like kvm2; helpers `addVM`, + `addTemplate`, `failWith`, `blockOn`, `callCount`, `callLog`, + `fakeErr(status int, msg string) *apiError`. Tests: `client_test.go` over + `httptest.NewTLSServer`: `TestClient_sendsTokenHeaderFromFileOnEveryRequest` + (rotate the file between two calls), `TestClient_missingTokenFileIsTransient`, + `TestClient_unparsableTokenFileIsPermanent`, + `TestClient_apiErrorFromStatusReasonAndBody` (serve each + `testdata/err_*.json` verbatim), `TestClient_decodesDataEnvelope`, + `TestClient_http1Only`, `TestClient_requestTimeoutIsTransient`, + `TestClient_cancelledContextReturnsPromptly`, + `TestClient_childSpanUnderParent`, `TestClient_noSpanWithoutParent`, + `TestClient_tokenNeverInErrorsOrLogs` (sentinel secret echoed by the + server in a 401 body), `TestClient_caFileTrustsPrivateCA`, + `TestClient_insecureSkipsVerification`, `TestClient_uploadStreamsMultipart` + (the server parses the multipart; the client never `io.ReadAll`s the + reader), `TestClient_destroyQueryFlags`. `observe_test.go`: + `TestObserve_everySeamMethodLogsOneLine` (reflection over `api`), + `TestObserve_detailOnlyAtV2`, `TestObserve_uploadBodyNeverLogged`, + `TestObserve_errorFieldsOnFailure`, `TestObserve_v0Silent`. + Verify: `go test -race -count=2 ./internal/provider/proxmox/`; + `grep -n 'ReadAll' internal/provider/proxmox/client.go` shows only the + bounded error-body read; `grep -n 'context.Background()' internal/provider/proxmox/*.go` + is empty outside tests. + Done when: the client passes its tests against the fixtures and the fake + compiles against the full `api` interface. Commit: `proxmox: thin HTTP + client, seam, log decorator and in-memory fake`. + +- **Phase 5 (implementer): Get, ListByTag, task helpers.** Read first: spec + entries 7, 8, 9, 12, 13; contract §2.2, §2.4, §4.7, §4.12; + `internal/provider/kvm2/{get.go,get_test.go,list.go,list_test.go}`; the + resources, config, status, agent and task fixtures. Create `task.go`: + `parseUPID(upid) (node string, err error)` (Permanent when malformed), + `waitTask(ctx, log, api, upid, budget) error` polling `TaskStatus` every + `budgets.taskPoll` (2s; tests shrink it) until `status == "stopped"`, + `exitstatus != "OK"` giving `transientf` with the message, the budget + giving `transientf`, ctx giving Transient; `waitUnlocked(ctx, api, node, + vmid, budget)` polling `VMConfig` until no `lock`. Create `get.go`: + `resolve(ctx, api, vmid) (vmResource, bool, error)` (one resources call, + `Type == "qemu"` and the vmid), the name check, `VMConfig` for the mark, + `lock` Transient, `unknown` Transient, `running` reads the agent + (`isAgentNotRunning` gives no IP, other errors Transient), `mapState`. + Create `list.go` per spec entry 12; `Instance` carries ID, UID, CreatedAt + only. Replace the two stubs. Tests: `task_test.go` `TestParseUPID_table`, + `TestWaitTask_table` (ok, error exitstatus, budget exceeded, ctx + cancelled, poll count), `TestWaitUnlocked_table`. `get_test.go`: + `TestGet_missingVMIsNotFound`, `TestGet_nameMismatchIsNotFound`, + `TestGet_unmarkedIsNotFound`, `TestGet_runningWithoutIPIsProvisioning`, + `TestGet_runningWithIPIsRunning`, `TestGet_agentNotRunningIsProvisioning`, + `TestGet_agentOtherErrorIsTransient`, `TestGet_ipFilterSkipsLoopbackAndLinkLocal`, + `TestGet_stoppedIsStopped`, `TestGet_pausedIsStopped`, + `TestGet_unknownStatusIsTransient`, `TestGet_lockedIsTransientNeverNotFound`, + `TestGet_unknownStateIsStopped`, `TestGet_readsUIDAndCreatedAtFromBackend`, + `TestGet_unmanagedDescriptionHasEmptyUID`, `TestGet_malformedIDIsPermanent`, + `TestGet_resourcesFailureIsTransient`, `TestGet_twoVMsWithOurNameIsPermanent`. + `list_test.go`: `TestListByTag_excludesUnmarkedVMs`, + `TestListByTag_emptyUIDStillListed`, `TestListByTag_candidatesByTagOrNamePrefix`, + `TestListByTag_skipsLXCAndTemplates`, `TestListByTag_resourcesFailureFailsWhole`, + `TestListByTag_unreadableVMSkipped`, `TestListByTag_unreadableVMWithEndedContextFails`, + `TestListByTag_unknownStatusStillAttempted`, `TestListByTag_doesNotReadStatusOrAgent`, + `TestListByTag_idsAcceptedByParse`. + Verify: `go test -race -count=2 ./internal/provider/proxmox/`; + `grep -n 'not implemented' internal/provider/proxmox/*.go` lists only + Create and Delete. + Done when: Get and ListByTag are real and every listed test passes. + Commit: `proxmox: Get, ListByTag and task polling`. + +- **Phase 6 (implementer): Create.** Read first: spec entries 2, 3, 4, 5, 6, + 10, 19; contract §2.1, §4.2, §4.10, §4.11, §5; + `internal/provider/kvm2/{create.go,create_test.go}`; + `internal/provider/nocloud`; F4, F7, F10, F12, F15, F16, F19, F20 in the + Phase 0 log. Create `seed.go`: `seedISOName(name)` is `-seed.iso`, + `seedVolID(storage, name)` is `:iso/-seed.iso`, + `uploadSeed(ctx, log, api, node, storage, req, cfg)` builds vendor-data + and the ISO into a `bytes.Buffer` (tens of KiB), deletes an existing + volume tolerating absence, uploads with the buffer's length, waits the + task with `budgets.upload`, logs `cloudInitBytes` only. Create `create.go` + implementing the ordered steps: placement validation (`node`, + `MachineTypes`, `Templates`, all Permanent naming field and value), the + per-name lock, `ClusterResources` once for both the name lookup (adoption + guard, then `waitUnlocked` if locked, then continue from the seed step if + stopped, return if running) and the template's node (absent template is + Permanent); allocation loop `for attempt := 0; attempt < maxProbes (32)` + calling `Clone` with `vmidFor(req.UID, rng, attempt)`, `isAlreadyExists` + moving to the next attempt, exhaustion Transient; `waitTask` with + `budgets.clone`; `uploadSeed`; `SetVMConfig` with `cores`, `memory`, + `agent=1`, `tags=proxy-operator`, `ide2=,media=cdrom`, and `net0` + rewritten from the cloned config with `bridge=` only when + configured (MAC kept); `bootDisk(cfg)` and `Resize` when `diskGiB` is + above the current `size=`; `Start` and `waitTask` with `budgets.start`; + `waitRunning` on `VMStatus` with `budgets.running`; return the ID. + `isNotExist` inside Create is Transient (classify with `op == "create"`). + Tests `create_test.go`: `TestCreate_repeatReturnsSameIDAndCreatesNothing`, + `TestCreate_resumesAfterEachPartialStep` (after clone task, after seed + upload, after config, after resize, after start, after running), + `TestCreate_returnsOnlyWhenRunning`, `TestCreate_runningWaitCapIsTransient`, + `TestCreate_rejectsUnknownPlacement`, `TestCreate_zoneOptionalWithOneNode`, + `TestCreate_regionIgnored`, `TestCreate_adoptionWithForeignUIDIsPermanent`, + `TestCreate_adoptsOnlyMarkedVMs`, `TestCreate_duplicateNameIsPermanent`, + `TestCreate_vmidCollisionProbesForward`, `TestCreate_vmidRangeExhaustedIsTransient`, + `TestCreate_waitsForCloneLockOnRetry`, `TestCreate_cloneTaskFailureIsTransient`, + `TestCreate_cloneTaskTimeoutIsTransient`, `TestCreate_markWrittenInCloneCall`, + `TestCreate_tagSetInConfigStep`, `TestCreate_seedAlwaysRebuilt`, + `TestCreate_seedUploadedBeforeConfig`, `TestCreate_configSetsCoresMemoryAgentAndSeed`, + `TestCreate_bridgeOverrideKeepsMAC`, `TestCreate_noBridgeOverrideLeavesNet0`, + `TestCreate_resizeOnlyWhenLarger`, `TestBootDisk_table`, + `TestCreate_startTaskFailureIsTransient`, `TestCreate_templateMissingIsPermanent`, + `TestCreate_notExistInsideCreateIsTransientNotNotFound`, + `TestCreate_cloudInitSentinelNeverLoggedOrInErrors` (V(2) capture, every + error path), `TestCreate_concurrentSameNameSerialised`, + `TestCreate_cancelledMidWaitReturnsPromptly`. + Verify: `grep -n 'CloudInit' internal/provider/proxmox/create.go internal/provider/proxmox/seed.go` + shows only `len(req.CloudInit)` and the pass-through into + `nocloud.BuildSeedISO`; `go test -race -count=2 ./internal/provider/proxmox/`; + `grep -n 'not implemented'` lists only Delete. + Done when: Create is real and every listed test passes. Commit: + `proxmox: Create by cloning a template, seed upload, config, start`. + +- **Phase 7 (implementer): Delete, Checker, CRD comment.** Read first: spec + entries 11, 16; contract §2.3, §4.3, §4.13, §8; + `internal/provider/kvm2/{delete.go,delete_test.go,check.go,check_test.go}`; + `internal/provider/check.go`; `api/v1alpha1/providerhealth_types.go`; F13, + F17 in the Phase 0 log. Create `delete.go`: parse the ID (Permanent), the + per-name lock, `resolve`; absent is a logged no-op; name mismatch or + unmarked is a logged no-op; `lock` Transient; `unknown` Transient; + `running` gets `Stop` and `waitTask` (`budgets.stop`); `Destroy` and + `waitTask` (`budgets.destroy`), `isNotExist` from `Destroy` being + success; `DeleteVolume` of the seed on the VM's node tolerating the F13 + absent message; nil. Create `check.go`: `var _ provider.Checker = + (*Provider)(nil)`; `reasonTemplateNotFound = "TemplateNotFound"`, + `reasonStorageNotFound = "StorageNotFound"`; `Check` fans out one + goroutine per node in config order (`wg.Go`), each `checkNode` running + `Version`, `NodeStatus`, `StorageStatus` (a 500 "does not exist" is + `StorageNotFound`; a storage whose `content` lacks `iso` is + `StorageNotFound` with a message saying so), `ClusterResources` for the + templates; `checkReason(err)`: ctx ended `Timeout`, 401 `Unauthorized`, + 403 `Forbidden`, other `*apiError` `APIError`, network `Unreachable`; + Details `pveVersion`, `kernel`, `isoStorage`, `isoStorageAvailGiB`, + `templates`; `Name` is the node, `Address` the base URL. Edit + `api/v1alpha1/providerhealth_types.go` (`kubernetes, gcp, kvm, kvm2 or + proxmox`) and run `make manifests`. Tests `delete_test.go`: + `TestDelete_malformedIDIsPermanent`, `TestDelete_twiceSucceeds`, + `TestDelete_absentIsNoOp`, `TestDelete_foreignNameIsNoOp`, + `TestDelete_unmarkedIsNoOp`, `TestDelete_runningVMStoppedFirst`, + `TestDelete_stoppedVMSkipsStop`, `TestDelete_lockedIsTransient`, + `TestDelete_unknownStatusIsTransient`, `TestDelete_removesSeedISO`, + `TestDelete_toleratesMissingISO`, `TestDelete_destroyTaskFailureIsTransient`, + `TestDelete_usesPurgeAndDestroyUnreferencedDisks`, + `TestDelete_concurrentSameNameSafe`, `TestDelete_noOpLinesAreSilentAtV0`; + in `get_test.go`: `TestGet_afterDeleteReturnsNotFound`, + `TestGet_duringDeleteStillReturnsInstance` (block the fake's `Destroy`). + `check_test.go`: `TestCheck_oneEntryPerNodeInConfigOrder`, + `TestCheck_nodesCheckedConcurrently`, `TestCheck_expiredContextReturnsPromptly`, + `TestCheck_reasonsTable`, `TestCheck_detailsKeys`, `TestCheck_logsVerdictPerNode`, + `TestCheckReason_table`. + Verify: `go test -race -count=2 ./internal/provider/proxmox/ ./internal/providerhealth/`; + `git diff --stat config/` shows only the CRD description line; + `grep -rn 'not implemented' internal/provider/proxmox/` is empty. + Done when: all four contract methods and `Check` are real. Commit: + `proxmox: Delete with seed cleanup, health Checker, CRD type comment`. + +- **Phase 8 (implementer): contract sweep, observability sweep, integration + test, Makefile, testing.md.** Read first: contract §11 and §9; spec + entries 17, 18, 20, 21; `internal/provider/kvm2/{provider_test.go,kvm2_integration_test.go}`; + `internal/provider/tracing_test.go`; `Makefile` integration targets; + `docs/testing.md` kvm2 rows and section. Create `provider_test.go`: the + shared `fixture` (fake, log capture, budgets shrunk), `assertProviderError`, + `TestProvider_cancelledContextReturnsPromptly` (each method blocked on + its first seam call), `TestProvider_everyErrorIsWrappedWithConfiguredName`, + `TestLogging_v0SilentV1OutcomeV2Detail` (V(1) lines `proxmox vm ensured + running`, `proxmox vm fetched`, `proxmox vm deleted`, `proxmox vms + listed`, `proxmox node checked`, every line with `provider`, instance + lines with `vmid` and `name`; V(2) has `cloudInitBytes` and never the + text), `TestTracing_httpSpansUnderOperationSpan` (a real `httpAPI` + against an `httptest` server playing the fixtures for a Get, wrapped in + `provider.WithTracing`, with a span recorder: every HTTP span is a child + of `provider.get` and no attribute carries the token). Create + `proxmox_integration_test.go` (`//go:build integration`): + `loadIntegrationEnv` with the `PROXMOX_TEST_*` variables, `providerConfig()`, + the `integrationLog` pattern from kvm2; `TestProxmox_lifecycle` asserting + in order: `New` returns within 100 ms without a request; `Check` Ready + with `pveVersion`, `isoStorageAvailGiB` and `templates`; `Create` returns + `vms//` with the vmid in range; a second `Create` returns the + same ID and the resources count is unchanged; `Get` reaches Running with + an IPv4 within the budget, UID and CreatedAt as written; `ListByTag` + contains it; a TCP connect to the guest's port is logged only; `Delete` + nil; `Get` NotFound; `Delete` again nil; the ISO storage has no + `-seed.iso` and resources has no VM with the name; an HTTP span is + a child of `provider.create` and no attribute carries the cloud-init + text; `PROXMOX_TEST_KEEP` skips the teardown and prints the `qm destroy` + and ISO delete commands. `TestProxmoxCheck_realHost` creates nothing. Edit + `Makefile`: `test-integration-proxmox` (`go test -race -tags=integration + -count=1 -v -timeout 30m -run TestProxmox ./internal/provider/proxmox/`). + Edit `docs/testing.md`: rows for the package in the suites table and a + "The proxmox integration test" section. Update spec entry 21's tick list + with the final test names. + Verify: `go test -race -count=2 ./internal/provider/proxmox/`; + `go vet -tags=integration ./... && go vet -tags=e2e ./...`; + `make -n test-integration-proxmox`; every §11 item has a test name in + spec entry 21 that exists. + Done when: the package covers every §11 item by name, the integration + test compiles, the target exists. Commit: `proxmox: contract and + observability tests, integration test, testing docs`. + +- **Phase 9 (implementer): deployment and documentation.** Read first: + contract §12 items 11 to 13; spec entries 22, 23, 24; + `config/manager/manager.yaml`, `config/manager/providers_config.yaml`, + `config/samples/{providers-config.yaml,proxy_kvm2.yaml,kustomization.yaml}`; + `Makefile` `kvm-ssh-secret`; `docs/kvm2-provider.md` headings; + `docs/providers-existing.md`; `docs/architecture.md` lines mentioning + kvm2; `README.md`; `docs/providers.md`; `CLAUDE.md` Gotchas. Edit + `config/manager/manager.yaml`: an optional Secret mount `proxmox-token` at + `/etc/proxy-operator/proxmox` (key `token`). Edit `Makefile`: + `PROXMOX_TOKEN_FILE ?= $(HOME)/.config/proxmox-token`, + `proxmox-token-secret` creating `proxmox-token` from `--from-file=token=`. + Edit both providers-config files: a commented `proxmox-home` entry (`url: + https://192.168.2.8:8006`, `tokenFile`, `caFile` or + `insecureSkipTLSVerify`, `nodes`, `isoStorage`, `storage`, `bridge`, + `vmidRange`, `templates`, `machineTypes`). Create + `config/samples/proxy_proxmox.yaml` (provider `proxmox-home`, zone the + node, machine type `small`, image the template key, the Squid cloud-init + from `proxy_kvm2.yaml`) and list it in `config/samples/kustomization.yaml`. + Create `docs/proxmox-provider.md` with sections: Proxmox prerequisites + (the template commands from this plan, no cloudinit drive on ide2, the + guest agent or outbound access, the bridge, the privilege list from F17, + reserving the VMID range), Cluster-side setup (the Secret, the config + entry), Config reference (spec entry 15's table), Placement (Region + ignored, catalogue-edit warning), What lands on Proxmox (VMID scheme, + description mark, tag, seed ISO, config keys, `qm unlock` for a stuck + clone), Logging, Tracing, Timeouts, Health check, Error classes, Known + gaps (spec entry 24), Testing. Edit `docs/providers-existing.md`: a fifth + column in the three tables and a proxmox section mirroring kvm2's. Edit + `docs/architecture.md`: components row, a proxmox block in §5, the span + note in §10, a §11 row (and the "All four providers" count), Decisions + bullets. Edit `README.md`, `docs/providers.md`, `CLAUDE.md` Gotchas + (`make test-integration-proxmox`). + Verify: `bin/kustomize build config/default >/dev/null && bin/kustomize build config/samples >/dev/null`; + a test `TestLoadConfig_proxmoxSampleEntry` loads the uncommented block + from `config/samples/providers-config.yaml` through `provider.LoadConfig`; + a link sweep over the edited docs finds no missing target; + `grep -c '—' docs/proxmox-provider.md` is 0. + Done when: manifests build, the sample loads, docs are linked and + link-clean. Commit: `proxmox: deployment files, sample, user guide and + doc rows`. + +- **Phase 10 (orchestrator with the user): review, real-host validation, + changelog, MR.** Read first: `docs/plans-executions/2026-09-17-1515-kvm2-review-fixes.md`; + `.claude/agents/go-operator-reviewer.md`; the latest report under + `docs/reviews/` for the shape; the spec; this plan's Risks. Do: run + `go-operator-reviewer` on the branch with the spec and the contract as + inputs, then `/code-review high`, `make lint` (if the local toolchain + panics in staticcheck as it did during kvm2, record it and run + `bin/golangci-lint run ./internal/provider/proxmox/...` with staticcheck + disabled), `make test`. Fix accepted findings with regression tests in + one commit; record rejected ones under `docs/reviews/-proxmox-provider-review/`. + Hand the user the host checklist (template ready, no leftover VMs in the + range, ISO storage has space, the token's privileges per F17) and the + `PROXMOX_TEST_*` export lines; with the go-ahead run + `make test-integration-proxmox` and record the clone duration and time to + IP. Optionally the in-cluster pass on kind: build and load the image, + `make proxmox-token-secret`, a scratch ConfigMap with the `proxmox-home` + entry, apply `proxy_proxmox.yaml`, wait for `Ready`, `kubectl get + providerhealth proxmox-home -o yaml`, a `curl -x` through the proxy, + delete and confirm the VM and ISO are gone; scratch overrides are not + committed. After the user confirms: CHANGELOG entry + (`date "+%Y-%m-%d %H:%M %Z"`), push with `-u` using the token helper from + memory (`~/.config/gitea-token`), open the MR with the Gitea API + (`POST /api/v1/repos/kacerr/egress-proxies-operator/pulls` with `head: + feat/proxmox-provider`, `base: main`, the body ending with the + attribution line; `pr.json` in the scratchpad, never committed). Never + merge. + Verify: the integration run's ordered assertions pass on 192.168.2.8; + `make test` green; the review report lists every finding as accepted or + rejected. + Done when: the MR URL is printed and the execution log's last box is + ticked. Commit: `proxmox: review fixes, real-host validation record, + changelog`. + +## Verification + +- Per phase: the `Verify:` clause, plus `go build ./... && go test -short ./...` + before every commit and `git diff --stat` reviewed for files outside the + phase's list. +- Contract mapping: after Phase 8 every rule of contract §2 and §8 has a + named test in the package, and spec entry 21 maps the §11 list 1:1 to + existing test names. +- Fixture discipline: every message `classify` or a matcher helper compares + against is loaded from `testdata/` in `TestClassify_table` or + `TestAPIError_matchers`; a string literal in `errors.go` without a fixture + is a review finding. +- Secrets: `TestClient_tokenNeverInErrorsOrLogs` and + `TestCreate_cloudInitSentinelNeverLoggedOrInErrors` run at V(2), and the + integration test asserts no span attribute carries the cloud-init text. +- End to end: the integration run (Phase 8's assertions) on 192.168.2.8 and + the optional kind pass in Phase 10. + +## Risks + +- **Error text location and exact wording (F1 to F6)** decide the whole + NotFound path; if messages vary by storage type or PVE minor version, the + matchers get one fixture per variant and the guide names the tested + version. +- **Clone cleanup (F15).** A failed or interrupted clone may leave a VM with + `lock=clone` that never clears; Create then waits out `budgets.clone` and + returns Transient every time. The guide documents `qm unlock`; the + reconciler keeps retrying rather than latching. +- **`SDN.Use` (F17).** PVE 8 refuses `net0` writes to non-root users without + it; a missing privilege is a 403 Permanent that latches the Proxy until + its spec changes. The guide's privilege list is the mitigation. +- **The ide2 seed and a template with a cloudinit drive** both claim + cloud-init; the Proxy's user-data would be ignored. Documented + prerequisite; the Checker cannot see it. +- **VMID range shared with humans.** A human VM inside the range costs a + probe; a human VM carrying our name is Permanent. The guide says to + reserve the range. +- **Node-local ISO storage after a migration** leaves the seed on the + original node; Delete tolerates its absence and the ISO is a documented + blind spot. Shared ISO storage avoids it. +- **Blocking Create** holds a reconcile worker for up to the clone budget; + the guide sets expectations. +- **The 30s request timeout on the seed upload** is fine for tens of KiB; a + multi-megabyte `vendorData` would trip it. Documented. +- **Two operator installs** on one Proxmox cluster delete each other's VMs + (contract §13); documented in entry 24. +- **kvm2 edits in Phase 3** touch a merged provider; the moved tests are the + guard, and `go vet -tags=integration` keeps its integration test compiling. From 21678191575b21051cd5f7959be6369b26ed57b5 Mon Sep 17 00:00:00 2001 From: kacerr Date: Fri, 18 Sep 2026 15:49:14 +0000 Subject: [PATCH 02/13] docs: retarget the proxmox plan at Proxmox VE 9.2 The host is 9.2.20, not 8.x. Three corrections that would have failed at runtime: - VM.Monitor was removed in PVE 9.0, so it is dropped from the role and the agent read rests on VM.GuestAgent.Audit alone. - The GC-assist tag becomes proxy_operator: current tag validation takes only lowercase letters, digits and underscore, so proxy-operator would have been rejected on every Create. The description mark keeps its hyphenated keys; it is free text, not a tag. - HTTP/2 is documented as absent from pveproxy on every release, so the client disabling it is a fact rather than a guess. Two new Phase 0 facts: F22 for the 9.2 unified authentication-failure message, which invalidates any 401 text remembered from 8.x, and F23 for the node /tmp tmpfs move. F12 records that the upload endpoint takes iso, vztmpl or import on 9.x and still refuses snippets, which is what keeps the seed ISO route the only API-only path for user-data. Co-Authored-By: Claude Opus 5 (1M context) --- .../plans/2026-09-18-1502-proxmox-provider.md | 71 +++++++++++++------ 1 file changed, 51 insertions(+), 20 deletions(-) diff --git a/docs/plans/2026-09-18-1502-proxmox-provider.md b/docs/plans/2026-09-18-1502-proxmox-provider.md index 9dfed1c..d7c42bd 100644 --- a/docs/plans/2026-09-18-1502-proxmox-provider.md +++ b/docs/plans/2026-09-18-1502-proxmox-provider.md @@ -7,10 +7,13 @@ The provider contract (`docs/provider-contract.md`), the spec template (`docs/provider-spec-template.md`) and the kvm2 build proved the workflow: spec first, then implementation against the contract, one commit per step. -This plan adds a third VM-style backend, `proxmox`, on the Proxmox VE 8.x +This plan adds a third VM-style backend, `proxmox`, on the Proxmox VE 9.x REST API at `https://:8006/api2/json`. The user has a real host at -192.168.2.8 and prepares the API token and a cloud-init-capable template -before Phase 0 (checklist below). +192.168.2.8 running **9.2.20** (Debian 13 Trixie, QEMU 11) and prepares the +API token and a cloud-init-capable template before Phase 0 (checklist +below). Everything below is written against 9.2; where 8.x differed the +difference is called out, because the appendix and the guide must name the +version they were proven on. The implementation runs in sessions whose context gets compacted, so every phase is self-contained: it names what to read first, what it creates or @@ -32,13 +35,14 @@ committed together with the code. Follow `CLAUDE.md` for the log format. ## Before Phase 0: what the user prepares on 192.168.2.8 -Run as root on the Proxmox node. Drop any privilege `pveum` rejects on the -installed version (`VM.GuestAgent.Audit` exists from 8.2; older releases use -`VM.Monitor` for the agent read) and note it for F17. +Run as root on the Proxmox node. **`VM.Monitor` was removed in PVE 9.0**, so +the agent read rests on `VM.GuestAgent.Audit` alone and the role below must +not mention `VM.Monitor`. Drop any privilege `pveum` still rejects and note +it for F17. ```bash pveum user add proxy-operator@pve --comment "egress-proxies-operator" -pveum role add ProxyOperator -privs "VM.Allocate VM.Clone VM.Audit VM.Config.CDROM VM.Config.CPU VM.Config.Memory VM.Config.Disk VM.Config.Network VM.Config.Options VM.PowerMgmt VM.Monitor VM.GuestAgent.Audit Datastore.Audit Datastore.AllocateSpace Datastore.AllocateTemplate Sys.Audit SDN.Use" +pveum role add ProxyOperator -privs "VM.Allocate VM.Clone VM.Audit VM.Config.CDROM VM.Config.CPU VM.Config.Memory VM.Config.Disk VM.Config.Network VM.Config.Options VM.PowerMgmt VM.GuestAgent.Audit Datastore.Audit Datastore.AllocateSpace Datastore.AllocateTemplate Sys.Audit SDN.Use" pveum aclmod / -user proxy-operator@pve -role ProxyOperator pveum user token add proxy-operator@pve operator --privsep 0 # The secret is printed once. Save it on the workstation as one line: @@ -71,7 +75,9 @@ else Phase 0 reads from the API. ## Design decisions made with the user (not re-opened) -- **Backend and client.** Proxmox VE 8.x REST API, JSON, HTTP/1.1 only. +- **Backend and client.** Proxmox VE 9.x REST API, JSON, HTTP/1.1 only + (pveproxy offers no HTTP/2 on any release; only Proxmox Backup Server + does). Thin client on `net/http` (`client-kind: thin-client`), no SDK, no new dependency. Auth header `Authorization: PVEAPIToken=@!=`, read from a @@ -110,9 +116,13 @@ else Phase 0 reads from the API. mark window). Content, one per line: `proxy-operator-managed=true`, `proxy-operator-uid=`, `proxy-operator-created-at=`. `created-at: written-at-create`. The config step after the clone also sets - tag `proxy-operator` for UI filtering only. ListByTag: one + tag `proxy_operator` for UI filtering only. **Underscore, not hyphen:** + current PVE tag validation accepts only lowercase letters, digits and + underscore, and a hyphen is rejected; the underscore form is valid under + both the strict and the permissive reading, so it is used regardless of + what F9 finds. ListByTag: one `GET /cluster/resources?type=vm`; candidates are entries with tag - `proxy-operator` or name prefix `proxy-`; a `GET /nodes/{node}/qemu/{vmid}/config` + `proxy_operator` or name prefix `proxy-`; a `GET /nodes/{node}/qemu/{vmid}/config` per candidate for the description; unmarked skipped; one unreadable VM skipped with a V(1) line unless ctx ended; the resources call failing fails the whole list (`list-partial: fail-whole`). @@ -229,8 +239,11 @@ else Phase 0 reads from the API. `status` values (`running`, `stopped`, `unknown`), `type` (`qemu`, `lxc`), `lock`, `tags` (separator), `template`; whether `lock` appears there during a clone. -- F9. Tag rules: allowed characters, case, length, `tags` separator on - `PUT config` and in `/cluster/resources`. +- F9. Tag rules on 9.2: confirm a hyphen is rejected and `proxy_operator` + accepted, plus case folding, length, and the `tags` separator on + `PUT config` and in `/cluster/resources`. The underscore form stays either + way; this fact only decides what the guide tells users about their own + tags. - F10. `description` on `POST .../clone`: accepted, survives verbatim (newlines), max length; whether `name` accepts the 22-character `proxy-` form. @@ -239,7 +252,9 @@ else Phase 0 reads from the API. - F12. Upload endpoint: field names (`content`, `filename`, file part name `filename`), the filename rule (`.iso` suffix, no slashes), the response (UPID), the privilege it needs, and whether re-uploading an existing name - is refused (then delete-then-upload) or overwrites. + is refused (then delete-then-upload) or overwrites. On 9.x `content` + accepts `iso`, `vztmpl` or `import` and still refuses `snippets`, which is + why the seed travels as an ISO; confirm `iso` is what an `.iso` needs. - F13. `DELETE /nodes/{node}/storage/{storage}/content/{volid}`: response shape (UPID or null) and the message for a missing volume. - F14. `status/current` on a stopped VM: is `qmpstatus` present; on a paused @@ -251,16 +266,29 @@ else Phase 0 reads from the API. (`OK` or the error string), and how long a full clone of the template takes. - F17. Privileges: which of the prepared list the token really needs, in - particular `SDN.Use` on `/sdn/zones/localnetwork/` for the `net0` - write and `VM.GuestAgent.Audit` or `VM.Monitor` for the agent read. -- F18. HTTP/2 is not offered (`curl --http2 -I` falls back or fails), so the - client disables it. + particular the exact ACL path that satisfies `SDN.Use` for the `net0` + write (the 9.x docs describe it as "access SDN vnets and local network + bridges" without spelling out the path for a plain bridge), and + `VM.GuestAgent.Audit` for the agent read. Also whether the config step + needs `VM.Config.Cloudinit` even though only `ide2` is written. +- F18. HTTP/2 is not offered by pveproxy on any release (documented), so the + client disables it; confirm with `curl --http2 -I` that the connection + falls back to HTTP/1.1. - F19. The template's disk key (`scsi0` or `virtio0`), its `boot` line, that ide2 is free and no `cloudinit` drive exists. - F20. `resize` on a stopped VM: request shape, the message when the target is smaller or equal, and that it is synchronous. - F21. Agent response shape: `result[].name`, `result[]["ip-addresses"][]` with `ip-address`, `ip-address-type`. +- F22. The 401 body and reason phrase on 9.2. PVE 9.2 unified the + authentication-failure message so a failed login no longer leaks whether + the user exists, so any 401 text remembered from 8.x is wrong. Capture it + fresh into `err_unauthorized.json`. +- F23. Where the upload endpoint stages the request body, given that PVE 9 + moved the node's `/tmp` to a tmpfs capped at half of RAM and cleans `/tmp` + and `/var/tmp` periodically. A seed ISO is tens of KiB so this cannot + bite in practice, but the observation belongs in the guide next to the + size limit. ## Proxmox endpoint map for the seam @@ -602,7 +630,7 @@ idioms (`fixture`, `assertProviderError`, `logCapture`). calling `Clone` with `vmidFor(req.UID, rng, attempt)`, `isAlreadyExists` moving to the next attempt, exhaustion Transient; `waitTask` with `budgets.clone`; `uploadSeed`; `SetVMConfig` with `cores`, `memory`, - `agent=1`, `tags=proxy-operator`, `ide2=,media=cdrom`, and `net0` + `agent=1`, `tags=proxy_operator`, `ide2=,media=cdrom`, and `net0` rewritten from the cloned config with `bridge=` only when configured (MAC kept); `bootDisk(cfg)` and `Resize` when `diskGiB` is above the current `size=`; `Start` and `waitTask` with `budgets.start`; @@ -817,12 +845,15 @@ idioms (`fixture`, `assertProviderError`, `logCapture`). - **Error text location and exact wording (F1 to F6)** decide the whole NotFound path; if messages vary by storage type or PVE minor version, the matchers get one fixture per variant and the guide names the tested - version. + version. This is not hypothetical: 9.2 rewrote the authentication-failure + message (F22), so fixtures captured from an 8.x host would already be + wrong. Every matched string comes from 192.168.2.8 at 9.2.20, and the + guide says so. - **Clone cleanup (F15).** A failed or interrupted clone may leave a VM with `lock=clone` that never clears; Create then waits out `budgets.clone` and returns Transient every time. The guide documents `qm unlock`; the reconciler keeps retrying rather than latching. -- **`SDN.Use` (F17).** PVE 8 refuses `net0` writes to non-root users without +- **`SDN.Use` (F17).** PVE 9 refuses `net0` writes to non-root users without it; a missing privilege is a 403 Permanent that latches the Proxy until its spec changes. The guide's privilege list is the mitigation. - **The ide2 seed and a template with a cloudinit drive** both claim From ff94fe70bbf9b390665b91ac7da74e9425c871be Mon Sep 17 00:00:00 2001 From: kacerr Date: Wed, 23 Sep 2026 08:22:41 +0000 Subject: [PATCH 03/13] proxmox: captured API fixtures and host facts from pv1-1 (PVE 9.2.20) Phase 0 of the proxmox provider plan. 55 fixtures under internal/provider/proxmox/testdata, each the status, reason phrase and raw body of a real response from the user's host, and the execution log answering the plan's facts F1 to F23. Recon ran a clone of the Debian 13 template through the whole path the provider will take, twice: untagged and on VLAN 4. On both, the NoCloud seed on ide2 installed the guest agent and an address came back about 50 seconds after start. It also found two bugs in the plan: a VM being cloned has no name in /cluster/resources, so name-first lookups would read it as gone, and a crash-retry would clone a duplicate. Those and eight smaller corrections are listed for the spec. Decisions made with the user: seeds go to a dedicated proxy-seeds storage where the token can delete them; proxies land on a VLAN set in config (vlanTag); the API URL is https://192.168.0.8:8006, the address in the node certificate. Co-Authored-By: Claude Opus 5.5 (1M context) --- .../2026-09-18-1502-proxmox-provider.md | 282 ++++++++++++++++++ .../agent_network_get_interfaces.json | 5 + .../agent_network_get_interfaces_vlan.json | 5 + .../proxmox/testdata/clone_response.json | 5 + .../proxmox/testdata/cluster_resources.json | 5 + .../cluster_resources_during_clone.json | 5 + .../cluster_resources_during_destroy.json | 5 + .../testdata/config_during_destroy.json | 5 + .../content_delete_again_response.json | 5 + .../testdata/content_delete_response.json | 5 + .../proxmox/testdata/destroy_response.json | 5 + .../testdata/err_agent_not_running.json | 5 + .../testdata/err_agent_vm_stopped.json | 5 + .../err_agent_vm_stopped_agent_enabled.json | 5 + .../proxmox/testdata/err_bad_param.json | 5 + .../testdata/err_clone_bad_storage.json | 5 + .../err_config_while_clone_locked.json | 5 + .../proxmox/testdata/err_destroy_missing.json | 5 + .../proxmox/testdata/err_forbidden.json | 5 + .../testdata/err_forbidden_volume_delete.json | 5 + .../proxmox/testdata/err_newid_in_use.json | 5 + .../proxmox/testdata/err_no_credentials.json | 5 + .../proxmox/testdata/err_tag_hyphen.json | 5 + .../proxmox/testdata/err_tag_invalid.json | 5 + .../proxmox/testdata/err_unauthorized.json | 5 + .../err_unauthorized_unknown_token.json | 5 + .../proxmox/testdata/err_upload_bad_name.json | 5 + .../proxmox/testdata/err_vm_missing.json | 5 + .../testdata/err_vm_missing_status.json | 5 + .../proxmox/testdata/node_status.json | 5 + .../proxmox/testdata/resize_response.json | 5 + .../testdata/resize_same_size_response.json | 5 + .../testdata/resize_shrink_response.json | 5 + .../proxmox/testdata/set_config_response.json | 5 + .../testdata/set_config_vlan_response.json | 5 + .../proxmox/testdata/start_response.json | 5 + .../start_while_clone_locked_response.json | 5 + .../testdata/status_current_paused.json | 5 + .../testdata/status_current_running.json | 5 + .../testdata/status_current_stopped.json | 5 + .../proxmox/testdata/stop_response.json | 5 + .../proxmox/testdata/storage_content_iso.json | 5 + .../proxmox/testdata/storage_status.json | 5 + .../proxmox/testdata/task_status_clone.json | 5 + .../proxmox/testdata/task_status_running.json | 5 + .../task_status_start_waited_for_lock_ok.json | 5 + .../testdata/task_status_stopped_error.json | 5 + .../testdata/task_status_stopped_ok.json | 5 + .../testdata/task_status_volume_missing.json | 5 + .../testdata/upload_overwrite_response.json | 5 + .../proxmox/testdata/upload_response.json | 5 + .../testdata/upload_response_proxy_seeds.json | 5 + .../provider/proxmox/testdata/version.json | 5 + .../proxmox/testdata/vm_config_clone.json | 5 + .../proxmox/testdata/vm_config_template.json | 5 + .../testdata/vm_config_template_ubuntu.json | 5 + 56 files changed, 557 insertions(+) create mode 100644 docs/plans-executions/2026-09-18-1502-proxmox-provider.md create mode 100644 internal/provider/proxmox/testdata/agent_network_get_interfaces.json create mode 100644 internal/provider/proxmox/testdata/agent_network_get_interfaces_vlan.json create mode 100644 internal/provider/proxmox/testdata/clone_response.json create mode 100644 internal/provider/proxmox/testdata/cluster_resources.json create mode 100644 internal/provider/proxmox/testdata/cluster_resources_during_clone.json create mode 100644 internal/provider/proxmox/testdata/cluster_resources_during_destroy.json create mode 100644 internal/provider/proxmox/testdata/config_during_destroy.json create mode 100644 internal/provider/proxmox/testdata/content_delete_again_response.json create mode 100644 internal/provider/proxmox/testdata/content_delete_response.json create mode 100644 internal/provider/proxmox/testdata/destroy_response.json create mode 100644 internal/provider/proxmox/testdata/err_agent_not_running.json create mode 100644 internal/provider/proxmox/testdata/err_agent_vm_stopped.json create mode 100644 internal/provider/proxmox/testdata/err_agent_vm_stopped_agent_enabled.json create mode 100644 internal/provider/proxmox/testdata/err_bad_param.json create mode 100644 internal/provider/proxmox/testdata/err_clone_bad_storage.json create mode 100644 internal/provider/proxmox/testdata/err_config_while_clone_locked.json create mode 100644 internal/provider/proxmox/testdata/err_destroy_missing.json create mode 100644 internal/provider/proxmox/testdata/err_forbidden.json create mode 100644 internal/provider/proxmox/testdata/err_forbidden_volume_delete.json create mode 100644 internal/provider/proxmox/testdata/err_newid_in_use.json create mode 100644 internal/provider/proxmox/testdata/err_no_credentials.json create mode 100644 internal/provider/proxmox/testdata/err_tag_hyphen.json create mode 100644 internal/provider/proxmox/testdata/err_tag_invalid.json create mode 100644 internal/provider/proxmox/testdata/err_unauthorized.json create mode 100644 internal/provider/proxmox/testdata/err_unauthorized_unknown_token.json create mode 100644 internal/provider/proxmox/testdata/err_upload_bad_name.json create mode 100644 internal/provider/proxmox/testdata/err_vm_missing.json create mode 100644 internal/provider/proxmox/testdata/err_vm_missing_status.json create mode 100644 internal/provider/proxmox/testdata/node_status.json create mode 100644 internal/provider/proxmox/testdata/resize_response.json create mode 100644 internal/provider/proxmox/testdata/resize_same_size_response.json create mode 100644 internal/provider/proxmox/testdata/resize_shrink_response.json create mode 100644 internal/provider/proxmox/testdata/set_config_response.json create mode 100644 internal/provider/proxmox/testdata/set_config_vlan_response.json create mode 100644 internal/provider/proxmox/testdata/start_response.json create mode 100644 internal/provider/proxmox/testdata/start_while_clone_locked_response.json create mode 100644 internal/provider/proxmox/testdata/status_current_paused.json create mode 100644 internal/provider/proxmox/testdata/status_current_running.json create mode 100644 internal/provider/proxmox/testdata/status_current_stopped.json create mode 100644 internal/provider/proxmox/testdata/stop_response.json create mode 100644 internal/provider/proxmox/testdata/storage_content_iso.json create mode 100644 internal/provider/proxmox/testdata/storage_status.json create mode 100644 internal/provider/proxmox/testdata/task_status_clone.json create mode 100644 internal/provider/proxmox/testdata/task_status_running.json create mode 100644 internal/provider/proxmox/testdata/task_status_start_waited_for_lock_ok.json create mode 100644 internal/provider/proxmox/testdata/task_status_stopped_error.json create mode 100644 internal/provider/proxmox/testdata/task_status_stopped_ok.json create mode 100644 internal/provider/proxmox/testdata/task_status_volume_missing.json create mode 100644 internal/provider/proxmox/testdata/upload_overwrite_response.json create mode 100644 internal/provider/proxmox/testdata/upload_response.json create mode 100644 internal/provider/proxmox/testdata/upload_response_proxy_seeds.json create mode 100644 internal/provider/proxmox/testdata/version.json create mode 100644 internal/provider/proxmox/testdata/vm_config_clone.json create mode 100644 internal/provider/proxmox/testdata/vm_config_template.json create mode 100644 internal/provider/proxmox/testdata/vm_config_template_ubuntu.json diff --git a/docs/plans-executions/2026-09-18-1502-proxmox-provider.md b/docs/plans-executions/2026-09-18-1502-proxmox-provider.md new file mode 100644 index 0000000..058a1c6 --- /dev/null +++ b/docs/plans-executions/2026-09-18-1502-proxmox-provider.md @@ -0,0 +1,282 @@ +# Execution: proxmox, a Proxmox VE provider built from the provider contract + +Pairs with [docs/plans/2026-09-18-1502-proxmox-provider.md](../plans/2026-09-18-1502-proxmox-provider.md). + +## Status + +- [x] Phase 0 — Plan file, branch, host recon, fixtures +- [ ] Phase 1 — The spec (`docs/proxmox-provider-spec.md`) +- [ ] Phase 2 — Config block, registration, package skeleton +- [ ] Phase 3 — Extract `nocloud` and `keyedlock` from kvm2 +- [ ] Phase 4 — Seam, HTTP client, decorator, fake +- [ ] Phase 5 — Get, ListByTag, task helpers +- [ ] Phase 6 — Create +- [ ] Phase 7 — Delete, Checker, CRD comment +- [ ] Phase 8 — Contract and observability sweep, integration test, testing.md +- [ ] Phase 9 — Deployment and documentation +- [ ] Phase 10 — Review, real-host validation, changelog, MR + +## Phase 0 — Plan file, branch, host recon, fixtures + +The plan was committed alone on `main` on 2026-09-18 (`708989c`, then +`2167819` retargeting it at PVE 9.2). `origin/main` had not moved in the +meantime, so the branch was cut straight from local `main`: + +```bash +git fetch -q origin && git rev-list --left-right --count origin/main...main # 0 2 +git switch -c feat/proxmox-provider +``` + +The host is **pv1-1**, a single node (no cluster peers), `pve-manager/9.2.20`, +kernel `7.0.14-17-pve`, QEMU 11.0. The user prepared the token +(`~/.config/proxmox-token`, privsep off) and the CA (`~/.config/proxmox-ca.pem`) +before the session. + +**TLS: 192.168.2.8 is not in the node certificate.** The first probe failed +with curl exit 60. The CA is right (`openssl s_client -CAfile` verifies the +chain), but the leaf's SANs are `127.0.0.1, ::1, localhost, 192.168.0.8, +pv1-1, pv1-1.home.lab`, and neither name resolves on the workstation. +192.168.0.8 and 192.168.2.8 present the same certificate fingerprint: the +node sits on `vmbr0` (192.168.0.8/24, untagged) with VLAN interfaces +`vmbr0.2` (192.168.2.8/24) and `vmbr0.4` (192.168.4.8/24). All recon kept +full verification by connecting to 192.168.2.8 while verifying the +certificate as `pv1-1.home.lab`: + +```bash +curl -s --resolve pv1-1.home.lab:8006:192.168.2.8 --cacert ~/.config/proxmox-ca.pem \ + -H @<(printf 'Authorization: PVEAPIToken=%s\n' "$(tr -d '[:space:]' < ~/.config/proxmox-token)") \ + https://pv1-1.home.lab:8006/api2/json/version +# {"data":{"release":"9.2","version":"9.2.20","repoid":"49318c671b82f31e"}} +``` + +Every capture went through a scratch helper wrapping exactly that call. It +passes the token through a file descriptor so it never reaches argv or +output, and writes `{"status","reason","body"}` to +`internal/provider/proxmox/testdata/.json`, refusing to save a body +that contains `PVEAPIToken`. + +A real NoCloud seed was used instead of the plan's 1 KiB placeholder, so the +agent could be installed and F21 captured from a live guest. No ISO tool is +installed locally, so a throwaway module in the scratchpad reused the +repository's `github.com/kdomanski/iso9660 v0.4.0` from the module cache and +copied kvm2's `guestAgentScript` verbatim: + +```bash +GOFLAGS=-mod=mod GOPROXY=off GOSUMDB=off go run . proxy-reconaaaaaaaaaaa proxy-reconaaaaaaaaaaa-seed.iso +# 45056 bytes, ISO 9660 'cidata' +``` + +Write experiments ran against VMID 199999 (`proxy-reconaaaaaaaaaaa`), a +clone of template 9001. Sequence: validation error; clone; resources, config +write and start while the clone lock was held; status and agent while +running bare; stop; config read; clone onto the taken VMID; tag tests; +upload twice plus a bad name; attach `ide2`, set cores and memory; three +resizes; start and agent poll; suspend and resume; a clone onto a storage +without image support; stop; destroy with `purge=1&destroy-unreferenced-disks=1`; +destroy again; ISO delete. + +### Facts F1 to F23 (cited as Evidence by the spec) + +- **F1.** Errors carry the text twice: in the HTTP reason phrase and as the + body's `message`, which ends in `\n`. A 400 adds `errors`, a map from + parameter to message. A 401 has the reason phrase only and an empty body. + The client should use the trimmed body `message` and fall back to the + reason phrase. Headers are ordinary CRLF. +- **F2.** Missing VM, identical for `GET config`, `GET status/current` and + `DELETE`: `500 Configuration file 'nodes/pv1-1/qemu-server/999999.conf' does not exist`. +- **F3.** A start issued while the VM is clone-locked is **accepted** (200 + with a UPID). The task log reads `trying to acquire lock... OK` and the VM + starts once the clone finishes. Setting a lock through the API was not + tried. +- **F4.** Clone onto a taken VMID, synchronous: + `500 unable to create VM 199999: config file already exists`. +- **F5.** Agent enabled but not running: `500 QEMU guest agent is not running`. + Agent not enabled in the config: `500 No QEMU guest agent configured`. + Stopped VM with `agent: 1`: `500 VM 199999 is not running`. +- **F6.** Not produced. `local-lvm` is thin and overprovisions, so + `quota: absent`. +- **F7.** Config write while clone-locked: `500 VM is locked (clone)`. +- **F8.** `/cluster/resources?type=vm` entries carry `vmid, name, node, + status, type, template, tags, maxcpu, maxmem, maxdisk, uptime` and more; + `lock` appears only when set. Status values seen: `running`, `stopped`, + and `unknown` during a clone. **Mid-clone the entry has no `name`**: only + `type, vmid, id, node, lock: clone, status: unknown`. +- **F9.** Hyphenated tags are **accepted** and stored as given. Uppercase is + folded (`Proxy_Operator` → `proxy_operator`), the separator is `;`, and + characters such as a space or `!` give + `400 … tags: invalid format - invalid characters in tag`. The datacenter + has no `tag-style` or `user-tag-access` set. +- **F10.** `description` on clone is accepted and survives verbatim, + newlines included. The 22-character `proxy-` name is accepted. +- **F11.** The clone's `meta` is copied from the template + (`creation-qemu=11.0.3,ctime=1790145676`, identical to 9001), so it says + nothing about when the clone was made. `written-at-create` is required. +- **F12.** Upload: multipart with field `content=iso` and the file part named + `filename`, carrying the target file name. Returns a UPID for an + `imgcopy` task. A name without `.iso` gives + `400 … filename: wrong file extension`. **Re-uploading an existing name + silently overwrites it** (the task log shows a plain `cp`). + `Datastore.AllocateTemplate` suffices. +- **F13.** Deleting a volume needs `Datastore.Allocate` on + `/storage/`; without it the call gives + `403 Permission check failed (/storage/local, Datastore.Allocate)`. With it + (on `proxy-seeds`), the delete **runs as an `imgdel` task** and returns a + UPID. Deleting a volume that no longer exists is also accepted and its + task still ends `OK`; the log reads `disk image '…' does not exist`, then + `Removed volume`. Seed cleanup is idempotent with no message matching. +- **F14.** Stopped: `status: stopped, qmpstatus: stopped`. Paused: + `status: running, qmpstatus: paused` in `status/current`, while + `/cluster/resources` shows plain `running`. +- **F15.** A clone onto a storage without image support is rejected up + front (`400 … storage 'local' does not support vm images`) and leaves no + VM. A clone failing mid-task was not observed. +- **F16.** Task status fields: `status` (`running`, `stopped`), `exitstatus` + (`OK` or the error text), `type, id, node, user, tokenid, upid, starttime, + pid, pstart`. A full clone of the 3 GiB template onto `local-lvm` finished + within about two seconds. +- **F17.** The planned role, granted on `/`, covered clone, config, resize, + start, stop, suspend, agent read, upload and destroy. **Missing: + `Datastore.Allocate` for deleting the seed ISO.** `VM.Config.Cloudinit` is + not needed for the `ide2` write. `SDN.Use` was held but never exercised, + because no call rewrote `net0`. +- **F18.** HTTP/1.1 only: `curl --http2` negotiates `HTTP/1.1`. +- **F19.** Template 9001 (`debian-13-template`): `scsi0` on `local-lvm`, + `boot: order=scsi0`, `ide2` free, `agent: 1`, OVMF, q35, `net0` untagged on + `vmbr0`. **Attaching `ide2` makes Proxmox append it to the boot order** + (`order=scsi0;ide2`). Template 9000 (Ubuntu) is unsuitable: it has a + Proxmox cloud-init drive on `ide2` and VLAN tag 4 on `net0`. +- **F20.** Resize **runs as a task** (returns a UPID). The same size is an + `OK` no-op; shrinking fails inside the task with `exitstatus` + `shrinking disks is not supported`. +- **F21.** Agent response: `result[]` of `{name, ip-addresses[{ip-address, + ip-address-type}]}`. The live guest reported `lo` and `ens18` with + `192.168.0.150`, 50 seconds after start, via DHCP on untagged `vmbr0`. + This proves the NoCloud-seed-on-`ide2` route end to end on this host. +- **F22.** A wrong secret and an unknown token id both give + `401 Authentication failed!`; no credentials gives `401 No ticket`. +- **F23.** Uploads are staged in `/var/tmp/pveupload-*` and then copied to + `/var/lib/vz/template/iso/`, so the tmpfs `/tmp` is not involved. + +### Corrections to the plan that Phase 1 must fold into the spec + +1. **Check `lock` before the name.** A mid-clone VM has no name in + `/cluster/resources` (F8), so name-first logic turns a clone in progress + into NotFound. Get, Delete and Create's lookup must treat a locked or + nameless entry at our VMID as in-progress (Transient or wait), never + absent. +2. **Inspect a taken VMID before probing forward.** After a crash mid-clone, + the retry's name lookup misses the nameless VM, the clone onto the + deterministic VMID fails with F4, and forward probing would clone a + duplicate. On F4, read that VMID: `lock: clone` means wait for it, and our + UID in the description means adopt it. Probe forward only for a VM that + is demonstrably someone else's. +3. **A start queues behind a lock** instead of failing (F3). +4. **Resize is asynchronous** (F20); wait on its task. +5. **Drop delete-before-upload** for the seed: upload overwrites (F12). + **Seed deletion is a task** (F13): wait on it. A repeat delete succeeds, + so Delete needs no absent-volume matcher. +6. **Get reads `status/current`**, not only `/cluster/resources`, to tell + paused from running (F14). +7. **Tag rationale**: hyphens are accepted on 9.2.20 (F9). `proxy_operator` + stays, since it is valid everywhere, but the claim that hyphens are + rejected is removed. +8. **The boot-disk logic must skip `ide2`**, which Proxmox adds to the boot + order (F19). The plan's rule already excludes it; add a test with + `order=scsi0;ide2`. +9. **Three agent messages, not one** (F5): `QEMU guest agent is not running` + (booting, or the install failed), `No QEMU guest agent configured` (our + `agent=1` was removed), and `VM is not running` (stopped between + the status read and the agent call). The spec decides each class; none + may become NotFound. +10. **Internet on the proxy network is a hard prerequisite**, twice over: the + vendor-data agent install needs a package mirror, and the proxy itself + needs egress. The guide says so, and recommends baking + `qemu-guest-agent` into the template with `installGuestAgent: false` to + drop the mirror dependency from IP discovery. + +### Decisions made with the user in Phase 0 + +- **Seeds go to a dedicated directory storage**, `proxy-seeds`, with ISO + content only. The token gets `Datastore.Allocate` there and nowhere else, + so it can delete seeds but never backups or other ISOs on `local`. The + sample config's `isoStorage` becomes `proxy-seeds`. The guide must say that + an ACL on a more specific path replaces the inherited one for that user, + so the role granted on `/storage/proxy-seeds` must repeat + `Datastore.AllocateTemplate`, `AllocateSpace` and `Audit` alongside + `Allocate`. +- **Proxies land on a VLAN** set in config. `ProxmoxConfig` gains an + optional `vlanTag` (1 to 4094), applied when Create rewrites `net0` + together with the optional bridge override, keeping the MAC. Unset leaves + `net0` as cloned. This is the first call that exercises `SDN.Use` (F17). +- **The API URL is `https://192.168.0.8:8006`**, the address in the node + certificate, so `caFile` verifies with no provider change. No + `tlsServerName` option. The operator's cluster must be able to reach + 192.168.0.x. + +### Seed storage and VLAN 4 + +The user created `proxy-seeds` (`dir`, ISO content only), granted the token +`Datastore.Allocate`, `AllocateSpace`, `AllocateTemplate` and `Audit` on +`/storage/proxy-seeds` through a role `ProxyOperatorSeeds`, and freed the +leftover recon ISO on `local`. An upload to `proxy-seeds` and two deletes of +a throwaway ISO there produced `content_delete_response.json`, +`content_delete_again_response.json` and `task_status_volume_missing.json`. + +The user gave the VLAN as "vlan 4, bridge: vmbr0.4@vmbr0". That is the +host's own VLAN interface name as `ip link` prints it; a VM NIC cannot +attach to it because it is not a bridge. On the VLAN-aware `vmbr0` the +Proxmox form is `bridge=vmbr0,tag=4`, which is what template 9000 uses, so +the provider's `vlanTag: 4` appends `tag=4` to the cloned `net0`. + +A second clone of 9001 at VMID 199999 got its seed from `proxy-seeds` and +`net0` rewritten from `virtio=BC:24:11:EE:48:83,bridge=vmbr0` to +`virtio=BC:24:11:EE:48:83,bridge=vmbr0,tag=4`. **The write succeeded, so the +token's `SDN.Use` on `/` covers a VLAN tag on `vmbr0`** +(`set_config_vlan_response.json`). + +**The first VLAN 4 attempt never produced an agent** in seven minutes of +polling (every 13 seconds, 33 polls). The only signal was the agent call, so +the cause could not be seen from the API. A debug seed added a `recon` user +(user-data only, with a new instance id so cloud-init re-ran), and the user +logged in on the console: DHCP worked, but VLAN 4 had no route to the +internet because of a routing misconfiguration on the Proxmox host, so the +vendor-data `apt-get install qemu-guest-agent` could not run. The user fixed +the routing. + +**Rerun from scratch after the fix**, same sequence the provider will use +(destroy and seed delete first, then a fresh clone and seed, `instance-id` +equal to the name): + +| Step | Elapsed | +|---|---| +| clone task done | 4 s | +| seed upload task done | 4 s | +| config (`ide2`, `net0` with `tag=4`, tag) | 4 s | +| start task done | 5 s | +| agent reports `192.168.4.147` | 47 s after start | + +The guest was reachable from the workstation (ping, TCP 22), as the contract +requires of a proxy address. Stopping it gave the F5 stopped-VM fixture; +destroy and seed delete then left the host with only its original VMs 100, +9000 and 9001, no ISOs on `local` or `proxy-seeds`, and no `local-lvm` +volumes of 199999. The capture is `agent_network_get_interfaces_vlan.json`. + +### Worth noting + +- The token file is mode 664; 600 would be tighter. +- Fixtures: 48 files, all valid JSON, none containing `PVEAPIToken`. + Several names differ from the plan's list because what they captured + differed from what was expected: `start_while_clone_locked_response.json` + (the start succeeded), `task_status_stopped_error.json` (from the shrink, + not a start), `upload_overwrite_response.json` (success, not an error), + `resize_same_size_response.json` and `resize_shrink_response.json` + (UPIDs), `err_forbidden_volume_delete.json` (the 403 from F13). + `err_no_space.json` was not produced (F6). +- The host is back to its original state; see the table above. +- The scratch seed builder (`seedgen` in the session scratchpad) took the + user-data file and instance id as arguments by the end, for the debug + login. It is not part of the repository; Phase 3's `nocloud` package + replaces it. +- Time to IP was about 50 s on both networks, with the agent installed at + first boot. Budgets in the plan (running wait 60 s for the VM state, the + agent discovered by `Get` polling) are consistent with this. diff --git a/internal/provider/proxmox/testdata/agent_network_get_interfaces.json b/internal/provider/proxmox/testdata/agent_network_get_interfaces.json new file mode 100644 index 0000000..3ab9842 --- /dev/null +++ b/internal/provider/proxmox/testdata/agent_network_get_interfaces.json @@ -0,0 +1,5 @@ +{ + "status": 200, + "reason": "OK", + "body": "{\"data\":{\"result\":[{\"statistics\":{\"tx-errs\":0,\"rx-dropped\":0,\"rx-bytes\":7101,\"rx-packets\":86,\"rx-errs\":0,\"tx-dropped\":0,\"tx-packets\":86,\"tx-bytes\":7101},\"name\":\"lo\",\"ip-addresses\":[{\"ip-address\":\"127.0.0.1\",\"ip-address-type\":\"ipv4\",\"prefix\":8},{\"ip-address\":\"::1\",\"ip-address-type\":\"ipv6\",\"prefix\":128}],\"hardware-address\":\"00:00:00:00:00:00\"},{\"hardware-address\":\"bc:24:11:48:11:46\",\"ip-addresses\":[{\"ip-address\":\"192.168.0.150\",\"ip-address-type\":\"ipv4\",\"prefix\":24},{\"ip-address\":\"fe80::be24:11ff:fe48:1146\",\"ip-address-type\":\"ipv6\",\"prefix\":64}],\"name\":\"ens18\",\"statistics\":{\"rx-dropped\":0,\"rx-bytes\":29688403,\"rx-packets\":6744,\"tx-errs\":0,\"tx-bytes\":221044,\"tx-dropped\":0,\"tx-packets\":2970,\"rx-errs\":0}}]}}" +} \ No newline at end of file diff --git a/internal/provider/proxmox/testdata/agent_network_get_interfaces_vlan.json b/internal/provider/proxmox/testdata/agent_network_get_interfaces_vlan.json new file mode 100644 index 0000000..8092193 --- /dev/null +++ b/internal/provider/proxmox/testdata/agent_network_get_interfaces_vlan.json @@ -0,0 +1,5 @@ +{ + "status": 200, + "reason": "OK", + "body": "{\"data\":{\"result\":[{\"ip-addresses\":[{\"prefix\":8,\"ip-address-type\":\"ipv4\",\"ip-address\":\"127.0.0.1\"},{\"prefix\":128,\"ip-address-type\":\"ipv6\",\"ip-address\":\"::1\"}],\"hardware-address\":\"00:00:00:00:00:00\",\"statistics\":{\"tx-errs\":0,\"rx-packets\":86,\"rx-bytes\":7101,\"rx-dropped\":0,\"rx-errs\":0,\"tx-dropped\":0,\"tx-packets\":86,\"tx-bytes\":7101},\"name\":\"lo\"},{\"ip-addresses\":[{\"ip-address-type\":\"ipv4\",\"ip-address\":\"192.168.4.147\",\"prefix\":24},{\"ip-address\":\"fe80::be24:11ff:fea9:b4eb\",\"ip-address-type\":\"ipv6\",\"prefix\":64}],\"hardware-address\":\"bc:24:11:a9:b4:eb\",\"statistics\":{\"rx-errs\":0,\"tx-bytes\":319460,\"tx-dropped\":0,\"tx-packets\":4544,\"tx-errs\":0,\"rx-bytes\":29576554,\"rx-dropped\":0,\"rx-packets\":6027},\"name\":\"ens18\"}]}}" +} \ No newline at end of file diff --git a/internal/provider/proxmox/testdata/clone_response.json b/internal/provider/proxmox/testdata/clone_response.json new file mode 100644 index 0000000..52c1161 --- /dev/null +++ b/internal/provider/proxmox/testdata/clone_response.json @@ -0,0 +1,5 @@ +{ + "status": 200, + "reason": "OK", + "body": "{\"data\":\"UPID:pv1-1:0010F9DC:02801DA1:6AB37871:qmclone:9001:proxy-operator@pve!operator:\"}" +} \ No newline at end of file diff --git a/internal/provider/proxmox/testdata/cluster_resources.json b/internal/provider/proxmox/testdata/cluster_resources.json new file mode 100644 index 0000000..a4d3dab --- /dev/null +++ b/internal/provider/proxmox/testdata/cluster_resources.json @@ -0,0 +1,5 @@ +{ + "status": 200, + "reason": "OK", + "body": "{\"data\":[{\"maxmem\":2147483648,\"netin\":0,\"uptime\":0,\"disk\":0,\"netout\":0,\"mem\":0,\"maxdisk\":34359738368,\"maxcpu\":2,\"type\":\"qemu\",\"memhost\":0,\"template\":0,\"id\":\"qemu/100\",\"cpu\":0,\"name\":\"ubuntu-vm-1\",\"vmid\":100,\"status\":\"stopped\",\"diskwrite\":0,\"diskread\":0,\"node\":\"pv1-1\"},{\"node\":\"pv1-1\",\"diskread\":0,\"diskwrite\":0,\"status\":\"stopped\",\"vmid\":9000,\"cpu\":0,\"name\":\"ubuntu-2604-template\",\"template\":1,\"id\":\"qemu/9000\",\"memhost\":0,\"type\":\"qemu\",\"maxcpu\":2,\"maxdisk\":34359738368,\"mem\":0,\"disk\":0,\"netout\":0,\"uptime\":0,\"netin\":0,\"maxmem\":2147483648},{\"uptime\":0,\"netout\":0,\"disk\":0,\"netin\":0,\"maxmem\":2147483648,\"maxdisk\":3221225472,\"mem\":0,\"id\":\"qemu/9001\",\"template\":1,\"name\":\"debian-13-template\",\"cpu\":0,\"type\":\"qemu\",\"maxcpu\":2,\"memhost\":0,\"diskwrite\":0,\"diskread\":0,\"node\":\"pv1-1\",\"vmid\":9001,\"status\":\"stopped\"}]}" +} \ No newline at end of file diff --git a/internal/provider/proxmox/testdata/cluster_resources_during_clone.json b/internal/provider/proxmox/testdata/cluster_resources_during_clone.json new file mode 100644 index 0000000..847be8c --- /dev/null +++ b/internal/provider/proxmox/testdata/cluster_resources_during_clone.json @@ -0,0 +1,5 @@ +{ + "status": 200, + "reason": "OK", + "body": "{\"data\":[{\"template\":0,\"id\":\"qemu/100\",\"cpu\":0,\"name\":\"ubuntu-vm-1\",\"type\":\"qemu\",\"maxcpu\":2,\"memhost\":0,\"diskread\":0,\"diskwrite\":0,\"node\":\"pv1-1\",\"vmid\":100,\"status\":\"stopped\",\"uptime\":0,\"disk\":0,\"netout\":0,\"netin\":0,\"maxmem\":2147483648,\"maxdisk\":34359738368,\"mem\":0},{\"type\":\"qemu\",\"vmid\":199999,\"lock\":\"clone\",\"status\":\"unknown\",\"id\":\"qemu/199999\",\"node\":\"pv1-1\"},{\"diskwrite\":0,\"diskread\":0,\"node\":\"pv1-1\",\"vmid\":9000,\"status\":\"stopped\",\"id\":\"qemu/9000\",\"template\":1,\"name\":\"ubuntu-2604-template\",\"cpu\":0,\"maxcpu\":2,\"type\":\"qemu\",\"memhost\":0,\"maxdisk\":34359738368,\"mem\":0,\"uptime\":0,\"netout\":0,\"disk\":0,\"netin\":0,\"maxmem\":2147483648},{\"type\":\"qemu\",\"maxcpu\":2,\"memhost\":0,\"template\":1,\"id\":\"qemu/9001\",\"cpu\":0,\"name\":\"debian-13-template\",\"vmid\":9001,\"status\":\"stopped\",\"diskwrite\":0,\"diskread\":0,\"node\":\"pv1-1\",\"netin\":0,\"maxmem\":2147483648,\"uptime\":0,\"disk\":0,\"netout\":0,\"mem\":0,\"maxdisk\":3221225472}]}" +} \ No newline at end of file diff --git a/internal/provider/proxmox/testdata/cluster_resources_during_destroy.json b/internal/provider/proxmox/testdata/cluster_resources_during_destroy.json new file mode 100644 index 0000000..1a74e32 --- /dev/null +++ b/internal/provider/proxmox/testdata/cluster_resources_during_destroy.json @@ -0,0 +1,5 @@ +{ + "status": 200, + "reason": "OK", + "body": "{\"data\":[{\"id\":\"qemu/100\",\"template\":0,\"name\":\"ubuntu-vm-1\",\"cpu\":0,\"type\":\"qemu\",\"maxcpu\":2,\"memhost\":0,\"diskwrite\":0,\"diskread\":0,\"node\":\"pv1-1\",\"vmid\":100,\"status\":\"stopped\",\"uptime\":0,\"netout\":0,\"disk\":0,\"maxmem\":2147483648,\"netin\":0,\"maxdisk\":34359738368,\"mem\":0},{\"tags\":\"proxy_operator\",\"mem\":0,\"maxdisk\":4294967296,\"netin\":0,\"maxmem\":1073741824,\"uptime\":0,\"netout\":0,\"disk\":0,\"vmid\":199999,\"status\":\"stopped\",\"diskread\":0,\"diskwrite\":0,\"node\":\"pv1-1\",\"maxcpu\":1,\"type\":\"qemu\",\"memhost\":0,\"id\":\"qemu/199999\",\"template\":0,\"name\":\"proxy-reconaaaaaaaaaaa\",\"cpu\":0},{\"uptime\":0,\"netout\":0,\"disk\":0,\"netin\":0,\"maxmem\":2147483648,\"maxdisk\":34359738368,\"mem\":0,\"id\":\"qemu/9000\",\"template\":1,\"name\":\"ubuntu-2604-template\",\"cpu\":0,\"maxcpu\":2,\"type\":\"qemu\",\"memhost\":0,\"diskwrite\":0,\"diskread\":0,\"node\":\"pv1-1\",\"vmid\":9000,\"status\":\"stopped\"},{\"mem\":0,\"maxdisk\":3221225472,\"maxmem\":2147483648,\"netin\":0,\"uptime\":0,\"disk\":0,\"netout\":0,\"vmid\":9001,\"status\":\"stopped\",\"diskwrite\":0,\"diskread\":0,\"node\":\"pv1-1\",\"maxcpu\":2,\"type\":\"qemu\",\"memhost\":0,\"template\":1,\"id\":\"qemu/9001\",\"cpu\":0,\"name\":\"debian-13-template\"}]}" +} \ No newline at end of file diff --git a/internal/provider/proxmox/testdata/config_during_destroy.json b/internal/provider/proxmox/testdata/config_during_destroy.json new file mode 100644 index 0000000..fbd9f5b --- /dev/null +++ b/internal/provider/proxmox/testdata/config_during_destroy.json @@ -0,0 +1,5 @@ +{ + "status": 200, + "reason": "OK", + "body": "{\"data\":{\"meta\":\"creation-qemu=11.0.3,ctime=1790145676\",\"description\":\"proxy-operator-managed=true\\nproxy-operator-uid=00000000-0000-0000-0000-00000000recon\\nproxy-operator-created-at=2026-09-23T07:00:00Z\",\"ide2\":\"local:iso/proxy-reconaaaaaaaaaaa-seed.iso,media=cdrom,size=44K\",\"cores\":1,\"memory\":\"1024\",\"scsihw\":\"virtio-scsi-pci\",\"net0\":\"virtio=BC:24:11:48:11:46,bridge=vmbr0\",\"efidisk0\":\"local-lvm:vm-199999-disk-0,efitype=4m,pre-enrolled-keys=0,size=4M\",\"scsi0\":\"local-lvm:vm-199999-disk-1,size=4G\",\"boot\":\"order=scsi0;ide2\",\"vmgenid\":\"2dd339a4-d7fb-4d47-96fd-5a61a25f808d\",\"digest\":\"98a4fff988b5508a7e1f6aa77c34768812d2382c\",\"name\":\"proxy-reconaaaaaaaaaaa\",\"vga\":\"serial0\",\"serial0\":\"socket\",\"smbios1\":\"uuid=f2ade202-ffdb-4a89-9867-3d419384471d\",\"machine\":\"q35\",\"bios\":\"ovmf\",\"tags\":\"proxy_operator\",\"agent\":\"1\",\"ostype\":\"l26\"}}" +} \ No newline at end of file diff --git a/internal/provider/proxmox/testdata/content_delete_again_response.json b/internal/provider/proxmox/testdata/content_delete_again_response.json new file mode 100644 index 0000000..8a3a67f --- /dev/null +++ b/internal/provider/proxmox/testdata/content_delete_again_response.json @@ -0,0 +1,5 @@ +{ + "status": 200, + "reason": "OK", + "body": "{\"data\":\"UPID:pv1-1:00111FF7:02853E62:6AB38591:imgdel:proxy-seeds:proxy-operator@pve!operator:\"}" +} \ No newline at end of file diff --git a/internal/provider/proxmox/testdata/content_delete_response.json b/internal/provider/proxmox/testdata/content_delete_response.json new file mode 100644 index 0000000..7828e53 --- /dev/null +++ b/internal/provider/proxmox/testdata/content_delete_response.json @@ -0,0 +1,5 @@ +{ + "status": 200, + "reason": "OK", + "body": "{\"data\":\"UPID:pv1-1:00111FF3:02853D30:6AB3858E:imgdel:proxy-seeds:proxy-operator@pve!operator:\"}" +} \ No newline at end of file diff --git a/internal/provider/proxmox/testdata/destroy_response.json b/internal/provider/proxmox/testdata/destroy_response.json new file mode 100644 index 0000000..ebb9c3f --- /dev/null +++ b/internal/provider/proxmox/testdata/destroy_response.json @@ -0,0 +1,5 @@ +{ + "status": 200, + "reason": "OK", + "body": "{\"data\":\"UPID:pv1-1:00110564:0281A657:6AB37C5E:qmdestroy:199999:proxy-operator@pve!operator:\"}" +} \ No newline at end of file diff --git a/internal/provider/proxmox/testdata/err_agent_not_running.json b/internal/provider/proxmox/testdata/err_agent_not_running.json new file mode 100644 index 0000000..71e101d --- /dev/null +++ b/internal/provider/proxmox/testdata/err_agent_not_running.json @@ -0,0 +1,5 @@ +{ + "status": 500, + "reason": "QEMU guest agent is not running", + "body": "{\"message\":\"QEMU guest agent is not running\\n\",\"data\":null}" +} \ No newline at end of file diff --git a/internal/provider/proxmox/testdata/err_agent_vm_stopped.json b/internal/provider/proxmox/testdata/err_agent_vm_stopped.json new file mode 100644 index 0000000..d96fc06 --- /dev/null +++ b/internal/provider/proxmox/testdata/err_agent_vm_stopped.json @@ -0,0 +1,5 @@ +{ + "status": 500, + "reason": "No QEMU guest agent configured", + "body": "{\"message\":\"No QEMU guest agent configured\\n\",\"data\":null}" +} \ No newline at end of file diff --git a/internal/provider/proxmox/testdata/err_agent_vm_stopped_agent_enabled.json b/internal/provider/proxmox/testdata/err_agent_vm_stopped_agent_enabled.json new file mode 100644 index 0000000..9916de9 --- /dev/null +++ b/internal/provider/proxmox/testdata/err_agent_vm_stopped_agent_enabled.json @@ -0,0 +1,5 @@ +{ + "status": 500, + "reason": "VM 199999 is not running", + "body": "{\"message\":\"VM 199999 is not running\\n\",\"data\":null}" +} \ No newline at end of file diff --git a/internal/provider/proxmox/testdata/err_bad_param.json b/internal/provider/proxmox/testdata/err_bad_param.json new file mode 100644 index 0000000..28c780f --- /dev/null +++ b/internal/provider/proxmox/testdata/err_bad_param.json @@ -0,0 +1,5 @@ +{ + "status": 400, + "reason": "Parameter verification failed.", + "body": "{\"data\":null,\"message\":\"Parameter verification failed.\\n\",\"errors\":{\"newid\":\"type check ('integer') failed - got 'abc'\"}}" +} \ No newline at end of file diff --git a/internal/provider/proxmox/testdata/err_clone_bad_storage.json b/internal/provider/proxmox/testdata/err_clone_bad_storage.json new file mode 100644 index 0000000..ff24da9 --- /dev/null +++ b/internal/provider/proxmox/testdata/err_clone_bad_storage.json @@ -0,0 +1,5 @@ +{ + "status": 400, + "reason": "Parameter verification failed.", + "body": "{\"errors\":{\"storage\":\"storage 'local' does not support vm images\"},\"message\":\"Parameter verification failed.\\n\",\"data\":null}" +} \ No newline at end of file diff --git a/internal/provider/proxmox/testdata/err_config_while_clone_locked.json b/internal/provider/proxmox/testdata/err_config_while_clone_locked.json new file mode 100644 index 0000000..f6ccf05 --- /dev/null +++ b/internal/provider/proxmox/testdata/err_config_while_clone_locked.json @@ -0,0 +1,5 @@ +{ + "status": 500, + "reason": "VM is locked (clone)", + "body": "{\"data\":null,\"message\":\"VM is locked (clone)\\n\"}" +} \ No newline at end of file diff --git a/internal/provider/proxmox/testdata/err_destroy_missing.json b/internal/provider/proxmox/testdata/err_destroy_missing.json new file mode 100644 index 0000000..cd094f5 --- /dev/null +++ b/internal/provider/proxmox/testdata/err_destroy_missing.json @@ -0,0 +1,5 @@ +{ + "status": 500, + "reason": "Configuration file 'nodes/pv1-1/qemu-server/199999.conf' does not exist", + "body": "{\"data\":null,\"message\":\"Configuration file 'nodes/pv1-1/qemu-server/199999.conf' does not exist\\n\"}" +} \ No newline at end of file diff --git a/internal/provider/proxmox/testdata/err_forbidden.json b/internal/provider/proxmox/testdata/err_forbidden.json new file mode 100644 index 0000000..ed5e2b3 --- /dev/null +++ b/internal/provider/proxmox/testdata/err_forbidden.json @@ -0,0 +1,5 @@ +{ + "status": 403, + "reason": "Permission check failed (/nodes/pv1-1, Sys.Syslog)", + "body": "{\"message\":\"Permission check failed (/nodes/pv1-1, Sys.Syslog)\\n\",\"data\":null}" +} \ No newline at end of file diff --git a/internal/provider/proxmox/testdata/err_forbidden_volume_delete.json b/internal/provider/proxmox/testdata/err_forbidden_volume_delete.json new file mode 100644 index 0000000..f77c1b9 --- /dev/null +++ b/internal/provider/proxmox/testdata/err_forbidden_volume_delete.json @@ -0,0 +1,5 @@ +{ + "status": 403, + "reason": "Permission check failed (/storage/local, Datastore.Allocate)", + "body": "{\"data\":null,\"message\":\"Permission check failed (/storage/local, Datastore.Allocate)\\n\"}" +} \ No newline at end of file diff --git a/internal/provider/proxmox/testdata/err_newid_in_use.json b/internal/provider/proxmox/testdata/err_newid_in_use.json new file mode 100644 index 0000000..78848c9 --- /dev/null +++ b/internal/provider/proxmox/testdata/err_newid_in_use.json @@ -0,0 +1,5 @@ +{ + "status": 500, + "reason": "unable to create VM 199999: config file already exists", + "body": "{\"data\":null,\"message\":\"unable to create VM 199999: config file already exists\\n\"}" +} \ No newline at end of file diff --git a/internal/provider/proxmox/testdata/err_no_credentials.json b/internal/provider/proxmox/testdata/err_no_credentials.json new file mode 100644 index 0000000..a40f962 --- /dev/null +++ b/internal/provider/proxmox/testdata/err_no_credentials.json @@ -0,0 +1,5 @@ +{ + "status": 401, + "reason": "No ticket", + "body": "" +} \ No newline at end of file diff --git a/internal/provider/proxmox/testdata/err_tag_hyphen.json b/internal/provider/proxmox/testdata/err_tag_hyphen.json new file mode 100644 index 0000000..926d1e0 --- /dev/null +++ b/internal/provider/proxmox/testdata/err_tag_hyphen.json @@ -0,0 +1,5 @@ +{ + "status": 200, + "reason": "OK", + "body": "{\"data\":null}" +} \ No newline at end of file diff --git a/internal/provider/proxmox/testdata/err_tag_invalid.json b/internal/provider/proxmox/testdata/err_tag_invalid.json new file mode 100644 index 0000000..2644075 --- /dev/null +++ b/internal/provider/proxmox/testdata/err_tag_invalid.json @@ -0,0 +1,5 @@ +{ + "status": 400, + "reason": "Parameter verification failed.", + "body": "{\"errors\":{\"tags\":\"invalid format - invalid characters in tag\\n\"},\"data\":null,\"message\":\"Parameter verification failed.\\n\"}" +} \ No newline at end of file diff --git a/internal/provider/proxmox/testdata/err_unauthorized.json b/internal/provider/proxmox/testdata/err_unauthorized.json new file mode 100644 index 0000000..6668b45 --- /dev/null +++ b/internal/provider/proxmox/testdata/err_unauthorized.json @@ -0,0 +1,5 @@ +{ + "status": 401, + "reason": "Authentication failed!", + "body": "" +} \ No newline at end of file diff --git a/internal/provider/proxmox/testdata/err_unauthorized_unknown_token.json b/internal/provider/proxmox/testdata/err_unauthorized_unknown_token.json new file mode 100644 index 0000000..6668b45 --- /dev/null +++ b/internal/provider/proxmox/testdata/err_unauthorized_unknown_token.json @@ -0,0 +1,5 @@ +{ + "status": 401, + "reason": "Authentication failed!", + "body": "" +} \ No newline at end of file diff --git a/internal/provider/proxmox/testdata/err_upload_bad_name.json b/internal/provider/proxmox/testdata/err_upload_bad_name.json new file mode 100644 index 0000000..01cbd68 --- /dev/null +++ b/internal/provider/proxmox/testdata/err_upload_bad_name.json @@ -0,0 +1,5 @@ +{ + "status": 400, + "reason": "Parameter verification failed.", + "body": "{\"message\":\"Parameter verification failed.\\n\",\"data\":null,\"errors\":{\"filename\":\"wrong file extension\"}}" +} \ No newline at end of file diff --git a/internal/provider/proxmox/testdata/err_vm_missing.json b/internal/provider/proxmox/testdata/err_vm_missing.json new file mode 100644 index 0000000..8ad56e5 --- /dev/null +++ b/internal/provider/proxmox/testdata/err_vm_missing.json @@ -0,0 +1,5 @@ +{ + "status": 500, + "reason": "Configuration file 'nodes/pv1-1/qemu-server/999999.conf' does not exist", + "body": "{\"message\":\"Configuration file 'nodes/pv1-1/qemu-server/999999.conf' does not exist\\n\",\"data\":null}" +} \ No newline at end of file diff --git a/internal/provider/proxmox/testdata/err_vm_missing_status.json b/internal/provider/proxmox/testdata/err_vm_missing_status.json new file mode 100644 index 0000000..f249284 --- /dev/null +++ b/internal/provider/proxmox/testdata/err_vm_missing_status.json @@ -0,0 +1,5 @@ +{ + "status": 500, + "reason": "Configuration file 'nodes/pv1-1/qemu-server/999999.conf' does not exist", + "body": "{\"data\":null,\"message\":\"Configuration file 'nodes/pv1-1/qemu-server/999999.conf' does not exist\\n\"}" +} \ No newline at end of file diff --git a/internal/provider/proxmox/testdata/node_status.json b/internal/provider/proxmox/testdata/node_status.json new file mode 100644 index 0000000..79afdbe --- /dev/null +++ b/internal/provider/proxmox/testdata/node_status.json @@ -0,0 +1,5 @@ +{ + "status": 200, + "reason": "OK", + "body": "{\"data\":{\"rootfs\":{\"total\":100861726720,\"avail\":88717860864,\"used\":6973128704,\"free\":93888598016},\"pveversion\":\"pve-manager/9.2.20/49318c671b82f31e\",\"current-kernel\":{\"machine\":\"x86_64\",\"sysname\":\"Linux\",\"release\":\"7.0.14-17-pve\",\"version\":\"#1 SMP PREEMPT_DYNAMIC PMX 7.0.14-17 (2026-09-10T10:16Z)\"},\"cpu\":0,\"cpuinfo\":{\"mhz\":\"1100.947\",\"user_hz\":100,\"cpus\":16,\"sockets\":1,\"model\":\"AMD Ryzen 9 PRO 8945HS w/ Radeon 780M Graphics\",\"vendor\":\"AuthenticAMD\",\"family\":\"25\",\"hvm\":\"1\",\"flags\":\"fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush mmx fxsr sse sse2 ht syscall nx mmxext fxsr_opt pdpe1gb rdtscp lm constant_tsc rep_good amd_lbr_v2 nopl xtopology nonstop_tsc cpuid extd_apicid aperfmperf rapl pni pclmulqdq monitor ssse3 fma cx16 sse4_1 sse4_2 x2apic movbe popcnt aes xsave avx f16c rdrand lahf_lm cmp_legacy svm extapic cr8_legacy abm sse4a misalignsse 3dnowprefetch osvw ibs skinit wdt tce topoext perfctr_core perfctr_nb bpext perfctr_llc mwaitx cpuid_fault cpb cat_l3 cdp_l3 hw_pstate ssbd mba perfmon_v2 ibrs ibpb stibp ibrs_enhanced vmmcall fsgsbase bmi1 avx2 smep bmi2 erms invpcid cqm rdt_a avx512f avx512dq rdseed adx smap avx512ifma clflushopt clwb avx512cd sha_ni avx512bw avx512vl xsaveopt xsavec xgetbv1 xsaves cqm_llc cqm_occup_llc cqm_mbm_total cqm_mbm_local user_shstk avx512_bf16 clzero irperf xsaveerptr rdpru wbnoinvd cppc arat npt lbrv svm_lock nrip_save tsc_scale vmcb_clean flushbyasid decodeassists pausefilter pfthreshold vgif x2avic v_spec_ctrl vnmi avx512vbmi umip pku ospke avx512_vbmi2 gfni vaes vpclmulqdq avx512_vnni avx512_bitalg avx512_vpopcntdq rdpid overflow_recov succor smca fsrm flush_l1d amd_lbr_pmc_freeze\",\"cores\":8},\"kversion\":\"Linux 7.0.14-17-pve #1 SMP PREEMPT_DYNAMIC PMX 7.0.14-17 (2026-09-10T10:16Z)\",\"wait\":0,\"boot-info\":{\"mode\":\"efi\",\"secureboot\":1},\"loadavg\":[\"0.05\",\"0.12\",\"0.10\"],\"memory\":{\"used\":2054795264,\"available\":29252005888,\"total\":31306801152,\"free\":28277043200},\"idle\":0,\"uptime\":419233,\"swap\":{\"used\":0,\"total\":8589930496,\"free\":8589930496},\"ksm\":{\"shared\":0}}}" +} \ No newline at end of file diff --git a/internal/provider/proxmox/testdata/resize_response.json b/internal/provider/proxmox/testdata/resize_response.json new file mode 100644 index 0000000..92811fc --- /dev/null +++ b/internal/provider/proxmox/testdata/resize_response.json @@ -0,0 +1,5 @@ +{ + "status": 200, + "reason": "OK", + "body": "{\"data\":\"UPID:pv1-1:0010FB74:02804090:6AB378CA:resize:199999:proxy-operator@pve!operator:\"}" +} \ No newline at end of file diff --git a/internal/provider/proxmox/testdata/resize_same_size_response.json b/internal/provider/proxmox/testdata/resize_same_size_response.json new file mode 100644 index 0000000..3e0ce7f --- /dev/null +++ b/internal/provider/proxmox/testdata/resize_same_size_response.json @@ -0,0 +1,5 @@ +{ + "status": 200, + "reason": "OK", + "body": "{\"data\":\"UPID:pv1-1:0010FB6F:02804085:6AB378CA:resize:199999:proxy-operator@pve!operator:\"}" +} \ No newline at end of file diff --git a/internal/provider/proxmox/testdata/resize_shrink_response.json b/internal/provider/proxmox/testdata/resize_shrink_response.json new file mode 100644 index 0000000..05b5156 --- /dev/null +++ b/internal/provider/proxmox/testdata/resize_shrink_response.json @@ -0,0 +1,5 @@ +{ + "status": 200, + "reason": "OK", + "body": "{\"data\":\"UPID:pv1-1:0010FB72:0280408B:6AB378CA:resize:199999:proxy-operator@pve!operator:\"}" +} \ No newline at end of file diff --git a/internal/provider/proxmox/testdata/set_config_response.json b/internal/provider/proxmox/testdata/set_config_response.json new file mode 100644 index 0000000..926d1e0 --- /dev/null +++ b/internal/provider/proxmox/testdata/set_config_response.json @@ -0,0 +1,5 @@ +{ + "status": 200, + "reason": "OK", + "body": "{\"data\":null}" +} \ No newline at end of file diff --git a/internal/provider/proxmox/testdata/set_config_vlan_response.json b/internal/provider/proxmox/testdata/set_config_vlan_response.json new file mode 100644 index 0000000..926d1e0 --- /dev/null +++ b/internal/provider/proxmox/testdata/set_config_vlan_response.json @@ -0,0 +1,5 @@ +{ + "status": 200, + "reason": "OK", + "body": "{\"data\":null}" +} \ No newline at end of file diff --git a/internal/provider/proxmox/testdata/start_response.json b/internal/provider/proxmox/testdata/start_response.json new file mode 100644 index 0000000..c065b22 --- /dev/null +++ b/internal/provider/proxmox/testdata/start_response.json @@ -0,0 +1,5 @@ +{ + "status": 200, + "reason": "OK", + "body": "{\"data\":\"UPID:pv1-1:0010FBF1:02804D0D:6AB378EA:qmstart:199999:proxy-operator@pve!operator:\"}" +} \ No newline at end of file diff --git a/internal/provider/proxmox/testdata/start_while_clone_locked_response.json b/internal/provider/proxmox/testdata/start_while_clone_locked_response.json new file mode 100644 index 0000000..a247f1f --- /dev/null +++ b/internal/provider/proxmox/testdata/start_while_clone_locked_response.json @@ -0,0 +1,5 @@ +{ + "status": 200, + "reason": "OK", + "body": "{\"data\":\"UPID:pv1-1:0010F9F0:02801DAF:6AB37871:qmstart:199999:proxy-operator@pve!operator:\"}" +} \ No newline at end of file diff --git a/internal/provider/proxmox/testdata/status_current_paused.json b/internal/provider/proxmox/testdata/status_current_paused.json new file mode 100644 index 0000000..458263d --- /dev/null +++ b/internal/provider/proxmox/testdata/status_current_paused.json @@ -0,0 +1,5 @@ +{ + "status": 200, + "reason": "OK", + "body": "{\"data\":{\"serial\":1,\"proxmox-support\":{\"pbs-dirty-bitmap-migration\":true,\"query-bitmap-info\":true,\"backup-fleecing\":true,\"pbs-masterkey\":true,\"pbs-dirty-bitmap-savevm\":true,\"backup-max-workers\":true,\"backup-access-api\":true,\"pbs-dirty-bitmap\":true,\"pbs-library-version\":\"2.0.3 (UNKNOWN)\"},\"maxdisk\":4294967296,\"vmid\":199999,\"ha\":{\"managed\":0},\"diskwrite\":373281280,\"uptime\":858,\"ballooninfo\":{\"mem_swapped_out\":0,\"mem_swapped_in\":0,\"max_mem\":1073741824,\"minor_page_faults\":266583,\"actual\":1073741824,\"last_update\":1790147647,\"major_page_faults\":599,\"free_mem\":414076928,\"total_mem\":943988736},\"running-qemu\":\"11.0.3\",\"disk\":0,\"cpus\":1,\"name\":\"proxy-reconaaaaaaaaaaa\",\"blockstat\":{\"ide2\":{\"rd_merged\":0,\"wr_total_time_ns\":0,\"failed_flush_operations\":0,\"account_failed\":true,\"account_invalid\":true,\"failed_wr_operations\":0,\"unmap_bytes\":0,\"invalid_rd_operations\":0,\"wr_highest_offset\":0,\"failed_zone_append_operations\":0,\"wr_operations\":0,\"unmap_merged\":0,\"wr_bytes\":0,\"idle_time_ns\":847849524087,\"unmap_total_time_ns\":0,\"zone_append_total_time_ns\":0,\"wr_merged\":0,\"rd_total_time_ns\":1255768,\"zone_append_bytes\":0,\"zone_append_operations\":0,\"invalid_unmap_operations\":0,\"unmap_operations\":0,\"invalid_zone_append_operations\":0,\"invalid_wr_operations\":0,\"flush_operations\":0,\"rd_operations\":61,\"invalid_flush_operations\":0,\"zone_append_merged\":0,\"failed_unmap_operations\":0,\"failed_rd_operations\":0,\"flush_total_time_ns\":0,\"rd_bytes\":174304,\"timed_stats\":[]},\"efidisk0\":{\"wr_highest_offset\":15360,\"unmap_merged\":0,\"wr_operations\":0,\"failed_zone_append_operations\":0,\"wr_bytes\":0,\"rd_merged\":0,\"failed_flush_operations\":0,\"wr_total_time_ns\":0,\"account_invalid\":true,\"account_failed\":true,\"invalid_rd_operations\":0,\"unmap_bytes\":0,\"failed_wr_operations\":0,\"invalid_wr_operations\":0,\"invalid_zone_append_operations\":0,\"flush_operations\":0,\"invalid_flush_operations\":0,\"rd_operations\":0,\"failed_rd_operations\":0,\"zone_append_merged\":0,\"failed_unmap_operations\":0,\"timed_stats\":[],\"rd_bytes\":0,\"flush_total_time_ns\":0,\"wr_merged\":0,\"zone_append_total_time_ns\":0,\"unmap_total_time_ns\":0,\"rd_total_time_ns\":0,\"zone_append_operations\":0,\"zone_append_bytes\":0,\"unmap_operations\":0,\"invalid_unmap_operations\":0},\"scsi0\":{\"unmap_operations\":81,\"invalid_unmap_operations\":0,\"zone_append_bytes\":0,\"zone_append_operations\":0,\"rd_total_time_ns\":899575874,\"wr_merged\":0,\"zone_append_total_time_ns\":0,\"idle_time_ns\":18104802903,\"unmap_total_time_ns\":326592,\"timed_stats\":[],\"rd_bytes\":196229120,\"flush_total_time_ns\":47510703,\"failed_rd_operations\":0,\"failed_unmap_operations\":0,\"zone_append_merged\":0,\"invalid_flush_operations\":0,\"rd_operations\":8843,\"invalid_wr_operations\":0,\"flush_operations\":145,\"invalid_zone_append_operations\":0,\"invalid_rd_operations\":0,\"unmap_bytes\":100188160,\"failed_wr_operations\":0,\"account_invalid\":true,\"account_failed\":true,\"failed_flush_operations\":0,\"wr_total_time_ns\":4726294270,\"rd_merged\":0,\"wr_bytes\":373281280,\"unmap_merged\":0,\"wr_operations\":1449,\"failed_zone_append_operations\":0,\"wr_highest_offset\":4294967296},\"pflash0\":{\"unmap_operations\":0,\"invalid_unmap_operations\":0,\"zone_append_operations\":0,\"zone_append_bytes\":0,\"rd_total_time_ns\":0,\"wr_merged\":0,\"zone_append_total_time_ns\":0,\"unmap_total_time_ns\":0,\"timed_stats\":[],\"rd_bytes\":0,\"flush_total_time_ns\":0,\"failed_rd_operations\":0,\"failed_unmap_operations\":0,\"zone_append_merged\":0,\"invalid_flush_operations\":0,\"rd_operations\":0,\"flush_operations\":0,\"invalid_wr_operations\":0,\"invalid_zone_append_operations\":0,\"unmap_bytes\":0,\"invalid_rd_operations\":0,\"failed_wr_operations\":0,\"account_invalid\":true,\"account_failed\":true,\"failed_flush_operations\":0,\"wr_total_time_ns\":0,\"rd_merged\":0,\"wr_bytes\":0,\"unmap_merged\":0,\"failed_zone_append_operations\":0,\"wr_operations\":0,\"wr_highest_offset\":0}},\"agent\":1,\"pid\":1113088,\"tags\":\"proxy_operator\",\"pressureiosome\":0,\"diskread\":196403424,\"pressurememoryfull\":0,\"nics\":{\"tap199999i0\":{\"netin\":30673586,\"netout\":240050}},\"qmpstatus\":\"paused\",\"status\":\"running\",\"netin\":30673586,\"balloon\":1073741824,\"netout\":240050,\"cpu\":0,\"clipboard\":null,\"pressurememorysome\":0,\"pressurecpufull\":0,\"running-machine\":\"pc-q35-11.0+pve0\",\"mem\":529911808,\"pressurecpusome\":0,\"freemem\":414076928,\"maxmem\":1073741824,\"pressureiofull\":0,\"memhost\":674607104}}" +} \ No newline at end of file diff --git a/internal/provider/proxmox/testdata/status_current_running.json b/internal/provider/proxmox/testdata/status_current_running.json new file mode 100644 index 0000000..3831ebf --- /dev/null +++ b/internal/provider/proxmox/testdata/status_current_running.json @@ -0,0 +1,5 @@ +{ + "status": 200, + "reason": "OK", + "body": "{\"data\":{\"running-machine\":\"pc-q35-11.0+pve0\",\"mem\":190930944,\"pressurecpufull\":0,\"pressurememorysome\":0,\"pressureiofull\":0,\"memhost\":337240064,\"pressurecpusome\":0,\"maxmem\":2147483648,\"freemem\":1808162816,\"pressurememoryfull\":0,\"nics\":{\"tap199999i0\":{\"netout\":0,\"netin\":255}},\"diskread\":148379648,\"clipboard\":null,\"status\":\"running\",\"qmpstatus\":\"running\",\"netin\":255,\"balloon\":2147483648,\"netout\":0,\"cpu\":0,\"agent\":1,\"pid\":1112650,\"name\":\"proxy-reconaaaaaaaaaaa\",\"blockstat\":{\"scsi0\":{\"zone_append_operations\":0,\"zone_append_bytes\":0,\"invalid_unmap_operations\":0,\"unmap_operations\":6,\"idle_time_ns\":4599584011,\"unmap_total_time_ns\":24404,\"zone_append_total_time_ns\":0,\"wr_merged\":0,\"rd_total_time_ns\":466322772,\"zone_append_merged\":0,\"failed_unmap_operations\":0,\"failed_rd_operations\":0,\"flush_total_time_ns\":4612531,\"rd_bytes\":148379648,\"timed_stats\":[],\"flush_operations\":19,\"invalid_wr_operations\":0,\"invalid_zone_append_operations\":0,\"rd_operations\":4279,\"invalid_flush_operations\":0,\"account_failed\":true,\"account_invalid\":true,\"failed_wr_operations\":0,\"invalid_rd_operations\":0,\"unmap_bytes\":282624,\"rd_merged\":0,\"wr_total_time_ns\":190079426,\"failed_flush_operations\":0,\"wr_bytes\":6373888,\"wr_highest_offset\":1533730816,\"failed_zone_append_operations\":0,\"wr_operations\":220,\"unmap_merged\":0},\"efidisk0\":{\"rd_merged\":0,\"failed_flush_operations\":0,\"wr_total_time_ns\":0,\"account_invalid\":true,\"account_failed\":true,\"unmap_bytes\":0,\"invalid_rd_operations\":0,\"failed_wr_operations\":0,\"wr_highest_offset\":9216,\"unmap_merged\":0,\"failed_zone_append_operations\":0,\"wr_operations\":0,\"wr_bytes\":0,\"wr_merged\":0,\"zone_append_total_time_ns\":0,\"unmap_total_time_ns\":0,\"rd_total_time_ns\":0,\"zone_append_bytes\":0,\"zone_append_operations\":0,\"unmap_operations\":0,\"invalid_unmap_operations\":0,\"invalid_wr_operations\":0,\"invalid_zone_append_operations\":0,\"flush_operations\":0,\"invalid_flush_operations\":0,\"rd_operations\":0,\"failed_rd_operations\":0,\"zone_append_merged\":0,\"failed_unmap_operations\":0,\"timed_stats\":[],\"rd_bytes\":0,\"flush_total_time_ns\":0},\"pflash0\":{\"wr_total_time_ns\":0,\"failed_flush_operations\":0,\"rd_merged\":0,\"failed_wr_operations\":0,\"unmap_bytes\":0,\"invalid_rd_operations\":0,\"account_failed\":true,\"account_invalid\":true,\"wr_operations\":0,\"failed_zone_append_operations\":0,\"unmap_merged\":0,\"wr_highest_offset\":0,\"wr_bytes\":0,\"rd_total_time_ns\":0,\"unmap_total_time_ns\":0,\"zone_append_total_time_ns\":0,\"wr_merged\":0,\"invalid_unmap_operations\":0,\"unmap_operations\":0,\"zone_append_operations\":0,\"zone_append_bytes\":0,\"rd_operations\":0,\"invalid_flush_operations\":0,\"invalid_zone_append_operations\":0,\"invalid_wr_operations\":0,\"flush_operations\":0,\"flush_total_time_ns\":0,\"rd_bytes\":0,\"timed_stats\":[],\"failed_unmap_operations\":0,\"zone_append_merged\":0,\"failed_rd_operations\":0}},\"pressureiosome\":0.01,\"vmid\":199999,\"ha\":{\"managed\":0},\"diskwrite\":6373888,\"serial\":1,\"proxmox-support\":{\"pbs-dirty-bitmap-savevm\":true,\"pbs-masterkey\":true,\"backup-fleecing\":true,\"pbs-dirty-bitmap-migration\":true,\"query-bitmap-info\":true,\"backup-access-api\":true,\"pbs-dirty-bitmap\":true,\"pbs-library-version\":\"2.0.3 (UNKNOWN)\",\"backup-max-workers\":true},\"maxdisk\":3221225472,\"ballooninfo\":{\"actual\":2147483648,\"mem_swapped_out\":0,\"mem_swapped_in\":0,\"max_mem\":2147483648,\"minor_page_faults\":122998,\"major_page_faults\":616,\"free_mem\":1808162816,\"total_mem\":1999093760,\"last_update\":1790146701},\"uptime\":26,\"running-qemu\":\"11.0.3\",\"disk\":0,\"cpus\":2}}" +} \ No newline at end of file diff --git a/internal/provider/proxmox/testdata/status_current_stopped.json b/internal/provider/proxmox/testdata/status_current_stopped.json new file mode 100644 index 0000000..75ab196 --- /dev/null +++ b/internal/provider/proxmox/testdata/status_current_stopped.json @@ -0,0 +1,5 @@ +{ + "status": 200, + "reason": "OK", + "body": "{\"data\":{\"name\":\"ubuntu-vm-1\",\"mem\":0,\"maxmem\":2147483648,\"memhost\":0,\"maxdisk\":34359738368,\"serial\":1,\"ha\":{\"managed\":0},\"vmid\":100,\"cpu\":0,\"cpus\":2,\"netout\":0,\"disk\":0,\"qmpstatus\":\"stopped\",\"status\":\"stopped\",\"netin\":0,\"uptime\":0,\"clipboard\":null}}" +} \ No newline at end of file diff --git a/internal/provider/proxmox/testdata/stop_response.json b/internal/provider/proxmox/testdata/stop_response.json new file mode 100644 index 0000000..adf6b22 --- /dev/null +++ b/internal/provider/proxmox/testdata/stop_response.json @@ -0,0 +1,5 @@ +{ + "status": 200, + "reason": "OK", + "body": "{\"data\":\"UPID:pv1-1:0010FAC5:02802A52:6AB37891:qmstop:199999:proxy-operator@pve!operator:\"}" +} \ No newline at end of file diff --git a/internal/provider/proxmox/testdata/storage_content_iso.json b/internal/provider/proxmox/testdata/storage_content_iso.json new file mode 100644 index 0000000..ae47169 --- /dev/null +++ b/internal/provider/proxmox/testdata/storage_content_iso.json @@ -0,0 +1,5 @@ +{ + "status": 200, + "reason": "OK", + "body": "{\"data\":[{\"size\":45056,\"volid\":\"local:iso/proxy-reconaaaaaaaaaaa-seed.iso\",\"format\":\"iso\",\"content\":\"iso\",\"ctime\":1790146742}]}" +} \ No newline at end of file diff --git a/internal/provider/proxmox/testdata/storage_status.json b/internal/provider/proxmox/testdata/storage_status.json new file mode 100644 index 0000000..40bff44 --- /dev/null +++ b/internal/provider/proxmox/testdata/storage_status.json @@ -0,0 +1,5 @@ +{ + "status": 200, + "reason": "OK", + "body": "{\"data\":{\"total\":100861726720,\"active\":1,\"enabled\":1,\"used\":6973128704,\"avail\":88717860864,\"type\":\"dir\",\"shared\":0,\"content\":\"iso,import,backup,vztmpl\"}}" +} \ No newline at end of file diff --git a/internal/provider/proxmox/testdata/task_status_clone.json b/internal/provider/proxmox/testdata/task_status_clone.json new file mode 100644 index 0000000..576a28c --- /dev/null +++ b/internal/provider/proxmox/testdata/task_status_clone.json @@ -0,0 +1,5 @@ +{ + "status": 200, + "reason": "OK", + "body": "{\"data\":{\"upid\":\"UPID:pv1-1:0010F9DC:02801DA1:6AB37871:qmclone:9001:proxy-operator@pve!operator:\",\"exitstatus\":\"OK\",\"status\":\"stopped\",\"pid\":1112540,\"pstart\":41950625,\"user\":\"proxy-operator@pve\",\"node\":\"pv1-1\",\"type\":\"qmclone\",\"starttime\":1790146673,\"id\":\"9001\",\"tokenid\":\"operator\"}}" +} \ No newline at end of file diff --git a/internal/provider/proxmox/testdata/task_status_running.json b/internal/provider/proxmox/testdata/task_status_running.json new file mode 100644 index 0000000..eb497ca --- /dev/null +++ b/internal/provider/proxmox/testdata/task_status_running.json @@ -0,0 +1,5 @@ +{ + "status": 200, + "reason": "OK", + "body": "{\"data\":{\"pstart\":41962765,\"upid\":\"UPID:pv1-1:0010FBF1:02804D0D:6AB378EA:qmstart:199999:proxy-operator@pve!operator:\",\"pid\":1113073,\"node\":\"pv1-1\",\"user\":\"proxy-operator@pve\",\"type\":\"qmstart\",\"status\":\"running\",\"tokenid\":\"operator\",\"id\":\"199999\",\"starttime\":1790146794}}" +} \ No newline at end of file diff --git a/internal/provider/proxmox/testdata/task_status_start_waited_for_lock_ok.json b/internal/provider/proxmox/testdata/task_status_start_waited_for_lock_ok.json new file mode 100644 index 0000000..fa45604 --- /dev/null +++ b/internal/provider/proxmox/testdata/task_status_start_waited_for_lock_ok.json @@ -0,0 +1,5 @@ +{ + "status": 200, + "reason": "OK", + "body": "{\"data\":{\"exitstatus\":\"OK\",\"status\":\"stopped\",\"upid\":\"UPID:pv1-1:0010F9F0:02801DAF:6AB37871:qmstart:199999:proxy-operator@pve!operator:\",\"type\":\"qmstart\",\"tokenid\":\"operator\",\"id\":\"199999\",\"starttime\":1790146673,\"pstart\":41950639,\"pid\":1112560,\"node\":\"pv1-1\",\"user\":\"proxy-operator@pve\"}}" +} \ No newline at end of file diff --git a/internal/provider/proxmox/testdata/task_status_stopped_error.json b/internal/provider/proxmox/testdata/task_status_stopped_error.json new file mode 100644 index 0000000..fd2ebf6 --- /dev/null +++ b/internal/provider/proxmox/testdata/task_status_stopped_error.json @@ -0,0 +1,5 @@ +{ + "status": 200, + "reason": "OK", + "body": "{\"data\":{\"type\":\"resize\",\"tokenid\":\"operator\",\"id\":\"199999\",\"starttime\":1790146762,\"pstart\":41959563,\"pid\":1112946,\"user\":\"proxy-operator@pve\",\"node\":\"pv1-1\",\"exitstatus\":\"shrinking disks is not supported\",\"status\":\"stopped\",\"upid\":\"UPID:pv1-1:0010FB72:0280408B:6AB378CA:resize:199999:proxy-operator@pve!operator:\"}}" +} \ No newline at end of file diff --git a/internal/provider/proxmox/testdata/task_status_stopped_ok.json b/internal/provider/proxmox/testdata/task_status_stopped_ok.json new file mode 100644 index 0000000..cbfc5a0 --- /dev/null +++ b/internal/provider/proxmox/testdata/task_status_stopped_ok.json @@ -0,0 +1,5 @@ +{ + "status": 200, + "reason": "OK", + "body": "{\"data\":{\"exitstatus\":\"OK\",\"status\":\"stopped\",\"upid\":\"UPID:pv1-1:0010FAC5:02802A52:6AB37891:qmstop:199999:proxy-operator@pve!operator:\",\"type\":\"qmstop\",\"starttime\":1790146705,\"id\":\"199999\",\"tokenid\":\"operator\",\"pid\":1112773,\"pstart\":41953874,\"user\":\"proxy-operator@pve\",\"node\":\"pv1-1\"}}" +} \ No newline at end of file diff --git a/internal/provider/proxmox/testdata/task_status_volume_missing.json b/internal/provider/proxmox/testdata/task_status_volume_missing.json new file mode 100644 index 0000000..e39b846 --- /dev/null +++ b/internal/provider/proxmox/testdata/task_status_volume_missing.json @@ -0,0 +1,5 @@ +{ + "status": 200, + "reason": "OK", + "body": "{\"data\":{\"user\":\"proxy-operator@pve\",\"node\":\"pv1-1\",\"pstart\":42286690,\"pid\":1122295,\"id\":\"proxy-seeds\",\"tokenid\":\"operator\",\"starttime\":1790150033,\"type\":\"imgdel\",\"upid\":\"UPID:pv1-1:00111FF7:02853E62:6AB38591:imgdel:proxy-seeds:proxy-operator@pve!operator:\",\"status\":\"stopped\",\"exitstatus\":\"OK\"}}" +} \ No newline at end of file diff --git a/internal/provider/proxmox/testdata/upload_overwrite_response.json b/internal/provider/proxmox/testdata/upload_overwrite_response.json new file mode 100644 index 0000000..30726ac --- /dev/null +++ b/internal/provider/proxmox/testdata/upload_overwrite_response.json @@ -0,0 +1,5 @@ +{ + "status": 200, + "reason": "OK", + "body": "{\"data\":\"UPID:pv1-1:0010FB2D:028038B0:6AB378B6:imgcopy::proxy-operator@pve!operator:\"}" +} \ No newline at end of file diff --git a/internal/provider/proxmox/testdata/upload_response.json b/internal/provider/proxmox/testdata/upload_response.json new file mode 100644 index 0000000..68c8a1f --- /dev/null +++ b/internal/provider/proxmox/testdata/upload_response.json @@ -0,0 +1,5 @@ +{ + "status": 200, + "reason": "OK", + "body": "{\"data\":\"UPID:pv1-1:0010FB27:0280377E:6AB378B3:imgcopy::proxy-operator@pve!operator:\"}" +} \ No newline at end of file diff --git a/internal/provider/proxmox/testdata/upload_response_proxy_seeds.json b/internal/provider/proxmox/testdata/upload_response_proxy_seeds.json new file mode 100644 index 0000000..9e44092 --- /dev/null +++ b/internal/provider/proxmox/testdata/upload_response_proxy_seeds.json @@ -0,0 +1,5 @@ +{ + "status": 200, + "reason": "OK", + "body": "{\"data\":\"UPID:pv1-1:00110D02:0282B09D:6AB37F08:imgcopy::proxy-operator@pve!operator:\"}" +} \ No newline at end of file diff --git a/internal/provider/proxmox/testdata/version.json b/internal/provider/proxmox/testdata/version.json new file mode 100644 index 0000000..5361259 --- /dev/null +++ b/internal/provider/proxmox/testdata/version.json @@ -0,0 +1,5 @@ +{ + "status": 200, + "reason": "OK", + "body": "{\"data\":{\"version\":\"9.2.20\",\"repoid\":\"49318c671b82f31e\",\"release\":\"9.2\"}}" +} \ No newline at end of file diff --git a/internal/provider/proxmox/testdata/vm_config_clone.json b/internal/provider/proxmox/testdata/vm_config_clone.json new file mode 100644 index 0000000..293844e --- /dev/null +++ b/internal/provider/proxmox/testdata/vm_config_clone.json @@ -0,0 +1,5 @@ +{ + "status": 200, + "reason": "OK", + "body": "{\"data\":{\"boot\":\"order=scsi0\",\"vmgenid\":\"2dd339a4-d7fb-4d47-96fd-5a61a25f808d\",\"digest\":\"e747d94a5aac26cf6528b43835600b1bc4e82dda\",\"scsi0\":\"local-lvm:vm-199999-disk-1,size=3G\",\"serial0\":\"socket\",\"name\":\"proxy-reconaaaaaaaaaaa\",\"vga\":\"serial0\",\"cores\":2,\"meta\":\"creation-qemu=11.0.3,ctime=1790145676\",\"description\":\"proxy-operator-managed=true\\nproxy-operator-uid=00000000-0000-0000-0000-00000000recon\\nproxy-operator-created-at=2026-09-23T07:00:00Z\",\"efidisk0\":\"local-lvm:vm-199999-disk-0,efitype=4m,pre-enrolled-keys=0,size=4M\",\"memory\":\"2048\",\"scsihw\":\"virtio-scsi-pci\",\"net0\":\"virtio=BC:24:11:48:11:46,bridge=vmbr0\",\"agent\":\"1\",\"ostype\":\"l26\",\"bios\":\"ovmf\",\"machine\":\"q35\",\"smbios1\":\"uuid=f2ade202-ffdb-4a89-9867-3d419384471d\"}}" +} \ No newline at end of file diff --git a/internal/provider/proxmox/testdata/vm_config_template.json b/internal/provider/proxmox/testdata/vm_config_template.json new file mode 100644 index 0000000..fa291b1 --- /dev/null +++ b/internal/provider/proxmox/testdata/vm_config_template.json @@ -0,0 +1,5 @@ +{ + "status": 200, + "reason": "OK", + "body": "{\"data\":{\"ostype\":\"l26\",\"agent\":\"1\",\"smbios1\":\"uuid=5b131064-9841-4bc7-982e-5fb5b2e677eb\",\"machine\":\"q35\",\"bios\":\"ovmf\",\"scsi0\":\"local-lvm:base-9001-disk-2,size=3G\",\"digest\":\"03bdc5d457ce2165bb633505d2ff0cd94340dbc2\",\"vmgenid\":\"9a42b153-7965-4601-864d-59fc2d71ecd4\",\"boot\":\"order=scsi0\",\"vga\":\"serial0\",\"template\":1,\"name\":\"debian-13-template\",\"serial0\":\"socket\",\"meta\":\"creation-qemu=11.0.3,ctime=1790145676\",\"cores\":2,\"net0\":\"virtio=BC:24:11:93:25:82,bridge=vmbr0\",\"scsihw\":\"virtio-scsi-pci\",\"memory\":\"2048\",\"efidisk0\":\"local-lvm:base-9001-disk-0,efitype=4m,pre-enrolled-keys=0,size=4M\"}}" +} \ No newline at end of file diff --git a/internal/provider/proxmox/testdata/vm_config_template_ubuntu.json b/internal/provider/proxmox/testdata/vm_config_template_ubuntu.json new file mode 100644 index 0000000..321fdb4 --- /dev/null +++ b/internal/provider/proxmox/testdata/vm_config_template_ubuntu.json @@ -0,0 +1,5 @@ +{ + "status": 200, + "reason": "OK", + "body": "{\"data\":{\"ciuser\":\"ubuntu\",\"smbios1\":\"uuid=6ba36ff6-9d1f-457b-8cdc-a44448f5bd35\",\"ostype\":\"l26\",\"memory\":\"2048\",\"scsihw\":\"virtio-scsi-pci\",\"net0\":\"virtio=BC:24:11:2F:97:A0,bridge=vmbr0,tag=4\",\"ide2\":\"local-lvm:vm-9000-cloudinit,media=cdrom\",\"cores\":2,\"meta\":\"creation-qemu=11.0.0,ctime=1789719724\",\"ipconfig0\":\"ip=dhcp\",\"serial0\":\"socket\",\"name\":\"ubuntu-2604-template\",\"template\":1,\"vga\":\"serial0\",\"boot\":\"order=scsi0\",\"vmgenid\":\"37d16d5c-957f-4c83-aab3-b403513e466c\",\"digest\":\"8820e9bb34a576d0f54b94cfe3e69fbfdf13a8ce\",\"scsi0\":\"local-lvm:base-9000-disk-0,size=32G\"}}" +} \ No newline at end of file From 4e974f30f6e52d2ff5e4f6796e6e239cac8c098d Mon Sep 17 00:00:00 2001 From: kacerr Date: Wed, 23 Sep 2026 09:18:29 +0000 Subject: [PATCH 04/13] docs: proxmox provider spec Phase 1 of the proxmox provider plan: docs/proxmox-provider-spec.md answers all 24 entries of the provider spec template against the contract, citing the Phase 0 facts from pv1-1 as evidence. It folds in the ten corrections recon found. The two that matter most: Create inspects a taken VMID, waiting out a clone lock, before probing past it, so a crash-retry adopts its own half-cloned VM instead of cloning a duplicate; and Get checks the lock before the name, because a VM being cloned has no name yet. The plan now opens with a note that the spec wins where they disagree. Co-Authored-By: Claude Opus 5.5 (1M context) --- .../2026-09-18-1502-proxmox-provider.md | 51 +- .../plans/2026-09-18-1502-proxmox-provider.md | 9 + docs/proxmox-provider-spec.md | 856 ++++++++++++++++++ 3 files changed, 915 insertions(+), 1 deletion(-) create mode 100644 docs/proxmox-provider-spec.md diff --git a/docs/plans-executions/2026-09-18-1502-proxmox-provider.md b/docs/plans-executions/2026-09-18-1502-proxmox-provider.md index 058a1c6..ec9cc10 100644 --- a/docs/plans-executions/2026-09-18-1502-proxmox-provider.md +++ b/docs/plans-executions/2026-09-18-1502-proxmox-provider.md @@ -5,7 +5,7 @@ Pairs with [docs/plans/2026-09-18-1502-proxmox-provider.md](../plans/2026-09-18- ## Status - [x] Phase 0 — Plan file, branch, host recon, fixtures -- [ ] Phase 1 — The spec (`docs/proxmox-provider-spec.md`) +- [x] Phase 1 — The spec (`docs/proxmox-provider-spec.md`) - [ ] Phase 2 — Config block, registration, package skeleton - [ ] Phase 3 — Extract `nocloud` and `keyedlock` from kvm2 - [ ] Phase 4 — Seam, HTTP client, decorator, fake @@ -280,3 +280,52 @@ volumes of 199999. The capture is `agent_network_get_interfaces_vlan.json`. - Time to IP was about 50 s on both networks, with the agent installed at first boot. Budgets in the plan (running wait 60 s for the VM state, the agent discovered by `Get` polling) are consistent with this. + +## Phase 1 — The spec + +Wrote [docs/proxmox-provider-spec.md](../proxmox-provider-spec.md), all 24 +entries of the template, from the plan, the contract and the Phase 0 facts. +The ten Phase 0 corrections are folded in, and the plan now opens with a +note that the spec wins where the two disagree. + +Judgment calls made here, each recorded in its entry: + +- **Create's allocation loop** (entry 2) runs an ownership check on any + taken VMID: wait out a lock or a nameless entry, then adopt when the name + and UID are ours, skip when the name differs, refuse as Permanent when the + name is ours but the mark is not. This is the fix for the duplicate-VM bug. +- **Get's check order** (entry 9): lock, then name and mark from the config, + then `status/current`, then the agent. `lock: suspended` alone maps to + Stopped, since a hibernated VM never resumes by itself. +- **Agent messages** (entry 7): `QEMU guest agent is not running` is + Provisioning, `VM is not running` is Transient, and + `No QEMU guest agent configured` is **Stopped**, so the reconciler replaces + a VM that can never report an address instead of latching the Proxy. +- **Templates are checked before cloning** (entry 4): not a template, or a + Proxmox cloud-init drive in any slot, is Permanent. This makes the "no + cloud-init drive" prerequisite enforceable instead of a silent failure. +- **`isoStorage` is required**, with no default (entry 15), because the token + must hold `Datastore.Allocate` on it. +- **TLS verification failures are Permanent**, with a Checker reason + `CertificateInvalid` (entries 13, 16), since the fix is the URL or CA file. +- **The CA is loaded lazily and cached**; the token is read per request + (entry 14). + +Checks, as the plan's Verify clause asks: + +```bash +grep -c '^## [0-9]*\. ' docs/proxmox-provider-spec.md # 24 +grep -n '^\*\*Answer:\*\*$' docs/proxmox-provider-spec.md # nothing +grep -c '—' docs/proxmox-provider-spec.md # 0 +``` + +plus a script comparing every `` ``: `` `` pair against the +contract's §4.20 table (no mismatches, every axis answered) and a list of +the F numbers cited (all 23). + +Worth noting: the first pass stated two things as fact that recon could not +show and was corrected before commit. That an unreachable node's guests stay +listed as `unknown` is Proxmox's documented design, unverifiable on a single +node; entry 13 now says so. And the matcher for `VM is not running` +is a whole-message regexp, because ` is not running` alone also matches +`QEMU guest agent is not running`. diff --git a/docs/plans/2026-09-18-1502-proxmox-provider.md b/docs/plans/2026-09-18-1502-proxmox-provider.md index d7c42bd..b0e9704 100644 --- a/docs/plans/2026-09-18-1502-proxmox-provider.md +++ b/docs/plans/2026-09-18-1502-proxmox-provider.md @@ -4,6 +4,15 @@ ## Context +> **Superseded in places by the spec.** Phase 0 recon found ten corrections +> to the design below (listed in the execution log), and Phase 1 folded them +> into [docs/proxmox-provider-spec.md](../proxmox-provider-spec.md). Where a +> phase step here disagrees with the spec (the Create allocation loop, Get's +> check order, resize and seed delete as tasks, no delete before upload, +> `isoStorage` required, `vlanTag`, the agent messages), **the spec wins**. +> Test names in the phases below remain the minimum; spec entry 21 adds the +> ones for the corrections. + The provider contract (`docs/provider-contract.md`), the spec template (`docs/provider-spec-template.md`) and the kvm2 build proved the workflow: spec first, then implementation against the contract, one commit per step. diff --git a/docs/proxmox-provider-spec.md b/docs/proxmox-provider-spec.md new file mode 100644 index 0000000..8b2bd2f --- /dev/null +++ b/docs/proxmox-provider-spec.md @@ -0,0 +1,856 @@ +# Provider spec: proxmox + +Filled from [provider-spec-template.md](provider-spec-template.md) against +[provider-contract.md](provider-contract.md). This is the design the +`proxmox` package is built from. Where an entry names an axis in backticks, +the answer is one of that axis's profiles in contract §4. + +Backend facts come from the Phase 0 recon on the user's host, recorded in +[the execution log](plans-executions/2026-09-18-1502-proxmox-provider.md) as +facts F1 to F23, with the raw responses under +`internal/provider/proxmox/testdata/`. An Evidence line citing `F` points +there. Where this spec and the plan +([docs/plans/2026-09-18-1502-proxmox-provider.md](plans/2026-09-18-1502-proxmox-provider.md)) +disagree, this spec wins: it folds in the ten corrections Phase 0 found. + +## Header + +- **Type**: `proxmox` +- **Example configured name**: `proxmox-home` +- **Backend and API version this spec was written against**: Proxmox VE + 9.2.20 (`pve-manager/9.2.20/49318c671b82f31e`, kernel `7.0.14-17-pve`, + QEMU 11.0.3) on the single node `pv1-1`, API at + `https://192.168.0.8:8006/api2/json`. +- **Client**: `thin-client`: `net/http` and `encoding/json` from the standard + library, with the transport wrapped by the already-required + `go.opentelemetry.io/contrib/instrumentation/net/http/otelhttp`. No new + dependency. The seed ISO uses `github.com/kdomanski/iso9660`, already + required by kvm2. +- **Author and date**: Claude (orchestrator session), 2026-09-23. + +## 1. Identity and providerID + +**Answer:** `vms//`, where `` is the VM's numeric id and +`` is `req.Name`. The ID is location-free: the node is looked up in +`GET /cluster/resources?type=vm` on every call, so a VM migrated between +nodes stays addressable. Parsing is strict: exactly three segments, the +literal `vms`, a decimal VMID without leading zeros between 100 and +999999999, and a non-empty name without `/`. Anything else is `ErrPermanent` +from `Get` and `Delete`. The name is carried so that `Get` and `Delete` can +tell our VM from another VM that later received the same VMID (entry 2). No +config value is pasted into the ID, so no ID-safety regexp is needed; node +names and storage ids are still validated at load (entry 15) because they +end up in request paths. + +**Evidence:** contract §6 and §4.10 (the ID must not depend on the location +when instances can migrate); F8 (`node` is a field of every resource entry). + +## 2. Primary key (`primary-key`) + +**Answer:** `allocated-id`. Proxmox addresses VMs by VMID; `name` is only a +label and is not unique. The VMID is derived from the Proxy UID so a retry +lands on the same number: `vmidFor(uid, attempt)` is +`min + (fnv1a64(uid) + attempt) mod (max - min + 1)` over `vmidRange` +(entry 15). Create, under the per-name lock (entry 19): + +1. Reads `/cluster/resources` once. Entries with `type: qemu`, `template: 0` + and `name == req.Name` are the by-name candidates. Two or more is + `ErrPermanent` naming every VMID: an admin must remove the duplicate. + Exactly one goes to the ownership check below and is adopted or refused. +2. With none, probes `attempt = 0, 1, …` up to 32: + - VMID free in the listing: clone onto it. If the clone fails with + `unable to create VM : config file already exists` (F4), someone + took it between the listing and the clone; run the ownership check on + it. + - VMID taken: run the ownership check. + - Exhausted: `ErrTransient` naming the range. +3. The **ownership check** on a VMID: if its resource entry carries a + `lock` or has no `name` (a clone in progress lists neither, F8), wait + until it is unlocked (entry 10, clone budget). Then read its config. It + is **ours** when `name == req.Name` and the description carries the mark + with `uid` equal to `req.UID` or empty: adopt it and resume Create at the + seed step (entry 10). It is **foreign** when the name differs: move to + the next attempt. The same name with no mark, or with a different + non-empty `uid`, is `ErrPermanent` naming the VMID and both UIDs; the + operator never takes over a VM it did not create. + +The wait in step 3 is what closes the duplicate-VM bug Phase 0 found: after +a crash mid-clone, the retry's by-name lookup misses the nameless VM, but +attempt 0 lands on the same VMID, waits for the clone to finish, and adopts +it instead of probing past it. + +`Get` and `Delete` check the name and mark at the VMID in the ID. A VM there +with another name means ours is gone and the VMID was reused: `Get` answers +`ErrNotFound`, `Delete` succeeds without touching it. + +**Evidence:** F4 (taken VMID fails synchronously); F8 (a mid-clone entry has +no name); contract §4.10 (allocate-then-verify, deterministic from the UID) +and §13 (adoption not verified: closed here). + +## 3. Placement mapping + +**Answer:** three fields are catalogue keys checked before any backend call, +one is ignored: + +| Field | Meaning | Required | Unknown value | +|---|---|---|---| +| `Zone` | a node from `nodes` | only when more than one node is configured; with one node an empty zone selects it | `ErrPermanent`: `placement.zone "x" is not in proxmox.nodes` | +| `MachineType` | a key of `machineTypes` (`cores`, `memoryMiB`, optional `diskGiB`) | yes | `ErrPermanent`: `placement.machineType "x" is not in proxmox.machineTypes` | +| `Image` | a key of `templates` (a template VMID) | yes | `ErrPermanent`: `placement.image "x" is not in proxmox.templates` | +| `Region` | ignored | | still part of the spec hash: changing it replaces the VM | + +The guide documents the ignored field and that catalogue edits affect only +new VMs. + +**Evidence:** contract §2.1 (reject early, MAY ignore with documentation). + +## 4. Image source + +**Answer:** a template VM prepared by the admin, referenced by a catalogue +key: `templates..vmid`. Create makes a **full clone** (`full=1`), so a +VM owns copies of its disks and keeps nothing linked to the template: the +template may be edited, replaced or deleted without affecting running VMs. +Before cloning, Create reads the template's config and refuses it with +`ErrPermanent` when it is not a template (`template` is not 1) or when any +disk slot holds a Proxmox cloud-init drive (a value containing `cloudinit`), +which would compete with the provider's seed (entry 5). The template must +boot from its first disk, run cloud-init with the NoCloud datasource, and +either ship `qemu-guest-agent` or be able to install it (entry 7). + +Clone parameters: `newid`, `name`, `description` (the mark, entry 8), +`full=1`, `storage` when configured (empty clones onto the template's own +disk storage), and `target` when the chosen node differs from the template's +node. A cross-node clone needs the template on shared storage; this host has +one node, so that path is untested (entry 24). + +Catalogue edit warning for the guide: changing a key's `vmid` changes only +VMs created afterwards. Add a new key rather than repointing one in use, so +the Proxies that name it keep a meaning you can reason about. + +**Evidence:** F19 (template 9001: `scsi0`, `boot: order=scsi0`, `ide2` +free, `agent: 1`; template 9000 carries a cloud-init drive on `ide2`); F15 +(a clone onto a storage without image support is refused up front); F16 (a +full clone of 9001 took about two seconds). + +## 5. Cloud-init delivery + +**Answer:** a NoCloud seed ISO built in-process by the shared +`internal/provider/nocloud` package (extracted from kvm2 in Phase 3): +volume label `cidata`, files `user-data` (the Proxy's `CloudInit`, byte for +byte, never parsed), `meta-data` (`instance-id: ` and +`local-hostname: `) and `vendor-data` (the guest agent script, joined +with the configured `vendorData` as a MIME multipart when both are set; +identical to kvm2's rules). The builder stages under `os.TempDir()` (`/tmp`, +the manager's emptyDir) and writes into a `bytes.Buffer`; a seed is tens of +KiB, so buffering is within contract §1.4. + +The ISO is uploaded with `POST /nodes/{node}/storage/{isoStorage}/upload` +as multipart (`content=iso`, file part `filename` named +`-seed.iso`) and the returned `imgcopy` task is waited on. An upload +**overwrites** an existing file of the same name, so no delete precedes it; +the seed is rebuilt and re-uploaded on every Create attempt, since a +previous attempt may have been cut off. It is attached as +`ide2: :iso/-seed.iso,media=cdrom`; Proxmox then appends +`ide2` to the boot order on its own, which is harmless because the boot +disk stays first. + +Only the Proxmox API is used: the storage upload endpoint accepts `iso`, +`vztmpl` or `import`, never `snippets`, so `cicustom` would need shell +access to the node. Size: the request timeout (entry 18) bounds the upload; +a seed with a multi-megabyte `vendorData` would need a larger +`requestTimeoutSeconds`. + +**Evidence:** F12 (upload fields, UPID, overwrite, `.iso` suffix required); +F19 (`ide2` appended to the boot order); F21 (the seed on `ide2` installed +the agent on both the untagged network and VLAN 4); F23 (the upload is +staged in `/var/tmp` on the node, not the tmpfs `/tmp`); contract §2.1 and +§11 (byte for byte, sentinel test). + +## 6. Machine type mapping + +**Answer:** a catalogue: `machineTypes.` is `{cores, memoryMiB, +diskGiB}`. `cores` and `memoryMiB` are written in the config step (entry +10). `diskGiB` is optional: when set and larger than the cloned boot disk, +the disk is grown with `PUT /nodes/{node}/qemu/{vmid}/resize` to the +absolute size `G`, and the returned task is waited on; when unset +or not larger, the disk keeps the template's size and no call is made. The +provider never shrinks a disk. Cloud-init's growpart expands the root +filesystem at first boot. + +The boot disk is the first entry of the config's `boot: order=…` that is a +disk key (`scsiN`, `virtioN`, `sataN`, `ideN`) and not a CD-ROM, else the +`bootdisk` key, else the first present of `scsi0`, `virtio0`, `sata0`, +`ide0`. Its current size comes from the `size=` option of that disk's value. +No boot disk, when `diskGiB` is set, is `ErrPermanent`. + +**Evidence:** F20 (resize is a task; the same size is an `OK` no-op; a +shrink fails in the task with `shrinking disks is not supported`); F19 +(`boot: order=scsi0`, becoming `order=scsi0;ide2` once the seed is +attached). + +## 7. Network and IP discovery (`readiness`, `ip-source`) + +**Answer:** `readiness`: `provisioning-window`. `ip-source`: `discovered`, +from the QEMU guest agent only (`GET /nodes/{node}/qemu/{vmid}/agent/network-get-interfaces`); +the API offers no ARP or DHCP-lease source. The address reported is the first +IPv4 of the first non-loopback interface, in the agent's order, that is not +loopback, link-local or unspecified. + +The NIC is the template's `net0`, rewritten in the config step only when the +config sets `bridge` or `vlanTag`: the value's comma-separated options are +kept in order with the model and MAC untouched, `bridge=` and `tag=` are +replaced or appended. With neither set, `net0` is left as cloned. + +The agent answers: + +| Agent response | Meaning | Get reports | +|---|---|---| +| 200 with a routable IPv4 | booted | Running with that IP | +| 200 without one | network still coming up | Provisioning | +| 500 `QEMU guest agent is not running` | booting, or the agent install has not finished or failed | Provisioning | +| 500 `VM is not running` | the VM stopped between the status read and this call | `ErrTransient` | +| 500 `No QEMU guest agent configured` | `agent: 1` was removed from the config; this VM can never report an address | Stopped, so the reconciler replaces it with a VM that has the agent on | +| anything else | cannot tell | `ErrTransient` | + +The guest needs the agent. The vendor-data script installs it at first boot +unless `installGuestAgent: false`, and that install needs a package mirror: +**the proxy network must have a route to the internet**, which an egress +proxy needs anyway. Baking `qemu-guest-agent` into the template and setting +`installGuestAgent: false` removes the mirror from IP discovery. `Port` is +not used by the provider. Reachability is a prerequisite in the guide: the +operator's pod network and the clients must route to the proxy network (on +the user's host, VLAN 4, `192.168.4.0/24`). + +**Evidence:** F5 (all three agent messages); F21 (agent response shape; an +address about 50 seconds after start on the untagged network and on VLAN 4); +the Phase 0 log's VLAN 4 section (no internet route meant no agent); +contract §1.3 and §2.2 (Running only with an IP). + +## 8. GC marks and CreatedAt (`gc-mark`, `created-at`) + +**Answer:** `gc-mark`: `written-metadata`. `created-at`: `written-at-create`. +The mark is three lines in the VM's `description`, passed in the clone call +itself, so there is no mark window: + +``` +proxy-operator-managed=true +proxy-operator-uid= +proxy-operator-created-at= +``` + +The keys are `provider.LabelManaged`, `provider.LabelUID` and +`proxy-operator-created-at`; the managed value is `provider.LabelManagedYes`. +The parser reads `key=value` lines, tolerates CRLF, blank lines and any +other lines an admin adds, and treats the VM as managed only when the +managed value is exactly `true`. A missing `uid` line gives an empty UID. A +`created-at` that does not parse gives the zero time, which GC treats as +old; that is accepted. The clone's own `meta: …,ctime=…` is not usable: +Proxmox copies it from the template. + +The config step also sets the tag `proxy_operator`, for filtering in the +Proxmox UI and as a ListByTag candidate hint (entry 12). It is not the mark: +tags cannot be passed to the clone call. Hyphens are accepted in tags on +9.2.20, but the underscore form is kept because it is valid under every +tag rule Proxmox has had. Uppercase is folded to lowercase by Proxmox. + +The seed ISO carries no mark; its name `-seed.iso` identifies it +(entry 11). + +**Evidence:** F10 (the description survives the clone verbatim, newlines +included); F11 (the clone's `meta` is the template's); F9 (tag rules); +contract §2.1 (mark in the create call) and §4.5. + +## 9. State mapping (`provisioning-state`) + +**Answer:** `provisioning-state`: `create-blocks`. Create returns only once +`status/current` reports the VM running (entry 10), so a VM the provider +cloned but never started is only ever seen by a Create retry, which adopts +and starts it; the reconciler has no ID for it to `Get`. + +`Get` checks in this order; the first match decides: + +| Check | Source | Result | +|---|---|---| +| VMID absent | `/cluster/resources` | `ErrNotFound` | +| `lock` set, other than `suspended` | resources entry (a clone in progress also has no name) | `ErrTransient` | +| `lock: suspended` (hibernated to disk) | resources entry | Stopped | +| `status: unknown` (node not reachable) | resources entry | `ErrTransient` | +| config `does not exist` | `GET …/config` | `ErrNotFound` (deleted between the two calls) | +| config `lock` set | config | `ErrTransient` | +| `name` differs from the ID's name | config | `ErrNotFound`, logged: the VMID was reused | +| no mark in the description | config | `ErrNotFound`, logged: not ours | +| `status: stopped` | `status/current` | Stopped | +| `status: running`, `qmpstatus: running` | `status/current` | the agent decides (entry 7) | +| `status: running`, `qmpstatus: prelaunch` | `status/current` | Provisioning | +| `status: running`, any other `qmpstatus` (`paused`, `suspended`, `io-error`, …) | `status/current` | Stopped | +| any other `status` | `status/current` | Stopped | + +`status/current` is read, not only the resources entry, because a paused VM +shows as plain `running` in `/cluster/resources`. `UID` and `CreatedAt` come +from the description on every call. A running `Get` makes four calls: +resources, config, status and agent. + +**Evidence:** F8 (resources fields; mid-clone entries lack a name, carry +`lock: clone` and `status: unknown`); F14 (stopped and paused as seen by +`status/current` and by resources); F2 (the `does not exist` message); +contract §2.2 and §4.11. + +## 10. Create: blocking and task tracking (`create-return`, `async-ops`) + +**Answer:** `create-return`: `blocking`. `async-ops`: `task-handle` and +`backend-lock`. Clone, upload, resize, start, stop, destroy and volume +delete each return a UPID; the provider polls +`GET /nodes/{node}/tasks/{upid}/status` every second until `status` is +`stopped`, then reads `exitstatus`: `OK` is success, anything else is +`ErrTransient` carrying that text. The node comes from the UPID +(`UPID::…`). The handle is held within the call only: Create blocks +until the VM runs, so no handle outlives it, and a failure surfaces as +Create's own error with the task's message. Locks: a VM locked by a task is +waited on in Create (the ownership check) and reported as `ErrTransient` by +`Get` and `Delete`. A start issued while a clone holds the lock queues +behind it rather than failing; the provider never relies on that, because it +waits for the clone task first. + +Steps, each idempotent, in order, under the per-name lock (entry 19): + +1. Validate placement (entry 3). No backend call yet. +2. `GET /cluster/resources`. Find the template's node; a missing template + is `ErrPermanent`. Read the template's config and refuse it as entry 4 + says. +3. Find or allocate the VMID (entry 2). An adopted VM that is running + returns its ID at once. An adopted VM that is stopped continues at step 5. +4. Clone and wait (clone budget). +5. Build the seed and upload it to the VM's node, overwriting; wait + (upload budget). +6. `PUT /nodes/{node}/qemu/{vmid}/config` (synchronous) with `cores`, + `memory`, `agent=1`, `tags=proxy_operator`, + `ide2=:iso/-seed.iso,media=cdrom` and, when configured, + the rewritten `net0` (entry 7). +7. Resize when `diskGiB` is larger than the boot disk (entry 6); wait + (resize budget). +8. `POST …/status/start`; wait (start budget). +9. Poll `status/current` every second until `status: running` and + `qmpstatus: running`, up to the running budget; expiry is + `ErrTransient`, and so is a `stopped` status after the start. +10. Return `vms//`. + +What a retry finds: + +| Previous attempt stopped after | The retry | +|---|---| +| nothing | allocates, clones | +| submitting the clone | finds the VMID locked and nameless, waits, adopts, continues at 5 | +| the clone | adopts the stopped VM, continues at 5 | +| any of steps 5 to 7 | adopts, repeats 5 to 7 (upload overwrites, config is a replace, resize is skipped once large enough) | +| the start | adopts the running VM, returns its ID | + +A Proxmox `does not exist` inside Create (a VM or volume vanishing between +steps) is `ErrTransient`, never `ErrNotFound`. The log line for a Create +records `cloudInitBytes`, never the content. + +**Evidence:** F3 (a start queues behind the clone lock); F7 (a config write +under the clone lock is refused with `VM is locked (clone)`); F16 (task +status fields); F20 (resize is a task); the Phase 0 log's VLAN 4 rerun +(clone to started VM in about five seconds); contract §2.1, §4.2, §4.12 and +§5 ("do not return NotFound from Create"). + +## 11. Delete and secondary resources (`delete-return`, `secondary-resources`) + +**Answer:** `delete-return`: `waits`. `secondary-resources`: `explicit` for +the seed ISO; the VM's disks are removed by the destroy call itself, with +`purge=1&destroy-unreferenced-disks=1`. Under the per-name lock: + +1. Parse the ID (`ErrPermanent` if malformed). +2. `/cluster/resources`. VMID absent: skip to step 7, trying every configured + node because the VM's node is no longer known. +3. `lock` set or `status: unknown`: `ErrTransient`. A clone or another task + still running finishes first; the reconciler calls Delete again after + `DeletionPoll`. +4. Read the config. `does not exist`: skip to step 7. A different name, or + no mark: the VM is not ours; log it and skip to step 7 without touching + it. +5. Status other than `stopped`: `POST …/status/stop`, wait (stop budget). + Stop works on a paused VM too. +6. `DELETE /nodes/{node}/qemu/{vmid}?purge=1&destroy-unreferenced-disks=1`, + wait (destroy budget). `does not exist` is success. +7. `DELETE /nodes/{node}/storage/{isoStorage}/content/:iso/-seed.iso`, + wait (seed-delete budget). A missing volume is still a successful task, + so no message is matched. On the try-every-node path a node whose + request fails is logged and skipped. +8. Return nil. + +Everything is gone when Delete returns nil, so `Get` afterwards is +`ErrNotFound`. A concurrent `Get` during a Delete keeps seeing the VM (as +Stopped once it is stopped) until the destroy task removes its config, +because destroying sets no lock and leaves the resource entry in place until +the end. A second Delete waits on the lock, then finds nothing. + +GC blind spot: a VM destroyed outside the operator leaves its seed ISO, which +ListByTag cannot see. The guide says seeds are named `proxy-*-seed.iso` on +`isoStorage` and any without a matching VM may be removed. + +**Evidence:** F2 (a second destroy gives `does not exist`); F13 (volume +delete is an `imgdel` task, needs `Datastore.Allocate`, and succeeds on a +missing volume); the Phase 0 capture of resources and config mid-destroy +(`cluster_resources_during_destroy.json`, `config_during_destroy.json`); +contract §2.3. + +## 12. ListByTag scope and partial failure (`list-partial`) + +**Answer:** `fail-whole` for the one listing, with single unreadable VMs +skipped. Scope: every VM the cluster knows, from one +`GET /cluster/resources?type=vm`, which covers every node, including +unreachable ones, whose guests Proxmox lists with `status: unknown` (entry +13). Candidates are entries +with `type: qemu`, `template: 0`, and either the tag `proxy_operator` (tags +split on `;`, `,` or space) or a name starting with `proxy-`. A nameless +entry (a clone in progress) is not a candidate: its Create is live and it is +younger than `MinAge`. For each candidate the config is read and the +description parsed; managed VMs become an `Instance` with `ID`, `UID` and +`CreatedAt` only, which is all GC reads (contract §2.4); no status or agent +call is made. A candidate whose config cannot be read (a node down, a VM +destroyed mid-sweep) is skipped with a V(1) line, unless ctx ended, which +fails the call. The resources call failing fails the whole call. Filtering +by the mark is client-side; the tag narrows the candidates. Cost: one +resources call plus one config read per candidate per sweep. + +**Evidence:** F8 (resources fields, tags separated by `;`, nameless mid-clone +entries); F9 (tags case-folded); contract §2.4 and §4.7. + +## 13. Error classification (`not-found`, `quota`) + +**Answer:** `not-found`: `error-code`, the 500 message +`Configuration file 'nodes//qemu-server/.conf' does not exist`, +plus `list-and-miss` on `/cluster/resources`. The listing can miss a VM only +when it is gone: it is built from the cluster filesystem, which holds every +guest's config cluster-wide, and Proxmox lists the guests of an unreachable +node with `status: unknown` rather than dropping them. That last point is +Proxmox's documented design, not something the single-node host could show; +Phase 10 or a second node verifies it. `quota`: `absent`. The thin `local-lvm` storage +overprovisions, so no out-of-space response could be produced; capacity +failures are `ErrTransient`, and the Checker reports free space in +`Details` instead (entry 16). + +The client carries each failure as `apiError{Status, Message, Errors}`: +`Message` is the body's `message` with the trailing newline trimmed, or the +reason phrase when the body has none (a 401); `Errors` is the 400 body's +per-parameter map. One `classify(op string, err error)`: + +| Backend response | Class | Why | +|---|---|---| +| 401 `Authentication failed!`, 401 `No ticket` | Permanent | the token or its secret is wrong; fixed in the Secret | +| 403 `Permission check failed (, )` | Permanent | the token lacks a privilege; the message names it | +| 400 `Parameter verification failed.` with `errors` | Permanent | the request is wrong; the message includes the field errors | +| other 4xx | Permanent | a request Proxmox will never accept | +| 500 `Configuration file '…' does not exist` | NotFound (Transient when `op == "create"`) | the VM is gone | +| 500 `VM is locked ()` | Transient | a task holds the VM | +| 500 `unable to create VM : config file already exists` | handled by the allocator (entry 2); Transient if it escapes | the VMID is taken | +| 500 `QEMU guest agent is not running`, `No QEMU guest agent configured`, `VM is not running` | handled by `Get` (entry 7); Transient if they escape | agent state | +| any other 5xx, including pveproxy's 595 and 596 | Transient | unknown means retry | +| task `exitstatus` other than `OK` | Transient, with the text | the task failed; the next attempt retries | +| TLS verification failure (`x509.UnknownAuthorityError`, `x509.HostnameError`, `x509.CertificateInvalidError`) | Permanent | wrong URL or CA file; the admin must act | +| token file missing (`fs.ErrNotExist`) | Transient | the Secret has not landed yet | +| token file present but not `@!=` | Permanent | wrong content in the Secret | +| `context.Canceled`, `context.DeadlineExceeded`, `net.OpError`, `io.EOF`, `io.ErrUnexpectedEOF` | Transient | network or cancellation | +| an error already wrapped with a class | that class | validation and builders classify at the source | +| anything else | Transient | contract default | + +Every matched string is backed by a captured response under `testdata/`; +the table test loads them rather than repeating the text. Matching is on +fixed substrings anchored to the 500 status: `does not exist`, +`VM is locked`, `config file already exists`, `QEMU guest agent is not +running`, `No QEMU guest agent configured`, and for the stopped VM the whole +message against `^VM [0-9]+ is not running$`. The agent messages are tested +in that order, because the last is a suffix of the first. + +**Evidence:** F1 (error text in the reason phrase and the body; 401 has no +body); F2, F4, F5, F7 (the 500 messages); F6 (no out-of-space response +could be produced on thin storage); F22 (both 401 texts); fixtures +`err_forbidden.json`, `err_bad_param.json`, `err_tag_invalid.json`, +`err_upload_bad_name.json`; the Phase 0 TLS finding (`192.168.2.8` not in +the node certificate); contract §5. + +## 14. Credentials and TLS (`credentials`) + +**Answer:** `mounted-files`. The config names `tokenFile`, one line +`@!=`, sent as +`Authorization: PVEAPIToken=`. It is expected in the optional +Secret `proxmox-token`, key `token`, mounted at +`/etc/proxy-operator/proxmox`. The file is read on **every request**, so +rotating the Secret takes effect without a restart. `New` stats it and logs +a warning when it is missing, but never fails on it and never dials. + +TLS: `ca-file`. `caFile` names a PEM bundle, typically the node's +`/etc/pve/pve-root-ca.pem`, carried in the same Secret as key `ca.pem`. It +is loaded on first use and cached, so rotating the CA needs a restart; a +missing CA file is `ErrTransient` until it lands, one that does not parse is +`ErrPermanent`. Without `caFile` the system roots are used. The URL's host +must be a name or address in the node certificate's SANs: on the user's +host that is `192.168.0.8` or `pv1-1.home.lab`, not `192.168.2.8`. +`insecureSkipTLSVerify: true` disables verification and is logged as a +warning by `New`. + +Minimum privileges, as verified on 9.2.20, for `proxy-operator@pve` with a +token without privilege separation: + +- a role on `/` with `VM.Allocate VM.Clone VM.Audit VM.Config.CDROM + VM.Config.CPU VM.Config.Memory VM.Config.Disk VM.Config.Network + VM.Config.Options VM.PowerMgmt VM.GuestAgent.Audit Datastore.Audit + Datastore.AllocateSpace Datastore.AllocateTemplate Sys.Audit SDN.Use`; +- a role on `/storage/` with `Datastore.Allocate + Datastore.AllocateTemplate Datastore.AllocateSpace Datastore.Audit`, + because deleting a volume needs `Datastore.Allocate` and a grant on a more + specific path replaces the inherited one. A dedicated ISO-only storage + keeps that privilege away from backups. + +`VM.Monitor` no longer exists in PVE 9 and is not needed. +`VM.Config.Cloudinit` is not needed. `SDN.Use` on `/` covers a VLAN tag on +`vmbr0`. + +Hygiene: the token never appears in logs, span attributes or error strings. +`apiError.Error()` carries status and message only, never headers or URLs. +Tested with a sentinel secret that a test server echoes in a 401 body. + +**Evidence:** F17 (privileges exercised and the one missing); the Phase 0 +log's seed storage section; F22 (auth failure texts); contract §7. + +## 15. Config block + +**Answer:** a new block `proxmox:` with the Go type `provider.ProxmoxConfig` +in `internal/provider/config_proxmox.go`: + +| Field | Required | Default | Validation | Ends up in | +|---|---|---|---|---| +| `url` | yes | | `https` URL with a host and no path other than `/` | request base, `…/api2/json` appended | +| `tokenFile` | yes | | non-empty | `Authorization` header | +| `caFile` | | system roots | non-empty when set | TLS roots | +| `insecureSkipTLSVerify` | | false | | TLS, warning at `New` | +| `nodes` | yes, at least one | | unique, `^[a-z0-9]([a-z0-9-]*[a-z0-9])?$` | request paths, clone `target` | +| `isoStorage` | yes | | Proxmox storage id `^[a-zA-Z][a-zA-Z0-9_.-]*$` | upload and delete paths, `ide2` | +| `storage` | | the template's disk storage | storage id regexp when set | clone `storage` | +| `bridge` | | as cloned | non-empty when set | `net0` `bridge=` | +| `vlanTag` | | as cloned | 1 to 4094 when set | `net0` `tag=` | +| `vmidRange.min`, `vmidRange.max` | | 100000, 199999 | min at least 100, max at most 999999999, min below max | `vmidFor` | +| `templates..vmid` | at least one template | | at least 100 | clone source | +| `machineTypes.` | at least one | | `cores` and `memoryMiB` above zero, `diskGiB` not negative | config step, resize | +| `requestTimeoutSeconds` | | 30 | not negative | HTTP client timeout | +| `vendorData` | | | | seed `vendor-data` | +| `installGuestAgent` | | true | | seed `vendor-data` | + +`isoStorage` has no default on purpose: the token must be allowed to delete +volumes there (entry 14), so the choice must be explicit. `nodes` holds node +names as Proxmox knows them (`pv1-1`), not addresses; the one URL reaches the +whole cluster. + +Wiring: `ProviderConfig.Proxmox *ProxmoxConfig` with tag +`proxmox,omitempty`; `setBlocks` reports `proxmox`; the allow-list in +`Config.validate` and the constructor map in `cmd/main.go` both gain +`proxmox` in the same commit; `validateProxmox(prefix, c)` in +`config_proxmox.go` with `config_proxmox_test.go`. + +**Evidence:** contract §10; the Phase 0 decisions (dedicated seed storage, +`vlanTag`, URL at `192.168.0.8`); F13 (why `isoStorage` needs +`Datastore.Allocate`). + +## 16. Checker + +**Answer:** one target per `nodes` entry, `Name` the node, `Address` the +URL, checked concurrently, returned in config order. The check shares the +provider's HTTP client; each request is independent, so a check that runs +out of time cancels only its own requests. Per node: + +1. `GET /version` (`pveVersion`). +2. `GET /nodes/{node}/status` (the node answers; `kernel`). +3. `GET /nodes/{node}/storage/{isoStorage}/status`: the storage exists, is + active, and lists `iso` in its content (`isoStorageAvailGiB`). +4. `GET /cluster/resources?type=vm`: every configured template exists and + is a template (`templates` as `/`). + +Reasons, shared first: `Reachable`; `Timeout` (the round's ctx ended, +whichever request was running); `Unreachable` (TCP connect or TLS handshake +failed for a network reason, or pveproxy answered 595 or 596); +`Unauthorized` (401); `Forbidden` (403, the message names the privilege); +`APIError` (any other failure). Provider-specific, each fixed differently by +an admin: + +- `CertificateInvalid`: TLS verification failed. Fix `url` or `caFile`, not + the host. +- `CredentialsUnavailable`: the token file is missing or malformed. Fix the + Secret, not Proxmox. +- `StorageNotFound`: `isoStorage` is missing, inactive, or has no `iso` + content. Fix the storage on the node. +- `TemplateNotFound`: a configured template VMID does not exist or is not a + template. Fix the template or the catalogue. + +Details: `pveVersion`, `kernel`, `tokenID` (the part before `=`, never the +secret), `isoStorage`, `isoStorageAvailGiB`, `templates`, filled as far as +the check got. Cost: four requests per node per round. The `status.type` +comment in `api/v1alpha1/providerhealth_types.go` gains `proxmox` and the +CRD is regenerated. + +**Evidence:** contract §8; fixtures `version.json`, `node_status.json`, +`storage_status.json`, `cluster_resources.json`. + +## 17. Client and child spans (`client-kind`, `child-spans`) + +**Answer:** `client-kind`: `thin-client`. `child-spans`: `transport-wrapper`. + +The seam is the interface `api`, one method per endpoint, `ctx` first: + +| Method | HTTP | Returns | +|---|---|---| +| `Version` | `GET /version` | version info | +| `NodeStatus` | `GET /nodes/{node}/status` | node status | +| `StorageStatus` | `GET /nodes/{node}/storage/{storage}/status` | storage status | +| `ClusterResources` | `GET /cluster/resources?type=vm` | entries | +| `VMConfig` | `GET /nodes/{node}/qemu/{vmid}/config` | config as `map[string]string` | +| `SetVMConfig` | `PUT /nodes/{node}/qemu/{vmid}/config` | nothing (synchronous) | +| `Clone` | `POST /nodes/{node}/qemu/{vmid}/clone` | UPID | +| `Resize` | `PUT /nodes/{node}/qemu/{vmid}/resize` | UPID | +| `Start` | `POST /nodes/{node}/qemu/{vmid}/status/start` | UPID | +| `Stop` | `POST /nodes/{node}/qemu/{vmid}/status/stop` | UPID | +| `VMStatus` | `GET /nodes/{node}/qemu/{vmid}/status/current` | status and qmpstatus | +| `Destroy` | `DELETE /nodes/{node}/qemu/{vmid}?purge=1&destroy-unreferenced-disks=1` | UPID | +| `TaskStatus` | `GET /nodes/{node}/tasks/{upid}/status` | status and exitstatus | +| `AgentNetworkInterfaces` | `GET /nodes/{node}/qemu/{vmid}/agent/network-get-interfaces` | interfaces | +| `UploadISO` | `POST /nodes/{node}/storage/{storage}/upload` (multipart) | UPID | +| `DeleteVolume` | `DELETE /nodes/{node}/storage/{storage}/content/{volid}` | UPID | + +`httpAPI` is the production adapter; `fakeAPI` in `fake_test.go` keeps VMs, +tasks and ISOs in memory, simulates locks (a clone leaves a nameless locked +entry until its task completes), task polls, per-method errors and blocking, +and records a call log. Scalars in a config are rendered as strings by the +adapter (`cores` arrives as a JSON number, `memory` as a string). + +The transport is `tracing.RestConfigWrapper(opts...)(base)`, a parent-gated +`otelhttp` transport, so each request is an `HTTP ` span under +`provider.create`, `provider.get`, `provider.delete`, `provider.list` or the +monitor's `providerhealth.check`, and background traffic makes no root +spans. The `observed` decorator over `api` logs one line per call and starts +no span, so nothing is counted twice. Span attributes are otelhttp's +standard ones; the token is in a header, which otelhttp does not record. +Tests: a span recorder asserts the HTTP spans are children of the operation +span and carry no token; a reflection test asserts every `api` method logs +exactly one line. + +**Evidence:** contract §4.15, §4.18, §9; `RestConfigWrapper` and +`parentGatedTransport` in `internal/tracing/transport.go`; F18 (HTTP/1.1 +only). + +## 18. Timeouts (`timeouts`) + +**Answer:** `per-call`. + +| Bound | Budget | +|---|---| +| TCP dial | 10 s | +| TLS handshake | 10 s | +| one request, response included (`http.Client.Timeout`) | `requestTimeoutSeconds`, default 30 s | +| clone task, and the wait for a VM's lock during the ownership check | 10 m | +| upload task | 2 m | +| resize task | 2 m | +| start task | 2 m | +| stop task | 2 m | +| destroy task | 5 m | +| seed delete task | 1 m | +| wait for `running` after the start | 60 s | +| task and status poll interval | 1 s | + +The budgets live in one `budgets` struct that tests shrink. A task wait that +runs out is `ErrTransient` naming the UPID, and the task keeps running on +Proxmox; the next attempt finds its result through the ownership check or +the lock. A cancelled `ctx` cancels only that request: the transport keeps +its idle connections (`MaxIdleConnsPerHost` 4, `IdleConnTimeout` 90 s), and +no other call is disturbed. HTTP/2 is disabled explicitly +(`ForceAttemptHTTP2: false`, an empty `TLSNextProto`). + +**Evidence:** contract §1.5 and §4.16; F16 (a full clone of the 3 GiB +template took about two seconds; tasks finish within a second or two); F21 +(about 50 s from start to an address, which the reconciler's polling covers, +not the running wait). + +## 19. Concurrency + +**Answer:** shared state is the HTTP client (safe for concurrent use), the +lazily loaded CA pool (loaded once under a `sync.Once`-style guard), and a +lock per instance name from the shared `internal/provider/keyedlock` package +(extracted from kvm2 in Phase 3): a one-slot semaphore per key, waitable +with ctx, removed from the map once unused. Create and Delete for one name +run under that name's lock, so the reconciler's Delete and GC's Delete never +race on stop and destroy, and a Delete cannot interleave with a Create's +steps. Get and ListByTag take no lock. Two Creates for different Proxies +that land on the same VMID are safe: one clone wins, the other gets the +`config file already exists` answer, waits in the ownership check for the +winner's clone to finish, sees a different name, and probes on. Three +reconciles, a GC sweep and a check round at once are safe. + +**Evidence:** contract §1.2 (concurrency); F4; entry 2. + +## 20. Logging (`logging`) + +**Answer:** `tiered`. Logger from `logf.FromContext(ctx)` on every call, +named `proxmox`, with `provider` on every line and `vmid`, `name` and +`node` on every line about an instance. + +- V(1): one line per operation outcome (`proxmox vm ensured running`, + `proxmox vm fetched` with state, ip and uid, `proxmox vm deleted`, + `proxmox vms listed` with counts, `proxmox node checked` with the reason), + one `proxmox api call` line per seam call from the decorator (`call`, + `node`, `vmid`, `storage`, `upid`, `duration`, and on failure + `httpStatus` and `error`), a `proxmox task failed` line with the UPID and + `exitstatus`, and the no-op lines of Delete and ListByTag (foreign VM + left alone, unreadable candidate skipped). +- V(2): request parameters (without `description`'s content at V(1) and + never the upload body), config summaries, agent addresses, the resolved + Create parameters with `cloudInitBytes`. +- No V(5) wire tier. +- Never logged at any level: `CloudInit`, the token, the CA content, the + upload body. + +**Evidence:** contract §4.17 and §9. + +## 21. Tests (`backend-test`) + +**Answer:** `backend-test`: `integration-tag`. Unit tests run over `fakeAPI` +behind the seam, the HTTP client over `httptest.NewTLSServer` serving the +captured fixtures, and `classify` over the fixtures directly. The +integration test is `proxmox_integration_test.go` behind +`//go:build integration`, skipped unless `PROXMOX_TEST_URL`, +`PROXMOX_TEST_TOKEN_FILE`, `PROXMOX_TEST_NODE`, +`PROXMOX_TEST_TEMPLATE_VMID` and `PROXMOX_TEST_ISO_STORAGE` are set; +optional `PROXMOX_TEST_CA_FILE`, `PROXMOX_TEST_INSECURE`, +`PROXMOX_TEST_STORAGE`, `PROXMOX_TEST_BRIDGE`, `PROXMOX_TEST_VLAN_TAG`, +`PROXMOX_TEST_VMID_MIN`, `PROXMOX_TEST_VMID_MAX`, `PROXMOX_TEST_TIMEOUT` +(default 20m), `PROXMOX_TEST_KEEP`, `PROXMOX_TEST_LOG_LEVEL`; run with +`make test-integration-proxmox`. It asserts, in order: `New` returns without +a request; `Check` reports the node Ready with details; `Create` returns +`vms//` with the VMID in range; a second `Create` returns the +same ID and the VM count is unchanged; `Get` reaches Running with an IPv4 +within the budget and its `UID` and `CreatedAt` equal what Create wrote; +`ListByTag` contains it; `Delete` returns nil; `Get` is `ErrNotFound`; +`Delete` again returns nil; no VM with the name and no `-seed.iso` +remain; an HTTP span is a child of `provider.create` and no span attribute +carries the cloud-init text. + +Contract §11 list, with the planned test for each item (Phase 8 ticks them +once they exist): + +- [ ] repeat `Create`: `TestCreate_repeatReturnsSameIDAndCreatesNothing` +- [ ] `Get` after `Delete`: `TestGet_afterDeleteReturnsNotFound` +- [ ] `Delete` twice: `TestDelete_twiceSucceeds` +- [ ] malformed ID: `TestParseProviderID_malformed`, `TestGet_malformedIDIsPermanent`, `TestDelete_malformedIDIsPermanent` +- [ ] unknown state maps to `StateStopped`: `TestMapState_table`, `TestGet_unknownStateIsStopped` +- [ ] `StateRunning` requires an IP: `TestGet_runningWithoutIPIsProvisioning`, `TestGet_runningWithIPIsRunning` +- [ ] `UID` and `CreatedAt` read back: `TestGet_readsUIDAndCreatedAtFromBackend` +- [ ] `ListByTag` excludes unmarked instances: `TestListByTag_excludesUnmarkedVMs`, `TestListByTag_emptyUIDStillListed` +- [ ] `Get` during an in-progress delete: `TestGet_duringDeleteStillReturnsInstance` +- [ ] `Get` during an asynchronous operation or lock: `TestGet_lockedIsTransientNeverNotFound`, `TestGet_midCloneNamelessEntryIsTransient` +- [ ] never started is never `StateStopped`: `TestCreate_returnsOnlyWhenRunning` +- [ ] cancelled `ctx` returns promptly: `TestProvider_cancelledContextReturnsPromptly`, `TestClient_cancelledContextReturnsPromptly` +- [ ] `classify` table with captured responses: `TestClassify_table` +- [ ] Checker reasons, expired `ctx`, stable order: `TestCheck_reasonsTable`, `TestCheck_expiredContextReturnsPromptly`, `TestCheck_oneEntryPerNodeInConfigOrder` +- [ ] sentinel tests for `CloudInit` and the credential: `TestCreate_cloudInitSentinelNeverLoggedOrInErrors`, `TestClient_tokenNeverInErrorsOrLogs` +- [ ] child spans under the operation span: `TestTracing_httpSpansUnderOperationSpan`, `TestClient_childSpanUnderParent` +- [ ] real-backend lifecycle test: `TestProxmox_lifecycle` and `TestProxmoxCheck_realHost` in `proxmox_integration_test.go` + +Tests that cover the Phase 0 corrections specifically, in addition to the +plan's per-phase lists: + +- `TestCreate_resumesMidCloneVMAtDeterministicVMID`: a crash after the clone + was submitted; the retry waits for the lock and adopts, creating nothing. +- `TestCreate_takenVMIDInspectedBeforeProbing`: a foreign VM at attempt 0 is + skipped; ours at attempt 0 is adopted. +- `TestCreate_cloneAlreadyExistsRaceRunsOwnershipCheck`. +- `TestCreate_templateWithCloudInitDriveIsPermanent`. +- `TestCreate_seedUploadOverwritesWithoutDelete`. +- `TestCreate_resizeWaitsForTask`, `TestCreate_resizeSkippedWhenNotLarger`. +- `TestCreate_vlanTagAppendedKeepsMACAndOptions`. +- `TestBootDisk_table`, with a row for `order=scsi0;ide2`. +- `TestGet_pausedIsStoppedFromStatusCurrent`. +- `TestGet_agentMessagesTable` (the three messages and a 200 without IPv4). +- `TestDelete_seedDeleteIsTaskAndIdempotent`. +- `TestClient_x509FailureIsPermanent`. + +**Evidence:** contract §11; docs/testing.md conventions. + +## 22. Deployment + +**Answer:** a new optional Secret and its mount, two Makefile targets, and +samples: + +- `config/manager/manager.yaml`: an optional Secret `proxmox-token` mounted + read-only at `/etc/proxy-operator/proxmox`, keys `token` and `ca.pem`. + `/tmp` is already an emptyDir. +- Makefile: `proxmox-token-secret`, creating or updating `proxmox-token` + from `PROXMOX_TOKEN_FILE` (default `$(HOME)/.config/proxmox-token`) and + `PROXMOX_CA_FILE` (default `$(HOME)/.config/proxmox-ca.pem`), shaped like + `kvm-ssh-secret`; and `test-integration-proxmox` (`go test -race + -tags=integration -count=1 -v -timeout 30m -run TestProxmox + ./internal/provider/proxmox/`). +- `config/manager/providers_config.yaml` and + `config/samples/providers-config.yaml`: a commented `proxmox-home` entry + for the user's host: `url: https://192.168.0.8:8006`, the two file paths, + `nodes: [pv1-1]`, `isoStorage: proxy-seeds`, `vlanTag: 4`, + `templates: {debian-13: {vmid: 9001}}`, a `small` machine type. +- `config/samples/proxy_proxmox.yaml`, listed in + `config/samples/kustomization.yaml`: provider `proxmox-home`, zone + `pv1-1`, machine type `small`, image `debian-13`, the Squid cloud-init + from `proxy_kvm2.yaml`. +- RBAC: none; the provider does not talk to the cluster's API. +- `hack/providers-dev.yaml`: no entry (needs credentials). + +**Evidence:** contract §12. + +## 23. Docs + +**Answer:** `docs/proxmox-provider.md` with sections: Proxmox prerequisites +(the template recipe and what disqualifies a template, the dedicated seed +storage, the user, both roles and the token, the proxy network with DHCP and +internet, reserving the VMID range), Cluster-side setup (the Secret, the +config entry, the URL-must-match-the-certificate rule), Config reference +(entry 15), Placement (Region ignored, catalogue-edit warning), What lands on +Proxmox (the VMID scheme, the description mark, the tag, the seed ISO, the +config keys written, `qm unlock` for a stuck clone), Logging, Tracing, +Timeouts, Health check (entry 16), Error classes (entry 13), Known gaps +(entry 24), Testing. A fifth column in the three tables of +`docs/providers-existing.md` and a proxmox section. `docs/architecture.md`: +components row, a `proxmox` block in §5, the span note in §10, a §11 row and +the provider count, Decisions bullets (VMID from the UID hash with the +ownership check, the mark in the description, the seed by API upload, the +node resolved per call). A README row. `docs/providers.md` link. +`docs/testing.md` rows and a proxmox integration section. `CLAUDE.md` +Gotchas: `make test-integration-proxmox`. A `CHANGELOG.md` entry after the +user confirms. + +**Evidence:** contract §12. + +## 24. Known limits accepted at design time + +**Answer:** ten limits, each with its consequence and workaround: + +- **Create blocks** until the VM runs, holding one of three reconcile + workers. On the user's host that is about five seconds plus the clone; + large templates on slow storage take longer. +- **A clone lock that never clears** (a node crash mid-clone) makes every + Create for that Proxy wait out the clone budget and return + `ErrTransient`. Clear it with `qm unlock ` or destroy the VM. +- **Seed ISOs of VMs destroyed outside the operator** are invisible to GC. + Remove `proxy-*-seed.iso` files without a matching VM by hand. +- **The VMID range is shared with humans.** A VM created by hand inside it + costs a probe; a hand-made VM named like a proxy is refused as + `ErrPermanent`. Reserve the range. +- **Cross-node clones need shared storage** and are untested, because the + host has one node. +- **A VM whose agent was switched off is replaced**, not repaired (entry 7). +- **The proxy network needs internet** for the agent install, unless the + template ships the agent. +- **The CA is cached** after first use; rotating it needs a restart. The + token rotates freely. +- **A node that stops answering** makes `Get` and `Delete` for its VMs + `ErrTransient` until it returns, so deleting such a Proxy waits. +- Contract §13 limits that apply unchanged: config read once per process, + no installation identity in GC (two operators sharing a cluster and a + VMID range delete each other's VMs), an unconfigured provider latches the + Proxy. + +**Evidence:** contract §13; entries 2, 7, 10, 11, 14 above. From 6e996ae2c72030264170891b7d77ce38acce4ac5 Mon Sep 17 00:00:00 2001 From: kacerr Date: Wed, 23 Sep 2026 09:42:46 +0000 Subject: [PATCH 05/13] proxmox: config block, IDs, marks, state mapping, classify, registration Phase 2 of the proxmox provider plan. Adds the proxmox: config block with its validation, registers the type in the config allow-list and the constructor map, and starts internal/provider/proxmox: the api seam and its data types, providerIDs (vms//) with the UID-derived VMID, the description mark, state mapping, and error classification. Tests read the responses captured from pv1-1 in Phase 0 rather than repeating Proxmox's wording, and every matcher is run against every captured error so none fires on the wrong message. The contract methods are Transient stubs until Phases 5 to 7. Co-Authored-By: Claude Opus 5.5 (1M context) --- cmd/main.go | 6 +- .../2026-09-18-1502-proxmox-provider.md | 54 ++++- internal/provider/config.go | 11 +- internal/provider/config_proxmox.go | 179 +++++++++++++++ internal/provider/config_proxmox_test.go | 132 +++++++++++ internal/provider/config_test.go | 20 ++ internal/provider/proxmox/api.go | 119 ++++++++++ internal/provider/proxmox/api_test.go | 97 ++++++++ internal/provider/proxmox/errors.go | 192 ++++++++++++++++ internal/provider/proxmox/errors_test.go | 197 +++++++++++++++++ internal/provider/proxmox/fixtures_test.go | 57 +++++ internal/provider/proxmox/id.go | 42 ++++ internal/provider/proxmox/id_test.go | 89 ++++++++ internal/provider/proxmox/mark.go | 54 +++++ internal/provider/proxmox/mark_test.go | 64 ++++++ internal/provider/proxmox/proxmox.go | 173 +++++++++++++++ internal/provider/proxmox/proxmox_test.go | 208 ++++++++++++++++++ internal/provider/proxmox/state.go | 52 +++++ internal/provider/proxmox/state_test.go | 114 ++++++++++ 19 files changed, 1856 insertions(+), 4 deletions(-) create mode 100644 internal/provider/config_proxmox.go create mode 100644 internal/provider/config_proxmox_test.go create mode 100644 internal/provider/proxmox/api.go create mode 100644 internal/provider/proxmox/api_test.go create mode 100644 internal/provider/proxmox/errors.go create mode 100644 internal/provider/proxmox/errors_test.go create mode 100644 internal/provider/proxmox/fixtures_test.go create mode 100644 internal/provider/proxmox/id.go create mode 100644 internal/provider/proxmox/id_test.go create mode 100644 internal/provider/proxmox/mark.go create mode 100644 internal/provider/proxmox/mark_test.go create mode 100644 internal/provider/proxmox/proxmox.go create mode 100644 internal/provider/proxmox/proxmox_test.go create mode 100644 internal/provider/proxmox/state.go create mode 100644 internal/provider/proxmox/state_test.go diff --git a/cmd/main.go b/cmd/main.go index 4bacd2f..5a0ff77 100644 --- a/cmd/main.go +++ b/cmd/main.go @@ -61,6 +61,7 @@ import ( "gitea.home.hrajfrisbee.cz/kacerr/egress-proxies-operator/internal/provider/kubernetes" "gitea.home.hrajfrisbee.cz/kacerr/egress-proxies-operator/internal/provider/kvm" "gitea.home.hrajfrisbee.cz/kacerr/egress-proxies-operator/internal/provider/kvm2" + "gitea.home.hrajfrisbee.cz/kacerr/egress-proxies-operator/internal/provider/proxmox" "gitea.home.hrajfrisbee.cz/kacerr/egress-proxies-operator/internal/provider/registry" "gitea.home.hrajfrisbee.cz/kacerr/egress-proxies-operator/internal/providerhealth" "gitea.home.hrajfrisbee.cz/kacerr/egress-proxies-operator/internal/tracing" @@ -211,8 +212,9 @@ func main() { "gcp": func(ctx context.Context, pc provider.ProviderConfig) (provider.Provider, error) { return gcp.NewWithWireOptions(ctx, pc, gcp.WireLogOptions{FullPayloads: gcpWireFullPayloads}) }, - "kvm": kvm.New, - "kvm2": kvm2.New, + "kvm": kvm.New, + "kvm2": kvm2.New, + "proxmox": proxmox.New, }) if err != nil { setupLog.Error(err, "Failed to build providers") diff --git a/docs/plans-executions/2026-09-18-1502-proxmox-provider.md b/docs/plans-executions/2026-09-18-1502-proxmox-provider.md index ec9cc10..24d7fcb 100644 --- a/docs/plans-executions/2026-09-18-1502-proxmox-provider.md +++ b/docs/plans-executions/2026-09-18-1502-proxmox-provider.md @@ -6,7 +6,7 @@ Pairs with [docs/plans/2026-09-18-1502-proxmox-provider.md](../plans/2026-09-18- - [x] Phase 0 — Plan file, branch, host recon, fixtures - [x] Phase 1 — The spec (`docs/proxmox-provider-spec.md`) -- [ ] Phase 2 — Config block, registration, package skeleton +- [x] Phase 2 — Config block, registration, package skeleton - [ ] Phase 3 — Extract `nocloud` and `keyedlock` from kvm2 - [ ] Phase 4 — Seam, HTTP client, decorator, fake - [ ] Phase 5 — Get, ListByTag, task helpers @@ -329,3 +329,55 @@ listed as `unknown` is Proxmox's documented design, unverifiable on a single node; entry 13 now says so. And the matcher for `VM is not running` is a whole-message regexp, because ` is not running` alone also matches `QEMU guest agent is not running`. + +## Phase 2 — Config block, registration, package skeleton + +Added the `proxmox:` block (`internal/provider/config_proxmox.go`), wired it +into `config.go` (the field, `setBlocks`, the allow-list and the dispatch: +three `"proxmox"` sites) and registered `"proxmox": proxmox.New` in +`cmd/main.go`, so the two type lists changed in one commit. The package +`internal/provider/proxmox` now holds the seam interface and its data types +(`api.go`), `id.go`, `mark.go`, `state.go`, `errors.go` and the constructor +(`proxmox.go`). The four contract methods are stubs that answer +`ErrTransient`, so a Proxy on a configured `proxmox` entry retries instead of +latching until Phases 5 to 7 land. `New` passes a nil `api` until Phase 4 +builds the HTTP client. + +Every test that depends on Proxmox's wording or JSON shape reads the Phase 0 +captures through `fixtures_test.go` (`readCapture`, `loadAPIError`, +`decodeData`), so no message string is repeated in a test. The one exception +is `TestClassify_table`'s synthetic rows (a 404, 502, 595, 596, a storage +message), for statuses recon never produced. + +Deviations from the plan's Phase 2 list, both making it smaller: + +- **No temporary `locks.go`.** Only Create and Delete take the per-name lock, + and they arrive after Phase 3 extracts `internal/provider/keyedlock`, so + copying kvm2's lock now would only be deleted next phase. +- **No `provider.Checker` assertion yet.** `cmd/main.go` type-asserts the + Checker and would call a stub; the assertion comes with the real `Check` + in Phase 7. + +Added beyond the plan: `api_test.go` decodes the captured responses into the +seam's types, so a misspelled JSON tag fails now rather than in Phase 5, and +`TestParseDescription_fromClone` reads the mark back from the description of +the VM cloned in Phase 0. + +A mutation check confirmed the suite can fail. Matching `VM is not +running` as a plain suffix (the first spec draft's mistake) failed +`TestAPIError_matchers` on the agent-not-running capture; treating a paused +VM as running failed `TestMapState_table` and `TestMapState_fromCaptures`. +Both mutations were reverted and the files compared identical to backups. + +```bash +gofmt -w cmd/main.go # realigned the constructor map for the longer key +grep -c '"proxmox"' internal/provider/config.go # 3 +go build ./... && go vet ./... +go test -race -count=1 ./internal/provider/... ./cmd/... # all ok +go test -short -count=1 ./... # all ok +go test -count=1 -v ./internal/provider/proxmox/ | grep -c '^=== RUN' # 152 +``` + +Worth noting: the `validProxmoxYAML` constant is the shape Phase 9's commented +sample entry should take, with the user's values (`192.168.0.8`, `pv1-1`, +`proxy-seeds`, `vlanTag: 4`, template 9001). diff --git a/internal/provider/config.go b/internal/provider/config.go index 07fe0cd..64c7260 100644 --- a/internal/provider/config.go +++ b/internal/provider/config.go @@ -25,6 +25,8 @@ type ProviderConfig struct { // KVM2 is the same block shape as KVM, for the kvm2 provider type: one // or more libvirt/KVM hypervisors reached over SSH. KVM2 *KVMConfig `json:"kvm2,omitempty"` + // Proxmox is a Proxmox VE cluster reached through its REST API. + Proxmox *ProxmoxConfig `json:"proxmox,omitempty"` } // setBlocks names the type-specific blocks that are non-nil, so validate can @@ -43,6 +45,9 @@ func (p ProviderConfig) setBlocks() []string { if p.KVM2 != nil { out = append(out, "kvm2") } + if p.Proxmox != nil { + out = append(out, "proxmox") + } return out } @@ -106,7 +111,7 @@ func (c *Config) validate() error { prefix := fmt.Sprintf("providers[%d] %q", i, p.Name) switch p.Type { - case "kubernetes", "gcp", "kvm", "kvm2": + case "kubernetes", "gcp", "kvm", "kvm2", "proxmox": case "": return fmt.Errorf("%s: type is required", prefix) default: @@ -130,6 +135,10 @@ func (c *Config) validate() error { if err := validateKVMBlock(prefix, "kvm2", p.KVM2); err != nil { return err } + case "proxmox": + if err := validateProxmox(prefix, p.Proxmox); err != nil { + return err + } } } return nil diff --git a/internal/provider/config_proxmox.go b/internal/provider/config_proxmox.go new file mode 100644 index 0000000..18c877b --- /dev/null +++ b/internal/provider/config_proxmox.go @@ -0,0 +1,179 @@ +package provider + +import ( + "fmt" + "net/url" + "regexp" +) + +// ProxmoxConfig configures a proxmox provider instance: one Proxmox VE +// cluster reached through its REST API with an API token. VMs are full +// clones of prepared templates, seeded with a NoCloud ISO uploaded to +// ISOStorage. The design is in docs/proxmox-provider-spec.md. +type ProxmoxConfig struct { + // URL is the API endpoint, https://:8006. The host must be a name + // or address in the node certificate. Required. + URL string `json:"url"` + // TokenFile holds one line, @!=, read on + // every request. Required. + TokenFile string `json:"tokenFile"` + // CAFile is a PEM bundle trusted for the API's certificate, typically the + // node's /etc/pve/pve-root-ca.pem. Empty means the system roots. + CAFile string `json:"caFile,omitempty"` + // InsecureSkipTLSVerify disables certificate verification. + InsecureSkipTLSVerify bool `json:"insecureSkipTLSVerify,omitempty"` + // Nodes are the Proxmox node names placement.zone selects from. + // Required, at least one, unique. + Nodes []string `json:"nodes"` + // ISOStorage is the storage seed ISOs are uploaded to and deleted from; + // the token needs Datastore.Allocate on it. Required. + ISOStorage string `json:"isoStorage"` + // Storage is the target storage of the full clone. Empty clones onto the + // template's own disk storage. + Storage string `json:"storage,omitempty"` + // Bridge replaces the bridge of the cloned net0. Empty keeps it. + Bridge string `json:"bridge,omitempty"` + // VLANTag sets the VLAN tag of the cloned net0, 1 to 4094. Zero keeps + // whatever the template has. + VLANTag int `json:"vlanTag,omitempty"` + // VMIDRange is where VMIDs are allocated. Default 100000 to 199999. + VMIDRange *ProxmoxVMIDRange `json:"vmidRange,omitempty"` + // Templates are the clone sources placement.image names. Required, at + // least one. + Templates map[string]ProxmoxTemplate `json:"templates"` + // MachineTypes are the size profiles placement.machineType names. + // Required, at least one. + MachineTypes map[string]ProxmoxMachineType `json:"machineTypes"` + // RequestTimeoutSeconds bounds one API request. Default 30. + RequestTimeoutSeconds int `json:"requestTimeoutSeconds,omitempty"` + // VendorData is optional cloud-init vendor-data placed on every seed next + // to the Proxy's own user-data, as for the kvm providers. + VendorData string `json:"vendorData,omitempty"` + // InstallGuestAgent adds a vendor-data script that installs and starts + // qemu-guest-agent, which is how the provider learns a VM's address. + // Default true. Turn it off for templates that ship the agent. + InstallGuestAgent *bool `json:"installGuestAgent,omitempty"` +} + +// ProxmoxVMIDRange bounds the VMIDs the provider allocates, both inclusive. +type ProxmoxVMIDRange struct { + Min int `json:"min"` + Max int `json:"max"` +} + +// ProxmoxTemplate is one clone source: a VM converted to a template. +type ProxmoxTemplate struct { + VMID int `json:"vmid"` +} + +// ProxmoxMachineType is a VM size profile. DiskGiB grows the cloned boot +// disk when larger than the template's; zero keeps the template's size. +type ProxmoxMachineType struct { + Cores int32 `json:"cores"` + MemoryMiB int64 `json:"memoryMiB"` + DiskGiB int64 `json:"diskGiB,omitempty"` +} + +// Proxmox's own limits on VMIDs and VLAN tags. +const ( + ProxmoxMinVMID = 100 + ProxmoxMaxVMID = 999999999 + proxmoxMaxVLANTag = 4094 +) + +var ( + // proxmoxNodeName is a Proxmox node name: a DNS label, as pvecm allows. + proxmoxNodeName = regexp.MustCompile(`^[a-z0-9]([a-z0-9-]*[a-z0-9])?$`) + // proxmoxStorageID is a Proxmox storage identifier. + proxmoxStorageID = regexp.MustCompile(`^[a-zA-Z][a-zA-Z0-9_.-]*$`) +) + +func validateProxmox(prefix string, c *ProxmoxConfig) error { + if c == nil { + return fmt.Errorf("%s: proxmox block is required", prefix) + } + if err := validateProxmoxURL(c.URL); err != nil { + return fmt.Errorf("%s: proxmox.url %w", prefix, err) + } + if c.TokenFile == "" { + return fmt.Errorf("%s: proxmox.tokenFile is required", prefix) + } + if len(c.Nodes) == 0 { + return fmt.Errorf("%s: proxmox.nodes must list at least one node", prefix) + } + seen := make(map[string]bool, len(c.Nodes)) + for i, n := range c.Nodes { + if !proxmoxNodeName.MatchString(n) { + return fmt.Errorf("%s: proxmox.nodes[%d] %q is not a Proxmox node name (lowercase letters, digits and dashes)", prefix, i, n) + } + if seen[n] { + return fmt.Errorf("%s: proxmox.nodes[%d]: duplicate node %q", prefix, i, n) + } + seen[n] = true + } + if c.ISOStorage == "" { + return fmt.Errorf("%s: proxmox.isoStorage is required: name a storage with iso content where the token holds Datastore.Allocate", prefix) + } + if !proxmoxStorageID.MatchString(c.ISOStorage) { + return fmt.Errorf("%s: proxmox.isoStorage %q is not a Proxmox storage id", prefix, c.ISOStorage) + } + if c.Storage != "" && !proxmoxStorageID.MatchString(c.Storage) { + return fmt.Errorf("%s: proxmox.storage %q is not a Proxmox storage id", prefix, c.Storage) + } + if c.VLANTag < 0 || c.VLANTag > proxmoxMaxVLANTag { + return fmt.Errorf("%s: proxmox.vlanTag must be between 1 and %d (got %d)", prefix, proxmoxMaxVLANTag, c.VLANTag) + } + if r := c.VMIDRange; r != nil { + if r.Min < ProxmoxMinVMID || r.Max > ProxmoxMaxVMID || r.Min >= r.Max { + return fmt.Errorf("%s: proxmox.vmidRange must satisfy %d <= min < max <= %d (got %d to %d)", prefix, ProxmoxMinVMID, ProxmoxMaxVMID, r.Min, r.Max) + } + } + if len(c.Templates) == 0 { + return fmt.Errorf("%s: proxmox.templates must define at least one template", prefix) + } + for key, t := range c.Templates { + if key == "" { + return fmt.Errorf("%s: proxmox.templates has an empty key", prefix) + } + if t.VMID < ProxmoxMinVMID { + return fmt.Errorf("%s: proxmox.templates[%q].vmid must be at least %d (got %d)", prefix, key, ProxmoxMinVMID, t.VMID) + } + } + if len(c.MachineTypes) == 0 { + return fmt.Errorf("%s: proxmox.machineTypes must define at least one profile", prefix) + } + for key, mt := range c.MachineTypes { + if key == "" { + return fmt.Errorf("%s: proxmox.machineTypes has an empty key", prefix) + } + if mt.Cores <= 0 || mt.MemoryMiB <= 0 { + return fmt.Errorf("%s: proxmox.machineTypes[%q] needs cores > 0 and memoryMiB > 0", prefix, key) + } + if mt.DiskGiB < 0 { + return fmt.Errorf("%s: proxmox.machineTypes[%q].diskGiB must not be negative", prefix, key) + } + } + if c.RequestTimeoutSeconds < 0 { + return fmt.Errorf("%s: proxmox.requestTimeoutSeconds must not be negative", prefix) + } + return nil +} + +// validateProxmoxURL accepts only an https URL with a host and no path, so +// the client can append /api2/json itself. +func validateProxmoxURL(s string) error { + if s == "" { + return fmt.Errorf("is required") + } + u, err := url.Parse(s) + if err != nil { + return fmt.Errorf("does not parse: %w", err) + } + if u.Scheme != "https" || u.Host == "" { + return fmt.Errorf("must be an https URL with a host (got %q)", s) + } + if u.Path != "" && u.Path != "/" || u.RawQuery != "" || u.Fragment != "" { + return fmt.Errorf("must have no path, query or fragment; the client appends /api2/json (got %q)", s) + } + return nil +} diff --git a/internal/provider/config_proxmox_test.go b/internal/provider/config_proxmox_test.go new file mode 100644 index 0000000..d623928 --- /dev/null +++ b/internal/provider/config_proxmox_test.go @@ -0,0 +1,132 @@ +package provider + +import ( + "strings" + "testing" +) + +const validProxmoxYAML = ` +providers: + - name: proxmox-home + type: proxmox + proxmox: + url: https://192.168.0.8:8006 + tokenFile: /etc/proxy-operator/proxmox/token + caFile: /etc/proxy-operator/proxmox/ca.pem + nodes: [pv1-1, pv1-2] + isoStorage: proxy-seeds + storage: local-lvm + bridge: vmbr0 + vlanTag: 4 + vmidRange: {min: 150000, max: 150999} + templates: + debian-13: {vmid: 9001} + machineTypes: + small: {cores: 1, memoryMiB: 1024} + large: {cores: 4, memoryMiB: 4096, diskGiB: 20} + requestTimeoutSeconds: 45 + installGuestAgent: false +` + +func TestLoadConfig_proxmoxValid(t *testing.T) { + t.Parallel() + cfg, err := LoadConfig([]byte(validProxmoxYAML)) + if err != nil { + t.Fatalf("LoadConfig() error = %v, want nil", err) + } + c := cfg.Providers[0].Proxmox + if c == nil { + t.Fatal("providers[0].proxmox = nil") + } + if c.URL != "https://192.168.0.8:8006" || c.ISOStorage != "proxy-seeds" || c.VLANTag != 4 { + t.Errorf("proxmox = %+v, want url, isoStorage and vlanTag parsed", c) + } + if len(c.Nodes) != 2 || c.Nodes[1] != "pv1-2" { + t.Errorf("nodes = %v, want [pv1-1 pv1-2]", c.Nodes) + } + if c.VMIDRange == nil || c.VMIDRange.Min != 150000 || c.VMIDRange.Max != 150999 { + t.Errorf("vmidRange = %+v, want 150000 to 150999", c.VMIDRange) + } + if c.Templates["debian-13"].VMID != 9001 { + t.Errorf("templates[debian-13] = %+v, want vmid 9001", c.Templates["debian-13"]) + } + if mt := c.MachineTypes["large"]; mt.Cores != 4 || mt.MemoryMiB != 4096 || mt.DiskGiB != 20 { + t.Errorf("machineTypes[large] = %+v, want {4 4096 20}", mt) + } + if c.InstallGuestAgent == nil || *c.InstallGuestAgent { + t.Errorf("installGuestAgent = %v, want explicit false", c.InstallGuestAgent) + } +} + +func TestValidateProxmox_table(t *testing.T) { + t.Parallel() + block := func(body string) string { + return "providers:\n - name: proxmox-home\n type: proxmox\n proxmox:\n" + indent(body, 6) + } + const base = ` +url: https://192.168.0.8:8006 +tokenFile: /t +nodes: [pv1-1] +isoStorage: proxy-seeds +templates: {debian-13: {vmid: 9001}} +machineTypes: {small: {cores: 1, memoryMiB: 1024}} +` + with := func(old, repl string) string { + if !strings.Contains(base, old) { + t.Fatalf("base has no %q", old) + } + return block(strings.Replace(base, old, repl, 1)) + } + tests := []struct { + name string + yaml string + wantErrSub string + }{ + {name: "valid minimal", yaml: block(base)}, + {name: "valid full", yaml: validProxmoxYAML}, + {name: "valid with trailing slash", yaml: with("url: https://192.168.0.8:8006", "url: https://192.168.0.8:8006/")}, + {name: "missing block", yaml: "providers:\n - name: proxmox-home\n type: proxmox\n", wantErrSub: "proxmox block is required"}, + {name: "missing url", yaml: with("url: https://192.168.0.8:8006\n", ""), wantErrSub: "proxmox.url is required"}, + {name: "http url", yaml: with("https://", "http://"), wantErrSub: "must be an https URL"}, + {name: "url with api path", yaml: with("8006", "8006/api2/json"), wantErrSub: "must have no path"}, + {name: "missing tokenFile", yaml: with("tokenFile: /t\n", ""), wantErrSub: "proxmox.tokenFile is required"}, + {name: "no nodes", yaml: with("nodes: [pv1-1]", "nodes: []"), wantErrSub: "proxmox.nodes must list at least one node"}, + {name: "duplicate node", yaml: with("nodes: [pv1-1]", "nodes: [pv1-1, pv1-1]"), wantErrSub: `duplicate node "pv1-1"`}, + {name: "node with a slash", yaml: with("nodes: [pv1-1]", "nodes: [pv1/1]"), wantErrSub: "is not a Proxmox node name"}, + {name: "node in uppercase", yaml: with("nodes: [pv1-1]", "nodes: [PV1]"), wantErrSub: "is not a Proxmox node name"}, + {name: "missing isoStorage", yaml: with("isoStorage: proxy-seeds\n", ""), wantErrSub: "proxmox.isoStorage is required"}, + {name: "bad isoStorage id", yaml: with("isoStorage: proxy-seeds", "isoStorage: 1seeds"), wantErrSub: "proxmox.isoStorage \"1seeds\" is not a Proxmox storage id"}, + {name: "bad storage id", yaml: with("isoStorage: proxy-seeds", "isoStorage: proxy-seeds\nstorage: a/b"), wantErrSub: "proxmox.storage \"a/b\""}, + {name: "vlanTag too large", yaml: with("isoStorage: proxy-seeds", "isoStorage: proxy-seeds\nvlanTag: 4095"), wantErrSub: "proxmox.vlanTag must be between 1 and 4094"}, + {name: "vlanTag negative", yaml: with("isoStorage: proxy-seeds", "isoStorage: proxy-seeds\nvlanTag: -1"), wantErrSub: "proxmox.vlanTag"}, + {name: "vmidRange min below 100", yaml: with("isoStorage: proxy-seeds", "isoStorage: proxy-seeds\nvmidRange: {min: 50, max: 500}"), wantErrSub: "proxmox.vmidRange must satisfy"}, + {name: "vmidRange min above max", yaml: with("isoStorage: proxy-seeds", "isoStorage: proxy-seeds\nvmidRange: {min: 2000, max: 1000}"), wantErrSub: "proxmox.vmidRange must satisfy"}, + {name: "vmidRange max too large", yaml: with("isoStorage: proxy-seeds", "isoStorage: proxy-seeds\nvmidRange: {min: 100, max: 1000000000}"), wantErrSub: "proxmox.vmidRange must satisfy"}, + {name: "no templates", yaml: with("templates: {debian-13: {vmid: 9001}}", "templates: {}"), wantErrSub: "proxmox.templates must define at least one template"}, + {name: "template vmid below 100", yaml: with("vmid: 9001", "vmid: 99"), wantErrSub: `proxmox.templates["debian-13"].vmid must be at least 100`}, + {name: "no machineTypes", yaml: with("machineTypes: {small: {cores: 1, memoryMiB: 1024}}", "machineTypes: {}"), wantErrSub: "proxmox.machineTypes must define at least one profile"}, + {name: "cores zero", yaml: with("cores: 1", "cores: 0"), wantErrSub: `proxmox.machineTypes["small"] needs cores > 0`}, + {name: "memory zero", yaml: with("memoryMiB: 1024", "memoryMiB: 0"), wantErrSub: "memoryMiB > 0"}, + {name: "negative diskGiB", yaml: with("memoryMiB: 1024", "memoryMiB: 1024, diskGiB: -5"), wantErrSub: "diskGiB must not be negative"}, + {name: "negative timeout", yaml: with("isoStorage: proxy-seeds", "isoStorage: proxy-seeds\nrequestTimeoutSeconds: -1"), wantErrSub: "proxmox.requestTimeoutSeconds must not be negative"}, + {name: "unknown key is rejected", yaml: with("isoStorage: proxy-seeds", "isoStorage: proxy-seeds\nsshKeyFile: /k"), wantErrSub: "parsing providers config"}, + } + for _, tc := range tests { + t.Run(tc.name, func(t *testing.T) { + t.Parallel() + _, err := LoadConfig([]byte(tc.yaml)) + if tc.wantErrSub == "" { + if err != nil { + t.Fatalf("LoadConfig() error = %v, want nil", err) + } + return + } + if err == nil { + t.Fatalf("LoadConfig() error = nil, want error containing %q", tc.wantErrSub) + } + if !strings.Contains(err.Error(), tc.wantErrSub) { + t.Errorf("LoadConfig() error = %q, want substring %q", err.Error(), tc.wantErrSub) + } + }) + } +} diff --git a/internal/provider/config_test.go b/internal/provider/config_test.go index f82999c..4b81ddd 100644 --- a/internal/provider/config_test.go +++ b/internal/provider/config_test.go @@ -138,6 +138,26 @@ providers: sshKeyFile: /key`, wantErrSub: "type is kvm2 but a kvm block is set", }, + { + name: "proxmox type with kvm block", + yaml: ` +providers: + - name: p1 + type: proxmox + kvm: + sshKeyFile: /key`, + wantErrSub: "type is proxmox but a kvm block is set", + }, + { + name: "kvm type with proxmox block", + yaml: ` +providers: + - name: p1 + type: kvm + proxmox: + url: https://pve:8006`, + wantErrSub: "type is kvm but a proxmox block is set", + }, { name: "strict mode rejects unknown top-level key", yaml: ` diff --git a/internal/provider/proxmox/api.go b/internal/provider/proxmox/api.go new file mode 100644 index 0000000..e5e479f --- /dev/null +++ b/internal/provider/proxmox/api.go @@ -0,0 +1,119 @@ +package proxmox + +import ( + "context" + "io" +) + +// api is the seam over the Proxmox VE REST API: one method per endpoint the +// provider uses, ctx first (spec entry 17). Calls that start a Proxmox task +// return its UPID; the caller waits on it with TaskStatus. +type api interface { + Version(ctx context.Context) (versionInfo, error) + NodeStatus(ctx context.Context, node string) (nodeStatus, error) + StorageStatus(ctx context.Context, node, storage string) (storageStatus, error) + ClusterResources(ctx context.Context) ([]vmResource, error) + VMConfig(ctx context.Context, node string, vmid int) (vmConfig, error) + SetVMConfig(ctx context.Context, node string, vmid int, params map[string]string) error + Clone(ctx context.Context, node string, vmid int, p cloneParams) (upid string, err error) + Resize(ctx context.Context, node string, vmid int, disk, size string) (upid string, err error) + Start(ctx context.Context, node string, vmid int) (upid string, err error) + Stop(ctx context.Context, node string, vmid int) (upid string, err error) + VMStatus(ctx context.Context, node string, vmid int) (vmStatus, error) + Destroy(ctx context.Context, node string, vmid int) (upid string, err error) + TaskStatus(ctx context.Context, node, upid string) (taskStatus, error) + AgentNetworkInterfaces(ctx context.Context, node string, vmid int) ([]agentInterface, error) + UploadISO(ctx context.Context, node, storage, filename string, r io.Reader, size int64) (upid string, err error) + DeleteVolume(ctx context.Context, node, storage, volid string) (upid string, err error) +} + +// versionInfo is GET /version. +type versionInfo struct { + Version string `json:"version"` + Release string `json:"release"` + RepoID string `json:"repoid"` +} + +// nodeStatus is the part of GET /nodes/{node}/status the Checker reports. +type nodeStatus struct { + PVEVersion string `json:"pveversion"` + CurrentKernel struct { + Release string `json:"release"` + } `json:"current-kernel"` +} + +// storageStatus is GET /nodes/{node}/storage/{storage}/status. Content is a +// comma-separated list such as "iso,import,backup,vztmpl". +type storageStatus struct { + Type string `json:"type"` + Content string `json:"content"` + Active int `json:"active"` + Enabled int `json:"enabled"` + Shared int `json:"shared"` + Total int64 `json:"total"` + Used int64 `json:"used"` + Avail int64 `json:"avail"` +} + +// vmResource is one entry of GET /cluster/resources?type=vm. A VM being +// cloned is listed without a name, with lock "clone" and status "unknown" +// (spec entry 9). Tags are separated by ";". +type vmResource struct { + VMID int `json:"vmid"` + Name string `json:"name"` + Node string `json:"node"` + Type string `json:"type"` + Status string `json:"status"` + Lock string `json:"lock"` + Tags string `json:"tags"` + Template int `json:"template"` +} + +// vmConfig is GET /nodes/{node}/qemu/{vmid}/config with every value +// rendered as a string: Proxmox sends some scalars as JSON numbers (cores, +// template) and others as strings (memory). +type vmConfig map[string]string + +// vmStatus is GET /nodes/{node}/qemu/{vmid}/status/current. QMPStatus tells +// a paused VM from a running one; the resources listing cannot. +type vmStatus struct { + Status string `json:"status"` + QMPStatus string `json:"qmpstatus"` + Name string `json:"name"` + Lock string `json:"lock"` + Uptime int64 `json:"uptime"` +} + +// taskStatus is GET /nodes/{node}/tasks/{upid}/status. ExitStatus is "OK" +// or the failure's text once Status is "stopped". +type taskStatus struct { + UPID string `json:"upid"` + Type string `json:"type"` + ID string `json:"id"` + Node string `json:"node"` + Status string `json:"status"` + ExitStatus string `json:"exitstatus"` +} + +// agentInterface is one entry of the guest agent's network-get-interfaces +// result. +type agentInterface struct { + Name string `json:"name"` + IPAddresses []agentIP `json:"ip-addresses"` +} + +type agentIP struct { + Address string `json:"ip-address"` + Type string `json:"ip-address-type"` + Prefix int `json:"prefix"` +} + +// cloneParams are the parameters of POST .../clone the provider sets. +// Storage and Target are omitted when empty. +type cloneParams struct { + NewID int + Name string + Description string + Storage string + Target string +} diff --git a/internal/provider/proxmox/api_test.go b/internal/provider/proxmox/api_test.go new file mode 100644 index 0000000..b91d2b1 --- /dev/null +++ b/internal/provider/proxmox/api_test.go @@ -0,0 +1,97 @@ +package proxmox + +import ( + "strings" + "testing" +) + +// TestAPITypes_decodeCaptures decodes the Phase 0 responses into the seam's +// types, so a misspelled JSON tag fails here and not in a later phase. +func TestAPITypes_decodeCaptures(t *testing.T) { + t.Parallel() + + t.Run("version", func(t *testing.T) { + t.Parallel() + if v := decodeData[versionInfo](t, "version"); v.Version != "9.2.20" || v.Release != "9.2" || v.RepoID == "" { + t.Errorf("version = %+v", v) + } + }) + t.Run("node status", func(t *testing.T) { + t.Parallel() + n := decodeData[nodeStatus](t, "node_status") + if !strings.HasPrefix(n.PVEVersion, "pve-manager/9.2.20") || n.CurrentKernel.Release != "7.0.14-17-pve" { + t.Errorf("node status = %+v", n) + } + }) + t.Run("storage status", func(t *testing.T) { + t.Parallel() + s := decodeData[storageStatus](t, "storage_status") + if s.Type != "dir" || !strings.Contains(s.Content, "iso") || s.Active != 1 || s.Avail <= 0 || s.Total < s.Avail { + t.Errorf("storage status = %+v", s) + } + }) + t.Run("cluster resources", func(t *testing.T) { + t.Parallel() + rs := decodeData[[]vmResource](t, "cluster_resources") + byID := map[int]vmResource{} + for _, r := range rs { + byID[r.VMID] = r + } + if r := byID[9001]; r.Name != "debian-13-template" || r.Template != 1 || r.Node != "pv1-1" || r.Type != "qemu" || r.Status != "stopped" { + t.Errorf("resources[9001] = %+v", r) + } + if r := byID[100]; r.Template != 0 || r.Name != "ubuntu-vm-1" { + t.Errorf("resources[100] = %+v", r) + } + }) + t.Run("cluster resources during clone", func(t *testing.T) { + t.Parallel() + for _, r := range decodeData[[]vmResource](t, "cluster_resources_during_clone") { + if r.VMID != 199999 { + continue + } + if r.Name != "" || r.Lock != "clone" || r.Status != "unknown" { + t.Errorf("mid-clone entry = %+v, want no name, lock clone, status unknown", r) + } + return + } + t.Error("no entry for VMID 199999 in the mid-clone capture") + }) + t.Run("cluster resources tags", func(t *testing.T) { + t.Parallel() + for _, r := range decodeData[[]vmResource](t, "cluster_resources_during_destroy") { + if r.VMID == 199999 && r.Tags != "proxy_operator" { + t.Errorf("tags = %q, want proxy_operator", r.Tags) + } + } + }) + t.Run("status current", func(t *testing.T) { + t.Parallel() + if s := decodeData[vmStatus](t, "status_current_paused"); s.Status != "running" || s.QMPStatus != "paused" { + t.Errorf("paused status = %+v", s) + } + if s := decodeData[vmStatus](t, "status_current_stopped"); s.Status != "stopped" || s.QMPStatus != "stopped" { + t.Errorf("stopped status = %+v", s) + } + }) + t.Run("task status", func(t *testing.T) { + t.Parallel() + if s := decodeData[taskStatus](t, "task_status_stopped_error"); s.Status != "stopped" || s.ExitStatus != "shrinking disks is not supported" || s.Type != "resize" { + t.Errorf("failed task = %+v", s) + } + if s := decodeData[taskStatus](t, "task_status_running"); s.Status != "running" || s.ExitStatus != "" || s.Node != "pv1-1" { + t.Errorf("running task = %+v", s) + } + if s := decodeData[taskStatus](t, "task_status_volume_missing"); s.ExitStatus != "OK" { + t.Errorf("deleting a missing volume = %+v, want exitstatus OK", s) + } + }) + t.Run("UPID responses", func(t *testing.T) { + t.Parallel() + for _, c := range []string{"clone_response", "start_response", "stop_response", "destroy_response", "resize_response", "upload_response", "content_delete_response"} { + if upid := decodeData[string](t, c); !strings.HasPrefix(upid, "UPID:pv1-1:") { + t.Errorf("%s = %q, want a UPID of node pv1-1", c, upid) + } + } + }) +} diff --git a/internal/provider/proxmox/errors.go b/internal/provider/proxmox/errors.go new file mode 100644 index 0000000..3dbf074 --- /dev/null +++ b/internal/provider/proxmox/errors.go @@ -0,0 +1,192 @@ +package proxmox + +import ( + "crypto/tls" + "crypto/x509" + "encoding/json" + "errors" + "fmt" + "maps" + "net/http" + "regexp" + "slices" + "strings" + + "gitea.home.hrajfrisbee.cz/kacerr/egress-proxies-operator/internal/provider" +) + +// permanentf and transientf classify an error at the source, so classify +// keeps that class no matter what wraps it later. +func permanentf(format string, args ...any) error { + return fmt.Errorf("%w: %s", provider.ErrPermanent, fmt.Sprintf(format, args...)) +} + +func transientf(format string, args ...any) error { + return fmt.Errorf("%w: %s", provider.ErrTransient, fmt.Sprintf(format, args...)) +} + +// apiError is a non-2xx answer from the API. It carries the status and the +// message only, never headers or the request URL, so no credential can leak +// through it (spec entry 14). +type apiError struct { + Status int + Message string + // Errors is a 400's per-parameter map. + Errors map[string]string +} + +func (e *apiError) Error() string { + msg := fmt.Sprintf("proxmox API %d: %s", e.Status, e.Message) + if len(e.Errors) == 0 { + return msg + } + fields := make([]string, 0, len(e.Errors)) + for _, k := range slices.Sorted(maps.Keys(e.Errors)) { + fields = append(fields, k+": "+e.Errors[k]) + } + return msg + " (" + strings.Join(fields, "; ") + ")" +} + +// newAPIError builds an apiError from a response: the body's message with +// its trailing newline trimmed, or the status reason phrase when the body +// has none, as for a 401 (spec entry 13, fact F1). +func newAPIError(status int, reason string, body []byte) *apiError { + var b struct { + Message string `json:"message"` + Errors map[string]string `json:"errors"` + } + _ = json.Unmarshal(body, &b) + e := &apiError{Status: status, Message: strings.TrimSpace(b.Message)} + if e.Message == "" { + e.Message = strings.TrimSpace(reason) + } + if e.Message == "" { + e.Message = http.StatusText(status) + } + if len(b.Errors) > 0 { + e.Errors = make(map[string]string, len(b.Errors)) + for k, v := range b.Errors { + e.Errors[k] = strings.TrimSpace(v) + } + } + return e +} + +// The 500 messages the provider acts on, each backed by a captured response +// under testdata/. +var ( + vmConfigMissing = regexp.MustCompile(`^Configuration file '[^']*' does not exist$`) + vmNotRunning = regexp.MustCompile(`^VM [0-9]+ is not running$`) +) + +const ( + msgLocked = "VM is locked" + msgAlreadyExists = "config file already exists" + msgAgentNotRunning = "QEMU guest agent is not running" + msgAgentNotConfigured = "No QEMU guest agent configured" +) + +// server500 returns the message of err when it is, or wraps, an apiError +// with status 500. +func server500(err error) (string, bool) { + var ae *apiError + if !errors.As(err, &ae) || ae.Status != http.StatusInternalServerError { + return "", false + } + return ae.Message, true +} + +// isNotExist reports the missing-VM answer: the VM's config file is gone. +func isNotExist(err error) bool { + msg, ok := server500(err) + return ok && vmConfigMissing.MatchString(msg) +} + +// isLocked reports a VM held by a running task ("VM is locked (clone)"). +func isLocked(err error) bool { + msg, ok := server500(err) + return ok && strings.HasPrefix(msg, msgLocked) +} + +// isAlreadyExists reports a clone onto a VMID that is taken. +func isAlreadyExists(err error) bool { + msg, ok := server500(err) + return ok && strings.HasSuffix(msg, msgAlreadyExists) +} + +// isAgentNotRunning reports a VM with the agent enabled whose agent does not +// answer yet. +func isAgentNotRunning(err error) bool { + msg, ok := server500(err) + return ok && msg == msgAgentNotRunning +} + +// isAgentNotConfigured reports a VM whose config no longer enables the agent. +func isAgentNotConfigured(err error) bool { + msg, ok := server500(err) + return ok && msg == msgAgentNotConfigured +} + +// isVMNotRunning reports an agent call on a VM that is stopped. It is +// matched on the whole message because " is not running" alone is also a +// suffix of the agent-not-running answer. +func isVMNotRunning(err error) bool { + msg, ok := server500(err) + return ok && vmNotRunning.MatchString(msg) +} + +// classify maps any error reaching the provider's boundary to one of the +// four contract sentinels, per spec entry 13. op is the contract operation: +// a vanished VM inside Create is Transient, because the contract forbids +// NotFound from Create. +func classify(op string, err error) error { + if err == nil { + return provider.ErrTransient + } + for _, class := range []error{provider.ErrNotFound, provider.ErrQuotaExceeded, provider.ErrPermanent, provider.ErrTransient} { + if errors.Is(err, class) { + return class + } + } + var ae *apiError + if errors.As(err, &ae) { + return classifyAPI(op, err, ae) + } + if isTLSVerification(err) { + return provider.ErrPermanent + } + // Everything else retries: context cancellation and deadlines, + // net.OpError, io.EOF, and anything unrecognised. + return provider.ErrTransient +} + +func classifyAPI(op string, err error, ae *apiError) error { + switch { + case ae.Status >= 400 && ae.Status < 500: + // 401 and 403 are credentials or privileges, 400 a request Proxmox + // will never accept: all fixed by an admin, not by retrying. + return provider.ErrPermanent + case isNotExist(err): + if op == "create" { + return provider.ErrTransient + } + return provider.ErrNotFound + default: + // Other 500s (a lock, a taken VMID escaping the allocator, agent + // states escaping Get), pveproxy's 595 and 596, and any other status. + return provider.ErrTransient + } +} + +// isTLSVerification reports a certificate the client refused: the URL does +// not match the certificate, or the CA is wrong. An admin fixes the URL or +// caFile; retrying cannot. +func isTLSVerification(err error) bool { + var ( + verr *tls.CertificateVerificationError + unknown x509.UnknownAuthorityError + hostname x509.HostnameError + invalid x509.CertificateInvalidError + ) + return errors.As(err, &verr) || errors.As(err, &unknown) || errors.As(err, &hostname) || errors.As(err, &invalid) +} diff --git a/internal/provider/proxmox/errors_test.go b/internal/provider/proxmox/errors_test.go new file mode 100644 index 0000000..f3e8fc3 --- /dev/null +++ b/internal/provider/proxmox/errors_test.go @@ -0,0 +1,197 @@ +package proxmox + +import ( + "context" + "crypto/tls" + "crypto/x509" + "errors" + "fmt" + "io" + "net" + "net/url" + "strings" + "testing" + + "gitea.home.hrajfrisbee.cz/kacerr/egress-proxies-operator/internal/provider" +) + +func TestClassify_table(t *testing.T) { + t.Parallel() + fixture := func(name string) error { return loadAPIError(t, name) } + tests := []struct { + name string + op string + err error + want error + }{ + {name: "nil is transient", op: "get", err: nil, want: provider.ErrTransient}, + + {name: "missing VM config on get", op: "get", err: fixture("err_vm_missing"), want: provider.ErrNotFound}, + {name: "missing VM status on get", op: "get", err: fixture("err_vm_missing_status"), want: provider.ErrNotFound}, + {name: "missing VM on delete", op: "delete", err: fixture("err_destroy_missing"), want: provider.ErrNotFound}, + {name: "missing VM on list", op: "list", err: fixture("err_vm_missing"), want: provider.ErrNotFound}, + {name: "missing VM inside create is transient", op: "create", err: fixture("err_vm_missing"), want: provider.ErrTransient}, + {name: "wrapped missing VM keeps its class", op: "get", err: fmt.Errorf("reading config: %w", fixture("err_vm_missing")), want: provider.ErrNotFound}, + + {name: "locked by a clone", op: "create", err: fixture("err_config_while_clone_locked"), want: provider.ErrTransient}, + {name: "VMID taken escaping the allocator", op: "create", err: fixture("err_newid_in_use"), want: provider.ErrTransient}, + {name: "agent not running escaping Get", op: "get", err: fixture("err_agent_not_running"), want: provider.ErrTransient}, + {name: "agent not configured escaping Get", op: "get", err: fixture("err_agent_vm_stopped"), want: provider.ErrTransient}, + {name: "VM not running escaping Get", op: "get", err: fixture("err_agent_vm_stopped_agent_enabled"), want: provider.ErrTransient}, + + {name: "wrong secret", op: "get", err: fixture("err_unauthorized"), want: provider.ErrPermanent}, + {name: "unknown token id", op: "list", err: fixture("err_unauthorized_unknown_token"), want: provider.ErrPermanent}, + {name: "no credentials", op: "list", err: fixture("err_no_credentials"), want: provider.ErrPermanent}, + {name: "missing privilege", op: "get", err: fixture("err_forbidden"), want: provider.ErrPermanent}, + {name: "missing Datastore.Allocate", op: "delete", err: fixture("err_forbidden_volume_delete"), want: provider.ErrPermanent}, + {name: "parameter verification", op: "create", err: fixture("err_bad_param"), want: provider.ErrPermanent}, + {name: "invalid tag", op: "create", err: fixture("err_tag_invalid"), want: provider.ErrPermanent}, + {name: "upload without .iso", op: "create", err: fixture("err_upload_bad_name"), want: provider.ErrPermanent}, + {name: "clone onto a storage without images", op: "create", err: fixture("err_clone_bad_storage"), want: provider.ErrPermanent}, + {name: "any other 4xx", op: "get", err: &apiError{Status: 404, Message: "Not Found"}, want: provider.ErrPermanent}, + + {name: "other 500", op: "get", err: &apiError{Status: 500, Message: "got timeout"}, want: provider.ErrTransient}, + {name: "storage does not exist is not a missing VM", op: "delete", err: &apiError{Status: 500, Message: "storage 'x' does not exist"}, want: provider.ErrTransient}, + {name: "502", op: "get", err: &apiError{Status: 502, Message: "Bad Gateway"}, want: provider.ErrTransient}, + {name: "pveproxy 595", op: "get", err: &apiError{Status: 595, Message: "No route to host"}, want: provider.ErrTransient}, + {name: "pveproxy 596", op: "list", err: &apiError{Status: 596, Message: "Connection timed out"}, want: provider.ErrTransient}, + + {name: "unknown authority", op: "get", err: x509.UnknownAuthorityError{}, want: provider.ErrPermanent}, + {name: "hostname mismatch", op: "create", err: x509.HostnameError{Certificate: &x509.Certificate{}, Host: "192.168.2.8"}, want: provider.ErrPermanent}, + {name: "expired certificate", op: "list", err: x509.CertificateInvalidError{Reason: x509.Expired}, want: provider.ErrPermanent}, + {name: "tls verification wrapped in url.Error", op: "get", err: &url.Error{Op: "Get", URL: "https://pve", Err: &tls.CertificateVerificationError{Err: x509.UnknownAuthorityError{}}}, want: provider.ErrPermanent}, + + {name: "context canceled", op: "get", err: context.Canceled, want: provider.ErrTransient}, + {name: "context deadline", op: "get", err: context.DeadlineExceeded, want: provider.ErrTransient}, + {name: "net.OpError", op: "get", err: &net.OpError{Op: "dial", Net: "tcp", Err: errors.New("connection refused")}, want: provider.ErrTransient}, + {name: "io.EOF", op: "list", err: io.EOF, want: provider.ErrTransient}, + {name: "io.ErrUnexpectedEOF", op: "list", err: io.ErrUnexpectedEOF, want: provider.ErrTransient}, + {name: "anything else is transient", op: "get", err: errors.New("anything"), want: provider.ErrTransient}, + + {name: "pre-classified NotFound stays", op: "create", err: fmt.Errorf("x: %w", provider.ErrNotFound), want: provider.ErrNotFound}, + {name: "pre-classified Quota stays", op: "create", err: fmt.Errorf("x: %w", provider.ErrQuotaExceeded), want: provider.ErrQuotaExceeded}, + {name: "pre-classified Permanent stays", op: "get", err: permanentf("bad %s", "thing"), want: provider.ErrPermanent}, + {name: "pre-classified Transient beats a 4xx underneath", op: "get", err: fmt.Errorf("%w: %w", provider.ErrTransient, fixture("err_forbidden")), want: provider.ErrTransient}, + } + for _, tc := range tests { + t.Run(tc.name, func(t *testing.T) { + t.Parallel() + if got := classify(tc.op, tc.err); !errors.Is(got, tc.want) { + t.Errorf("classify(%q, %v) = %v, want %v", tc.op, tc.err, got, tc.want) + } + }) + } +} + +// TestAPIError_matchers runs every matcher over every captured error, so a +// matcher that also fires on another message fails here. +func TestAPIError_matchers(t *testing.T) { + t.Parallel() + matchers := map[string]func(error) bool{ + "isNotExist": isNotExist, + "isLocked": isLocked, + "isAlreadyExists": isAlreadyExists, + "isAgentNotRunning": isAgentNotRunning, + "isAgentNotConfigured": isAgentNotConfigured, + "isVMNotRunning": isVMNotRunning, + } + tests := []struct { + capture string + want string // the one matcher that fires, or "" for none + }{ + {"err_vm_missing", "isNotExist"}, + {"err_vm_missing_status", "isNotExist"}, + {"err_destroy_missing", "isNotExist"}, + {"err_config_while_clone_locked", "isLocked"}, + {"err_newid_in_use", "isAlreadyExists"}, + {"err_agent_not_running", "isAgentNotRunning"}, + {"err_agent_vm_stopped", "isAgentNotConfigured"}, + {"err_agent_vm_stopped_agent_enabled", "isVMNotRunning"}, + {"err_unauthorized", ""}, + {"err_forbidden", ""}, + {"err_bad_param", ""}, + {"err_clone_bad_storage", ""}, + } + for _, tc := range tests { + t.Run(tc.capture, func(t *testing.T) { + t.Parallel() + err := fmt.Errorf("wrapped: %w", loadAPIError(t, tc.capture)) + for name, match := range matchers { + if got, want := match(err), name == tc.want; got != want { + t.Errorf("%s(%v) = %v, want %v", name, err, got, want) + } + } + }) + } + if isNotExist(errors.New("Configuration file 'x' does not exist")) { + t.Error("isNotExist matched an error that is not an apiError") + } +} + +func TestNewAPIError_fromCaptures(t *testing.T) { + t.Parallel() + tests := []struct { + capture string + wantStatus int + wantMessage string + wantErrors map[string]string + }{ + {capture: "err_vm_missing", wantStatus: 500, wantMessage: "Configuration file 'nodes/pv1-1/qemu-server/999999.conf' does not exist"}, + {capture: "err_unauthorized", wantStatus: 401, wantMessage: "Authentication failed!"}, + {capture: "err_no_credentials", wantStatus: 401, wantMessage: "No ticket"}, + {capture: "err_forbidden", wantStatus: 403, wantMessage: "Permission check failed (/nodes/pv1-1, Sys.Syslog)"}, + {capture: "err_bad_param", wantStatus: 400, wantMessage: "Parameter verification failed.", wantErrors: map[string]string{"newid": "type check ('integer') failed - got 'abc'"}}, + } + for _, tc := range tests { + t.Run(tc.capture, func(t *testing.T) { + t.Parallel() + got := loadAPIError(t, tc.capture) + if got.Status != tc.wantStatus || got.Message != tc.wantMessage { + t.Errorf("newAPIError = {%d %q}, want {%d %q}", got.Status, got.Message, tc.wantStatus, tc.wantMessage) + } + if len(got.Errors) != len(tc.wantErrors) { + t.Fatalf("Errors = %v, want %v", got.Errors, tc.wantErrors) + } + for k, v := range tc.wantErrors { + if got.Errors[k] != v { + t.Errorf("Errors[%q] = %q, want %q", k, got.Errors[k], v) + } + } + }) + } +} + +func TestNewAPIError_fallbacks(t *testing.T) { + t.Parallel() + if got := newAPIError(503, "", []byte("")); got.Message != "Service Unavailable" { + t.Errorf("no reason and no JSON body: Message = %q, want the status text", got.Message) + } + if got := newAPIError(500, "reason text", []byte(`{"data":null}`)); got.Message != "reason text" { + t.Errorf("body without message: Message = %q, want the reason phrase", got.Message) + } +} + +func TestAPIError_errorString(t *testing.T) { + t.Parallel() + got := loadAPIError(t, "err_bad_param").Error() + want := "proxmox API 400: Parameter verification failed. (newid: type check ('integer') failed - got 'abc')" + if got != want { + t.Errorf("Error() = %q, want %q", got, want) + } + e := &apiError{Status: 400, Message: "m", Errors: map[string]string{"b": "2", "a": "1"}} + if got := e.Error(); !strings.HasSuffix(got, "(a: 1; b: 2)") { + t.Errorf("Error() = %q, want field errors sorted by key", got) + } +} + +func TestPermanentfAndTransientf(t *testing.T) { + t.Parallel() + perm := permanentf("node %q missing", "x") + if !errors.Is(perm, provider.ErrPermanent) || perm.Error() != `provider: permanent error: node "x" missing` { + t.Errorf("permanentf() = %q, want ErrPermanent with the message", perm) + } + trans := transientf("retry %d", 3) + if !errors.Is(trans, provider.ErrTransient) || trans.Error() != "provider: transient error: retry 3" { + t.Errorf("transientf() = %q, want ErrTransient with the message", trans) + } +} diff --git a/internal/provider/proxmox/fixtures_test.go b/internal/provider/proxmox/fixtures_test.go new file mode 100644 index 0000000..a49bc32 --- /dev/null +++ b/internal/provider/proxmox/fixtures_test.go @@ -0,0 +1,57 @@ +package proxmox + +import ( + "encoding/json" + "os" + "path/filepath" + "testing" +) + +// capture is one response recorded from the real host in Phase 0: the +// status, the reason phrase after the status code, and the raw body. +type capture struct { + Status int `json:"status"` + Reason string `json:"reason"` + Body string `json:"body"` +} + +// readCapture loads testdata/.json and fails the test when it is +// missing, so every string a test relies on is backed by a real response. +func readCapture(t *testing.T, name string) capture { + t.Helper() + raw, err := os.ReadFile(filepath.Join("testdata", name+".json")) + if err != nil { + t.Fatalf("reading capture %s: %v", name, err) + } + var c capture + if err := json.Unmarshal(raw, &c); err != nil { + t.Fatalf("parsing capture %s: %v", name, err) + } + return c +} + +// loadAPIError turns an error capture into the apiError the client builds. +func loadAPIError(t *testing.T, name string) *apiError { + t.Helper() + c := readCapture(t, name) + if c.Status < 400 { + t.Fatalf("capture %s has status %d, want an error status", name, c.Status) + } + return newAPIError(c.Status, c.Reason, []byte(c.Body)) +} + +// decodeData decodes the "data" member of a successful capture into T. +func decodeData[T any](t *testing.T, name string) T { + t.Helper() + c := readCapture(t, name) + if c.Status != 200 { + t.Fatalf("capture %s has status %d, want 200", name, c.Status) + } + var env struct { + Data T `json:"data"` + } + if err := json.Unmarshal([]byte(c.Body), &env); err != nil { + t.Fatalf("decoding capture %s: %v", name, err) + } + return env.Data +} diff --git a/internal/provider/proxmox/id.go b/internal/provider/proxmox/id.go new file mode 100644 index 0000000..f0c48d3 --- /dev/null +++ b/internal/provider/proxmox/id.go @@ -0,0 +1,42 @@ +package proxmox + +import ( + "hash/fnv" + "strconv" + "strings" + + "gitea.home.hrajfrisbee.cz/kacerr/egress-proxies-operator/internal/provider" +) + +const idPrefix = "vms" + +// formatProviderID gives vms//: location-free, because VMs can +// migrate between nodes (spec entry 1). +func formatProviderID(vmid int, name string) string { + return idPrefix + "/" + strconv.Itoa(vmid) + "/" + name +} + +// parseProviderID is strict: exactly three segments, the literal "vms", a +// decimal VMID without leading zeros in Proxmox's range, and a non-empty +// name. Anything else is ErrPermanent. +func parseProviderID(id string) (int, string, error) { + parts := strings.Split(id, "/") + if len(parts) != 3 || parts[0] != idPrefix || parts[2] == "" { + return 0, "", permanentf("malformed providerID %q, want vms//", id) + } + vmid, err := strconv.Atoi(parts[1]) + if err != nil || strconv.Itoa(vmid) != parts[1] || vmid < provider.ProxmoxMinVMID || vmid > provider.ProxmoxMaxVMID { + return 0, "", permanentf("malformed providerID %q: %q is not a VMID", id, parts[1]) + } + return vmid, parts[2], nil +} + +// vmidFor is the attempt-th VMID candidate for a Proxy UID: a stable hash of +// the UID placed in [min, max], moving forward and wrapping as attempt grows, +// so a retry of the same Create lands on the same VMID (spec entry 2). +func vmidFor(uid string, min, max, attempt int) int { + h := fnv.New64a() + _, _ = h.Write([]byte(uid)) + span := uint64(max - min + 1) + return min + int((h.Sum64()%span+uint64(attempt)%span)%span) +} diff --git a/internal/provider/proxmox/id_test.go b/internal/provider/proxmox/id_test.go new file mode 100644 index 0000000..e86a84a --- /dev/null +++ b/internal/provider/proxmox/id_test.go @@ -0,0 +1,89 @@ +package proxmox + +import ( + "errors" + "testing" + + "gitea.home.hrajfrisbee.cz/kacerr/egress-proxies-operator/internal/provider" +) + +func TestFormatProviderID_roundTrip(t *testing.T) { + t.Parallel() + for _, vmid := range []int{100, 150042, 999999999} { + id := formatProviderID(vmid, "proxy-abcdefghijklmnop") + gotVMID, gotName, err := parseProviderID(id) + if err != nil || gotVMID != vmid || gotName != "proxy-abcdefghijklmnop" { + t.Errorf("parseProviderID(%q) = %d, %q, %v; want %d, proxy-abcdefghijklmnop, nil", id, gotVMID, gotName, err, vmid) + } + } + if got := formatProviderID(150042, "proxy-x"); got != "vms/150042/proxy-x" { + t.Errorf("formatProviderID() = %q, want vms/150042/proxy-x", got) + } +} + +func TestParseProviderID_malformed(t *testing.T) { + t.Parallel() + for _, id := range []string{ + "", + "vms", + "vms/150042", + "vms/150042/", + "vms//proxy-x", + "vm/150042/proxy-x", + "hosts/beelink/domains/proxy-x", + "vms/150042/proxy-x/extra", + "vms/abc/proxy-x", + "vms/0150042/proxy-x", + "vms/+150042/proxy-x", + "vms/-5/proxy-x", + "vms/99/proxy-x", + "vms/1000000000/proxy-x", + } { + t.Run(id, func(t *testing.T) { + t.Parallel() + _, _, err := parseProviderID(id) + if !errors.Is(err, provider.ErrPermanent) { + t.Errorf("parseProviderID(%q) error = %v, want ErrPermanent", id, err) + } + }) + } +} + +func TestVMIDFor_deterministicAndInRange(t *testing.T) { + t.Parallel() + uids := []string{"3f1c2d7e-8a44-4f1b-9a6c-1d2e3f4a5b6c", "00000000-0000-0000-0000-000000000000", "x"} + for _, uid := range uids { + first := vmidFor(uid, 100000, 199999, 0) + if first < 100000 || first > 199999 { + t.Errorf("vmidFor(%q) = %d, want within 100000..199999", uid, first) + } + if again := vmidFor(uid, 100000, 199999, 0); again != first { + t.Errorf("vmidFor(%q) = %d then %d, want the same VMID every time", uid, first, again) + } + } + if vmidFor(uids[0], 100000, 199999, 0) == vmidFor(uids[1], 100000, 199999, 0) { + t.Error("two different UIDs got the same first VMID; the hash does not spread") + } +} + +func TestVMIDFor_probesForwardAndWraps(t *testing.T) { + t.Parallel() + const min, max = 500, 509 + uid := "3f1c2d7e-8a44-4f1b-9a6c-1d2e3f4a5b6c" + start := vmidFor(uid, min, max, 0) + seen := map[int]bool{} + for attempt := range 10 { + got := vmidFor(uid, min, max, attempt) + want := min + (start-min+attempt)%10 + if got != want { + t.Errorf("attempt %d: vmidFor = %d, want %d (one step forward, wrapping at max)", attempt, got, want) + } + seen[got] = true + } + if len(seen) != 10 { + t.Errorf("10 attempts over a range of 10 visited %d VMIDs, want all 10", len(seen)) + } + if got := vmidFor(uid, min, max, 10); got != start { + t.Errorf("attempt 10 = %d, want to wrap back to %d", got, start) + } +} diff --git a/internal/provider/proxmox/mark.go b/internal/provider/proxmox/mark.go new file mode 100644 index 0000000..76954f0 --- /dev/null +++ b/internal/provider/proxmox/mark.go @@ -0,0 +1,54 @@ +package proxmox + +import ( + "strings" + "time" + + "gitea.home.hrajfrisbee.cz/kacerr/egress-proxies-operator/internal/provider" +) + +// createdAtKey is the third line of the mark; the first two use the +// contract's label names. +const createdAtKey = "proxy-operator-created-at" + +// instanceMark is the GC mark as read back from a VM's description (spec +// entry 8). +type instanceMark struct { + Managed bool + UID string + CreatedAt time.Time +} + +// formatDescription renders the mark written in the clone call. +func formatDescription(uid string, createdAt time.Time) string { + return provider.LabelManaged + "=" + provider.LabelManagedYes + "\n" + + provider.LabelUID + "=" + uid + "\n" + + createdAtKey + "=" + createdAt.UTC().Format(time.RFC3339) +} + +// parseDescription reads key=value lines, tolerating CRLF, blank lines and +// lines an admin added. The first occurrence of a key wins. Managed only +// when the value is exactly LabelManagedYes; an unparsable created-at gives +// the zero time, which GC treats as old. +func parseDescription(desc string) instanceMark { + var m instanceMark + seen := map[string]bool{} + for line := range strings.SplitSeq(desc, "\n") { + key, value, ok := strings.Cut(strings.TrimSpace(line), "=") + if !ok || seen[key] { + continue + } + seen[key] = true + switch key { + case provider.LabelManaged: + m.Managed = value == provider.LabelManagedYes + case provider.LabelUID: + m.UID = value + case createdAtKey: + if t, err := time.Parse(time.RFC3339, value); err == nil { + m.CreatedAt = t + } + } + } + return m +} diff --git a/internal/provider/proxmox/mark_test.go b/internal/provider/proxmox/mark_test.go new file mode 100644 index 0000000..56004e3 --- /dev/null +++ b/internal/provider/proxmox/mark_test.go @@ -0,0 +1,64 @@ +package proxmox + +import ( + "testing" + "time" +) + +func TestFormatDescription_keys(t *testing.T) { + t.Parallel() + created := time.Date(2026, 9, 23, 8, 0, 0, 0, time.FixedZone("CEST", 2*3600)) + got := formatDescription("3f1c2d7e-uid", created) + want := "proxy-operator-managed=true\nproxy-operator-uid=3f1c2d7e-uid\nproxy-operator-created-at=2026-09-23T06:00:00Z" + if got != want { + t.Errorf("formatDescription() =\n%s\nwant\n%s", got, want) + } + m := parseDescription(got) + if !m.Managed || m.UID != "3f1c2d7e-uid" || !m.CreatedAt.Equal(created) { + t.Errorf("parseDescription(formatDescription()) = %+v, want managed, the UID and the time back", m) + } +} + +// TestParseDescription_fromClone reads the mark back from the config of the +// VM cloned in Phase 0, as Proxmox stored it. +func TestParseDescription_fromClone(t *testing.T) { + t.Parallel() + cfg := decodeData[map[string]any](t, "vm_config_clone") + desc, _ := cfg["description"].(string) + m := parseDescription(desc) + want := time.Date(2026, 9, 23, 7, 0, 0, 0, time.UTC) + if !m.Managed || m.UID != "00000000-0000-0000-0000-00000000recon" || !m.CreatedAt.Equal(want) { + t.Errorf("parseDescription(captured) = %+v, want the recon mark", m) + } +} + +func TestParseDescription_table(t *testing.T) { + t.Parallel() + ts := time.Date(2026, 9, 23, 7, 0, 0, 0, time.UTC) + tests := []struct { + name string + desc string + want instanceMark + }{ + {name: "empty", desc: "", want: instanceMark{}}, + {name: "unmarked free text", desc: "Ubuntu VM for testing", want: instanceMark{}}, + {name: "managed with UID", desc: "proxy-operator-managed=true\nproxy-operator-uid=u1\nproxy-operator-created-at=2026-09-23T07:00:00Z", want: instanceMark{Managed: true, UID: "u1", CreatedAt: ts}}, + {name: "managed with empty UID", desc: "proxy-operator-managed=true\nproxy-operator-uid=", want: instanceMark{Managed: true}}, + {name: "managed without UID line", desc: "proxy-operator-managed=true", want: instanceMark{Managed: true}}, + {name: "managed value must be exactly true", desc: "proxy-operator-managed=yes\nproxy-operator-uid=u1", want: instanceMark{UID: "u1"}}, + {name: "unparsable created-at gives zero time", desc: "proxy-operator-managed=true\nproxy-operator-created-at=yesterday", want: instanceMark{Managed: true}}, + {name: "CRLF line endings", desc: "proxy-operator-managed=true\r\nproxy-operator-uid=u1\r\n", want: instanceMark{Managed: true, UID: "u1"}}, + {name: "admin lines kept around the mark", desc: "owner: team-crawl\n\nproxy-operator-managed=true\nnote=do not touch\nproxy-operator-uid=u1", want: instanceMark{Managed: true, UID: "u1"}}, + {name: "first occurrence of a key wins", desc: "proxy-operator-uid=u1\nproxy-operator-uid=u2\nproxy-operator-managed=true", want: instanceMark{Managed: true, UID: "u1"}}, + {name: "surrounding spaces trimmed", desc: " proxy-operator-managed=true \n proxy-operator-uid=u1", want: instanceMark{Managed: true, UID: "u1"}}, + } + for _, tc := range tests { + t.Run(tc.name, func(t *testing.T) { + t.Parallel() + got := parseDescription(tc.desc) + if got.Managed != tc.want.Managed || got.UID != tc.want.UID || !got.CreatedAt.Equal(tc.want.CreatedAt) { + t.Errorf("parseDescription(%q) = %+v, want %+v", tc.desc, got, tc.want) + } + }) + } +} diff --git a/internal/provider/proxmox/proxmox.go b/internal/provider/proxmox/proxmox.go new file mode 100644 index 0000000..d3875b5 --- /dev/null +++ b/internal/provider/proxmox/proxmox.go @@ -0,0 +1,173 @@ +// Package proxmox implements the provider contract on a Proxmox VE cluster +// through its REST API: VMs are full clones of prepared templates, seeded +// with a NoCloud ISO uploaded to a storage and addressed through the guest +// agent. The design is in docs/proxmox-provider-spec.md. +package proxmox + +import ( + "context" + "os" + "time" + + "github.com/go-logr/logr" + logf "sigs.k8s.io/controller-runtime/pkg/log" + + "gitea.home.hrajfrisbee.cz/kacerr/egress-proxies-operator/internal/provider" +) + +const ( + defaultVMIDMin = 100000 + defaultVMIDMax = 199999 + defaultRequestTimeoutSeconds = 30 +) + +// budgets bounds every wait of the provider (spec entry 18); tests shrink +// them. +type budgets struct { + clone time.Duration // the clone task, and waiting for a VM's lock + upload time.Duration + resize time.Duration + start time.Duration + stop time.Duration + destroy time.Duration + seedDelete time.Duration + running time.Duration // from the start task's end to status running + poll time.Duration // task and status poll interval +} + +func defaultBudgets() budgets { + return budgets{ + clone: 10 * time.Minute, + upload: 2 * time.Minute, + resize: 2 * time.Minute, + start: 2 * time.Minute, + stop: 2 * time.Minute, + destroy: 5 * time.Minute, + seedDelete: time.Minute, + running: 60 * time.Second, + poll: time.Second, + } +} + +// Provider is one configured proxmox entry: one cluster reached through one +// API URL with one token. +type Provider struct { + name string + cfg provider.ProxmoxConfig + api api + now func() time.Time + budgets budgets +} + +var _ provider.Provider = (*Provider)(nil) + +// New builds a Provider from pc.Proxmox. It never makes a request: the token +// file is only checked for existence, and a missing one is a warning, not an +// error, because the Secret mount is optional and may land later. +func New(ctx context.Context, pc provider.ProviderConfig) (provider.Provider, error) { + cfg := provider.ProxmoxConfig{} + if pc.Proxmox != nil { + cfg = *pc.Proxmox + } + // The HTTP client behind the api seam arrives in Phase 4 of the plan; + // until then every method is a stub that makes no request. + p := newProvider(pc.Name, cfg, nil) + log := p.logger(ctx) + if _, err := os.Stat(p.cfg.TokenFile); err != nil { + log.Info("WARNING: token file is not readable; every call fails until it is", "path", p.cfg.TokenFile, "reason", err.Error()) + } + if p.cfg.InsecureSkipTLSVerify { + log.Info("WARNING: insecureSkipTLSVerify is set; the API's certificate is not verified") + } + return p, nil +} + +// newProvider is the constructor tests use, with a in-memory api. +func newProvider(name string, cfg provider.ProxmoxConfig, a api) *Provider { + return &Provider{ + name: name, + cfg: withDefaults(cfg), + api: a, + now: time.Now, + budgets: defaultBudgets(), + } +} + +// withDefaults returns cfg with every optional field filled in, so the rest +// of the package never re-checks for an empty value. Slices, maps and +// pointers are copied, not shared with the caller. +func withDefaults(cfg provider.ProxmoxConfig) provider.ProxmoxConfig { + cfg.Nodes = append([]string(nil), cfg.Nodes...) + if cfg.VMIDRange == nil { + cfg.VMIDRange = &provider.ProxmoxVMIDRange{Min: defaultVMIDMin, Max: defaultVMIDMax} + } else { + r := *cfg.VMIDRange + cfg.VMIDRange = &r + } + if cfg.RequestTimeoutSeconds == 0 { + cfg.RequestTimeoutSeconds = defaultRequestTimeoutSeconds + } + install := true + if cfg.InstallGuestAgent != nil { + install = *cfg.InstallGuestAgent + } + cfg.InstallGuestAgent = &install + templates := make(map[string]provider.ProxmoxTemplate, len(cfg.Templates)) + for k, v := range cfg.Templates { + templates[k] = v + } + cfg.Templates = templates + machineTypes := make(map[string]provider.ProxmoxMachineType, len(cfg.MachineTypes)) + for k, v := range cfg.MachineTypes { + machineTypes[k] = v + } + cfg.MachineTypes = machineTypes + return cfg +} + +func (p *Provider) logger(ctx context.Context) logr.Logger { + return logf.FromContext(ctx).WithName("proxmox").WithValues("provider", p.name) +} + +// node resolves placement.zone to a configured node. An empty zone is +// allowed only when exactly one node is configured. +func (p *Provider) node(zone string) (string, error) { + if zone == "" { + if len(p.cfg.Nodes) == 1 { + return p.cfg.Nodes[0], nil + } + return "", permanentf("placement.zone is required when proxmox.nodes lists more than one node") + } + for _, n := range p.cfg.Nodes { + if n == zone { + return n, nil + } + } + return "", permanentf("placement.zone %q is not in proxmox.nodes", zone) +} + +// wrapErr is the single exit for errors: it classifies err and wraps it with +// the configured provider name, op and id. +func (p *Provider) wrapErr(op, id string, err error) error { + return provider.Wrap(classify(op, err), op, p.name, id, err) +} + +// The four contract methods are stubs until the phases that implement them +// (Get and ListByTag in Phase 5, Create in 6, Delete in 7). Each answers +// Transient, so a Proxy on this provider retries rather than latching. + +func (p *Provider) Create(_ context.Context, req provider.CreateRequest) (string, error) { + return "", p.wrapErr("create", "", transientf("proxmox Create is not implemented yet (request %s)", req.Name)) +} + +func (p *Provider) Get(_ context.Context, id string) (*provider.Instance, error) { + return nil, p.wrapErr("get", id, transientf("proxmox Get is not implemented yet")) +} + +func (p *Provider) Delete(_ context.Context, id string) error { + return p.wrapErr("delete", id, transientf("proxmox Delete is not implemented yet")) +} + +func (p *Provider) ListByTag(_ context.Context) ([]provider.Instance, error) { + return nil, p.wrapErr("list", "", transientf("proxmox ListByTag is not implemented yet")) +} diff --git a/internal/provider/proxmox/proxmox_test.go b/internal/provider/proxmox/proxmox_test.go new file mode 100644 index 0000000..1b25f70 --- /dev/null +++ b/internal/provider/proxmox/proxmox_test.go @@ -0,0 +1,208 @@ +package proxmox + +import ( + "context" + "errors" + "strings" + "sync" + "testing" + "time" + + "github.com/go-logr/logr/funcr" + logf "sigs.k8s.io/controller-runtime/pkg/log" + + "gitea.home.hrajfrisbee.cz/kacerr/egress-proxies-operator/internal/provider" +) + +// captureContext collects funcr output at V(2). +func captureContext() (context.Context, *[]string) { + lines := &[]string{} + var mu sync.Mutex + log := funcr.New(func(prefix, args string) { + mu.Lock() + defer mu.Unlock() + *lines = append(*lines, prefix+" "+args) + }, funcr.Options{Verbosity: 2}) + return logf.IntoContext(context.Background(), log), lines +} + +func testConfig() provider.ProxmoxConfig { + return provider.ProxmoxConfig{ + URL: "https://192.0.2.8:8006", + TokenFile: "/nonexistent/proxmox/token", + Nodes: []string{"pv1-1"}, + ISOStorage: "proxy-seeds", + Templates: map[string]provider.ProxmoxTemplate{"debian-13": {VMID: 9001}}, + MachineTypes: map[string]provider.ProxmoxMachineType{"small": {Cores: 1, MemoryMiB: 1024}}, + } +} + +func TestNew_doesNotDialAndWarnsOnMissingToken(t *testing.T) { + t.Parallel() + ctx, lines := captureContext() + cfg := testConfig() + start := time.Now() + p, err := New(ctx, provider.ProviderConfig{Name: "proxmox-home", Type: "proxmox", Proxmox: &cfg}) + if err != nil || p == nil { + t.Fatalf("New() = %v, %v; want a provider", p, err) + } + if elapsed := time.Since(start); elapsed > 100*time.Millisecond { + t.Errorf("New() took %v, want under 100ms (it must not make a request)", elapsed) + } + var warnings int + for _, l := range *lines { + if strings.Contains(l, "WARNING:") && strings.Contains(l, "token file") { + warnings++ + } + if !strings.Contains(l, `"provider"="proxmox-home"`) { + t.Errorf("log line lacks the provider value: %s", l) + } + } + if warnings != 1 { + t.Errorf("got %d token WARNING: lines, want 1:\n%s", warnings, strings.Join(*lines, "\n")) + } +} + +func TestNew_insecureSkipTLSVerifyWarns(t *testing.T) { + t.Parallel() + ctx, lines := captureContext() + cfg := testConfig() + cfg.InsecureSkipTLSVerify = true + if _, err := New(ctx, provider.ProviderConfig{Name: "proxmox-home", Type: "proxmox", Proxmox: &cfg}); err != nil { + t.Fatalf("New() error = %v, want nil", err) + } + for _, l := range *lines { + if strings.Contains(l, "WARNING:") && strings.Contains(l, "not verified") { + return + } + } + t.Errorf("no insecureSkipTLSVerify warning in:\n%s", strings.Join(*lines, "\n")) +} + +func TestNew_nilBlockDoesNotPanic(t *testing.T) { + t.Parallel() + ctx, _ := captureContext() + p, err := New(ctx, provider.ProviderConfig{Name: "proxmox-empty", Type: "proxmox"}) + if err != nil { + t.Fatalf("New() error = %v, want nil", err) + } + pp, ok := p.(*Provider) + if !ok { + t.Fatalf("New() = %T, want *Provider", p) + } + if pp.name != "proxmox-empty" || pp.cfg.VMIDRange == nil || pp.now == nil { + t.Errorf("provider = %+v, want its name, defaults and a clock", pp) + } +} + +func TestWithDefaults_table(t *testing.T) { + t.Parallel() + off := false + tests := []struct { + name string + in provider.ProxmoxConfig + wantRange provider.ProxmoxVMIDRange + wantTimeout int + wantAgent bool + }{ + {name: "empty fills every default", wantRange: provider.ProxmoxVMIDRange{Min: 100000, Max: 199999}, wantTimeout: 30, wantAgent: true}, + { + name: "explicit values kept", + in: provider.ProxmoxConfig{VMIDRange: &provider.ProxmoxVMIDRange{Min: 500, Max: 600}, RequestTimeoutSeconds: 45, InstallGuestAgent: &off}, + wantRange: provider.ProxmoxVMIDRange{Min: 500, Max: 600}, + wantTimeout: 45, + wantAgent: false, + }, + } + for _, tc := range tests { + t.Run(tc.name, func(t *testing.T) { + t.Parallel() + got := withDefaults(tc.in) + if *got.VMIDRange != tc.wantRange || got.RequestTimeoutSeconds != tc.wantTimeout || *got.InstallGuestAgent != tc.wantAgent { + t.Errorf("withDefaults() = range %+v timeout %d agent %v; want %+v %d %v", + *got.VMIDRange, got.RequestTimeoutSeconds, *got.InstallGuestAgent, tc.wantRange, tc.wantTimeout, tc.wantAgent) + } + }) + } +} + +func TestWithDefaults_doesNotShareWithCaller(t *testing.T) { + t.Parallel() + in := testConfig() + in.VMIDRange = &provider.ProxmoxVMIDRange{Min: 500, Max: 600} + got := withDefaults(in) + got.Nodes[0] = "changed" + got.VMIDRange.Min = 1 + got.Templates["debian-13"] = provider.ProxmoxTemplate{VMID: 1} + got.MachineTypes["small"] = provider.ProxmoxMachineType{} + if in.Nodes[0] != "pv1-1" || in.VMIDRange.Min != 500 || in.Templates["debian-13"].VMID != 9001 || in.MachineTypes["small"].Cores != 1 { + t.Errorf("mutating the result changed the caller's config: %+v", in) + } +} + +func TestNode_table(t *testing.T) { + t.Parallel() + tests := []struct { + name string + nodes []string + zone string + want string + wantErr string + }{ + {name: "one node and empty zone selects it", nodes: []string{"pv1-1"}, zone: "", want: "pv1-1"}, + {name: "several nodes and empty zone is permanent", nodes: []string{"pv1-1", "pv1-2"}, zone: "", wantErr: "placement.zone is required"}, + {name: "known node", nodes: []string{"pv1-1", "pv1-2"}, zone: "pv1-2", want: "pv1-2"}, + {name: "unknown node is permanent", nodes: []string{"pv1-1"}, zone: "pv9", wantErr: `placement.zone "pv9" is not in proxmox.nodes`}, + } + for _, tc := range tests { + t.Run(tc.name, func(t *testing.T) { + t.Parallel() + cfg := testConfig() + cfg.Nodes = tc.nodes + got, err := newProvider("proxmox-home", cfg, nil).node(tc.zone) + if tc.wantErr == "" { + if err != nil || got != tc.want { + t.Errorf("node(%q) = %q, %v; want %q, nil", tc.zone, got, err, tc.want) + } + return + } + if !errors.Is(err, provider.ErrPermanent) || !strings.Contains(err.Error(), tc.wantErr) { + t.Errorf("node(%q) error = %v, want ErrPermanent containing %q", tc.zone, err, tc.wantErr) + } + }) + } +} + +func TestWrapErr_carriesProviderNameOpAndID(t *testing.T) { + t.Parallel() + p := newProvider("proxmox-home", testConfig(), nil) + err := p.wrapErr("get", "vms/150042/proxy-x", loadAPIError(t, "err_vm_missing")) + var pe *provider.Error + if !errors.As(err, &pe) { + t.Fatalf("wrapErr() = %T, want *provider.Error", err) + } + if pe.Provider != "proxmox-home" || pe.Op != "get" || pe.ID != "vms/150042/proxy-x" || !errors.Is(err, provider.ErrNotFound) { + t.Errorf("wrapErr() = %+v, want configured name, op, id and class NotFound", pe) + } + var ae *apiError + if !errors.As(err, &ae) || ae.Status != 500 { + t.Errorf("wrapErr() lost the underlying apiError: %v", err) + } +} + +// TestStubs_answerTransient keeps a Proxy on this provider retrying, not +// latched, until the methods are implemented in later phases. +func TestStubs_answerTransient(t *testing.T) { + t.Parallel() + ctx := context.Background() + p := newProvider("proxmox-home", testConfig(), nil) + _, createErr := p.Create(ctx, provider.CreateRequest{Name: "proxy-x"}) + _, getErr := p.Get(ctx, "vms/150042/proxy-x") + deleteErr := p.Delete(ctx, "vms/150042/proxy-x") + _, listErr := p.ListByTag(ctx) + for op, err := range map[string]error{"create": createErr, "get": getErr, "delete": deleteErr, "list": listErr} { + if !errors.Is(err, provider.ErrTransient) { + t.Errorf("%s error = %v, want ErrTransient", op, err) + } + } +} diff --git a/internal/provider/proxmox/state.go b/internal/provider/proxmox/state.go new file mode 100644 index 0000000..4807d7a --- /dev/null +++ b/internal/provider/proxmox/state.go @@ -0,0 +1,52 @@ +package proxmox + +import ( + "net/netip" + + "gitea.home.hrajfrisbee.cz/kacerr/egress-proxies-operator/internal/provider" +) + +// mapState turns status/current's status and qmpstatus plus the discovered +// address into the contract's state (spec entry 9). A VM is Running only +// with an address; prelaunch is still starting; any other qmpstatus of a +// running VM (paused, suspended, io-error, …) and any status not recognised +// is Stopped, which the reconciler answers by replacing the VM. An empty +// qmpstatus is taken as running: the field is missing, not a state. +func mapState(status, qmpStatus, ip string) provider.InstanceState { + if status != "running" { + return provider.StateStopped + } + switch qmpStatus { + case "running", "": + if ip == "" { + return provider.StateProvisioning + } + return provider.StateRunning + case "prelaunch": + return provider.StateProvisioning + default: + return provider.StateStopped + } +} + +// firstIPv4 picks the first usable IPv4 from the guest agent's interfaces, +// in the agent's order: not on the loopback interface, not loopback, +// link-local or unspecified. Empty means no address yet. +func firstIPv4(ifaces []agentInterface) string { + for _, iface := range ifaces { + if iface.Name == "lo" { + continue + } + for _, a := range iface.IPAddresses { + if a.Type != "ipv4" { + continue + } + addr, err := netip.ParseAddr(a.Address) + if err != nil || !addr.Is4() || addr.IsLoopback() || addr.IsLinkLocalUnicast() || addr.IsUnspecified() { + continue + } + return addr.String() + } + } + return "" +} diff --git a/internal/provider/proxmox/state_test.go b/internal/provider/proxmox/state_test.go new file mode 100644 index 0000000..55de709 --- /dev/null +++ b/internal/provider/proxmox/state_test.go @@ -0,0 +1,114 @@ +package proxmox + +import ( + "testing" + + "gitea.home.hrajfrisbee.cz/kacerr/egress-proxies-operator/internal/provider" +) + +func TestMapState_table(t *testing.T) { + t.Parallel() + tests := []struct { + status, qmp, ip string + want provider.InstanceState + }{ + {"running", "running", "192.168.4.147", provider.StateRunning}, + {"running", "running", "", provider.StateProvisioning}, + {"running", "", "192.168.4.147", provider.StateRunning}, + {"running", "", "", provider.StateProvisioning}, + {"running", "prelaunch", "", provider.StateProvisioning}, + {"running", "paused", "192.168.4.147", provider.StateStopped}, + {"running", "suspended", "", provider.StateStopped}, + {"running", "io-error", "", provider.StateStopped}, + {"running", "something-new", "", provider.StateStopped}, + {"stopped", "stopped", "", provider.StateStopped}, + {"stopped", "", "", provider.StateStopped}, + {"unknown", "", "", provider.StateStopped}, + {"", "", "", provider.StateStopped}, + } + for _, tc := range tests { + t.Run(tc.status+"/"+tc.qmp+"/"+tc.ip, func(t *testing.T) { + t.Parallel() + if got := mapState(tc.status, tc.qmp, tc.ip); got != tc.want { + t.Errorf("mapState(%q, %q, %q) = %q, want %q", tc.status, tc.qmp, tc.ip, got, tc.want) + } + }) + } +} + +// TestMapState_fromCaptures feeds status/current as Proxmox reported it for +// a running, a stopped and a paused VM. +func TestMapState_fromCaptures(t *testing.T) { + t.Parallel() + tests := []struct { + capture string + ip string + want provider.InstanceState + }{ + {"status_current_running", "192.168.0.150", provider.StateRunning}, + {"status_current_running", "", provider.StateProvisioning}, + {"status_current_stopped", "", provider.StateStopped}, + {"status_current_paused", "192.168.0.150", provider.StateStopped}, + } + for _, tc := range tests { + t.Run(tc.capture+"/"+tc.ip, func(t *testing.T) { + t.Parallel() + s := decodeData[vmStatus](t, tc.capture) + if got := mapState(s.Status, s.QMPStatus, tc.ip); got != tc.want { + t.Errorf("mapState(%q, %q, %q) = %q, want %q", s.Status, s.QMPStatus, tc.ip, got, tc.want) + } + }) + } +} + +func TestFirstIPv4_filters(t *testing.T) { + t.Parallel() + ip := func(addr, typ string) agentIP { return agentIP{Address: addr, Type: typ} } + tests := []struct { + name string + ifaces []agentInterface + want string + }{ + {name: "none", want: ""}, + {name: "loopback interface only", ifaces: []agentInterface{{Name: "lo", IPAddresses: []agentIP{ip("127.0.0.1", "ipv4")}}}, want: ""}, + {name: "ipv6 only", ifaces: []agentInterface{{Name: "ens18", IPAddresses: []agentIP{ip("fe80::1", "ipv6"), ip("2001:db8::5", "ipv6")}}}, want: ""}, + {name: "link-local skipped", ifaces: []agentInterface{{Name: "ens18", IPAddresses: []agentIP{ip("169.254.3.4", "ipv4"), ip("192.168.4.20", "ipv4")}}}, want: "192.168.4.20"}, + {name: "unspecified skipped", ifaces: []agentInterface{{Name: "ens18", IPAddresses: []agentIP{ip("0.0.0.0", "ipv4")}}}, want: ""}, + {name: "loopback address on another interface skipped", ifaces: []agentInterface{{Name: "dummy0", IPAddresses: []agentIP{ip("127.0.0.2", "ipv4")}}}, want: ""}, + {name: "type and literal disagree", ifaces: []agentInterface{{Name: "ens18", IPAddresses: []agentIP{ip("fe80::1", "ipv4"), ip("10.0.0.9", "ipv6")}}}, want: ""}, + {name: "unparsable skipped", ifaces: []agentInterface{{Name: "ens18", IPAddresses: []agentIP{ip("not-an-ip", "ipv4"), ip("10.0.0.9", "ipv4")}}}, want: "10.0.0.9"}, + {name: "agent order decides", ifaces: []agentInterface{ + {Name: "ens18", IPAddresses: []agentIP{ip("192.168.4.20", "ipv4")}}, + {Name: "ens19", IPAddresses: []agentIP{ip("10.0.0.9", "ipv4")}}, + }, want: "192.168.4.20"}, + } + for _, tc := range tests { + t.Run(tc.name, func(t *testing.T) { + t.Parallel() + if got := firstIPv4(tc.ifaces); got != tc.want { + t.Errorf("firstIPv4() = %q, want %q", got, tc.want) + } + }) + } +} + +// TestFirstIPv4_fromCaptures reads the guest agent's answer from the two +// Phase 0 runs, untagged and on VLAN 4. +func TestFirstIPv4_fromCaptures(t *testing.T) { + t.Parallel() + tests := map[string]string{ + "agent_network_get_interfaces": "192.168.0.150", + "agent_network_get_interfaces_vlan": "192.168.4.147", + } + for capture, want := range tests { + t.Run(capture, func(t *testing.T) { + t.Parallel() + res := decodeData[struct { + Result []agentInterface `json:"result"` + }](t, capture) + if got := firstIPv4(res.Result); got != want { + t.Errorf("firstIPv4(%s) = %q, want %q", capture, got, want) + } + }) + } +} From 0a6c8624f30c94f7279a8cbdfa7d5f47fc39e85b Mon Sep 17 00:00:00 2001 From: kacerr Date: Wed, 23 Sep 2026 14:01:27 +0000 Subject: [PATCH 06/13] provider: extract nocloud seed builder and keyedlock from kvm2 kvm2's NoCloud seed builder moves to internal/provider/nocloud (exported, with an ErrEmptyName sentinel instead of permanentf) and its keyed locks to internal/provider/keyedlock, tests moved under their existing names. kvm2 behaviour is unchanged. proxmox gains the per-name lock and the pure seed helpers (seedISOName, seedVolID, buildSeed); unlike kvm2 (review finding S5.1), a seed staging failure there is Transient, not Permanent. Co-Authored-By: Claude Opus 5.5 (1M context) --- docs/architecture.md | 1 + docs/kvm2-provider-spec.md | 2 +- docs/kvm2-provider.md | 5 +- .../2026-09-18-1502-proxmox-provider.md | 68 ++++++++- docs/providers-existing.md | 10 +- internal/provider/keyedlock/keyedlock.go | 60 ++++++++ internal/provider/keyedlock/keyedlock_test.go | 90 ++++++++++++ internal/provider/kvm2/create.go | 5 +- internal/provider/kvm2/create_test.go | 5 +- internal/provider/kvm2/image.go | 2 +- internal/provider/kvm2/kvm2.go | 63 +------- internal/provider/kvm2/kvm2_test.go | 81 ---------- internal/provider/{kvm2 => nocloud}/seed.go | 36 +++-- .../provider/{kvm2 => nocloud}/seed_test.go | 29 ++-- internal/provider/proxmox/proxmox.go | 29 +++- internal/provider/proxmox/proxmox_test.go | 21 +++ internal/provider/proxmox/seed.go | 43 ++++++ internal/provider/proxmox/seed_test.go | 139 ++++++++++++++++++ 18 files changed, 504 insertions(+), 185 deletions(-) create mode 100644 internal/provider/keyedlock/keyedlock.go create mode 100644 internal/provider/keyedlock/keyedlock_test.go rename internal/provider/{kvm2 => nocloud}/seed.go (77%) rename internal/provider/{kvm2 => nocloud}/seed_test.go (91%) create mode 100644 internal/provider/proxmox/seed.go create mode 100644 internal/provider/proxmox/seed_test.go diff --git a/docs/architecture.md b/docs/architecture.md index af8de07..03d458f 100644 --- a/docs/architecture.md +++ b/docs/architecture.md @@ -11,6 +11,7 @@ | gcp provider | `internal/provider/gcp` | library | — | Compute Engine VMs, four API calls, fire-and-forget ops | | kvm provider | `internal/provider/kvm` | library | — | VMs on libvirt hosts over SSH-forwarded sockets; image cache, overlay disks, NoCloud seeds | | kvm2 provider | `internal/provider/kvm2` | library | — | the same backend, built clean-room from the provider contract as a second type: per-RPC child spans, pool-space quota, per-call budgets, its own metadata namespace, credentials re-read on every dial | +| Shared provider helpers | `internal/provider/nocloud`, `internal/provider/keyedlock` | library | — | the NoCloud seed ISO and vendor-data builder, and one-slot semaphores by key; used by kvm2 and proxmox | | Health engine | `internal/health` | Runnable | yes | through-the-proxy probes, thresholds, transition events | | Lease store | `internal/lease` | Runnable (expiry sweep) | no | in-memory leases + cooldowns, single mutex | | Discovery API | `internal/discovery` | Runnable | no | HTTP list/lease/release/report on `:8090` | diff --git a/docs/kvm2-provider-spec.md b/docs/kvm2-provider-spec.md index d8821b8..f07b75e 100644 --- a/docs/kvm2-provider-spec.md +++ b/docs/kvm2-provider-spec.md @@ -511,7 +511,7 @@ every 15 s) marks the connection dead on failure. per host, dialled lazily, under a mutex with a per-host dial mutex so concurrent callers dial once and share the result), the per-`(host, volume)` download locks, and a lock per instance name. Both lock sets are one instance -each of `keyedLocks`: a one-slot semaphore per key, so a waiter gives up when +each of `keyedlock.Locks` (`internal/provider/keyedlock`): a one-slot semaphore per key, so a waiter gives up when its ctx ends, reference-counted so an entry is removed from the map once no caller holds or waits for it and the map does not grow with every name ever seen, while two waiters on one key always share one semaphore. Create and diff --git a/docs/kvm2-provider.md b/docs/kvm2-provider.md index 287297f..b8e8140 100644 --- a/docs/kvm2-provider.md +++ b/docs/kvm2-provider.md @@ -252,7 +252,8 @@ UIDs, and a marked domain with a matching or empty `uid` is adopted. The namespace is kvm2's own, so the `kvm` provider's domains are invisible to it (see *Migrating from kvm*). -The seed (`buildSeedISO` in `seed.go`) is an ISO 9660 image labelled +The seed (`nocloud.BuildSeedISO`, shared with the proxmox provider in +`internal/provider/nocloud`) is an ISO 9660 image labelled `cidata` with three files. `user-data` is the Proxy's `cloudInit`, **byte for byte**: any format cloud-init accepts works, including a Jinja template, a MIME archive, a shell script or a gzipped payload; the @@ -267,7 +268,7 @@ previous attempt may have been cut off mid-upload and a truncated seed would boot a VM without cloud-init. The guest agent rides along in `vendor-data` as a **shell script** -(`guestAgentScript` in `seed.go`), not a cloud-config, on purpose: +(`nocloud.GuestAgentScript`), not a cloud-config, on purpose: cloud-init merges a vendor-data cloud-config under the user-data with mergers that keep the user-data's value for every top-level key both define and never append lists, so a Proxy carrying its own `packages:` diff --git a/docs/plans-executions/2026-09-18-1502-proxmox-provider.md b/docs/plans-executions/2026-09-18-1502-proxmox-provider.md index 24d7fcb..3c5e57e 100644 --- a/docs/plans-executions/2026-09-18-1502-proxmox-provider.md +++ b/docs/plans-executions/2026-09-18-1502-proxmox-provider.md @@ -7,7 +7,7 @@ Pairs with [docs/plans/2026-09-18-1502-proxmox-provider.md](../plans/2026-09-18- - [x] Phase 0 — Plan file, branch, host recon, fixtures - [x] Phase 1 — The spec (`docs/proxmox-provider-spec.md`) - [x] Phase 2 — Config block, registration, package skeleton -- [ ] Phase 3 — Extract `nocloud` and `keyedlock` from kvm2 +- [x] Phase 3 — Extract `nocloud` and `keyedlock` from kvm2 - [ ] Phase 4 — Seam, HTTP client, decorator, fake - [ ] Phase 5 — Get, ListByTag, task helpers - [ ] Phase 6 — Create @@ -381,3 +381,69 @@ go test -count=1 -v ./internal/provider/proxmox/ | grep -c '^=== RUN' # 152 Worth noting: the `validProxmoxYAML` constant is the shape Phase 9's commented sample entry should take, with the user's values (`192.168.0.8`, `pv1-1`, `proxy-seeds`, `vlanTag: 4`, template 9001). + +## Phase 3 — Extract `nocloud` and `keyedlock` from kvm2 + +kvm2's `seed.go` and `seed_test.go` moved to `internal/provider/nocloud` +with `git mv`, so history follows them. The package exports `BuildSeedISO`, +`BuildVendorData`, `GuestAgentScript`, `SeedLabel` and a new `ErrEmptyName` +sentinel in place of kvm2's `permanentf`. kvm2's `keyedLocks` became +`internal/provider/keyedlock` (`Locks`, `New`, `Acquire`), and its three +`TestKeyedLocks_*` tests moved with it under their old names. kvm2 now +imports both packages, and its `create.go` still wraps every seed-build error +with `permanentf`, so kvm2's behaviour is unchanged. proxmox gained the +`nameLocks` field and `lockName`. + +The test counts match the plan's Verify clause: kvm2 dropped from 465 to 447 +`=== RUN` entries, which is the 12 moved tests plus the 6 subtests of +`TestSeedISO_userDataByteForByte`. `nocloud` (15) and `keyedlock` (3) carry +exactly those 18. + +```bash +git mv internal/provider/kvm2/seed.go internal/provider/nocloud/seed.go +git mv internal/provider/kvm2/seed_test.go internal/provider/nocloud/seed_test.go +go test -count=1 -v ./internal/provider/kvm2/ | grep -c '^=== RUN' # 465 before, 447 after +grep -rn 'iso9660' internal/provider/kvm2/ # nothing +go build ./... && go vet ./... && go vet -tags=integration ./... +go test -race -count=1 ./internal/provider/... # all ok, run three times +go test -short -count=1 ./... # all ok +``` + +**Pulled forward from Phase 6 so proxmox imports `nocloud` now**, as the +"Done when" line requires: `seed.go` with `seedISOName`, `seedVolID` and +`buildSeed`, the pure part of the seed step. `uploadSeed` still belongs to +Phase 6, which calls `buildSeed`. The new tests are `TestSeedNames_table`, +`TestBuildSeed_table` (4 rows), +`TestBuildSeed_emptyNameIsPermanentWithoutPayload`, +`TestBuildSeed_stagingFailureIsTransient` and +`TestLockName_givesUpWithContextAsTransient`. proxmox went from 152 to 161 +test entries. + +**proxmox does not copy kvm2's open review finding S5.1** +(`docs/reviews/2026-09-17-kvm2-provider-review/sweep.md`). In kvm2, a +staging failure such as a missing `/tmp` or a full emptyDir becomes +Permanent, and that latches the Proxy. `buildSeed` makes only +`ErrEmptyName` Permanent and wraps any other error with `%w`, so +`classify` reads it as Transient. The sentinel makes the same three-line +fix possible in kvm2, but this phase leaves kvm2's behaviour unchanged. + +Worth noting: + +- The older `kvm` provider has its own `seed.go` with a different + signature (`buildSeedISO` returns `[]byte`). The plan scopes the + extraction to kvm2, so `kvm` is untouched. +- `GuestAgentScript` now says the operator reads the address "from the + hypervisor" instead of "through libvirt", in both the Go comment and the + script's comment line inside the guest. The script's commands are + unchanged. +- The docs now name the new packages: `kvm2-provider.md`, + `kvm2-provider-spec.md` entry 19, the kvm2 columns and section of + `providers-existing.md`, plus a "Shared provider helpers" row in the + `architecture.md` component table. `docs/reviews/` keeps the old names, + because it records the code as it was reviewed. +- `TestCheck_hostsCheckedConcurrently` in kvm2 failed once under the full + `-race` suite (latency 199.96 ms against a 200 ms floor). It passed 20 + runs out of 20 in isolation, with and without this phase's changes. The + test starts its 200 ms release timers before `Check` starts its clock, so + under load the measured latency can come in just under 200 ms. The flake + predates this phase, and the test is not touched here. diff --git a/docs/providers-existing.md b/docs/providers-existing.md index b5667eb..80cb104 100644 --- a/docs/providers-existing.md +++ b/docs/providers-existing.md @@ -58,7 +58,7 @@ against. |---|---|---|---|---| | Create verifies the adopted instance's UID | no | no | no | **yes**: a domain without the kvm2 mark or with a different non-empty UID is Permanent; a marked one with a matching or empty UID is adopted (`create`) | | Placement used | none | Zone, MachineType, Image | Zone (host), MachineType, Image | Zone (host), MachineType, Image | -| CloudInit delivered | **no**: ignored | yes: `user-data` metadata | yes: NoCloud seed ISO built in-process (`buildSeedISO`) | yes: NoCloud seed ISO built in-process (`buildSeedISO`) | +| CloudInit delivered | **no**: ignored | yes: `user-data` metadata | yes: NoCloud seed ISO built in-process (`buildSeedISO`) | yes: NoCloud seed ISO built in-process (`nocloud.BuildSeedISO`) | | Port used | yes: Squid config and containerPort (`squidConf`) | no: firewall is manual | no | no | | States produced | Provisioning, Running, Terminated | all four | Provisioning, Running, Stopped | Provisioning, Running, Stopped | | Deleting instance still reports its state | yes: a Pod being deleted reports its phase | n/a | n/a | n/a: Delete waits, and a concurrent Get sees the domain until it is undefined | @@ -68,7 +68,7 @@ against. | Credential rotation without restart | n/a | via ADC | no: cached after first successful load | yes: both files read on every dial (`loadCredentials`) | | Connection handling | client-go | SDK client, never closed | one pooled SSH plus libvirt connection per host, 15s keepalive (`hostPool`) | one pooled SSH plus libvirt connection per host, 15s keepalive (`hostTable`); dedicated connections for the image upload and the check (`dialFresh`) | | Unit-test double | controller-runtime fake client | `fakeAPI` behind `instancesAPI`, plus `httptest` for the real SDK client in `check_test.go` | `fakeAPI` behind `libvirtAPI`, plus an in-process `x/crypto/ssh` server in `conn_test.go` | `fakeBackend` behind `backend` (`fake_test.go`), plus an in-process `x/crypto/ssh` server (`sshserver_test.go`) and go-libvirt's own mock for the adapter | -| Proxy software | generated Squid config (`pod.go`) | from `cloudInit`; the image must run cloud-init | from `cloudInit`, plus a vendor-data script that installs the guest agent (`seed.go`) | from `cloudInit`, plus a vendor-data script that installs the guest agent (`seed.go`) | +| Proxy software | generated Squid config (`pod.go`) | from `cloudInit`; the image must run cloud-init | from `cloudInit`, plus a vendor-data script that installs the guest agent (`seed.go`) | from `cloudInit`, plus a vendor-data script that installs the guest agent (`nocloud.GuestAgentScript`) | | IP reachable from the operator | Pod network, so in-cluster only (not from `make run-dev` on a laptop) | external IP, once a firewall rule allows the cluster's egress IP | LAN address on the host's bridge; the operator needs a route to that LAN | LAN address on the host's bridge; the operator needs a route to that LAN | | Distinct egress IP per proxy | no: the cluster's egress | yes: ephemeral external IP | whatever the host's LAN routes through; not per VM by itself | whatever the host's LAN routes through; not per VM by itself | @@ -347,8 +347,10 @@ same libvirt. production adapter and the budgets), `ssh.go` (credentials, the forwarded socket, keepalive), `hosts.go` (the pooled table and the production dialer), `observe.go` (the span and log decorator), `image.go` (cache, - download, quota precheck), `seed.go`, `xml.go`, `volume.go`, - `metadata.go`, `id.go`, `state.go`, `errors.go`, `check.go`. The seam is + download, quota precheck), `xml.go`, `volume.go`, `metadata.go`, + `id.go`, `state.go`, `errors.go`, `check.go`; the seed ISO comes from + the shared `internal/provider/nocloud` and both lock sets are + `internal/provider/keyedlock`. The seam is `backend`: the twenty go-libvirt calls the provider uses, each with `ctx` first (go-libvirt's own methods take none and block), plus `Close`. `libvirtBackend` adapts `*libvirt.Libvirt` and runs every call through diff --git a/internal/provider/keyedlock/keyedlock.go b/internal/provider/keyedlock/keyedlock.go new file mode 100644 index 0000000..7e25ba1 --- /dev/null +++ b/internal/provider/keyedlock/keyedlock.go @@ -0,0 +1,60 @@ +// Package keyedlock provides one-slot semaphores by key, shared by the +// providers that serialise work per instance name or per cached image +// (kvm2, proxmox). +package keyedlock + +import ( + "context" + "sync" +) + +// Locks is a set of one-slot semaphores by key: a semaphore rather than a +// mutex so a waiter can give up with its ctx. Each entry counts the callers +// holding or waiting for it and is dropped from the map at zero, so the map +// does not grow with every key ever seen while two waiters on one key still +// always share one semaphore. The zero value is not usable; call New. +type Locks struct { + mu sync.Mutex + locks map[string]*lock +} + +type lock struct { + sem chan struct{} + refs int +} + +func New() *Locks { + return &Locks{locks: make(map[string]*lock)} +} + +// Acquire takes the lock for key, or gives up with ctx's error. The +// returned func releases it and must be called exactly once. +func (k *Locks) Acquire(ctx context.Context, key string) (release func(), err error) { + k.mu.Lock() + l, ok := k.locks[key] + if !ok { + l = &lock{sem: make(chan struct{}, 1)} + k.locks[key] = l + } + l.refs++ + k.mu.Unlock() + select { + case l.sem <- struct{}{}: + return func() { + <-l.sem + k.unref(key, l) + }, nil + case <-ctx.Done(): + k.unref(key, l) + return nil, ctx.Err() + } +} + +func (k *Locks) unref(key string, l *lock) { + k.mu.Lock() + defer k.mu.Unlock() + l.refs-- + if l.refs == 0 { + delete(k.locks, key) + } +} diff --git a/internal/provider/keyedlock/keyedlock_test.go b/internal/provider/keyedlock/keyedlock_test.go new file mode 100644 index 0000000..7a77aa4 --- /dev/null +++ b/internal/provider/keyedlock/keyedlock_test.go @@ -0,0 +1,90 @@ +package keyedlock + +import ( + "context" + "errors" + "sync" + "sync/atomic" + "testing" + "time" +) + +func lockCount(k *Locks) int { + k.mu.Lock() + defer k.mu.Unlock() + return len(k.locks) +} + +func TestKeyedLocks_entryRemovedWhenUncontended(t *testing.T) { + t.Parallel() + k := New() + release, err := k.Acquire(context.Background(), "a") + if err != nil { + t.Fatalf("acquire error = %v", err) + } + if lockCount(k) != 1 { + t.Errorf("entries while held = %d, want 1", lockCount(k)) + } + release() + if lockCount(k) != 0 { + t.Errorf("entries after release = %d, want 0", lockCount(k)) + } + if _, err := k.Acquire(context.Background(), "a"); err != nil { + t.Fatalf("acquire after release error = %v", err) + } +} + +func TestKeyedLocks_waitersShareOneLock(t *testing.T) { + t.Parallel() + k := New() + const n = 20 + var inside, overlaps atomic.Int32 + var wg sync.WaitGroup + for range n { + wg.Go(func() { + release, err := k.Acquire(context.Background(), "shared") + if err != nil { + t.Errorf("acquire error = %v", err) + return + } + if inside.Add(1) > 1 { + overlaps.Add(1) + } + time.Sleep(time.Millisecond) + inside.Add(-1) + release() + }) + } + wg.Wait() + if overlaps.Load() != 0 { + t.Errorf("%d holders overlapped on one key", overlaps.Load()) + } + if lockCount(k) != 0 { + t.Errorf("entries after every holder released = %d, want 0", lockCount(k)) + } +} + +func TestKeyedLocks_waiterGivesUpWithContext(t *testing.T) { + t.Parallel() + k := New() + release, err := k.Acquire(context.Background(), "a") + if err != nil { + t.Fatalf("acquire error = %v", err) + } + ctx, cancel := context.WithTimeout(context.Background(), 20*time.Millisecond) + defer cancel() + start := time.Now() + if _, err := k.Acquire(ctx, "a"); !errors.Is(err, context.DeadlineExceeded) { + t.Fatalf("waiter error = %v, want DeadlineExceeded", err) + } + if time.Since(start) > time.Second { + t.Errorf("waiter took %v, want to give up with its ctx", time.Since(start)) + } + if lockCount(k) != 1 { + t.Errorf("entries with the holder still in = %d, want 1: the waiter's exit must not drop the holder's entry", lockCount(k)) + } + release() + if lockCount(k) != 0 { + t.Errorf("entries after the holder released = %d, want 0", lockCount(k)) + } +} diff --git a/internal/provider/kvm2/create.go b/internal/provider/kvm2/create.go index 0de7e45..0778ba4 100644 --- a/internal/provider/kvm2/create.go +++ b/internal/provider/kvm2/create.go @@ -11,6 +11,7 @@ import ( "github.com/go-logr/logr" "gitea.home.hrajfrisbee.cz/kacerr/egress-proxies-operator/internal/provider" + "gitea.home.hrajfrisbee.cz/kacerr/egress-proxies-operator/internal/provider/nocloud" ) // The wait after DomainCreate (spec entry 18): Create returns only once the @@ -191,12 +192,12 @@ func (p *Provider) writeSeed(ctx context.Context, log logr.Logger, be backend, p return libvirt.StorageVol{}, err } installAgent := p.cfg.InstallGuestAgent == nil || *p.cfg.InstallGuestAgent - vendorData, err := buildVendorData(p.cfg.VendorData, installAgent) + vendorData, err := nocloud.BuildVendorData(p.cfg.VendorData, installAgent) if err != nil { return libvirt.StorageVol{}, permanentf("building vendor-data for %s: %v", req.Name, err) } var iso bytes.Buffer - if err := buildSeedISO(req.Name, req.CloudInit, vendorData, &iso); err != nil { + if err := nocloud.BuildSeedISO(req.Name, req.CloudInit, vendorData, &iso); err != nil { return libvirt.StorageVol{}, permanentf("building seed ISO for %s: %v", req.Name, err) } xmlDoc, err := buildVolumeXML(volumeSpec{Name: name, CapacityBytes: int64(iso.Len()), Format: provider.KVMFormatRaw}) diff --git a/internal/provider/kvm2/create_test.go b/internal/provider/kvm2/create_test.go index 66fb26c..218c0ba 100644 --- a/internal/provider/kvm2/create_test.go +++ b/internal/provider/kvm2/create_test.go @@ -12,6 +12,7 @@ import ( "github.com/digitalocean/go-libvirt" "gitea.home.hrajfrisbee.cz/kacerr/egress-proxies-operator/internal/provider" + "gitea.home.hrajfrisbee.cz/kacerr/egress-proxies-operator/internal/provider/nocloud" ) func TestCreate_repeatReturnsSameIDAndCreatesNothing(t *testing.T) { @@ -95,7 +96,7 @@ func TestCreate_resumesAfterEachPartialStep(t *testing.T) { }, check: func(t *testing.T, fx *fixture) { got := fx.fake.vol(seed) - if got == nil || len(got.content) <= 3 || !bytes.Contains(got.content, []byte(seedLabel)) { + if got == nil || len(got.content) <= 3 || !bytes.Contains(got.content, []byte(nocloud.SeedLabel)) { t.Errorf("seed volume not rebuilt") } }, @@ -461,7 +462,7 @@ func TestCreate_seedAlwaysRebuilt(t *testing.T) { if got := fx.fake.callCount("StorageVolUpload"); got != uploads+1 { t.Errorf("StorageVolUpload calls = %d, want %d", got, uploads+1) } - if len(second.content) != len(first.content) || !bytes.Contains(second.content, []byte(seedLabel)) { + if len(second.content) != len(first.content) || !bytes.Contains(second.content, []byte(nocloud.SeedLabel)) { t.Errorf("rebuilt seed is %d bytes, first was %d; want a complete seed of the same size", len(second.content), len(first.content)) } } diff --git a/internal/provider/kvm2/image.go b/internal/provider/kvm2/image.go index f0d6664..f7bcae5 100644 --- a/internal/provider/kvm2/image.go +++ b/internal/provider/kvm2/image.go @@ -73,7 +73,7 @@ func resolveImage(entry string, cfg provider.KVMConfig, image string) (imageSour // returns promptly (contract §1.5) instead of waiting out a 20-minute // transfer. func (p *Provider) lockImage(ctx context.Context, hostName, volume string) (func(), error) { - release, err := p.imageLocks.acquire(ctx, hostName+"/"+volume) + release, err := p.imageLocks.Acquire(ctx, hostName+"/"+volume) if err != nil { return nil, fmt.Errorf("%w: waiting for image %s on host %s: %w", provider.ErrTransient, volume, hostName, err) } diff --git a/internal/provider/kvm2/kvm2.go b/internal/provider/kvm2/kvm2.go index 11090b8..e7a13e0 100644 --- a/internal/provider/kvm2/kvm2.go +++ b/internal/provider/kvm2/kvm2.go @@ -9,13 +9,13 @@ import ( "fmt" "net/http" "os" - "sync" "time" "github.com/go-logr/logr" logf "sigs.k8s.io/controller-runtime/pkg/log" "gitea.home.hrajfrisbee.cz/kacerr/egress-proxies-operator/internal/provider" + "gitea.home.hrajfrisbee.cz/kacerr/egress-proxies-operator/internal/provider/keyedlock" "gitea.home.hrajfrisbee.cz/kacerr/egress-proxies-operator/internal/tracing" ) @@ -41,12 +41,12 @@ type Provider struct { runningPoll time.Duration // imageLocks serialises image fetches per (host, volume), spec entry 19. - imageLocks *keyedLocks + imageLocks *keyedlock.Locks // nameLocks serialises Create and Delete per instance name, spec entry // 19, so two Deletes never race on destroy and undefine and a Delete // cannot interleave with a Create's steps. - nameLocks *keyedLocks + nameLocks *keyedlock.Locks } // providerOption adjusts what newProvider builds; tests inject an HTTP @@ -111,8 +111,8 @@ func newProvider(name string, cfg provider.KVMConfig, d dialer, opts ...provider http: http.DefaultClient, runningWait: defaultRunningWait, runningPoll: defaultRunningPoll, - imageLocks: newKeyedLocks(), - nameLocks: newKeyedLocks(), + imageLocks: keyedlock.New(), + nameLocks: keyedlock.New(), } for _, opt := range opts { opt(p) @@ -187,60 +187,9 @@ func (p *Provider) wrapErr(op, id string, err error) error { // so a caller whose ctx ends while another caller's Create is downloading // an image returns promptly. func (p *Provider) lockName(ctx context.Context, name string) (func(), error) { - release, err := p.nameLocks.acquire(ctx, name) + release, err := p.nameLocks.Acquire(ctx, name) if err != nil { return nil, fmt.Errorf("%w: waiting for the lock on %s: %w", provider.ErrTransient, name, err) } return release, nil } - -// keyedLocks is a set of one-slot semaphores by key (spec entry 19): a -// semaphore rather than a mutex so a waiter can give up with its ctx. Each -// entry counts the callers holding or waiting for it and is dropped from -// the map at zero, so the map does not grow with every name ever seen while -// two waiters on one key still always share one semaphore. -type keyedLocks struct { - mu sync.Mutex - locks map[string]*keyedLock -} - -type keyedLock struct { - sem chan struct{} - refs int -} - -func newKeyedLocks() *keyedLocks { - return &keyedLocks{locks: make(map[string]*keyedLock)} -} - -// acquire takes the lock for key, or gives up with ctx's error. The -// returned func releases it and must be called exactly once. -func (k *keyedLocks) acquire(ctx context.Context, key string) (func(), error) { - k.mu.Lock() - l, ok := k.locks[key] - if !ok { - l = &keyedLock{sem: make(chan struct{}, 1)} - k.locks[key] = l - } - l.refs++ - k.mu.Unlock() - select { - case l.sem <- struct{}{}: - return func() { - <-l.sem - k.unref(key, l) - }, nil - case <-ctx.Done(): - k.unref(key, l) - return nil, ctx.Err() - } -} - -func (k *keyedLocks) unref(key string, l *keyedLock) { - k.mu.Lock() - defer k.mu.Unlock() - l.refs-- - if l.refs == 0 { - delete(k.locks, key) - } -} diff --git a/internal/provider/kvm2/kvm2_test.go b/internal/provider/kvm2/kvm2_test.go index 909d00d..a04c9ae 100644 --- a/internal/provider/kvm2/kvm2_test.go +++ b/internal/provider/kvm2/kvm2_test.go @@ -5,7 +5,6 @@ import ( "errors" "strings" "sync" - "sync/atomic" "testing" "time" @@ -267,86 +266,6 @@ func TestWithDefaults_doesNotAliasInput(t *testing.T) { } } -func lockCount(k *keyedLocks) int { - k.mu.Lock() - defer k.mu.Unlock() - return len(k.locks) -} - -func TestKeyedLocks_entryRemovedWhenUncontended(t *testing.T) { - t.Parallel() - k := newKeyedLocks() - release, err := k.acquire(context.Background(), "a") - if err != nil { - t.Fatalf("acquire error = %v", err) - } - if lockCount(k) != 1 { - t.Errorf("entries while held = %d, want 1", lockCount(k)) - } - release() - if lockCount(k) != 0 { - t.Errorf("entries after release = %d, want 0", lockCount(k)) - } - if _, err := k.acquire(context.Background(), "a"); err != nil { - t.Fatalf("acquire after release error = %v", err) - } -} - -func TestKeyedLocks_waitersShareOneLock(t *testing.T) { - t.Parallel() - k := newKeyedLocks() - const n = 20 - var inside, overlaps atomic.Int32 - var wg sync.WaitGroup - for range n { - wg.Go(func() { - release, err := k.acquire(context.Background(), "shared") - if err != nil { - t.Errorf("acquire error = %v", err) - return - } - if inside.Add(1) > 1 { - overlaps.Add(1) - } - time.Sleep(time.Millisecond) - inside.Add(-1) - release() - }) - } - wg.Wait() - if overlaps.Load() != 0 { - t.Errorf("%d holders overlapped on one key", overlaps.Load()) - } - if lockCount(k) != 0 { - t.Errorf("entries after every holder released = %d, want 0", lockCount(k)) - } -} - -func TestKeyedLocks_waiterGivesUpWithContext(t *testing.T) { - t.Parallel() - k := newKeyedLocks() - release, err := k.acquire(context.Background(), "a") - if err != nil { - t.Fatalf("acquire error = %v", err) - } - ctx, cancel := context.WithTimeout(context.Background(), 20*time.Millisecond) - defer cancel() - start := time.Now() - if _, err := k.acquire(ctx, "a"); !errors.Is(err, context.DeadlineExceeded) { - t.Fatalf("waiter error = %v, want DeadlineExceeded", err) - } - if time.Since(start) > time.Second { - t.Errorf("waiter took %v, want to give up with its ctx", time.Since(start)) - } - if lockCount(k) != 1 { - t.Errorf("entries with the holder still in = %d, want 1: the waiter's exit must not drop the holder's entry", lockCount(k)) - } - release() - if lockCount(k) != 0 { - t.Errorf("entries after the holder released = %d, want 0", lockCount(k)) - } -} - func TestWrapErr_carriesProviderNameOpAndID(t *testing.T) { t.Parallel() p := newProvider("kvm2-home", twoHosts(), &stubDialer{}) diff --git a/internal/provider/kvm2/seed.go b/internal/provider/nocloud/seed.go similarity index 77% rename from internal/provider/kvm2/seed.go rename to internal/provider/nocloud/seed.go index 4c81292..77d2d56 100644 --- a/internal/provider/kvm2/seed.go +++ b/internal/provider/nocloud/seed.go @@ -1,4 +1,7 @@ -package kvm2 +// Package nocloud builds cloud-init NoCloud seed ISOs in process, shared by +// the providers that attach a seed as a CD-ROM (kvm2, proxmox). It never +// parses or logs the payload. +package nocloud import ( "bytes" @@ -12,19 +15,22 @@ import ( "github.com/kdomanski/iso9660" ) -// seedLabel is the volume label cloud-init's NoCloud datasource looks for. -const seedLabel = "cidata" +// SeedLabel is the volume label cloud-init's NoCloud datasource looks for. +const SeedLabel = "cidata" -// guestAgentScript is the vendor-data payload that gets qemu-guest-agent -// running so the provider can read the VM's address through libvirt. It is +// ErrEmptyName is returned by BuildSeedISO for an empty instance name. +var ErrEmptyName = errors.New("seed ISO: instance name is empty") + +// GuestAgentScript is the vendor-data payload that gets qemu-guest-agent +// running so the provider can read the VM's address from the hypervisor. It is // a shell script and not a cloud-config on purpose: cloud-init merges a // vendor-data cloud-config under the user-data with mergers that keep the // user-data's value for every top-level key and never append lists, so a // Proxy with its own packages: would silently lose the agent. Scripts // bypass the merge. -const guestAgentScript = `#!/bin/sh +const GuestAgentScript = `#!/bin/sh # Added by egress-proxies-operator: installs and starts qemu-guest-agent so -# the operator can learn this VM's IP address through libvirt. +# the operator can learn this VM's IP address from the hypervisor. # To leave this out, set installGuestAgent: false in the provider's config. set -e if command -v qemu-ga >/dev/null 2>&1 || [ -x /usr/sbin/qemu-ga ]; then @@ -53,15 +59,15 @@ else fi ` -// buildSeedISO writes a NoCloud seed to w: user-data byte for byte, +// BuildSeedISO writes a NoCloud seed to w: user-data byte for byte, // meta-data naming the instance, and vendor-data (an empty file when there // is none, so cloud-init does not look elsewhere). iso9660.NewWriter stages // the files in os.MkdirTemp("", ""), that is under os.TempDir(): $TMPDIR // when set, else /tmp, the manager's 64Mi emptyDir. The staging dir is // removed before returning. Nothing about the payload is logged here. -func buildSeedISO(name, userData, vendorData string, w io.Writer) (err error) { +func BuildSeedISO(name, userData, vendorData string, w io.Writer) (err error) { if name == "" { - return permanentf("seed ISO: instance name is empty") + return ErrEmptyName } iw, err := iso9660.NewWriter() if err != nil { @@ -83,30 +89,30 @@ func buildSeedISO(name, userData, vendorData string, w io.Writer) (err error) { return fmt.Errorf("adding %s to seed ISO: %w", f.name, err) } } - if err := iw.WriteTo(w, seedLabel); err != nil { + if err := iw.WriteTo(w, SeedLabel); err != nil { return fmt.Errorf("writing seed ISO: %w", err) } return nil } -// buildVendorData composes the seed's vendor-data from the configured +// BuildVendorData composes the seed's vendor-data from the configured // document and the guest agent script. With both, they travel as separate // parts of a MIME multipart, so cloud-init handles each by its own type; the // MIME-Version header comes first because cloud-init detects multipart by // finding "mime-version:" within the first 4096 bytes. -func buildVendorData(configured string, installAgent bool) (string, error) { +func BuildVendorData(configured string, installAgent bool) (string, error) { switch { case !installAgent: return configured, nil case strings.TrimSpace(configured) == "": - return guestAgentScript, nil + return GuestAgentScript, nil } var buf bytes.Buffer mw := multipart.NewWriter(&buf) fmt.Fprintf(&buf, "MIME-Version: 1.0\r\nContent-Type: multipart/mixed; boundary=%q\r\n\r\n", mw.Boundary()) parts := []struct{ contentType, body string }{ {"text/plain", configured}, - {"text/x-shellscript", guestAgentScript}, + {"text/x-shellscript", GuestAgentScript}, } for _, p := range parts { pw, err := mw.CreatePart(textproto.MIMEHeader{"Content-Type": {p.contentType}}) diff --git a/internal/provider/kvm2/seed_test.go b/internal/provider/nocloud/seed_test.go similarity index 91% rename from internal/provider/kvm2/seed_test.go rename to internal/provider/nocloud/seed_test.go index e0bb40e..2ac2575 100644 --- a/internal/provider/kvm2/seed_test.go +++ b/internal/provider/nocloud/seed_test.go @@ -1,8 +1,9 @@ -package kvm2 +package nocloud import ( "bufio" "bytes" + "errors" "io" "mime" "mime/multipart" @@ -18,8 +19,8 @@ import ( func readSeed(t *testing.T, name, userData, vendorData string) (map[string]string, string) { t.Helper() var buf bytes.Buffer - if err := buildSeedISO(name, userData, vendorData, &buf); err != nil { - t.Fatalf("buildSeedISO: %v", err) + if err := BuildSeedISO(name, userData, vendorData, &buf); err != nil { + t.Fatalf("BuildSeedISO: %v", err) } img, err := iso9660.OpenImage(bytes.NewReader(buf.Bytes())) if err != nil { @@ -57,8 +58,8 @@ func TestSeedISO_filesAndLabel(t *testing.T) { userData := "#cloud-config\npackages: [squid]\n" vendorData := "#!/bin/sh\necho hi\n" files, label := readSeed(t, "proxy-abc", userData, vendorData) - if label != seedLabel { - t.Errorf("label = %q, want %q", label, seedLabel) + if label != SeedLabel { + t.Errorf("label = %q, want %q", label, SeedLabel) } want := map[string]string{ "user-data": userData, @@ -117,8 +118,8 @@ func TestSeedISO_tempDirRemoved(t *testing.T) { t.Fatalf("os.TempDir() = %q, want %q", got, tmp) } var buf bytes.Buffer - if err := buildSeedISO("proxy-abc", "#cloud-config\n", guestAgentScript, &buf); err != nil { - t.Fatalf("buildSeedISO: %v", err) + if err := BuildSeedISO("proxy-abc", "#cloud-config\n", GuestAgentScript, &buf); err != nil { + t.Fatalf("BuildSeedISO: %v", err) } if buf.Len() == 0 { t.Fatal("seed ISO is empty") @@ -151,8 +152,8 @@ func TestSeedISO_emptyVendorDataStillHasFile(t *testing.T) { func TestSeedISO_rejectsEmptyName(t *testing.T) { t.Parallel() var buf bytes.Buffer - if err := buildSeedISO("", "#cloud-config\n", "", &buf); err == nil { - t.Fatal("buildSeedISO with empty name succeeded") + if err := BuildSeedISO("", "#cloud-config\n", "", &buf); !errors.Is(err, ErrEmptyName) { + t.Fatalf("BuildSeedISO with empty name error = %v, want ErrEmptyName", err) } if buf.Len() != 0 { t.Errorf("wrote %d bytes alongside the error", buf.Len()) @@ -161,7 +162,7 @@ func TestSeedISO_rejectsEmptyName(t *testing.T) { func TestVendorData_isShellScriptNotCloudConfig(t *testing.T) { t.Parallel() - got, err := buildVendorData("", true) + got, err := BuildVendorData("", true) if err != nil { t.Fatal(err) } @@ -193,7 +194,7 @@ func bufioReader(s string) *bufio.Reader { func TestVendorData_multipartWhenExtraProvided(t *testing.T) { t.Parallel() configured := "#cloud-config\nntp:\n enabled: true\n" - got, err := buildVendorData(configured, true) + got, err := BuildVendorData(configured, true) if err != nil { t.Fatal(err) } @@ -229,7 +230,7 @@ func TestVendorData_multipartWhenExtraProvided(t *testing.T) { } parts = append(parts, part{p.Header.Get("Content-Type"), string(body)}) } - want := []part{{"text/plain", configured}, {"text/x-shellscript", guestAgentScript}} + want := []part{{"text/plain", configured}, {"text/x-shellscript", GuestAgentScript}} if len(parts) != len(want) { t.Fatalf("got %d parts, want %d", len(parts), len(want)) } @@ -242,7 +243,7 @@ func TestVendorData_multipartWhenExtraProvided(t *testing.T) { func TestVendorData_offWithoutConfiguredIsEmpty(t *testing.T) { t.Parallel() - got, err := buildVendorData("", false) + got, err := BuildVendorData("", false) if err != nil { t.Fatal(err) } @@ -254,7 +255,7 @@ func TestVendorData_offWithoutConfiguredIsEmpty(t *testing.T) { func TestVendorData_offWithConfiguredPassesThrough(t *testing.T) { t.Parallel() configured := "#cloud-config\nntp:\n enabled: true" - got, err := buildVendorData(configured, false) + got, err := BuildVendorData(configured, false) if err != nil { t.Fatal(err) } diff --git a/internal/provider/proxmox/proxmox.go b/internal/provider/proxmox/proxmox.go index d3875b5..1b19826 100644 --- a/internal/provider/proxmox/proxmox.go +++ b/internal/provider/proxmox/proxmox.go @@ -6,6 +6,7 @@ package proxmox import ( "context" + "fmt" "os" "time" @@ -13,6 +14,7 @@ import ( logf "sigs.k8s.io/controller-runtime/pkg/log" "gitea.home.hrajfrisbee.cz/kacerr/egress-proxies-operator/internal/provider" + "gitea.home.hrajfrisbee.cz/kacerr/egress-proxies-operator/internal/provider/keyedlock" ) const ( @@ -57,6 +59,11 @@ type Provider struct { api api now func() time.Time budgets budgets + + // nameLocks serialises Create and Delete per instance name (spec entry + // 19), so the reconciler's Delete and GC's Delete never race on stop and + // destroy, and a Delete cannot interleave with a Create's steps. + nameLocks *keyedlock.Locks } var _ provider.Provider = (*Provider)(nil) @@ -85,11 +92,12 @@ func New(ctx context.Context, pc provider.ProviderConfig) (provider.Provider, er // newProvider is the constructor tests use, with a in-memory api. func newProvider(name string, cfg provider.ProxmoxConfig, a api) *Provider { return &Provider{ - name: name, - cfg: withDefaults(cfg), - api: a, - now: time.Now, - budgets: defaultBudgets(), + name: name, + cfg: withDefaults(cfg), + api: a, + now: time.Now, + budgets: defaultBudgets(), + nameLocks: keyedlock.New(), } } @@ -152,6 +160,17 @@ func (p *Provider) wrapErr(op, id string, err error) error { return provider.Wrap(classify(op, err), op, p.name, id, err) } +// lockName takes the per-name lock of spec entry 19, or gives up with ctx, +// so a caller whose ctx ends while another caller's Create waits on a clone +// returns promptly. +func (p *Provider) lockName(ctx context.Context, name string) (func(), error) { + release, err := p.nameLocks.Acquire(ctx, name) + if err != nil { + return nil, fmt.Errorf("%w: waiting for the lock on %s: %w", provider.ErrTransient, name, err) + } + return release, nil +} + // The four contract methods are stubs until the phases that implement them // (Get and ListByTag in Phase 5, Create in 6, Delete in 7). Each answers // Transient, so a Proxy on this provider retries rather than latching. diff --git a/internal/provider/proxmox/proxmox_test.go b/internal/provider/proxmox/proxmox_test.go index 1b25f70..1197571 100644 --- a/internal/provider/proxmox/proxmox_test.go +++ b/internal/provider/proxmox/proxmox_test.go @@ -206,3 +206,24 @@ func TestStubs_answerTransient(t *testing.T) { } } } + +func TestLockName_givesUpWithContextAsTransient(t *testing.T) { + t.Parallel() + p := newProvider("proxmox-home", testConfig(), nil) + release, err := p.lockName(context.Background(), "proxy-abc") + if err != nil { + t.Fatalf("lockName error = %v", err) + } + defer release() + other, err := p.lockName(context.Background(), "proxy-def") + if err != nil { + t.Fatalf("lockName on another name error = %v, want it free", err) + } + other() + ctx, cancel := context.WithTimeout(context.Background(), 20*time.Millisecond) + defer cancel() + _, err = p.lockName(ctx, "proxy-abc") + if !errors.Is(err, provider.ErrTransient) || !errors.Is(err, context.DeadlineExceeded) { + t.Errorf("lockName on a held name error = %v, want ErrTransient wrapping DeadlineExceeded", err) + } +} diff --git a/internal/provider/proxmox/seed.go b/internal/provider/proxmox/seed.go new file mode 100644 index 0000000..cb7dc20 --- /dev/null +++ b/internal/provider/proxmox/seed.go @@ -0,0 +1,43 @@ +package proxmox + +import ( + "bytes" + "errors" + "fmt" + + "gitea.home.hrajfrisbee.cz/kacerr/egress-proxies-operator/internal/provider" + "gitea.home.hrajfrisbee.cz/kacerr/egress-proxies-operator/internal/provider/nocloud" +) + +// seedISOName is the seed's file name on the ISO storage. Upload refuses a +// name without the .iso suffix (F12), and GC's blind spot is documented by +// this pattern (spec entry 11). +func seedISOName(name string) string { + return name + "-seed.iso" +} + +// seedVolID is the seed's volume id, as ide2 references it and as the +// storage content delete takes it. +func seedVolID(storage, name string) string { + return storage + ":iso/" + seedISOName(name) +} + +// buildSeed renders the NoCloud seed for req in memory (spec entry 5): a +// seed is tens of KiB. Nothing about its content is logged or put in an +// error. Only an empty name is Permanent; a staging failure (no /tmp, a +// full emptyDir) is left unclassified, so classify makes it Transient and +// the reconciler retries instead of latching the Proxy. +func (p *Provider) buildSeed(req provider.CreateRequest) (*bytes.Buffer, error) { + vendorData, err := nocloud.BuildVendorData(p.cfg.VendorData, *p.cfg.InstallGuestAgent) + if err != nil { + return nil, fmt.Errorf("building vendor-data for %s: %w", req.Name, err) + } + var iso bytes.Buffer + if err := nocloud.BuildSeedISO(req.Name, req.CloudInit, vendorData, &iso); err != nil { + if errors.Is(err, nocloud.ErrEmptyName) { + return nil, permanentf("building seed ISO: %v", err) + } + return nil, fmt.Errorf("building seed ISO for %s: %w", req.Name, err) + } + return &iso, nil +} diff --git a/internal/provider/proxmox/seed_test.go b/internal/provider/proxmox/seed_test.go new file mode 100644 index 0000000..1ccc611 --- /dev/null +++ b/internal/provider/proxmox/seed_test.go @@ -0,0 +1,139 @@ +package proxmox + +import ( + "bytes" + "errors" + "io" + "strings" + "testing" + + "github.com/kdomanski/iso9660" + + "gitea.home.hrajfrisbee.cz/kacerr/egress-proxies-operator/internal/provider" + "gitea.home.hrajfrisbee.cz/kacerr/egress-proxies-operator/internal/provider/nocloud" +) + +// seedFiles opens a built seed and returns its label and files by name. +func seedFiles(t *testing.T, iso *bytes.Buffer) (string, map[string]string) { + t.Helper() + img, err := iso9660.OpenImage(bytes.NewReader(iso.Bytes())) + if err != nil { + t.Fatalf("OpenImage: %v", err) + } + label, err := img.Label() + if err != nil { + t.Fatalf("Label: %v", err) + } + root, err := img.RootDir() + if err != nil { + t.Fatalf("RootDir: %v", err) + } + children, err := root.GetChildren() + if err != nil { + t.Fatalf("GetChildren: %v", err) + } + files := make(map[string]string, len(children)) + for _, c := range children { + data, err := io.ReadAll(c.Reader()) + if err != nil { + t.Fatalf("reading %s: %v", c.Name(), err) + } + files[c.Name()] = string(data) + } + return label, files +} + +func TestSeedNames_table(t *testing.T) { + t.Parallel() + if got, want := seedISOName("proxy-abc"), "proxy-abc-seed.iso"; got != want { + t.Errorf("seedISOName = %q, want %q", got, want) + } + if got, want := seedVolID("proxy-seeds", "proxy-abc"), "proxy-seeds:iso/proxy-abc-seed.iso"; got != want { + t.Errorf("seedVolID = %q, want %q", got, want) + } +} + +func TestBuildSeed_table(t *testing.T) { + t.Parallel() + off := false + userData := "#cloud-config\r\npackages: [squid]\r\nwrite_files:\n - content: \"caf\xc3\xa9\"\n" + configured := "#cloud-config\nntp:\n enabled: true\n" + tests := []struct { + name string + vendorData string + install *bool + want func(t *testing.T, got string) + }{ + {name: "agent script by default", want: func(t *testing.T, got string) { + if got != nocloud.GuestAgentScript { + t.Errorf("vendor-data is not the guest agent script: %q", got) + } + }}, + {name: "configured joined with the agent script", vendorData: configured, want: func(t *testing.T, got string) { + if !strings.HasPrefix(got, "MIME-Version: 1.0\r\n") || !strings.Contains(got, configured) || !strings.Contains(got, nocloud.GuestAgentScript) { + t.Errorf("vendor-data is not a multipart of both documents: %q", got) + } + }}, + {name: "agent off passes configured through", vendorData: configured, install: &off, want: func(t *testing.T, got string) { + if got != configured { + t.Errorf("vendor-data = %q, want the configured document", got) + } + }}, + {name: "agent off with nothing configured is empty", install: &off, want: func(t *testing.T, got string) { + if got != "" { + t.Errorf("vendor-data = %q, want empty", got) + } + }}, + } + for _, tc := range tests { + t.Run(tc.name, func(t *testing.T) { + t.Parallel() + cfg := testConfig() + cfg.VendorData = tc.vendorData + cfg.InstallGuestAgent = tc.install + p := newProvider("proxmox-home", cfg, nil) + iso, err := p.buildSeed(provider.CreateRequest{Name: "proxy-abc", CloudInit: userData}) + if err != nil { + t.Fatalf("buildSeed error = %v", err) + } + label, files := seedFiles(t, iso) + if label != nocloud.SeedLabel { + t.Errorf("label = %q, want %q", label, nocloud.SeedLabel) + } + if files["user-data"] != userData { + t.Errorf("user-data changed:\n got %q\nwant %q", files["user-data"], userData) + } + if want := "instance-id: proxy-abc\nlocal-hostname: proxy-abc\n"; files["meta-data"] != want { + t.Errorf("meta-data = %q, want %q", files["meta-data"], want) + } + tc.want(t, files["vendor-data"]) + }) + } +} + +func TestBuildSeed_emptyNameIsPermanentWithoutPayload(t *testing.T) { + t.Parallel() + const sentinel = "SENTINEL-cloud-init-payload" + p := newProvider("proxmox-home", testConfig(), nil) + _, err := p.buildSeed(provider.CreateRequest{CloudInit: sentinel}) + if !errors.Is(err, provider.ErrPermanent) { + t.Fatalf("buildSeed error = %v, want ErrPermanent", err) + } + if strings.Contains(err.Error(), sentinel) { + t.Errorf("error carries the cloud-init payload: %v", err) + } +} + +// Not parallel: it points TMPDIR at a path that cannot exist for the +// process. +func TestBuildSeed_stagingFailureIsTransient(t *testing.T) { + t.Setenv("TMPDIR", t.TempDir()+"/missing") + p := newProvider("proxmox-home", testConfig(), nil) + _, err := p.buildSeed(provider.CreateRequest{Name: "proxy-abc", CloudInit: "#cloud-config\n"}) + if err == nil { + t.Fatal("buildSeed with no staging dir succeeded") + } + if got := classify("create", err); got != provider.ErrTransient { + t.Errorf("classify(buildSeed error) = %v, want ErrTransient: %v", got, err) + } +} From e05d9949744050ad931980bfcca53eb273c4f16a Mon Sep 17 00:00:00 2001 From: kacerr Date: Wed, 23 Sep 2026 14:47:50 +0000 Subject: [PATCH 07/13] proxmox: thin HTTP client, seam, log decorator and in-memory fake httpAPI implements the sixteen seam methods over net/http: the token is read per request, non-2xx answers become apiError from the body message or the reason phrase, echoed secrets are redacted, the CA loads lazily, HTTP/2 is off, and uploads stream with an exact Content-Length. The observed decorator logs one V(1) line per call and starts no span; New wires both. fakeAPI simulates clone locks, tasks and ISOs with the captured error texts. Co-Authored-By: Claude Opus 5.5 (1M context) --- .../2026-09-18-1502-proxmox-provider.md | 109 ++- docs/proxmox-provider-spec.md | 5 +- internal/provider/proxmox/client.go | 378 ++++++++ internal/provider/proxmox/client_test.go | 874 +++++++++++++++++ internal/provider/proxmox/fake_test.go | 893 ++++++++++++++++++ internal/provider/proxmox/observe.go | 259 +++++ internal/provider/proxmox/observe_test.go | 229 +++++ internal/provider/proxmox/proxmox.go | 10 +- internal/provider/proxmox/proxmox_test.go | 25 + 9 files changed, 2775 insertions(+), 7 deletions(-) create mode 100644 internal/provider/proxmox/client.go create mode 100644 internal/provider/proxmox/client_test.go create mode 100644 internal/provider/proxmox/fake_test.go create mode 100644 internal/provider/proxmox/observe.go create mode 100644 internal/provider/proxmox/observe_test.go diff --git a/docs/plans-executions/2026-09-18-1502-proxmox-provider.md b/docs/plans-executions/2026-09-18-1502-proxmox-provider.md index 3c5e57e..ca16638 100644 --- a/docs/plans-executions/2026-09-18-1502-proxmox-provider.md +++ b/docs/plans-executions/2026-09-18-1502-proxmox-provider.md @@ -8,7 +8,7 @@ Pairs with [docs/plans/2026-09-18-1502-proxmox-provider.md](../plans/2026-09-18- - [x] Phase 1 — The spec (`docs/proxmox-provider-spec.md`) - [x] Phase 2 — Config block, registration, package skeleton - [x] Phase 3 — Extract `nocloud` and `keyedlock` from kvm2 -- [ ] Phase 4 — Seam, HTTP client, decorator, fake +- [x] Phase 4 — Seam, HTTP client, decorator, fake - [ ] Phase 5 — Get, ListByTag, task helpers - [ ] Phase 6 — Create - [ ] Phase 7 — Delete, Checker, CRD comment @@ -447,3 +447,110 @@ Worth noting: test starts its 200 ms release timers before `Check` starts its clock, so under load the measured latency can come in just under 200 ms. The flake predates this phase, and the test is not touched here. + +## Phase 4 — Seam, HTTP client, decorator, fake + +`client.go` holds `httpAPI`, the thin `net/http` adapter for all sixteen +seam methods. It reads the token file on every request, turns any non-2xx +answer into `*apiError` from the body's `message` or the reason phrase +(F1), and wraps its transport in `tracing.RestConfigWrapper`. `observe.go` +is the logging decorator: one V(1) `proxmox api call` line per call, +detail at V(2), and no span, because the transport already traces each +request. `New` now builds `observe(newHTTPAPI(...))` instead of passing a +nil `api`, still without touching the network or the token file. +`fake_test.go` holds `fakeAPI`, which Phases 5 to 7 build on. + +Where this phase departs from the plan's Phase 4 text: + +- **The `http.Client` is built on first use, not in `newHTTPAPI`.** Spec + entry 14 wins over the plan: the CA is loaded lazily, a missing CA file + is Transient and retried until the file lands, one without a certificate + is Permanent, and the pool is cached only once it loads. A transport's + `RootCAs` cannot change after first use, so the whole client waits for + the CA (`TestClient_caFileLoadedLazily`). +- **The upload is not an `io.Pipe` with a goroutine.** The multipart + framing (the `content=iso` field, the `filename` part header and the + closing boundary) is rendered into two small buffers, and the body is + `io.MultiReader(head, io.LimitReader(r, size), tail)` with an exact + `Content-Length`. The seed is still streamed and never held by the + client. There is also no goroutine to leak, and the request carries a + length, as Phase 0's `curl -F` upload did. A chunked upload to pveproxy + was never tried. `TestClient_uploadStreamsMultipart` proves the + streaming with a reader that stalls after its first 256 KiB until the + server has seen file bytes. +- **The fake's `DeleteVolume` tolerates a missing volume**, as spec + correction 5 and F13 say, instead of the plan's error. +- **Added:** secret redaction in `apiError` (contract §7), because + otherwise a server that echoes the header would put the secret into + `Proxy` status; an `errCredentials` sentinel on token-file errors, so + Phase 7's Checker can report `CredentialsUnavailable`; and the V(1) + identifiers `newid` and `name` on a clone, `filename` on an upload and + `volid` on a volume delete. Spec entry 20 now lists them. + +**The volid's `/` travels as `%2F`.** Neither the plan nor the spec says +how `{volid}` is encoded in the content-delete path. The Phase 0 session's +transcript shows every delete went through a helper encoding the whole +volid: + +```bash +enc() { python3 -c "import urllib.parse,sys; print(urllib.parse.quote(sys.argv[1], safe=''))" "$1"; } +curl ... -X DELETE ".../storage/proxy-seeds/content/$(enc 'proxy-seeds:iso/proxy-reconaaaaaaaaaaa-seed.iso')" +``` + +So `apiPath` runs `url.PathEscape` over each segment +(`TestClient_volIDSlashIsEscaped`). + +`net/http` cannot send a custom reason phrase, so the client tests replay +every capture byte for byte through a hijacked connection (`serveCapture`). +`TestClient_apiErrorFromStatusReasonAndBody` globs `testdata/err_*.json` +and skips `err_tag_hyphen.json`, which is a 200 despite its name: hyphenated +tags were accepted (F9). + +Tests beyond the plan's list: `TestClient_methodPathAndParams` (the method, +path, query and form of every seam method, with a reflection check that no +method lacks a row), `TestClient_cloneOmitsEmptyStorageAndTarget`, +`TestClient_volIDSlashIsEscaped`, `TestClient_caFileLoadedLazily`, +`TestClient_x509FailureIsPermanent` (from spec entry 21), +`TestDecodeConfig_scalarsAsStrings`, `TestReadToken_trimsTrailingNewline`, +`TestObserve_startsNoSpan`, `TestObserve_identifiersAtV1`, +`TestNew_wiresObservedHTTPClient`, and six `TestFakeAPI_*` tests. +`TestFakeAPI_errorsMatchCaptures` compares the fake's error texts with the +captures, so every test that goes through the fake exercises the real +wording. Texts that recon never produced start with `fake:`. + +A mutation check confirmed the new tests can fail: dropping `redact`, +buffering the upload with `io.ReadAll`, and logging V(2) detail at V(1) +each failed their test. Removing only the empty `TLSNextProto` map did +**not** fail `TestClient_http1Only`. A transport with a custom +`TLSClientConfig` and `DialContext` never offers h2 unless +`ForceAttemptHTTP2` is set, so the map is a second guard. With both +guards removed the test failed with `client spoke HTTP/2`. Every mutation +was reverted, and the files were compared identical to backups. + +```bash +go test -race -count=2 ./internal/provider/proxmox/ # ok, run three times +grep -n 'ReadAll' internal/provider/proxmox/client.go # only the 64 KiB error-body read +grep -n 'context.Background()' internal/provider/proxmox/*.go | grep -v _test.go # nothing +go build ./... && go vet ./... && go vet -tags=integration ./... +go test -short -count=1 ./... # all ok +go test -count=1 -v ./internal/provider/proxmox/ | grep -c '^=== RUN' # 161 before, 289 after +``` + +Worth noting: + +- `bin/golangci-lint run ./internal/provider/proxmox/...` panics inside + staticcheck's IR builder (`unexpected expr: *ast.KeyValueExpr` in the + Go 1.27 standard library's `poll` package). This is a toolchain problem, + not this code. With `--disable staticcheck` the new files' findings are + goconst on form keys (kvm2 carries 50 of the same) and two unparam hits + on the fake's `addTemplate` and `resourceByID`, which Phases 5 and 6 will + call with other VMIDs. The mapsloop and gocyclo findings in `proxmox.go` + and `api_test.go` date from Phase 2. +- Nothing in this phase talked to the host. The client has only met + `httptest` servers replaying the captures. The first real request is + Phase 10's integration run, which is where the `Content-Length` upload + and the `%2F` volid get proven against pveproxy. +- The fake refuses to destroy a running VM with an uncaptured + `fake: VM is running - destroy failed`, so Phase 7's + stop-before-destroy order is enforced by the fake even though the real + text is unknown. diff --git a/docs/proxmox-provider-spec.md b/docs/proxmox-provider-spec.md index 8b2bd2f..9e1dda4 100644 --- a/docs/proxmox-provider-spec.md +++ b/docs/proxmox-provider-spec.md @@ -696,8 +696,9 @@ named `proxmox`, with `provider` on every line and `vmid`, `name` and `proxmox vm fetched` with state, ip and uid, `proxmox vm deleted`, `proxmox vms listed` with counts, `proxmox node checked` with the reason), one `proxmox api call` line per seam call from the decorator (`call`, - `node`, `vmid`, `storage`, `upid`, `duration`, and on failure - `httpStatus` and `error`), a `proxmox task failed` line with the UPID and + `node`, `vmid`, `storage`, `upid`, `duration`, plus `newid` and `name` + on a clone, `filename` on an upload and `volid` on a volume delete, and + on failure `httpStatus` and `error`), a `proxmox task failed` line with the UPID and `exitstatus`, and the no-op lines of Delete and ListByTag (foreign VM left alone, unreadable candidate skipped). - V(2): request parameters (without `description`'s content at V(1) and diff --git a/internal/provider/proxmox/client.go b/internal/provider/proxmox/client.go new file mode 100644 index 0000000..c1df283 --- /dev/null +++ b/internal/provider/proxmox/client.go @@ -0,0 +1,378 @@ +package proxmox + +import ( + "bytes" + "context" + "crypto/tls" + "crypto/x509" + "encoding/json" + "errors" + "fmt" + "io" + "io/fs" + "mime/multipart" + "net" + "net/http" + "net/url" + "os" + "regexp" + "strconv" + "strings" + "sync" + "time" + + "gitea.home.hrajfrisbee.cz/kacerr/egress-proxies-operator/internal/provider" + "gitea.home.hrajfrisbee.cz/kacerr/egress-proxies-operator/internal/tracing" +) + +const ( + dialTimeout = 10 * time.Second + tlsHandshakeTimeout = 10 * time.Second + idleConnTimeout = 90 * time.Second + maxIdleConnsPerHost = 4 + // maxErrorBody caps how much of a failed response is read for its + // message; Proxmox error bodies are a few hundred bytes. + maxErrorBody = 64 << 10 +) + +// errCredentials marks a token file that is missing or unusable, so the +// Checker can report CredentialsUnavailable instead of an API failure. +var errCredentials = errors.New("proxmox token file unusable") + +// tokenPattern is @!= (spec entry 14). +var tokenPattern = regexp.MustCompile(`^[^@\s]+@[^!\s]+![^=\s]+=\S+$`) + +// httpAPI is the production adapter behind the api seam: a thin net/http +// client for /api2/json (spec entry 17). It is safe for concurrent use. +type httpAPI struct { + base *url.URL + tokenFile string + caFile string + insecure bool + // timeout bounds one request, response included. Read when the client + // is built; tests shrink it before the first call. + timeout time.Duration + wrap func(http.RoundTripper) http.RoundTripper + + // mu guards client, built on first use because the CA is loaded lazily + // and only cached once it loads (spec entry 14). + mu sync.Mutex + client *http.Client +} + +var _ api = (*httpAPI)(nil) + +// newHTTPAPI builds the adapter without touching the network or any file. +func newHTTPAPI(cfg provider.ProxmoxConfig, opts ...tracing.Option) (*httpAPI, error) { + u, err := url.Parse(cfg.URL) + if err != nil { + return nil, fmt.Errorf("proxmox.url: %w", err) + } + return &httpAPI{ + base: u.JoinPath("api2", "json"), + tokenFile: cfg.TokenFile, + caFile: cfg.CAFile, + insecure: cfg.InsecureSkipTLSVerify, + timeout: time.Duration(cfg.RequestTimeoutSeconds) * time.Second, + wrap: tracing.RestConfigWrapper(opts...), + }, nil +} + +// httpClient returns the shared client, building it once the CA loads. A +// missing CA file is Transient and retried on the next call; one without a +// certificate is Permanent. +func (a *httpAPI) httpClient() (*http.Client, error) { + a.mu.Lock() + defer a.mu.Unlock() + if a.client != nil { + return a.client, nil + } + tlsCfg := &tls.Config{MinVersion: tls.VersionTLS12, InsecureSkipVerify: a.insecure} //nolint:gosec // opt-in, warned about at startup + if a.caFile != "" && !a.insecure { + pem, err := os.ReadFile(a.caFile) + if err != nil { + return nil, fmt.Errorf("%w: reading CA file: %w", provider.ErrTransient, err) + } + pool := x509.NewCertPool() + if !pool.AppendCertsFromPEM(pem) { + return nil, permanentf("CA file %s holds no PEM certificate", a.caFile) + } + tlsCfg.RootCAs = pool + } + transport := &http.Transport{ + Proxy: http.ProxyFromEnvironment, + DialContext: (&net.Dialer{Timeout: dialTimeout, KeepAlive: 30 * time.Second}).DialContext, + TLSClientConfig: tlsCfg, + TLSHandshakeTimeout: tlsHandshakeTimeout, + ResponseHeaderTimeout: a.timeout, + IdleConnTimeout: idleConnTimeout, + MaxIdleConnsPerHost: maxIdleConnsPerHost, + // pveproxy speaks HTTP/1.1 only (F18); a non-nil empty map keeps + // the transport from offering h2. + ForceAttemptHTTP2: false, + TLSNextProto: map[string]func(string, *tls.Conn) http.RoundTripper{}, + } + a.client = &http.Client{Transport: a.wrap(transport), Timeout: a.timeout} + return a.client, nil +} + +// readToken reads the token file on every request, so a rotated Secret +// takes effect at once. Neither error names the file's content. +func readToken(path string) (string, error) { + raw, err := os.ReadFile(path) + if errors.Is(err, fs.ErrNotExist) { + return "", fmt.Errorf("%w: %w: %s does not exist yet", provider.ErrTransient, errCredentials, path) + } + if err != nil { + return "", fmt.Errorf("%w: reading %s: %w", errCredentials, path, err) + } + token := strings.TrimSpace(string(raw)) + if !tokenPattern.MatchString(token) { + return "", fmt.Errorf("%w: %w: %s is not @!=", provider.ErrPermanent, errCredentials, path) + } + return token, nil +} + +// apiPath joins path segments, escaping each, so a volid's "/" travels as +// %2F as Phase 0 sent it. +func apiPath(segments ...string) string { + escaped := make([]string, len(segments)) + for i, s := range segments { + escaped[i] = url.PathEscape(s) + } + return strings.Join(escaped, "/") +} + +func vmPath(node string, vmid int, rest ...string) string { + return apiPath(append([]string{"nodes", node, "qemu", strconv.Itoa(vmid)}, rest...)...) +} + +// do sends one request and decodes the response's data member into out, +// which may be nil. POST and PUT send form as the body; every method sends +// query in the URL. Any non-2xx answer is an *apiError. +func (a *httpAPI) do(ctx context.Context, method, path string, query, form url.Values, out any) error { + var body io.Reader + contentType := "" + if form != nil { + body = strings.NewReader(form.Encode()) + contentType = "application/x-www-form-urlencoded" + } + return a.send(ctx, method, path, query, body, -1, contentType, out) +} + +// send is do with a caller-built body. length is the body's size, or -1 +// when do built it from a string the request can measure itself. +func (a *httpAPI) send(ctx context.Context, method, path string, query url.Values, body io.Reader, length int64, contentType string, out any) error { + token, err := readToken(a.tokenFile) + if err != nil { + return err + } + client, err := a.httpClient() + if err != nil { + return err + } + u := a.base.JoinPath(path) + u.RawQuery = query.Encode() + req, err := http.NewRequestWithContext(ctx, method, u.String(), body) + if err != nil { + return fmt.Errorf("building %s %s: %w", method, path, err) + } + if length >= 0 { + req.ContentLength = length + } + if contentType != "" { + req.Header.Set("Content-Type", contentType) + } + req.Header.Set("Accept", "application/json") + req.Header.Set("Authorization", "PVEAPIToken="+token) + resp, err := client.Do(req) + if err != nil { + return err + } + defer func() { + _, _ = io.Copy(io.Discard, io.LimitReader(resp.Body, maxErrorBody)) + _ = resp.Body.Close() + }() + if resp.StatusCode < 200 || resp.StatusCode > 299 { + raw, _ := io.ReadAll(io.LimitReader(resp.Body, maxErrorBody)) + reason := strings.TrimPrefix(resp.Status, strconv.Itoa(resp.StatusCode)) + return redact(newAPIError(resp.StatusCode, reason, raw), token) + } + var env struct { + Data json.RawMessage `json:"data"` + } + if err := json.NewDecoder(resp.Body).Decode(&env); err != nil { + return fmt.Errorf("decoding %s %s: %w", method, path, err) + } + if out == nil || len(env.Data) == 0 { + return nil + } + if err := json.Unmarshal(env.Data, out); err != nil { + return fmt.Errorf("decoding %s %s data: %w", method, path, err) + } + return nil +} + +// redact removes the token's secret from an error Proxmox echoed back, so +// it cannot reach a log line, a span or a Proxy's status (spec entry 14). +func redact(e *apiError, token string) *apiError { + _, secret, _ := strings.Cut(token, "=") + if secret == "" { + return e + } + e.Message = strings.ReplaceAll(e.Message, secret, "") + for k, v := range e.Errors { + e.Errors[k] = strings.ReplaceAll(v, secret, "") + } + return e +} + +// decodeConfig renders every config value as a string: Proxmox sends some +// scalars as JSON numbers (cores) and others as strings (memory). A null +// value is dropped. +func decodeConfig(raw map[string]json.RawMessage) (vmConfig, error) { + cfg := make(vmConfig, len(raw)) + for k, v := range raw { + switch { + case len(v) == 0 || string(v) == "null": + continue + case v[0] == '"': + var s string + if err := json.Unmarshal(v, &s); err != nil { + return nil, fmt.Errorf("config key %s: %w", k, err) + } + cfg[k] = s + default: + cfg[k] = string(v) + } + } + return cfg, nil +} + +func (a *httpAPI) Version(ctx context.Context) (v versionInfo, err error) { + err = a.do(ctx, http.MethodGet, "version", nil, nil, &v) + return v, err +} + +func (a *httpAPI) NodeStatus(ctx context.Context, node string) (s nodeStatus, err error) { + err = a.do(ctx, http.MethodGet, apiPath("nodes", node, "status"), nil, nil, &s) + return s, err +} + +func (a *httpAPI) StorageStatus(ctx context.Context, node, storage string) (s storageStatus, err error) { + err = a.do(ctx, http.MethodGet, apiPath("nodes", node, "storage", storage, "status"), nil, nil, &s) + return s, err +} + +func (a *httpAPI) ClusterResources(ctx context.Context) (rs []vmResource, err error) { + err = a.do(ctx, http.MethodGet, "cluster/resources", url.Values{"type": {"vm"}}, nil, &rs) + return rs, err +} + +func (a *httpAPI) VMConfig(ctx context.Context, node string, vmid int) (vmConfig, error) { + var raw map[string]json.RawMessage + if err := a.do(ctx, http.MethodGet, vmPath(node, vmid, "config"), nil, nil, &raw); err != nil { + return nil, err + } + return decodeConfig(raw) +} + +func (a *httpAPI) SetVMConfig(ctx context.Context, node string, vmid int, params map[string]string) error { + form := url.Values{} + for k, v := range params { + form.Set(k, v) + } + return a.do(ctx, http.MethodPut, vmPath(node, vmid, "config"), nil, form, nil) +} + +func (a *httpAPI) Clone(ctx context.Context, node string, vmid int, p cloneParams) (upid string, err error) { + form := url.Values{ + "newid": {strconv.Itoa(p.NewID)}, + "name": {p.Name}, + "description": {p.Description}, + "full": {"1"}, + } + if p.Storage != "" { + form.Set("storage", p.Storage) + } + if p.Target != "" { + form.Set("target", p.Target) + } + err = a.do(ctx, http.MethodPost, vmPath(node, vmid, "clone"), nil, form, &upid) + return upid, err +} + +func (a *httpAPI) Resize(ctx context.Context, node string, vmid int, disk, size string) (upid string, err error) { + form := url.Values{"disk": {disk}, "size": {size}} + err = a.do(ctx, http.MethodPut, vmPath(node, vmid, "resize"), nil, form, &upid) + return upid, err +} + +func (a *httpAPI) Start(ctx context.Context, node string, vmid int) (upid string, err error) { + err = a.do(ctx, http.MethodPost, vmPath(node, vmid, "status", "start"), nil, url.Values{}, &upid) + return upid, err +} + +func (a *httpAPI) Stop(ctx context.Context, node string, vmid int) (upid string, err error) { + err = a.do(ctx, http.MethodPost, vmPath(node, vmid, "status", "stop"), nil, url.Values{}, &upid) + return upid, err +} + +func (a *httpAPI) VMStatus(ctx context.Context, node string, vmid int) (s vmStatus, err error) { + err = a.do(ctx, http.MethodGet, vmPath(node, vmid, "status", "current"), nil, nil, &s) + return s, err +} + +// Destroy purges the VM from backup jobs and HA and removes disks no config +// references, so nothing of it is left behind (spec entry 11). +func (a *httpAPI) Destroy(ctx context.Context, node string, vmid int) (upid string, err error) { + query := url.Values{"purge": {"1"}, "destroy-unreferenced-disks": {"1"}} + err = a.do(ctx, http.MethodDelete, vmPath(node, vmid), query, nil, &upid) + return upid, err +} + +func (a *httpAPI) TaskStatus(ctx context.Context, node, upid string) (s taskStatus, err error) { + err = a.do(ctx, http.MethodGet, apiPath("nodes", node, "tasks", upid, "status"), nil, nil, &s) + return s, err +} + +func (a *httpAPI) AgentNetworkInterfaces(ctx context.Context, node string, vmid int) ([]agentInterface, error) { + var out struct { + Result []agentInterface `json:"result"` + } + err := a.do(ctx, http.MethodGet, vmPath(node, vmid, "agent", "network-get-interfaces"), nil, nil, &out) + return out.Result, err +} + +// UploadISO streams r as the file part of a multipart upload (F12). The +// multipart framing is rendered up front, so the request carries an exact +// Content-Length, as Phase 0's curl -F upload did, while r itself is read +// in the transport's chunks and never held in memory. +func (a *httpAPI) UploadISO(ctx context.Context, node, storage, filename string, r io.Reader, size int64) (upid string, err error) { + var frame bytes.Buffer + mw := multipart.NewWriter(&frame) + if err := mw.WriteField("content", "iso"); err != nil { + return "", fmt.Errorf("framing upload: %w", err) + } + if _, err := mw.CreateFormFile("filename", filename); err != nil { + return "", fmt.Errorf("framing upload: %w", err) + } + head := bytes.Clone(frame.Bytes()) + frame.Reset() + if err := mw.Close(); err != nil { + return "", fmt.Errorf("framing upload: %w", err) + } + tail := frame.Bytes() + if r == nil { + r = bytes.NewReader(nil) + } + body := io.MultiReader(bytes.NewReader(head), io.LimitReader(r, size), bytes.NewReader(tail)) + length := int64(len(head)) + size + int64(len(tail)) + err = a.send(ctx, http.MethodPost, apiPath("nodes", node, "storage", storage, "upload"), nil, body, length, mw.FormDataContentType(), &upid) + return upid, err +} + +func (a *httpAPI) DeleteVolume(ctx context.Context, node, storage, volid string) (upid string, err error) { + err = a.do(ctx, http.MethodDelete, apiPath("nodes", node, "storage", storage, "content", volid), nil, nil, &upid) + return upid, err +} diff --git a/internal/provider/proxmox/client_test.go b/internal/provider/proxmox/client_test.go new file mode 100644 index 0000000..3295047 --- /dev/null +++ b/internal/provider/proxmox/client_test.go @@ -0,0 +1,874 @@ +package proxmox + +import ( + "bytes" + "context" + "encoding/json" + "encoding/pem" + "errors" + "fmt" + "io" + stdlog "log" + "net" + "net/http" + "net/http/httptest" + "net/url" + "os" + "path/filepath" + "reflect" + "strings" + "sync" + "sync/atomic" + "testing" + "time" + + sdktrace "go.opentelemetry.io/otel/sdk/trace" + "go.opentelemetry.io/otel/sdk/trace/tracetest" + + "gitea.home.hrajfrisbee.cz/kacerr/egress-proxies-operator/internal/provider" + "gitea.home.hrajfrisbee.cz/kacerr/egress-proxies-operator/internal/tracing" +) + +const testToken = "proxy-operator@pve!operator=0b4e1c2a-5d6f-4a7b-8c9d-0e1f2a3b4c5d" + +// clientFixture is a TLS test server, a CA file trusting it, a token file +// and the config pointing at them. +type clientFixture struct { + srv *httptest.Server + cfg provider.ProxmoxConfig + tokenFile string + caFile string +} + +// newClientFixture starts h on a TLS server whose error log is discarded: +// the tests that fail a handshake on purpose would otherwise print it. +func newClientFixture(t *testing.T, h http.HandlerFunc) *clientFixture { + t.Helper() + srv := httptest.NewUnstartedServer(h) + srv.Config.ErrorLog = stdlog.New(io.Discard, "", 0) + srv.StartTLS() + t.Cleanup(srv.Close) + return fixtureFor(t, srv) +} + +func fixtureFor(t *testing.T, srv *httptest.Server) *clientFixture { + t.Helper() + dir := t.TempDir() + fx := &clientFixture{srv: srv, tokenFile: filepath.Join(dir, "token"), caFile: filepath.Join(dir, "ca.pem")} + writeFile(t, fx.tokenFile, testToken+"\n") + writeFile(t, fx.caFile, string(pem.EncodeToMemory(&pem.Block{Type: "CERTIFICATE", Bytes: srv.Certificate().Raw}))) + fx.cfg = provider.ProxmoxConfig{URL: srv.URL, TokenFile: fx.tokenFile, CAFile: fx.caFile, RequestTimeoutSeconds: 30} + return fx +} + +func (fx *clientFixture) api(t *testing.T, opts ...tracing.Option) *httpAPI { + t.Helper() + a, err := newHTTPAPI(fx.cfg, opts...) + if err != nil { + t.Fatalf("newHTTPAPI error = %v", err) + } + return a +} + +func writeFile(t *testing.T, path, content string) { + t.Helper() + if err := os.WriteFile(path, []byte(content), 0o600); err != nil { + t.Fatalf("writing %s: %v", path, err) + } +} + +// serveCapture writes a Phase 0 capture back byte for byte, reason phrase +// included, which net/http's ResponseWriter cannot do. +func serveCapture(t *testing.T, w http.ResponseWriter, c capture) { + t.Helper() + conn, buf, err := w.(http.Hijacker).Hijack() + if err != nil { + t.Errorf("hijack: %v", err) + return + } + defer func() { _ = conn.Close() }() + _, _ = fmt.Fprintf(buf, "HTTP/1.1 %d %s\r\nContent-Type: application/json;charset=UTF-8\r\nContent-Length: %d\r\nConnection: close\r\n\r\n%s", + c.Status, c.Reason, len(c.Body), c.Body) + _ = buf.Flush() +} + +func serveJSON(w http.ResponseWriter, body string) { + w.Header().Set("Content-Type", "application/json;charset=UTF-8") + _, _ = io.WriteString(w, body) +} + +func assertClass(t *testing.T, err, class error) { + t.Helper() + if got := classify("get", err); got != class { + t.Errorf("classify(%v) = %v, want %v", err, got, class) + } +} + +func TestClient_sendsTokenHeaderFromFileOnEveryRequest(t *testing.T) { + t.Parallel() + var ( + mu sync.Mutex + headers []string + ) + fx := newClientFixture(t, func(w http.ResponseWriter, r *http.Request) { + mu.Lock() + headers = append(headers, r.Header.Get("Authorization")) + mu.Unlock() + serveJSON(w, readCapture(t, "version").Body) + }) + a := fx.api(t) + const rotated = "proxy-operator@pve!operator=11111111-2222-3333-4444-555555555555" + for i, token := range []string{testToken, rotated} { + writeFile(t, fx.tokenFile, token+"\n") + v, err := a.Version(context.Background()) + if err != nil || v.Version != "9.2.20" { + t.Fatalf("call %d: Version() = %+v, %v", i, v, err) + } + } + want := []string{"PVEAPIToken=" + testToken, "PVEAPIToken=" + rotated} + mu.Lock() + defer mu.Unlock() + if !reflect.DeepEqual(headers, want) { + t.Errorf("Authorization headers = %q, want %q", headers, want) + } +} + +func TestClient_missingTokenFileIsTransient(t *testing.T) { + t.Parallel() + var requests atomic.Int32 + fx := newClientFixture(t, func(http.ResponseWriter, *http.Request) { requests.Add(1) }) + fx.cfg.TokenFile = filepath.Join(t.TempDir(), "absent") + _, err := fx.api(t).Version(context.Background()) + if !errors.Is(err, provider.ErrTransient) || !errors.Is(err, errCredentials) { + t.Errorf("error = %v, want ErrTransient and errCredentials", err) + } + if n := requests.Load(); n != 0 { + t.Errorf("server saw %d requests, want none without a token", n) + } +} + +func TestClient_unparsableTokenFileIsPermanent(t *testing.T) { + t.Parallel() + const sentinel = "SENTINEL-7f3e" + tests := []struct{ name, content string }{ + {name: "empty", content: ""}, + {name: "secret only", content: sentinel}, + {name: "no secret", content: "proxy-operator@pve!operator"}, + {name: "no token id", content: "proxy-operator@pve=" + sentinel}, + {name: "no realm", content: "proxy-operator!operator=" + sentinel}, + {name: "two lines", content: "proxy-operator@pve!operator=" + sentinel + "\nsecond=line"}, + } + for _, tc := range tests { + t.Run(tc.name, func(t *testing.T) { + t.Parallel() + var requests atomic.Int32 + fx := newClientFixture(t, func(http.ResponseWriter, *http.Request) { requests.Add(1) }) + writeFile(t, fx.tokenFile, tc.content) + _, err := fx.api(t).Version(context.Background()) + if !errors.Is(err, provider.ErrPermanent) || !errors.Is(err, errCredentials) { + t.Errorf("error = %v, want ErrPermanent and errCredentials", err) + } + if err != nil && strings.Contains(err.Error(), sentinel) { + t.Errorf("error carries the file's content: %v", err) + } + if n := requests.Load(); n != 0 { + t.Errorf("server saw %d requests, want none", n) + } + }) + } +} + +// TestClient_apiErrorFromStatusReasonAndBody serves every captured error +// verbatim and expects the apiError the captures describe (F1): the body's +// message, or the reason phrase when the body is empty. +func TestClient_apiErrorFromStatusReasonAndBody(t *testing.T) { + t.Parallel() + paths, err := filepath.Glob(filepath.Join("testdata", "err_*.json")) + if err != nil || len(paths) == 0 { + t.Fatalf("no error captures: %v", err) + } + for _, path := range paths { + name := strings.TrimSuffix(filepath.Base(path), ".json") + if readCapture(t, name).Status < 400 { + continue // err_tag_hyphen: hyphens turned out to be accepted (F9) + } + t.Run(name, func(t *testing.T) { + t.Parallel() + c := readCapture(t, name) + fx := newClientFixture(t, func(w http.ResponseWriter, _ *http.Request) { serveCapture(t, w, c) }) + _, err := fx.api(t).Version(context.Background()) + var ae *apiError + if !errors.As(err, &ae) { + t.Fatalf("error = %v (%T), want *apiError", err, err) + } + want := loadAPIError(t, name) + if ae.Status != want.Status || ae.Message != want.Message || !reflect.DeepEqual(ae.Errors, want.Errors) { + t.Errorf("apiError = %+v, want %+v", ae, want) + } + if ae.Message == "" || strings.HasSuffix(ae.Message, "\n") { + t.Errorf("message %q is empty or untrimmed", ae.Message) + } + }) + } +} + +func TestClient_decodesDataEnvelope(t *testing.T) { + t.Parallel() + ctx := context.Background() + tests := []struct { + capture string + call func(a *httpAPI) (any, error) + check func(t *testing.T, got any) + }{ + {capture: "version", call: func(a *httpAPI) (any, error) { return a.Version(ctx) }, check: func(t *testing.T, got any) { + if v := got.(versionInfo); v.Version != "9.2.20" { + t.Errorf("version = %+v", v) + } + }}, + {capture: "node_status", call: func(a *httpAPI) (any, error) { return a.NodeStatus(ctx, "pv1-1") }, check: func(t *testing.T, got any) { + if s := got.(nodeStatus); s.CurrentKernel.Release != "7.0.14-17-pve" { + t.Errorf("node status = %+v", s) + } + }}, + {capture: "storage_status", call: func(a *httpAPI) (any, error) { return a.StorageStatus(ctx, "pv1-1", "local") }, check: func(t *testing.T, got any) { + if s := got.(storageStatus); s.Avail <= 0 || !strings.Contains(s.Content, "iso") { + t.Errorf("storage status = %+v", s) + } + }}, + {capture: "cluster_resources_during_clone", call: func(a *httpAPI) (any, error) { return a.ClusterResources(ctx) }, check: func(t *testing.T, got any) { + rs := got.([]vmResource) + for _, r := range rs { + if r.VMID == 199999 && r.Lock == "clone" && r.Name == "" { + return + } + } + t.Errorf("resources = %+v, want the nameless locked clone", rs) + }}, + {capture: "vm_config_clone", call: func(a *httpAPI) (any, error) { return a.VMConfig(ctx, "pv1-1", 199999) }, check: func(t *testing.T, got any) { + cfg := got.(vmConfig) + if cfg["cores"] != "2" || cfg["memory"] != "2048" || cfg["agent"] != "1" || cfg["boot"] != "order=scsi0" { + t.Errorf("config scalars = cores %q memory %q agent %q boot %q", cfg["cores"], cfg["memory"], cfg["agent"], cfg["boot"]) + } + if m := parseDescription(cfg["description"]); !m.Managed || m.UID == "" { + t.Errorf("description %q did not survive decoding", cfg["description"]) + } + }}, + {capture: "status_current_paused", call: func(a *httpAPI) (any, error) { return a.VMStatus(ctx, "pv1-1", 199999) }, check: func(t *testing.T, got any) { + if s := got.(vmStatus); s.Status != "running" || s.QMPStatus != "paused" { + t.Errorf("status = %+v", s) + } + }}, + {capture: "task_status_stopped_error", call: func(a *httpAPI) (any, error) { return a.TaskStatus(ctx, "pv1-1", "UPID:pv1-1:x") }, check: func(t *testing.T, got any) { + if s := got.(taskStatus); s.Status != "stopped" || s.ExitStatus != "shrinking disks is not supported" { + t.Errorf("task = %+v", s) + } + }}, + {capture: "agent_network_get_interfaces", call: func(a *httpAPI) (any, error) { return a.AgentNetworkInterfaces(ctx, "pv1-1", 199999) }, check: func(t *testing.T, got any) { + if ip := firstIPv4(got.([]agentInterface)); ip != "192.168.0.150" { + t.Errorf("first IPv4 = %q", ip) + } + }}, + {capture: "set_config_response", call: func(a *httpAPI) (any, error) { + return nil, a.SetVMConfig(ctx, "pv1-1", 199999, map[string]string{"cores": "1"}) + }, check: func(*testing.T, any) {}}, + {capture: "clone_response", call: func(a *httpAPI) (any, error) { return a.Clone(ctx, "pv1-1", 9001, cloneParams{NewID: 199999}) }, check: wantUPID("qmclone")}, + {capture: "resize_response", call: func(a *httpAPI) (any, error) { return a.Resize(ctx, "pv1-1", 199999, "scsi0", "4G") }, check: wantUPID("resize")}, + {capture: "start_response", call: func(a *httpAPI) (any, error) { return a.Start(ctx, "pv1-1", 199999) }, check: wantUPID("qmstart")}, + {capture: "stop_response", call: func(a *httpAPI) (any, error) { return a.Stop(ctx, "pv1-1", 199999) }, check: wantUPID("qmstop")}, + {capture: "destroy_response", call: func(a *httpAPI) (any, error) { return a.Destroy(ctx, "pv1-1", 199999) }, check: wantUPID("qmdestroy")}, + {capture: "upload_response", call: func(a *httpAPI) (any, error) { + return a.UploadISO(ctx, "pv1-1", "local", "x.iso", strings.NewReader("iso"), 3) + }, check: wantUPID("imgcopy")}, + {capture: "content_delete_response", call: func(a *httpAPI) (any, error) { + return a.DeleteVolume(ctx, "pv1-1", "proxy-seeds", "proxy-seeds:iso/x.iso") + }, check: wantUPID("imgdel")}, + } + for _, tc := range tests { + t.Run(tc.capture, func(t *testing.T) { + t.Parallel() + c := readCapture(t, tc.capture) + fx := newClientFixture(t, func(w http.ResponseWriter, r *http.Request) { + _, _ = io.Copy(io.Discard, r.Body) + serveCapture(t, w, c) + }) + got, err := tc.call(fx.api(t)) + if err != nil { + t.Fatalf("call error = %v", err) + } + tc.check(t, got) + }) + } +} + +func wantUPID(kind string) func(t *testing.T, got any) { + return func(t *testing.T, got any) { + t.Helper() + if upid := got.(string); !strings.HasPrefix(upid, "UPID:pv1-1:") || !strings.Contains(upid, ":"+kind+":") { + t.Errorf("upid = %q, want a %s UPID of pv1-1", upid, kind) + } + } +} + +// seenRequest is what a test server records of one request. +type seenRequest struct { + method, path, escapedPath string + query, form url.Values + contentType string +} + +func recordingServer(t *testing.T, seen chan<- seenRequest) *clientFixture { + t.Helper() + return newClientFixture(t, func(w http.ResponseWriter, r *http.Request) { + req := seenRequest{method: r.Method, path: r.URL.Path, escapedPath: r.URL.EscapedPath(), query: r.URL.Query(), contentType: r.Header.Get("Content-Type")} + if r.Method == http.MethodPost || r.Method == http.MethodPut { + if err := r.ParseForm(); err != nil { + t.Errorf("parsing form: %v", err) + } + req.form = r.PostForm + } + seen <- req + serveJSON(w, `{"data":null}`) + }) +} + +// TestClient_methodPathAndParams pins the HTTP method, path and parameters +// of every seam method to the endpoint map of spec entry 17. +func TestClient_methodPathAndParams(t *testing.T) { + t.Parallel() + ctx := context.Background() + const upid = "UPID:pv1-1:0010FBF1:02804D0D:6AB378EA:qmstart:199999:proxy-operator@pve!operator:" + tests := map[string]struct { + call func(a *httpAPI) error + method string + path string + query url.Values + form url.Values + }{ + "Version": {call: func(a *httpAPI) error { _, err := a.Version(ctx); return err }, method: "GET", path: "/api2/json/version"}, + "NodeStatus": {call: func(a *httpAPI) error { _, err := a.NodeStatus(ctx, "pv1-1"); return err }, + method: "GET", path: "/api2/json/nodes/pv1-1/status"}, + "StorageStatus": {call: func(a *httpAPI) error { _, err := a.StorageStatus(ctx, "pv1-1", "proxy-seeds"); return err }, + method: "GET", path: "/api2/json/nodes/pv1-1/storage/proxy-seeds/status"}, + "ClusterResources": {call: func(a *httpAPI) error { _, err := a.ClusterResources(ctx); return err }, + method: "GET", path: "/api2/json/cluster/resources", query: url.Values{"type": {"vm"}}}, + "VMConfig": {call: func(a *httpAPI) error { _, err := a.VMConfig(ctx, "pv1-1", 150042); return err }, + method: "GET", path: "/api2/json/nodes/pv1-1/qemu/150042/config"}, + "SetVMConfig": {call: func(a *httpAPI) error { + return a.SetVMConfig(ctx, "pv1-1", 150042, map[string]string{"cores": "1", "ide2": "proxy-seeds:iso/proxy-x-seed.iso,media=cdrom"}) + }, method: "PUT", path: "/api2/json/nodes/pv1-1/qemu/150042/config", + form: url.Values{"cores": {"1"}, "ide2": {"proxy-seeds:iso/proxy-x-seed.iso,media=cdrom"}}}, + "Clone": {call: func(a *httpAPI) error { + _, err := a.Clone(ctx, "pv1-1", 9001, cloneParams{NewID: 150042, Name: "proxy-x", Description: "a=1\nb=2", Storage: "local-lvm", Target: "pv1-2"}) + return err + }, method: "POST", path: "/api2/json/nodes/pv1-1/qemu/9001/clone", + form: url.Values{"newid": {"150042"}, "name": {"proxy-x"}, "description": {"a=1\nb=2"}, "full": {"1"}, "storage": {"local-lvm"}, "target": {"pv1-2"}}}, + "Resize": {call: func(a *httpAPI) error { _, err := a.Resize(ctx, "pv1-1", 150042, "scsi0", "8G"); return err }, + method: "PUT", path: "/api2/json/nodes/pv1-1/qemu/150042/resize", form: url.Values{"disk": {"scsi0"}, "size": {"8G"}}}, + "Start": {call: func(a *httpAPI) error { _, err := a.Start(ctx, "pv1-1", 150042); return err }, + method: "POST", path: "/api2/json/nodes/pv1-1/qemu/150042/status/start", form: url.Values{}}, + "Stop": {call: func(a *httpAPI) error { _, err := a.Stop(ctx, "pv1-1", 150042); return err }, + method: "POST", path: "/api2/json/nodes/pv1-1/qemu/150042/status/stop", form: url.Values{}}, + "VMStatus": {call: func(a *httpAPI) error { _, err := a.VMStatus(ctx, "pv1-1", 150042); return err }, + method: "GET", path: "/api2/json/nodes/pv1-1/qemu/150042/status/current"}, + "Destroy": {call: func(a *httpAPI) error { _, err := a.Destroy(ctx, "pv1-1", 150042); return err }, + method: "DELETE", path: "/api2/json/nodes/pv1-1/qemu/150042", query: url.Values{"purge": {"1"}, "destroy-unreferenced-disks": {"1"}}}, + "TaskStatus": {call: func(a *httpAPI) error { _, err := a.TaskStatus(ctx, "pv1-1", upid); return err }, + method: "GET", path: "/api2/json/nodes/pv1-1/tasks/" + upid + "/status"}, + "AgentNetworkInterfaces": {call: func(a *httpAPI) error { _, err := a.AgentNetworkInterfaces(ctx, "pv1-1", 150042); return err }, + method: "GET", path: "/api2/json/nodes/pv1-1/qemu/150042/agent/network-get-interfaces"}, + "UploadISO": {call: func(a *httpAPI) error { + _, err := a.UploadISO(ctx, "pv1-1", "proxy-seeds", "proxy-x-seed.iso", strings.NewReader("iso"), 3) + return err + }, method: "POST", path: "/api2/json/nodes/pv1-1/storage/proxy-seeds/upload"}, // multipart, see TestClient_uploadStreamsMultipart + "DeleteVolume": {call: func(a *httpAPI) error { + _, err := a.DeleteVolume(ctx, "pv1-1", "proxy-seeds", "proxy-seeds:iso/proxy-x-seed.iso") + return err + }, method: "DELETE", path: "/api2/json/nodes/pv1-1/storage/proxy-seeds/content/proxy-seeds:iso/proxy-x-seed.iso"}, + } + for m := range reflect.TypeFor[api]().Methods() { + if _, ok := tests[m.Name]; !ok { + t.Errorf("no row for seam method %s", m.Name) + } + } + for name, tc := range tests { + t.Run(name, func(t *testing.T) { + t.Parallel() + seen := make(chan seenRequest, 1) + fx := recordingServer(t, seen) + if err := tc.call(fx.api(t)); err != nil { + t.Fatalf("call error = %v", err) + } + got := <-seen + if got.method != tc.method || got.path != tc.path { + t.Errorf("request = %s %s, want %s %s", got.method, got.path, tc.method, tc.path) + } + if len(got.query) != 0 || len(tc.query) != 0 { + if !reflect.DeepEqual(got.query, tc.query) { + t.Errorf("query = %v, want %v", got.query, tc.query) + } + } + if tc.form == nil { + if got.form != nil && len(got.form) != 0 { + t.Errorf("form = %v, want none", got.form) + } + return + } + if got.contentType != "application/x-www-form-urlencoded" { + t.Errorf("content type = %q, want a urlencoded form", got.contentType) + } + for k, want := range tc.form { + if !reflect.DeepEqual(got.form[k], want) { + t.Errorf("form %s = %q, want %q", k, got.form[k], want) + } + } + if len(got.form) != len(tc.form) { + t.Errorf("form = %v, want exactly %v", got.form, tc.form) + } + }) + } +} + +func TestClient_cloneOmitsEmptyStorageAndTarget(t *testing.T) { + t.Parallel() + seen := make(chan seenRequest, 1) + fx := recordingServer(t, seen) + if _, err := fx.api(t).Clone(context.Background(), "pv1-1", 9001, cloneParams{NewID: 150042, Name: "proxy-x"}); err != nil { + t.Fatalf("Clone error = %v", err) + } + got := <-seen + if _, ok := got.form["storage"]; ok { + t.Errorf("form = %v, want no storage", got.form) + } + if _, ok := got.form["target"]; ok { + t.Errorf("form = %v, want no target", got.form) + } +} + +// TestClient_destroyQueryFlags checks that destroy leaves nothing behind: +// the purge and unreferenced-disk flags travel in the query of the DELETE +// (spec entry 11). +func TestClient_destroyQueryFlags(t *testing.T) { + t.Parallel() + seen := make(chan seenRequest, 1) + fx := recordingServer(t, seen) + if _, err := fx.api(t).Destroy(context.Background(), "pv1-1", 150042); err != nil { + t.Fatalf("Destroy error = %v", err) + } + got := <-seen + if got.method != http.MethodDelete || got.query.Get("purge") != "1" || got.query.Get("destroy-unreferenced-disks") != "1" { + t.Errorf("request = %s %s?%s, want DELETE with purge=1 and destroy-unreferenced-disks=1", got.method, got.path, got.query.Encode()) + } +} + +// TestClient_volIDSlashIsEscaped sends the volid's "/" as %2F, the way +// Phase 0 deleted volumes, so the router sees one path segment. +func TestClient_volIDSlashIsEscaped(t *testing.T) { + t.Parallel() + seen := make(chan seenRequest, 1) + fx := recordingServer(t, seen) + if _, err := fx.api(t).DeleteVolume(context.Background(), "pv1-1", "proxy-seeds", "proxy-seeds:iso/proxy-x-seed.iso"); err != nil { + t.Fatalf("DeleteVolume error = %v", err) + } + if got := <-seen; !strings.HasSuffix(got.escapedPath, "/content/proxy-seeds:iso%2Fproxy-x-seed.iso") { + t.Errorf("escaped path = %q, want the volid as one segment with %%2F", got.escapedPath) + } +} + +func TestClient_http1Only(t *testing.T) { + t.Parallel() + protos := make(chan int, 2) + srv := httptest.NewUnstartedServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) { + protos <- r.ProtoMajor + serveJSON(w, readCapture(t, "version").Body) + })) + srv.EnableHTTP2 = true + srv.StartTLS() + t.Cleanup(srv.Close) + + resp, err := srv.Client().Get(srv.URL) + if err != nil { + t.Fatalf("sanity request error = %v", err) + } + _ = resp.Body.Close() + if p := <-protos; p != 2 { + t.Fatalf("sanity: server spoke HTTP/%d to an h2 client, want 2", p) + } + if _, err := fixtureFor(t, srv).api(t).Version(context.Background()); err != nil { + t.Fatalf("Version error = %v", err) + } + if p := <-protos; p != 1 { + t.Errorf("client spoke HTTP/%d, want HTTP/1.1 (F18)", p) + } +} + +func TestClient_requestTimeoutIsTransient(t *testing.T) { + t.Parallel() + fx := newClientFixture(t, func(_ http.ResponseWriter, r *http.Request) { + select { + case <-r.Context().Done(): + case <-time.After(5 * time.Second): + } + }) + a := fx.api(t) + a.timeout = 100 * time.Millisecond + start := time.Now() + _, err := a.Version(context.Background()) + if elapsed := time.Since(start); elapsed > 2*time.Second { + t.Errorf("Version took %v, want the 100ms request timeout to cut it", elapsed) + } + var ne net.Error + if !errors.As(err, &ne) || !ne.Timeout() { + t.Errorf("error = %v, want a timeout", err) + } + assertClass(t, err, provider.ErrTransient) +} + +// TestClient_cancelledContextReturnsPromptly also checks that the +// cancellation touches only that request: the next call on the same client +// succeeds. +func TestClient_cancelledContextReturnsPromptly(t *testing.T) { + t.Parallel() + fx := newClientFixture(t, func(w http.ResponseWriter, r *http.Request) { + if strings.HasSuffix(r.URL.Path, "/status") { + select { + case <-r.Context().Done(): + case <-time.After(5 * time.Second): + } + return + } + serveJSON(w, readCapture(t, "version").Body) + }) + a := fx.api(t) + ctx, cancel := context.WithCancel(context.Background()) + time.AfterFunc(50*time.Millisecond, cancel) + start := time.Now() + _, err := a.NodeStatus(ctx, "pv1-1") + if elapsed := time.Since(start); elapsed > time.Second { + t.Errorf("NodeStatus took %v after cancel, want prompt", elapsed) + } + if !errors.Is(err, context.Canceled) { + t.Errorf("error = %v, want context.Canceled", err) + } + assertClass(t, err, provider.ErrTransient) + if _, err := a.Version(context.Background()); err != nil { + t.Errorf("next call error = %v, want the client unaffected", err) + } +} + +func tracedFixture(t *testing.T) (*clientFixture, *tracetest.SpanRecorder, *sdktrace.TracerProvider, func() []string) { + t.Helper() + var ( + mu sync.Mutex + traceparent []string + ) + fx := newClientFixture(t, func(w http.ResponseWriter, r *http.Request) { + mu.Lock() + traceparent = append(traceparent, r.Header.Get("Traceparent")) + mu.Unlock() + serveJSON(w, readCapture(t, "version").Body) + }) + sr := tracetest.NewSpanRecorder() + tp := sdktrace.NewTracerProvider(sdktrace.WithSpanProcessor(sr)) + t.Cleanup(func() { _ = tp.Shutdown(context.Background()) }) + sent := func() []string { + mu.Lock() + defer mu.Unlock() + return append([]string(nil), traceparent...) + } + return fx, sr, tp, sent +} + +func TestClient_childSpanUnderParent(t *testing.T) { + t.Parallel() + fx, sr, tp, traceparent := tracedFixture(t) + a := fx.api(t, tracing.WithTracerProvider(tp)) + ctx, parent := tp.Tracer("test").Start(context.Background(), "provider.get") + if _, err := a.Version(ctx); err != nil { + t.Fatalf("Version error = %v", err) + } + parent.End() + + var child sdktrace.ReadOnlySpan + for _, s := range sr.Ended() { + if s.Name() != "provider.get" { + child = s + } + } + if child == nil { + t.Fatalf("no HTTP span recorded; spans: %d", len(sr.Ended())) + } + if child.Parent().SpanID() != parent.SpanContext().SpanID() || child.SpanContext().TraceID() != parent.SpanContext().TraceID() { + t.Errorf("HTTP span %q is not a child of provider.get", child.Name()) + } + secret := testToken[strings.Index(testToken, "=")+1:] + for _, kv := range child.Attributes() { + if strings.Contains(kv.Value.Emit(), secret) { + t.Errorf("span attribute %s carries the token", kv.Key) + } + } + if sent := traceparent(); len(sent) != 1 || !strings.Contains(sent[0], child.SpanContext().TraceID().String()) { + t.Errorf("traceparent sent = %q, want the trace id", sent) + } +} + +func TestClient_noSpanWithoutParent(t *testing.T) { + t.Parallel() + fx, sr, tp, traceparent := tracedFixture(t) + if _, err := fx.api(t, tracing.WithTracerProvider(tp)).Version(context.Background()); err != nil { + t.Fatalf("Version error = %v", err) + } + if n := len(sr.Ended()); n != 0 { + t.Errorf("got %d spans without a parent, want none", n) + } + if sent := traceparent(); len(sent) != 1 || sent[0] != "" { + t.Errorf("traceparent sent = %q, want none", sent) + } +} + +// TestClient_tokenNeverInErrorsOrLogs has the server echo the secret back +// in every place an error's text comes from; neither the error nor the +// decorator's V(2) lines may carry it. +func TestClient_tokenNeverInErrorsOrLogs(t *testing.T) { + t.Parallel() + const secret = "SENTINEL-SECRET-9d41" + token := "proxy-operator@pve!operator=" + secret + tests := []struct { + name string + resp func(auth string) capture + }{ + {name: "401 body echoes the header", resp: func(auth string) capture { + return capture{Status: 401, Reason: "Authentication failed!", Body: `{"data":null,"message":"bad credentials: ` + auth + `\n"}`} + }}, + {name: "401 reason echoes the header", resp: func(auth string) capture { + return capture{Status: 401, Reason: "bad credentials " + auth} + }}, + {name: "400 errors echo the header", resp: func(auth string) capture { + return capture{Status: 400, Reason: "Parameter verification failed.", Body: `{"data":null,"message":"Parameter verification failed.\n","errors":{"token":"` + auth + `"}}`} + }}, + } + for _, tc := range tests { + t.Run(tc.name, func(t *testing.T) { + t.Parallel() + fx := newClientFixture(t, func(w http.ResponseWriter, r *http.Request) { + serveCapture(t, w, tc.resp(r.Header.Get("Authorization"))) + }) + writeFile(t, fx.tokenFile, token) + logs := &logCapture{} + ctx := logs.context(2) + _, err := observe(fx.api(t), "proxmox-home").Version(ctx) + var ae *apiError + if !errors.As(err, &ae) { + t.Fatalf("error = %v, want an apiError", err) + } + if !strings.Contains(err.Error(), "") { + t.Errorf("error = %v, want the echoed secret redacted", err) + } + for _, s := range append([]string{err.Error(), fmt.Sprintf("%+v", err), fmt.Sprintf("%#v", ae)}, logs.all()...) { + if strings.Contains(s, secret) { + t.Errorf("secret leaked: %s", s) + } + } + if len(logs.all()) != 1 { + t.Errorf("got %d log lines, want the one call line", len(logs.all())) + } + }) + } +} + +func TestClient_caFileTrustsPrivateCA(t *testing.T) { + t.Parallel() + fx := newClientFixture(t, func(w http.ResponseWriter, _ *http.Request) { serveJSON(w, readCapture(t, "version").Body) }) + if _, err := fx.api(t).Version(context.Background()); err != nil { + t.Errorf("with the CA file: error = %v, want nil", err) + } + fx.cfg.CAFile = "" + _, err := fx.api(t).Version(context.Background()) + if !isTLSVerification(err) { + t.Errorf("with system roots: error = %v, want a verification failure", err) + } +} + +// TestClient_caFileLoadedLazily covers spec entry 14: a missing CA file is +// Transient and retried until it lands; one without a certificate is +// Permanent. +func TestClient_caFileLoadedLazily(t *testing.T) { + t.Parallel() + fx := newClientFixture(t, func(w http.ResponseWriter, _ *http.Request) { serveJSON(w, readCapture(t, "version").Body) }) + pemBytes, err := os.ReadFile(fx.caFile) + if err != nil { + t.Fatal(err) + } + fx.cfg.CAFile = filepath.Join(t.TempDir(), "ca.pem") + a := fx.api(t) + if _, err := a.Version(context.Background()); !errors.Is(err, provider.ErrTransient) { + t.Errorf("missing CA file: error = %v, want ErrTransient", err) + } + writeFile(t, fx.cfg.CAFile, "not a certificate") + if _, err := a.Version(context.Background()); !errors.Is(err, provider.ErrPermanent) { + t.Errorf("CA file without a certificate: error = %v, want ErrPermanent", err) + } + writeFile(t, fx.cfg.CAFile, string(pemBytes)) + if _, err := a.Version(context.Background()); err != nil { + t.Errorf("CA file landed: error = %v, want nil", err) + } +} + +func TestClient_x509FailureIsPermanent(t *testing.T) { + t.Parallel() + tests := []struct { + name string + mutate func(fx *clientFixture) + }{ + {name: "unknown authority", mutate: func(fx *clientFixture) { fx.cfg.CAFile = "" }}, + {name: "host not in the certificate", mutate: func(fx *clientFixture) { + fx.cfg.URL = strings.Replace(fx.srv.URL, "127.0.0.1", "localhost", 1) + }}, + } + for _, tc := range tests { + t.Run(tc.name, func(t *testing.T) { + t.Parallel() + var requests atomic.Int32 + fx := newClientFixture(t, func(http.ResponseWriter, *http.Request) { requests.Add(1) }) + tc.mutate(fx) + _, err := fx.api(t).Version(context.Background()) + if !isTLSVerification(err) { + t.Errorf("error = %v, want a verification failure", err) + } + assertClass(t, err, provider.ErrPermanent) + if n := requests.Load(); n != 0 { + t.Errorf("server saw %d requests, want none past the handshake", n) + } + }) + } +} + +func TestClient_insecureSkipsVerification(t *testing.T) { + t.Parallel() + fx := newClientFixture(t, func(w http.ResponseWriter, _ *http.Request) { serveJSON(w, readCapture(t, "version").Body) }) + fx.cfg.CAFile = "" + fx.cfg.InsecureSkipTLSVerify = true + if _, err := fx.api(t).Version(context.Background()); err != nil { + t.Errorf("error = %v, want nil with verification off", err) + } +} + +// gatedReader hands out first, then waits for release before handing out +// rest. A client that read the whole body before sending would never get +// past first. +type gatedReader struct { + first, rest []byte + release <-chan struct{} + waited bool +} + +func (g *gatedReader) Read(p []byte) (int, error) { + if len(g.first) > 0 { + n := copy(p, g.first) + g.first = g.first[n:] + return n, nil + } + if !g.waited { + select { + case <-g.release: + case <-time.After(5 * time.Second): + return 0, errors.New("the server saw nothing of the file before the reader was drained: the body is not streamed") + } + g.waited = true + } + if len(g.rest) == 0 { + return 0, io.EOF + } + n := copy(p, g.rest) + g.rest = g.rest[n:] + return n, nil +} + +func TestClient_uploadStreamsMultipart(t *testing.T) { + t.Parallel() + first := bytes.Repeat([]byte("a"), 256<<10) + rest := bytes.Repeat([]byte("b"), 256<<10) + sawFirst := make(chan struct{}) + type upload struct { + contentLength int64 + chunked bool + content, name string + fileName, bytes string + } + got := make(chan upload, 1) + fx := newClientFixture(t, func(w http.ResponseWriter, r *http.Request) { + u := upload{contentLength: r.ContentLength, chunked: len(r.TransferEncoding) > 0} + mr, err := r.MultipartReader() + if err != nil { + t.Errorf("multipart reader: %v", err) + return + } + for { + part, err := mr.NextPart() + if err == io.EOF { + break + } + if err != nil { + t.Errorf("next part: %v", err) + return + } + switch part.FormName() { + case "content": + b, _ := io.ReadAll(part) + u.content = string(b) + case "filename": + u.name, u.fileName = part.FormName(), part.FileName() + var buf bytes.Buffer + if _, err := io.CopyN(&buf, part, 1); err != nil { + t.Errorf("reading the file part: %v", err) + return + } + close(sawFirst) + _, _ = io.Copy(&buf, part) + u.bytes = buf.String() + } + } + got <- u + serveJSON(w, readCapture(t, "upload_response").Body) + }) + body := &gatedReader{first: first, rest: rest, release: sawFirst} + size := int64(len(first) + len(rest)) + upid, err := fx.api(t).UploadISO(context.Background(), "pv1-1", "proxy-seeds", "proxy-x-seed.iso", body, size) + if err != nil { + t.Fatalf("UploadISO error = %v", err) + } + if !strings.Contains(upid, ":imgcopy:") { + t.Errorf("upid = %q, want the imgcopy task", upid) + } + u := <-got + if u.chunked || u.contentLength <= size { + t.Errorf("Content-Length %d (chunked %v), want an exact length above the file's %d bytes", u.contentLength, u.chunked, size) + } + if u.content != "iso" || u.name != "filename" || u.fileName != "proxy-x-seed.iso" { + t.Errorf("fields content=%q part=%q file=%q, want iso, filename, proxy-x-seed.iso (F12)", u.content, u.name, u.fileName) + } + if u.bytes != string(first)+string(rest) { + t.Errorf("server got %d file bytes, want the %d sent", len(u.bytes), size) + } +} + +func TestDecodeConfig_scalarsAsStrings(t *testing.T) { + t.Parallel() + in := map[string]json.RawMessage{"cores": json.RawMessage(`2`), "memory": json.RawMessage(`"2048"`), "template": json.RawMessage(`1`), "balloon": json.RawMessage(`null`), "description": json.RawMessage(`"a\nb"`)} + got, err := decodeConfig(in) + if err != nil { + t.Fatalf("decodeConfig error = %v", err) + } + want := vmConfig{"cores": "2", "memory": "2048", "template": "1", "description": "a\nb"} + if !reflect.DeepEqual(got, want) { + t.Errorf("decodeConfig = %v, want %v", got, want) + } +} + +func TestReadToken_trimsTrailingNewline(t *testing.T) { + t.Parallel() + path := filepath.Join(t.TempDir(), "token") + writeFile(t, path, " "+testToken+"\r\n") + if got, err := readToken(path); err != nil || got != testToken { + t.Errorf("readToken = %q, %v; want the token without whitespace", got, err) + } +} diff --git a/internal/provider/proxmox/fake_test.go b/internal/provider/proxmox/fake_test.go new file mode 100644 index 0000000..62242e6 --- /dev/null +++ b/internal/provider/proxmox/fake_test.go @@ -0,0 +1,893 @@ +package proxmox + +import ( + "context" + "errors" + "fmt" + "io" + "maps" + "slices" + "strconv" + "strings" + "sync" + "testing" +) + +// fakeAPI is the in-memory cluster of spec entry 17: VMs, tasks and ISOs, +// with lock simulation (a clone leaves a nameless locked entry until its +// task completes), task polls, per-method errors and blocking, and a call +// log. Error texts Proxmox was seen to send are reproduced exactly and +// checked against the captures by TestFakeAPI_errorsMatchCaptures; texts +// recon never produced start with "fake:". +type fakeAPI struct { + mu sync.Mutex + vms map[int]*fakeVM + tasks map[string]*fakeTask + isos map[string]map[string][]byte // storage, then file name + nodes map[string]nodeStatus + storages map[string]storageStatus + version versionInfo + taskSeq int + + // hooks returns the error a method fails with before touching state. + hooks map[string]func() error + // block makes a method wait on its channel (or ctx) before running. + block map[string]chan struct{} + calls map[string]int + log []string + + // taskPolls is how many TaskStatus polls answer "running" before a + // task stops. + taskPolls int + // cloneFailsWith and startFailsWith, when set, are the exitstatus of + // every clone or start task. A failed clone removes its VM. + cloneFailsWith, startFailsWith string + // agentNotRunning makes the agent of every running VM with agent=1 + // answer "QEMU guest agent is not running". + agentNotRunning bool + // cloneLeavesLock keeps lock=clone after a clone task ends, as an + // interrupted clone would (F15). + cloneLeavesLock bool +} + +type fakeVM struct { + node, name, status, qmpstatus, lock string + template bool + config map[string]string + ifaces []agentInterface +} + +type fakeTask struct { + node string + polls int + exit string + onDone func() + kind string + id string + done bool +} + +const ( + fakeNode = "pv1-1" + fakeISOStorage = "proxy-seeds" +) + +var _ api = (*fakeAPI)(nil) + +func newFakeAPI() *fakeAPI { + var node nodeStatus + node.PVEVersion = "pve-manager/9.2.20/49318c671b82f31e" + node.CurrentKernel.Release = "7.0.14-17-pve" + return &fakeAPI{ + vms: make(map[int]*fakeVM), + tasks: make(map[string]*fakeTask), + isos: make(map[string]map[string][]byte), + nodes: map[string]nodeStatus{fakeNode: node}, + storages: map[string]storageStatus{fakeISOStorage: {Type: "dir", Content: "iso", Active: 1, Enabled: 1, Total: 100 << 30, Used: 10 << 30, Avail: 90 << 30}}, + version: versionInfo{Version: "9.2.20", Release: "9.2", RepoID: "49318c671b82f31e"}, + hooks: make(map[string]func() error), + block: make(map[string]chan struct{}), + calls: make(map[string]int), + } +} + +// addVM places a stopped or running VM cloned from a template like 9001 on +// fakeNode, without a description; tests add the mark they need. +func (f *fakeAPI) addVM(vmid int, name, status string) *fakeVM { + f.mu.Lock() + defer f.mu.Unlock() + vm := &fakeVM{ + node: fakeNode, + name: name, + status: status, + qmpstatus: status, + config: map[string]string{ + "agent": "1", + "boot": "order=scsi0", + "cores": "2", + "memory": "2048", + "net0": "virtio=BC:24:11:48:11:46,bridge=vmbr0", + "scsi0": fmt.Sprintf("local-lvm:vm-%d-disk-1,size=3G", vmid), + }, + } + f.vms[vmid] = vm + return vm +} + +// addTemplate places a template shaped like 9001 (F19) on fakeNode. +func (f *fakeAPI) addTemplate(vmid int, name string) *fakeVM { + vm := f.addVM(vmid, name, "stopped") + f.mu.Lock() + defer f.mu.Unlock() + vm.template = true + vm.config["scsi0"] = fmt.Sprintf("local-lvm:base-%d-disk-1,size=3G", vmid) + return vm +} + +func (f *fakeAPI) failWith(method string, err error) { + f.mu.Lock() + defer f.mu.Unlock() + f.hooks[method] = func() error { return err } +} + +func (f *fakeAPI) blockOn(method string) chan struct{} { + f.mu.Lock() + defer f.mu.Unlock() + ch := make(chan struct{}) + f.block[method] = ch + return ch +} + +func (f *fakeAPI) callCount(method string) int { + f.mu.Lock() + defer f.mu.Unlock() + return f.calls[method] +} + +func (f *fakeAPI) callLog() []string { + f.mu.Lock() + defer f.mu.Unlock() + return slices.Clone(f.log) +} + +// iso returns an uploaded file's content. +func (f *fakeAPI) iso(storage, filename string) ([]byte, bool) { + f.mu.Lock() + defer f.mu.Unlock() + b, ok := f.isos[storage][filename] + return b, ok +} + +// vmSnapshot returns a copy of the VM at vmid, for assertions. +func (f *fakeAPI) vmSnapshot(vmid int) (fakeVM, bool) { + f.mu.Lock() + defer f.mu.Unlock() + vm, ok := f.vms[vmid] + if !ok { + return fakeVM{}, false + } + c := *vm + c.config = maps.Clone(vm.config) + c.ifaces = slices.Clone(vm.ifaces) + return c, true +} + +func fakeErr(status int, msg string) *apiError { + return &apiError{Status: status, Message: msg} +} + +// The 500s recon captured (F2, F4, F5, F7). +func errVMMissing(node string, vmid int) *apiError { + return fakeErr(500, fmt.Sprintf("Configuration file 'nodes/%s/qemu-server/%d.conf' does not exist", node, vmid)) +} + +func errLocked(lock string) *apiError { return fakeErr(500, "VM is locked ("+lock+")") } + +func errNewIDInUse(vmid int) *apiError { + return fakeErr(500, fmt.Sprintf("unable to create VM %d: config file already exists", vmid)) +} + +func errVMNotRunning(vmid int) *apiError { + return fakeErr(500, fmt.Sprintf("VM %d is not running", vmid)) +} + +// enter records the call, honours a block and runs the hook. It is called +// without f.mu held; state access happens afterwards under the lock. +func (f *fakeAPI) enter(ctx context.Context, method string) error { + f.mu.Lock() + f.calls[method]++ + f.log = append(f.log, method) + block := f.block[method] + hook := f.hooks[method] + f.mu.Unlock() + if block != nil { + select { + case <-block: + case <-ctx.Done(): + return ctx.Err() + } + } + if hook != nil { + return hook() + } + return nil +} + +// vm looks a VM up the way Proxmox does, by node and VMID: a VM on another +// node has no config file on this one. +func (f *fakeAPI) vm(node string, vmid int) (*fakeVM, error) { + vm, ok := f.vms[vmid] + if !ok || vm.node != node { + return nil, errVMMissing(node, vmid) + } + return vm, nil +} + +// newTask registers a task that stops after f.taskPolls running answers, +// then runs onDone under f.mu. The UPID has the shape parseUPID reads. +func (f *fakeAPI) newTask(node, kind, id, exit string, onDone func()) string { + f.taskSeq++ + upid := fmt.Sprintf("UPID:%s:%08X:%08X:%08X:%s:%s:proxy-operator@pve!operator:", node, 0x100000+f.taskSeq, 0x2800000+f.taskSeq, 0x6AB37800+f.taskSeq, kind, id) + f.tasks[upid] = &fakeTask{node: node, exit: exit, onDone: onDone, kind: kind, id: id} + return upid +} + +func (f *fakeAPI) Version(ctx context.Context) (versionInfo, error) { + if err := f.enter(ctx, "Version"); err != nil { + return versionInfo{}, err + } + f.mu.Lock() + defer f.mu.Unlock() + return f.version, nil +} + +func (f *fakeAPI) NodeStatus(ctx context.Context, node string) (nodeStatus, error) { + if err := f.enter(ctx, "NodeStatus"); err != nil { + return nodeStatus{}, err + } + f.mu.Lock() + defer f.mu.Unlock() + s, ok := f.nodes[node] + if !ok { + return nodeStatus{}, fakeErr(500, "fake: no such node '"+node+"'") + } + return s, nil +} + +func (f *fakeAPI) StorageStatus(ctx context.Context, node, storage string) (storageStatus, error) { + if err := f.enter(ctx, "StorageStatus"); err != nil { + return storageStatus{}, err + } + f.mu.Lock() + defer f.mu.Unlock() + s, ok := f.storages[storage] + if !ok { + return storageStatus{}, fakeErr(500, "fake: storage '"+storage+"' does not exist") + } + return s, nil +} + +// ClusterResources lists VMs by VMID. A VM being cloned is listed as F8 +// saw it: no name, no tags, lock clone and status unknown. +func (f *fakeAPI) ClusterResources(ctx context.Context) ([]vmResource, error) { + if err := f.enter(ctx, "ClusterResources"); err != nil { + return nil, err + } + f.mu.Lock() + defer f.mu.Unlock() + out := make([]vmResource, 0, len(f.vms)) + for _, vmid := range slices.Sorted(maps.Keys(f.vms)) { + vm := f.vms[vmid] + r := vmResource{VMID: vmid, Name: vm.name, Node: vm.node, Type: "qemu", Status: vm.status, Lock: vm.lock, Tags: vm.config["tags"]} + if vm.template { + r.Template = 1 + } + if vm.lock == "clone" { + r.Name, r.Tags, r.Status = "", "", "unknown" + } + out = append(out, r) + } + return out, nil +} + +func (f *fakeAPI) VMConfig(ctx context.Context, node string, vmid int) (vmConfig, error) { + if err := f.enter(ctx, "VMConfig"); err != nil { + return nil, err + } + f.mu.Lock() + defer f.mu.Unlock() + vm, err := f.vm(node, vmid) + if err != nil { + return nil, err + } + cfg := vmConfig(maps.Clone(vm.config)) + if vm.name != "" { + cfg["name"] = vm.name + } + if vm.lock != "" { + cfg["lock"] = vm.lock + } + if vm.template { + cfg["template"] = "1" + } + return cfg, nil +} + +// SetVMConfig is refused while the VM is locked (F7). "delete" removes a +// comma-separated list of keys; "name" renames. +func (f *fakeAPI) SetVMConfig(ctx context.Context, node string, vmid int, params map[string]string) error { + if err := f.enter(ctx, "SetVMConfig"); err != nil { + return err + } + f.mu.Lock() + defer f.mu.Unlock() + vm, err := f.vm(node, vmid) + if err != nil { + return err + } + if vm.lock != "" { + return errLocked(vm.lock) + } + for k, v := range params { + switch k { + case "delete": + for key := range strings.SplitSeq(v, ",") { + delete(vm.config, key) + } + case "name": + vm.name = v + default: + vm.config[k] = v + } + } + return nil +} + +// Clone creates the VM at once, locked and nameless in the listing, and a +// task that clears the lock, or removes the VM when cloneFailsWith is set. +// A taken newid fails synchronously (F4). +func (f *fakeAPI) Clone(ctx context.Context, node string, vmid int, p cloneParams) (string, error) { + if err := f.enter(ctx, "Clone"); err != nil { + return "", err + } + f.mu.Lock() + defer f.mu.Unlock() + src, err := f.vm(node, vmid) + if err != nil { + return "", err + } + if _, taken := f.vms[p.NewID]; taken { + return "", errNewIDInUse(p.NewID) + } + target := node + if p.Target != "" { + target = p.Target + } + cfg := maps.Clone(src.config) + for k, v := range cfg { + cfg[k] = strings.ReplaceAll(v, fmt.Sprintf("base-%d-", vmid), fmt.Sprintf("vm-%d-", p.NewID)) + } + if p.Description != "" { + cfg["description"] = p.Description + } + vm := &fakeVM{node: target, name: p.Name, status: "stopped", qmpstatus: "stopped", lock: "clone", config: cfg} + f.vms[p.NewID] = vm + exit := "OK" + if f.cloneFailsWith != "" { + exit = f.cloneFailsWith + } + leaveLock := f.cloneLeavesLock + return f.newTask(node, "qmclone", strconv.Itoa(vmid), exit, func() { + switch { + case exit != "OK": + if f.vms[p.NewID] == vm { + delete(f.vms, p.NewID) + } + case !leaveLock: + vm.lock = "" + } + }), nil +} + +// Resize runs as a task (F20): the same size is a no-op, a smaller one +// fails inside the task with the captured exitstatus. +func (f *fakeAPI) Resize(ctx context.Context, node string, vmid int, disk, size string) (string, error) { + if err := f.enter(ctx, "Resize"); err != nil { + return "", err + } + f.mu.Lock() + defer f.mu.Unlock() + vm, err := f.vm(node, vmid) + if err != nil { + return "", err + } + if vm.lock != "" { + return "", errLocked(vm.lock) + } + cur, ok := vm.config[disk] + if !ok { + return "", fakeErr(500, "fake: disk '"+disk+"' does not exist") + } + curBytes, ok := fakeParseSize(diskSizeOption(cur)) + if !ok { + return "", fakeErr(500, "fake: disk '"+disk+"' has no size") + } + want, ok := fakeParseSize(strings.TrimPrefix(size, "+")) + if !ok { + return "", fakeErr(400, "fake: bad size '"+size+"'") + } + if strings.HasPrefix(size, "+") { + want += curBytes + } + exit := "OK" + if want < curBytes { + exit = "shrinking disks is not supported" + } + return f.newTask(node, "resize", strconv.Itoa(vmid), exit, func() { + if exit == "OK" { + vm.config[disk] = strings.Replace(cur, "size="+diskSizeOption(cur), "size="+fakeFormatSize(want), 1) + } + }), nil +} + +// Start is accepted even while a clone holds the lock (F3). +func (f *fakeAPI) Start(ctx context.Context, node string, vmid int) (string, error) { + if err := f.enter(ctx, "Start"); err != nil { + return "", err + } + f.mu.Lock() + defer f.mu.Unlock() + vm, err := f.vm(node, vmid) + if err != nil { + return "", err + } + exit := "OK" + if f.startFailsWith != "" { + exit = f.startFailsWith + } + return f.newTask(node, "qmstart", strconv.Itoa(vmid), exit, func() { + if exit == "OK" { + vm.status, vm.qmpstatus = "running", "running" + } + }), nil +} + +func (f *fakeAPI) Stop(ctx context.Context, node string, vmid int) (string, error) { + if err := f.enter(ctx, "Stop"); err != nil { + return "", err + } + f.mu.Lock() + defer f.mu.Unlock() + vm, err := f.vm(node, vmid) + if err != nil { + return "", err + } + return f.newTask(node, "qmstop", strconv.Itoa(vmid), "OK", func() { + vm.status, vm.qmpstatus = "stopped", "stopped" + }), nil +} + +func (f *fakeAPI) VMStatus(ctx context.Context, node string, vmid int) (vmStatus, error) { + if err := f.enter(ctx, "VMStatus"); err != nil { + return vmStatus{}, err + } + f.mu.Lock() + defer f.mu.Unlock() + vm, err := f.vm(node, vmid) + if err != nil { + return vmStatus{}, err + } + return vmStatus{Status: vm.status, QMPStatus: vm.qmpstatus, Name: vm.name, Lock: vm.lock}, nil +} + +// Destroy refuses a locked or running VM and removes it when its task +// ends. +func (f *fakeAPI) Destroy(ctx context.Context, node string, vmid int) (string, error) { + if err := f.enter(ctx, "Destroy"); err != nil { + return "", err + } + f.mu.Lock() + defer f.mu.Unlock() + vm, err := f.vm(node, vmid) + if err != nil { + return "", err + } + if vm.lock != "" { + return "", errLocked(vm.lock) + } + if vm.status == "running" { + return "", fakeErr(500, fmt.Sprintf("fake: VM %d is running - destroy failed", vmid)) + } + return f.newTask(node, "qmdestroy", strconv.Itoa(vmid), "OK", func() { + if f.vms[vmid] == vm { + delete(f.vms, vmid) + } + }), nil +} + +// TaskStatus answers running for f.taskPolls polls, then stopped with the +// task's exitstatus, running its effect once. +func (f *fakeAPI) TaskStatus(ctx context.Context, node, upid string) (taskStatus, error) { + if err := f.enter(ctx, "TaskStatus"); err != nil { + return taskStatus{}, err + } + f.mu.Lock() + defer f.mu.Unlock() + t, ok := f.tasks[upid] + if !ok || t.node != node { + return taskStatus{}, fakeErr(500, "fake: no such task '"+upid+"'") + } + t.polls++ + s := taskStatus{UPID: upid, Type: t.kind, ID: t.id, Node: t.node, Status: "running"} + if !t.done && t.polls > f.taskPolls { + t.done = true + if t.onDone != nil { + t.onDone() + } + } + if t.done { + s.Status, s.ExitStatus = "stopped", t.exit + } + return s, nil +} + +// AgentNetworkInterfaces answers with the three messages of F5 in the +// order Proxmox checks them. +func (f *fakeAPI) AgentNetworkInterfaces(ctx context.Context, node string, vmid int) ([]agentInterface, error) { + if err := f.enter(ctx, "AgentNetworkInterfaces"); err != nil { + return nil, err + } + f.mu.Lock() + defer f.mu.Unlock() + vm, err := f.vm(node, vmid) + if err != nil { + return nil, err + } + if agent := vm.config["agent"]; agent != "1" && !strings.HasPrefix(agent, "1,") && !strings.Contains(agent, "enabled=1") { + return nil, fakeErr(500, msgAgentNotConfigured) + } + if vm.status != "running" { + return nil, errVMNotRunning(vmid) + } + if f.agentNotRunning { + return nil, fakeErr(500, msgAgentNotRunning) + } + return slices.Clone(vm.ifaces), nil +} + +// UploadISO records the bytes and overwrites an existing file silently +// (F12). A body that disagrees with size fails, as the real Content-Length +// would. +func (f *fakeAPI) UploadISO(ctx context.Context, node, storage, filename string, r io.Reader, size int64) (string, error) { + if err := f.enter(ctx, "UploadISO"); err != nil { + return "", err + } + var content []byte + if r != nil { + var err error + if content, err = io.ReadAll(r); err != nil { + return "", err + } + } + f.mu.Lock() + defer f.mu.Unlock() + if _, ok := f.storages[storage]; !ok { + return "", fakeErr(500, "fake: storage '"+storage+"' does not exist") + } + if int64(len(content)) != size { + return "", fakeErr(400, fmt.Sprintf("fake: body has %d bytes, size says %d", len(content), size)) + } + if f.isos[storage] == nil { + f.isos[storage] = make(map[string][]byte) + } + f.isos[storage][filename] = content + return f.newTask(node, "imgcopy", "", "OK", nil), nil +} + +// DeleteVolume is a task that ends OK whether or not the volume exists +// (F13). +func (f *fakeAPI) DeleteVolume(ctx context.Context, node, storage, volid string) (string, error) { + if err := f.enter(ctx, "DeleteVolume"); err != nil { + return "", err + } + f.mu.Lock() + defer f.mu.Unlock() + if _, ok := f.storages[storage]; !ok { + return "", fakeErr(500, "fake: storage '"+storage+"' does not exist") + } + filename := strings.TrimPrefix(volid, storage+":iso/") + return f.newTask(node, "imgdel", storage, "OK", func() { + delete(f.isos[storage], filename) + }), nil +} + +// diskSizeOption returns the size= option of a disk value. +func diskSizeOption(disk string) string { + for opt := range strings.SplitSeq(disk, ",") { + if v, ok := strings.CutPrefix(opt, "size="); ok { + return v + } + } + return "" +} + +// fakeParseSize reads Proxmox's [KMGT] sizes into bytes. +func fakeParseSize(s string) (int64, bool) { + if s == "" { + return 0, false + } + shift := map[byte]uint{'K': 10, 'M': 20, 'G': 30, 'T': 40} + unit, ok := shift[s[len(s)-1]] + if ok { + s = s[:len(s)-1] + } + n, err := strconv.ParseInt(s, 10, 64) + if err != nil || n < 0 { + return 0, false + } + return n << unit, true +} + +func fakeFormatSize(b int64) string { + if b%(1<<30) == 0 { + return strconv.FormatInt(b>>30, 10) + "G" + } + return strconv.FormatInt(b>>20, 10) + "M" +} + +// TestFakeAPI_errorsMatchCaptures keeps the fake's error texts equal to +// what Proxmox sent, so the provider's matchers are exercised against real +// wording in every test that goes through the fake. +func TestFakeAPI_errorsMatchCaptures(t *testing.T) { + t.Parallel() + tests := []struct { + capture string + fake *apiError + matcher func(error) bool + }{ + {capture: "err_vm_missing", fake: errVMMissing("pv1-1", 999999), matcher: isNotExist}, + {capture: "err_destroy_missing", fake: errVMMissing("pv1-1", 199999), matcher: isNotExist}, + {capture: "err_config_while_clone_locked", fake: errLocked("clone"), matcher: isLocked}, + {capture: "err_newid_in_use", fake: errNewIDInUse(199999), matcher: isAlreadyExists}, + {capture: "err_agent_vm_stopped_agent_enabled", fake: errVMNotRunning(199999), matcher: isVMNotRunning}, + {capture: "err_agent_not_running", fake: fakeErr(500, msgAgentNotRunning), matcher: isAgentNotRunning}, + {capture: "err_agent_vm_stopped", fake: fakeErr(500, msgAgentNotConfigured), matcher: isAgentNotConfigured}, + } + for _, tc := range tests { + t.Run(tc.capture, func(t *testing.T) { + t.Parallel() + want := loadAPIError(t, tc.capture) + if tc.fake.Status != want.Status || tc.fake.Message != want.Message { + t.Errorf("fake error = %d %q, capture = %d %q", tc.fake.Status, tc.fake.Message, want.Status, want.Message) + } + if !tc.matcher(tc.fake) { + t.Errorf("matcher does not accept the fake's %q", tc.fake.Message) + } + }) + } +} + +// pollTask polls a fake task until it stops and returns its exitstatus. +func pollTask(t *testing.T, f *fakeAPI, upid string) string { + t.Helper() + for range 10 { + s, err := f.TaskStatus(context.Background(), fakeNode, upid) + if err != nil { + t.Fatalf("TaskStatus(%s) error = %v", upid, err) + } + if s.Status == "stopped" { + return s.ExitStatus + } + } + t.Fatalf("task %s still running after 10 polls", upid) + return "" +} + +func resourceByID(t *testing.T, f *fakeAPI, vmid int) (vmResource, bool) { + t.Helper() + rs, err := f.ClusterResources(context.Background()) + if err != nil { + t.Fatalf("ClusterResources error = %v", err) + } + for _, r := range rs { + if r.VMID == vmid { + return r, true + } + } + return vmResource{}, false +} + +func TestFakeAPI_cloneLocksUntilTaskEnds(t *testing.T) { + t.Parallel() + ctx := context.Background() + f := newFakeAPI() + f.addTemplate(9001, "debian-13-template") + f.taskPolls = 1 + upid, err := f.Clone(ctx, fakeNode, 9001, cloneParams{NewID: 150042, Name: "proxy-x", Description: "d"}) + if err != nil { + t.Fatalf("Clone error = %v", err) + } + if r, _ := resourceByID(t, f, 150042); r.Name != "" || r.Lock != "clone" || r.Status != "unknown" { + t.Errorf("mid-clone entry = %+v, want nameless, lock clone, status unknown (F8)", r) + } + if err := f.SetVMConfig(ctx, fakeNode, 150042, map[string]string{"cores": "1"}); !isLocked(err) { + t.Errorf("SetVMConfig mid-clone error = %v, want the lock answer (F7)", err) + } + if _, err := f.Clone(ctx, fakeNode, 9001, cloneParams{NewID: 150042, Name: "proxy-y"}); !isAlreadyExists(err) { + t.Errorf("second Clone onto 150042 error = %v, want config file already exists (F4)", err) + } + if s, _ := f.TaskStatus(ctx, fakeNode, upid); s.Status != "running" { + t.Errorf("first poll = %+v, want running with taskPolls 1", s) + } + if exit := pollTask(t, f, upid); exit != "OK" { + t.Errorf("clone exitstatus = %q, want OK", exit) + } + r, _ := resourceByID(t, f, 150042) + if r.Name != "proxy-x" || r.Lock != "" || r.Status != "stopped" || r.Template != 0 { + t.Errorf("cloned entry = %+v, want named, unlocked, stopped", r) + } + cfg, err := f.VMConfig(ctx, fakeNode, 150042) + if err != nil || cfg["description"] != "d" || cfg["scsi0"] != "local-lvm:vm-150042-disk-1,size=3G" || cfg["template"] != "" { + t.Errorf("clone config = %v, %v", cfg, err) + } +} + +func TestFakeAPI_cloneFailureAndLeftLock(t *testing.T) { + t.Parallel() + ctx := context.Background() + t.Run("failed clone removes the VM", func(t *testing.T) { + t.Parallel() + f := newFakeAPI() + f.addTemplate(9001, "debian-13-template") + f.cloneFailsWith = "clone failed: out of space" + upid, _ := f.Clone(ctx, fakeNode, 9001, cloneParams{NewID: 150042, Name: "proxy-x"}) + if exit := pollTask(t, f, upid); exit != "clone failed: out of space" { + t.Errorf("exitstatus = %q", exit) + } + if _, ok := resourceByID(t, f, 150042); ok { + t.Error("failed clone left its VM") + } + }) + t.Run("interrupted clone keeps the lock", func(t *testing.T) { + t.Parallel() + f := newFakeAPI() + f.addTemplate(9001, "debian-13-template") + f.cloneLeavesLock = true + upid, _ := f.Clone(ctx, fakeNode, 9001, cloneParams{NewID: 150042, Name: "proxy-x"}) + pollTask(t, f, upid) + if r, _ := resourceByID(t, f, 150042); r.Lock != "clone" { + t.Errorf("entry = %+v, want lock clone kept", r) + } + }) +} + +func TestFakeAPI_powerAgentAndDestroy(t *testing.T) { + t.Parallel() + ctx := context.Background() + f := newFakeAPI() + vm := f.addVM(150042, "proxy-x", "stopped") + vm.ifaces = []agentInterface{{Name: "ens18", IPAddresses: []agentIP{{Address: "192.168.4.147", Type: "ipv4", Prefix: 24}}}} + + if _, err := f.AgentNetworkInterfaces(ctx, fakeNode, 150042); !isVMNotRunning(err) { + t.Errorf("agent on a stopped VM error = %v, want VM is not running", err) + } + upid, _ := f.Start(ctx, fakeNode, 150042) + pollTask(t, f, upid) + if s, _ := f.VMStatus(ctx, fakeNode, 150042); s.Status != "running" || s.QMPStatus != "running" { + t.Errorf("status after start = %+v", s) + } + f.agentNotRunning = true + if _, err := f.AgentNetworkInterfaces(ctx, fakeNode, 150042); !isAgentNotRunning(err) { + t.Errorf("agent error = %v, want agent not running", err) + } + f.agentNotRunning = false + if ifaces, err := f.AgentNetworkInterfaces(ctx, fakeNode, 150042); err != nil || firstIPv4(ifaces) != "192.168.4.147" { + t.Errorf("agent = %v, %v", ifaces, err) + } + if _, err := f.Destroy(ctx, fakeNode, 150042); err == nil { + t.Error("Destroy of a running VM succeeded") + } + upid, _ = f.Stop(ctx, fakeNode, 150042) + pollTask(t, f, upid) + upid, err := f.Destroy(ctx, fakeNode, 150042) + if err != nil { + t.Fatalf("Destroy error = %v", err) + } + pollTask(t, f, upid) + if _, err := f.VMConfig(ctx, fakeNode, 150042); !isNotExist(err) { + t.Errorf("VMConfig after destroy error = %v, want does not exist", err) + } + if _, err := f.VMStatus(ctx, "pv1-2", 150042); !isNotExist(err) { + t.Errorf("lookup on another node error = %v, want does not exist", err) + } +} + +func TestFakeAPI_resizeIsATask(t *testing.T) { + t.Parallel() + tests := []struct { + name string + size string + wantExit string + wantDisk string + }{ + {name: "grow absolute", size: "8G", wantExit: "OK", wantDisk: "local-lvm:vm-150042-disk-1,size=8G"}, + {name: "grow relative", size: "+1G", wantExit: "OK", wantDisk: "local-lvm:vm-150042-disk-1,size=4G"}, + {name: "same size", size: "3G", wantExit: "OK", wantDisk: "local-lvm:vm-150042-disk-1,size=3G"}, + {name: "shrink fails in the task", size: "2G", wantExit: loadTaskExit(t, "task_status_stopped_error"), wantDisk: "local-lvm:vm-150042-disk-1,size=3G"}, + } + for _, tc := range tests { + t.Run(tc.name, func(t *testing.T) { + t.Parallel() + f := newFakeAPI() + f.addVM(150042, "proxy-x", "stopped") + upid, err := f.Resize(context.Background(), fakeNode, 150042, "scsi0", tc.size) + if err != nil { + t.Fatalf("Resize error = %v", err) + } + if exit := pollTask(t, f, upid); exit != tc.wantExit { + t.Errorf("exitstatus = %q, want %q", exit, tc.wantExit) + } + if vm, _ := f.vmSnapshot(150042); vm.config["scsi0"] != tc.wantDisk { + t.Errorf("scsi0 = %q, want %q", vm.config["scsi0"], tc.wantDisk) + } + }) + } +} + +func loadTaskExit(t *testing.T, name string) string { + t.Helper() + return decodeData[taskStatus](t, name).ExitStatus +} + +func TestFakeAPI_uploadOverwritesAndDeleteIsIdempotent(t *testing.T) { + t.Parallel() + ctx := context.Background() + f := newFakeAPI() + for _, body := range []string{"first", "second"} { + upid, err := f.UploadISO(ctx, fakeNode, fakeISOStorage, "proxy-x-seed.iso", strings.NewReader(body), int64(len(body))) + if err != nil { + t.Fatalf("UploadISO error = %v", err) + } + pollTask(t, f, upid) + } + if b, _ := f.iso(fakeISOStorage, "proxy-x-seed.iso"); string(b) != "second" { + t.Errorf("iso = %q, want the second upload (F12 overwrite)", b) + } + if _, err := f.UploadISO(ctx, fakeNode, fakeISOStorage, "x.iso", strings.NewReader("abc"), 4); err == nil { + t.Error("UploadISO with a wrong size succeeded") + } + for range 2 { + upid, err := f.DeleteVolume(ctx, fakeNode, fakeISOStorage, seedVolID(fakeISOStorage, "proxy-x")) + if err != nil { + t.Fatalf("DeleteVolume error = %v", err) + } + if exit := pollTask(t, f, upid); exit != "OK" { + t.Errorf("delete exitstatus = %q, want OK even when absent (F13)", exit) + } + } + if _, ok := f.iso(fakeISOStorage, "proxy-x-seed.iso"); ok { + t.Error("iso still present after delete") + } +} + +func TestFakeAPI_hooksBlocksAndLog(t *testing.T) { + t.Parallel() + f := newFakeAPI() + boom := errors.New("boom") + f.failWith("Version", boom) + if _, err := f.Version(context.Background()); !errors.Is(err, boom) { + t.Errorf("Version error = %v, want the hook's", err) + } + f.blockOn("ClusterResources") + ctx, cancel := context.WithCancel(context.Background()) + cancel() + if _, err := f.ClusterResources(ctx); !errors.Is(err, context.Canceled) { + t.Errorf("blocked call with ended ctx error = %v, want context.Canceled", err) + } + if got := f.callLog(); !slices.Equal(got, []string{"Version", "ClusterResources"}) { + t.Errorf("call log = %v", got) + } + if f.callCount("Version") != 1 { + t.Errorf("Version count = %d, want 1", f.callCount("Version")) + } +} diff --git a/internal/provider/proxmox/observe.go b/internal/provider/proxmox/observe.go new file mode 100644 index 0000000..7ef38e2 --- /dev/null +++ b/internal/provider/proxmox/observe.go @@ -0,0 +1,259 @@ +package proxmox + +import ( + "context" + "errors" + "io" + "strconv" + "time" + + logf "sigs.k8s.io/controller-runtime/pkg/log" +) + +// observed is the logging decorator of spec entry 20: one V(1) +// "proxmox api call" line per seam call, with request and response detail +// at V(2). It starts no span: the transport already traces each request +// (spec entry 17), so nothing is counted twice. +type observed struct { + inner api + provider string +} + +var _ api = (*observed)(nil) + +func observe(inner api, providerName string) api { + return &observed{inner: inner, provider: providerName} +} + +// target is what a call names at V(1); zero fields are left out. +type target struct { + node string + vmid int + storage string + upid string + extra []any +} + +// detail returns V(2)-only key/value pairs. It runs only when V(2) is on. +type detail func() []any + +// call wraps one seam call. fn returns the V(2) detail and may fill in +// t.upid from the call's answer. +func (o *observed) call(ctx context.Context, name string, t *target, fn func() (detail, error)) error { + log := logf.FromContext(ctx).WithName("proxmox").WithValues("provider", o.provider) + start := time.Now() + more, err := fn() + kv := make([]any, 0, 16) + kv = append(kv, "call", name) + if t.node != "" { + kv = append(kv, "node", t.node) + } + if t.vmid != 0 { + kv = append(kv, "vmid", t.vmid) + } + if t.storage != "" { + kv = append(kv, "storage", t.storage) + } + if t.upid != "" { + kv = append(kv, "upid", t.upid) + } + kv = append(kv, t.extra...) + kv = append(kv, "duration", time.Since(start)) + if err != nil { + var ae *apiError + if errors.As(err, &ae) { + kv = append(kv, "httpStatus", ae.Status) + } + kv = append(kv, "error", err.Error()) + } + if more != nil && log.V(2).Enabled() { + kv = append(kv, more()...) + } + log.V(1).Info("proxmox api call", kv...) + return err +} + +func (o *observed) Version(ctx context.Context) (v versionInfo, err error) { + err = o.call(ctx, "Version", &target{}, func() (detail, error) { + v, err = o.inner.Version(ctx) + return func() []any { return []any{"version", v.Version} }, err + }) + return v, err +} + +func (o *observed) NodeStatus(ctx context.Context, node string) (s nodeStatus, err error) { + err = o.call(ctx, "NodeStatus", &target{node: node}, func() (detail, error) { + s, err = o.inner.NodeStatus(ctx, node) + return func() []any { return []any{"pveVersion", s.PVEVersion, "kernel", s.CurrentKernel.Release} }, err + }) + return s, err +} + +func (o *observed) StorageStatus(ctx context.Context, node, storage string) (s storageStatus, err error) { + err = o.call(ctx, "StorageStatus", &target{node: node, storage: storage}, func() (detail, error) { + s, err = o.inner.StorageStatus(ctx, node, storage) + return func() []any { + return []any{"type", s.Type, "content", s.Content, "active", s.Active, "avail", s.Avail} + }, err + }) + return s, err +} + +func (o *observed) ClusterResources(ctx context.Context) (rs []vmResource, err error) { + err = o.call(ctx, "ClusterResources", &target{}, func() (detail, error) { + rs, err = o.inner.ClusterResources(ctx) + return func() []any { return []any{"count", len(rs), "vms", resourceSummary(rs)} }, err + }) + return rs, err +} + +func (o *observed) VMConfig(ctx context.Context, node string, vmid int) (cfg vmConfig, err error) { + err = o.call(ctx, "VMConfig", &target{node: node, vmid: vmid}, func() (detail, error) { + cfg, err = o.inner.VMConfig(ctx, node, vmid) + return func() []any { return []any{"config", configSummary(cfg)} }, err + }) + return cfg, err +} + +func (o *observed) SetVMConfig(ctx context.Context, node string, vmid int, params map[string]string) error { + return o.call(ctx, "SetVMConfig", &target{node: node, vmid: vmid}, func() (detail, error) { + return func() []any { return []any{"params", params} }, o.inner.SetVMConfig(ctx, node, vmid, params) + }) +} + +// Clone names the new VM at V(1); the description, which holds the mark, +// only at V(2). +func (o *observed) Clone(ctx context.Context, node string, vmid int, p cloneParams) (upid string, err error) { + t := &target{node: node, vmid: vmid, extra: []any{"newid", p.NewID, "name", p.Name}} + err = o.call(ctx, "Clone", t, func() (detail, error) { + upid, err = o.inner.Clone(ctx, node, vmid, p) + t.upid = upid + return func() []any { + return []any{"description", p.Description, "storage", p.Storage, "target", p.Target} + }, err + }) + return upid, err +} + +func (o *observed) Resize(ctx context.Context, node string, vmid int, disk, size string) (upid string, err error) { + t := &target{node: node, vmid: vmid} + err = o.call(ctx, "Resize", t, func() (detail, error) { + upid, err = o.inner.Resize(ctx, node, vmid, disk, size) + t.upid = upid + return func() []any { return []any{"disk", disk, "size", size} }, err + }) + return upid, err +} + +func (o *observed) Start(ctx context.Context, node string, vmid int) (upid string, err error) { + t := &target{node: node, vmid: vmid} + err = o.call(ctx, "Start", t, func() (detail, error) { + upid, err = o.inner.Start(ctx, node, vmid) + t.upid = upid + return nil, err + }) + return upid, err +} + +func (o *observed) Stop(ctx context.Context, node string, vmid int) (upid string, err error) { + t := &target{node: node, vmid: vmid} + err = o.call(ctx, "Stop", t, func() (detail, error) { + upid, err = o.inner.Stop(ctx, node, vmid) + t.upid = upid + return nil, err + }) + return upid, err +} + +func (o *observed) VMStatus(ctx context.Context, node string, vmid int) (s vmStatus, err error) { + err = o.call(ctx, "VMStatus", &target{node: node, vmid: vmid}, func() (detail, error) { + s, err = o.inner.VMStatus(ctx, node, vmid) + return func() []any { return []any{"status", s.Status, "qmpstatus", s.QMPStatus, "lock", s.Lock} }, err + }) + return s, err +} + +func (o *observed) Destroy(ctx context.Context, node string, vmid int) (upid string, err error) { + t := &target{node: node, vmid: vmid} + err = o.call(ctx, "Destroy", t, func() (detail, error) { + upid, err = o.inner.Destroy(ctx, node, vmid) + t.upid = upid + return nil, err + }) + return upid, err +} + +func (o *observed) TaskStatus(ctx context.Context, node, upid string) (s taskStatus, err error) { + err = o.call(ctx, "TaskStatus", &target{node: node, upid: upid}, func() (detail, error) { + s, err = o.inner.TaskStatus(ctx, node, upid) + return func() []any { return []any{"status", s.Status, "exitstatus", s.ExitStatus} }, err + }) + return s, err +} + +func (o *observed) AgentNetworkInterfaces(ctx context.Context, node string, vmid int) (ifaces []agentInterface, err error) { + err = o.call(ctx, "AgentNetworkInterfaces", &target{node: node, vmid: vmid}, func() (detail, error) { + ifaces, err = o.inner.AgentNetworkInterfaces(ctx, node, vmid) + return func() []any { return []any{"addresses", agentAddresses(ifaces)} }, err + }) + return ifaces, err +} + +// UploadISO logs the file name and size, never the body: a seed carries +// CloudInit. +func (o *observed) UploadISO(ctx context.Context, node, storage, filename string, r io.Reader, size int64) (upid string, err error) { + t := &target{node: node, storage: storage, extra: []any{"filename", filename}} + err = o.call(ctx, "UploadISO", t, func() (detail, error) { + upid, err = o.inner.UploadISO(ctx, node, storage, filename, r, size) + t.upid = upid + return func() []any { return []any{"bytes", size} }, err + }) + return upid, err +} + +func (o *observed) DeleteVolume(ctx context.Context, node, storage, volid string) (upid string, err error) { + t := &target{node: node, storage: storage, extra: []any{"volid", volid}} + err = o.call(ctx, "DeleteVolume", t, func() (detail, error) { + upid, err = o.inner.DeleteVolume(ctx, node, storage, volid) + t.upid = upid + return nil, err + }) + return upid, err +} + +// resourceSummary is one "::" per entry, with the lock +// appended when set. +func resourceSummary(rs []vmResource) []string { + out := make([]string, len(rs)) + for i, r := range rs { + s := strconv.Itoa(r.VMID) + ":" + r.Name + ":" + r.Status + if r.Lock != "" { + s += ":lock=" + r.Lock + } + out[i] = s + } + return out +} + +// summaryKeys are the config keys the provider reads or writes. +var summaryKeys = []string{"name", "lock", "tags", "description", "cores", "memory", "agent", "net0", "ide2", "boot", "bootdisk", "scsi0", "virtio0", "sata0", "ide0", "template"} + +func configSummary(cfg vmConfig) map[string]string { + out := make(map[string]string, len(summaryKeys)) + for _, k := range summaryKeys { + if v, ok := cfg[k]; ok { + out[k] = v + } + } + return out +} + +func agentAddresses(ifaces []agentInterface) []string { + var out []string + for _, iface := range ifaces { + for _, a := range iface.IPAddresses { + out = append(out, iface.Name+"="+a.Address) + } + } + return out +} diff --git a/internal/provider/proxmox/observe_test.go b/internal/provider/proxmox/observe_test.go new file mode 100644 index 0000000..64ff65a --- /dev/null +++ b/internal/provider/proxmox/observe_test.go @@ -0,0 +1,229 @@ +package proxmox + +import ( + "context" + "errors" + "reflect" + "strconv" + "strings" + "sync" + "testing" + + "github.com/go-logr/logr/funcr" + sdktrace "go.opentelemetry.io/otel/sdk/trace" + "go.opentelemetry.io/otel/sdk/trace/tracetest" + logf "sigs.k8s.io/controller-runtime/pkg/log" +) + +// logCapture collects funcr output, safe for concurrent use. +type logCapture struct { + mu sync.Mutex + lines []string +} + +// context returns a ctx whose logger writes into c at verbosity. +func (c *logCapture) context(verbosity int) context.Context { + log := funcr.New(func(prefix, args string) { + c.mu.Lock() + defer c.mu.Unlock() + c.lines = append(c.lines, prefix+" "+args) + }, funcr.Options{Verbosity: verbosity}) + return logf.IntoContext(context.Background(), log) +} + +func (c *logCapture) all() []string { + c.mu.Lock() + defer c.mu.Unlock() + return append([]string(nil), c.lines...) +} + +func (c *logCapture) matching(substr string) []string { + var out []string + for _, l := range c.all() { + if strings.Contains(l, substr) { + out = append(out, l) + } + } + return out +} + +const callLine = `"msg"="proxmox api call"` + +func newObserveFixture(verbosity int) (*fakeAPI, api, *logCapture, context.Context) { + fake := newFakeAPI() + logs := &logCapture{} + return fake, observe(fake, "proxmox-home"), logs, logs.context(verbosity) +} + +func TestObserve_everySeamMethodLogsOneLine(t *testing.T) { + t.Parallel() + for m := range reflect.TypeFor[api]().Methods() { + t.Run(m.Name, func(t *testing.T) { + t.Parallel() + fake, a, logs, ctx := newObserveFixture(1) + args := make([]reflect.Value, m.Type.NumIn()) + args[0] = reflect.ValueOf(ctx) + for j := 1; j < m.Type.NumIn(); j++ { + args[j] = reflect.Zero(m.Type.In(j)) + } + reflect.ValueOf(a).MethodByName(m.Name).Call(args) + + lines := logs.matching(callLine) + if len(lines) != 1 || len(logs.all()) != 1 { + t.Fatalf("got %d call lines of %d, want exactly 1:\n%s", len(lines), len(logs.all()), strings.Join(logs.all(), "\n")) + } + if !strings.HasPrefix(lines[0], "proxmox ") { + t.Errorf("line is not from the proxmox logger: %s", lines[0]) + } + for _, want := range []string{`"call"="` + m.Name + `"`, `"provider"="proxmox-home"`, `"duration"=`, `"level"=1`} { + if !strings.Contains(lines[0], want) { + t.Errorf("line lacks %s: %s", want, lines[0]) + } + } + if n := fake.callCount(m.Name); n != 1 { + t.Errorf("inner %s called %d times, want 1", m.Name, n) + } + }) + } +} + +// TestObserve_startsNoSpan keeps the decorator from duplicating the +// transport's HTTP spans (spec entry 17). +func TestObserve_startsNoSpan(t *testing.T) { + t.Parallel() + sr := tracetest.NewSpanRecorder() + tp := sdktrace.NewTracerProvider(sdktrace.WithSpanProcessor(sr)) + t.Cleanup(func() { _ = tp.Shutdown(context.Background()) }) + fake, a, _, ctx := newObserveFixture(1) + fake.addVM(150042, "proxy-x", "running") + ctx, parent := tp.Tracer("test").Start(ctx, "provider.get") + _, _ = a.ClusterResources(ctx) + _, _ = a.VMConfig(ctx, fakeNode, 150042) + parent.End() + if ended := sr.Ended(); len(ended) != 1 || ended[0].Name() != "provider.get" { + t.Errorf("spans = %d, want only the parent", len(ended)) + } +} + +func TestObserve_identifiersAtV1(t *testing.T) { + t.Parallel() + fake, a, logs, ctx := newObserveFixture(1) + fake.addTemplate(9001, "debian-13-template") + upid, err := a.Clone(ctx, fakeNode, 9001, cloneParams{NewID: 150042, Name: "proxy-x", Description: "proxy-operator-managed=true"}) + if err != nil { + t.Fatalf("Clone error = %v", err) + } + _, _ = a.TaskStatus(ctx, fakeNode, upid) + _, _ = a.DeleteVolume(ctx, fakeNode, fakeISOStorage, seedVolID(fakeISOStorage, "proxy-x")) + lines := logs.matching(callLine) + if len(lines) != 3 { + t.Fatalf("got %d lines, want 3:\n%s", len(lines), strings.Join(logs.all(), "\n")) + } + wants := [][]string{ + {`"call"="Clone" "node"="pv1-1" "vmid"=9001 "upid"="` + upid + `" "newid"=150042 "name"="proxy-x"`}, + {`"call"="TaskStatus" "node"="pv1-1" "upid"="` + upid + `"`}, + {`"call"="DeleteVolume" "node"="pv1-1" "storage"="proxy-seeds"`, `"volid"="proxy-seeds:iso/proxy-x-seed.iso"`, `"upid"="UPID:pv1-1:`}, + } + for i, want := range wants { + for _, w := range want { + if !strings.Contains(lines[i], w) { + t.Errorf("line %d lacks %s:\n%s", i, w, lines[i]) + } + } + } + if strings.Contains(lines[0], "description") { + t.Errorf("the description reached V(1): %s", lines[0]) + } +} + +func TestObserve_detailOnlyAtV2(t *testing.T) { + t.Parallel() + tests := []struct { + name string + verbosity int + want bool + }{ + {name: "V1 has no detail", verbosity: 1, want: false}, + {name: "V2 has detail", verbosity: 2, want: true}, + } + for _, tc := range tests { + t.Run(tc.name, func(t *testing.T) { + t.Parallel() + fake, a, logs, ctx := newObserveFixture(tc.verbosity) + fake.addTemplate(9001, "debian-13-template") + vm := fake.addVM(150042, "proxy-x", "running") + vm.ifaces = []agentInterface{{Name: "ens18", IPAddresses: []agentIP{{Address: "192.168.4.147", Type: "ipv4"}}}} + _, _ = a.ClusterResources(ctx) + _, _ = a.VMConfig(ctx, fakeNode, 150042) + _ = a.SetVMConfig(ctx, fakeNode, 9001, map[string]string{"cores": "3"}) + _, _ = a.Clone(ctx, fakeNode, 9001, cloneParams{NewID: 150043, Name: "proxy-y", Description: "mark-line"}) + _, _ = a.AgentNetworkInterfaces(ctx, fakeNode, 150042) + all := strings.Join(logs.all(), "\n") + for _, detail := range []string{`"vms"=["9001:debian-13-template:stopped" "150042:proxy-x:running"]`, `"config"={`, `"params"={"cores"="3"}`, `"description"="mark-line"`, `"addresses"=["ens18=192.168.4.147"]`} { + if got := strings.Contains(all, detail); got != tc.want { + t.Errorf("%s logged = %v, want %v:\n%s", detail, got, tc.want, all) + } + } + if got := len(logs.matching(callLine)); got != 5 { + t.Errorf("got %d call lines, want 5", got) + } + }) + } +} + +func TestObserve_uploadBodyNeverLogged(t *testing.T) { + t.Parallel() + const sentinel = "SENTINEL-CLOUDINIT-51c9" + fake, a, logs, ctx := newObserveFixture(2) + body := "#cloud-config\npassword: " + sentinel + "\n" + if _, err := a.UploadISO(ctx, fakeNode, fakeISOStorage, "proxy-x-seed.iso", strings.NewReader(body), int64(len(body))); err != nil { + t.Fatalf("UploadISO error = %v", err) + } + if got, _ := fake.iso(fakeISOStorage, "proxy-x-seed.iso"); string(got) != body { + t.Errorf("fake got %q, want the body passed through", got) + } + all := strings.Join(logs.all(), "\n") + if strings.Contains(all, sentinel) { + t.Errorf("upload body leaked into logs:\n%s", all) + } + lines := logs.matching(`"call"="UploadISO"`) + if len(lines) != 1 || !strings.Contains(lines[0], `"filename"="proxy-x-seed.iso"`) || !strings.Contains(lines[0], `"bytes"=`+strconv.Itoa(len(body))) { + t.Errorf("upload line lacks filename or bytes: %v", lines) + } +} + +func TestObserve_errorFieldsOnFailure(t *testing.T) { + t.Parallel() + fake, a, logs, ctx := newObserveFixture(1) + _, err := a.VMConfig(ctx, fakeNode, 999999) + if !isNotExist(err) { + t.Fatalf("error = %v, want the does-not-exist apiError through the decorator", err) + } + lines := logs.matching(callLine) + if len(lines) != 1 { + t.Fatalf("got %d lines, want 1", len(lines)) + } + for _, want := range []string{`"httpStatus"=500`, `"error"="proxmox API 500: Configuration file 'nodes/pv1-1/qemu-server/999999.conf' does not exist"`} { + if !strings.Contains(lines[0], want) { + t.Errorf("line lacks %s: %s", want, lines[0]) + } + } + + fake.failWith("Version", errors.New("dial tcp: connection refused")) + _, _ = a.Version(ctx) + lines = logs.matching(`"call"="Version"`) + if len(lines) != 1 || strings.Contains(lines[0], "httpStatus") || !strings.Contains(lines[0], `"error"="dial tcp: connection refused"`) { + t.Errorf("non-API failure line = %v, want the error without httpStatus", lines) + } +} + +func TestObserve_v0Silent(t *testing.T) { + t.Parallel() + fake, a, logs, ctx := newObserveFixture(0) + fake.addVM(150042, "proxy-x", "running") + _, _ = a.VMConfig(ctx, fakeNode, 150042) + _, _ = a.VMConfig(ctx, fakeNode, 999999) + if lines := logs.all(); len(lines) != 0 { + t.Errorf("got %d lines at V(0), want none:\n%s", len(lines), strings.Join(lines, "\n")) + } +} diff --git a/internal/provider/proxmox/proxmox.go b/internal/provider/proxmox/proxmox.go index 1b19826..66fe882 100644 --- a/internal/provider/proxmox/proxmox.go +++ b/internal/provider/proxmox/proxmox.go @@ -76,9 +76,11 @@ func New(ctx context.Context, pc provider.ProviderConfig) (provider.Provider, er if pc.Proxmox != nil { cfg = *pc.Proxmox } - // The HTTP client behind the api seam arrives in Phase 4 of the plan; - // until then every method is a stub that makes no request. - p := newProvider(pc.Name, cfg, nil) + client, err := newHTTPAPI(withDefaults(cfg)) + if err != nil { + return nil, fmt.Errorf("proxmox provider %s: %w", pc.Name, err) + } + p := newProvider(pc.Name, cfg, observe(client, pc.Name)) log := p.logger(ctx) if _, err := os.Stat(p.cfg.TokenFile); err != nil { log.Info("WARNING: token file is not readable; every call fails until it is", "path", p.cfg.TokenFile, "reason", err.Error()) @@ -89,7 +91,7 @@ func New(ctx context.Context, pc provider.ProviderConfig) (provider.Provider, er return p, nil } -// newProvider is the constructor tests use, with a in-memory api. +// newProvider is the constructor tests use, with an in-memory api. func newProvider(name string, cfg provider.ProxmoxConfig, a api) *Provider { return &Provider{ name: name, diff --git a/internal/provider/proxmox/proxmox_test.go b/internal/provider/proxmox/proxmox_test.go index 1197571..2add898 100644 --- a/internal/provider/proxmox/proxmox_test.go +++ b/internal/provider/proxmox/proxmox_test.go @@ -95,6 +95,31 @@ func TestNew_nilBlockDoesNotPanic(t *testing.T) { } } +// TestNew_wiresObservedHTTPClient checks that production calls go through +// the logging decorator to the HTTP client, with /api2/json appended and +// the request timeout from the config. +func TestNew_wiresObservedHTTPClient(t *testing.T) { + t.Parallel() + ctx, _ := captureContext() + cfg := testConfig() + cfg.RequestTimeoutSeconds = 45 + p, err := New(ctx, provider.ProviderConfig{Name: "proxmox-home", Type: "proxmox", Proxmox: &cfg}) + if err != nil { + t.Fatalf("New() error = %v", err) + } + o, ok := p.(*Provider).api.(*observed) + if !ok { + t.Fatalf("api = %T, want *observed", p.(*Provider).api) + } + h, ok := o.inner.(*httpAPI) + if !ok { + t.Fatalf("observed.inner = %T, want *httpAPI", o.inner) + } + if o.provider != "proxmox-home" || h.base.String() != "https://192.0.2.8:8006/api2/json" || h.timeout != 45*time.Second || h.tokenFile != cfg.TokenFile { + t.Errorf("wiring = provider %q base %q timeout %v token file %q", o.provider, h.base, h.timeout, h.tokenFile) + } +} + func TestWithDefaults_table(t *testing.T) { t.Parallel() off := false From 803716459af16a2c9c0c242aa09d361e472c7957 Mon Sep 17 00:00:00 2001 From: kacerr Date: Wed, 23 Sep 2026 15:11:20 +0000 Subject: [PATCH 08/13] proxmox: Get, ListByTag and task polling Get follows spec entry 9's check order (lock and unknown status before the name, suspended is Stopped after the name and mark checks), reads status/current so paused is Stopped, and maps the three agent messages of entry 7. ListByTag reads one config per candidate and fails whole only on the resources call or an ended ctx. waitTask polls the UPID's node every budgets.poll, logs failed tasks at V(1) and counts "WARNINGS: " as success; waitUnlocked serves Create's ownership check. Co-Authored-By: Claude Opus 5.5 (1M context) --- .../2026-09-18-1502-proxmox-provider.md | 105 +++- docs/proxmox-provider-spec.md | 14 +- internal/provider/proxmox/fake_test.go | 22 +- internal/provider/proxmox/get.go | 132 +++++ internal/provider/proxmox/get_test.go | 476 ++++++++++++++++++ internal/provider/proxmox/list.go | 81 +++ internal/provider/proxmox/list_test.go | 227 +++++++++ internal/provider/proxmox/provider_test.go | 103 ++++ internal/provider/proxmox/proxmox.go | 14 +- internal/provider/proxmox/proxmox_test.go | 6 +- internal/provider/proxmox/task.go | 109 ++++ internal/provider/proxmox/task_test.go | 233 +++++++++ 12 files changed, 1497 insertions(+), 25 deletions(-) create mode 100644 internal/provider/proxmox/get.go create mode 100644 internal/provider/proxmox/get_test.go create mode 100644 internal/provider/proxmox/list.go create mode 100644 internal/provider/proxmox/list_test.go create mode 100644 internal/provider/proxmox/provider_test.go create mode 100644 internal/provider/proxmox/task.go create mode 100644 internal/provider/proxmox/task_test.go diff --git a/docs/plans-executions/2026-09-18-1502-proxmox-provider.md b/docs/plans-executions/2026-09-18-1502-proxmox-provider.md index ca16638..d2e7308 100644 --- a/docs/plans-executions/2026-09-18-1502-proxmox-provider.md +++ b/docs/plans-executions/2026-09-18-1502-proxmox-provider.md @@ -9,7 +9,7 @@ Pairs with [docs/plans/2026-09-18-1502-proxmox-provider.md](../plans/2026-09-18- - [x] Phase 2 — Config block, registration, package skeleton - [x] Phase 3 — Extract `nocloud` and `keyedlock` from kvm2 - [x] Phase 4 — Seam, HTTP client, decorator, fake -- [ ] Phase 5 — Get, ListByTag, task helpers +- [x] Phase 5 — Get, ListByTag, task helpers - [ ] Phase 6 — Create - [ ] Phase 7 — Delete, Checker, CRD comment - [ ] Phase 8 — Contract and observability sweep, integration test, testing.md @@ -554,3 +554,106 @@ Worth noting: `fake: VM is running - destroy failed`, so Phase 7's stop-before-destroy order is enforced by the fake even though the real text is unknown. + +## Phase 5 — Get, ListByTag, task helpers + +`task.go` holds `parseUPID`, a shared `pollUntil` loop, `waitTask` and +`waitUnlocked`. `get.go` holds `Get`, the guest-agent read and `resolve` +(one resources call, QEMU entries only, which Delete will reuse). `list.go` +holds `ListByTag` and its candidate rule. The Get and ListByTag stubs are +gone, and `TestStubs_answerTransient` now covers only Create and Delete. + +The spec overrides the plan in the places the Phase 4 notes listed. Get +checks in entry 9's order, so a lock or `status: unknown` in the resources +entry is Transient before the name is ever read, and a running Get makes +four calls (`TestGet_runningMakesFourCalls`). The three agent messages +follow entry 7. The poll interval is `budgets.poll`, and a failed task logs +a V(1) `proxmox task failed` line with `upid` and `exitstatus`. + +Judgment calls, most of them spec gaps: + +- **`WARNINGS: ` counts as a successful task.** Proxmox reports a task + that finished but logged warnings with that exitstatus and treats it as + finished, not failed. Under the spec's old "anything but `OK` is + Transient" rule, a start that logged a warning would make Create retry + forever. Recon never produced one, so this row relies on how Proxmox is + known to behave, not on a capture. The spec's entry 10 text and entry 13 + table now say so. +- **A `suspended` lock still goes through the name and mark checks.** The + entry 9 table puts "lock suspended is Stopped" before the config rows. + Taken literally, a hibernated VM that reused our VMID would be reported + as ours and Stopped, and Delete would then skip it as foreign in a loop. + Get therefore reads the config, which it needs anyway for UID and + CreatedAt, applies the name and mark checks, and only then answers + Stopped. It skips `status/current` and the agent. Entry 9 now says this. +- **`TestGet_twoVMsWithOurNameIsPermanent` became + `TestGet_otherVMWithOurNameIsIgnored`.** Spec entries 2 and 9 address Get + by VMID, and only Create refuses a duplicate name. Get answering + Permanent would latch a working Proxy as Failed because of a VM someone + else created. +- **`TestGet_unmanagedDescriptionHasEmptyUID` tests a mark with no `uid` + line.** The plan's name read literally ("unmanaged") would contradict + "no mark is NotFound". +- **Agent errors outside the entry 7 table are forced to Transient**, even + a 403 or a `does not exist`. Otherwise `classify` would make them + Permanent or NotFound, and the spec says "anything else: Transient". +- **`waitUnlocked` returns the unlocked config**, which saves Create's + ownership check one read. When the VM disappears while it waits (a + failed clone), it returns the does-not-exist error and lets Create + decide. + +Pulled forward and left behind: + +- **`provider_test.go` now exists** with `newFixture`, `testBudgets`, + `addOurVM` and `assertProviderError`, because every Get and ListByTag + test needs them. The plan created this file in Phase 8. Phase 8 adds the + contract sweep to it. +- **`waitRunning` is left to Phase 6.** The plan's package layout puts it + in `task.go`, but Phase 5 lists no test for it and only Create uses it. + It should be built on `pollUntil`. +- **The fake gained two things.** `fakeVM.kind` makes an entry listed as + `lxc` invisible to the qemu endpoints. `hookOn` is a general per-method + hook, and `failWith` is now built on it. Tests use `hookOn` (through + `mutateOn` in `get_test.go`) to change a VM between two of Get's calls. + +A mutation check confirmed the new tests can fail. Each of these failed a +test: classifying agent errors instead of forcing Transient; dropping the +config lock check; dropping the `suspended` special case; ignoring an ended +ctx in ListByTag; treating `WARNINGS` as failure; and not mapping +`No QEMU guest agent configured` to Stopped. Removing the resources-entry +lock and `unknown` checks together failed only +`TestGet_unknownStatusIsTransient`. The fake reports a lock in the config +as well, so the config check catches every lock row, including a mid-clone +VM. The resources check stays because the spec orders it first and +because it spares a config read. Every mutation was reverted, and the +files were compared identical to backups. + +```bash +go test -race -count=2 ./internal/provider/proxmox/ # ok +grep -n 'not implemented' internal/provider/proxmox/*.go | grep -v _test # Create and Delete only +go build ./... && go vet ./... && go vet -tags=integration ./... +go test -race -count=1 ./internal/provider/... ./cmd/... # all ok +go test -short -count=1 ./... # all ok +go test -count=1 -v ./internal/provider/proxmox/ | grep -c '^=== RUN' # 289 before, 393 after +``` + +Lint ran with staticcheck disabled, because staticcheck panics on +Go 1.27: + +```bash +bin/golangci-lint run --disable staticcheck ./internal/provider/proxmox/... +``` + +Worth noting: + +- The lint findings in the new files are goconst on test strings and + logcheck on `get`, `agentAddress` and `waitTask` taking both a ctx and a + logger. That shape copies kvm2, which has five of the same findings. The + errcheck hits on `assertGetFails` were fixed by returning the error text. + The two unparam hits and the mapsloop hits predate this phase. +- `waitUnlocked` waits on any lock, `suspended` included. Phase 6's + ownership check should decide whether a hibernated VM at our VMID is + worth waiting out the clone budget for, or should be refused at once. +- Nothing in this phase talked to the host. Every Proxmox message a test + relies on comes from `testdata/` or from fake errors that + `TestFakeAPI_errorsMatchCaptures` ties to the captures. diff --git a/docs/proxmox-provider-spec.md b/docs/proxmox-provider-spec.md index 9e1dda4..84712ab 100644 --- a/docs/proxmox-provider-spec.md +++ b/docs/proxmox-provider-spec.md @@ -285,6 +285,11 @@ and starts it; the reconciler has no ID for it to `Get`. | `status: running`, any other `qmpstatus` (`paused`, `suspended`, `io-error`, …) | `status/current` | Stopped | | any other `status` | `status/current` | Stopped | +A `suspended` lock decides the state but not ownership: the config is still +read, and a foreign name or a missing mark is still `ErrNotFound`, so a +hibernated VM that reused our VMID is never reported as ours. Only +`status/current` and the agent are skipped for it. + `status/current` is read, not only the resources entry, because a paused VM shows as plain `running` in `/cluster/resources`. `UID` and `CreatedAt` come from the description on every call. A running `Get` makes four calls: @@ -301,9 +306,10 @@ contract §2.2 and §4.11. `backend-lock`. Clone, upload, resize, start, stop, destroy and volume delete each return a UPID; the provider polls `GET /nodes/{node}/tasks/{upid}/status` every second until `status` is -`stopped`, then reads `exitstatus`: `OK` is success, anything else is -`ErrTransient` carrying that text. The node comes from the UPID -(`UPID::…`). The handle is held within the call only: Create blocks +`stopped`, then reads `exitstatus`: `OK` is success, and so is +`WARNINGS: `, which Proxmox itself counts as a finished task (logged at +V(1)); anything else is `ErrTransient` carrying that text. The node comes +from the UPID (`UPID::…`). The handle is held within the call only: Create blocks until the VM runs, so no handle outlives it, and a failure surfaces as Create's own error with the task's message. Locks: a VM locked by a task is waited on in Create (the ownership check) and reported as `ErrTransient` by @@ -447,7 +453,7 @@ per-parameter map. One `classify(op string, err error)`: | 500 `unable to create VM : config file already exists` | handled by the allocator (entry 2); Transient if it escapes | the VMID is taken | | 500 `QEMU guest agent is not running`, `No QEMU guest agent configured`, `VM is not running` | handled by `Get` (entry 7); Transient if they escape | agent state | | any other 5xx, including pveproxy's 595 and 596 | Transient | unknown means retry | -| task `exitstatus` other than `OK` | Transient, with the text | the task failed; the next attempt retries | +| task `exitstatus` other than `OK` or `WARNINGS: ` | Transient, with the text | the task failed; the next attempt retries | | TLS verification failure (`x509.UnknownAuthorityError`, `x509.HostnameError`, `x509.CertificateInvalidError`) | Permanent | wrong URL or CA file; the admin must act | | token file missing (`fs.ErrNotExist`) | Transient | the Secret has not landed yet | | token file present but not `@!=` | Permanent | wrong content in the Secret | diff --git a/internal/provider/proxmox/fake_test.go b/internal/provider/proxmox/fake_test.go index 62242e6..52206d0 100644 --- a/internal/provider/proxmox/fake_test.go +++ b/internal/provider/proxmox/fake_test.go @@ -52,9 +52,12 @@ type fakeAPI struct { type fakeVM struct { node, name, status, qmpstatus, lock string - template bool - config map[string]string - ifaces []agentInterface + // kind is the resource type; empty is "qemu". Any other kind, such as + // "lxc", is listed but invisible to the qemu endpoints. + kind string + template bool + config map[string]string + ifaces []agentInterface } type fakeTask struct { @@ -125,9 +128,15 @@ func (f *fakeAPI) addTemplate(vmid int, name string) *fakeVM { } func (f *fakeAPI) failWith(method string, err error) { + f.hookOn(method, func() error { return err }) +} + +// hookOn runs fn at the start of every call to method, without f.mu held; +// a non-nil result fails the call. A hook that changes state takes f.mu. +func (f *fakeAPI) hookOn(method string, fn func() error) { f.mu.Lock() defer f.mu.Unlock() - f.hooks[method] = func() error { return err } + f.hooks[method] = fn } func (f *fakeAPI) blockOn(method string) chan struct{} { @@ -217,7 +226,7 @@ func (f *fakeAPI) enter(ctx context.Context, method string) error { // node has no config file on this one. func (f *fakeAPI) vm(node string, vmid int) (*fakeVM, error) { vm, ok := f.vms[vmid] - if !ok || vm.node != node { + if !ok || vm.node != node || vm.kind != "" { return nil, errVMMissing(node, vmid) } return vm, nil @@ -279,6 +288,9 @@ func (f *fakeAPI) ClusterResources(ctx context.Context) ([]vmResource, error) { for _, vmid := range slices.Sorted(maps.Keys(f.vms)) { vm := f.vms[vmid] r := vmResource{VMID: vmid, Name: vm.name, Node: vm.node, Type: "qemu", Status: vm.status, Lock: vm.lock, Tags: vm.config["tags"]} + if vm.kind != "" { + r.Type = vm.kind + } if vm.template { r.Template = 1 } diff --git a/internal/provider/proxmox/get.go b/internal/provider/proxmox/get.go new file mode 100644 index 0000000..b0a2379 --- /dev/null +++ b/internal/provider/proxmox/get.go @@ -0,0 +1,132 @@ +package proxmox + +import ( + "context" + "errors" + "fmt" + + "github.com/go-logr/logr" + + "gitea.home.hrajfrisbee.cz/kacerr/egress-proxies-operator/internal/provider" +) + +// lockSuspended is the lock of a VM hibernated to disk: it is not a task in +// progress, and the VM stays that way until someone resumes it. +const lockSuspended = "suspended" + +// Get is spec entries 7 to 9: the VMID's resource entry, then its config +// for the name and the mark, then status/current and, for a running VM, the +// guest agent. A running Get makes four calls. +func (p *Provider) Get(ctx context.Context, id string) (*provider.Instance, error) { + vmid, name, err := parseProviderID(id) + if err != nil { + return nil, p.wrapErr("get", id, err) + } + log := p.logger(ctx).WithValues("vmid", vmid, "name", name) + inst, err := p.get(ctx, log, vmid, name) + if err != nil { + err = p.wrapErr("get", id, err) + if errors.Is(err, provider.ErrNotFound) { + log.V(1).Info("proxmox vm not found", "reason", err.Error()) + } + return nil, err + } + return inst, nil +} + +func (p *Provider) get(ctx context.Context, log logr.Logger, vmid int, name string) (*provider.Instance, error) { + r, ok, err := p.resolve(ctx, vmid) + if err != nil { + return nil, err + } + if !ok { + return nil, fmt.Errorf("%w: VM %d is not in the cluster's resources", provider.ErrNotFound, vmid) + } + log = log.WithValues("node", r.Node) + // A mid-clone entry has no name and status unknown, so the lock is + // checked before anything that would read it as absent or foreign. + if r.Lock != "" && r.Lock != lockSuspended { + return nil, transientf("VM %d is locked (%s)", vmid, r.Lock) + } + if r.Status == "unknown" { + return nil, transientf("VM %d has status unknown: its node %s is not answering", vmid, r.Node) + } + cfg, err := p.api.VMConfig(ctx, r.Node, vmid) + if err != nil { + return nil, fmt.Errorf("reading config of VM %d: %w", vmid, err) + } + if lock := cfg["lock"]; lock != "" && lock != lockSuspended { + return nil, transientf("VM %d is locked (%s)", vmid, lock) + } + if cfg["name"] != name { + return nil, fmt.Errorf("%w: VM %d is now named %q: the VMID was reused", provider.ErrNotFound, vmid, cfg["name"]) + } + mark := parseDescription(cfg["description"]) + if !mark.Managed { + return nil, fmt.Errorf("%w: VM %d carries no proxy-operator mark", provider.ErrNotFound, vmid) + } + inst := &provider.Instance{ID: formatProviderID(vmid, name), UID: mark.UID, CreatedAt: mark.CreatedAt} + + // A hibernated VM is Stopped whatever status/current says; it needs no + // further call. + if r.Lock == lockSuspended || cfg["lock"] == lockSuspended { + inst.State = provider.StateStopped + log.V(1).Info("proxmox vm fetched", "state", inst.State, "lock", lockSuspended, "uid", inst.UID) + return inst, nil + } + st, err := p.api.VMStatus(ctx, r.Node, vmid) + if err != nil { + return nil, fmt.Errorf("reading status of VM %d: %w", vmid, err) + } + if st.Status == "running" && (st.QMPStatus == "running" || st.QMPStatus == "") { + ip, state, err := p.agentAddress(ctx, log, r.Node, vmid) + if err != nil { + return nil, err + } + inst.IP = ip + inst.State = state + } else { + inst.State = mapState(st.Status, st.QMPStatus, "") + } + log.V(1).Info("proxmox vm fetched", "state", inst.State, "ip", inst.IP, "uid", inst.UID, "status", st.Status, "qmpstatus", st.QMPStatus) + return inst, nil +} + +// agentAddress asks the guest agent of a running VM for its address and +// turns its answers into a state (spec entry 7). Any answer it cannot place +// is Transient, whatever its status: an address read never latches or +// loses a Proxy. +func (p *Provider) agentAddress(ctx context.Context, log logr.Logger, node string, vmid int) (string, provider.InstanceState, error) { + ifaces, err := p.api.AgentNetworkInterfaces(ctx, node, vmid) + switch { + case err == nil: + ip := firstIPv4(ifaces) + log.V(2).Info("proxmox guest agent answered", "interfaces", len(ifaces), "ip", ip) + return ip, mapState("running", "running", ip), nil + case isAgentNotRunning(err): + return "", provider.StateProvisioning, nil + case isAgentNotConfigured(err): + // agent=1 was removed: this VM can never report an address, so it + // is replaced by one that can. + return "", provider.StateStopped, nil + default: + // Includes "VM is not running": it stopped since the status + // read, and the next Get sees that directly. + return "", "", fmt.Errorf("%w: reading the guest agent of VM %d: %w", provider.ErrTransient, vmid, err) + } +} + +// resolve finds a VMID in the cluster's resources. Only QEMU VMs count: a +// container at the VMID means ours is gone. +func (p *Provider) resolve(ctx context.Context, vmid int) (vmResource, bool, error) { + rs, err := p.api.ClusterResources(ctx) + if err != nil { + return vmResource{}, false, fmt.Errorf("listing cluster resources: %w", err) + } + for _, r := range rs { + if r.VMID == vmid && r.Type == "qemu" { + return r, true, nil + } + } + return vmResource{}, false, nil +} diff --git a/internal/provider/proxmox/get_test.go b/internal/provider/proxmox/get_test.go new file mode 100644 index 0000000..ca7de3b --- /dev/null +++ b/internal/provider/proxmox/get_test.go @@ -0,0 +1,476 @@ +package proxmox + +import ( + "errors" + "slices" + "strings" + "testing" + "time" + + "gitea.home.hrajfrisbee.cz/kacerr/egress-proxies-operator/internal/provider" +) + +// mutateOn changes the fake's state under its lock at the start of every +// call to method: a VM changing between two of Get's calls. +func mutateOn(f *fakeAPI, method string, fn func()) { + f.hookOn(method, func() error { + f.mu.Lock() + defer f.mu.Unlock() + fn() + return nil + }) +} + +// capturedIfaces is the agent's answer from the Phase 0 guest: lo and ens18 +// with 192.168.0.150. +func capturedIfaces(t *testing.T) []agentInterface { + t.Helper() + return decodeData[struct { + Result []agentInterface `json:"result"` + }](t, "agent_network_get_interfaces").Result +} + +// assertGetFails runs Get on fx.id(), checks the error's class and returns +// its text. +func assertGetFails(t *testing.T, fx *fixture, class error) string { + t.Helper() + inst, err := fx.p.Get(fx.ctx, fx.id()) + if inst != nil { + t.Errorf("Get = %+v, want nil", inst) + } + assertProviderError(t, err, "get", class) + return err.Error() +} + +func TestGet_missingVMIsNotFound(t *testing.T) { + t.Parallel() + fx := newFixture(t, 1) + fx.addOurVM(testVMID+1, testName, "running") + assertGetFails(t, fx, provider.ErrNotFound) + lines := fx.logs.matching(`"msg"="proxmox vm not found"`) + if len(lines) != 1 || !strings.Contains(lines[0], `"vmid"=150042`) || !strings.Contains(lines[0], `"name"="`+testName+`"`) { + t.Errorf("not-found lines = %v, want one with vmid and name", lines) + } + if got := fx.fake.callLog(); !slices.Equal(got, []string{"ClusterResources"}) { + t.Errorf("calls = %v, want only the resources listing", got) + } +} + +func TestGet_containerAtVMIDIsNotFound(t *testing.T) { + t.Parallel() + fx := newFixture(t, 1) + fx.addOurVM(testVMID, testName, "running").kind = "lxc" + assertGetFails(t, fx, provider.ErrNotFound) +} + +func TestGet_nameMismatchIsNotFound(t *testing.T) { + t.Parallel() + fx := newFixture(t, 1) + fx.addOurVM(testVMID, "proxy-someoneelse00000", "running") + msg := assertGetFails(t, fx, provider.ErrNotFound) + if !strings.Contains(msg, "reused") { + t.Errorf("error = %s, want it to say the VMID was reused", msg) + } + if n := fx.fake.callCount("VMStatus"); n != 0 { + t.Errorf("VMStatus calls = %d, want 0 for a VM that is not ours", n) + } + if lines := fx.logs.matching(`"msg"="proxmox vm not found"`); len(lines) != 1 { + t.Errorf("not-found lines = %v, want one", lines) + } +} + +func TestGet_unmarkedIsNotFound(t *testing.T) { + t.Parallel() + tests := []struct { + name string + description string + }{ + {name: "no description"}, + {name: "managed false", description: "proxy-operator-managed=false\nproxy-operator-uid=" + testUID}, + {name: "admin notes only", description: "hand made, do not touch"}, + } + for _, tc := range tests { + t.Run(tc.name, func(t *testing.T) { + t.Parallel() + fx := newFixture(t, 1) + vm := fx.fake.addVM(testVMID, testName, "running") + if tc.description != "" { + vm.config["description"] = tc.description + } + msg := assertGetFails(t, fx, provider.ErrNotFound) + if !strings.Contains(msg, "no proxy-operator mark") { + t.Errorf("error = %s, want it to name the missing mark", msg) + } + }) + } +} + +// TestGet_otherVMWithOurNameIsIgnored replaces the plan's +// TestGet_twoVMsWithOurNameIsPermanent: spec entries 2 and 9 address Get by +// VMID, and only Create treats a duplicate name as Permanent. +func TestGet_otherVMWithOurNameIsIgnored(t *testing.T) { + t.Parallel() + fx := newFixture(t, 1) + fx.addOurVM(testVMID, testName, "stopped") + fx.addOurVM(testVMID+7, testName, "running") + if inst := fx.get(fx.id()); inst.State != provider.StateStopped || inst.ID != fx.id() { + t.Errorf("Get = %+v, want the VM at the ID's VMID, Stopped", inst) + } +} + +func TestGet_vmOnAnotherNodeIsFound(t *testing.T) { + t.Parallel() + fx := newFixture(t, 1) + vm := fx.addOurVM(testVMID, testName, "running") + vm.node = "pv1-2" + vm.ifaces = capturedIfaces(t) + inst := fx.get(fx.id()) + if inst.State != provider.StateRunning || inst.IP != "192.168.0.150" { + t.Errorf("Get = %+v, want Running with 192.168.0.150 read from pv1-2", inst) + } + if lines := fx.logs.matching(`"msg"="proxmox vm fetched"`); len(lines) != 1 || !strings.Contains(lines[0], `"node"="pv1-2"`) { + t.Errorf("fetched lines = %v, want the resolved node", lines) + } +} + +func TestGet_runningWithoutIPIsProvisioning(t *testing.T) { + t.Parallel() + fx := newFixture(t, 1) + fx.addOurVM(testVMID, testName, "running") + if inst := fx.get(fx.id()); inst.State != provider.StateProvisioning || inst.IP != "" { + t.Errorf("Get = %+v, want Provisioning with no IP", inst) + } +} + +func TestGet_runningWithIPIsRunning(t *testing.T) { + t.Parallel() + fx := newFixture(t, 1) + fx.addOurVM(testVMID, testName, "running").ifaces = capturedIfaces(t) + inst := fx.get(fx.id()) + if inst.State != provider.StateRunning || inst.IP != "192.168.0.150" || inst.ID != fx.id() { + t.Errorf("Get = %+v, want Running with 192.168.0.150", inst) + } + lines := fx.logs.matching(`"msg"="proxmox vm fetched"`) + if len(lines) != 1 { + t.Fatalf("fetched lines = %v, want one", lines) + } + for _, kv := range []string{`"provider"="proxmox-home"`, `"vmid"=150042`, `"name"="` + testName + `"`, `"node"="pv1-1"`, `"state"="Running"`, `"ip"="192.168.0.150"`, `"uid"="` + testUID + `"`} { + if !strings.Contains(lines[0], kv) { + t.Errorf("fetched line lacks %s: %s", kv, lines[0]) + } + } +} + +func TestGet_runningMakesFourCalls(t *testing.T) { + t.Parallel() + fx := newFixture(t, 1) + fx.addOurVM(testVMID, testName, "running").ifaces = capturedIfaces(t) + fx.get(fx.id()) + want := []string{"ClusterResources", "VMConfig", "VMStatus", "AgentNetworkInterfaces"} + if got := fx.fake.callLog(); !slices.Equal(got, want) { + t.Errorf("calls = %v, want %v", got, want) + } +} + +func TestGet_agentNotRunningIsProvisioning(t *testing.T) { + t.Parallel() + fx := newFixture(t, 1) + fx.addOurVM(testVMID, testName, "running").ifaces = capturedIfaces(t) + fx.fake.agentNotRunning = true + if inst := fx.get(fx.id()); inst.State != provider.StateProvisioning || inst.IP != "" { + t.Errorf("Get = %+v, want Provisioning while the agent is not running", inst) + } +} + +// TestGet_agentMessagesTable is spec entry 7's table, with the agent's +// answers taken from the captures. +func TestGet_agentMessagesTable(t *testing.T) { + t.Parallel() + tests := []struct { + name string + agentErr error + ifaces []agentInterface + wantState provider.InstanceState + wantClass error + }{ + {name: "200 with a routable IPv4", ifaces: capturedIfaces(t), wantState: provider.StateRunning}, + { + name: "200 without an IPv4", + ifaces: []agentInterface{{Name: "lo", IPAddresses: []agentIP{{Address: "127.0.0.1", Type: "ipv4"}}}, {Name: "ens18", IPAddresses: []agentIP{{Address: "fe80::1", Type: "ipv6"}}}}, + wantState: provider.StateProvisioning, + }, + {name: "agent not running", agentErr: loadAPIError(t, "err_agent_not_running"), wantState: provider.StateProvisioning}, + {name: "no agent configured", agentErr: loadAPIError(t, "err_agent_vm_stopped"), wantState: provider.StateStopped}, + {name: "VM stopped since the status read", agentErr: loadAPIError(t, "err_agent_vm_stopped_agent_enabled"), wantClass: provider.ErrTransient}, + } + for _, tc := range tests { + t.Run(tc.name, func(t *testing.T) { + t.Parallel() + fx := newFixture(t, 1) + fx.addOurVM(testVMID, testName, "running").ifaces = tc.ifaces + if tc.agentErr != nil { + fx.fake.failWith("AgentNetworkInterfaces", tc.agentErr) + } + if tc.wantClass != nil { + assertGetFails(t, fx, tc.wantClass) + return + } + inst := fx.get(fx.id()) + if inst.State != tc.wantState { + t.Errorf("Get state = %s, want %s", inst.State, tc.wantState) + } + if (inst.State == provider.StateRunning) != (inst.IP != "") { + t.Errorf("Get = %+v: an IP goes with Running and only with Running", inst) + } + }) + } +} + +func TestGet_agentOtherErrorIsTransient(t *testing.T) { + t.Parallel() + tests := []struct { + name string + err error + }{ + {name: "forbidden", err: loadAPIError(t, "err_forbidden")}, + {name: "VM vanished since the status read", err: loadAPIError(t, "err_vm_missing")}, + {name: "pveproxy 596", err: fakeErr(596, "Broken pipe")}, + {name: "other 500", err: fakeErr(500, "fake: guest-network-get-interfaces timed out")}, + {name: "not an api error", err: errors.New("read tcp: connection reset by peer")}, + } + for _, tc := range tests { + t.Run(tc.name, func(t *testing.T) { + t.Parallel() + fx := newFixture(t, 1) + fx.addOurVM(testVMID, testName, "running") + fx.fake.failWith("AgentNetworkInterfaces", tc.err) + assertGetFails(t, fx, provider.ErrTransient) + }) + } +} + +func TestGet_ipFilterSkipsLoopbackAndLinkLocal(t *testing.T) { + t.Parallel() + fx := newFixture(t, 1) + fx.addOurVM(testVMID, testName, "running").ifaces = []agentInterface{ + {Name: "lo", IPAddresses: []agentIP{{Address: "127.0.0.1", Type: "ipv4"}}}, + {Name: "ens18", IPAddresses: []agentIP{{Address: "169.254.10.2", Type: "ipv4"}, {Address: "fe80::be24:11ff:fe48:1146", Type: "ipv6"}}}, + {Name: "ens19", IPAddresses: []agentIP{{Address: "2001:db8::5", Type: "ipv6"}, {Address: "192.168.4.147", Type: "ipv4"}}}, + } + if inst := fx.get(fx.id()); inst.State != provider.StateRunning || inst.IP != "192.168.4.147" { + t.Errorf("Get = %+v, want Running with 192.168.4.147", inst) + } +} + +func TestGet_stoppedIsStopped(t *testing.T) { + t.Parallel() + fx := newFixture(t, 1) + fx.addOurVM(testVMID, testName, "stopped") + if inst := fx.get(fx.id()); inst.State != provider.StateStopped || inst.IP != "" || inst.UID != testUID { + t.Errorf("Get = %+v, want Stopped with uid %s", inst, testUID) + } + if n := fx.fake.callCount("AgentNetworkInterfaces"); n != 0 { + t.Errorf("agent calls = %d, want 0 for a stopped VM", n) + } +} + +func TestGet_pausedIsStopped(t *testing.T) { + t.Parallel() + for _, qmp := range []string{"paused", "suspended", "io-error", "postmigrate", "internal-error"} { + t.Run(qmp, func(t *testing.T) { + t.Parallel() + fx := newFixture(t, 1) + fx.addOurVM(testVMID, testName, "running").qmpstatus = qmp + if inst := fx.get(fx.id()); inst.State != provider.StateStopped { + t.Errorf("Get state with qmpstatus %s = %s, want Stopped", qmp, inst.State) + } + if n := fx.fake.callCount("AgentNetworkInterfaces"); n != 0 { + t.Errorf("agent calls = %d, want 0", n) + } + }) + } +} + +// TestGet_pausedIsStoppedFromStatusCurrent uses F14's paused VM: the +// resources listing shows plain running, only status/current tells. +func TestGet_pausedIsStoppedFromStatusCurrent(t *testing.T) { + t.Parallel() + st := decodeData[vmStatus](t, "status_current_paused") + if st.Status != "running" || st.QMPStatus != "paused" { + t.Fatalf("capture status_current_paused = %+v, want running and paused", st) + } + fx := newFixture(t, 1) + vm := fx.addOurVM(testVMID, testName, st.Status) + vm.qmpstatus = st.QMPStatus + vm.ifaces = capturedIfaces(t) + if r, _ := resourceByID(t, fx.fake, testVMID); r.Status != "running" { + t.Fatalf("resources status = %q, want running as F14 saw", r.Status) + } + if inst := fx.get(fx.id()); inst.State != provider.StateStopped || inst.IP != "" { + t.Errorf("Get = %+v, want Stopped with no IP", inst) + } +} + +func TestGet_prelaunchIsProvisioning(t *testing.T) { + t.Parallel() + fx := newFixture(t, 1) + vm := fx.addOurVM(testVMID, testName, "running") + vm.qmpstatus = "prelaunch" + vm.ifaces = capturedIfaces(t) + if inst := fx.get(fx.id()); inst.State != provider.StateProvisioning || inst.IP != "" { + t.Errorf("Get = %+v, want Provisioning", inst) + } +} + +func TestGet_suspendedLockIsStopped(t *testing.T) { + t.Parallel() + fx := newFixture(t, 1) + fx.addOurVM(testVMID, testName, "stopped").lock = lockSuspended + inst := fx.get(fx.id()) + if inst.State != provider.StateStopped || inst.UID != testUID { + t.Errorf("Get = %+v, want Stopped with uid", inst) + } + if got := fx.fake.callLog(); !slices.Equal(got, []string{"ClusterResources", "VMConfig"}) { + t.Errorf("calls = %v, want resources and config only", got) + } +} + +func TestGet_unknownStatusIsTransient(t *testing.T) { + t.Parallel() + fx := newFixture(t, 1) + fx.addOurVM(testVMID, testName, "running").status = "unknown" + msg := assertGetFails(t, fx, provider.ErrTransient) + if !strings.Contains(msg, "pv1-1") { + t.Errorf("error = %s, want it to name the node", msg) + } + if n := fx.fake.callCount("VMConfig"); n != 0 { + t.Errorf("VMConfig calls = %d, want 0 while the node is unknown", n) + } +} + +func TestGet_lockedIsTransientNeverNotFound(t *testing.T) { + t.Parallel() + for _, lock := range []string{"backup", "migrate", "snapshot", "rollback", "create"} { + t.Run("resources lock "+lock, func(t *testing.T) { + t.Parallel() + fx := newFixture(t, 1) + fx.addOurVM(testVMID, testName, "running").lock = lock + msg := assertGetFails(t, fx, provider.ErrTransient) + if !strings.Contains(msg, "locked ("+lock+")") { + t.Errorf("error = %s, want it to name the lock", msg) + } + }) + } + t.Run("lock taken between the resources and config reads", func(t *testing.T) { + t.Parallel() + fx := newFixture(t, 1) + vm := fx.addOurVM(testVMID, testName, "running") + mutateOn(fx.fake, "VMConfig", func() { vm.lock = "backup" }) + assertGetFails(t, fx, provider.ErrTransient) + if n := fx.fake.callCount("VMStatus"); n != 0 { + t.Errorf("VMStatus calls = %d, want 0", n) + } + }) +} + +// TestGet_midCloneNamelessEntryIsTransient drives a real fake clone: while +// it runs the entry has no name (F8), which must not read as NotFound. +func TestGet_midCloneNamelessEntryIsTransient(t *testing.T) { + t.Parallel() + fx := newFixture(t, 1) + fx.fake.addTemplate(9001, "debian-13-template") + upid, err := fx.fake.Clone(fx.ctx, fakeNode, 9001, cloneParams{NewID: testVMID, Name: testName, Description: formatDescription(testUID, testCreatedAt)}) + if err != nil { + t.Fatalf("Clone error = %v", err) + } + if r, _ := resourceByID(t, fx.fake, testVMID); r.Name != "" || r.Lock != "clone" { + t.Fatalf("mid-clone entry = %+v, want nameless and locked", r) + } + assertGetFails(t, fx, provider.ErrTransient) + pollTask(t, fx.fake, upid) + if inst := fx.get(fx.id()); inst.State != provider.StateStopped || inst.UID != testUID { + t.Errorf("Get after the clone = %+v, want Stopped with uid", inst) + } +} + +func TestGet_vanishedBetweenCallsIsNotFound(t *testing.T) { + t.Parallel() + for _, method := range []string{"VMConfig", "VMStatus"} { + t.Run("before "+method, func(t *testing.T) { + t.Parallel() + fx := newFixture(t, 1) + fx.addOurVM(testVMID, testName, "running") + mutateOn(fx.fake, method, func() { delete(fx.fake.vms, testVMID) }) + assertGetFails(t, fx, provider.ErrNotFound) + }) + } +} + +func TestGet_unknownStateIsStopped(t *testing.T) { + t.Parallel() + fx := newFixture(t, 1) + vm := fx.addOurVM(testVMID, testName, "running") + vm.status, vm.qmpstatus = "hibernating-v2", "hibernating-v2" + if inst := fx.get(fx.id()); inst.State != provider.StateStopped { + t.Errorf("Get state = %s, want Stopped for an unrecognised status", inst.State) + } +} + +func TestGet_readsUIDAndCreatedAtFromBackend(t *testing.T) { + t.Parallel() + fx := newFixture(t, 1) + created := time.Date(2026, 9, 1, 8, 30, 0, 0, time.UTC) + fx.addOurVM(testVMID, testName, "stopped").config["description"] = formatDescription("uid-from-backend", created) + inst := fx.get(fx.id()) + if inst.UID != "uid-from-backend" || !inst.CreatedAt.Equal(created) { + t.Errorf("Get = uid %q createdAt %v, want uid-from-backend and %v", inst.UID, inst.CreatedAt, created) + } +} + +// TestGet_unmanagedDescriptionHasEmptyUID: a mark without a uid line is +// still ours, with an empty UID, which callers must tolerate. +func TestGet_unmanagedDescriptionHasEmptyUID(t *testing.T) { + t.Parallel() + fx := newFixture(t, 1) + fx.addOurVM(testVMID, testName, "stopped").config["description"] = "proxy-operator-managed=true\r\nadmin note\r\n" + inst := fx.get(fx.id()) + if inst.UID != "" || !inst.CreatedAt.IsZero() || inst.State != provider.StateStopped { + t.Errorf("Get = %+v, want empty uid, zero createdAt, Stopped", inst) + } +} + +func TestGet_malformedIDIsPermanent(t *testing.T) { + t.Parallel() + for _, id := range []string{"", "vms/150042", "vms/0150042/" + testName, "vm/150042/" + testName, "vms/abc/" + testName, "vms/150042/a/b"} { + t.Run(id, func(t *testing.T) { + t.Parallel() + fx := newFixture(t, 1) + _, err := fx.p.Get(fx.ctx, id) + assertProviderError(t, err, "get", provider.ErrPermanent) + if calls := fx.fake.callLog(); len(calls) != 0 { + t.Errorf("calls = %v, want none", calls) + } + }) + } +} + +func TestGet_resourcesFailureIsTransient(t *testing.T) { + t.Parallel() + tests := []struct { + name string + err error + }{ + {name: "pveproxy 595", err: fakeErr(595, "Connection refused")}, + {name: "other 500", err: fakeErr(500, "fake: cluster filesystem not ready")}, + {name: "network", err: errors.New("dial tcp 192.168.0.8:8006: connect: no route to host")}, + } + for _, tc := range tests { + t.Run(tc.name, func(t *testing.T) { + t.Parallel() + fx := newFixture(t, 1) + fx.addOurVM(testVMID, testName, "running") + fx.fake.failWith("ClusterResources", tc.err) + assertGetFails(t, fx, provider.ErrTransient) + }) + } +} diff --git a/internal/provider/proxmox/list.go b/internal/provider/proxmox/list.go new file mode 100644 index 0000000..4bb634a --- /dev/null +++ b/internal/provider/proxmox/list.go @@ -0,0 +1,81 @@ +package proxmox + +import ( + "context" + "strings" + + "gitea.home.hrajfrisbee.cz/kacerr/egress-proxies-operator/internal/provider" +) + +// proxyTag is the tag Create sets for the Proxmox UI; ListByTag uses it, +// with the name prefix provider.NameFromUID gives, only to pick candidates. +// The mark decides. +const ( + proxyTag = "proxy_operator" + proxyNamePrefix = "proxy-" +) + +// ListByTag is spec entry 12: one resources call covers every node, then +// one config read per candidate for the mark. GC reads only ID, UID and +// CreatedAt (contract §2.4), so no status or agent call is made. The +// resources call failing fails the list; one unreadable candidate is +// skipped, unless ctx ended, which fails the list rather than pass off a +// partial view as complete. +func (p *Provider) ListByTag(ctx context.Context) ([]provider.Instance, error) { + log := p.logger(ctx) + rs, err := p.api.ClusterResources(ctx) + if err != nil { + return nil, p.wrapErr("list", "", err) + } + var ( + out []provider.Instance + candidates, skipped int + ) + for _, r := range rs { + if !isCandidate(r) { + continue + } + candidates++ + vlog := log.WithValues("vmid", r.VMID, "name", r.Name, "node", r.Node) + cfg, err := p.api.VMConfig(ctx, r.Node, r.VMID) + if err != nil { + if ctx.Err() != nil { + return nil, p.wrapErr("list", "", err) + } + skipped++ + vlog.V(1).Info("skipping unreadable proxmox vm", "error", err.Error()) + continue + } + mark := parseDescription(cfg["description"]) + if !mark.Managed { + continue + } + inst := provider.Instance{ID: formatProviderID(r.VMID, r.Name), UID: mark.UID, CreatedAt: mark.CreatedAt} + vlog.V(2).Info("proxmox listed instance", "id", inst.ID, "uid", inst.UID, "createdAt", inst.CreatedAt) + out = append(out, inst) + } + log.V(1).Info("proxmox vms listed", "vms", len(rs), "candidates", candidates, "managed", len(out), "skipped", skipped) + return out, nil +} + +// isCandidate picks the resource entries worth a config read: QEMU VMs, not +// templates, named, and either tagged or named like a proxy. A nameless +// entry is a clone in progress, whose Create is still running. +func isCandidate(r vmResource) bool { + if r.Type != "qemu" || r.Template != 0 || r.Name == "" { + return false + } + return strings.HasPrefix(r.Name, proxyNamePrefix) || hasTag(r.Tags, proxyTag) +} + +// hasTag reports whether tags, as Proxmox lists them (";"-separated, though +// "," and spaces are accepted on input), contains tag. Proxmox folds tags to +// lower case. +func hasTag(tags, tag string) bool { + for _, t := range strings.FieldsFunc(tags, func(r rune) bool { return r == ';' || r == ',' || r == ' ' }) { + if strings.EqualFold(t, tag) { + return true + } + } + return false +} diff --git a/internal/provider/proxmox/list_test.go b/internal/provider/proxmox/list_test.go new file mode 100644 index 0000000..422d979 --- /dev/null +++ b/internal/provider/proxmox/list_test.go @@ -0,0 +1,227 @@ +package proxmox + +import ( + "context" + "errors" + "slices" + "strings" + "testing" + + "gitea.home.hrajfrisbee.cz/kacerr/egress-proxies-operator/internal/provider" +) + +func listIDs(list []provider.Instance) []string { + ids := make([]string, len(list)) + for i, inst := range list { + ids[i] = inst.ID + } + return ids +} + +func (fx *fixture) list() []provider.Instance { + fx.t.Helper() + list, err := fx.p.ListByTag(fx.ctx) + if err != nil { + fx.t.Fatalf("ListByTag() error = %v", err) + } + return list +} + +func TestListByTag_excludesUnmarkedVMs(t *testing.T) { + t.Parallel() + fx := newFixture(t, 1) + fx.addOurVM(testVMID, testName, "running") + fx.fake.addVM(150100, "proxy-nomark0000000000", "running") + fx.fake.addVM(150101, "proxy-managedfalse0000", "stopped").config["description"] = "proxy-operator-managed=false\nproxy-operator-uid=x" + fx.fake.addVM(150102, "tagged-by-hand", "running").config["tags"] = proxyTag + list := fx.list() + if got := listIDs(list); !slices.Equal(got, []string{fx.id()}) { + t.Fatalf("ListByTag = %v, want only %s", got, fx.id()) + } + if list[0].UID != testUID || !list[0].CreatedAt.Equal(testCreatedAt) || list[0].State != "" || list[0].IP != "" { + t.Errorf("listed = %+v, want uid and createdAt only", list[0]) + } + lines := fx.logs.matching(`"msg"="proxmox vms listed"`) + if len(lines) != 1 || !strings.Contains(lines[0], `"candidates"=4`) || !strings.Contains(lines[0], `"managed"=1`) || !strings.Contains(lines[0], `"skipped"=0`) { + t.Errorf("listed line = %v, want candidates 4, managed 1, skipped 0", lines) + } +} + +func TestListByTag_emptyUIDStillListed(t *testing.T) { + t.Parallel() + fx := newFixture(t, 1) + fx.addOurVM(testVMID, testName, "stopped").config["description"] = "proxy-operator-managed=true" + list := fx.list() + if len(list) != 1 || list[0].UID != "" || list[0].ID != fx.id() { + t.Errorf("ListByTag = %+v, want the VM listed with an empty UID", list) + } +} + +func TestListByTag_candidatesByTagOrNamePrefix(t *testing.T) { + t.Parallel() + fx := newFixture(t, 1) + // Named like a proxy, no tag: a candidate. + fx.addOurVM(150001, "proxy-notag00000000000", "running").config["tags"] = "" + // Tagged, renamed by an admin: a candidate. Tags as Proxmox lists them. + fx.addOurVM(150002, "renamed", "running").config["tags"] = "web;proxy_operator" + // Marked but neither tagged nor named like a proxy: never read. + fx.addOurVM(150003, "unrelated", "running").config["tags"] = "web" + got := listIDs(fx.list()) + want := []string{formatProviderID(150001, "proxy-notag00000000000"), formatProviderID(150002, "renamed")} + if !slices.Equal(got, want) { + t.Errorf("ListByTag = %v, want %v", got, want) + } + if n := fx.fake.callCount("VMConfig"); n != 2 { + t.Errorf("VMConfig calls = %d, want 2, one per candidate", n) + } +} + +func TestHasTag_table(t *testing.T) { + t.Parallel() + tests := []struct { + tags string + want bool + }{ + {tags: "proxy_operator", want: true}, + {tags: "a;proxy_operator;b", want: true}, + {tags: "a,proxy_operator", want: true}, + {tags: "a proxy_operator", want: true}, + {tags: "Proxy_Operator", want: true}, + {tags: ""}, + {tags: "proxy_operator2"}, + {tags: "proxy-operator"}, + } + for _, tc := range tests { + if got := hasTag(tc.tags, proxyTag); got != tc.want { + t.Errorf("hasTag(%q) = %v, want %v", tc.tags, got, tc.want) + } + } +} + +func TestListByTag_skipsLXCAndTemplates(t *testing.T) { + t.Parallel() + fx := newFixture(t, 1) + fx.addOurVM(150001, "proxy-container0000000", "running").kind = "lxc" + fx.addOurVM(150002, "proxy-template00000000", "stopped").template = true + if list := fx.list(); len(list) != 0 { + t.Errorf("ListByTag = %+v, want nothing", list) + } + if n := fx.fake.callCount("VMConfig"); n != 0 { + t.Errorf("VMConfig calls = %d, want 0", n) + } +} + +func TestListByTag_midCloneEntryIsNotACandidate(t *testing.T) { + t.Parallel() + fx := newFixture(t, 1) + fx.fake.addTemplate(9001, "debian-13-template") + if _, err := fx.fake.Clone(fx.ctx, fakeNode, 9001, cloneParams{NewID: testVMID, Name: testName, Description: formatDescription(testUID, testCreatedAt)}); err != nil { + t.Fatalf("Clone error = %v", err) + } + if list := fx.list(); len(list) != 0 { + t.Errorf("ListByTag = %+v, want nothing while the clone runs", list) + } +} + +func TestListByTag_resourcesFailureFailsWhole(t *testing.T) { + t.Parallel() + fx := newFixture(t, 1) + fx.addOurVM(testVMID, testName, "running") + fx.fake.failWith("ClusterResources", fakeErr(595, "Connection refused")) + list, err := fx.p.ListByTag(fx.ctx) + if list != nil { + t.Errorf("ListByTag = %+v, want nil", list) + } + assertProviderError(t, err, "list", provider.ErrTransient) +} + +func TestListByTag_unreadableVMSkipped(t *testing.T) { + t.Parallel() + fx := newFixture(t, 1) + fx.addOurVM(150001, "proxy-destroyedmidswee", "running") + fx.addOurVM(testVMID, testName, "running") + // The first candidate is destroyed between the listing and its config + // read. + mutateOn(fx.fake, "VMConfig", func() { delete(fx.fake.vms, 150001) }) + if got := listIDs(fx.list()); !slices.Equal(got, []string{fx.id()}) { + t.Errorf("ListByTag = %v, want only %s", got, fx.id()) + } + lines := fx.logs.matching(`"msg"="skipping unreadable proxmox vm"`) + if len(lines) != 1 || !strings.Contains(lines[0], `"vmid"=150001`) || !strings.Contains(lines[0], "does not exist") { + t.Errorf("skip lines = %v, want one naming 150001 and the error", lines) + } + if lines := fx.logs.matching(`"skipped"=1`); len(lines) != 1 { + t.Errorf("listed line with skipped 1 = %v", lines) + } +} + +func TestListByTag_unreadableVMWithEndedContextFails(t *testing.T) { + t.Parallel() + fx := newFixture(t, 1) + fx.addOurVM(testVMID, testName, "running") + ctx, cancel := context.WithCancel(fx.ctx) + defer cancel() + fx.fake.hookOn("VMConfig", func() error { + cancel() + return context.Canceled + }) + list, err := fx.p.ListByTag(ctx) + if list != nil { + t.Errorf("ListByTag = %+v, want nil", list) + } + assertProviderError(t, err, "list", provider.ErrTransient) + if !errors.Is(err, context.Canceled) { + t.Errorf("error = %v, want it to wrap context.Canceled", err) + } +} + +func TestListByTag_unknownStatusStillAttempted(t *testing.T) { + t.Parallel() + fx := newFixture(t, 1) + fx.addOurVM(testVMID, testName, "running").status = "unknown" + if got := listIDs(fx.list()); !slices.Equal(got, []string{fx.id()}) { + t.Errorf("ListByTag = %v, want %s despite status unknown", got, fx.id()) + } +} + +func TestListByTag_doesNotReadStatusOrAgent(t *testing.T) { + t.Parallel() + fx := newFixture(t, 1) + fx.addOurVM(testVMID, testName, "running").ifaces = capturedIfaces(t) + fx.addOurVM(150001, "proxy-second0000000000", "stopped") + fx.list() + want := []string{"ClusterResources", "VMConfig", "VMConfig"} + if got := fx.fake.callLog(); !slices.Equal(got, want) { + t.Errorf("calls = %v, want %v", got, want) + } +} + +func TestListByTag_idsAcceptedByParse(t *testing.T) { + t.Parallel() + fx := newFixture(t, 1) + fx.addOurVM(testVMID, testName, "running") + fx.addOurVM(999999999, "proxy-topofrange000000", "stopped") + fx.addOurVM(100, "proxy-bottomofrange000", "stopped") + list := fx.list() + if len(list) != 3 { + t.Fatalf("ListByTag = %+v, want 3", list) + } + for _, inst := range list { + vmid, name, err := parseProviderID(inst.ID) + if err != nil || formatProviderID(vmid, name) != inst.ID { + t.Errorf("parseProviderID(%q) = %d, %q, %v; want a round trip", inst.ID, vmid, name, err) + } + } +} + +func TestListByTag_quietAtV0(t *testing.T) { + t.Parallel() + fx := newFixture(t, 0) + fx.addOurVM(testVMID, testName, "running") + fx.addOurVM(150001, "proxy-destroyedmidswee", "running") + mutateOn(fx.fake, "VMConfig", func() { delete(fx.fake.vms, 150001) }) + fx.list() + if lines := fx.logs.all(); len(lines) != 0 { + t.Errorf("V(0) lines = %v, want none", lines) + } +} diff --git a/internal/provider/proxmox/provider_test.go b/internal/provider/proxmox/provider_test.go new file mode 100644 index 0000000..3e9dc4b --- /dev/null +++ b/internal/provider/proxmox/provider_test.go @@ -0,0 +1,103 @@ +package proxmox + +import ( + "context" + "errors" + "testing" + "time" + + "gitea.home.hrajfrisbee.cz/kacerr/egress-proxies-operator/internal/provider" +) + +const ( + testProvider = "proxmox-home" + testUID = "0b9c6f3e-2d4a-4e1b-9c7f-5a8d3e2f1b60" + testVMID = 150042 +) + +var ( + testName = provider.NameFromUID(testUID) + testCreatedAt = time.Date(2026, 9, 23, 10, 0, 0, 0, time.UTC) +) + +// fixture is a Provider over fakeAPI, logging into a capture, with every +// budget shrunk so waits take milliseconds. +type fixture struct { + t *testing.T + p *Provider + fake *fakeAPI + logs *logCapture + ctx context.Context +} + +func newFixture(t *testing.T, verbosity int) *fixture { + t.Helper() + fx := &fixture{t: t, fake: newFakeAPI(), logs: &logCapture{}} + fx.p = newProvider(testProvider, testConfig(), fx.fake) + fx.p.budgets = testBudgets() + fx.ctx = fx.logs.context(verbosity) + return fx +} + +func testBudgets() budgets { + return budgets{ + clone: time.Second, + upload: time.Second, + resize: time.Second, + start: time.Second, + stop: time.Second, + destroy: time.Second, + seedDelete: time.Second, + running: time.Second, + poll: time.Millisecond, + } +} + +// addOurVM places a VM carrying the mark for testUID and the tag, as +// Create leaves it. +func (fx *fixture) addOurVM(vmid int, name, status string) *fakeVM { + fx.t.Helper() + vm := fx.fake.addVM(vmid, name, status) + vm.config["description"] = formatDescription(testUID, testCreatedAt) + vm.config["tags"] = proxyTag + return vm +} + +// id is the providerID of the VM addOurVM(testVMID, testName, …) places. +func (fx *fixture) id() string { return formatProviderID(testVMID, testName) } + +func (fx *fixture) get(id string) *provider.Instance { + fx.t.Helper() + inst, err := fx.p.Get(fx.ctx, id) + if err != nil { + fx.t.Fatalf("Get(%s) error = %v", id, err) + } + return inst +} + +// assertProviderError checks that err is a *provider.Error from this +// provider for op, in class and in no other class. +func assertProviderError(t *testing.T, err error, op string, class error) { + t.Helper() + if err == nil { + t.Fatalf("error = nil, want op %s class %v", op, class) + } + var pe *provider.Error + if !errors.As(err, &pe) { + t.Fatalf("error = %T (%v), want *provider.Error", err, err) + } + if pe.Provider != testProvider { + t.Errorf("error.Provider = %q, want %q", pe.Provider, testProvider) + } + if pe.Op != op { + t.Errorf("error.Op = %q, want %q", pe.Op, op) + } + if !errors.Is(err, class) { + t.Errorf("error = %v, want class %v", err, class) + } + for _, other := range []error{provider.ErrNotFound, provider.ErrQuotaExceeded, provider.ErrPermanent, provider.ErrTransient} { + if other != class && errors.Is(err, other) { + t.Errorf("error = %v, also matches %v", err, other) + } + } +} diff --git a/internal/provider/proxmox/proxmox.go b/internal/provider/proxmox/proxmox.go index 66fe882..1be34e8 100644 --- a/internal/provider/proxmox/proxmox.go +++ b/internal/provider/proxmox/proxmox.go @@ -173,22 +173,14 @@ func (p *Provider) lockName(ctx context.Context, name string) (func(), error) { return release, nil } -// The four contract methods are stubs until the phases that implement them -// (Get and ListByTag in Phase 5, Create in 6, Delete in 7). Each answers -// Transient, so a Proxy on this provider retries rather than latching. +// Create and Delete are stubs until the phases that implement them (Create +// in 6, Delete in 7). Each answers Transient, so a Proxy on this provider +// retries rather than latching. func (p *Provider) Create(_ context.Context, req provider.CreateRequest) (string, error) { return "", p.wrapErr("create", "", transientf("proxmox Create is not implemented yet (request %s)", req.Name)) } -func (p *Provider) Get(_ context.Context, id string) (*provider.Instance, error) { - return nil, p.wrapErr("get", id, transientf("proxmox Get is not implemented yet")) -} - func (p *Provider) Delete(_ context.Context, id string) error { return p.wrapErr("delete", id, transientf("proxmox Delete is not implemented yet")) } - -func (p *Provider) ListByTag(_ context.Context) ([]provider.Instance, error) { - return nil, p.wrapErr("list", "", transientf("proxmox ListByTag is not implemented yet")) -} diff --git a/internal/provider/proxmox/proxmox_test.go b/internal/provider/proxmox/proxmox_test.go index 2add898..42d540e 100644 --- a/internal/provider/proxmox/proxmox_test.go +++ b/internal/provider/proxmox/proxmox_test.go @@ -216,16 +216,14 @@ func TestWrapErr_carriesProviderNameOpAndID(t *testing.T) { } // TestStubs_answerTransient keeps a Proxy on this provider retrying, not -// latched, until the methods are implemented in later phases. +// latched, until Create and Delete are implemented in later phases. func TestStubs_answerTransient(t *testing.T) { t.Parallel() ctx := context.Background() p := newProvider("proxmox-home", testConfig(), nil) _, createErr := p.Create(ctx, provider.CreateRequest{Name: "proxy-x"}) - _, getErr := p.Get(ctx, "vms/150042/proxy-x") deleteErr := p.Delete(ctx, "vms/150042/proxy-x") - _, listErr := p.ListByTag(ctx) - for op, err := range map[string]error{"create": createErr, "get": getErr, "delete": deleteErr, "list": listErr} { + for op, err := range map[string]error{"create": createErr, "delete": deleteErr} { if !errors.Is(err, provider.ErrTransient) { t.Errorf("%s error = %v, want ErrTransient", op, err) } diff --git a/internal/provider/proxmox/task.go b/internal/provider/proxmox/task.go new file mode 100644 index 0000000..3ebe972 --- /dev/null +++ b/internal/provider/proxmox/task.go @@ -0,0 +1,109 @@ +package proxmox + +import ( + "context" + "fmt" + "regexp" + "time" + + "github.com/go-logr/logr" + + "gitea.home.hrajfrisbee.cz/kacerr/egress-proxies-operator/internal/provider" +) + +// upidPattern is Proxmox's task id: UPID::::: +// ::: with hex pid, pstart and starttime. +var upidPattern = regexp.MustCompile(`^UPID:([a-zA-Z0-9]([a-zA-Z0-9-]*[a-zA-Z0-9])?):[0-9A-Fa-f]{8}:[0-9A-Fa-f]{8,9}:[0-9A-Fa-f]{8}:[^:\s]+:[^:\s]*:[^:\s]+:$`) + +// parseUPID returns the node a task runs on, which is where its status is +// read (spec entry 10). A malformed UPID is Permanent: Proxmox never sends +// one, so retrying cannot help. +func parseUPID(upid string) (string, error) { + m := upidPattern.FindStringSubmatch(upid) + if m == nil { + return "", permanentf("malformed task id %q", upid) + } + return m[1], nil +} + +// taskSucceeded reads a stopped task's exitstatus the way Proxmox does: +// "OK", or "WARNINGS: " for a task that finished but logged warnings. +func taskSucceeded(exit string) bool { + return exit == "OK" || taskWarnings.MatchString(exit) +} + +var taskWarnings = regexp.MustCompile(`^WARNINGS: [0-9]+$`) + +// pollUntil calls check every interval until it reports done or fails. Once +// budget has passed or ctx has ended it gives up with ErrTransient naming +// what it waited for; the backend keeps working, and the next attempt picks +// the result up. +func pollUntil(ctx context.Context, interval, budget time.Duration, what string, check func() (bool, error)) error { + deadline := time.NewTimer(budget) + defer deadline.Stop() + tick := time.NewTicker(interval) + defer tick.Stop() + for { + done, err := check() + if err != nil || done { + return err + } + select { + case <-ctx.Done(): + return fmt.Errorf("%w: waiting for %s: %w", provider.ErrTransient, what, ctx.Err()) + case <-deadline.C: + return transientf("waiting for %s: not done after %v", what, budget) + case <-tick.C: + } + } +} + +// waitTask polls a task on the node its UPID names until it stops. A +// failed task is ErrTransient carrying its exitstatus, and logged at V(1) +// (spec entries 10 and 20). +func (p *Provider) waitTask(ctx context.Context, log logr.Logger, upid string, budget time.Duration) error { + node, err := parseUPID(upid) + if err != nil { + return err + } + var exit string + err = pollUntil(ctx, p.budgets.poll, budget, "task "+upid, func() (bool, error) { + s, err := p.api.TaskStatus(ctx, node, upid) + if err != nil { + return false, fmt.Errorf("reading task %s: %w", upid, err) + } + exit = s.ExitStatus + return s.Status == "stopped", nil + }) + if err != nil { + return err + } + if !taskSucceeded(exit) { + log.V(1).Info("proxmox task failed", "upid", upid, "exitstatus", exit) + return transientf("proxmox task %s failed: %s", upid, exit) + } + if exit != "OK" { + log.V(1).Info("proxmox task finished with warnings", "upid", upid, "exitstatus", exit) + } + return nil +} + +// waitUnlocked polls a VM's config until no task holds its lock and returns +// that config, for Create's ownership check (spec entry 2). A VM that +// disappears meanwhile, as a failed clone does, returns the does-not-exist +// error for the caller to judge. +func (p *Provider) waitUnlocked(ctx context.Context, node string, vmid int, budget time.Duration) (vmConfig, error) { + var cfg vmConfig + err := pollUntil(ctx, p.budgets.poll, budget, fmt.Sprintf("the lock on VM %d", vmid), func() (bool, error) { + c, err := p.api.VMConfig(ctx, node, vmid) + if err != nil { + return false, fmt.Errorf("reading config of VM %d: %w", vmid, err) + } + cfg = c + return c["lock"] == "", nil + }) + if err != nil { + return nil, err + } + return cfg, nil +} diff --git a/internal/provider/proxmox/task_test.go b/internal/provider/proxmox/task_test.go new file mode 100644 index 0000000..f25d401 --- /dev/null +++ b/internal/provider/proxmox/task_test.go @@ -0,0 +1,233 @@ +package proxmox + +import ( + "context" + "errors" + "strings" + "testing" + "time" + + "gitea.home.hrajfrisbee.cz/kacerr/egress-proxies-operator/internal/provider" +) + +// addTask registers a fake task on node that ends with exit. +func addTask(f *fakeAPI, node, exit string) string { + f.mu.Lock() + defer f.mu.Unlock() + return f.newTask(node, "qmstart", "150042", exit, nil) +} + +func TestParseUPID_table(t *testing.T) { + t.Parallel() + tests := []struct { + name string + upid string + wantNode string + }{ + {name: "captured clone", upid: decodeData[string](t, "clone_response"), wantNode: "pv1-1"}, + {name: "captured task status", upid: decodeData[taskStatus](t, "task_status_stopped_ok").UPID, wantNode: "pv1-1"}, + {name: "captured upload", upid: decodeData[string](t, "upload_response_proxy_seeds"), wantNode: "pv1-1"}, + {name: "fake task on another node", upid: addTask(newFakeAPI(), "pv1-2", "OK"), wantNode: "pv1-2"}, + {name: "empty", upid: ""}, + {name: "prefix only", upid: "UPID:"}, + {name: "lower-case prefix", upid: "upid:pv1-1:0010F9DC:02801DA1:6AB37871:qmclone:9001:root@pam:"}, + {name: "no trailing colon", upid: "UPID:pv1-1:0010F9DC:02801DA1:6AB37871:qmclone:9001:root@pam"}, + {name: "slash in node", upid: "UPID:pv1/1:0010F9DC:02801DA1:6AB37871:qmclone:9001:root@pam:"}, + {name: "pid not hex", upid: "UPID:pv1-1:0010F9DZ:02801DA1:6AB37871:qmclone:9001:root@pam:"}, + {name: "missing user", upid: "UPID:pv1-1:0010F9DC:02801DA1:6AB37871:qmclone:9001::"}, + } + for _, tc := range tests { + t.Run(tc.name, func(t *testing.T) { + t.Parallel() + node, err := parseUPID(tc.upid) + if tc.wantNode == "" { + if !errors.Is(err, provider.ErrPermanent) { + t.Errorf("parseUPID(%q) = %q, %v; want ErrPermanent", tc.upid, node, err) + } + return + } + if err != nil || node != tc.wantNode { + t.Errorf("parseUPID(%q) = %q, %v; want %q", tc.upid, node, err, tc.wantNode) + } + }) + } +} + +func TestWaitTask_table(t *testing.T) { + t.Parallel() + failedExit := loadTaskExit(t, "task_status_stopped_error") + tests := []struct { + name string + node string + exit string + upid string // overrides the registered task's UPID + polls int + budget time.Duration + cancelled bool + statusErr error + wantClass error // nil for success + wantCalls int + wantInErr string + wantLog string + }{ + {name: "ok on the first poll", exit: "OK", wantCalls: 1}, + {name: "polls until stopped", exit: "OK", polls: 3, wantCalls: 4}, + {name: "polls the node the UPID names", node: "pv1-2", exit: "OK", wantCalls: 1}, + {name: "warnings count as success", exit: "WARNINGS: 2", wantCalls: 1, wantLog: `"msg"="proxmox task finished with warnings"`}, + { + name: "failed exitstatus is transient and logged", exit: failedExit, wantClass: provider.ErrTransient, wantCalls: 1, + wantInErr: failedExit, wantLog: `"msg"="proxmox task failed"`, + }, + {name: "budget exceeded names the UPID", exit: "OK", polls: 1 << 30, budget: 20 * time.Millisecond, wantClass: provider.ErrTransient, wantCalls: -1, wantInErr: "UPID:pv1-1:"}, + {name: "ended ctx is transient", exit: "OK", polls: 1 << 30, cancelled: true, wantClass: provider.ErrTransient, wantCalls: 1, wantInErr: context.Canceled.Error()}, + {name: "malformed UPID makes no call", upid: "UPID:bogus", wantClass: provider.ErrPermanent, wantCalls: 0}, + {name: "status read failing is returned", exit: "OK", statusErr: loadAPIError(t, "err_forbidden"), wantClass: provider.ErrPermanent, wantCalls: 1, wantInErr: "Permission check failed"}, + } + for _, tc := range tests { + t.Run(tc.name, func(t *testing.T) { + t.Parallel() + fx := newFixture(t, 1) + fx.fake.taskPolls = tc.polls + if tc.statusErr != nil { + fx.fake.failWith("TaskStatus", tc.statusErr) + } + node := tc.node + if node == "" { + node = fakeNode + } + upid := addTask(fx.fake, node, tc.exit) + if tc.upid != "" { + upid = tc.upid + } + budget := tc.budget + if budget == 0 { + budget = time.Minute + } + ctx := fx.ctx + if tc.cancelled { + var cancel context.CancelFunc + ctx, cancel = context.WithCancel(ctx) + cancel() + } + start := time.Now() + err := fx.p.waitTask(ctx, fx.p.logger(ctx), upid, budget) + if elapsed := time.Since(start); elapsed > 5*time.Second { + t.Errorf("waitTask took %v", elapsed) + } + if tc.wantClass == nil { + if err != nil { + t.Fatalf("waitTask error = %v, want nil", err) + } + } else if classify("create", err) != tc.wantClass { + t.Fatalf("waitTask error = %v, want class %v", err, tc.wantClass) + } + if tc.wantInErr != "" && !strings.Contains(err.Error(), tc.wantInErr) { + t.Errorf("waitTask error = %q, want it to contain %q", err, tc.wantInErr) + } + if got := fx.fake.callCount("TaskStatus"); tc.wantCalls >= 0 && got != tc.wantCalls { + t.Errorf("TaskStatus calls = %d, want %d", got, tc.wantCalls) + } + if tc.wantLog != "" { + lines := fx.logs.matching(tc.wantLog) + if len(lines) != 1 || !strings.Contains(lines[0], `"upid"="`+upid+`"`) || !strings.Contains(lines[0], `"exitstatus"="`+tc.exit+`"`) { + t.Errorf("log lines %q = %v, want one with upid and exitstatus", tc.wantLog, lines) + } + } + }) + } +} + +func TestWaitTask_quietAtV0(t *testing.T) { + t.Parallel() + fx := newFixture(t, 0) + err := fx.p.waitTask(fx.ctx, fx.p.logger(fx.ctx), addTask(fx.fake, fakeNode, "boom"), time.Minute) + if err == nil { + t.Fatal("waitTask on a failed task returned nil") + } + if lines := fx.logs.all(); len(lines) != 0 { + t.Errorf("V(0) lines = %v, want none", lines) + } +} + +func TestWaitUnlocked_table(t *testing.T) { + t.Parallel() + tests := []struct { + name string + lock string + // onCall runs on the n-th VMConfig call (1-based) under f.mu. + onCall func(f *fakeAPI, n int) + budget time.Duration + cancelled bool + wantErr func(error) bool + wantCalls int + }{ + {name: "unlocked answers at once", wantCalls: 1}, + { + name: "waits until the lock clears", lock: "clone", wantCalls: 3, + onCall: func(f *fakeAPI, n int) { + if n == 3 { + f.vms[testVMID].lock = "" + } + }, + }, + { + name: "lock that never clears is transient", lock: "clone", budget: 20 * time.Millisecond, wantCalls: -1, + wantErr: func(err error) bool { + return errors.Is(err, provider.ErrTransient) && strings.Contains(err.Error(), "the lock on VM 150042") + }, + }, + { + name: "VM removed by a failed clone gives does not exist", lock: "clone", wantCalls: 2, + onCall: func(f *fakeAPI, n int) { + if n == 2 { + delete(f.vms, testVMID) + } + }, + wantErr: isNotExist, + }, + { + name: "ended ctx is transient", lock: "clone", cancelled: true, wantCalls: 1, + wantErr: func(err error) bool { + return errors.Is(err, provider.ErrTransient) && errors.Is(err, context.Canceled) + }, + }, + } + for _, tc := range tests { + t.Run(tc.name, func(t *testing.T) { + t.Parallel() + fx := newFixture(t, 1) + fx.addOurVM(testVMID, testName, "stopped").lock = tc.lock + var n int + fx.fake.hookOn("VMConfig", func() error { + fx.fake.mu.Lock() + defer fx.fake.mu.Unlock() + n++ + if tc.onCall != nil { + tc.onCall(fx.fake, n) + } + return nil + }) + budget := tc.budget + if budget == 0 { + budget = time.Minute + } + ctx := fx.ctx + if tc.cancelled { + var cancel context.CancelFunc + ctx, cancel = context.WithCancel(ctx) + cancel() + } + cfg, err := fx.p.waitUnlocked(ctx, fakeNode, testVMID, budget) + if tc.wantErr == nil { + if err != nil || cfg["name"] != testName || cfg["lock"] != "" { + t.Errorf("waitUnlocked = %v, %v; want the unlocked config", cfg, err) + } + } else if !tc.wantErr(err) || cfg != nil { + t.Errorf("waitUnlocked = %v, %v; want no config and the expected error", cfg, err) + } + if got := fx.fake.callCount("VMConfig"); tc.wantCalls >= 0 && got != tc.wantCalls { + t.Errorf("VMConfig calls = %d, want %d", got, tc.wantCalls) + } + }) + } +} From 34f132d421f8ad9973e10120ea5fcbfbca674e27 Mon Sep 17 00:00:00 2001 From: kacerr Date: Wed, 23 Sep 2026 15:32:31 +0000 Subject: [PATCH 09/13] proxmox: Create by cloning a template, seed upload, config, start Create validates placement, then under the per-name lock adopts the VM a previous attempt left (by name, or at a taken VMID after the ownership check of spec entry 2) or full-clones the template onto the UID's deterministic VMID with the mark in the clone call. It then uploads the seed over any existing file, writes cores, memory, agent, tag, ide2 and net0 when configured, grows the boot disk when diskGiB is larger, starts, and waits for running/running. waitUnlocked no longer waits on a suspended lock: a hibernated VM is judged at once, so ours is Permanent and a foreign one is probed past. The template is checked only on the allocation path, and its shape (net0, boot disk size) is refused before any clone exists. Spec entries 2, 4 and 10 updated to match. Co-Authored-By: Claude Opus 5.5 (1M context) --- .../2026-09-18-1502-proxmox-provider.md | 134 +- docs/proxmox-provider-spec.md | 28 +- internal/provider/proxmox/create.go | 425 ++++++ internal/provider/proxmox/create_test.go | 1269 +++++++++++++++++ internal/provider/proxmox/proxmox.go | 10 +- internal/provider/proxmox/proxmox_test.go | 11 +- internal/provider/proxmox/seed.go | 22 + internal/provider/proxmox/task.go | 23 +- internal/provider/proxmox/task_test.go | 6 +- 9 files changed, 1900 insertions(+), 28 deletions(-) create mode 100644 internal/provider/proxmox/create.go create mode 100644 internal/provider/proxmox/create_test.go diff --git a/docs/plans-executions/2026-09-18-1502-proxmox-provider.md b/docs/plans-executions/2026-09-18-1502-proxmox-provider.md index d2e7308..2eac296 100644 --- a/docs/plans-executions/2026-09-18-1502-proxmox-provider.md +++ b/docs/plans-executions/2026-09-18-1502-proxmox-provider.md @@ -10,7 +10,7 @@ Pairs with [docs/plans/2026-09-18-1502-proxmox-provider.md](../plans/2026-09-18- - [x] Phase 3 — Extract `nocloud` and `keyedlock` from kvm2 - [x] Phase 4 — Seam, HTTP client, decorator, fake - [x] Phase 5 — Get, ListByTag, task helpers -- [ ] Phase 6 — Create +- [x] Phase 6 — Create - [ ] Phase 7 — Delete, Checker, CRD comment - [ ] Phase 8 — Contract and observability sweep, integration test, testing.md - [ ] Phase 9 — Deployment and documentation @@ -657,3 +657,135 @@ Worth noting: - Nothing in this phase talked to the host. Every Proxmox message a test relies on comes from `testdata/` or from fake errors that `TestFakeAPI_errorsMatchCaptures` ties to the captures. + +## Phase 6 — Create + +`create.go` holds `Create` and its steps: placement, the by-name lookup, +the template check, the VMID probe with its ownership check (`inspect`), +the clone, and `provision` (seed, config, resize, start). `configParams`, +`rewriteNet0`, `resizeTarget`, `bootDisk` and `diskBytes` are pure helpers +with their own table tests. `seed.go` gains `uploadSeed`, which uploads +over any existing file with no delete first (F12). `task.go` gains +`waitRunning`, built on `pollUntil` with `budgets.running`. It finishes +only at `running`/`running`, and a `stopped` status after the start is +Transient at once. The Create stub is gone, and `TestStubs_answerTransient` +now covers only Delete. + +The spec won over the plan wherever the Phase 6 notes said it would. The +plan's allocation loop is replaced by entry 2: a taken VMID and a clone +answering F4 both go through the ownership check before any probing, and a +locked or nameless entry is waited on with `waitUnlocked` and the clone +budget. The seed is uploaded without a delete. The resize is a task. `net0` +changes only when `bridge` or `vlanTag` is set, and `tag=` is replaced or +appended in place. + +Judgment calls, with spec entries 2, 4 and 10 updated to match: + +- **The Phase 5 open question: a `suspended` lock is not waited on.** + `waitUnlocked` now returns at once when it sees `suspended`, because a + hibernated VM is not a task and would otherwise burn the 10-minute clone + budget on every attempt. The ownership check then judges the VM the way + `Get` does. A foreign hibernated VM is probed past. Ours is Permanent + ("resume or destroy it"), following kvm2's precedent for a paused or + pmsuspended domain in Create. Transient was the alternative: it would + self-heal once an admin acts, but it would retry indefinitely on + something only an admin can fix. `TestWaitUnlocked_table` has a new row, + and `TestCreate_hibernatedVMIsNotWaitedOn` covers both cases. +- **The template is checked only when a clone is needed.** Spec entry 10 + read the template's config in step 2, before the by-name lookup. A retry + for a VM that already exists would then fail Permanent if the template + had been deleted since, which entry 4 explicitly allows. The template + check now runs after the by-name lookup misses, so a repeat Create costs + three calls: resources, config and status + (`TestCreate_adoptionDoesNotNeedTheTemplate`). +- **The template's shape is checked before the clone.** No `net0` while + `bridge` or `vlanTag` is set, or no boot disk (or none with a readable + `size=`) while `diskGiB` is set, is Permanent. Found after the clone, the + VM would already carry the bad config, so fixing the template could not + help that Proxy's next attempt. `template()` runs the same + `configParams` and `resizeTarget` over the template's config that + `provision` later runs over the clone's. +- **An empty `req.Name` is Permanent at placement.** A nameless request + would match nameless mid-clone entries in the by-name lookup. +- **Adopted VMs get one `status/current` read** rather than trusting the + listing's status, which may be stale after a lock wait. A running VM, + whatever its `qmpstatus`, returns its ID at once, as entry 10 says. `Get` + then reports prelaunch as Provisioning and paused as Stopped. +- **After the clone task, Create reads the config through + `waitUnlocked`.** That read supplies the `net0` and boot disk the later + steps need, and it turns a clone that ended but kept its lock (F15) into + a Transient lock wait rather than a refused config write. +- **An F4 whose VMID then resolves to no QEMU VM probes on.** Either the + racer's clone failed and removed it, or a container took it. +- **Errors carry no ID** (`wrapErr("create", "", …)`), since the VMID is + known only after allocation. The messages name the VMID wherever one + exists. + +Test names: every plan and entry 21 name exists except +`TestCreate_resizeOnlyWhenLarger`. It is covered by the spec's pair: +`TestCreate_resizeWaitsForTask` for the larger case and +`TestCreate_resizeSkippedWhenNotLarger` for equal, smaller and unset. Tests +beyond both lists: `TestCreate_vmThatStopsAfterStartIsTransientAtOnce`, +`TestCreate_hibernatedVMIsNotWaitedOn`, +`TestCreate_templateShapeRefusedBeforeClone`, +`TestCreate_adoptionDoesNotNeedTheTemplate`, +`TestCreate_cloneTargetAndStorage` (read from the decorator's V(2) clone +line), `TestCreate_adoptedVMStaysOnItsNode`, `TestCreate_outcomeLine`, +`TestCreate_quietAtV0`, `TestRewriteNet0_table`, `TestBootDisk_capturedClone` +and `TestDiskBytes_table`. The only new test helpers are the small ones in +`create_test.go` (`slot`, `newCreateFixture`, `req`/`create`/`createFails`, +`assertRunning`, `callsBefore`, `clearLockOnVMConfig`). Everything else +reuses the Phase 5 fixture and fake. Two tests check config-write contents +without a recording wrapper. `TestCreate_markWrittenInCloneCall` reads the +VM at the start of `UploadISO`, which is after the clone and before the +config write. `TestCreate_noBridgeOverrideLeavesNet0` changes `net0` at the +start of `SetVMConfig`, so a write that carried `net0` would put the old +value back. + +A mutation check applied 18 mutations, and each failed at least one test. +They were: no lock wait in the ownership check; taken VMIDs never +inspected; the F4 race skipping the ownership check; no cloud-init drive +check; upload or resize task not waited on; resize at equal size; a +CD-ROM counted as a boot disk; prelaunch counted as running; the suspended +lock waited on; `net0` always written; `bridge=` appended instead of +replaced; duplicate names ignored; a running VM re-provisioned; `target` +always set; a foreign UID adopted; the tag dropped; probes not capped by +the range. The last one was first **missed**. A 4-VMID range with 32 +probes wraps, re-inspects each VM eight times and still ends Transient, so +`TestCreate_vmidRangeExhaustedIsTransient` now pins five config reads. +Five mutations did not compile on the first try and were redone. Every +mutation was reverted, and the files were compared identical to backups. + +```bash +go test -race -count=2 ./internal/provider/proxmox/ # ok +grep -n 'CloudInit' internal/provider/proxmox/create.go internal/provider/proxmox/seed.go +# seed.go: the pass-through into nocloud.BuildSeedISO; create.go: len(req.CloudInit) +grep -n 'not implemented' internal/provider/proxmox/*.go | grep -v _test # Delete only +grep -n 'context.Background()' internal/provider/proxmox/*.go | grep -v _test.go # nothing +go build ./... && go vet ./... && go vet -tags=integration ./... +go test -short -count=1 ./... # all ok but the parked kvm2 flake, below +go test -count=1 -v ./internal/provider/proxmox/ | grep -c '^=== RUN' # 393 before, 519 after +bin/golangci-lint run --disable staticcheck ./internal/provider/proxmox/... +``` + +Worth noting: + +- `go test -short ./...` failed once on the parked kvm2 flake + `TestCheck_hostsCheckedConcurrently` (measured 199.98 ms against a + 200 ms floor). Three reruns passed. This phase does not touch kvm2. +- The new lint findings follow the baseline's shape. There are logcheck + hits on `create`, `allocate` and `clone` (ctx plus logger, as in kvm2), + and goconst on test strings. One more unparam than the notes list: + `(*fakeAPI).iso`'s `storage` always receives `fakeISOStorage`, because + this phase is its first caller. `addTemplate`'s hit changed from a + constant VMID to an unused result. +- The config write sets `agent=1` and `tags=proxy_operator` as values. A + template's agent options (such as `fstrim_cloned_disks=1`) and its own + tags are replaced, not merged. That is what entry 10 specifies, and the + guide should say so. +- In the fake, a cross-node clone puts the config on the target at once. + Real Proxmox may move it at the end of the task, in which case a lock + wait on the target node would see `does not exist` and the next attempt + would find the VM. This stays untested, like every cross-node path + (entry 24). +- Nothing in this phase talked to the host. diff --git a/docs/proxmox-provider-spec.md b/docs/proxmox-provider-spec.md index 84712ab..b697917 100644 --- a/docs/proxmox-provider-spec.md +++ b/docs/proxmox-provider-spec.md @@ -72,7 +72,12 @@ lands on the same number: `vmidFor(uid, attempt)` is seed step (entry 10). It is **foreign** when the name differs: move to the next attempt. The same name with no mark, or with a different non-empty `uid`, is `ErrPermanent` naming the VMID and both UIDs; the - operator never takes over a VM it did not create. + operator never takes over a VM it did not create. A `suspended` lock (a + VM hibernated to disk) is not waited on: it is not a task and never + clears by itself. Ownership is judged at once, as `Get` does (entry 9): + a foreign hibernated VM is probed past, and ours is `ErrPermanent` + (resume or destroy it), like kvm2's paused domain. Container and + template entries at a VMID are foreign without a config read. The wait in step 3 is what closes the duplicate-VM bug Phase 0 found: after a crash mid-clone, the retry's by-name lookup misses the nameless VM, but @@ -113,7 +118,12 @@ template may be edited, replaced or deleted without affecting running VMs. Before cloning, Create reads the template's config and refuses it with `ErrPermanent` when it is not a template (`template` is not 1) or when any disk slot holds a Proxmox cloud-init drive (a value containing `cloudinit`), -which would compete with the provider's seed (entry 5). The template must +which would compete with the provider's seed (entry 5). Because the clone +copies the template's config, what the config or resize step would refuse +on the clone is refused on the template too, before any VM exists: no +`net0` while `bridge` or `vlanTag` is set, and no boot disk, or one +without a readable `size=`, while `diskGiB` is set. A Permanent error +after the clone would leave a VM that fixing the template cannot help. The template must boot from its first disk, run cloud-init with the NoCloud datasource, and either ship `qemu-guest-agent` or be able to install it (entry 7). @@ -320,11 +330,15 @@ waits for the clone task first. Steps, each idempotent, in order, under the per-name lock (entry 19): 1. Validate placement (entry 3). No backend call yet. -2. `GET /cluster/resources`. Find the template's node; a missing template - is `ErrPermanent`. Read the template's config and refuse it as entry 4 - says. -3. Find or allocate the VMID (entry 2). An adopted VM that is running - returns its ID at once. An adopted VM that is stopped continues at step 5. +2. `GET /cluster/resources` once, then look the name up (entry 2). +3. When the name finds nothing: find the template's node (a missing + template is `ErrPermanent`), read the template's config and refuse it as + entry 4 says, then allocate the VMID (entry 2). The template is checked + only here, because a VM that already exists no longer depends on it + (entry 4): a template deleted after its clone must not fail that + clone's adoption. An adopted VM, found by name or by the probe, gets one + `status/current` read. Running, it returns its ID at once. Stopped, it + continues at step 5. 4. Clone and wait (clone budget). 5. Build the seed and upload it to the VM's node, overwriting; wait (upload budget). diff --git a/internal/provider/proxmox/create.go b/internal/provider/proxmox/create.go new file mode 100644 index 0000000..df4c7ae --- /dev/null +++ b/internal/provider/proxmox/create.go @@ -0,0 +1,425 @@ +package proxmox + +import ( + "context" + "fmt" + "maps" + "regexp" + "slices" + "strconv" + "strings" + + "github.com/go-logr/logr" + + "gitea.home.hrajfrisbee.cz/kacerr/egress-proxies-operator/internal/provider" +) + +// maxProbes bounds the VMID probe of spec entry 2. +const maxProbes = 32 + +// diskKey is a Proxmox disk slot: the keys a boot disk, a CD-ROM or a +// cloud-init drive can sit on. +var diskKey = regexp.MustCompile(`^(ide|sata|scsi|virtio)[0-9]+$`) + +// ownedVM is a VM Create has cloned or adopted: where it lives and its +// config as last read, which the config and resize steps start from. +type ownedVM struct { + node string + vmid int + cfg vmConfig +} + +// Create is spec entry 10: validate placement, then, under the per-name +// lock, adopt the VM a previous attempt left or clone a new one, and carry +// it through seed, config, resize and start until it runs. +func (p *Provider) Create(ctx context.Context, req provider.CreateRequest) (string, error) { + node, mt, tmpl, err := p.placement(req) + if err != nil { + return "", p.wrapErr("create", "", err) + } + log := p.logger(ctx).WithValues("name", req.Name) + log.V(2).Info("proxmox create resolved", + "zone", node, "machineType", req.Placement.MachineType, "cores", mt.Cores, "memoryMiB", mt.MemoryMiB, + "diskGiB", mt.DiskGiB, "image", req.Placement.Image, "templateVMID", tmpl.VMID, "storage", p.cfg.Storage, + "isoStorage", p.cfg.ISOStorage, "bridge", p.cfg.Bridge, "vlanTag", p.cfg.VLANTag, "cloudInitBytes", len(req.CloudInit)) + + unlock, err := p.lockName(ctx, req.Name) + if err != nil { + return "", p.wrapErr("create", "", err) + } + defer unlock() + + vm, err := p.create(ctx, log, node, mt, tmpl, req) + if err != nil { + return "", p.wrapErr("create", "", err) + } + log.V(1).Info("proxmox vm ensured running", "vmid", vm.vmid, "node", vm.node) + return formatProviderID(vm.vmid, req.Name), nil +} + +// placement is step 1: every catalogue key is checked before any backend +// call, and an unknown one is Permanent naming the field and the value +// (spec entry 3). Region is ignored. +func (p *Provider) placement(req provider.CreateRequest) (string, provider.ProxmoxMachineType, provider.ProxmoxTemplate, error) { + var ( + mt provider.ProxmoxMachineType + tmpl provider.ProxmoxTemplate + ) + if req.Name == "" { + return "", mt, tmpl, permanentf("the request has no instance name") + } + node, err := p.node(req.Placement.Zone) + if err != nil { + return "", mt, tmpl, err + } + mt, ok := p.cfg.MachineTypes[req.Placement.MachineType] + if !ok { + return "", mt, tmpl, permanentf("placement.machineType %q is not in proxmox.machineTypes", req.Placement.MachineType) + } + tmpl, ok = p.cfg.Templates[req.Placement.Image] + if !ok { + return "", mt, tmpl, permanentf("placement.image %q is not in proxmox.templates", req.Placement.Image) + } + return node, mt, tmpl, nil +} + +// create is steps 2 to 9. The template is checked only when a clone is +// needed: a VM a previous attempt created no longer depends on it (spec +// entry 4), so a template removed since must not fail its adoption. +func (p *Provider) create(ctx context.Context, log logr.Logger, node string, mt provider.ProxmoxMachineType, tmpl provider.ProxmoxTemplate, req provider.CreateRequest) (ownedVM, error) { + rs, err := p.api.ClusterResources(ctx) + if err != nil { + return ownedVM{}, fmt.Errorf("listing cluster resources: %w", err) + } + vm, adopted, err := p.adoptByName(ctx, rs, req) + if err != nil { + return ownedVM{}, err + } + if !adopted { + src, err := p.template(ctx, rs, req.Placement.Image, tmpl.VMID, mt) + if err != nil { + return ownedVM{}, err + } + if vm, adopted, err = p.allocate(ctx, log, rs, node, src, req); err != nil { + return ownedVM{}, err + } + } + log = log.WithValues("vmid", vm.vmid, "node", vm.node) + if adopted { + st, err := p.api.VMStatus(ctx, vm.node, vm.vmid) + if err != nil { + return ownedVM{}, fmt.Errorf("reading status of VM %d: %w", vm.vmid, err) + } + log.V(1).Info("proxmox vm adopted", "status", st.Status, "qmpstatus", st.QMPStatus) + if st.Status == "running" { + return vm, nil + } + } + return vm, p.provision(ctx, log, vm, mt, req) +} + +// adoptByName is the by-name half of spec entry 2: two VMs with the name +// are for an admin to sort out, one goes through the ownership check. +func (p *Provider) adoptByName(ctx context.Context, rs []vmResource, req provider.CreateRequest) (ownedVM, bool, error) { + var found []vmResource + for _, r := range rs { + if r.Type == "qemu" && r.Template == 0 && r.Name == req.Name { + found = append(found, r) + } + } + switch len(found) { + case 0: + return ownedVM{}, false, nil + case 1: + return p.inspect(ctx, found[0], req) + } + vmids := make([]string, len(found)) + for i, r := range found { + vmids[i] = strconv.Itoa(r.VMID) + } + return ownedVM{}, false, permanentf("VMs %s are all named %s; remove all but one", strings.Join(vmids, ", "), req.Name) +} + +// inspect is the ownership check of spec entry 2 on a VMID that is taken. +// It waits out a task's lock (a clone in progress is also nameless), then +// reads the name and the mark: another name is someone else's VM, and our +// name without our mark is never taken over. +func (p *Provider) inspect(ctx context.Context, r vmResource, req provider.CreateRequest) (ownedVM, bool, error) { + cfg, err := p.waitUnlocked(ctx, r.Node, r.VMID, p.budgets.clone) + if err != nil { + return ownedVM{}, false, err + } + if cfg["name"] != req.Name { + return ownedVM{}, false, nil + } + mark := parseDescription(cfg["description"]) + switch { + case !mark.Managed: + return ownedVM{}, false, permanentf("VM %d is named %s but carries no proxy-operator mark; it was not created by this operator", r.VMID, req.Name) + case mark.UID != "" && mark.UID != req.UID: + return ownedVM{}, false, permanentf("VM %d named %s belongs to proxy uid %s, not %s", r.VMID, req.Name, mark.UID, req.UID) + case cfg["lock"] == lockSuspended: + return ownedVM{}, false, permanentf("VM %d named %s is hibernated (lock suspended); resume or destroy it in Proxmox", r.VMID, req.Name) + } + return ownedVM{node: r.Node, vmid: r.VMID, cfg: cfg}, true, nil +} + +// template finds the clone source and refuses one that is not a template +// or brings a Proxmox cloud-init drive, which would compete with the seed +// (spec entry 4). The clone copies the template's config, so what the +// config or resize step would refuse on the clone is refused here, while no +// VM exists that a fixed template could no longer help. +func (p *Provider) template(ctx context.Context, rs []vmResource, image string, vmid int, mt provider.ProxmoxMachineType) (vmResource, error) { + i := slices.IndexFunc(rs, func(r vmResource) bool { return r.VMID == vmid && r.Type == "qemu" }) + if i < 0 { + return vmResource{}, permanentf("placement.image %q: template VM %d does not exist", image, vmid) + } + src := rs[i] + cfg, err := p.api.VMConfig(ctx, src.Node, vmid) + if err != nil { + return vmResource{}, fmt.Errorf("reading config of template %d: %w", vmid, err) + } + if cfg["template"] != "1" { + return vmResource{}, permanentf("placement.image %q: VM %d is not a template", image, vmid) + } + for _, k := range slices.Sorted(maps.Keys(cfg)) { + if diskKey.MatchString(k) && strings.Contains(cfg[k], "cloudinit") { + return vmResource{}, permanentf("placement.image %q: template %d has a Proxmox cloud-init drive on %s; remove it, the provider attaches its own seed", image, vmid, k) + } + } + shape := ownedVM{node: src.Node, vmid: vmid, cfg: cfg} + if _, err := p.configParams(shape, mt, ""); err != nil { + return vmResource{}, err + } + if _, _, _, err := resizeTarget(shape, mt.DiskGiB); err != nil { + return vmResource{}, err + } + return src, nil +} + +// allocate is the probe of spec entry 2: the attempt-th VMID for the UID +// is cloned onto when the listing shows it free and put through the +// ownership check when it is taken, or when the clone finds it taken after +// all (F4). The boolean reports an adoption rather than a fresh clone. +func (p *Provider) allocate(ctx context.Context, log logr.Logger, rs []vmResource, node string, src vmResource, req provider.CreateRequest) (ownedVM, bool, error) { + taken := make(map[int]vmResource, len(rs)) + for _, r := range rs { + taken[r.VMID] = r + } + rng := p.cfg.VMIDRange + probes := min(maxProbes, rng.Max-rng.Min+1) + for attempt := range probes { + vmid := vmidFor(req.UID, rng.Min, rng.Max, attempt) + r, isTaken := taken[vmid] + if !isTaken { + vm, err := p.clone(ctx, log, node, src, vmid, req) + if !isAlreadyExists(err) { + return vm, false, err + } + var found bool + if r, found, err = p.resolve(ctx, vmid); err != nil { + return ownedVM{}, false, err + } + if !found { + log.V(1).Info("proxmox vmid taken by something other than a VM, probing on", "vmid", vmid, "attempt", attempt) + continue + } + } + if r.Type == "qemu" && r.Template == 0 { + vm, ours, err := p.inspect(ctx, r, req) + if err != nil || ours { + return vm, ours, err + } + } + log.V(1).Info("proxmox vmid taken by another vm, probing on", "vmid", vmid, "attempt", attempt) + } + return ownedVM{}, false, transientf("no free VMID for %s after %d probes in %d..%d", req.Name, probes, rng.Min, rng.Max) +} + +// clone is step 4: a full clone carrying the mark, so the VM is never +// unmarked (spec entry 8), then its unlocked config. A taken VMID comes +// back as the API's error for allocate to judge. +func (p *Provider) clone(ctx context.Context, log logr.Logger, node string, src vmResource, vmid int, req provider.CreateRequest) (ownedVM, error) { + params := cloneParams{ + NewID: vmid, + Name: req.Name, + Description: formatDescription(req.UID, p.now()), + Storage: p.cfg.Storage, + } + if node != src.Node { + params.Target = node + } + upid, err := p.api.Clone(ctx, src.Node, src.VMID, params) + if err != nil { + return ownedVM{}, fmt.Errorf("cloning template %d to VM %d: %w", src.VMID, vmid, err) + } + if err := p.waitTask(ctx, log, upid, p.budgets.clone); err != nil { + return ownedVM{}, fmt.Errorf("cloning template %d to VM %d: %w", src.VMID, vmid, err) + } + cfg, err := p.waitUnlocked(ctx, node, vmid, p.budgets.clone) + if err != nil { + return ownedVM{}, err + } + return ownedVM{node: node, vmid: vmid, cfg: cfg}, nil +} + +// provision is steps 5 to 9, each safe to repeat on a VM a previous attempt +// left stopped: the upload overwrites, the config write replaces, and the +// resize is skipped once the disk is large enough. Everything that can be +// refused from the config alone is checked before the first write. +func (p *Provider) provision(ctx context.Context, log logr.Logger, vm ownedVM, mt provider.ProxmoxMachineType, req provider.CreateRequest) error { + params, err := p.configParams(vm, mt, req.Name) + if err != nil { + return err + } + disk, size, grow, err := resizeTarget(vm, mt.DiskGiB) + if err != nil { + return err + } + if err := p.uploadSeed(ctx, log, vm.node, req); err != nil { + return err + } + if err := p.api.SetVMConfig(ctx, vm.node, vm.vmid, params); err != nil { + return fmt.Errorf("configuring VM %d: %w", vm.vmid, err) + } + if grow { + upid, err := p.api.Resize(ctx, vm.node, vm.vmid, disk, size) + if err != nil { + return fmt.Errorf("resizing %s of VM %d to %s: %w", disk, vm.vmid, size, err) + } + if err := p.waitTask(ctx, log, upid, p.budgets.resize); err != nil { + return fmt.Errorf("resizing %s of VM %d to %s: %w", disk, vm.vmid, size, err) + } + } + upid, err := p.api.Start(ctx, vm.node, vm.vmid) + if err != nil { + return fmt.Errorf("starting VM %d: %w", vm.vmid, err) + } + if err := p.waitTask(ctx, log, upid, p.budgets.start); err != nil { + return fmt.Errorf("starting VM %d: %w", vm.vmid, err) + } + return p.waitRunning(ctx, vm.node, vm.vmid) +} + +// configParams is step 6's write: size, agent, tag and seed, and net0 only +// when the config sets a bridge or a VLAN tag (spec entry 7). +func (p *Provider) configParams(vm ownedVM, mt provider.ProxmoxMachineType, name string) (map[string]string, error) { + params := map[string]string{ + "cores": strconv.Itoa(int(mt.Cores)), + "memory": strconv.FormatInt(mt.MemoryMiB, 10), + "agent": "1", + "tags": proxyTag, + "ide2": seedVolID(p.cfg.ISOStorage, name) + ",media=cdrom", + } + if p.cfg.Bridge == "" && p.cfg.VLANTag == 0 { + return params, nil + } + net0 := vm.cfg["net0"] + if net0 == "" { + return nil, permanentf("VM %d has no net0 to apply proxmox.bridge or proxmox.vlanTag to; the template needs a network device", vm.vmid) + } + params["net0"] = rewriteNet0(net0, p.cfg.Bridge, p.cfg.VLANTag) + return params, nil +} + +// rewriteNet0 replaces or appends bridge= and tag= in a net0 value, keeping +// every other option, the model and the MAC in place. An empty bridge or a +// zero tag leaves that option alone. +func rewriteNet0(net0, bridge string, tag int) string { + want := map[string]string{} + if bridge != "" { + want["bridge"] = bridge + } + if tag != 0 { + want["tag"] = strconv.Itoa(tag) + } + opts := strings.Split(net0, ",") + for i, opt := range opts { + key, _, _ := strings.Cut(opt, "=") + if v, ok := want[key]; ok { + opts[i] = key + "=" + v + delete(want, key) + } + } + for _, key := range []string{"bridge", "tag"} { + if v, ok := want[key]; ok { + opts = append(opts, key+"="+v) + } + } + return strings.Join(opts, ",") +} + +// resizeTarget decides step 7: grow the boot disk to diskGiB when that is +// larger than its size= (spec entry 6). A disk is never shrunk. +func resizeTarget(vm ownedVM, diskGiB int64) (disk, size string, grow bool, err error) { + if diskGiB == 0 { + return "", "", false, nil + } + disk, ok := bootDisk(vm.cfg) + if !ok { + return "", "", false, permanentf("VM %d has no boot disk to grow to %dG", vm.vmid, diskGiB) + } + cur, ok := diskBytes(vm.cfg[disk]) + if !ok { + return "", "", false, permanentf("boot disk %s of VM %d has no readable size in %q", disk, vm.vmid, vm.cfg[disk]) + } + if cur >= diskGiB<<30 { + return "", "", false, nil + } + return disk, strconv.FormatInt(diskGiB, 10) + "G", true, nil +} + +// bootDisk is the first disk of boot: order=… that is not a CD-ROM, else +// the legacy bootdisk key, else the first present of scsi0, virtio0, +// sata0 and ide0. Proxmox appends the seed's ide2 to the order (F19), so +// CD-ROMs are skipped. +func bootDisk(cfg vmConfig) (string, bool) { + isDisk := func(k string) bool { + v, ok := cfg[k] + return ok && diskKey.MatchString(k) && diskOption(v, "media") != "cdrom" + } + for dev := range strings.SplitSeq(diskOption(cfg["boot"], "order"), ";") { + if isDisk(dev) { + return dev, true + } + } + if k := cfg["bootdisk"]; isDisk(k) { + return k, true + } + for _, k := range []string{"scsi0", "virtio0", "sata0", "ide0"} { + if isDisk(k) { + return k, true + } + } + return "", false +} + +// diskOption returns the value of key= among a config value's +// comma-separated options. +func diskOption(value, key string) string { + for opt := range strings.SplitSeq(value, ",") { + if v, ok := strings.CutPrefix(opt, key+"="); ok { + return v + } + } + return "" +} + +// diskBytes reads a disk value's size=, in Proxmox's [KMGT] notation +// (bytes without a suffix). +func diskBytes(disk string) (int64, bool) { + s := diskOption(disk, "size") + if s == "" { + return 0, false + } + shift := map[byte]uint{'K': 10, 'M': 20, 'G': 30, 'T': 40} + unit, ok := shift[s[len(s)-1]] + if ok { + s = s[:len(s)-1] + } + n, err := strconv.ParseFloat(s, 64) + if err != nil || n < 0 { + return 0, false + } + return int64(n * float64(int64(1)<= n { + if vm, ok := f.vms[vmid]; ok { + vm.lock = "" + } + } + }) +} + +func TestCreate_repeatReturnsSameIDAndCreatesNothing(t *testing.T) { + t.Parallel() + fx := newCreateFixture(t, 1) + first := fx.create(fx.req()) + if first != formatProviderID(slot(0), testName) { + t.Errorf("Create = %q, want the attempt-0 VMID %d", first, slot(0)) + } + before := len(fx.fake.callLog()) + + if second := fx.create(fx.req()); second != first { + t.Errorf("second Create = %q, want %q", second, first) + } + repeat := fx.fake.callLog()[before:] + if !slices.Equal(repeat, []string{"ClusterResources", "VMConfig", "VMStatus"}) { + t.Errorf("repeat Create made %v, want only the listing, the config and the status reads", repeat) + } + if got := fx.fake.callCount("Clone"); got != 1 { + t.Errorf("Clone calls = %d, want 1", got) + } + if len(fx.logs.matching(`"msg"="proxmox vm adopted"`)) != 1 { + t.Errorf("no adoption line:\n%s", strings.Join(fx.logs.all(), "\n")) + } +} + +func TestCreate_resumesAfterEachPartialStep(t *testing.T) { + t.Parallel() + tests := []struct { + name string + machineType string + setup func(fx *fixture, vm *fakeVM) + wantCalls map[string]int + }{ + { + name: "clone finished", + setup: func(*fixture, *fakeVM) {}, + wantCalls: map[string]int{"UploadISO": 1, "SetVMConfig": 1, "Start": 1}, + }, + { + name: "seed uploaded", + setup: func(fx *fixture, _ *fakeVM) { + upid, _ := fx.fake.UploadISO(context.Background(), fakeNode, fakeISOStorage, seedISOName(testName), strings.NewReader("cut"), 3) + pollTask(fx.t, fx.fake, upid) + }, + wantCalls: map[string]int{"UploadISO": 2, "SetVMConfig": 1, "Start": 1}, + }, + { + name: "config written", + setup: func(_ *fixture, vm *fakeVM) { + vm.config["ide2"] = seedVolID(fakeISOStorage, testName) + ",media=cdrom" + vm.config["boot"] = "order=scsi0;ide2" + vm.config["tags"] = proxyTag + }, + wantCalls: map[string]int{"UploadISO": 1, "SetVMConfig": 1, "Start": 1}, + }, + { + name: "disk resized", + machineType: "big", + setup: func(_ *fixture, vm *fakeVM) { + vm.config["scsi0"] = "local-lvm:vm-1-disk-1,size=8G" + }, + wantCalls: map[string]int{"UploadISO": 1, "SetVMConfig": 1, "Resize": 0, "Start": 1}, + }, + { + name: "start submitted, still booting", + setup: func(_ *fixture, vm *fakeVM) { + vm.status, vm.qmpstatus = "running", "prelaunch" + }, + wantCalls: map[string]int{"UploadISO": 0, "SetVMConfig": 0, "Start": 0}, + }, + { + name: "running", + setup: func(_ *fixture, vm *fakeVM) { + vm.status, vm.qmpstatus = "running", "running" + }, + wantCalls: map[string]int{"UploadISO": 0, "SetVMConfig": 0, "Start": 0}, + }, + } + for _, tc := range tests { + t.Run(tc.name, func(t *testing.T) { + t.Parallel() + fx := newCreateFixture(t, 1) + fx.p.cfg.MachineTypes["big"] = provider.ProxmoxMachineType{Cores: 1, MemoryMiB: 1024, DiskGiB: 8} + vm := fx.addOurVM(slot(0), testName, "stopped") + delete(vm.config, "tags") + tc.setup(fx, vm) + req := fx.req() + if tc.machineType != "" { + req.Placement.MachineType = tc.machineType + } + if id := fx.create(req); id != formatProviderID(slot(0), testName) { + t.Errorf("Create = %q, want the adopted VM", id) + } + if got := fx.fake.callCount("Clone"); got != 0 { + t.Errorf("Clone calls = %d, want 0", got) + } + for method, want := range tc.wantCalls { + if got := fx.fake.callCount(method); got != want { + t.Errorf("%s calls = %d, want %d", method, got, want) + } + } + if vm, _ := fx.fake.vmSnapshot(slot(0)); vm.status != "running" { + t.Errorf("VM status = %s, want running", vm.status) + } + if b, _ := fx.fake.iso(fakeISOStorage, seedISOName(testName)); tc.wantCalls["UploadISO"] > 0 && !bytes.Contains(b, []byte(nocloud.SeedLabel)) { + t.Errorf("seed on the storage is not a rebuilt ISO (%d bytes)", len(b)) + } + }) + } +} + +// TestCreate_resumesMidCloneVMAtDeterministicVMID is the crash Phase 0 +// found: the clone was submitted, the listing shows the VM nameless and +// locked, and the retry must wait for it and adopt it rather than probe +// past it and clone a duplicate. +func TestCreate_resumesMidCloneVMAtDeterministicVMID(t *testing.T) { + t.Parallel() + fx := newCreateFixture(t, 1) + vm := fx.addOurVM(slot(0), testName, "stopped") + delete(vm.config, "tags") + vm.lock = "clone" + clearLockOnVMConfig(fx.fake, slot(0), 4) + + if id := fx.create(fx.req()); id != formatProviderID(slot(0), testName) { + t.Errorf("Create = %q, want the mid-clone VM at %d", id, slot(0)) + } + if got := fx.fake.callCount("Clone"); got != 0 { + t.Errorf("Clone calls = %d, want 0: the retry cloned a duplicate", got) + } + if got := fx.fake.callCount("VMConfig"); got < 4 { + t.Errorf("VMConfig calls = %d, want the locked VM polled until unlocked", got) + } + fx.assertRunning(slot(0)) +} + +func TestCreate_takenVMIDInspectedBeforeProbing(t *testing.T) { + t.Parallel() + tests := []struct { + name string + setup func(fx *fixture) + // wantVMID is the attempt the VM ends up on; -1 is an error. + wantAttempt int + wantClass error + // wantConfigReads is how many VMConfig calls precede the first + // Clone: the template, plus one per inspected VM. + wantConfigReads int + }{ + { + name: "foreign VM probes on after its config is read", + setup: func(fx *fixture) { fx.fake.addVM(slot(0), "someone-else", "running") }, + wantAttempt: 1, + wantConfigReads: 2, + }, + { + name: "container probes on without a read", + setup: func(fx *fixture) { + fx.fake.addVM(slot(0), "ct", "running").kind = "lxc" + }, + wantAttempt: 1, + wantConfigReads: 1, + }, + { + name: "template probes on without a read", + setup: func(fx *fixture) { fx.fake.addTemplate(slot(0), "other-template") }, + wantAttempt: 1, + wantConfigReads: 1, + }, + { + name: "ours, nameless in the listing, is adopted", + setup: func(fx *fixture) { + fx.addOurVM(slot(0), testName, "stopped").lock = "clone" + clearLockOnVMConfig(fx.fake, slot(0), 1) + }, + wantAttempt: 0, + wantConfigReads: 2, + }, + { + name: "someone else's clone with our name is permanent", + setup: func(fx *fixture) { + vm := fx.fake.addVM(slot(0), testName, "stopped") + vm.config["description"] = formatDescription("another-uid", testCreatedAt) + vm.lock = "clone" + clearLockOnVMConfig(fx.fake, slot(0), 1) + }, + wantAttempt: -1, + wantClass: provider.ErrPermanent, + wantConfigReads: 2, + }, + } + for _, tc := range tests { + t.Run(tc.name, func(t *testing.T) { + t.Parallel() + fx := newCreateFixture(t, 1) + tc.setup(fx) + if tc.wantAttempt < 0 { + fx.createFails(fx.req(), tc.wantClass) + } else if id := fx.create(fx.req()); id != formatProviderID(slot(tc.wantAttempt), testName) { + t.Errorf("Create = %q, want attempt %d (VMID %d)", id, tc.wantAttempt, slot(tc.wantAttempt)) + } + if got := callsBefore(fx.fake.callLog(), "VMConfig", "Clone"); got != tc.wantConfigReads { + t.Errorf("VMConfig calls before the clone = %d, want %d", got, tc.wantConfigReads) + } + }) + } +} + +// TestCreate_cloneAlreadyExistsRaceRunsOwnershipCheck: the listing shows +// the VMID free, but another Create takes it before our clone lands (F4). +func TestCreate_cloneAlreadyExistsRaceRunsOwnershipCheck(t *testing.T) { + t.Parallel() + tests := []struct { + name string + racerName string + racerUID string + wantAttempt int + wantClones int + }{ + {name: "another proxy's clone is waited on and probed past", racerName: "proxy-racer", racerUID: "racer-uid", wantAttempt: 1, wantClones: 2}, + {name: "our own clone is adopted", racerName: testName, racerUID: testUID, wantAttempt: 0, wantClones: 1}, + } + for _, tc := range tests { + t.Run(tc.name, func(t *testing.T) { + t.Parallel() + fx := newCreateFixture(t, 1) + var raced bool + fx.fake.hookOn("Clone", func() error { + if !raced { + raced = true + vm := fx.fake.addVM(slot(0), tc.racerName, "stopped") + fx.fake.mu.Lock() + vm.config["description"] = formatDescription(tc.racerUID, testCreatedAt) + vm.lock = "clone" + fx.fake.mu.Unlock() + } + return nil + }) + clearLockOnVMConfig(fx.fake, slot(0), 3) + + if id := fx.create(fx.req()); id != formatProviderID(slot(tc.wantAttempt), testName) { + t.Errorf("Create = %q, want attempt %d", id, tc.wantAttempt) + } + if got := fx.fake.callCount("Clone"); got != tc.wantClones { + t.Errorf("Clone calls = %d, want %d", got, tc.wantClones) + } + if vm, _ := fx.fake.vmSnapshot(slot(0)); vm.name != tc.racerName { + t.Errorf("VM %d is named %q, want the racer's %q left alone", slot(0), vm.name, tc.racerName) + } + }) + } +} + +func TestCreate_returnsOnlyWhenRunning(t *testing.T) { + t.Parallel() + fx := newCreateFixture(t, 1) + var polls int + mutateOn(fx.fake, "VMStatus", func() { + polls++ + if vm, ok := fx.fake.vms[slot(0)]; ok && polls < 3 { + vm.qmpstatus = "prelaunch" + } else if ok { + vm.qmpstatus = "running" + } + }) + fx.create(fx.req()) + if got := fx.fake.callCount("VMStatus"); got != 3 { + t.Errorf("VMStatus calls = %d, want 3: Create must wait through prelaunch", got) + } + fx.assertRunning(slot(0)) +} + +func TestCreate_vmThatStopsAfterStartIsTransientAtOnce(t *testing.T) { + t.Parallel() + fx := newCreateFixture(t, 1) + fx.p.budgets.running = time.Hour + mutateOn(fx.fake, "VMStatus", func() { + fx.fake.vms[slot(0)].status = "stopped" + }) + msg := fx.createFails(fx.req(), provider.ErrTransient) + if !strings.Contains(msg, "stopped right after its start") { + t.Errorf("error = %s, want it to say the VM stopped", msg) + } + if got := fx.fake.callCount("VMStatus"); got != 1 { + t.Errorf("VMStatus calls = %d, want 1", got) + } +} + +func TestCreate_runningWaitCapIsTransient(t *testing.T) { + t.Parallel() + fx := newCreateFixture(t, 1) + fx.p.budgets.running = 30 * time.Millisecond + mutateOn(fx.fake, "VMStatus", func() { + fx.fake.vms[slot(0)].qmpstatus = "prelaunch" + }) + msg := fx.createFails(fx.req(), provider.ErrTransient) + if !strings.Contains(msg, "VM "+strconv.Itoa(slot(0))+" to run") { + t.Errorf("error = %s, want it to name the running wait", msg) + } +} + +func TestCreate_rejectsUnknownPlacement(t *testing.T) { + t.Parallel() + tests := []struct { + name string + nodes []string + mutate func(*provider.CreateRequest) + want string + }{ + {name: "unknown zone", mutate: func(r *provider.CreateRequest) { r.Placement.Zone = "pv9-9" }, want: `placement.zone "pv9-9" is not in proxmox.nodes`}, + {name: "empty zone with two nodes", nodes: []string{"pv1-1", "pv1-2"}, mutate: func(*provider.CreateRequest) {}, want: "placement.zone is required"}, + {name: "unknown machine type", mutate: func(r *provider.CreateRequest) { r.Placement.MachineType = "huge" }, want: `placement.machineType "huge" is not in proxmox.machineTypes`}, + {name: "empty machine type", mutate: func(r *provider.CreateRequest) { r.Placement.MachineType = "" }, want: `placement.machineType "" is not in proxmox.machineTypes`}, + {name: "unknown image", mutate: func(r *provider.CreateRequest) { r.Placement.Image = "arch" }, want: `placement.image "arch" is not in proxmox.templates`}, + {name: "no name", mutate: func(r *provider.CreateRequest) { r.Name = "" }, want: "no instance name"}, + } + for _, tc := range tests { + t.Run(tc.name, func(t *testing.T) { + t.Parallel() + fx := newCreateFixture(t, 1) + if tc.nodes != nil { + fx.p.cfg.Nodes = tc.nodes + } + req := fx.req() + tc.mutate(&req) + if msg := fx.createFails(req, provider.ErrPermanent); !strings.Contains(msg, tc.want) { + t.Errorf("error = %s, want it to contain %s", msg, tc.want) + } + if got := fx.fake.callLog(); len(got) != 0 { + t.Errorf("calls = %v, want none before placement is valid", got) + } + }) + } +} + +func TestCreate_zoneOptionalWithOneNode(t *testing.T) { + t.Parallel() + for _, zone := range []string{"", fakeNode} { + t.Run("zone "+strconv.Quote(zone), func(t *testing.T) { + t.Parallel() + fx := newCreateFixture(t, 1) + req := fx.req() + req.Placement.Zone = zone + fx.create(req) + if vm := fx.assertRunning(slot(0)); vm.node != fakeNode { + t.Errorf("VM node = %s, want %s", vm.node, fakeNode) + } + }) + } +} + +func TestCreate_regionIgnored(t *testing.T) { + t.Parallel() + fx := newCreateFixture(t, 1) + req := fx.req() + req.Placement.Region = "eu-west-1" + if id := fx.create(req); id != formatProviderID(slot(0), testName) { + t.Errorf("Create = %q, want the same VM as without a region", id) + } +} + +func TestCreate_adoptionWithForeignUIDIsPermanent(t *testing.T) { + t.Parallel() + fx := newCreateFixture(t, 1) + fx.fake.addVM(testVMID, testName, "running").config["description"] = formatDescription("another-uid", testCreatedAt) + msg := fx.createFails(fx.req(), provider.ErrPermanent) + for _, want := range []string{"150042", "another-uid", testUID} { + if !strings.Contains(msg, want) { + t.Errorf("error = %s, want it to name %s", msg, want) + } + } + if got := fx.fake.callCount("Clone") + fx.fake.callCount("Start"); got != 0 { + t.Errorf("Clone and Start calls = %d, want 0", got) + } +} + +func TestCreate_adoptsOnlyMarkedVMs(t *testing.T) { + t.Parallel() + tests := []struct { + name string + description string + wantClass error + }{ + {name: "no description is refused", description: "", wantClass: provider.ErrPermanent}, + {name: "admin notes only are refused", description: "owner=alice\nticket=OPS-1", wantClass: provider.ErrPermanent}, + {name: "managed false is refused", description: provider.LabelManaged + "=false\n" + provider.LabelUID + "=" + testUID, wantClass: provider.ErrPermanent}, + {name: "mark with an empty uid is adopted", description: provider.LabelManaged + "=true"}, + {name: "mark with our uid is adopted", description: formatDescription(testUID, testCreatedAt)}, + } + for _, tc := range tests { + t.Run(tc.name, func(t *testing.T) { + t.Parallel() + fx := newCreateFixture(t, 1) + if tc.description != "" { + fx.fake.addVM(testVMID, testName, "stopped").config["description"] = tc.description + } else { + fx.fake.addVM(testVMID, testName, "stopped") + } + if tc.wantClass != nil { + if msg := fx.createFails(fx.req(), tc.wantClass); !strings.Contains(msg, "no proxy-operator mark") { + t.Errorf("error = %s, want it to say the VM is unmarked", msg) + } + if vm, _ := fx.fake.vmSnapshot(testVMID); vm.status != "stopped" || fx.fake.callCount("SetVMConfig") != 0 { + t.Error("an unmarked VM was touched") + } + return + } + if id := fx.create(fx.req()); id != fx.id() { + t.Errorf("Create = %q, want the adopted %q", id, fx.id()) + } + }) + } +} + +// TestCreate_hibernatedVMIsNotWaitedOn: a suspended lock is not a task and +// never clears, so the ownership check judges the VM at once instead of +// waiting out the clone budget. +func TestCreate_hibernatedVMIsNotWaitedOn(t *testing.T) { + t.Parallel() + t.Run("ours is permanent", func(t *testing.T) { + t.Parallel() + fx := newCreateFixture(t, 1) + fx.p.budgets.clone = time.Hour + fx.addOurVM(testVMID, testName, "stopped").lock = lockSuspended + if msg := fx.createFails(fx.req(), provider.ErrPermanent); !strings.Contains(msg, "hibernated") { + t.Errorf("error = %s, want it to say the VM is hibernated", msg) + } + if got := fx.fake.callCount("VMConfig"); got != 1 { + t.Errorf("VMConfig calls = %d, want 1", got) + } + }) + t.Run("foreign is probed past", func(t *testing.T) { + t.Parallel() + fx := newCreateFixture(t, 1) + fx.p.budgets.clone = time.Hour + fx.fake.addVM(slot(0), "someone-else", "stopped").lock = lockSuspended + if id := fx.create(fx.req()); id != formatProviderID(slot(1), testName) { + t.Errorf("Create = %q, want attempt 1", id) + } + }) +} + +func TestCreate_duplicateNameIsPermanent(t *testing.T) { + t.Parallel() + fx := newCreateFixture(t, 1) + fx.addOurVM(testVMID, testName, "running") + fx.addOurVM(testVMID+7, testName, "stopped") + msg := fx.createFails(fx.req(), provider.ErrPermanent) + if !strings.Contains(msg, "150042, 150049") { + t.Errorf("error = %s, want it to name both VMIDs", msg) + } + if got := fx.fake.callLog(); !slices.Equal(got, []string{"ClusterResources"}) { + t.Errorf("calls = %v, want only the listing", got) + } +} + +func TestCreate_vmidCollisionProbesForward(t *testing.T) { + t.Parallel() + fx := newCreateFixture(t, 1) + fx.fake.addVM(slot(0), "someone-else", "running") + fx.fake.addVM(slot(1), "someone-else-too", "stopped") + if id := fx.create(fx.req()); id != formatProviderID(slot(2), testName) { + t.Errorf("Create = %q, want attempt 2 (VMID %d)", id, slot(2)) + } + if got := len(fx.logs.matching("probing on")); got != 2 { + t.Errorf("probe lines = %d, want 2", got) + } +} + +func TestCreate_vmidRangeExhaustedIsTransient(t *testing.T) { + t.Parallel() + fx := newCreateFixture(t, 1) + fx.p.cfg.VMIDRange = &provider.ProxmoxVMIDRange{Min: 100000, Max: 100003} + for vmid := 100000; vmid <= 100003; vmid++ { + fx.fake.addVM(vmid, "someone-"+strconv.Itoa(vmid), "running") + } + msg := fx.createFails(fx.req(), provider.ErrTransient) + if !strings.Contains(msg, "no free VMID") || !strings.Contains(msg, "100000..100003") { + t.Errorf("error = %s, want it to name the range", msg) + } + if got := fx.fake.callCount("Clone"); got != 0 { + t.Errorf("Clone calls = %d, want 0", got) + } + if got := fx.fake.callCount("VMConfig"); got != 5 { + t.Errorf("VMConfig calls = %d, want the template and each of the 4 VMIDs once", got) + } +} + +func TestCreate_waitsForCloneLockOnRetry(t *testing.T) { + t.Parallel() + fx := newCreateFixture(t, 1) + fx.p.budgets.clone = 30 * time.Millisecond + fx.fake.cloneLeavesLock = true + if msg := fx.createFails(fx.req(), provider.ErrTransient); !strings.Contains(msg, "the lock on VM") { + t.Errorf("first error = %s, want the lock wait to run out", msg) + } + fx.fake.mu.Lock() + fx.fake.cloneLeavesLock = false + fx.fake.mu.Unlock() + fx.p.budgets.clone = time.Second + clearLockOnVMConfig(fx.fake, slot(0), 4) + + if id := fx.create(fx.req()); id != formatProviderID(slot(0), testName) { + t.Errorf("retry Create = %q, want the locked VM adopted", id) + } + if got := fx.fake.callCount("Clone"); got != 1 { + t.Errorf("Clone calls = %d, want 1", got) + } + fx.assertRunning(slot(0)) +} + +func TestCreate_cloneTaskFailureIsTransient(t *testing.T) { + t.Parallel() + fx := newCreateFixture(t, 1) + fx.fake.cloneFailsWith = "clone failed: storage 'local-lvm' is full" + msg := fx.createFails(fx.req(), provider.ErrTransient) + if !strings.Contains(msg, "clone failed: storage 'local-lvm' is full") { + t.Errorf("error = %s, want the task's exitstatus", msg) + } + if len(fx.logs.matching(`"msg"="proxmox task failed"`)) != 1 { + t.Errorf("no task-failed line:\n%s", strings.Join(fx.logs.all(), "\n")) + } + fx.fake.mu.Lock() + fx.fake.cloneFailsWith = "" + fx.fake.mu.Unlock() + if id := fx.create(fx.req()); id != formatProviderID(slot(0), testName) { + t.Errorf("retry Create = %q, want attempt 0 again", id) + } +} + +func TestCreate_cloneTaskTimeoutIsTransient(t *testing.T) { + t.Parallel() + fx := newCreateFixture(t, 1) + fx.p.budgets.clone = 30 * time.Millisecond + fx.fake.taskPolls = 1 << 30 + msg := fx.createFails(fx.req(), provider.ErrTransient) + if !strings.Contains(msg, "waiting for task UPID:"+fakeNode) || !strings.Contains(msg, "not done after") { + t.Errorf("error = %s, want it to name the task and the budget", msg) + } +} + +// TestCreate_markWrittenInCloneCall reads the VM when the seed upload +// starts, between the clone and the config write: the mark is already +// there, so it came with the clone. +func TestCreate_markWrittenInCloneCall(t *testing.T) { + t.Parallel() + fx := newCreateFixture(t, 1) + now := time.Date(2026, 9, 23, 12, 30, 0, 0, time.UTC) + fx.p.now = func() time.Time { return now } + var atUpload string + mutateOn(fx.fake, "UploadISO", func() { atUpload = fx.fake.vms[slot(0)].config["description"] }) + fx.create(fx.req()) + want := formatDescription(testUID, now) + if atUpload != want { + t.Errorf("description after the clone = %q, want %q", atUpload, want) + } + vm, _ := fx.fake.vmSnapshot(slot(0)) + if m := parseDescription(vm.config["description"]); !m.Managed || m.UID != testUID || !m.CreatedAt.Equal(now) { + t.Errorf("mark read back = %+v", m) + } +} + +func TestCreate_tagSetInConfigStep(t *testing.T) { + t.Parallel() + fx := newCreateFixture(t, 1) + atUpload := "unset" + mutateOn(fx.fake, "UploadISO", func() { atUpload = fx.fake.vms[slot(0)].config["tags"] }) + fx.create(fx.req()) + if atUpload != "" { + t.Errorf("tags after the clone = %q, want none: tags are not a clone parameter", atUpload) + } + if vm, _ := fx.fake.vmSnapshot(slot(0)); vm.config["tags"] != proxyTag { + t.Errorf("tags = %q, want %q", vm.config["tags"], proxyTag) + } +} + +func TestCreate_seedAlwaysRebuilt(t *testing.T) { + t.Parallel() + fx := newCreateFixture(t, 1) + fx.create(fx.req()) + fx.fake.mu.Lock() + fx.fake.vms[slot(0)].status = "stopped" + fx.fake.isos[fakeISOStorage][seedISOName(testName)] = []byte("cut") + fx.fake.mu.Unlock() + + fx.create(fx.req()) + if got := fx.fake.callCount("UploadISO"); got != 2 { + t.Errorf("UploadISO calls = %d, want 2", got) + } + if again, _ := fx.fake.iso(fakeISOStorage, seedISOName(testName)); !bytes.Contains(again, []byte(nocloud.SeedLabel)) { + t.Errorf("seed after the retry has %d bytes and no %s label, want it rebuilt", len(again), nocloud.SeedLabel) + } +} + +func TestCreate_seedUploadOverwritesWithoutDelete(t *testing.T) { + t.Parallel() + fx := newCreateFixture(t, 1) + fx.fake.mu.Lock() + fx.fake.isos[fakeISOStorage] = map[string][]byte{seedISOName(testName): []byte("stale")} + fx.fake.mu.Unlock() + fx.create(fx.req()) + if got := fx.fake.callCount("DeleteVolume"); got != 0 { + t.Errorf("DeleteVolume calls = %d, want 0: an upload overwrites (F12)", got) + } + b, _ := fx.fake.iso(fakeISOStorage, seedISOName(testName)) + if !bytes.Contains(b, []byte(nocloud.SeedLabel)) { + t.Errorf("seed has %d bytes and no %s label, want the new ISO", len(b), nocloud.SeedLabel) + } +} + +func TestCreate_seedUploadedBeforeConfig(t *testing.T) { + t.Parallel() + fx := newCreateFixture(t, 1) + fx.fake.taskPolls = 2 + fx.create(fx.req()) + log := fx.fake.callLog() + up, cfg := slices.Index(log, "UploadISO"), slices.Index(log, "SetVMConfig") + if up < 0 || cfg < up { + t.Fatalf("call log = %v, want UploadISO before SetVMConfig", log) + } + if got := callsBefore(log[up:], "TaskStatus", "SetVMConfig"); got != 3 { + t.Errorf("upload task polls before the config write = %d, want 3", got) + } +} + +func TestCreate_configSetsCoresMemoryAgentAndSeed(t *testing.T) { + t.Parallel() + fx := newCreateFixture(t, 1) + fx.fake.mu.Lock() + tmpl := fx.fake.vms[testTemplateVMID].config + tmpl["agent"], tmpl["cores"], tmpl["memory"] = "0", "4", "8192" + fx.fake.mu.Unlock() + fx.create(fx.req()) + vm, _ := fx.fake.vmSnapshot(slot(0)) + want := map[string]string{ + "cores": "1", + "memory": "1024", + "agent": "1", + "ide2": "proxy-seeds:iso/" + testName + "-seed.iso,media=cdrom", + "tags": proxyTag, + } + for k, v := range want { + if vm.config[k] != v { + t.Errorf("%s = %q, want %q", k, vm.config[k], v) + } + } +} + +func TestCreate_bridgeOverrideKeepsMAC(t *testing.T) { + t.Parallel() + fx := newCreateFixture(t, 1) + fx.p.cfg.Bridge = "vmbr4" + fx.create(fx.req()) + if vm, _ := fx.fake.vmSnapshot(slot(0)); vm.config["net0"] != "virtio=BC:24:11:48:11:46,bridge=vmbr4" { + t.Errorf("net0 = %q, want the MAC kept and the bridge replaced", vm.config["net0"]) + } +} + +// TestCreate_noBridgeOverrideLeavesNet0 changes net0 just before the config +// write: a write that carried net0 would put the cloned value back. +func TestCreate_noBridgeOverrideLeavesNet0(t *testing.T) { + t.Parallel() + fx := newCreateFixture(t, 1) + const changed = "virtio=BC:24:11:48:11:46,bridge=vmbr9" + mutateOn(fx.fake, "SetVMConfig", func() { fx.fake.vms[slot(0)].config["net0"] = changed }) + fx.create(fx.req()) + if vm, _ := fx.fake.vmSnapshot(slot(0)); vm.config["net0"] != changed { + t.Errorf("net0 = %q, want it left out of the config write", vm.config["net0"]) + } +} + +func TestCreate_vlanTagAppendedKeepsMACAndOptions(t *testing.T) { + t.Parallel() + fx := newCreateFixture(t, 1) + fx.p.cfg.VLANTag = 4 + fx.fake.mu.Lock() + fx.fake.vms[testTemplateVMID].config["net0"] = "virtio=BC:24:11:48:11:46,bridge=vmbr0,firewall=1,mtu=1500" + fx.fake.mu.Unlock() + fx.create(fx.req()) + if vm, _ := fx.fake.vmSnapshot(slot(0)); vm.config["net0"] != "virtio=BC:24:11:48:11:46,bridge=vmbr0,firewall=1,mtu=1500,tag=4" { + t.Errorf("net0 = %q, want tag=4 appended and everything else in order", vm.config["net0"]) + } +} + +func TestRewriteNet0_table(t *testing.T) { + t.Parallel() + const mac = "virtio=BC:24:11:48:11:46" + tests := []struct { + name string + net0 string + bridge string + tag int + want string + }{ + {name: "nothing configured", net0: mac + ",bridge=vmbr0", want: mac + ",bridge=vmbr0"}, + {name: "bridge replaced in place", net0: mac + ",bridge=vmbr0,firewall=1", bridge: "vmbr4", want: mac + ",bridge=vmbr4,firewall=1"}, + {name: "tag appended", net0: mac + ",bridge=vmbr0", tag: 4, want: mac + ",bridge=vmbr0,tag=4"}, + {name: "tag replaced in place", net0: mac + ",bridge=vmbr0,tag=2,firewall=1", tag: 4, want: mac + ",bridge=vmbr0,tag=4,firewall=1"}, + {name: "both appended", net0: mac, bridge: "vmbr4", tag: 4, want: mac + ",bridge=vmbr4,tag=4"}, + {name: "the captured VLAN template", net0: "virtio=BC:24:11:2F:97:A0,bridge=vmbr0,tag=4", bridge: "vmbr1", tag: 7, want: "virtio=BC:24:11:2F:97:A0,bridge=vmbr1,tag=7"}, + } + for _, tc := range tests { + t.Run(tc.name, func(t *testing.T) { + t.Parallel() + if got := rewriteNet0(tc.net0, tc.bridge, tc.tag); got != tc.want { + t.Errorf("rewriteNet0(%q, %q, %d) = %q, want %q", tc.net0, tc.bridge, tc.tag, got, tc.want) + } + }) + } +} + +// TestCreate_templateShapeRefusedBeforeClone: the clone copies the +// template's config, so what the config or resize step would refuse on the +// clone is refused on the template, before a VM exists that a fixed +// template could no longer help. +func TestCreate_templateShapeRefusedBeforeClone(t *testing.T) { + t.Parallel() + tests := []struct { + name string + setup func(fx *fixture, tmpl map[string]string) + want string + bigger bool + }{ + { + name: "no net0 with a bridge configured", + setup: func(fx *fixture, tmpl map[string]string) { + fx.p.cfg.Bridge = "vmbr4" + delete(tmpl, "net0") + }, + want: "no net0", + }, + { + name: "no boot disk with diskGiB set", + setup: func(_ *fixture, tmpl map[string]string) { + delete(tmpl, "scsi0") + }, + want: "no boot disk", + bigger: true, + }, + { + name: "boot disk without a size", + setup: func(_ *fixture, tmpl map[string]string) { + tmpl["scsi0"] = "local-lvm:base-9001-disk-1" + }, + want: "no readable size", + bigger: true, + }, + } + for _, tc := range tests { + t.Run(tc.name, func(t *testing.T) { + t.Parallel() + fx := newCreateFixture(t, 1) + fx.p.cfg.MachineTypes["big"] = provider.ProxmoxMachineType{Cores: 1, MemoryMiB: 1024, DiskGiB: 8} + fx.fake.mu.Lock() + tc.setup(fx, fx.fake.vms[testTemplateVMID].config) + fx.fake.mu.Unlock() + req := fx.req() + if tc.bigger { + req.Placement.MachineType = "big" + } + if msg := fx.createFails(req, provider.ErrPermanent); !strings.Contains(msg, tc.want) { + t.Errorf("error = %s, want %s", msg, tc.want) + } + if got := fx.fake.callCount("Clone"); got != 0 { + t.Errorf("Clone calls = %d, want 0", got) + } + }) + } +} + +func TestCreate_resizeWaitsForTask(t *testing.T) { + t.Parallel() + fx := newCreateFixture(t, 1) + fx.p.cfg.MachineTypes["big"] = provider.ProxmoxMachineType{Cores: 1, MemoryMiB: 1024, DiskGiB: 8} + fx.fake.taskPolls = 2 + req := fx.req() + req.Placement.MachineType = "big" + fx.create(req) + log := fx.fake.callLog() + rs := slices.Index(log, "Resize") + if rs < 0 { + t.Fatalf("call log = %v, want a Resize", log) + } + if got := callsBefore(log[rs:], "TaskStatus", "Start"); got != 3 { + t.Errorf("resize task polls before the start = %d, want 3", got) + } + if vm, _ := fx.fake.vmSnapshot(slot(0)); vm.config["scsi0"] != "local-lvm:vm-"+strconv.Itoa(slot(0))+"-disk-1,size=8G" { + t.Errorf("scsi0 = %q, want size=8G", vm.config["scsi0"]) + } +} + +func TestCreate_resizeSkippedWhenNotLarger(t *testing.T) { + t.Parallel() + for _, diskGiB := range []int64{0, 2, 3} { + t.Run("diskGiB "+strconv.FormatInt(diskGiB, 10), func(t *testing.T) { + t.Parallel() + fx := newCreateFixture(t, 1) + fx.p.cfg.MachineTypes["sized"] = provider.ProxmoxMachineType{Cores: 1, MemoryMiB: 1024, DiskGiB: diskGiB} + req := fx.req() + req.Placement.MachineType = "sized" + fx.create(req) + if got := fx.fake.callCount("Resize"); got != 0 { + t.Errorf("Resize calls = %d, want 0 for a 3G disk", got) + } + if vm, _ := fx.fake.vmSnapshot(slot(0)); diskOption(vm.config["scsi0"], "size") != "3G" { + t.Errorf("scsi0 = %q, want 3G kept", vm.config["scsi0"]) + } + }) + } +} + +func TestBootDisk_table(t *testing.T) { + t.Parallel() + const ( + disk = "local-lvm:vm-1-disk-1,size=3G" + seed = "proxy-seeds:iso/proxy-x-seed.iso,media=cdrom" + cdrom = "none,media=cdrom" + ) + tests := []struct { + name string + cfg vmConfig + want string + wantOK bool + }{ + {name: "order names the disk", cfg: vmConfig{"boot": "order=scsi0", "scsi0": disk}, want: "scsi0", wantOK: true}, + {name: "seed appended by Proxmox", cfg: vmConfig{"boot": "order=scsi0;ide2", "scsi0": disk, "ide2": seed}, want: "scsi0", wantOK: true}, + {name: "cdrom first in order", cfg: vmConfig{"boot": "order=ide2;scsi0", "scsi0": disk, "ide2": cdrom}, want: "scsi0", wantOK: true}, + {name: "network first in order", cfg: vmConfig{"boot": "order=net0;virtio0", "net0": "virtio=BC:24:11:48:11:46", "virtio0": disk}, want: "virtio0", wantOK: true}, + {name: "second disk first in order", cfg: vmConfig{"boot": "order=scsi1;scsi0", "scsi0": disk, "scsi1": disk}, want: "scsi1", wantOK: true}, + {name: "order names a missing disk", cfg: vmConfig{"boot": "order=scsi3;sata0", "sata0": disk}, want: "sata0", wantOK: true}, + {name: "legacy boot with bootdisk", cfg: vmConfig{"boot": "cdn", "bootdisk": "virtio0", "virtio0": disk, "scsi0": disk}, want: "virtio0", wantOK: true}, + {name: "no boot line, first known slot", cfg: vmConfig{"sata0": disk, "ide0": disk}, want: "sata0", wantOK: true}, + {name: "ide0 disk is a disk", cfg: vmConfig{"ide0": disk, "ide2": seed}, want: "ide0", wantOK: true}, + {name: "efidisk is not a boot disk", cfg: vmConfig{"efidisk0": "local-lvm:vm-1-disk-0,size=4M", "ide2": seed}}, + {name: "cdrom only", cfg: vmConfig{"boot": "order=ide0", "ide0": cdrom}}, + {name: "empty", cfg: vmConfig{}}, + } + for _, tc := range tests { + t.Run(tc.name, func(t *testing.T) { + t.Parallel() + got, ok := bootDisk(tc.cfg) + if got != tc.want || ok != tc.wantOK { + t.Errorf("bootDisk = %q, %v; want %q, %v", got, ok, tc.want, tc.wantOK) + } + }) + } +} + +func TestBootDisk_capturedClone(t *testing.T) { + t.Parallel() + cfg := capturedConfig(t, "vm_config_clone") + if got, ok := bootDisk(cfg); got != "scsi0" || !ok { + t.Errorf("bootDisk of the captured clone = %q, %v; want scsi0", got, ok) + } + if got, ok := diskBytes(cfg["scsi0"]); got != 3<<30 || !ok { + t.Errorf("diskBytes of the captured scsi0 = %d, %v; want 3 GiB", got, ok) + } +} + +func TestDiskBytes_table(t *testing.T) { + t.Parallel() + tests := []struct { + disk string + want int64 + wantOK bool + }{ + {disk: "local-lvm:vm-1-disk-1,size=3G", want: 3 << 30, wantOK: true}, + {disk: "local-lvm:vm-1-disk-0,efitype=4m,size=4M", want: 4 << 20, wantOK: true}, + {disk: "local:1/vm-1-disk-0.qcow2,size=512K", want: 512 << 10, wantOK: true}, + {disk: "big:vm-1-disk-1,size=2T", want: 2 << 40, wantOK: true}, + {disk: "raw:vm-1-disk-1,size=1073741824", want: 1 << 30, wantOK: true}, + {disk: "odd:vm-1-disk-1,size=1.5G", want: 3 << 29, wantOK: true}, + {disk: "local-lvm:vm-1-disk-1"}, + {disk: "local-lvm:vm-1-disk-1,size="}, + {disk: "local-lvm:vm-1-disk-1,size=lots"}, + {disk: "local-lvm:vm-1-disk-1,size=-1G"}, + } + for _, tc := range tests { + t.Run(tc.disk, func(t *testing.T) { + t.Parallel() + got, ok := diskBytes(tc.disk) + if got != tc.want || ok != tc.wantOK { + t.Errorf("diskBytes(%q) = %d, %v; want %d, %v", tc.disk, got, ok, tc.want, tc.wantOK) + } + }) + } +} + +func TestCreate_startTaskFailureIsTransient(t *testing.T) { + t.Parallel() + fx := newCreateFixture(t, 1) + fx.fake.startFailsWith = "start failed: QEMU exited with code 1" + if msg := fx.createFails(fx.req(), provider.ErrTransient); !strings.Contains(msg, "start failed: QEMU exited with code 1") { + t.Errorf("error = %s, want the task's exitstatus", msg) + } +} + +func TestCreate_templateMissingIsPermanent(t *testing.T) { + t.Parallel() + fx := newFixture(t, 1) + msg := fx.createFails(fx.req(), provider.ErrPermanent) + if !strings.Contains(msg, `placement.image "debian-13": template VM 9001 does not exist`) { + t.Errorf("error = %s, want it to name the image and the VMID", msg) + } + if got := fx.fake.callCount("Clone"); got != 0 { + t.Errorf("Clone calls = %d, want 0", got) + } +} + +// capturedConfig loads a config capture the way the client decodes it. +func capturedConfig(t *testing.T, name string) vmConfig { + t.Helper() + cfg, err := decodeConfig(decodeData[map[string]json.RawMessage](t, name)) + if err != nil { + t.Fatalf("decoding %s: %v", name, err) + } + return cfg +} + +func TestCreate_templateWithCloudInitDriveIsPermanent(t *testing.T) { + t.Parallel() + tests := []struct { + name string + setup func(vm *fakeVM) + want string + }{ + { + name: "the captured Ubuntu template", + setup: func(vm *fakeVM) { + vm.config = capturedConfig(t, "vm_config_template_ubuntu") + }, + want: "cloud-init drive on ide2", + }, + { + name: "cloud-init drive on another slot", + setup: func(vm *fakeVM) { vm.config["sata1"] = "local-lvm:vm-9001-cloudinit,media=cdrom" }, + want: "cloud-init drive on sata1", + }, + { + name: "not a template", + setup: func(vm *fakeVM) { vm.template = false }, + want: "VM 9001 is not a template", + }, + } + for _, tc := range tests { + t.Run(tc.name, func(t *testing.T) { + t.Parallel() + fx := newCreateFixture(t, 1) + fx.fake.mu.Lock() + tc.setup(fx.fake.vms[testTemplateVMID]) + fx.fake.mu.Unlock() + if msg := fx.createFails(fx.req(), provider.ErrPermanent); !strings.Contains(msg, tc.want) { + t.Errorf("error = %s, want %s", msg, tc.want) + } + if got := fx.fake.callCount("Clone"); got != 0 { + t.Errorf("Clone calls = %d, want 0", got) + } + }) + } +} + +// TestCreate_adoptionDoesNotNeedTheTemplate: a template may be deleted once +// its clones exist (spec entry 4), so a retry that finds its VM must not +// fail on the template. +func TestCreate_adoptionDoesNotNeedTheTemplate(t *testing.T) { + t.Parallel() + for _, status := range []string{"running", "stopped"} { + t.Run(status, func(t *testing.T) { + t.Parallel() + fx := newFixture(t, 1) + fx.addOurVM(testVMID, testName, status) + if id := fx.create(fx.req()); id != fx.id() { + t.Errorf("Create = %q, want %q", id, fx.id()) + } + fx.assertRunning(testVMID) + }) + } +} + +func TestCreate_cloneTargetAndStorage(t *testing.T) { + t.Parallel() + tests := []struct { + name string + zone string + storage string + wantTarget string + wantStorage string + wantNode string + }{ + {name: "template's node, template's storage", zone: fakeNode, wantNode: fakeNode}, + {name: "another node, configured storage", zone: "pv1-2", storage: "fast-lvm", wantTarget: "pv1-2", wantStorage: "fast-lvm", wantNode: "pv1-2"}, + } + for _, tc := range tests { + t.Run(tc.name, func(t *testing.T) { + t.Parallel() + fx := newCreateFixture(t, 2) + fx.p.api = observe(fx.fake, testProvider) + fx.p.cfg.Nodes = []string{fakeNode, "pv1-2"} + fx.p.cfg.Storage = tc.storage + req := fx.req() + req.Placement.Zone = tc.zone + fx.create(req) + lines := fx.logs.matching(`"call"="Clone"`) + if len(lines) != 1 { + t.Fatalf("clone call lines = %d, want 1", len(lines)) + } + for _, want := range []string{`"target"="` + tc.wantTarget + `"`, `"storage"="` + tc.wantStorage + `"`, `"node"="` + fakeNode + `"`} { + if !strings.Contains(lines[0], want) { + t.Errorf("clone call line lacks %s: %s", want, lines[0]) + } + } + if vm := fx.assertRunning(slot(0)); vm.node != tc.wantNode { + t.Errorf("VM node = %s, want %s", vm.node, tc.wantNode) + } + }) + } +} + +// TestCreate_adoptedVMStaysOnItsNode: the ID is location-free, so a VM that +// migrated is carried on where it is, not where placement.zone points. +func TestCreate_adoptedVMStaysOnItsNode(t *testing.T) { + t.Parallel() + fx := newCreateFixture(t, 1) + fx.addOurVM(testVMID, testName, "stopped").node = "pv1-2" + if id := fx.create(fx.req()); id != fx.id() { + t.Errorf("Create = %q, want %q", id, fx.id()) + } + if vm := fx.assertRunning(testVMID); vm.node != "pv1-2" { + t.Errorf("VM node = %s, want pv1-2", vm.node) + } + if lines := fx.logs.matching(`"msg"="proxmox vm ensured running"`); len(lines) != 1 || !strings.Contains(lines[0], `"node"="pv1-2"`) { + t.Errorf("outcome lines = %v, want one naming pv1-2", lines) + } +} + +func TestCreate_notExistInsideCreateIsTransientNotNotFound(t *testing.T) { + t.Parallel() + for _, method := range []string{"SetVMConfig", "Start", "VMStatus"} { + t.Run(method, func(t *testing.T) { + t.Parallel() + fx := newCreateFixture(t, 1) + fx.fake.failWith(method, errVMMissing(fakeNode, slot(0))) + fx.createFails(fx.req(), provider.ErrTransient) + }) + } +} + +func TestCreate_cloudInitSentinelNeverLoggedOrInErrors(t *testing.T) { + t.Parallel() + const sentinel = "SENTINEL-s3cr3t-passw0rd-7f3a" + tests := []struct { + name string + setup func(fx *fixture) + }{ + {name: "success", setup: func(*fixture) {}}, + {name: "upload refused", setup: func(fx *fixture) { fx.fake.failWith("UploadISO", fakeErr(500, "fake: upload failed")) }}, + {name: "upload task failed", setup: func(fx *fixture) { + fx.fake.hookOn("TaskStatus", func() error { + if fx.fake.callCount("SetVMConfig") == 0 && fx.fake.callCount("UploadISO") > 0 { + return fakeErr(596, "fake: broken pipe") + } + return nil + }) + }}, + {name: "config refused", setup: func(fx *fixture) { fx.fake.failWith("SetVMConfig", errLocked("backup")) }}, + {name: "start failed", setup: func(fx *fixture) { fx.fake.startFailsWith = "start failed" }}, + } + for _, tc := range tests { + t.Run(tc.name, func(t *testing.T) { + t.Parallel() + fx := newCreateFixture(t, 2) + fx.p.api = observe(fx.fake, testProvider) + tc.setup(fx) + req := fx.req() + req.CloudInit = "#cloud-config\nwrite_files:\n - content: " + sentinel + "\n" + _, err := fx.p.Create(fx.ctx, req) + if err != nil && strings.Contains(err.Error(), sentinel) { + t.Errorf("error carries the cloud-init: %v", err) + } + lines := fx.logs.all() + for _, l := range lines { + if strings.Contains(l, sentinel) { + t.Errorf("log line carries the cloud-init: %s", l) + } + } + if got := fx.logs.matching(`"cloudInitBytes"=` + strconv.Itoa(len(req.CloudInit))); len(got) != 1 { + t.Errorf("cloudInitBytes lines = %d, want 1:\n%s", len(got), strings.Join(lines, "\n")) + } + }) + } +} + +func TestCreate_concurrentSameNameSerialised(t *testing.T) { + t.Parallel() + fx := newCreateFixture(t, 1) + release := fx.fake.blockOn("Clone") + ids := make([]string, 2) + errs := make([]error, 2) + var wg sync.WaitGroup + for i := range ids { + wg.Go(func() { ids[i], errs[i] = fx.p.Create(fx.ctx, fx.req()) }) + if i == 0 { + for fx.fake.callCount("Clone") == 0 { + time.Sleep(time.Millisecond) + } + } + } + close(release) + wg.Wait() + for i, err := range errs { + if err != nil { + t.Fatalf("Create %d error = %v", i, err) + } + } + if ids[0] != ids[1] { + t.Errorf("ids = %v, want the same VM for both", ids) + } + if got := fx.fake.callCount("Clone"); got != 1 { + t.Errorf("Clone calls = %d, want 1", got) + } +} + +func TestCreate_cancelledMidWaitReturnsPromptly(t *testing.T) { + t.Parallel() + fx := newCreateFixture(t, 1) + fx.p.budgets = defaultBudgets() + fx.p.budgets.poll = time.Millisecond + fx.fake.taskPolls = 1 << 30 + ctx, cancel := context.WithCancel(fx.ctx) + defer cancel() + fx.fake.hookOn("TaskStatus", func() error { + cancel() + return nil + }) + start := time.Now() + _, err := fx.p.Create(ctx, fx.req()) + if elapsed := time.Since(start); elapsed > 5*time.Second { + t.Errorf("Create took %v after cancellation", elapsed) + } + assertProviderError(t, err, "create", provider.ErrTransient) + if !errors.Is(err, context.Canceled) { + t.Errorf("error = %v, want it to wrap context.Canceled", err) + } +} + +func TestCreate_outcomeLine(t *testing.T) { + t.Parallel() + fx := newCreateFixture(t, 1) + fx.create(fx.req()) + lines := fx.logs.matching(`"msg"="proxmox vm ensured running"`) + if len(lines) != 1 { + t.Fatalf("outcome lines = %d, want 1:\n%s", len(lines), strings.Join(fx.logs.all(), "\n")) + } + for _, want := range []string{`"vmid"=` + strconv.Itoa(slot(0)), `"name"="` + testName + `"`, `"node"="` + fakeNode + `"`, `"provider"="` + testProvider + `"`} { + if !strings.Contains(lines[0], want) { + t.Errorf("outcome line lacks %s: %s", want, lines[0]) + } + } +} + +func TestCreate_quietAtV0(t *testing.T) { + t.Parallel() + fx := newCreateFixture(t, 0) + fx.p.api = observe(fx.fake, testProvider) + fx.create(fx.req()) + if lines := fx.logs.all(); len(lines) != 0 { + t.Errorf("V(0) lines = %v, want none", lines) + } +} diff --git a/internal/provider/proxmox/proxmox.go b/internal/provider/proxmox/proxmox.go index 1be34e8..5843857 100644 --- a/internal/provider/proxmox/proxmox.go +++ b/internal/provider/proxmox/proxmox.go @@ -173,14 +173,8 @@ func (p *Provider) lockName(ctx context.Context, name string) (func(), error) { return release, nil } -// Create and Delete are stubs until the phases that implement them (Create -// in 6, Delete in 7). Each answers Transient, so a Proxy on this provider -// retries rather than latching. - -func (p *Provider) Create(_ context.Context, req provider.CreateRequest) (string, error) { - return "", p.wrapErr("create", "", transientf("proxmox Create is not implemented yet (request %s)", req.Name)) -} - +// Delete is a stub until Phase 7. It answers Transient, so a Proxy on this +// provider retries rather than latching. func (p *Provider) Delete(_ context.Context, id string) error { return p.wrapErr("delete", id, transientf("proxmox Delete is not implemented yet")) } diff --git a/internal/provider/proxmox/proxmox_test.go b/internal/provider/proxmox/proxmox_test.go index 42d540e..502ff9d 100644 --- a/internal/provider/proxmox/proxmox_test.go +++ b/internal/provider/proxmox/proxmox_test.go @@ -216,17 +216,12 @@ func TestWrapErr_carriesProviderNameOpAndID(t *testing.T) { } // TestStubs_answerTransient keeps a Proxy on this provider retrying, not -// latched, until Create and Delete are implemented in later phases. +// latched, until Delete is implemented in Phase 7. func TestStubs_answerTransient(t *testing.T) { t.Parallel() - ctx := context.Background() p := newProvider("proxmox-home", testConfig(), nil) - _, createErr := p.Create(ctx, provider.CreateRequest{Name: "proxy-x"}) - deleteErr := p.Delete(ctx, "vms/150042/proxy-x") - for op, err := range map[string]error{"create": createErr, "delete": deleteErr} { - if !errors.Is(err, provider.ErrTransient) { - t.Errorf("%s error = %v, want ErrTransient", op, err) - } + if err := p.Delete(context.Background(), "vms/150042/proxy-x"); !errors.Is(err, provider.ErrTransient) { + t.Errorf("delete error = %v, want ErrTransient", err) } } diff --git a/internal/provider/proxmox/seed.go b/internal/provider/proxmox/seed.go index cb7dc20..dfdf9a0 100644 --- a/internal/provider/proxmox/seed.go +++ b/internal/provider/proxmox/seed.go @@ -2,9 +2,12 @@ package proxmox import ( "bytes" + "context" "errors" "fmt" + "github.com/go-logr/logr" + "gitea.home.hrajfrisbee.cz/kacerr/egress-proxies-operator/internal/provider" "gitea.home.hrajfrisbee.cz/kacerr/egress-proxies-operator/internal/provider/nocloud" ) @@ -41,3 +44,22 @@ func (p *Provider) buildSeed(req provider.CreateRequest) (*bytes.Buffer, error) } return &iso, nil } + +// uploadSeed puts a freshly built seed on the VM's node and waits for the +// copy (spec entry 5). An upload overwrites a file of the same name (F12), +// so what a cut-off attempt left is replaced without a delete first. +func (p *Provider) uploadSeed(ctx context.Context, log logr.Logger, node string, req provider.CreateRequest) error { + iso, err := p.buildSeed(req) + if err != nil { + return err + } + volid := seedVolID(p.cfg.ISOStorage, req.Name) + upid, err := p.api.UploadISO(ctx, node, p.cfg.ISOStorage, seedISOName(req.Name), iso, int64(iso.Len())) + if err != nil { + return fmt.Errorf("uploading seed %s: %w", volid, err) + } + if err := p.waitTask(ctx, log, upid, p.budgets.upload); err != nil { + return fmt.Errorf("uploading seed %s: %w", volid, err) + } + return nil +} diff --git a/internal/provider/proxmox/task.go b/internal/provider/proxmox/task.go index 3ebe972..542423e 100644 --- a/internal/provider/proxmox/task.go +++ b/internal/provider/proxmox/task.go @@ -89,7 +89,9 @@ func (p *Provider) waitTask(ctx context.Context, log logr.Logger, upid string, b } // waitUnlocked polls a VM's config until no task holds its lock and returns -// that config, for Create's ownership check (spec entry 2). A VM that +// that config, for Create's ownership check (spec entry 2). A suspended lock +// is returned at once: it marks a hibernated VM, not a task, and never +// clears on its own, so waiting would only burn the clone budget. A VM that // disappears meanwhile, as a failed clone does, returns the does-not-exist // error for the caller to judge. func (p *Provider) waitUnlocked(ctx context.Context, node string, vmid int, budget time.Duration) (vmConfig, error) { @@ -100,10 +102,27 @@ func (p *Provider) waitUnlocked(ctx context.Context, node string, vmid int, budg return false, fmt.Errorf("reading config of VM %d: %w", vmid, err) } cfg = c - return c["lock"] == "", nil + return c["lock"] == "" || c["lock"] == lockSuspended, nil }) if err != nil { return nil, err } return cfg, nil } + +// waitRunning polls status/current after the start task until QEMU runs +// the guest (spec entry 10, step 9). A VM that is stopped again is +// Transient at once: it will not come up on its own, and the next attempt +// adopts and starts it. +func (p *Provider) waitRunning(ctx context.Context, node string, vmid int) error { + return pollUntil(ctx, p.budgets.poll, p.budgets.running, fmt.Sprintf("VM %d to run", vmid), func() (bool, error) { + st, err := p.api.VMStatus(ctx, node, vmid) + if err != nil { + return false, fmt.Errorf("reading status of VM %d: %w", vmid, err) + } + if st.Status == "stopped" { + return false, transientf("VM %d stopped right after its start", vmid) + } + return st.Status == "running" && st.QMPStatus == "running", nil + }) +} diff --git a/internal/provider/proxmox/task_test.go b/internal/provider/proxmox/task_test.go index f25d401..a5c3b0b 100644 --- a/internal/provider/proxmox/task_test.go +++ b/internal/provider/proxmox/task_test.go @@ -160,8 +160,10 @@ func TestWaitUnlocked_table(t *testing.T) { cancelled bool wantErr func(error) bool wantCalls int + wantLock string }{ {name: "unlocked answers at once", wantCalls: 1}, + {name: "suspended lock is not waited on", lock: lockSuspended, budget: time.Hour, wantCalls: 1, wantLock: lockSuspended}, { name: "waits until the lock clears", lock: "clone", wantCalls: 3, onCall: func(f *fakeAPI, n int) { @@ -219,8 +221,8 @@ func TestWaitUnlocked_table(t *testing.T) { } cfg, err := fx.p.waitUnlocked(ctx, fakeNode, testVMID, budget) if tc.wantErr == nil { - if err != nil || cfg["name"] != testName || cfg["lock"] != "" { - t.Errorf("waitUnlocked = %v, %v; want the unlocked config", cfg, err) + if err != nil || cfg["name"] != testName || cfg["lock"] != tc.wantLock { + t.Errorf("waitUnlocked = %v, %v; want the config with lock %q", cfg, err, tc.wantLock) } } else if !tc.wantErr(err) || cfg != nil { t.Errorf("waitUnlocked = %v, %v; want no config and the expected error", cfg, err) From 2f89c1fffbdf1696beb9e6bcb0afad9f4384d4a9 Mon Sep 17 00:00:00 2001 From: kacerr Date: Wed, 23 Sep 2026 20:14:14 +0000 Subject: [PATCH 10/13] proxmox: Delete with seed cleanup, health Checker, CRD type comment Delete stops and destroys a VM of ours under the per-name lock, then deletes its seed on the VM's node, or on every node when no VM of ours was left. A hibernated VM of ours is Permanent, as in Create. The Checker verifies each node concurrently (version, node status, ISO storage, templates) and adds CertificateInvalid, CredentialsUnavailable, StorageNotFound and TemplateNotFound to the shared reasons. The ProviderHealth type comment gains proxmox. Co-Authored-By: Claude Opus 5.5 (1M context) --- api/v1alpha1/providerhealth_types.go | 2 +- .../crawl.example.com_providerhealths.yaml | 2 +- .../2026-09-18-1502-proxmox-provider.md | 152 +++- docs/proxmox-provider-spec.md | 68 +- internal/provider/proxmox/check.go | 218 ++++++ internal/provider/proxmox/check_test.go | 577 ++++++++++++++ internal/provider/proxmox/client.go | 8 +- internal/provider/proxmox/delete.go | 149 ++++ internal/provider/proxmox/delete_test.go | 717 ++++++++++++++++++ internal/provider/proxmox/fake_test.go | 15 +- internal/provider/proxmox/get_test.go | 58 ++ internal/provider/proxmox/proxmox.go | 6 - internal/provider/proxmox/proxmox_test.go | 10 - 13 files changed, 1936 insertions(+), 46 deletions(-) create mode 100644 internal/provider/proxmox/check.go create mode 100644 internal/provider/proxmox/check_test.go create mode 100644 internal/provider/proxmox/delete.go create mode 100644 internal/provider/proxmox/delete_test.go diff --git a/api/v1alpha1/providerhealth_types.go b/api/v1alpha1/providerhealth_types.go index c12e058..87ce8ac 100644 --- a/api/v1alpha1/providerhealth_types.go +++ b/api/v1alpha1/providerhealth_types.go @@ -68,7 +68,7 @@ type ProviderTarget struct { // ProviderHealthStatus is the result of the latest check of one provider. type ProviderHealthStatus struct { // type is the provider's type from the providers config: kubernetes, - // gcp, kvm or kvm2. + // gcp, kvm, kvm2 or proxmox. // +optional Type string `json:"type,omitempty"` diff --git a/config/crd/bases/crawl.example.com_providerhealths.yaml b/config/crd/bases/crawl.example.com_providerhealths.yaml index 2f207cf..d3f8084 100644 --- a/config/crd/bases/crawl.example.com_providerhealths.yaml +++ b/config/crd/bases/crawl.example.com_providerhealths.yaml @@ -183,7 +183,7 @@ spec: type: description: |- type is the provider's type from the providers config: kubernetes, - gcp, kvm or kvm2. + gcp, kvm, kvm2 or proxmox. type: string type: object type: object diff --git a/docs/plans-executions/2026-09-18-1502-proxmox-provider.md b/docs/plans-executions/2026-09-18-1502-proxmox-provider.md index 2eac296..9f403a2 100644 --- a/docs/plans-executions/2026-09-18-1502-proxmox-provider.md +++ b/docs/plans-executions/2026-09-18-1502-proxmox-provider.md @@ -11,7 +11,7 @@ Pairs with [docs/plans/2026-09-18-1502-proxmox-provider.md](../plans/2026-09-18- - [x] Phase 4 — Seam, HTTP client, decorator, fake - [x] Phase 5 — Get, ListByTag, task helpers - [x] Phase 6 — Create -- [ ] Phase 7 — Delete, Checker, CRD comment +- [x] Phase 7 — Delete, Checker, CRD comment - [ ] Phase 8 — Contract and observability sweep, integration test, testing.md - [ ] Phase 9 — Deployment and documentation - [ ] Phase 10 — Review, real-host validation, changelog, MR @@ -789,3 +789,153 @@ Worth noting: would find the VM. This stays untested, like every cross-node path (entry 24). - Nothing in this phase talked to the host. + +## Phase 7 — Delete, Checker, CRD comment + +`delete.go` holds `Delete`: parse, the per-name lock, `destroy` (resolve, +the lock and status checks, config, ownership, stop, destroy) and the seed +delete. `check.go` holds `Check`, which fans out one `checkNode` per node +with `wg.Go` and returns the verdicts in config order, plus `checkReason` +and the four provider reasons. The Delete stub and +`TestStubs_answerTransient` are gone, and `grep -rn 'not implemented' +internal/provider/proxmox/` is empty. All four contract methods and +`Check` are real. The `status.type` comment in +`api/v1alpha1/providerhealth_types.go` now ends `kvm2 or proxmox`, and +`make manifests` changed only that description line in the ProviderHealth +CRD. + +The spec won over the plan as the Phase 7 notes said. An absent VM still +gets its seed deleted, on every configured node. A task waits on the seed +delete, and nothing matches F13's absent-volume message. A paused VM gets +the stop too. The Checker's reasons include `CertificateInvalid` and +`CredentialsUnavailable`, and pveproxy's 595 and 596 are `Unreachable`. + +Open questions decided (spec entries 11 and 16 updated): + +- **Our hibernated VM (`lock: suspended`) is Permanent in Delete:** "resume + or destroy it in Proxmox", the same answer Create's ownership check + gives. Proxmox's destroy refuses a locked VM (the fake answers with F7's + `VM is locked (…)` text; the suspended variant was never captured), and + that lock never clears on its own. Transient would retry forever. A + Permanent from Delete in `reconcileManaged`'s Stopped branch goes through + `providerFailure` and latches the Proxy with the admin action in its + status. On the finalizer path `deletionFailure` keeps retrying, and the + retries can be seen. Nothing captured shows what a hibernated VM does on + a stop or start through the API, so there was no evidence for a + self-healing path (for example, start to resume, then stop and destroy). + A hibernated VM that is not ours is left alone. The resources-entry lock + check skips `suspended`, so ownership is judged from the config, as in + Get. +- **The Checker reads `p.cfg.TokenFile` itself, through the client's + `readToken`**, before its first request, and keeps only the part before + `=`. A seam method would break the rule of one method per endpoint, and + both `TestClient_methodPathAndParams` and + `TestObserve_everySeamMethodLogsOneLine` enumerate the interface. An + unusable file answers `CredentialsUnavailable` without a request. + `readToken`'s errors never carry the content. Every row of + `TestCheck_reasonsTable` runs at V(2) over the decorator with a sentinel + secret and asserts that the secret is in no message, detail or log line. + +Other judgment calls: + +- **The seed is deleted on the VM's node only when Delete destroyed a VM + of ours; otherwise on every node.** Entry 11 said "skip to step 7" for + a VM that is foreign, unmarked, or gone by the time the config is read, + without naming a node. In each of those cases our VM's node is unknown: + the node in the listing belongs to someone else's VM, or to one that + just vanished. A container at the VMID takes the same path. +- **Stop's `does not exist` means the VM is gone**, the same as Destroy's. + The seed is then deleted on its node, and Destroy is not called. +- **`errCAFile` joins `errCredentials` in `client.go`.** A missing CA file + or one without a certificate is `CertificateInvalid` ("fix `caFile`"). + Without the sentinel it would be `APIError`. Its class is unchanged + (Transient or Permanent, as entry 14 says). +- **Any 500 from the storage status call is `StorageNotFound`**, carrying + Proxmox's text. The plan matched a 500 "does not exist", but no response + for a missing storage was captured. Matching an invented string would go + against the fixture discipline, and every 500 there (missing, not + enabled on that node, offline) is fixed on the storage anyway. A 401, + 403, 595 or 596 keeps its shared reason. +- **`checkReason` puts `*net.OpError` before the deadline check**, so a + dial that runs out of its own 10 s budget is `Unreachable`, not + `Timeout`. A request timing out on its own while the round still runs + is `Timeout`. The round's ctx ending is `Timeout` whatever the error + says, and that guard is in `checkNode`, as in kvm2. +- **The outcome line `proxmox vm deleted` carries `destroyed`**, which is + false when there was no VM of ours left to destroy. It has no `node` in + that case. Entry 20 lists the new V(1) no-op lines. +- `templates` counts catalogue entries, not unique VMIDs. + +Test names. These plan names changed with the behaviour: +`TestDelete_absentIsNoOp` → `TestDelete_absentStillDeletesSeedOnEveryNode`, +`TestDelete_foreignNameIsNoOp` → `TestDelete_foreignNameLeavesVMAlone`, +`TestDelete_unmarkedIsNoOp` → `TestDelete_unmarkedLeavesVMAlone`, and +`TestDelete_toleratesMissingISO` merged into entry 21's +`TestDelete_seedDeleteIsTaskAndIdempotent`, which runs with the seed both +present and absent and ties the absent case to +`task_status_volume_missing.json`. Every other plan and entry 21 name +exists. `TestDelete_usesPurgeAndDestroyUnreferencedDisks` runs Delete over +the real `httpAPI` against the Phase 0 destroy captures +(`cluster_resources_during_destroy`, `config_during_destroy`, +`destroy_response`, `content_delete_response`, +`task_status_volume_missing`). `TestGet_duringDeleteStillReturnsInstance` +holds the destroy task in its first poll and first checks that the +mid-destroy captures show the VM listed, stopped and without a lock. +Tests beyond both lists: `TestDelete_containerAtVMIDIsLeftAlone`, +`TestDelete_hibernatedVMIsPermanentIfOurs`, +`TestDelete_seedDeleteFailureOnVMNodeFails`, +`TestDelete_seedDeleteNodeFailureSkippedWhenNodeUnknown`, +`TestDelete_goneMidwayIsSuccess`, `TestDelete_resourcesFailureIsTransient`, +`TestDelete_waitsOnTheNameLock` (a held name lock means no call at all, +which is what makes a Delete racing a Create wait), +`TestDelete_cancelledMidWaitReturnsPromptly`, `TestDelete_outcomeLine` and +`TestCheck_reasonsOverHTTPClient`, which takes the transport reasons from +real net/http and crypto/tls errors: unknown CA, hostname, missing CA file, +a replayed 401, 596, a closed server and a cut response. The fake gained +`destroyFailsWith`. New test helpers are only the small ones in +`delete_test.go` (`putSeed`, `seedPresent`, `delete`/`deleteFails`, +`seedDeleteNodes` read from the decorator's lines, `twoNodeFixture`) and +`check_test.go` (`newCheckFixture`, `assertNoSecret`, `setStorage`). +`TestCheck_nodesCheckedConcurrently` uses a barrier, not a timer: every +node's `Version` waits until all have arrived, so checking the nodes one +at a time fails the verdicts deterministically and does not measure +elapsed time. That avoids the kind of flake kvm2's 200 ms test has. + +The mutation check applied 34 mutations: 17 in `delete.go`, 16 in +`check.go` and 1 in `client.go`. Every one compiled and every one failed at +least one test. Three did not compile on the first try (an unused `upid` +or `tokenID`) and were redone. The first run of "never stop" did fail +tests, but it also **hung** `TestDelete_concurrentSameNameSafe`, whose +wait for the first `Stop` call could not end. The wait is now bounded by +the first Delete returning, and the rerun fails fast. One reasons row +("dial cut short by the round's deadline") was added before the run, +because no existing row would have caught dropping `checkNode`'s ctx +guard. Every mutation was reverted by a script that compared each file's +hash with the original. + +```bash +go test -race -count=2 ./internal/provider/proxmox/ ./internal/providerhealth/ # ok +make manifests && git diff --stat config/ # crawl.example.com_providerhealths.yaml | 2 +- +grep -rn 'not implemented' internal/provider/proxmox/ # nothing +go build ./... && go vet ./... && go vet -tags=integration ./... +go test -short -count=1 ./... # all ok +go test -count=1 -v ./internal/provider/proxmox/ | grep -c '^=== RUN' # 519 before, 649 after +bin/golangci-lint run --disable staticcheck --max-same-issues 0 --max-issues-per-linter 0 ./internal/provider/proxmox/... +``` + +Worth noting: + +- Lint defaults hide repeats: with `--max-same-issues 0` the logcheck hits + are 4 in `create.go`, 4 in the new `delete.go`, and 2, 1 and 1 in + `get.go`, `seed.go` and `task.go`, all the ctx-plus-logger shape. + Nothing else is new except goconst on test strings. A prealloc hit in + `delete_test.go` was fixed. +- `TestCreate_concurrentSameNameSerialised` has the same unbounded + `for callCount("Clone") == 0` wait that hung the Delete test under a + mutation. It is left for the Phase 8 sweep, because this phase does not + touch Create. +- Never seen on the host, for Phase 10: what the storage status call + answers for a missing or disabled storage, what destroy says for a + hibernated VM, and a stop on a paused VM. The try-every-node seed delete + and the multi-node Checker have only met the fake (entry 24). +- Nothing in this phase talked to the host. diff --git a/docs/proxmox-provider-spec.md b/docs/proxmox-provider-spec.md index b697917..7c22a78 100644 --- a/docs/proxmox-provider-spec.md +++ b/docs/proxmox-provider-spec.md @@ -381,22 +381,35 @@ the seed ISO; the VM's disks are removed by the destroy call itself, with `purge=1&destroy-unreferenced-disks=1`. Under the per-name lock: 1. Parse the ID (`ErrPermanent` if malformed). -2. `/cluster/resources`. VMID absent: skip to step 7, trying every configured - node because the VM's node is no longer known. -3. `lock` set or `status: unknown`: `ErrTransient`. A clone or another task - still running finishes first; the reconciler calls Delete again after - `DeletionPoll`. -4. Read the config. `does not exist`: skip to step 7. A different name, or - no mark: the VM is not ours; log it and skip to step 7 without touching - it. -5. Status other than `stopped`: `POST …/status/stop`, wait (stop budget). - Stop works on a paused VM too. +2. `/cluster/resources`. VMID absent, or held by a container: skip to + step 7, trying every configured node because the VM's node is no longer + known. +3. `lock` set, other than `suspended`, or `status: unknown`: `ErrTransient`. + A clone or another task still running finishes first; the reconciler + calls Delete again after `DeletionPoll`. +4. Read the config. `does not exist`: skip to step 7 on every node. A + different name, or no mark: the VM is not ours; log it at V(1) and skip + to step 7 on every node without touching it. Our VM with a `lock` other + than `suspended` (taken since the listing): `ErrTransient`. Our VM with + `lock: suspended` (hibernated): `ErrPermanent` asking the admin to resume + or destroy it, the same answer Create's ownership check gives. The + destroy refuses a locked VM, and that lock never clears on its own, so + Transient would retry forever; Permanent puts the admin action in the + Proxy's status, where `reconcileManaged` latches it and the finalizer + path keeps retrying it visibly. A hibernated VM that is not ours is left + alone like any other. +5. Status (from the listing) other than `stopped`: `POST …/status/stop`, + wait (stop budget). Stop works on a paused VM too, which the listing + shows as `running`. `does not exist` means the VM is gone. 6. `DELETE /nodes/{node}/qemu/{vmid}?purge=1&destroy-unreferenced-disks=1`, wait (destroy budget). `does not exist` is success. 7. `DELETE /nodes/{node}/storage/{isoStorage}/content/:iso/-seed.iso`, - wait (seed-delete budget). A missing volume is still a successful task, - so no message is matched. On the try-every-node path a node whose - request fails is logged and skipped. + wait (seed-delete budget), on the VM's node when steps 5 and 6 ran on a + VM of ours, and on every configured node in config order otherwise. A + missing volume is still a successful task, so no message is matched. On + the every-node path a node whose request or task fails is logged at V(1) + and skipped, unless ctx ended, which fails the call; on the VM's node a + failure fails the call (a 403 for `Datastore.Allocate` is Permanent). 8. Return nil. Everything is gone when Delete returns nil, so `Get` afterwards is @@ -593,18 +606,31 @@ failed for a network reason, or pveproxy answered 595 or 596); `APIError` (any other failure). Provider-specific, each fixed differently by an admin: -- `CertificateInvalid`: TLS verification failed. Fix `url` or `caFile`, not - the host. +- `CertificateInvalid`: TLS verification failed, or `caFile` is missing or + holds no certificate. Fix `url` or `caFile`, not the host. - `CredentialsUnavailable`: the token file is missing or malformed. Fix the Secret, not Proxmox. - `StorageNotFound`: `isoStorage` is missing, inactive, or has no `iso` - content. Fix the storage on the node. + content. Fix the storage on the node. Any 500 from the storage status + call counts as missing, whatever its wording (it does not exist, is not + enabled on that node, or is offline; no missing-storage response was + captured, so no text is matched), and the message carries Proxmox's + text. A 401, 403, 595 or 596 there keeps its shared reason. - `TemplateNotFound`: a configured template VMID does not exist or is not a template. Fix the template or the catalogue. Details: `pveVersion`, `kernel`, `tokenID` (the part before `=`, never the secret), `isoStorage`, `isoStorageAvailGiB`, `templates`, filled as far as -the check got. Cost: four requests per node per round. The `status.type` +the check got; `isoStorage` is always there. `templates` counts catalogue +entries. The Checker reads `tokenFile` itself, before the first request, +with the client's `readToken`: the seam stays one method per endpoint, and +a missing or malformed file answers `CredentialsUnavailable` without a +request. Only the part before `=` is kept. + +`Timeout` also covers a request that runs out of its own timeout while the +round is still running. A dial or a connection cut mid-response is +`Unreachable`, even when the dial ran out of its own 10 s budget, because +the node did not answer. Cost: four requests per node per round. The `status.type` comment in `api/v1alpha1/providerhealth_types.go` gains `proxmox` and the CRD is regenerated. @@ -713,14 +739,16 @@ named `proxmox`, with `provider` on every line and `vmid`, `name` and `node` on every line about an instance. - V(1): one line per operation outcome (`proxmox vm ensured running`, - `proxmox vm fetched` with state, ip and uid, `proxmox vm deleted`, + `proxmox vm fetched` with state, ip and uid, `proxmox vm deleted` with + `destroyed` (false when no VM of ours was left to destroy), `proxmox vms listed` with counts, `proxmox node checked` with the reason), one `proxmox api call` line per seam call from the decorator (`call`, `node`, `vmid`, `storage`, `upid`, `duration`, plus `newid` and `name` on a clone, `filename` on an upload and `volid` on a volume delete, and on failure `httpStatus` and `error`), a `proxmox task failed` line with the UPID and - `exitstatus`, and the no-op lines of Delete and ListByTag (foreign VM - left alone, unreadable candidate skipped). + `exitstatus`, and the no-op lines of Delete and ListByTag (VM already + gone, foreign or unmarked VM left alone, seed delete skipped on a node, + unreadable candidate skipped). - V(2): request parameters (without `description`'s content at V(1) and never the upload body), config summaries, agent addresses, the resolved Create parameters with `cloudInitBytes`. diff --git a/internal/provider/proxmox/check.go b/internal/provider/proxmox/check.go new file mode 100644 index 0000000..a4b7836 --- /dev/null +++ b/internal/provider/proxmox/check.go @@ -0,0 +1,218 @@ +package proxmox + +import ( + "context" + "errors" + "fmt" + "io" + "maps" + "net" + "net/http" + "slices" + "strconv" + "strings" + "sync" + "time" + + "gitea.home.hrajfrisbee.cz/kacerr/egress-proxies-operator/internal/provider" +) + +var _ provider.Checker = (*Provider)(nil) + +// Reasons only this provider can tell apart (spec entry 16). Each names a +// failure an admin fixes somewhere other than where the shared reasons +// point. +const ( + // reasonCertificateInvalid: TLS verification failed or the CA file is + // unusable; fix url or caFile, not the host. + reasonCertificateInvalid = "CertificateInvalid" + // reasonCredentialsUnavailable: the token file is missing or malformed; + // fix the Secret, not Proxmox. + reasonCredentialsUnavailable = "CredentialsUnavailable" + // reasonStorageNotFound: isoStorage is missing, inactive, or holds no + // iso content on the node; fix the storage on the node. + reasonStorageNotFound = "StorageNotFound" + // reasonTemplateNotFound: a configured template VMID does not exist or + // is not a template; fix the template or the catalogue. + reasonTemplateNotFound = "TemplateNotFound" +) + +// pveproxy's own answers when it cannot reach the node behind it. +const ( + statusProxyConnect = 595 + statusProxyBroken = 596 +) + +// Check verifies every configured node concurrently and returns one verdict +// per node in config order. The requests share the provider's HTTP client, +// but each is its own request, so a round that runs out of time cancels +// only its own requests and never a reconcile's. +func (p *Provider) Check(ctx context.Context) []provider.TargetStatus { + out := make([]provider.TargetStatus, len(p.cfg.Nodes)) + var wg sync.WaitGroup + for i, node := range p.cfg.Nodes { + wg.Go(func() { out[i] = p.checkNode(ctx, node) }) + } + wg.Wait() + return out +} + +func (p *Provider) checkNode(ctx context.Context, node string) provider.TargetStatus { + start := time.Now() + log := p.logger(ctx).WithValues("node", node) + ts := provider.TargetStatus{ + Name: node, + Address: p.cfg.URL, + Details: map[string]string{"isoStorage": p.cfg.ISOStorage}, + } + err := p.probeNode(ctx, node, ts.Details) + ts.Latency = time.Since(start) + switch { + case err == nil: + ts.Ready = true + ts.Reason = provider.ReasonReachable + case ctx.Err() != nil: + // The round ran out of time, whichever request was running. + ts.Reason = provider.ReasonTimeout + ts.Message = err.Error() + default: + ts.Reason = checkReason(err) + ts.Message = err.Error() + } + log.V(1).Info("proxmox node checked", "ready", ts.Ready, "reason", ts.Reason, "latency", ts.Latency) + return ts +} + +// probeNode runs spec entry 16's four requests and fills details as facts +// arrive, so a failure late in the sequence still reports what came before. +func (p *Provider) probeNode(ctx context.Context, node string, details map[string]string) error { + // The token file is read here only for its id: the secret after "=" + // is dropped at once and never reaches details, logs or errors. + token, err := readToken(p.cfg.TokenFile) + if err != nil { + return err + } + tokenID, _, _ := strings.Cut(token, "=") + details["tokenID"] = tokenID + + v, err := p.api.Version(ctx) + if err != nil { + return fmt.Errorf("reading the API version: %w", err) + } + details["pveVersion"] = v.Version + ns, err := p.api.NodeStatus(ctx, node) + if err != nil { + return fmt.Errorf("reading the status of node %s: %w", node, err) + } + details["kernel"] = ns.CurrentKernel.Release + if err := p.checkStorage(ctx, node, details); err != nil { + return err + } + return p.checkTemplates(ctx, details) +} + +// checkStorage reads isoStorage on the node. A 500 means Proxmox cannot +// use that storage on this node (it does not exist, is not enabled here, +// or is offline), whatever its wording; the message carries Proxmox's +// text. +func (p *Provider) checkStorage(ctx context.Context, node string, details map[string]string) error { + storage := p.cfg.ISOStorage + st, err := p.api.StorageStatus(ctx, node, storage) + var ae *apiError + if errors.As(err, &ae) && ae.Status == http.StatusInternalServerError { + return &checkFailure{reason: reasonStorageNotFound, msg: fmt.Sprintf("storage %s on node %s: %s", storage, node, ae.Message)} + } + if err != nil { + return fmt.Errorf("reading storage %s on node %s: %w", storage, node, err) + } + details["isoStorageAvailGiB"] = formatGiB(st.Avail) + if st.Active == 0 { + return &checkFailure{reason: reasonStorageNotFound, msg: fmt.Sprintf("storage %s is not active on node %s", storage, node)} + } + if !slices.Contains(strings.Split(st.Content, ","), "iso") { + return &checkFailure{reason: reasonStorageNotFound, msg: fmt.Sprintf("storage %s on node %s does not hold iso content (content: %s)", storage, node, st.Content)} + } + return nil +} + +// checkTemplates looks every catalogue entry up in the cluster's resources: +// a template may sit on any node, since a clone can target another. +func (p *Provider) checkTemplates(ctx context.Context, details map[string]string) error { + rs, err := p.api.ClusterResources(ctx) + if err != nil { + return fmt.Errorf("listing cluster resources: %w", err) + } + isTemplate := make(map[int]bool) + for _, r := range rs { + if r.Type == "qemu" { + isTemplate[r.VMID] = r.Template == 1 + } + } + var problems []string + for _, key := range slices.Sorted(maps.Keys(p.cfg.Templates)) { + vmid := p.cfg.Templates[key].VMID + switch tmpl, found := isTemplate[vmid]; { + case !found: + problems = append(problems, fmt.Sprintf("template %q: VM %d does not exist", key, vmid)) + case !tmpl: + problems = append(problems, fmt.Sprintf("template %q: VM %d is not a template", key, vmid)) + } + } + details["templates"] = fmt.Sprintf("%d/%d", len(p.cfg.Templates)-len(problems), len(p.cfg.Templates)) + if len(problems) > 0 { + return &checkFailure{reason: reasonTemplateNotFound, msg: strings.Join(problems, "; ")} + } + return nil +} + +// checkFailure is a verdict the backend answered but the check refuses: the +// storage or a template is not usable. +type checkFailure struct { + reason string + msg string +} + +func (e *checkFailure) Error() string { return e.msg } + +// checkReason maps a failed probe to a reason once checkNode has ruled out +// the round's ctx ending. A failed connect or a cut connection is +// Unreachable; a request timeout with the round still running is Timeout. +func checkReason(err error) string { + var ( + cf *checkFailure + ae *apiError + op *net.OpError + ne net.Error + ) + switch { + case errors.As(err, &cf): + return cf.reason + case errors.Is(err, errCredentials): + return reasonCredentialsUnavailable + case errors.Is(err, errCAFile), isTLSVerification(err): + return reasonCertificateInvalid + case errors.As(err, &ae): + switch ae.Status { + case http.StatusUnauthorized: + return provider.ReasonUnauthorized + case http.StatusForbidden: + return provider.ReasonForbidden + case statusProxyConnect, statusProxyBroken: + return provider.ReasonUnreachable + } + return provider.ReasonAPIError + case errors.As(err, &op): + // Checked before the deadline: a dial that runs out of its own + // budget is a node that did not answer. + return provider.ReasonUnreachable + case errors.Is(err, context.DeadlineExceeded), errors.Is(err, context.Canceled): + return provider.ReasonTimeout + case errors.As(err, &ne), errors.Is(err, io.EOF), errors.Is(err, io.ErrUnexpectedEOF): + return provider.ReasonUnreachable + } + return provider.ReasonAPIError +} + +func formatGiB(bytes int64) string { + return strconv.FormatFloat(float64(bytes)/(1<<30), 'f', 1, 64) +} diff --git a/internal/provider/proxmox/check_test.go b/internal/provider/proxmox/check_test.go new file mode 100644 index 0000000..79b9609 --- /dev/null +++ b/internal/provider/proxmox/check_test.go @@ -0,0 +1,577 @@ +package proxmox + +import ( + "context" + "crypto/x509" + "errors" + "fmt" + "io" + "maps" + "net" + "net/http" + "net/url" + "path/filepath" + "slices" + "strings" + "sync" + "syscall" + "testing" + "time" + + "gitea.home.hrajfrisbee.cz/kacerr/egress-proxies-operator/internal/provider" +) + +// checkSecret is the secret half of the Checker's token file; it must never +// reach a verdict or a log line. +const checkSecret = "SENTINEL-token-secret-4c1d" + +// newCheckFixture is an observed create fixture at V(2) with a token file +// whose secret is checkSecret. +func newCheckFixture(t *testing.T) *fixture { + t.Helper() + fx := newCreateFixture(t, 2) + fx.p.api = observe(fx.fake, testProvider) + fx.p.cfg.TokenFile = filepath.Join(t.TempDir(), "token") + writeFile(t, fx.p.cfg.TokenFile, "proxy-operator@pve!operator="+checkSecret+"\n") + return fx +} + +// assertNoSecret checks every verdict and every log line for checkSecret. +func assertNoSecret(t *testing.T, fx *fixture, got []provider.TargetStatus) { + t.Helper() + for _, ts := range got { + if strings.Contains(ts.Message, checkSecret) || slices.ContainsFunc(slices.Collect(maps.Values(ts.Details)), func(v string) bool { return strings.Contains(v, checkSecret) }) { + t.Errorf("verdict carries the token secret: %+v", ts) + } + } + for _, l := range fx.logs.all() { + if strings.Contains(l, checkSecret) { + t.Errorf("log line carries the token secret: %s", l) + } + } +} + +func TestCheck_oneEntryPerNodeInConfigOrder(t *testing.T) { + t.Parallel() + fx := newCheckFixture(t) + fx.p.cfg.Nodes = []string{"pv1-3", fakeNode, "pv1-2"} + fx.fake.nodes["pv1-3"] = fx.fake.nodes[fakeNode] + got := fx.p.Check(fx.ctx) + if len(got) != len(fx.p.cfg.Nodes) { + t.Fatalf("Check() returned %d targets, want %d", len(got), len(fx.p.cfg.Nodes)) + } + for i, node := range fx.p.cfg.Nodes { + ts := got[i] + if ts.Name != node || ts.Address != fx.p.cfg.URL { + t.Errorf("target %d = %s (%s), want %s (%s)", i, ts.Name, ts.Address, node, fx.p.cfg.URL) + } + if wantReady := node != "pv1-2"; ts.Ready != wantReady { + t.Errorf("target %s ready = %v, want %v (%s: %s)", node, ts.Ready, wantReady, ts.Reason, ts.Message) + } + } + if got[2].Reason != provider.ReasonAPIError || !strings.Contains(got[2].Message, "pv1-2") { + t.Errorf("pv1-2 verdict = %s: %s, want APIError naming the node", got[2].Reason, got[2].Message) + } +} + +// TestCheck_nodesCheckedConcurrently holds every node's first request +// until all of them have arrived; checked one at a time, the first would +// give up and fail its verdict. +func TestCheck_nodesCheckedConcurrently(t *testing.T) { + t.Parallel() + fx := newCheckFixture(t) + fx.p.cfg.Nodes = []string{fakeNode, "pv1-2", "pv1-3"} + for _, n := range fx.p.cfg.Nodes[1:] { + fx.fake.nodes[n] = fx.fake.nodes[fakeNode] + } + var ( + mu sync.Mutex + arrived int + ) + all := make(chan struct{}) + fx.fake.hookOn("Version", func() error { + mu.Lock() + if arrived++; arrived == len(fx.p.cfg.Nodes) { + close(all) + } + mu.Unlock() + select { + case <-all: + return nil + case <-time.After(2 * time.Second): + return errors.New("barrier: the nodes were not checked concurrently") + } + }) + for _, ts := range fx.p.Check(fx.ctx) { + if !ts.Ready { + t.Errorf("target %s = %s: %s, want Ready", ts.Name, ts.Reason, ts.Message) + } + } +} + +func TestCheck_expiredContextReturnsPromptly(t *testing.T) { + t.Parallel() + fx := newCheckFixture(t) + fx.p.cfg.Nodes = []string{fakeNode, "pv1-2"} + fx.fake.blockOn("Version") + ctx, cancel := context.WithTimeout(fx.ctx, 0) + defer cancel() + start := time.Now() + got := fx.p.Check(ctx) + if elapsed := time.Since(start); elapsed > time.Second { + t.Errorf("Check() took %v with an expired ctx, want under 1s", elapsed) + } + if len(got) != 2 { + t.Fatalf("Check() returned %d targets, want 2", len(got)) + } + for _, ts := range got { + if ts.Ready || ts.Reason != provider.ReasonTimeout || !strings.Contains(ts.Message, context.DeadlineExceeded.Error()) { + t.Errorf("target %s = ready %v reason %s (%s), want Timeout naming the deadline", ts.Name, ts.Ready, ts.Reason, ts.Message) + } + } +} + +func TestCheck_reasonsTable(t *testing.T) { + t.Parallel() + urlErr := func(err error) error { + return &url.Error{Op: "Get", URL: "https://192.0.2.8:8006/api2/json/version", Err: err} + } + tests := []struct { + name string + prep func(fx *fixture) + timeout time.Duration + cancel bool + + wantReady bool + wantReason string + wantMessage string + }{ + {name: "ok", wantReady: true, wantReason: provider.ReasonReachable}, + { + name: "token file missing", + prep: func(fx *fixture) { fx.p.cfg.TokenFile = filepath.Join(fx.t.TempDir(), "absent") }, + wantReason: reasonCredentialsUnavailable, + wantMessage: "does not exist yet", + }, + { + name: "token file malformed", + prep: func(fx *fixture) { writeFile(fx.t, fx.p.cfg.TokenFile, checkSecret+"\n") }, + wantReason: reasonCredentialsUnavailable, + wantMessage: "is not @!=", + }, + { + name: "token file gone by the time the client reads it", + prep: func(fx *fixture) { + fx.fake.failWith("NodeStatus", fmt.Errorf("%w: %w: /etc/proxy-operator/proxmox/token does not exist yet", provider.ErrTransient, errCredentials)) + }, + wantReason: reasonCredentialsUnavailable, + }, + { + name: "unknown certificate authority", + prep: func(fx *fixture) { fx.fake.failWith("Version", urlErr(x509.UnknownAuthorityError{})) }, + wantReason: reasonCertificateInvalid, + }, + { + name: "host not in the certificate", + prep: func(fx *fixture) { fx.fake.failWith("Version", urlErr(x509.HostnameError{Host: "192.168.2.8"})) }, + wantReason: reasonCertificateInvalid, + }, + { + name: "CA file unusable", + prep: func(fx *fixture) { + fx.fake.failWith("Version", fmt.Errorf("%w: %w: reading CA file: %w", provider.ErrTransient, errCAFile, syscall.ENOENT)) + }, + wantReason: reasonCertificateInvalid, + }, + { + name: "wrong secret", + prep: func(fx *fixture) { fx.fake.failWith("Version", loadAPIError(fx.t, "err_unauthorized")) }, + wantReason: provider.ReasonUnauthorized, + }, + { + name: "no credentials sent", + prep: func(fx *fixture) { fx.fake.failWith("Version", loadAPIError(fx.t, "err_no_credentials")) }, + wantReason: provider.ReasonUnauthorized, + }, + { + name: "privilege missing", + prep: func(fx *fixture) { fx.fake.failWith("NodeStatus", loadAPIError(fx.t, "err_forbidden")) }, + wantReason: provider.ReasonForbidden, + wantMessage: "Sys.Syslog", + }, + { + name: "pveproxy cannot reach the node", + prep: func(fx *fixture) { fx.fake.failWith("NodeStatus", fakeErr(595, "Connection refused")) }, + wantReason: provider.ReasonUnreachable, + }, + { + name: "pveproxy broken pipe", + prep: func(fx *fixture) { fx.fake.failWith("NodeStatus", fakeErr(596, "Broken pipe")) }, + wantReason: provider.ReasonUnreachable, + }, + { + name: "connection refused", + prep: func(fx *fixture) { + fx.fake.failWith("Version", urlErr(&net.OpError{Op: "dial", Net: "tcp", Err: syscall.ECONNREFUSED})) + }, + wantReason: provider.ReasonUnreachable, + }, + { + name: "node status 500", + prep: func(fx *fixture) { fx.fake.failWith("NodeStatus", fakeErr(500, "fake: pve-cluster not running")) }, + wantReason: provider.ReasonAPIError, + }, + { + name: "iso storage missing", + prep: func(fx *fixture) { + fx.fake.mu.Lock() + delete(fx.fake.storages, fakeISOStorage) + fx.fake.mu.Unlock() + }, + wantReason: reasonStorageNotFound, + wantMessage: "storage proxy-seeds on node pv1-1: fake: storage 'proxy-seeds' does not exist", + }, + { + name: "iso storage inactive", + prep: func(fx *fixture) { setStorage(fx, func(s *storageStatus) { s.Active = 0 }) }, + wantReason: reasonStorageNotFound, + wantMessage: "not active", + }, + { + name: "iso storage without iso content", + prep: func(fx *fixture) { setStorage(fx, func(s *storageStatus) { s.Content = "images,rootdir,isoish" }) }, + wantReason: reasonStorageNotFound, + wantMessage: "does not hold iso content (content: images,rootdir,isoish)", + }, + { + name: "iso storage forbidden", + prep: func(fx *fixture) { fx.fake.failWith("StorageStatus", loadAPIError(fx.t, "err_forbidden")) }, + wantReason: provider.ReasonForbidden, + }, + { + name: "template missing", + prep: func(fx *fixture) { + fx.fake.mu.Lock() + delete(fx.fake.vms, testTemplateVMID) + fx.fake.mu.Unlock() + }, + wantReason: reasonTemplateNotFound, + wantMessage: `template "debian-13": VM 9001 does not exist`, + }, + { + name: "template VMID holds a plain VM", + prep: func(fx *fixture) { fx.fake.vms[testTemplateVMID].template = false }, + wantReason: reasonTemplateNotFound, + wantMessage: `template "debian-13": VM 9001 is not a template`, + }, + { + name: "template VMID holds a container", + prep: func(fx *fixture) { fx.fake.vms[testTemplateVMID].kind = "lxc" }, + wantReason: reasonTemplateNotFound, + wantMessage: "does not exist", + }, + { + name: "resources listing fails", + prep: func(fx *fixture) { + fx.fake.failWith("ClusterResources", fakeErr(500, "fake: cluster filesystem not ready")) + }, + wantReason: provider.ReasonAPIError, + }, + { + name: "request outlives the deadline", + prep: func(fx *fixture) { fx.fake.blockOn("StorageStatus") }, + timeout: 50 * time.Millisecond, + wantReason: provider.ReasonTimeout, + }, + { + // The real client's dial sees the round's ctx end and fails as + // a dial error, which checkReason alone would call Unreachable. + name: "dial cut short by the round's deadline", + prep: func(fx *fixture) { + fx.fake.hookOn("Version", func() error { + time.Sleep(200 * time.Millisecond) + return urlErr(&net.OpError{Op: "dial", Net: "tcp", Err: context.DeadlineExceeded}) + }) + }, + timeout: 20 * time.Millisecond, + wantReason: provider.ReasonTimeout, + }, + { + name: "ctx cancelled mid check", + prep: func(fx *fixture) { fx.fake.blockOn("ClusterResources") }, + cancel: true, + wantReason: provider.ReasonTimeout, + }, + } + for _, tc := range tests { + t.Run(tc.name, func(t *testing.T) { + t.Parallel() + fx := newCheckFixture(t) + if tc.prep != nil { + tc.prep(fx) + } + ctx := fx.ctx + switch { + case tc.timeout > 0: + var cancel context.CancelFunc + ctx, cancel = context.WithTimeout(ctx, tc.timeout) + defer cancel() + case tc.cancel: + var cancel context.CancelFunc + ctx, cancel = context.WithCancel(ctx) + time.AfterFunc(50*time.Millisecond, cancel) + } + got := fx.p.Check(ctx) + if len(got) != 1 { + t.Fatalf("Check() returned %d targets, want 1", len(got)) + } + ts := got[0] + if ts.Ready != tc.wantReady || ts.Reason != tc.wantReason { + t.Errorf("verdict = ready %v reason %s (%s), want ready %v reason %s", ts.Ready, ts.Reason, ts.Message, tc.wantReady, tc.wantReason) + } + if !tc.wantReady && ts.Message == "" { + t.Error("failed check has no message") + } + if tc.wantReady && ts.Message != "" { + t.Errorf("successful check has a message: %q", ts.Message) + } + if tc.wantMessage != "" && !strings.Contains(ts.Message, tc.wantMessage) { + t.Errorf("message = %q, want it to contain %q", ts.Message, tc.wantMessage) + } + if ts.Details["isoStorage"] != fakeISOStorage { + t.Errorf("details = %v, want isoStorage whatever failed", ts.Details) + } + assertNoSecret(t, fx, got) + }) + } +} + +func setStorage(fx *fixture, fn func(s *storageStatus)) { + fx.fake.mu.Lock() + defer fx.fake.mu.Unlock() + s := fx.fake.storages[fakeISOStorage] + fn(&s) + fx.fake.storages[fakeISOStorage] = s +} + +// TestCheck_reasonsOverHTTPClient takes the transport-level reasons from +// the real client, so the errors checkReason sees are the ones net/http and +// crypto/tls produce, and the Ready row replays the Phase 0 captures. +func TestCheck_reasonsOverHTTPClient(t *testing.T) { + t.Parallel() + captures := func(t *testing.T) http.HandlerFunc { + return func(w http.ResponseWriter, r *http.Request) { + path := strings.TrimPrefix(r.URL.Path, "/api2/json/") + name := map[string]string{ + "version": "version", + "nodes/pv1-1/status": "node_status", + "nodes/pv1-1/storage/local/status": "storage_status", + "cluster/resources": "cluster_resources", + }[path] + if name == "" { + t.Errorf("unexpected request %s", path) + http.NotFound(w, r) + return + } + serveJSON(w, readCapture(t, name).Body) + } + } + tests := []struct { + name string + handler func(t *testing.T) http.HandlerFunc + mutate func(t *testing.T, cfx *clientFixture) + wantReason string + }{ + {name: "captures", handler: captures, wantReason: provider.ReasonReachable}, + {name: "unknown authority", handler: captures, mutate: func(_ *testing.T, cfx *clientFixture) { cfx.cfg.CAFile = "" }, wantReason: reasonCertificateInvalid}, + { + name: "host not in the certificate", + handler: captures, + mutate: func(_ *testing.T, cfx *clientFixture) { + cfx.cfg.URL = strings.Replace(cfx.srv.URL, "127.0.0.1", "localhost", 1) + }, + wantReason: reasonCertificateInvalid, + }, + { + name: "CA file missing", + handler: captures, + mutate: func(t *testing.T, cfx *clientFixture) { cfx.cfg.CAFile = filepath.Join(t.TempDir(), "ca.pem") }, + wantReason: reasonCertificateInvalid, + }, + { + name: "401 from pveproxy", + handler: func(t *testing.T) http.HandlerFunc { + return func(w http.ResponseWriter, _ *http.Request) { serveCapture(t, w, readCapture(t, "err_unauthorized")) } + }, + wantReason: provider.ReasonUnauthorized, + }, + { + name: "596 from pveproxy", + handler: func(t *testing.T) http.HandlerFunc { + return func(w http.ResponseWriter, _ *http.Request) { + serveCapture(t, w, capture{Status: 596, Reason: "Broken pipe"}) + } + }, + wantReason: provider.ReasonUnreachable, + }, + { + name: "server gone", + handler: captures, + mutate: func(_ *testing.T, cfx *clientFixture) { cfx.srv.Close() }, + wantReason: provider.ReasonUnreachable, + }, + { + name: "connection cut mid response", + handler: func(t *testing.T) http.HandlerFunc { + return func(w http.ResponseWriter, _ *http.Request) { + conn, _, err := w.(http.Hijacker).Hijack() + if err != nil { + t.Errorf("hijack: %v", err) + return + } + _, _ = io.WriteString(conn, "HTTP/1.1 200 OK\r\nContent-Length: 100\r\n\r\n{\"data\":") + _ = conn.Close() + } + }, + wantReason: provider.ReasonUnreachable, + }, + } + for _, tc := range tests { + t.Run(tc.name, func(t *testing.T) { + t.Parallel() + cfx := newClientFixture(t, tc.handler(t)) + if tc.mutate != nil { + tc.mutate(t, cfx) + } + cfg := testConfig() + cfg.ISOStorage = "local" + cfg.URL, cfg.TokenFile, cfg.CAFile = cfx.cfg.URL, cfx.cfg.TokenFile, cfx.cfg.CAFile + p := newProvider(testProvider, cfg, cfx.api(t)) + got := p.Check(context.Background()) + if len(got) != 1 { + t.Fatalf("Check() returned %d targets, want 1", len(got)) + } + if ts := got[0]; ts.Reason != tc.wantReason || ts.Ready != (tc.wantReason == provider.ReasonReachable) { + t.Errorf("verdict = ready %v reason %s (%s), want %s", ts.Ready, ts.Reason, ts.Message, tc.wantReason) + } + }) + } +} + +func TestCheck_detailsKeys(t *testing.T) { + t.Parallel() + full := map[string]string{ + "tokenID": "proxy-operator@pve!operator", "isoStorage": fakeISOStorage, "pveVersion": "9.2.20", + "kernel": "7.0.14-17-pve", "isoStorageAvailGiB": "90.0", "templates": "1/1", + } + tests := []struct { + name string + prep func(fx *fixture) + want []string + }{ + {name: "success", want: slices.Sorted(maps.Keys(full))}, + {name: "token unusable", prep: func(fx *fixture) { writeFile(fx.t, fx.p.cfg.TokenFile, "junk") }, want: []string{"isoStorage"}}, + {name: "node status fails", prep: func(fx *fixture) { fx.fake.failWith("NodeStatus", fakeErr(596, "Broken pipe")) }, want: []string{"isoStorage", "pveVersion", "tokenID"}}, + {name: "storage inactive", prep: func(fx *fixture) { setStorage(fx, func(s *storageStatus) { s.Active = 0 }) }, want: []string{"isoStorage", "isoStorageAvailGiB", "kernel", "pveVersion", "tokenID"}}, + {name: "template missing", prep: func(fx *fixture) { fx.fake.vms[testTemplateVMID].template = false }, want: slices.Sorted(maps.Keys(full))}, + } + for _, tc := range tests { + t.Run(tc.name, func(t *testing.T) { + t.Parallel() + fx := newCheckFixture(t) + if tc.prep != nil { + tc.prep(fx) + } + got := fx.p.Check(fx.ctx)[0] + if keys := slices.Sorted(maps.Keys(got.Details)); !slices.Equal(keys, tc.want) { + t.Errorf("details keys = %v, want %v", keys, tc.want) + } + for k, v := range got.Details { + if want, ok := full[k]; ok && tc.name == "success" && v != want { + t.Errorf("details[%s] = %q, want %q", k, v, want) + } + if v == "" { + t.Errorf("details[%s] is empty", k) + } + } + }) + } + t.Run("templates counts found over configured", func(t *testing.T) { + t.Parallel() + fx := newCheckFixture(t) + fx.p.cfg.Templates["ubuntu"] = provider.ProxmoxTemplate{VMID: 9002} + fx.p.cfg.Templates["debian-again"] = provider.ProxmoxTemplate{VMID: testTemplateVMID} + got := fx.p.Check(fx.ctx)[0] + if got.Details["templates"] != "2/3" || got.Reason != reasonTemplateNotFound || !strings.Contains(got.Message, `template "ubuntu": VM 9002`) { + t.Errorf("verdict = %s %q, details %v; want TemplateNotFound naming ubuntu, templates 2/3", got.Reason, got.Message, got.Details) + } + }) +} + +func TestCheck_logsVerdictPerNode(t *testing.T) { + t.Parallel() + fx := newCheckFixture(t) + fx.p.cfg.Nodes = []string{fakeNode, "pv1-2"} + fx.p.Check(fx.ctx) + checked := fx.logs.matching(`"msg"="proxmox node checked"`) + if len(checked) != 2 { + t.Fatalf("got %d verdict lines, want 2: %v", len(checked), checked) + } + for _, l := range checked { + switch { + case strings.Contains(l, `"node"="pv1-1"`): + if !strings.Contains(l, `"ready"=true`) || !strings.Contains(l, `"reason"="Reachable"`) || !strings.Contains(l, `"latency"=`) { + t.Errorf("pv1-1 verdict line = %s", l) + } + case strings.Contains(l, `"node"="pv1-2"`): + if !strings.Contains(l, `"ready"=false`) || !strings.Contains(l, `"reason"="APIError"`) { + t.Errorf("pv1-2 verdict line = %s", l) + } + default: + t.Errorf("verdict line without a node: %s", l) + } + if !strings.Contains(l, `"provider"="`+testProvider+`"`) || !strings.Contains(l, `"level"=1`) { + t.Errorf("verdict line lacks the provider or is not V(1): %s", l) + } + } +} + +func TestCheckReason_table(t *testing.T) { + t.Parallel() + dial := &url.Error{Op: "Get", URL: "https://192.0.2.8:8006/api2/json/version", Err: &net.OpError{Op: "dial", Net: "tcp", Err: context.DeadlineExceeded}} + tests := []struct { + name string + err error + want string + }{ + {name: "storage verdict", err: &checkFailure{reason: reasonStorageNotFound, msg: "x"}, want: reasonStorageNotFound}, + {name: "template verdict wrapped", err: fmt.Errorf("check: %w", &checkFailure{reason: reasonTemplateNotFound, msg: "x"}), want: reasonTemplateNotFound}, + {name: "token missing", err: fmt.Errorf("%w: %w: /t does not exist yet", provider.ErrTransient, errCredentials), want: reasonCredentialsUnavailable}, + {name: "token malformed", err: fmt.Errorf("%w: %w: /t is not a token", provider.ErrPermanent, errCredentials), want: reasonCredentialsUnavailable}, + {name: "CA file", err: fmt.Errorf("%w: %w: /ca holds no PEM certificate", provider.ErrPermanent, errCAFile), want: reasonCertificateInvalid}, + {name: "unknown authority", err: &url.Error{Err: x509.UnknownAuthorityError{}}, want: reasonCertificateInvalid}, + {name: "certificate invalid", err: &url.Error{Err: x509.CertificateInvalidError{Reason: x509.Expired}}, want: reasonCertificateInvalid}, + {name: "401", err: loadAPIError(t, "err_unauthorized"), want: provider.ReasonUnauthorized}, + {name: "403 wrapped", err: fmt.Errorf("reading: %w", loadAPIError(t, "err_forbidden_volume_delete")), want: provider.ReasonForbidden}, + {name: "595", err: fakeErr(595, "Connection refused"), want: provider.ReasonUnreachable}, + {name: "596", err: fakeErr(596, "Broken pipe"), want: provider.ReasonUnreachable}, + {name: "400", err: loadAPIError(t, "err_bad_param"), want: provider.ReasonAPIError}, + {name: "500", err: loadAPIError(t, "err_vm_missing"), want: provider.ReasonAPIError}, + {name: "dial ran out of its own budget", err: dial, want: provider.ReasonUnreachable}, + {name: "connection reset", err: &url.Error{Err: &net.OpError{Op: "read", Net: "tcp", Err: syscall.ECONNRESET}}, want: provider.ReasonUnreachable}, + {name: "request timeout", err: fmt.Errorf("reading: %w", &url.Error{Err: context.DeadlineExceeded}), want: provider.ReasonTimeout}, + {name: "cancelled", err: context.Canceled, want: provider.ReasonTimeout}, + {name: "server closed mid response", err: &url.Error{Err: io.ErrUnexpectedEOF}, want: provider.ReasonUnreachable}, + {name: "eof", err: io.EOF, want: provider.ReasonUnreachable}, + {name: "decode failure", err: errors.New("decoding GET version: invalid character '<'"), want: provider.ReasonAPIError}, + {name: "nil", err: nil, want: provider.ReasonAPIError}, + } + for _, tc := range tests { + t.Run(tc.name, func(t *testing.T) { + t.Parallel() + if got := checkReason(tc.err); got != tc.want { + t.Errorf("checkReason(%v) = %s, want %s", tc.err, got, tc.want) + } + }) + } +} diff --git a/internal/provider/proxmox/client.go b/internal/provider/proxmox/client.go index c1df283..940ac2c 100644 --- a/internal/provider/proxmox/client.go +++ b/internal/provider/proxmox/client.go @@ -39,6 +39,10 @@ const ( // Checker can report CredentialsUnavailable instead of an API failure. var errCredentials = errors.New("proxmox token file unusable") +// errCAFile marks a CA file that is missing or holds no certificate, so the +// Checker can report CertificateInvalid: the admin fixes caFile. +var errCAFile = errors.New("proxmox CA file unusable") + // tokenPattern is @!= (spec entry 14). var tokenPattern = regexp.MustCompile(`^[^@\s]+@[^!\s]+![^=\s]+=\S+$`) @@ -91,11 +95,11 @@ func (a *httpAPI) httpClient() (*http.Client, error) { if a.caFile != "" && !a.insecure { pem, err := os.ReadFile(a.caFile) if err != nil { - return nil, fmt.Errorf("%w: reading CA file: %w", provider.ErrTransient, err) + return nil, fmt.Errorf("%w: %w: reading CA file: %w", provider.ErrTransient, errCAFile, err) } pool := x509.NewCertPool() if !pool.AppendCertsFromPEM(pem) { - return nil, permanentf("CA file %s holds no PEM certificate", a.caFile) + return nil, fmt.Errorf("%w: %w: %s holds no PEM certificate", provider.ErrPermanent, errCAFile, a.caFile) } tlsCfg.RootCAs = pool } diff --git a/internal/provider/proxmox/delete.go b/internal/provider/proxmox/delete.go new file mode 100644 index 0000000..309324e --- /dev/null +++ b/internal/provider/proxmox/delete.go @@ -0,0 +1,149 @@ +package proxmox + +import ( + "context" + "fmt" + "time" + + "github.com/go-logr/logr" +) + +// Delete is spec entry 11: under the per-name lock, stop and destroy the VM +// when it is ours, then delete its seed. Everything already gone is +// success, and nothing of ours is left when nil is returned, so Get +// afterwards is NotFound. +func (p *Provider) Delete(ctx context.Context, id string) error { + vmid, name, err := parseProviderID(id) + if err != nil { + return p.wrapErr("delete", id, err) + } + log := p.logger(ctx).WithValues("vmid", vmid, "name", name) + unlock, err := p.lockName(ctx, name) + if err != nil { + return p.wrapErr("delete", id, err) + } + defer unlock() + + node, err := p.destroy(ctx, log, vmid, name) + if err != nil { + return p.wrapErr("delete", id, err) + } + if node == "" { + err = p.deleteSeedEverywhere(ctx, log, name) + } else { + log = log.WithValues("node", node) + err = p.deleteSeed(ctx, log, node, name) + } + if err != nil { + return p.wrapErr("delete", id, err) + } + log.V(1).Info("proxmox vm deleted", "destroyed", node != "") + return nil +} + +// destroy is steps 2 to 6. It returns the node of the VM it destroyed, or +// "" when there was no VM of ours to destroy: then the seed's node is not +// known, and the seed is looked for on every node. +func (p *Provider) destroy(ctx context.Context, log logr.Logger, vmid int, name string) (string, error) { + r, ok, err := p.resolve(ctx, vmid) + if err != nil { + return "", err + } + if !ok { + log.V(1).Info("proxmox vm already gone") + return "", nil + } + log = log.WithValues("node", r.Node) + if r.Lock != "" && r.Lock != lockSuspended { + return "", transientf("VM %d is locked (%s)", vmid, r.Lock) + } + if r.Status == "unknown" { + return "", transientf("VM %d has status unknown: its node %s is not answering", vmid, r.Node) + } + cfg, err := p.api.VMConfig(ctx, r.Node, vmid) + if isNotExist(err) { + log.V(1).Info("proxmox vm already gone") + return "", nil + } + if err != nil { + return "", fmt.Errorf("reading config of VM %d: %w", vmid, err) + } + if cfg["name"] != name { + log.V(1).Info("proxmox vm at this vmid is not ours, left alone", "vmName", cfg["name"]) + return "", nil + } + if !parseDescription(cfg["description"]).Managed { + log.V(1).Info("proxmox vm carries no proxy-operator mark, left alone") + return "", nil + } + switch lock := cfg["lock"]; lock { + case "": + case lockSuspended: + // Destroy refuses a locked VM and the lock never clears on its own, + // so only an admin can move this on, as in Create's ownership check. + return "", permanentf("VM %d named %s is hibernated (lock suspended); resume or destroy it in Proxmox", vmid, name) + default: + return "", transientf("VM %d is locked (%s)", vmid, lock) + } + + // The listing shows a paused VM as running, and Stop ends it too. + if r.Status != "stopped" { + gone, err := p.runTask(ctx, log, "stopping", vmid, p.budgets.stop, func() (string, error) { return p.api.Stop(ctx, r.Node, vmid) }) + if err != nil || gone { + return r.Node, err + } + } + if _, err := p.runTask(ctx, log, "destroying", vmid, p.budgets.destroy, func() (string, error) { return p.api.Destroy(ctx, r.Node, vmid) }); err != nil { + return "", err + } + return r.Node, nil +} + +// runTask submits a task on a VM and waits for it. A VM that is gone by the +// time the task is submitted reports gone: it is what the task was for. +func (p *Provider) runTask(ctx context.Context, log logr.Logger, what string, vmid int, budget time.Duration, submit func() (string, error)) (bool, error) { + upid, err := submit() + if isNotExist(err) { + log.V(1).Info("proxmox vm already gone") + return true, nil + } + if err != nil { + return false, fmt.Errorf("%s VM %d: %w", what, vmid, err) + } + if err := p.waitTask(ctx, log, upid, budget); err != nil { + return false, fmt.Errorf("%s VM %d: %w", what, vmid, err) + } + return false, nil +} + +// deleteSeed is step 7 on one node. A missing volume still ends in a +// successful task (F13), so nothing is matched. +func (p *Provider) deleteSeed(ctx context.Context, log logr.Logger, node, name string) error { + volid := seedVolID(p.cfg.ISOStorage, name) + upid, err := p.api.DeleteVolume(ctx, node, p.cfg.ISOStorage, volid) + if err != nil { + return fmt.Errorf("deleting seed %s on %s: %w", volid, node, err) + } + if err := p.waitTask(ctx, log, upid, p.budgets.seedDelete); err != nil { + return fmt.Errorf("deleting seed %s on %s: %w", volid, node, err) + } + return nil +} + +// deleteSeedEverywhere is step 7 when the VM's node is not known: every +// configured node is tried, and one that fails is logged and skipped, so a +// node that is down or lacks the storage cannot hold the Delete up. An +// ended ctx fails the call instead. +func (p *Provider) deleteSeedEverywhere(ctx context.Context, log logr.Logger, name string) error { + for _, node := range p.cfg.Nodes { + err := p.deleteSeed(ctx, log, node, name) + if err == nil { + continue + } + if ctx.Err() != nil { + return err + } + log.V(1).Info("proxmox seed delete failed on a node, skipped", "node", node, "reason", err.Error()) + } + return nil +} diff --git a/internal/provider/proxmox/delete_test.go b/internal/provider/proxmox/delete_test.go new file mode 100644 index 0000000..bd621c4 --- /dev/null +++ b/internal/provider/proxmox/delete_test.go @@ -0,0 +1,717 @@ +package proxmox + +import ( + "context" + "errors" + "net/http" + "slices" + "strconv" + "strings" + "sync" + "testing" + "time" + + "gitea.home.hrajfrisbee.cz/kacerr/egress-proxies-operator/internal/provider" +) + +// putSeed uploads testName's seed to the fake's ISO storage, as Create +// leaves it. +func (fx *fixture) putSeed() { + fx.t.Helper() + upid, err := fx.fake.UploadISO(context.Background(), fakeNode, fakeISOStorage, seedISOName(testName), strings.NewReader("seed"), 4) + if err != nil { + fx.t.Fatalf("UploadISO error = %v", err) + } + pollTask(fx.t, fx.fake, upid) +} + +func (fx *fixture) seedPresent() bool { + _, ok := fx.fake.iso(fakeISOStorage, seedISOName(testName)) + return ok +} + +// deleteFails runs Delete on fx.id(), checks the error's class and returns +// its text. +func (fx *fixture) deleteFails(class error) string { + fx.t.Helper() + err := fx.p.Delete(fx.ctx, fx.id()) + assertProviderError(fx.t, err, "delete", class) + return err.Error() +} + +func (fx *fixture) delete() { + fx.t.Helper() + if err := fx.p.Delete(fx.ctx, fx.id()); err != nil { + fx.t.Fatalf("Delete error = %v", err) + } +} + +// seedDeleteNodes returns the node of every DeleteVolume call, read from +// the decorator's lines; the fixture's api must be observed. +func (fx *fixture) seedDeleteNodes() []string { + lines := fx.logs.matching(`"call"="DeleteVolume"`) + nodes := make([]string, 0, len(lines)) + for _, l := range lines { + _, rest, _ := strings.Cut(l, `"node"="`) + node, _, _ := strings.Cut(rest, `"`) + nodes = append(nodes, node) + } + return nodes +} + +// twoNodeFixture is an observed fixture at V(1) with nodes pv1-1 and pv1-2. +func twoNodeFixture(t *testing.T) *fixture { + t.Helper() + fx := newFixture(t, 1) + fx.p.api = observe(fx.fake, testProvider) + fx.p.cfg.Nodes = []string{fakeNode, "pv1-2"} + return fx +} + +func TestDelete_malformedIDIsPermanent(t *testing.T) { + t.Parallel() + for _, id := range []string{"", "vms/150042", "vms/0150042/" + testName, "vm/150042/" + testName, "vms/abc/" + testName, "vms/150042/a/b", "vms/99/" + testName} { + t.Run(id, func(t *testing.T) { + t.Parallel() + fx := newFixture(t, 1) + assertProviderError(t, fx.p.Delete(fx.ctx, id), "delete", provider.ErrPermanent) + if calls := fx.fake.callLog(); len(calls) != 0 { + t.Errorf("calls = %v, want none", calls) + } + }) + } +} + +func TestDelete_twiceSucceeds(t *testing.T) { + t.Parallel() + fx := newFixture(t, 1) + fx.addOurVM(testVMID, testName, "running") + fx.putSeed() + fx.delete() + fx.delete() + if _, ok := fx.fake.vmSnapshot(testVMID); ok { + t.Error("VM still exists") + } + if fx.seedPresent() { + t.Error("seed still exists") + } + for method, want := range map[string]int{"Stop": 1, "Destroy": 1, "DeleteVolume": 2} { + if got := fx.fake.callCount(method); got != want { + t.Errorf("%s calls = %d, want %d", method, got, want) + } + } +} + +// TestDelete_absentStillDeletesSeedOnEveryNode replaces the plan's +// TestDelete_absentIsNoOp: spec entry 11 deletes the seed even when the VM +// is gone, on every node, because the VM's node is no longer known. +func TestDelete_absentStillDeletesSeedOnEveryNode(t *testing.T) { + t.Parallel() + fx := twoNodeFixture(t) + fx.putSeed() + fx.delete() + if got := fx.seedDeleteNodes(); !slices.Equal(got, []string{fakeNode, "pv1-2"}) { + t.Errorf("seed deleted on %v, want every node in config order", got) + } + if fx.seedPresent() { + t.Error("seed still exists") + } + for _, method := range []string{"VMConfig", "Stop", "Destroy"} { + if n := fx.fake.callCount(method); n != 0 { + t.Errorf("%s calls = %d, want 0 for an absent VM", method, n) + } + } + if lines := fx.logs.matching(`"msg"="proxmox vm already gone"`); len(lines) != 1 { + t.Errorf("already-gone lines = %v, want one", lines) + } +} + +func TestDelete_containerAtVMIDIsLeftAlone(t *testing.T) { + t.Parallel() + fx := twoNodeFixture(t) + fx.addOurVM(testVMID, testName, "running").kind = "lxc" + fx.delete() + if vm, ok := fx.fake.vmSnapshot(testVMID); !ok || vm.status != "running" { + t.Errorf("container = %+v, %v; want it untouched", vm, ok) + } + if got := fx.seedDeleteNodes(); len(got) != 2 { + t.Errorf("seed deleted on %v, want every node", got) + } +} + +// TestDelete_foreignNameLeavesVMAlone replaces the plan's +// TestDelete_foreignNameIsNoOp: the VM is not touched, but our seed is +// still deleted, on every node since ours is gone. +func TestDelete_foreignNameLeavesVMAlone(t *testing.T) { + t.Parallel() + fx := twoNodeFixture(t) + fx.addOurVM(testVMID, "proxy-someoneelse00000", "running") + fx.putSeed() + fx.delete() + if vm, ok := fx.fake.vmSnapshot(testVMID); !ok || vm.status != "running" { + t.Errorf("foreign VM = %+v, %v; want it running and untouched", vm, ok) + } + for _, method := range []string{"Stop", "Destroy", "SetVMConfig"} { + if n := fx.fake.callCount(method); n != 0 { + t.Errorf("%s calls = %d, want 0", method, n) + } + } + if got := fx.seedDeleteNodes(); !slices.Equal(got, []string{fakeNode, "pv1-2"}) { + t.Errorf("seed deleted on %v, want every node", got) + } + lines := fx.logs.matching(`"msg"="proxmox vm at this vmid is not ours, left alone"`) + if len(lines) != 1 || !strings.Contains(lines[0], `"vmName"="proxy-someoneelse00000"`) { + t.Errorf("no-op lines = %v, want one naming the other VM", lines) + } +} + +// TestDelete_unmarkedLeavesVMAlone replaces the plan's +// TestDelete_unmarkedIsNoOp, for the same reason. +func TestDelete_unmarkedLeavesVMAlone(t *testing.T) { + t.Parallel() + for name, desc := range map[string]string{ + "no description": "", + "managed false": "proxy-operator-managed=false\nproxy-operator-uid=" + testUID, + } { + t.Run(name, func(t *testing.T) { + t.Parallel() + fx := twoNodeFixture(t) + vm := fx.fake.addVM(testVMID, testName, "stopped") + if desc != "" { + vm.config["description"] = desc + } + fx.delete() + if _, ok := fx.fake.vmSnapshot(testVMID); !ok { + t.Error("unmarked VM was destroyed") + } + if n := fx.fake.callCount("Destroy"); n != 0 { + t.Errorf("Destroy calls = %d, want 0", n) + } + if got := fx.seedDeleteNodes(); len(got) != 2 { + t.Errorf("seed deleted on %v, want every node", got) + } + if lines := fx.logs.matching(`"msg"="proxmox vm carries no proxy-operator mark, left alone"`); len(lines) != 1 { + t.Errorf("no-op lines = %v, want one", lines) + } + }) + } +} + +// TestDelete_runningVMStoppedFirst covers a paused VM too: the listing +// shows it as running (F14), and Stop ends it. The fake refuses to destroy +// a running VM, and each task takes two polls, so a Destroy that does not +// wait for the stop fails. +func TestDelete_runningVMStoppedFirst(t *testing.T) { + t.Parallel() + for _, qmp := range []string{"running", "paused", "prelaunch"} { + t.Run(qmp, func(t *testing.T) { + t.Parallel() + fx := newFixture(t, 1) + fx.fake.taskPolls = 2 + fx.addOurVM(testVMID, testName, "running").qmpstatus = qmp + fx.delete() + log := fx.fake.callLog() + stop, destroy := slices.Index(log, "Stop"), slices.Index(log, "Destroy") + if stop < 0 || destroy < stop { + t.Fatalf("calls = %v, want Stop before Destroy", log) + } + if n := callsBefore(log[stop:], "TaskStatus", "Destroy"); n != fx.fake.taskPolls+1 { + t.Errorf("stop task polled %d times before Destroy, want %d", n, fx.fake.taskPolls+1) + } + if _, ok := fx.fake.vmSnapshot(testVMID); ok { + t.Error("VM still exists") + } + }) + } +} + +func TestDelete_stoppedVMSkipsStop(t *testing.T) { + t.Parallel() + fx := newFixture(t, 1) + fx.addOurVM(testVMID, testName, "stopped") + fx.delete() + want := []string{"ClusterResources", "VMConfig", "Destroy", "TaskStatus", "DeleteVolume", "TaskStatus"} + if got := fx.fake.callLog(); !slices.Equal(got, want) { + t.Errorf("calls = %v, want %v", got, want) + } +} + +func TestDelete_lockedIsTransient(t *testing.T) { + t.Parallel() + for _, lock := range []string{"backup", "migrate", "snapshot", "rollback", "create"} { + t.Run("resources lock "+lock, func(t *testing.T) { + t.Parallel() + fx := newFixture(t, 1) + fx.addOurVM(testVMID, testName, "running").lock = lock + fx.putSeed() + if msg := fx.deleteFails(provider.ErrTransient); !strings.Contains(msg, "locked ("+lock+")") { + t.Errorf("error = %s, want it to name the lock", msg) + } + if got := fx.fake.callLog(); !slices.Equal(got, []string{"UploadISO", "TaskStatus", "ClusterResources"}) { + t.Errorf("calls = %v, want only the listing after the seed setup", got) + } + if !fx.seedPresent() { + t.Error("seed deleted while the VM is locked") + } + }) + } + t.Run("mid-clone nameless entry", func(t *testing.T) { + t.Parallel() + fx := newFixture(t, 1) + fx.fake.addTemplate(9001, "debian-13-template") + if _, err := fx.fake.Clone(fx.ctx, fakeNode, 9001, cloneParams{NewID: testVMID, Name: testName, Description: formatDescription(testUID, testCreatedAt)}); err != nil { + t.Fatalf("Clone error = %v", err) + } + fx.deleteFails(provider.ErrTransient) + if _, ok := fx.fake.vmSnapshot(testVMID); !ok { + t.Error("mid-clone VM is gone") + } + }) + t.Run("lock taken between the resources and config reads", func(t *testing.T) { + t.Parallel() + fx := newFixture(t, 1) + vm := fx.addOurVM(testVMID, testName, "running") + mutateOn(fx.fake, "VMConfig", func() { vm.lock = "backup" }) + fx.deleteFails(provider.ErrTransient) + if n := fx.fake.callCount("Stop"); n != 0 { + t.Errorf("Stop calls = %d, want 0", n) + } + }) +} + +func TestDelete_unknownStatusIsTransient(t *testing.T) { + t.Parallel() + fx := newFixture(t, 1) + fx.addOurVM(testVMID, testName, "running").status = "unknown" + if msg := fx.deleteFails(provider.ErrTransient); !strings.Contains(msg, fakeNode) { + t.Errorf("error = %s, want it to name the node", msg) + } + if n := fx.fake.callCount("VMConfig"); n != 0 { + t.Errorf("VMConfig calls = %d, want 0 while the node is unknown", n) + } +} + +// TestDelete_hibernatedVMIsPermanentIfOurs matches Create's ownership +// check: Destroy refuses the suspended lock, which never clears on its own, +// so an admin must act. A hibernated VM that is not ours is left alone. +func TestDelete_hibernatedVMIsPermanentIfOurs(t *testing.T) { + t.Parallel() + t.Run("ours", func(t *testing.T) { + t.Parallel() + fx := newFixture(t, 1) + fx.addOurVM(testVMID, testName, "stopped").lock = lockSuspended + fx.putSeed() + if msg := fx.deleteFails(provider.ErrPermanent); !strings.Contains(msg, "resume or destroy it in Proxmox") { + t.Errorf("error = %s, want the admin action", msg) + } + for _, method := range []string{"Stop", "Destroy", "DeleteVolume"} { + if n := fx.fake.callCount(method); n != 0 { + t.Errorf("%s calls = %d, want 0", method, n) + } + } + if _, err := fx.fake.Destroy(fx.ctx, fakeNode, testVMID); !isLocked(err) { + t.Errorf("fake Destroy on the hibernated VM error = %v, want the lock answer", err) + } + }) + t.Run("someone else's", func(t *testing.T) { + t.Parallel() + fx := newFixture(t, 1) + fx.addOurVM(testVMID, "proxy-someoneelse00000", "stopped").lock = lockSuspended + fx.delete() + if vm, ok := fx.fake.vmSnapshot(testVMID); !ok || vm.lock != lockSuspended { + t.Errorf("foreign hibernated VM = %+v, %v; want it untouched", vm, ok) + } + }) +} + +func TestDelete_removesSeedISO(t *testing.T) { + t.Parallel() + fx := twoNodeFixture(t) + fx.addOurVM(testVMID, testName, "stopped").node = "pv1-2" + fx.putSeed() + fx.delete() + if fx.seedPresent() { + t.Error("seed still exists") + } + if got := fx.seedDeleteNodes(); !slices.Equal(got, []string{"pv1-2"}) { + t.Errorf("seed deleted on %v, want only the VM's node pv1-2", got) + } + lines := fx.logs.matching(`"call"="DeleteVolume"`) + if len(lines) != 1 || !strings.Contains(lines[0], `"volid"="`+seedVolID(fakeISOStorage, testName)+`"`) || !strings.Contains(lines[0], `"storage"="`+fakeISOStorage+`"`) { + t.Errorf("DeleteVolume lines = %v, want the seed's volid on the ISO storage", lines) + } +} + +// TestDelete_seedDeleteIsTaskAndIdempotent replaces the plan's +// TestDelete_toleratesMissingISO: Proxmox ends a delete of a missing volume +// with a successful imgdel task (F13), so nothing is matched, but the task +// is still waited on. +func TestDelete_seedDeleteIsTaskAndIdempotent(t *testing.T) { + t.Parallel() + missing := decodeData[taskStatus](t, "task_status_volume_missing") + if missing.Type != "imgdel" || missing.ExitStatus != "OK" { + t.Fatalf("capture task_status_volume_missing = %+v, want an imgdel task ending OK", missing) + } + for _, withSeed := range []bool{true, false} { + t.Run("seed present "+strconv.FormatBool(withSeed), func(t *testing.T) { + t.Parallel() + fx := newFixture(t, 1) + fx.addOurVM(testVMID, testName, "stopped") + if withSeed { + fx.putSeed() + } + fx.fake.taskPolls = 2 + fx.delete() + log := fx.fake.callLog() + last := len(log) - 1 + for log[last] != "DeleteVolume" { + last-- + } + if n := callsBefore(log[last:], "TaskStatus", ""); n != fx.fake.taskPolls+1 { + t.Errorf("seed delete task polled %d times, want %d: %v", n, fx.fake.taskPolls+1, log) + } + if fx.seedPresent() { + t.Error("seed still exists") + } + }) + } +} + +func TestDelete_seedDeleteFailureOnVMNodeFails(t *testing.T) { + t.Parallel() + tests := []struct { + name string + setup func(fx *fixture) + class error + }{ + {name: "token lacks Datastore.Allocate", setup: func(fx *fixture) { + fx.fake.failWith("DeleteVolume", loadAPIError(t, "err_forbidden_volume_delete")) + }, class: provider.ErrPermanent}, + {name: "pveproxy 596", setup: func(fx *fixture) { fx.fake.failWith("DeleteVolume", fakeErr(596, "Broken pipe")) }, class: provider.ErrTransient}, + {name: "task failed", setup: func(fx *fixture) { + fx.fake.hookOn("TaskStatus", func() error { + if fx.fake.callCount("DeleteVolume") > 0 { + return fakeErr(500, "fake: task log unreadable") + } + return nil + }) + }, class: provider.ErrTransient}, + } + for _, tc := range tests { + t.Run(tc.name, func(t *testing.T) { + t.Parallel() + fx := newFixture(t, 1) + fx.addOurVM(testVMID, testName, "stopped") + tc.setup(fx) + fx.deleteFails(tc.class) + if _, ok := fx.fake.vmSnapshot(testVMID); ok { + t.Error("VM still exists: the destroy should have run before the seed delete") + } + }) + } +} + +func TestDelete_seedDeleteNodeFailureSkippedWhenNodeUnknown(t *testing.T) { + t.Parallel() + t.Run("one node refuses", func(t *testing.T) { + t.Parallel() + fx := twoNodeFixture(t) + fx.putSeed() + var calls int + fx.fake.hookOn("DeleteVolume", func() error { + if calls++; calls == 1 { + return fakeErr(500, "fake: storage 'proxy-seeds' is not available on node 'pv1-1'") + } + return nil + }) + fx.delete() + if got := fx.seedDeleteNodes(); !slices.Equal(got, []string{fakeNode, "pv1-2"}) { + t.Errorf("seed deleted on %v, want both nodes tried", got) + } + lines := fx.logs.matching(`"msg"="proxmox seed delete failed on a node, skipped"`) + if len(lines) != 1 || !strings.Contains(lines[0], `"node"="pv1-1"`) || !strings.Contains(lines[0], "not available") { + t.Errorf("skip lines = %v, want one naming pv1-1 and the reason", lines) + } + if fx.seedPresent() { + t.Error("seed still exists after pv1-2's delete") + } + }) + t.Run("ended ctx fails the call", func(t *testing.T) { + t.Parallel() + fx := twoNodeFixture(t) + ctx, cancel := context.WithCancel(fx.ctx) + defer cancel() + fx.fake.hookOn("DeleteVolume", func() error { + cancel() + return context.Canceled + }) + err := fx.p.Delete(ctx, fx.id()) + assertProviderError(t, err, "delete", provider.ErrTransient) + if n := fx.fake.callCount("DeleteVolume"); n != 1 { + t.Errorf("DeleteVolume calls = %d, want 1: the ended ctx stops the loop", n) + } + }) +} + +func TestDelete_destroyTaskFailureIsTransient(t *testing.T) { + t.Parallel() + fx := newFixture(t, 1) + fx.addOurVM(testVMID, testName, "stopped") + fx.putSeed() + fx.fake.destroyFailsWith = "fake: unable to remove disk" + if msg := fx.deleteFails(provider.ErrTransient); !strings.Contains(msg, "unable to remove disk") { + t.Errorf("error = %s, want the task's exitstatus", msg) + } + if _, ok := fx.fake.vmSnapshot(testVMID); !ok { + t.Error("VM gone after a failed destroy task") + } + if !fx.seedPresent() { + t.Error("seed deleted although the VM is still there") + } + if lines := fx.logs.matching(`"msg"="proxmox task failed"`); len(lines) != 1 { + t.Errorf("task failed lines = %v, want one", lines) + } +} + +// TestDelete_goneMidwayIsSuccess: the VM vanishing at any step counts as +// done, and F2's does-not-exist answer is what each step then sees. +func TestDelete_goneMidwayIsSuccess(t *testing.T) { + t.Parallel() + tests := []struct { + name string + status string + method string + wantDestroy int + wantNodes []string + }{ + {name: "before the config read", status: "stopped", method: "VMConfig", wantNodes: []string{fakeNode, "pv1-2"}}, + {name: "before the stop", status: "running", method: "Stop", wantNodes: []string{fakeNode}}, + {name: "before the destroy", status: "stopped", method: "Destroy", wantDestroy: 1, wantNodes: []string{fakeNode}}, + } + for _, tc := range tests { + t.Run(tc.name, func(t *testing.T) { + t.Parallel() + fx := twoNodeFixture(t) + fx.addOurVM(testVMID, testName, tc.status) + mutateOn(fx.fake, tc.method, func() { delete(fx.fake.vms, testVMID) }) + fx.delete() + if n := fx.fake.callCount("Destroy"); n != tc.wantDestroy { + t.Errorf("Destroy calls = %d, want %d", n, tc.wantDestroy) + } + if got := fx.seedDeleteNodes(); !slices.Equal(got, tc.wantNodes) { + t.Errorf("seed deleted on %v, want %v", got, tc.wantNodes) + } + }) + } +} + +func TestDelete_resourcesFailureIsTransient(t *testing.T) { + t.Parallel() + fx := newFixture(t, 1) + fx.addOurVM(testVMID, testName, "running") + fx.fake.failWith("ClusterResources", fakeErr(595, "Connection refused")) + fx.deleteFails(provider.ErrTransient) + if n := fx.fake.callCount("DeleteVolume"); n != 0 { + t.Errorf("DeleteVolume calls = %d, want 0 when the VM's state is unknown", n) + } +} + +// TestDelete_usesPurgeAndDestroyUnreferencedDisks runs Delete over the real +// client against Phase 0's destroy captures: the DELETE carries both flags +// and the seed delete sends the volid as one escaped segment. +func TestDelete_usesPurgeAndDestroyUnreferencedDisks(t *testing.T) { + t.Parallel() + const ( + vmid = 199999 + name = "proxy-reconaaaaaaaaaaa" + ) + var ( + mu sync.Mutex + seen []string + ) + cfx := newClientFixture(t, func(w http.ResponseWriter, r *http.Request) { + path := strings.TrimPrefix(r.URL.EscapedPath(), "/api2/json/") + mu.Lock() + seen = append(seen, r.Method+" "+path+"?"+r.URL.RawQuery) + mu.Unlock() + capture := "" + switch { + case r.Method == http.MethodGet && path == "cluster/resources": + capture = "cluster_resources_during_destroy" + case r.Method == http.MethodGet && path == "nodes/pv1-1/qemu/199999/config": + capture = "config_during_destroy" + case r.Method == http.MethodDelete && path == "nodes/pv1-1/qemu/199999": + capture = "destroy_response" + case r.Method == http.MethodDelete && path == "nodes/pv1-1/storage/proxy-seeds/content/proxy-seeds:iso%2F"+name+"-seed.iso": + capture = "content_delete_response" + case r.Method == http.MethodGet && strings.Contains(path, ":imgdel:"): + capture = "task_status_volume_missing" + case r.Method == http.MethodGet && strings.Contains(path, ":qmdestroy:"): + capture = "task_status_stopped_ok" + default: + t.Errorf("unexpected request %s %s", r.Method, path) + http.Error(w, "unexpected", http.StatusNotImplemented) + return + } + serveJSON(w, readCapture(t, capture).Body) + }) + cfg := testConfig() + cfg.URL, cfg.TokenFile, cfg.CAFile = cfx.cfg.URL, cfx.cfg.TokenFile, cfx.cfg.CAFile + p := newProvider(testProvider, cfg, cfx.api(t)) + p.budgets = testBudgets() + if err := p.Delete(context.Background(), formatProviderID(vmid, name)); err != nil { + t.Fatalf("Delete error = %v", err) + } + mu.Lock() + defer mu.Unlock() + if !slices.Contains(seen, "DELETE nodes/pv1-1/qemu/199999?destroy-unreferenced-disks=1&purge=1") { + t.Errorf("requests = %v, want the destroy with purge=1 and destroy-unreferenced-disks=1", seen) + } + if !slices.ContainsFunc(seen, func(s string) bool { return strings.HasPrefix(s, "DELETE nodes/pv1-1/storage/proxy-seeds/content/") }) { + t.Errorf("requests = %v, want the seed delete", seen) + } + if slices.ContainsFunc(seen, func(s string) bool { return strings.Contains(s, "status/stop") }) { + t.Errorf("requests = %v, want no stop for the captured stopped VM", seen) + } +} + +func TestDelete_concurrentSameNameSafe(t *testing.T) { + t.Parallel() + fx := newFixture(t, 1) + fx.addOurVM(testVMID, testName, "running") + release := fx.fake.blockOn("Stop") + errs := make([]error, 2) + first := make(chan struct{}) + var wg sync.WaitGroup + wg.Go(func() { + defer close(first) + errs[0] = fx.p.Delete(fx.ctx, fx.id()) + }) + for fx.fake.callCount("Stop") == 0 { + select { + case <-first: + t.Fatalf("first Delete returned %v without stopping the VM", errs[0]) + case <-time.After(time.Millisecond): + } + } + wg.Go(func() { errs[1] = fx.p.Delete(fx.ctx, fx.id()) }) + close(release) + wg.Wait() + for i, err := range errs { + if err != nil { + t.Errorf("Delete %d error = %v", i, err) + } + } + for method, want := range map[string]int{"Stop": 1, "Destroy": 1} { + if got := fx.fake.callCount(method); got != want { + t.Errorf("%s calls = %d, want %d", method, got, want) + } + } +} + +// TestDelete_waitsOnTheNameLock: Delete takes the per-name lock before any +// call, so it cannot interleave with a Create or another Delete of that +// name (spec entry 19). +func TestDelete_waitsOnTheNameLock(t *testing.T) { + t.Parallel() + fx := newFixture(t, 1) + fx.addOurVM(testVMID, testName, "running") + release, err := fx.p.lockName(fx.ctx, testName) + if err != nil { + t.Fatalf("lockName error = %v", err) + } + defer release() + ctx, cancel := context.WithTimeout(fx.ctx, 20*time.Millisecond) + defer cancel() + err = fx.p.Delete(ctx, fx.id()) + assertProviderError(t, err, "delete", provider.ErrTransient) + if !errors.Is(err, context.DeadlineExceeded) { + t.Errorf("error = %v, want it to wrap DeadlineExceeded", err) + } + if calls := fx.fake.callLog(); len(calls) != 0 { + t.Errorf("calls = %v, want none while the name is locked", calls) + } +} + +func TestDelete_cancelledMidWaitReturnsPromptly(t *testing.T) { + t.Parallel() + fx := newFixture(t, 1) + fx.p.budgets = defaultBudgets() + fx.p.budgets.poll = time.Millisecond + fx.fake.taskPolls = 1 << 30 + fx.addOurVM(testVMID, testName, "running") + ctx, cancel := context.WithCancel(fx.ctx) + defer cancel() + fx.fake.hookOn("TaskStatus", func() error { + cancel() + return nil + }) + start := time.Now() + err := fx.p.Delete(ctx, fx.id()) + if elapsed := time.Since(start); elapsed > 5*time.Second { + t.Errorf("Delete took %v after cancellation", elapsed) + } + assertProviderError(t, err, "delete", provider.ErrTransient) + if !errors.Is(err, context.Canceled) { + t.Errorf("error = %v, want it to wrap context.Canceled", err) + } +} + +func TestDelete_outcomeLine(t *testing.T) { + t.Parallel() + tests := []struct { + name string + setup func(fx *fixture) + want []string + never string + }{ + { + name: "destroyed", + setup: func(fx *fixture) { fx.addOurVM(testVMID, testName, "stopped") }, + want: []string{`"node"="` + fakeNode + `"`, `"destroyed"=true`}, + }, + {name: "already gone", setup: func(*fixture) {}, want: []string{`"destroyed"=false`}, never: `"node"=`}, + } + for _, tc := range tests { + t.Run(tc.name, func(t *testing.T) { + t.Parallel() + fx := newFixture(t, 1) + tc.setup(fx) + fx.delete() + lines := fx.logs.matching(`"msg"="proxmox vm deleted"`) + if len(lines) != 1 { + t.Fatalf("outcome lines = %d, want 1:\n%s", len(lines), strings.Join(fx.logs.all(), "\n")) + } + for _, want := range append(tc.want, `"vmid"=150042`, `"name"="`+testName+`"`, `"provider"="`+testProvider+`"`) { + if !strings.Contains(lines[0], want) { + t.Errorf("outcome line lacks %s: %s", want, lines[0]) + } + } + if tc.never != "" && strings.Contains(lines[0], tc.never) { + t.Errorf("outcome line has %s: %s", tc.never, lines[0]) + } + }) + } +} + +func TestDelete_noOpLinesAreSilentAtV0(t *testing.T) { + t.Parallel() + tests := map[string]func(fx *fixture){ + "absent": func(*fixture) {}, + "foreign": func(fx *fixture) { fx.addOurVM(testVMID, "proxy-someoneelse00000", "running") }, + "unmarked": func(fx *fixture) { fx.fake.addVM(testVMID, testName, "running") }, + "ours": func(fx *fixture) { fx.addOurVM(testVMID, testName, "running") }, + } + for name, setup := range tests { + t.Run(name, func(t *testing.T) { + t.Parallel() + fx := newFixture(t, 0) + fx.p.api = observe(fx.fake, testProvider) + setup(fx) + fx.delete() + if lines := fx.logs.all(); len(lines) != 0 { + t.Errorf("V(0) lines = %v, want none", lines) + } + }) + } +} diff --git a/internal/provider/proxmox/fake_test.go b/internal/provider/proxmox/fake_test.go index 52206d0..594f569 100644 --- a/internal/provider/proxmox/fake_test.go +++ b/internal/provider/proxmox/fake_test.go @@ -39,9 +39,10 @@ type fakeAPI struct { // taskPolls is how many TaskStatus polls answer "running" before a // task stops. taskPolls int - // cloneFailsWith and startFailsWith, when set, are the exitstatus of - // every clone or start task. A failed clone removes its VM. - cloneFailsWith, startFailsWith string + // cloneFailsWith, startFailsWith and destroyFailsWith, when set, are + // the exitstatus of every clone, start or destroy task. A failed clone + // removes its VM; a failed destroy keeps it. + cloneFailsWith, startFailsWith, destroyFailsWith string // agentNotRunning makes the agent of every running VM with agent=1 // answer "QEMU guest agent is not running". agentNotRunning bool @@ -510,8 +511,12 @@ func (f *fakeAPI) Destroy(ctx context.Context, node string, vmid int) (string, e if vm.status == "running" { return "", fakeErr(500, fmt.Sprintf("fake: VM %d is running - destroy failed", vmid)) } - return f.newTask(node, "qmdestroy", strconv.Itoa(vmid), "OK", func() { - if f.vms[vmid] == vm { + exit := "OK" + if f.destroyFailsWith != "" { + exit = f.destroyFailsWith + } + return f.newTask(node, "qmdestroy", strconv.Itoa(vmid), exit, func() { + if exit == "OK" && f.vms[vmid] == vm { delete(f.vms, vmid) } }), nil diff --git a/internal/provider/proxmox/get_test.go b/internal/provider/proxmox/get_test.go index ca7de3b..39d672d 100644 --- a/internal/provider/proxmox/get_test.go +++ b/internal/provider/proxmox/get_test.go @@ -4,6 +4,7 @@ import ( "errors" "slices" "strings" + "sync" "testing" "time" @@ -474,3 +475,60 @@ func TestGet_resourcesFailureIsTransient(t *testing.T) { }) } } + +func TestGet_afterDeleteReturnsNotFound(t *testing.T) { + t.Parallel() + fx := newFixture(t, 1) + fx.addOurVM(testVMID, testName, "running").ifaces = capturedIfaces(t) + if inst := fx.get(fx.id()); inst.State != provider.StateRunning { + t.Fatalf("Get before Delete = %+v, want Running", inst) + } + if err := fx.p.Delete(fx.ctx, fx.id()); err != nil { + t.Fatalf("Delete error = %v", err) + } + assertGetFails(t, fx, provider.ErrNotFound) +} + +// TestGet_duringDeleteStillReturnsInstance holds the destroy task running: +// destroying sets no lock and keeps the VM listed until the task ends, as +// Phase 0's mid-destroy captures show, so Get sees it Stopped, not gone. +func TestGet_duringDeleteStillReturnsInstance(t *testing.T) { + t.Parallel() + if _, locked := capturedConfig(t, "config_during_destroy")["lock"]; locked { + t.Fatal("capture config_during_destroy has a lock; the fake models none") + } + listed := decodeData[[]vmResource](t, "cluster_resources_during_destroy") + if !slices.ContainsFunc(listed, func(r vmResource) bool { return r.VMID == 199999 && r.Status == "stopped" && r.Lock == "" }) { + t.Fatal("capture cluster_resources_during_destroy does not list VM 199999 stopped and unlocked") + } + + fx := newFixture(t, 1) + fx.addOurVM(testVMID, testName, "running") + inTask, release := make(chan struct{}), make(chan struct{}) + var once sync.Once + fx.fake.hookOn("TaskStatus", func() error { + if fx.fake.callCount("Destroy") > 0 { + once.Do(func() { + close(inTask) + <-release + }) + } + return nil + }) + done := make(chan error, 1) + go func() { done <- fx.p.Delete(fx.ctx, fx.id()) }() + select { + case <-inTask: + case err := <-done: + t.Fatalf("Delete returned %v before its destroy task was polled", err) + } + inst := fx.get(fx.id()) + if inst.State != provider.StateStopped || inst.UID != testUID || inst.ID != fx.id() { + t.Errorf("Get during the destroy = %+v, want the instance Stopped with its uid", inst) + } + close(release) + if err := <-done; err != nil { + t.Fatalf("Delete error = %v", err) + } + assertGetFails(t, fx, provider.ErrNotFound) +} diff --git a/internal/provider/proxmox/proxmox.go b/internal/provider/proxmox/proxmox.go index 5843857..97c6d58 100644 --- a/internal/provider/proxmox/proxmox.go +++ b/internal/provider/proxmox/proxmox.go @@ -172,9 +172,3 @@ func (p *Provider) lockName(ctx context.Context, name string) (func(), error) { } return release, nil } - -// Delete is a stub until Phase 7. It answers Transient, so a Proxy on this -// provider retries rather than latching. -func (p *Provider) Delete(_ context.Context, id string) error { - return p.wrapErr("delete", id, transientf("proxmox Delete is not implemented yet")) -} diff --git a/internal/provider/proxmox/proxmox_test.go b/internal/provider/proxmox/proxmox_test.go index 502ff9d..dd47efd 100644 --- a/internal/provider/proxmox/proxmox_test.go +++ b/internal/provider/proxmox/proxmox_test.go @@ -215,16 +215,6 @@ func TestWrapErr_carriesProviderNameOpAndID(t *testing.T) { } } -// TestStubs_answerTransient keeps a Proxy on this provider retrying, not -// latched, until Delete is implemented in Phase 7. -func TestStubs_answerTransient(t *testing.T) { - t.Parallel() - p := newProvider("proxmox-home", testConfig(), nil) - if err := p.Delete(context.Background(), "vms/150042/proxy-x"); !errors.Is(err, provider.ErrTransient) { - t.Errorf("delete error = %v, want ErrTransient", err) - } -} - func TestLockName_givesUpWithContextAsTransient(t *testing.T) { t.Parallel() p := newProvider("proxmox-home", testConfig(), nil) From 6e9ef26dcb856cd18d10425731ec08d6ef439f1c Mon Sep 17 00:00:00 2001 From: kacerr Date: Wed, 23 Sep 2026 20:42:54 +0000 Subject: [PATCH 11/13] proxmox: contract and observability tests, integration test, testing docs The contract sweep in provider_test.go: every method and Check returns within a second of a cancel, every wrapErr path carries the configured name, op and id, the V0/V1/V2 logging tiers and the no-op lines, and a Get over the real client whose HTTP spans are children of provider.get. proxmox_integration_test.go (-tags=integration) holds TestProxmox_lifecycle and TestProxmoxCheck_realHost; both compile and skip without PROXMOX_TEST_*, and have not run against a host yet. make test-integration-proxmox runs them. docs/testing.md gains the proxmox, nocloud and keyedlock rows, the Live Proxmox gaps and the integration section; spec entry 21's section 11 list is ticked. Waits that could hang the package under a mutation are bounded: the concurrent-Create loop and eight direct blocking calls in older tests. Co-Authored-By: Claude Opus 5.5 (1M context) --- Makefile | 4 + .../2026-09-18-1502-proxmox-provider.md | 184 +++++- docs/proxmox-provider-spec.md | 40 +- docs/testing.md | 130 ++++ internal/provider/proxmox/check_test.go | 6 +- internal/provider/proxmox/create_test.go | 20 +- internal/provider/proxmox/delete_test.go | 5 +- internal/provider/proxmox/provider_test.go | 595 +++++++++++++++++ .../proxmox/proxmox_integration_test.go | 624 ++++++++++++++++++ internal/provider/proxmox/proxmox_test.go | 2 +- internal/provider/proxmox/task_test.go | 9 +- 11 files changed, 1585 insertions(+), 34 deletions(-) create mode 100644 internal/provider/proxmox/proxmox_integration_test.go diff --git a/Makefile b/Makefile index fb685df..d5fd541 100644 --- a/Makefile +++ b/Makefile @@ -74,6 +74,10 @@ test-integration-kvm: ## Run the KVM provider test against a real hypervisor. Ne test-integration-kvm2: ## Run the kvm2 provider test against a real hypervisor. Needs KVM2_TEST_HOST/USER/SSH_KEY/BRIDGE/IMAGE_URL; see docs/testing.md. go test -race -tags=integration -count=1 -v -timeout 40m -run TestKVM2 ./internal/provider/kvm2/ +.PHONY: test-integration-proxmox +test-integration-proxmox: ## Run the proxmox provider test against a real Proxmox VE host. Needs PROXMOX_TEST_URL/TOKEN_FILE/NODE/TEMPLATE_VMID/ISO_STORAGE; see docs/testing.md. + go test -race -tags=integration -count=1 -v -timeout 30m -run TestProxmox ./internal/provider/proxmox/ + # TODO(user): To use a different vendor for e2e tests, modify the setup under 'tests/e2e'. # The default setup assumes Kind is pre-installed and builds/loads the Manager Docker image locally. # kubectl kuberc is disabled by default for test isolation; enable with: diff --git a/docs/plans-executions/2026-09-18-1502-proxmox-provider.md b/docs/plans-executions/2026-09-18-1502-proxmox-provider.md index 9f403a2..21e77f4 100644 --- a/docs/plans-executions/2026-09-18-1502-proxmox-provider.md +++ b/docs/plans-executions/2026-09-18-1502-proxmox-provider.md @@ -12,7 +12,7 @@ Pairs with [docs/plans/2026-09-18-1502-proxmox-provider.md](../plans/2026-09-18- - [x] Phase 5 — Get, ListByTag, task helpers - [x] Phase 6 — Create - [x] Phase 7 — Delete, Checker, CRD comment -- [ ] Phase 8 — Contract and observability sweep, integration test, testing.md +- [x] Phase 8 — Contract and observability sweep, integration test, testing.md - [ ] Phase 9 — Deployment and documentation - [ ] Phase 10 — Review, real-host validation, changelog, MR @@ -939,3 +939,185 @@ Worth noting: hibernated VM, and a stop on a paused VM. The try-every-node seed delete and the multi-node Checker have only met the fake (entry 24). - Nothing in this phase talked to the host. + +## Phase 8 — Contract and observability sweep, integration test, testing.md + +The sweep went into the existing `provider_test.go` and reuses the earlier +phases' helpers. The new helpers are `cancelWhileBlocked`, `within`, +`newLoggingFixture` with `lifecycle`/`sentinelReq`, `assertNeverLogged`, +`assertEveryV1Line` and `msgLine`. + +- `TestProvider_cancelledContextReturnsPromptly` blocks Create, Get, + Delete and ListByTag on `ClusterResources`, and Check on `Version`, + with `blockOn`. It asserts that the blocked call was the first in the + call log, then cancels. Each must return within 1 s, as a Transient + that still carries `context.Canceled` (Check: one target, `Timeout`). + Two more rows block mid-operation: Create on `UploadISO` and Delete on + `Stop`. +- `TestProvider_everyErrorIsWrappedWithConfiguredName` has 20 rows, one + or more behind every `wrapErr` call site: placement, the name lock with + an ended ctx, resources, a 403 on the clone, a failed start task, the + malformed, missing, locked and 401 Get, the hibernated VM, a failed + destroy task, a 403 on the seed delete, and the list with a ctx that + ends on a candidate. Each row also checks `Error.ID` (empty for + Create) and the `provider proxmox-home: ` prefix. +- `TestLogging_v0SilentV1OutcomeV2Detail` runs one whole lifecycle over + the observed fake with a token file: Create with a sentinel in + CloudInit, a Get that finds the captured address, ListByTag, Delete, + Delete of the gone VM, and Check. It runs at V(0), V(1) and V(2). V(0) + logs nothing. V(1) has each outcome line once, with its keys, and two + `proxmox vm deleted` lines: `destroyed=true` with node, then + `destroyed=false` without. Every line has the `proxmox` logger, the + provider and level 1, and no V(2) key. V(2) has the resolved line with + `cloudInitBytes`, agent addresses, the listed instance, and the + clone's description. The sentinel, `#cloud-config` and the token + secret appear at no level. +- `TestLogging_noOpLinesAtV1` (a name beyond the plan) covers entry 20's + no-op lines. Each carries provider and vmid: VM already gone, foreign, + unmarked, seed delete skipped on each of two nodes, unreadable + candidate. +- `TestTracing_httpSpansUnderOperationSpan` routes captures by path, as + `TestDelete_usesPurgeAndDestroyUnreferencedDisks` does. The captures + are `cluster_resources_during_destroy`, `config_during_destroy`, + `status_current_running` and `agent_network_get_interfaces`. A Get + runs over the real `httpAPI`, wrapped in `observe` and + `provider.WithTracing`, with the recorder's tracer passed through + `newHTTPAPI`'s opts. It asserts Running at 192.168.0.150, and exactly + four spans named `HTTP GET` (otelhttp v0.70's + `defaultTransportFormatter`), each a child of `provider.get` in its + trace. No attribute on any span carries the secret or `PVEAPIToken`. + +Judgment calls: + +- **"Instance lines carry vmid and name" is checked on the provider's + lines, not the decorator's.** Entry 20 gives `proxmox api call` the + fields `call`, `node`, `vmid`, `storage`, `upid` and `duration`, with + `name` only on a clone. So `assertEveryV1Line` exempts that line and + the two lines that are not about one instance (`vms listed`, `node + checked`), and requires vmid and name on all the others. +- **The integration test compares `CreatedAt` with the mark Create + wrote.** kvm2 only checks that it is non-zero. Entry 21 says "equal + what Create wrote", so the test reads the VM's config through the + bare client and parses its description. +- **Get's poll tolerates Transient.** kvm2's poll fails on any error. + Entry 7 makes every agent answer the provider cannot place Transient, + and a booting guest gives some. Those answers are logged and polled + through; any other class fails. +- **The integration machine type has `diskGiB: 4`.** The 3 GiB recon + template is then grown by the resize task on the real host. A larger + template skips the resize. +- **The seed check after Delete is one extra read-only GET in the test**: + `client.do` on `/nodes/{node}/storage/{storage}/content?content=iso`, + decoded the way `storage_content_iso.json` is shaped. No seam method + was added. The VM count and the by-name lookup use the existing + `ClusterResources`. +- **The cleanup finds the VM by name**, so a Create that failed halfway + is also removed. With `PROXMOX_TEST_KEEP` it prints `qm stop ` + before `qm destroy --purge`, because destroy refuses a running + VM. It then prints `pvesm free :iso/-seed.iso`. + +**Unbounded waits.** `TestCreate_concurrentSameNameSerialised` now uses +the `first` channel shape of `TestDelete_concurrentSameNameSafe`. The grep +found no other sleep loop; `client_test.go`'s `for {` reads multipart +parts until EOF. The mutation check found a different shape of the same +hang: tests that call a blocking operation directly on the test goroutine, +where the lock or `blockOn` is released only after the call returns. Under +"lockName ignores ctx", "Check ignores ctx" and "pollUntil ignores ctx" +the package ran into its 60 s timeout on +`TestDelete_waitsOnTheNameLock`, `TestLockName_givesUpWithContextAsTransient`, +`TestCheck_expiredContextReturnsPromptly`, two `TestCheck_reasonsTable` +rows, `TestCreate_cancelledMidWaitReturnsPromptly`, +`TestDelete_cancelledMidWaitReturnsPromptly`, and the ended-ctx rows of +`TestWaitTask_table` and `TestWaitUnlocked_table`. Each call now runs in +`within(t, 5s or 10s, …)`, which fails the test by name. After the fix +the same mutations fail those tests in seconds. + +The mutation check ran 14 mutations on production code. The script copied +each file, applied the edit (asserting that it matched exactly once), and +ran `go test -c` before counting a mutation. It then ran +`go test -count=1 -timeout=60s`, restored the file from the copy, and +asserted the sha256 matched. Afterwards `sha256sum -c` over the eight +files hashed up front was OK. `task.go`, added for M14, was covered by the +script's own assertion. `git status` showed no production file changed. +Results: + +- Caught by the new tests: + - Get drops the span context before its requests → + `TestTracing_httpSpansUnderOperationSpan`. + - Cloud-init logged at V(2) → `TestLogging_…` and + `TestCreate_cloudInitSentinel…`. + - Delete's destroy path skips `wrapErr` → the wrap test and 6 Delete + tests. + - ListByTag's ended-ctx path skips `wrapErr` → the wrap test's `list + ended on a candidate` row. + - Delete's resolve with `context.Background()` → the cancel test's + `delete` row. + - `lockName` with `context.Background()` → the wrap test's two lock + rows. + - Check's probe with `context.Background()` → the cancel test's + `check` row. + - Delete's logger without the name, `destroyed` always true, `vms + listed` at V(0), the clone description at V(1), and the list skip + line without vmid → `TestLogging_*`. +- Caught only by the older tests: `pollUntil` without its `ctx.Done()` + case → the four waits tests above. The first attempt deleted the case + and did not compile (the `provider` import became unused). The rerun + replaced it with a nil channel. +- Survived, and equivalent: `newHTTPAPI` without the tracer-provider + option. The transport starts a span only under a valid parent. With no + `WithTracerProvider`, otelhttp v0.70 takes the tracer from + `trace.SpanFromContext(r.Context()).TracerProvider()` + (`transport.go:99`), so the child always lands in the parent's + provider. No test can tell the two apart. + +`Makefile`: `test-integration-proxmox` next to the kvm2 target, with a +`##` help line naming the five required variables. `docs/testing.md` +gained: + +- rows for `nocloud`, `keyedlock` and five for `proxmox` (both packages + were missing); +- a "Live Proxmox, by default" bullet under "deliberately do NOT cover", + naming the cross-node paths and the three answers never captured; +- a "The proxmox integration test" section: variables, the eleven + assertions in order, the KEEP cleanup, the cost, and that it has not + run yet. + +In spec entry 21 every §11 box is ticked. The list's intro says the two +integration tests compile but have not run against a host. The cancel and +sentinel items name the new coverage. + +```bash +# the names in entry 21, each checked before ticking: 42 of 42 exist +awk '/^## 21\. Tests/,/^## 22\./' docs/proxmox-provider-spec.md | grep -o '`Test[A-Za-z0-9_]*`' | tr -d '`' | sort -u \ + | while read n; do grep -rqE "^func $n\(" internal/provider/proxmox/ && echo "ok $n" || echo "MISSING $n"; done +go test -race -count=2 ./internal/provider/proxmox/ # ok +go vet -tags=integration ./... && go vet -tags=e2e ./... # ok +make -n test-integration-proxmox +# go test -race -tags=integration -count=1 -v -timeout 30m -run TestProxmox ./internal/provider/proxmox/ +go test -tags=integration -count=1 -v -run TestProxmox ./internal/provider/proxmox/ # both SKIP: PROXMOX_TEST_URL is not set +go build ./... && go test -short -count=1 ./... # all ok +go test -count=1 -v ./internal/provider/proxmox/ | grep -c '^=== RUN' # 649 before, 688 after +bin/golangci-lint run --disable staticcheck --max-same-issues 0 --max-issues-per-linter 0 ./internal/provider/proxmox/... +bin/golangci-lint run --build-tags integration --disable staticcheck --max-same-issues 0 --max-issues-per-linter 0 ./internal/provider/proxmox/... +``` + +Worth noting: + +- Lint is at the baseline: + - the logcheck hits (4/4/2/1/1), mapsloop ×2 in `proxmox.go`, unparam + ×3 in `fake_test.go`, gocyclo in `api_test.go`, and goconst on test + strings. + - Two new gocyclo hits were removed, not accepted. + `TestLogging_v0SilentV1OutcomeV2Detail` was at 32; the per-line loop + became `assertEveryV1Line`. `TestProxmox_lifecycle` was at 40; steps + 6 and 7 became `assertMarkReadBack` and `assertListed`. kvm2's + lifecycle test is still at 39. +- The integration tests only compile so far. The first real run is Phase + 10's. That run meets the Datastore.Allocate point too: a token without + it on the ISO storage fails step 9 as Permanent. Phase 0 granted it on + `proxy-seeds`, and it is already a Phase 9 guide item. +- The parked items did not bite: kvm2's S5.1 and its 200 ms + `TestCheck_hostsCheckedConcurrently` passed in every + `go test -short ./...` run of this phase. +- `.claude/settings.json` stays out of the commit. Nothing in this phase + talked to the host. diff --git a/docs/proxmox-provider-spec.md b/docs/proxmox-provider-spec.md index 7c22a78..7af56d7 100644 --- a/docs/proxmox-provider-spec.md +++ b/docs/proxmox-provider-spec.md @@ -781,26 +781,28 @@ within the budget and its `UID` and `CreatedAt` equal what Create wrote; remain; an HTTP span is a child of `provider.create` and no span attribute carries the cloud-init text. -Contract §11 list, with the planned test for each item (Phase 8 ticks them -once they exist): +Contract §11 list, with the test for each item. Every name below exists +(Phase 8 checked each with a grep for `func (`). The two integration +tests compile under `-tags=integration` but have not run against a host +yet; Phase 10 runs them. -- [ ] repeat `Create`: `TestCreate_repeatReturnsSameIDAndCreatesNothing` -- [ ] `Get` after `Delete`: `TestGet_afterDeleteReturnsNotFound` -- [ ] `Delete` twice: `TestDelete_twiceSucceeds` -- [ ] malformed ID: `TestParseProviderID_malformed`, `TestGet_malformedIDIsPermanent`, `TestDelete_malformedIDIsPermanent` -- [ ] unknown state maps to `StateStopped`: `TestMapState_table`, `TestGet_unknownStateIsStopped` -- [ ] `StateRunning` requires an IP: `TestGet_runningWithoutIPIsProvisioning`, `TestGet_runningWithIPIsRunning` -- [ ] `UID` and `CreatedAt` read back: `TestGet_readsUIDAndCreatedAtFromBackend` -- [ ] `ListByTag` excludes unmarked instances: `TestListByTag_excludesUnmarkedVMs`, `TestListByTag_emptyUIDStillListed` -- [ ] `Get` during an in-progress delete: `TestGet_duringDeleteStillReturnsInstance` -- [ ] `Get` during an asynchronous operation or lock: `TestGet_lockedIsTransientNeverNotFound`, `TestGet_midCloneNamelessEntryIsTransient` -- [ ] never started is never `StateStopped`: `TestCreate_returnsOnlyWhenRunning` -- [ ] cancelled `ctx` returns promptly: `TestProvider_cancelledContextReturnsPromptly`, `TestClient_cancelledContextReturnsPromptly` -- [ ] `classify` table with captured responses: `TestClassify_table` -- [ ] Checker reasons, expired `ctx`, stable order: `TestCheck_reasonsTable`, `TestCheck_expiredContextReturnsPromptly`, `TestCheck_oneEntryPerNodeInConfigOrder` -- [ ] sentinel tests for `CloudInit` and the credential: `TestCreate_cloudInitSentinelNeverLoggedOrInErrors`, `TestClient_tokenNeverInErrorsOrLogs` -- [ ] child spans under the operation span: `TestTracing_httpSpansUnderOperationSpan`, `TestClient_childSpanUnderParent` -- [ ] real-backend lifecycle test: `TestProxmox_lifecycle` and `TestProxmoxCheck_realHost` in `proxmox_integration_test.go` +- [x] repeat `Create`: `TestCreate_repeatReturnsSameIDAndCreatesNothing` +- [x] `Get` after `Delete`: `TestGet_afterDeleteReturnsNotFound` +- [x] `Delete` twice: `TestDelete_twiceSucceeds` +- [x] malformed ID: `TestParseProviderID_malformed`, `TestGet_malformedIDIsPermanent`, `TestDelete_malformedIDIsPermanent` +- [x] unknown state maps to `StateStopped`: `TestMapState_table`, `TestGet_unknownStateIsStopped` +- [x] `StateRunning` requires an IP: `TestGet_runningWithoutIPIsProvisioning`, `TestGet_runningWithIPIsRunning` +- [x] `UID` and `CreatedAt` read back: `TestGet_readsUIDAndCreatedAtFromBackend` +- [x] `ListByTag` excludes unmarked instances: `TestListByTag_excludesUnmarkedVMs`, `TestListByTag_emptyUIDStillListed` +- [x] `Get` during an in-progress delete: `TestGet_duringDeleteStillReturnsInstance` +- [x] `Get` during an asynchronous operation or lock: `TestGet_lockedIsTransientNeverNotFound`, `TestGet_midCloneNamelessEntryIsTransient` +- [x] never started is never `StateStopped`: `TestCreate_returnsOnlyWhenRunning` +- [x] cancelled `ctx` returns promptly: `TestProvider_cancelledContextReturnsPromptly` (every method and `Check`, blocked on the first seam call, plus Create and Delete blocked mid-operation), `TestClient_cancelledContextReturnsPromptly` +- [x] `classify` table with captured responses: `TestClassify_table` +- [x] Checker reasons, expired `ctx`, stable order: `TestCheck_reasonsTable`, `TestCheck_expiredContextReturnsPromptly`, `TestCheck_oneEntryPerNodeInConfigOrder` +- [x] sentinel tests for `CloudInit` and the credential: `TestCreate_cloudInitSentinelNeverLoggedOrInErrors`, `TestClient_tokenNeverInErrorsOrLogs`, and `TestLogging_v0SilentV1OutcomeV2Detail` over a whole lifecycle at every level +- [x] child spans under the operation span: `TestTracing_httpSpansUnderOperationSpan`, `TestClient_childSpanUnderParent` +- [x] real-backend lifecycle test: `TestProxmox_lifecycle` and `TestProxmoxCheck_realHost` in `proxmox_integration_test.go` (compiled, not yet run) Tests that cover the Phase 0 corrections specifically, in addition to the plan's per-phase lists: diff --git a/docs/testing.md b/docs/testing.md index 0adf88b..5e29127 100644 --- a/docs/testing.md +++ b/docs/testing.md @@ -67,6 +67,13 @@ manual runs too. | | `image_test.go`, `create_test.go`, `get_test.go`, `delete_test.go`, `list_test.go`, `kvm2_test.go`, `provider_test.go` | image cache (fetch with the marker written last, cached, marker missing or without a digest, sha256 change or mismatch, 4xx/5xx, no Content-Length, short read, concurrent callers fetch once, cancel mid-stream, streamed without buffering, upload on a dedicated connection) and the pool-space quota precheck; Create resume after every partial step, returns only when running, adoption guard on UID, seed always rebuilt, cloud-init sentinel never logged; Get state and address mapping incl. agent-before-ARP; every Delete already-gone case, NVRAM flag, per-name lock; ListByTag across hosts, one down fails the whole call, unreadable domain skipped; `New` never dials and warns on missing files; every error wrapped with the configured name; logging tiers V0/V1/V2 | `httptest` image server + in-memory `fakeBackend` behind the `backend` seam | | | `observe_test.go`, `check_test.go` | every seam method traced and logged exactly once (guarded by reflection), child spans under the parent, detail only at V2, upload bodies never logged, error fields; health check: one entry per host in config order, hosts checked concurrently, expired ctx returns promptly, never touches the pooled connection, the reason table, detail keys | `fakeBackend` + `funcr` capture + span recorder | | | `kvm2_integration_test.go` (`-tags=integration`) | one real VM: `New` without a dial, `Check` Ready with details, Create, idempotent re-Create with an unchanged domain count, Running with IP and the metadata round-trip, ListByTag, Delete, NotFound, pool state afterwards, `kvm2.*` spans under `provider.create`; `TestKVM2Check_realHost` runs the health check against the host and creates nothing | a **real hypervisor** — see below | +| `internal/provider/nocloud` | `seed_test.go` | the NoCloud seed kvm2 and proxmox share: the ISO reopened (files, label, user-data byte for byte, temp dir removed, an empty vendor-data still a file, an empty name refused) and the vendor-data rules (a shell script, not cloud-config; multipart with configured extra vendor-data; the agent install switched off) | pure | +| `internal/provider/keyedlock` | `keyedlock_test.go` | the per-name lock kvm2 and proxmox share: the entry is removed once uncontended, waiters share one lock, a waiter gives up with its ctx | pure | +| `internal/provider/proxmox` | `id_test.go`, `mark_test.go`, `state_test.go`, `seed_test.go`, `errors_test.go`, `api_test.go` | pure pieces: providerID round-trip and malformed forms, the VMID derived from the UID (deterministic, in range, probing forward and wrapping), the description mark (keys, parsed from the captured clone, admin lines kept), state mapping and the IPv4 filter from captures, seed names and building (empty name Permanent without its payload, staging failure Transient); the `classify` table and the 500-message matchers against every captured error; the API types decoded from the captures | pure, over the responses Phase 0 captured from PVE 9.2.20 (`testdata/`) | +| | `client_test.go` | the thin HTTP client: the token read from its file on every request (missing Transient, unparsable Permanent), `apiError` from status, reason phrase and body, the data envelope, method, path and parameters of every seam call, the clone and destroy query shapes, the volid's slash escaped, HTTP/1.1 only, request timeout and cancel Transient and the client unaffected, a child span under a parent and none without, the token never in errors or logs, the CA file trusted and loaded lazily, an x509 failure Permanent, insecure mode, the upload streamed as multipart | `httptest.NewTLSServer` replaying the captures byte for byte | +| | `create_test.go`, `get_test.go`, `delete_test.go`, `list_test.go`, `task_test.go`, `proxmox_test.go`, `provider_test.go`, `fake_test.go` | Create: resume after every partial step and after a crash mid-clone, the VMID probe and ownership check, the template's shape refused before any clone, resize as a task, the seed overwritten without a delete, the net0 bridge and VLAN rewrite keeping the MAC, returns only when running, cloud-init sentinel never logged; Get: lock checked before name, the agent's messages, paused from `status/current`, hibernated, a reused VMID, four calls when running; Delete: every already-gone and not-ours case, the seed deleted on the VM's node or on every node, hibernated Permanent, the per-name lock; ListByTag: candidates by tag or name prefix, the mark decides, an unreadable VM skipped unless ctx ended; task and lock waits; `New` makes no request; the contract sweep: every method returns within a second of a cancel (blocked on its first seam call, and mid-operation), every error wrapped with the configured name, op and id, logging tiers V0/V1/V2 with the no-op lines | in-memory `fakeAPI` behind the `api` seam, its error texts checked against the captures | +| | `observe_test.go`, `check_test.go`, `provider_test.go` | the decorator: every seam method logs exactly one line (guarded by reflection), starts no span, identifiers at V1, detail at V2, upload body never logged, error fields; a Get through `provider.WithTracing` over the real client makes four `HTTP GET` spans, all children of `provider.get` and none carrying the token; health check: one entry per node in config order, nodes checked concurrently (a barrier, not a timer), expired ctx returns promptly, the reasons table at V2 with a sentinel secret, the transport reasons from real `net/http` and `crypto/tls` errors, detail keys | `fakeAPI` + `funcr` capture + span recorder; the real client over `httptest` for the tracing and transport-reason rows | +| | `proxmox_integration_test.go` (`-tags=integration`) | one real VM: `New` without a request, `Check` Ready with details, Create, idempotent re-Create with an unchanged VM count, Running with an IPv4 and the mark read back, ListByTag, Delete, NotFound, no VM and no seed left, HTTP spans under `provider.create`; `TestProxmoxCheck_realHost` runs the health check against the host and creates nothing. **Compiles, not yet run against a host** | a **real Proxmox VE host** — see below | | `internal/health` | `probe_test.go` | probes through a **real CONNECT-capable proxy stub** to a real TLS server: tunnel success, refused CONNECT, wrong status, dead proxy, plain-http forward | `httptest` + hijacked bidirectional tunnel | | | `engine_test.go` | thresholds, first-verdict, latency suppression matrix, dropped-event retry, stale-probe UID guard, tick scheduling/seeding/pruning, `Start` end-to-end | fake `client.Reader` + injected `probeFn` | | `internal/lease` | `store_test.go` | capacity, `MaxLeases=0`, all three selection tie-breaks, target-scoped vs global cooldowns, expiry via fake clock, report-on-retained-lease, idempotent release, **40 concurrent acquires never exceeding capacity** | injectable clock; the concurrency case is why `-race` matters | @@ -105,6 +112,25 @@ envtest suite the scaffold generated. boots and reports an address are provable only on a real host — that is the opt-in integration test below (one per provider). +- **Live Proxmox, by default**: the proxmox unit tests run over an + in-memory fake of the 16 seam calls, and the HTTP client over + `httptest` replaying responses captured from PVE 9.2.20. Whether a + clone, config write, seed upload and start yield a guest that boots and + reports an address is provable only on a real host (the proxmox + integration test below). Two kinds of path have met only the fake: + - **the cross-node paths**: a clone that targets another node, the seed + deleted on every node when the VM's node is unknown, a Check round over + several nodes. The test host is a single node, so the integration test + cannot reach them either + ([proxmox-provider-spec.md](proxmox-provider-spec.md) entry 24); + - **answers never captured**: what the storage status call returns for a + missing or disabled storage (every 500 there is taken as + `StorageNotFound`), what destroy says for a hibernated VM, and a stop + on a paused VM. The provider does not depend on their wording: any + 500 from the storage status is `StorageNotFound`, a hibernated VM is + refused before destroy is called, and a paused VM gets the same stop + as a running one. What the fake answers there is its own guess. + ## The KVM integration test (real hypervisor) `internal/provider/kvm/kvm_integration_test.go` compiles only under @@ -256,6 +282,110 @@ go test -tags=integration -count=1 -v -run TestKVM2Check ./internal/provider/kvm The test is not part of `make test` and not run in CI. Keep it compiling with `go vet -tags=integration ./...` when the provider's internals move. +## The proxmox integration test (real Proxmox VE host) + +`internal/provider/proxmox/proxmox_integration_test.go` compiles only +under `-tags=integration` and skips itself unless the environment names a +host. It deviates from the testcontainers convention for the same reason +as the kvm tests: Proxmox VE is a hypervisor, not something a container +can stand in for. It needs a node with a cloud-init template without a +Proxmox cloud-init drive and with the guest agent enabled, and a token +with the privileges of [proxmox-provider-spec.md](proxmox-provider-spec.md) +entry 14: the role on `/` from the plan's "Before Phase 0" section +([2026-09-18-1502-proxmox-provider.md](plans/2026-09-18-1502-proxmox-provider.md)), +plus `Datastore.Allocate` on the ISO storage. A passing run leaves +nothing behind: its VM and its seed ISO are gone. + +```sh +export PROXMOX_TEST_URL=https://192.168.0.8:8006 # host must be in the node certificate +export PROXMOX_TEST_TOKEN_FILE=$HOME/.config/proxmox-token # one line: user@realm!tokenid=secret +export PROXMOX_TEST_CA_FILE=$HOME/.config/proxmox-ca.pem # the node's pve-root-ca.pem +export PROXMOX_TEST_NODE=pv1-1 +export PROXMOX_TEST_TEMPLATE_VMID=9001 +export PROXMOX_TEST_ISO_STORAGE=proxy-seeds # holds iso content +make test-integration-proxmox +``` + +Required: `PROXMOX_TEST_URL`, `PROXMOX_TEST_TOKEN_FILE`, +`PROXMOX_TEST_NODE`, `PROXMOX_TEST_TEMPLATE_VMID` and +`PROXMOX_TEST_ISO_STORAGE`; the test skips when any is missing. Optional: +`PROXMOX_TEST_CA_FILE` (empty trusts the system roots), +`PROXMOX_TEST_INSECURE` (`true` skips certificate verification; the log +says so), `PROXMOX_TEST_STORAGE` (the clone's target storage; empty keeps +the template's), `PROXMOX_TEST_BRIDGE` and `PROXMOX_TEST_VLAN_TAG` (the +cloned `net0`'s bridge and tag; empty keeps the template's), +`PROXMOX_TEST_VMID_MIN` and `PROXMOX_TEST_VMID_MAX` (default +`100000`..`199999`), `PROXMOX_TEST_TIMEOUT` (default `20m`, the budget for +the whole lifecycle), `PROXMOX_TEST_KEEP` to leave the VM behind, and +`PROXMOX_TEST_LOG_LEVEL` (default `1`): at `1` every request is one +`proxmox api call` line with the call, node, VMID, UPID and duration, +plus the per-operation outcome lines; `2` adds request parameters, config +summaries and agent addresses; `0` leaves only `t.Log`. Provider, node, +name and VMID are stated once and left off every provider line. Without +`Datastore.Allocate` on the ISO storage the seed delete at the end of +Delete is a 403, and Delete fails as Permanent. + +What one run asserts, in order, each step logged with its duration: + +1. `New` returns within 100 ms and makes no request (no `proxmox api + call` line before the first operation). +2. `Check` reports the node `Ready` with reason `Reachable` and the + `pveVersion`, `kernel`, `tokenID`, `isoStorageAvailGiB` and + `templates` (`1/1`) details, which are logged. +3. Every later call goes through `provider.WithTracing`, with an + in-memory span recorder on the transport as well. +4. `Create` returns `vms//` with the VMID in range. The + machine type is 1 core, 1024 MiB and a 4 GiB disk, so a 3 GiB template + is grown by the resize task. +5. A second `Create` returns the same ID and the cluster's QEMU VM count + is unchanged. +6. `Get`, polled every 5 s, reaches `Running` with an IPv4 within the + budget (a Transient answer while the guest boots is logged and polled + through). Its `UID` and `CreatedAt` equal the mark in the VM's + description, read back from its config, and `CreatedAt` is within a + few minutes of now. The time from Create's return to the first address + is logged. +7. `ListByTag` contains the ID with that UID and CreatedAt. +8. A TCP connect to port 22 on the guest, logged only: the machine + running the test may not route to the VM's network. +9. `Delete` returns nil, `Get` is `ErrNotFound`, a second `Delete` is a + no-op. +10. No VM with the name is left in the cluster's resources, and no + `:iso/-seed.iso` in the storage's iso content. The + content is read with one extra read-only GET of + `/nodes/{node}/storage/{storage}/content?content=iso` made by the test + itself (`storage_content_iso.json` is its captured shape): the seam + has no method to list storage content, and does not get one for a + test. +11. Among the recorded spans, a `provider.create` span has at least one + `HTTP` child, and no span attribute carries the cloud-init text. + +With `PROXMOX_TEST_KEEP` steps 9 and 10 are skipped, and at the end the +log prints what to run as root on the node to clean up: +`qm stop `, `qm destroy --purge` and +`pvesm free :iso/-seed.iso`. Otherwise a `t.Cleanup` +runs the Delete even when an earlier assertion failed. It finds the VM by +name, so a Create that failed halfway is cleaned up too. + +Cost: one full clone of the template (about 2 s for the 3 GiB template on +pv1-1), one boot to an address (about 50 s), and a few seconds of +destroy; a run takes a couple of minutes. The Makefile target's 30 m +timeout covers the 20 m budget plus the cleanup. + +`TestProxmoxCheck_realHost` runs the provider health check against the +same host. It must find the node reachable, the ISO storage usable and +the template present, and it creates nothing. It needs the same +variables; to run only this test: + +```sh +go test -tags=integration -count=1 -v -run TestProxmoxCheck ./internal/provider/proxmox/ +``` + +**Neither test has run against a host yet.** Both compile +(`go vet -tags=integration ./...`); Phase 10 of the plan runs them on +pv1-1. Like the kvm tests they are not part of `make test` and not run in +CI. + ## The scaffolded `make test-e2e` suite `test/e2e/` is the kubebuilder-generated smoke suite (build image → kind diff --git a/internal/provider/proxmox/check_test.go b/internal/provider/proxmox/check_test.go index 79b9609..1e3bcbe 100644 --- a/internal/provider/proxmox/check_test.go +++ b/internal/provider/proxmox/check_test.go @@ -117,7 +117,8 @@ func TestCheck_expiredContextReturnsPromptly(t *testing.T) { ctx, cancel := context.WithTimeout(fx.ctx, 0) defer cancel() start := time.Now() - got := fx.p.Check(ctx) + var got []provider.TargetStatus + within(t, 5*time.Second, func() { got = fx.p.Check(ctx) }) if elapsed := time.Since(start); elapsed > time.Second { t.Errorf("Check() took %v with an expired ctx, want under 1s", elapsed) } @@ -321,7 +322,8 @@ func TestCheck_reasonsTable(t *testing.T) { ctx, cancel = context.WithCancel(ctx) time.AfterFunc(50*time.Millisecond, cancel) } - got := fx.p.Check(ctx) + var got []provider.TargetStatus + within(t, 5*time.Second, func() { got = fx.p.Check(ctx) }) if len(got) != 1 { t.Fatalf("Check() returned %d targets, want 1", len(got)) } diff --git a/internal/provider/proxmox/create_test.go b/internal/provider/proxmox/create_test.go index 73f919e..cce31f4 100644 --- a/internal/provider/proxmox/create_test.go +++ b/internal/provider/proxmox/create_test.go @@ -1196,15 +1196,20 @@ func TestCreate_concurrentSameNameSerialised(t *testing.T) { release := fx.fake.blockOn("Clone") ids := make([]string, 2) errs := make([]error, 2) + first := make(chan struct{}) var wg sync.WaitGroup - for i := range ids { - wg.Go(func() { ids[i], errs[i] = fx.p.Create(fx.ctx, fx.req()) }) - if i == 0 { - for fx.fake.callCount("Clone") == 0 { - time.Sleep(time.Millisecond) - } + wg.Go(func() { + defer close(first) + ids[0], errs[0] = fx.p.Create(fx.ctx, fx.req()) + }) + for fx.fake.callCount("Clone") == 0 { + select { + case <-first: + t.Fatalf("first Create returned %v without cloning", errs[0]) + case <-time.After(time.Millisecond): } } + wg.Go(func() { ids[1], errs[1] = fx.p.Create(fx.ctx, fx.req()) }) close(release) wg.Wait() for i, err := range errs { @@ -1233,7 +1238,8 @@ func TestCreate_cancelledMidWaitReturnsPromptly(t *testing.T) { return nil }) start := time.Now() - _, err := fx.p.Create(ctx, fx.req()) + var err error + within(t, 10*time.Second, func() { _, err = fx.p.Create(ctx, fx.req()) }) if elapsed := time.Since(start); elapsed > 5*time.Second { t.Errorf("Create took %v after cancellation", elapsed) } diff --git a/internal/provider/proxmox/delete_test.go b/internal/provider/proxmox/delete_test.go index bd621c4..b73d936 100644 --- a/internal/provider/proxmox/delete_test.go +++ b/internal/provider/proxmox/delete_test.go @@ -623,7 +623,7 @@ func TestDelete_waitsOnTheNameLock(t *testing.T) { defer release() ctx, cancel := context.WithTimeout(fx.ctx, 20*time.Millisecond) defer cancel() - err = fx.p.Delete(ctx, fx.id()) + within(t, 5*time.Second, func() { err = fx.p.Delete(ctx, fx.id()) }) assertProviderError(t, err, "delete", provider.ErrTransient) if !errors.Is(err, context.DeadlineExceeded) { t.Errorf("error = %v, want it to wrap DeadlineExceeded", err) @@ -647,7 +647,8 @@ func TestDelete_cancelledMidWaitReturnsPromptly(t *testing.T) { return nil }) start := time.Now() - err := fx.p.Delete(ctx, fx.id()) + var err error + within(t, 10*time.Second, func() { err = fx.p.Delete(ctx, fx.id()) }) if elapsed := time.Since(start); elapsed > 5*time.Second { t.Errorf("Delete took %v after cancellation", elapsed) } diff --git a/internal/provider/proxmox/provider_test.go b/internal/provider/proxmox/provider_test.go index 3e9dc4b..0ed7dff 100644 --- a/internal/provider/proxmox/provider_test.go +++ b/internal/provider/proxmox/provider_test.go @@ -3,10 +3,19 @@ package proxmox import ( "context" "errors" + "net/http" + "path/filepath" + "slices" + "strconv" + "strings" "testing" "time" + sdktrace "go.opentelemetry.io/otel/sdk/trace" + "go.opentelemetry.io/otel/sdk/trace/tracetest" + "gitea.home.hrajfrisbee.cz/kacerr/egress-proxies-operator/internal/provider" + "gitea.home.hrajfrisbee.cz/kacerr/egress-proxies-operator/internal/tracing" ) const ( @@ -101,3 +110,589 @@ func assertProviderError(t *testing.T, err error, op string, class error) { } } } + +// within runs fn and fails the test when it has not returned after d, so +// a call that ignores its ctx fails by name instead of hanging the package +// until the test binary's timeout. +func within(t *testing.T, d time.Duration, fn func()) { + t.Helper() + done := make(chan struct{}) + go func() { + defer close(done) + fn() + }() + select { + case <-done: + case <-time.After(d): + t.Fatalf("did not return within %v", d) + } +} + +// cancelWhileBlocked runs call with a ctx it cancels once call has reached +// method, which blocks until then, and fails unless call returns within a +// second of the cancel. Every wait is bounded, so a call that never reaches +// method or never returns fails instead of hanging the package. +func cancelWhileBlocked(t *testing.T, fx *fixture, method string, call func(ctx context.Context)) { + t.Helper() + release := fx.fake.blockOn(method) + defer close(release) + ctx, cancel := context.WithCancel(fx.ctx) + defer cancel() + returned := make(chan struct{}) + go func() { + defer close(returned) + call(ctx) + }() + deadline := time.After(5 * time.Second) + for fx.fake.callCount(method) == 0 { + select { + case <-returned: + t.Fatalf("returned before calling %s; calls: %v", method, fx.fake.callLog()) + case <-deadline: + t.Fatalf("%s was not called within 5s; calls: %v", method, fx.fake.callLog()) + case <-time.After(time.Millisecond): + } + } + cancel() + select { + case <-returned: + case <-time.After(time.Second): + t.Fatalf("did not return within 1s of the cancel while blocked in %s", method) + } +} + +func TestProvider_cancelledContextReturnsPromptly(t *testing.T) { + t.Parallel() + tests := []struct { + name string + blockOn string + // first: blockOn is the operation's first seam call. + first bool + setup func(fx *fixture) + op string + call func(ctx context.Context, fx *fixture) error + }{ + { + name: "create", blockOn: "ClusterResources", first: true, op: "create", + call: func(ctx context.Context, fx *fixture) error { _, err := fx.p.Create(ctx, fx.req()); return err }, + }, + { + name: "create mid upload", blockOn: "UploadISO", op: "create", + call: func(ctx context.Context, fx *fixture) error { _, err := fx.p.Create(ctx, fx.req()); return err }, + }, + { + name: "get", blockOn: "ClusterResources", first: true, op: "get", + setup: func(fx *fixture) { fx.addOurVM(testVMID, testName, "running") }, + call: func(ctx context.Context, fx *fixture) error { _, err := fx.p.Get(ctx, fx.id()); return err }, + }, + { + name: "delete", blockOn: "ClusterResources", first: true, op: "delete", + setup: func(fx *fixture) { fx.addOurVM(testVMID, testName, "running") }, + call: func(ctx context.Context, fx *fixture) error { return fx.p.Delete(ctx, fx.id()) }, + }, + { + name: "delete mid stop", blockOn: "Stop", op: "delete", + setup: func(fx *fixture) { fx.addOurVM(testVMID, testName, "running") }, + call: func(ctx context.Context, fx *fixture) error { return fx.p.Delete(ctx, fx.id()) }, + }, + { + name: "list", blockOn: "ClusterResources", first: true, op: "list", + setup: func(fx *fixture) { fx.addOurVM(testVMID, testName, "running") }, + call: func(ctx context.Context, fx *fixture) error { _, err := fx.p.ListByTag(ctx); return err }, + }, + } + for _, tc := range tests { + t.Run(tc.name, func(t *testing.T) { + t.Parallel() + fx := newCreateFixture(t, 1) + if tc.setup != nil { + tc.setup(fx) + } + var err error + cancelWhileBlocked(t, fx, tc.blockOn, func(ctx context.Context) { err = tc.call(ctx, fx) }) + if calls := fx.fake.callLog(); tc.first && calls[0] != tc.blockOn { + t.Errorf("first call = %s, want %s", calls[0], tc.blockOn) + } + assertProviderError(t, err, tc.op, provider.ErrTransient) + if !errors.Is(err, context.Canceled) { + t.Errorf("error = %v, want it to carry context.Canceled", err) + } + }) + } + t.Run("check", func(t *testing.T) { + t.Parallel() + fx := newCheckFixture(t) + var got []provider.TargetStatus + cancelWhileBlocked(t, fx, "Version", func(ctx context.Context) { got = fx.p.Check(ctx) }) + if calls := fx.fake.callLog(); calls[0] != "Version" { + t.Errorf("first call = %s, want Version", calls[0]) + } + if len(got) != 1 || got[0].Ready || got[0].Reason != provider.ReasonTimeout { + t.Errorf("Check() = %+v, want one target not ready with reason %s", got, provider.ReasonTimeout) + } + }) +} + +func TestProvider_everyErrorIsWrappedWithConfiguredName(t *testing.T) { + t.Parallel() + createWith := func(mutate func(*provider.CreateRequest)) func(context.Context, *fixture) error { + return func(ctx context.Context, fx *fixture) error { + req := fx.req() + if mutate != nil { + mutate(&req) + } + _, err := fx.p.Create(ctx, req) + return err + } + } + get := func(ctx context.Context, fx *fixture) error { _, err := fx.p.Get(ctx, fx.id()); return err } + del := func(ctx context.Context, fx *fixture) error { return fx.p.Delete(ctx, fx.id()) } + list := func(ctx context.Context, fx *fixture) error { _, err := fx.p.ListByTag(ctx); return err } + holdNameLock := func(fx *fixture, cancel context.CancelFunc) { + release, err := fx.p.nameLocks.Acquire(context.Background(), testName) + if err != nil { + fx.t.Fatalf("Acquire error = %v", err) + } + fx.t.Cleanup(release) + cancel() + } + broken := fakeErr(596, "Broken pipe") + tests := []struct { + name string + setup func(fx *fixture, cancel context.CancelFunc) + call func(ctx context.Context, fx *fixture) error + op string + id string + class error + }{ + {name: "create without a name", call: createWith(func(r *provider.CreateRequest) { r.Name = "" }), op: "create", class: provider.ErrPermanent}, + {name: "create unknown zone", call: createWith(func(r *provider.CreateRequest) { r.Placement.Zone = "pv9-9" }), op: "create", class: provider.ErrPermanent}, + {name: "create unknown image", call: createWith(func(r *provider.CreateRequest) { r.Placement.Image = "arch" }), op: "create", class: provider.ErrPermanent}, + {name: "create waiting on the name lock", setup: holdNameLock, call: createWith(nil), op: "create", class: provider.ErrTransient}, + { + name: "create resources unreadable", + setup: func(fx *fixture, _ context.CancelFunc) { fx.fake.failWith("ClusterResources", broken) }, + call: createWith(nil), op: "create", class: provider.ErrTransient, + }, + { + name: "create clone forbidden", + setup: func(fx *fixture, _ context.CancelFunc) { + fx.fake.failWith("Clone", loadAPIError(fx.t, "err_forbidden")) + }, + call: createWith(nil), op: "create", class: provider.ErrPermanent, + }, + { + name: "create start task fails", + setup: func(fx *fixture, _ context.CancelFunc) { fx.fake.startFailsWith = "fake: start failed" }, + call: createWith(nil), op: "create", class: provider.ErrTransient, + }, + { + name: "get malformed id", + call: func(ctx context.Context, fx *fixture) error { _, err := fx.p.Get(ctx, "vms/abc/"+testName); return err }, + op: "get", id: "vms/abc/" + testName, class: provider.ErrPermanent, + }, + {name: "get missing vm", call: get, op: "get", id: "self", class: provider.ErrNotFound}, + { + name: "get locked vm", + setup: func(fx *fixture, _ context.CancelFunc) { fx.addOurVM(testVMID, testName, "running").lock = "backup" }, + call: get, op: "get", id: "self", class: provider.ErrTransient, + }, + { + name: "get unauthorized", + setup: func(fx *fixture, _ context.CancelFunc) { + fx.fake.failWith("ClusterResources", loadAPIError(fx.t, "err_unauthorized")) + }, + call: get, op: "get", id: "self", class: provider.ErrPermanent, + }, + { + name: "delete malformed id", + call: func(ctx context.Context, fx *fixture) error { return fx.p.Delete(ctx, "vms/150042") }, + op: "delete", id: "vms/150042", class: provider.ErrPermanent, + }, + {name: "delete waiting on the name lock", setup: holdNameLock, call: del, op: "delete", id: "self", class: provider.ErrTransient}, + { + name: "delete hibernated vm", + setup: func(fx *fixture, _ context.CancelFunc) { + fx.addOurVM(testVMID, testName, "stopped").lock = lockSuspended + }, + call: del, op: "delete", id: "self", class: provider.ErrPermanent, + }, + { + name: "delete destroy task fails", + setup: func(fx *fixture, _ context.CancelFunc) { + fx.addOurVM(testVMID, testName, "stopped") + fx.fake.destroyFailsWith = "fake: destroy failed" + }, + call: del, op: "delete", id: "self", class: provider.ErrTransient, + }, + { + name: "delete seed forbidden", + setup: func(fx *fixture, _ context.CancelFunc) { + fx.addOurVM(testVMID, testName, "stopped") + fx.fake.failWith("DeleteVolume", loadAPIError(fx.t, "err_forbidden_volume_delete")) + }, + call: del, op: "delete", id: "self", class: provider.ErrPermanent, + }, + { + name: "list resources unreadable", + setup: func(fx *fixture, _ context.CancelFunc) { fx.fake.failWith("ClusterResources", broken) }, + call: list, op: "list", class: provider.ErrTransient, + }, + { + name: "list unauthorized", + setup: func(fx *fixture, _ context.CancelFunc) { + fx.fake.failWith("ClusterResources", loadAPIError(fx.t, "err_unauthorized")) + }, + call: list, op: "list", class: provider.ErrPermanent, + }, + { + name: "list ended on a candidate", + setup: func(fx *fixture, cancel context.CancelFunc) { + fx.addOurVM(testVMID, testName, "running") + fx.fake.hookOn("VMConfig", func() error { + cancel() + return broken + }) + }, + call: list, op: "list", class: provider.ErrTransient, + }, + } + for _, tc := range tests { + t.Run(tc.name, func(t *testing.T) { + t.Parallel() + fx := newCreateFixture(t, 1) + ctx, cancel := context.WithCancel(fx.ctx) + defer cancel() + if tc.setup != nil { + tc.setup(fx, cancel) + } + var err error + within(t, 5*time.Second, func() { err = tc.call(ctx, fx) }) + assertProviderError(t, err, tc.op, tc.class) + wantID := tc.id + if wantID == "self" { + wantID = fx.id() + } + var pe *provider.Error + if errors.As(err, &pe) && pe.ID != wantID { + t.Errorf("error.ID = %q, want %q", pe.ID, wantID) + } + if !strings.HasPrefix(err.Error(), "provider "+testProvider+": "+tc.op) { + t.Errorf("error = %q, want it to start with the configured name and op", err) + } + }) + } +} + +// logSentinel is in the CloudInit the logging lifecycle creates with; no +// log line at any level may carry it. +const logSentinel = "SENTINEL-cloud-init-7e2a" + +// newLoggingFixture is an observed create fixture with a token file, so +// one lifecycle covers every operation, Check and the decorator's lines. +func newLoggingFixture(t *testing.T, verbosity int) *fixture { + t.Helper() + fx := newCreateFixture(t, verbosity) + fx.p.api = observe(fx.fake, testProvider) + fx.p.cfg.TokenFile = filepath.Join(t.TempDir(), "token") + writeFile(t, fx.p.cfg.TokenFile, testToken+"\n") + return fx +} + +// sentinelReq is fx.req() with logSentinel in its CloudInit. +func (fx *fixture) sentinelReq() provider.CreateRequest { + req := fx.req() + req.CloudInit = "#cloud-config\nwrite_files:\n - path: /etc/sentinel\n content: " + logSentinel + "\n" + return req +} + +// lifecycle runs Create with sentinelReq, a Get that finds the address, +// ListByTag, Delete, Delete of the gone VM, and Check. +func (fx *fixture) lifecycle() { + fx.t.Helper() + id := fx.create(fx.sentinelReq()) + ifaces := capturedIfaces(fx.t) + fx.fake.mu.Lock() + fx.fake.vms[slot(0)].ifaces = ifaces + fx.fake.mu.Unlock() + if inst := fx.get(id); inst.State != provider.StateRunning { + fx.t.Fatalf("Get state = %s, want Running", inst.State) + } + if _, err := fx.p.ListByTag(fx.ctx); err != nil { + fx.t.Fatalf("ListByTag error = %v", err) + } + for range 2 { + if err := fx.p.Delete(fx.ctx, id); err != nil { + fx.t.Fatalf("Delete error = %v", err) + } + } + if ts := fx.p.Check(fx.ctx); len(ts) != 1 || !ts[0].Ready { + fx.t.Fatalf("Check = %+v, want one Ready target", ts) + } +} + +// assertNeverLogged checks every line for the cloud-init text and the +// token's secret. +func assertNeverLogged(t *testing.T, lines []string) { + t.Helper() + secret := testToken[strings.Index(testToken, "=")+1:] + for _, l := range lines { + for _, never := range []string{logSentinel, "#cloud-config", secret} { + if strings.Contains(l, never) { + t.Errorf("line carries %q: %s", never, l) + } + } + } +} + +func msgLine(msg string) string { return `"msg"="` + msg + `"` } + +// assertEveryV1Line checks that each line is from the proxmox logger at +// V(1) with the provider, and that each line about an instance, which is +// all but the decorator's, the list's and the check's, names its vmid and +// name. +func assertEveryV1Line(t *testing.T, lines []string, vmid, name string) { + t.Helper() + notInstance := []string{msgLine("proxmox api call"), msgLine("proxmox vms listed"), msgLine("proxmox node checked")} + for _, l := range lines { + if !strings.HasPrefix(l, "proxmox ") || !strings.Contains(l, `"provider"="`+testProvider+`"`) { + t.Errorf("line lacks the proxmox logger or the provider: %s", l) + } + if !strings.Contains(l, `"level"=1`) { + t.Errorf("line is not at V(1): %s", l) + } + instance := !slices.ContainsFunc(notInstance, func(m string) bool { return strings.Contains(l, m) }) + if instance && (!strings.Contains(l, vmid) || !strings.Contains(l, name)) { + t.Errorf("instance line lacks vmid or name: %s", l) + } + } +} + +func TestLogging_v0SilentV1OutcomeV2Detail(t *testing.T) { + t.Parallel() + vmid := `"vmid"=` + strconv.Itoa(slot(0)) + name := `"name"="` + testName + `"` + t.Run("v0 is silent", func(t *testing.T) { + t.Parallel() + fx := newLoggingFixture(t, 0) + fx.lifecycle() + if lines := fx.logs.all(); len(lines) != 0 { + t.Errorf("the lifecycle logged %d lines at V(0), want none:\n%s", len(lines), strings.Join(lines, "\n")) + } + }) + t.Run("v1 has one outcome per operation", func(t *testing.T) { + t.Parallel() + fx := newLoggingFixture(t, 1) + fx.lifecycle() + all := fx.logs.all() + dump := strings.Join(all, "\n") + outcomes := map[string][]string{ + "proxmox vm ensured running": {vmid, name, `"node"="pv1-1"`}, + "proxmox vm fetched": {vmid, name, `"state"="Running"`, `"ip"="192.168.0.150"`, `"uid"="` + testUID + `"`}, + "proxmox vms listed": {`"candidates"=1`, `"managed"=1`, `"skipped"=0`}, + "proxmox node checked": {`"node"="pv1-1"`, `"ready"=true`, `"reason"="Reachable"`}, + } + for msg, keys := range outcomes { + lines := fx.logs.matching(msgLine(msg)) + if len(lines) != 1 { + t.Errorf("lines %q = %d, want 1:\n%s", msg, len(lines), dump) + continue + } + for _, k := range keys { + if !strings.Contains(lines[0], k) { + t.Errorf("%q line lacks %s: %s", msg, k, lines[0]) + } + } + } + deleted := fx.logs.matching(msgLine("proxmox vm deleted")) + if len(deleted) != 2 { + t.Fatalf("deleted lines = %d, want 2 (destroyed, then already gone):\n%s", len(deleted), dump) + } + if l := deleted[0]; !strings.Contains(l, `"destroyed"=true`) || !strings.Contains(l, `"node"="pv1-1"`) || !strings.Contains(l, vmid) || !strings.Contains(l, name) { + t.Errorf("first deleted line = %s, want destroyed=true with node, vmid and name", l) + } + if l := deleted[1]; !strings.Contains(l, `"destroyed"=false`) || strings.Contains(l, `"node"=`) || !strings.Contains(l, vmid) || !strings.Contains(l, name) { + t.Errorf("second deleted line = %s, want destroyed=false with vmid and name and no node", l) + } + assertEveryV1Line(t, all, vmid, name) + for _, v2 := range []string{"cloudInitBytes", "proxmox create resolved", "proxmox guest agent answered", "proxmox listed instance", `"description"=`, `"config"=`, `"params"=`} { + if got := fx.logs.matching(v2); len(got) != 0 { + t.Errorf("V(1) carries the V(2) detail %s:\n%s", v2, strings.Join(got, "\n")) + } + } + assertNeverLogged(t, all) + }) + t.Run("v2 has detail and never the cloud-init", func(t *testing.T) { + t.Parallel() + fx := newLoggingFixture(t, 2) + fx.lifecycle() + resolved := fx.logs.matching(msgLine("proxmox create resolved")) + if len(resolved) != 1 { + t.Fatalf("resolved lines = %d, want 1", len(resolved)) + } + for _, k := range []string{name, `"cloudInitBytes"=` + strconv.Itoa(len(fx.sentinelReq().CloudInit)), `"machineType"="small"`, `"cores"=1`, `"memoryMiB"=1024`, `"templateVMID"=9001`, `"isoStorage"="proxy-seeds"`} { + if !strings.Contains(resolved[0], k) { + t.Errorf("resolved line lacks %s: %s", k, resolved[0]) + } + } + details := map[string]string{ + msgLine("proxmox guest agent answered"): `"ip"="192.168.0.150"`, + msgLine("proxmox listed instance"): `"uid"="` + testUID + `"`, + `"call"="Clone"`: `"description"="proxy-operator-managed=true`, + `"call"="VMConfig"`: `"config"=`, + `"call"="UploadISO"`: `"bytes"=`, + } + for line, k := range details { + got := fx.logs.matching(line) + if len(got) == 0 || !strings.Contains(got[0], k) { + t.Errorf("V(2) %s line lacks %s: %v", line, k, got) + } + } + assertNeverLogged(t, fx.logs.all()) + }) +} + +// TestLogging_noOpLinesAtV1: spec entry 20's no-op lines of Delete and +// ListByTag are there at V(1), each naming the instance. +func TestLogging_noOpLinesAtV1(t *testing.T) { + t.Parallel() + del := func(fx *fixture) error { return fx.p.Delete(fx.ctx, fx.id()) } + tests := []struct { + name string + setup func(fx *fixture) + call func(fx *fixture) error + msg string + want int + }{ + {name: "vm already gone", call: del, msg: "proxmox vm already gone", want: 1}, + { + name: "foreign vm", + setup: func(fx *fixture) { fx.addOurVM(testVMID, "proxy-someoneelse00000", "running") }, + call: del, msg: "proxmox vm at this vmid is not ours, left alone", want: 1, + }, + { + name: "unmarked vm", + setup: func(fx *fixture) { fx.fake.addVM(testVMID, testName, "running") }, + call: del, msg: "proxmox vm carries no proxy-operator mark, left alone", want: 1, + }, + { + name: "seed delete skipped on each node", + setup: func(fx *fixture) { + fx.p.cfg.Nodes = []string{fakeNode, "pv1-2"} + fx.fake.failWith("DeleteVolume", loadAPIError(fx.t, "err_forbidden_volume_delete")) + }, + call: del, msg: "proxmox seed delete failed on a node, skipped", want: 2, + }, + { + name: "unreadable candidate", + setup: func(fx *fixture) { + fx.addOurVM(testVMID, testName, "running") + fx.fake.failWith("VMConfig", fakeErr(596, "Broken pipe")) + }, + call: func(fx *fixture) error { _, err := fx.p.ListByTag(fx.ctx); return err }, + msg: "skipping unreadable proxmox vm", want: 1, + }, + } + for _, tc := range tests { + t.Run(tc.name, func(t *testing.T) { + t.Parallel() + fx := newFixture(t, 1) + if tc.setup != nil { + tc.setup(fx) + } + if err := tc.call(fx); err != nil { + t.Fatalf("error = %v, want nil", err) + } + lines := fx.logs.matching(msgLine(tc.msg)) + if len(lines) != tc.want { + t.Fatalf("lines %q = %d, want %d:\n%s", tc.msg, len(lines), tc.want, strings.Join(fx.logs.all(), "\n")) + } + for _, l := range lines { + for _, k := range []string{`"provider"="` + testProvider + `"`, `"vmid"=150042`, `"level"=1`} { + if !strings.Contains(l, k) { + t.Errorf("line lacks %s: %s", k, l) + } + } + } + }) + } +} + +// TestTracing_httpSpansUnderOperationSpan runs a Get through +// provider.WithTracing over the real client against Phase 0's captures: +// each of Get's four requests is an HTTP span under provider.get, and no +// span attribute carries the token. +func TestTracing_httpSpansUnderOperationSpan(t *testing.T) { + t.Parallel() + const ( + vmid = 199999 + name = "proxy-reconaaaaaaaaaaa" + ) + cfx := newClientFixture(t, func(w http.ResponseWriter, r *http.Request) { + path := strings.TrimPrefix(r.URL.Path, "/api2/json/") + capture := map[string]string{ + "cluster/resources": "cluster_resources_during_destroy", + "nodes/pv1-1/qemu/199999/config": "config_during_destroy", + "nodes/pv1-1/qemu/199999/status/current": "status_current_running", + "nodes/pv1-1/qemu/199999/agent/network-get-interfaces": "agent_network_get_interfaces", + }[path] + if r.Method != http.MethodGet || capture == "" { + t.Errorf("unexpected request %s %s", r.Method, path) + http.NotFound(w, r) + return + } + serveJSON(w, readCapture(t, capture).Body) + }) + sr := tracetest.NewSpanRecorder() + tp := sdktrace.NewTracerProvider(sdktrace.WithSpanProcessor(sr)) + t.Cleanup(func() { _ = tp.Shutdown(context.Background()) }) + opt := tracing.WithTracerProvider(tp) + + cfg := testConfig() + cfg.URL, cfg.TokenFile, cfg.CAFile = cfx.cfg.URL, cfx.cfg.TokenFile, cfx.cfg.CAFile + p := newProvider(testProvider, cfg, observe(cfx.api(t, opt), testProvider)) + p.budgets = testBudgets() + traced := provider.WithTracing(testProvider, p, opt) + + inst, err := traced.Get(context.Background(), formatProviderID(vmid, name)) + if err != nil { + t.Fatalf("Get error = %v", err) + } + if inst.State != provider.StateRunning || inst.IP != "192.168.0.150" { + t.Errorf("Get = %+v, want Running at 192.168.0.150", inst) + } + + var ( + op sdktrace.ReadOnlySpan + requests []sdktrace.ReadOnlySpan + ) + for _, s := range sr.Ended() { + if s.Name() == "provider.get" { + op = s + } else { + requests = append(requests, s) + } + } + if op == nil { + t.Fatalf("no provider.get span among %d", len(sr.Ended())) + } + if len(requests) != 4 { + t.Errorf("HTTP spans = %d, want one per request of Get (4)", len(requests)) + } + for _, s := range requests { + if s.Name() != "HTTP GET" { + t.Errorf("span %q, want HTTP GET (spec entry 17)", s.Name()) + } + if s.Parent().SpanID() != op.SpanContext().SpanID() || s.SpanContext().TraceID() != op.SpanContext().TraceID() { + t.Errorf("span %q is not a child of provider.get", s.Name()) + } + } + secret := testToken[strings.Index(testToken, "=")+1:] + for _, s := range sr.Ended() { + for _, kv := range s.Attributes() { + if v := kv.Value.Emit(); strings.Contains(v, secret) || strings.Contains(v, "PVEAPIToken") { + t.Errorf("span %s attribute %s carries the token: %s", s.Name(), kv.Key, v) + } + } + } +} diff --git a/internal/provider/proxmox/proxmox_integration_test.go b/internal/provider/proxmox/proxmox_integration_test.go new file mode 100644 index 0000000..b63ee8d --- /dev/null +++ b/internal/provider/proxmox/proxmox_integration_test.go @@ -0,0 +1,624 @@ +//go:build integration + +package proxmox + +import ( + "context" + "crypto/rand" + "encoding/hex" + "errors" + "fmt" + "net" + "net/http" + "net/url" + "os" + "strconv" + "strings" + "sync" + "testing" + "time" + + "github.com/go-logr/logr" + "github.com/go-logr/logr/funcr" + sdktrace "go.opentelemetry.io/otel/sdk/trace" + "go.opentelemetry.io/otel/sdk/trace/tracetest" + "k8s.io/apimachinery/pkg/types" + logf "sigs.k8s.io/controller-runtime/pkg/log" + + "gitea.home.hrajfrisbee.cz/kacerr/egress-proxies-operator/internal/provider" + "gitea.home.hrajfrisbee.cz/kacerr/egress-proxies-operator/internal/tracing" +) + +// The real-host test of spec entry 21. It runs only with -tags=integration +// and skips itself unless PROXMOX_TEST_* names a host; see docs/testing.md +// for the variables and what one run asserts. +const ( + integrationProvider = "proxmox-integration" + integrationImage = "test-template" + integrationMachine = "small" + integrationPort = 3128 + // integrationDiskGiB is above the 3 GiB of the recon template, so a run + // on such a template exercises the resize task; a larger template skips + // it. + integrationDiskGiB = 4 + + // integrationCloudInit is the user-data the VM boots with. Nothing + // asserts on the guest running it; it exists so the sentinel check can + // prove that no span attribute carries the cloud-init text. + integrationCloudInit = "#cloud-config\nruncmd:\n - echo proxmox-integration\n" + + defaultIntegrationTimeout = 20 * time.Minute + defaultIntegrationLogLvl = 1 + getPollInterval = 5 * time.Second + cleanupBudget = 5 * time.Minute +) + +type integrationEnv struct { + url, tokenFile, node, isoStorage string + caFile, storage, bridge string + templateVMID, vlanTag int + vmidMin, vmidMax int + insecure, keep bool + timeout time.Duration + logLevel int +} + +// loadIntegrationEnv reads the PROXMOX_TEST_* variables, skipping the test +// under -short or when a required one is missing. +func loadIntegrationEnv(t *testing.T) integrationEnv { + t.Helper() + if testing.Short() { + t.Skip("real-host test skipped under -short") + } + env := integrationEnv{timeout: defaultIntegrationTimeout, logLevel: defaultIntegrationLogLvl} + var templateVMID string + required := []struct { + name string + dst *string + }{ + {"PROXMOX_TEST_URL", &env.url}, + {"PROXMOX_TEST_TOKEN_FILE", &env.tokenFile}, + {"PROXMOX_TEST_NODE", &env.node}, + {"PROXMOX_TEST_TEMPLATE_VMID", &templateVMID}, + {"PROXMOX_TEST_ISO_STORAGE", &env.isoStorage}, + } + for _, r := range required { + v := os.Getenv(r.name) + if v == "" { + t.Skipf("%s is not set; the proxmox integration test needs a real Proxmox VE host (see docs/testing.md)", r.name) + } + *r.dst = v + } + env.templateVMID = envInt(t, "PROXMOX_TEST_TEMPLATE_VMID", 0) + env.caFile = os.Getenv("PROXMOX_TEST_CA_FILE") + env.storage = os.Getenv("PROXMOX_TEST_STORAGE") + env.bridge = os.Getenv("PROXMOX_TEST_BRIDGE") + env.vlanTag = envInt(t, "PROXMOX_TEST_VLAN_TAG", 0) + env.vmidMin = envInt(t, "PROXMOX_TEST_VMID_MIN", defaultVMIDMin) + env.vmidMax = envInt(t, "PROXMOX_TEST_VMID_MAX", defaultVMIDMax) + if env.vmidMin > env.vmidMax { + t.Fatalf("PROXMOX_TEST_VMID_MIN=%d is above PROXMOX_TEST_VMID_MAX=%d", env.vmidMin, env.vmidMax) + } + env.keep = os.Getenv("PROXMOX_TEST_KEEP") != "" + if v := os.Getenv("PROXMOX_TEST_INSECURE"); v != "" { + b, err := strconv.ParseBool(v) + if err != nil { + t.Fatalf("PROXMOX_TEST_INSECURE=%q: %v", v, err) + } + env.insecure = b + } + if v := os.Getenv("PROXMOX_TEST_TIMEOUT"); v != "" { + d, err := time.ParseDuration(v) + if err != nil { + t.Fatalf("PROXMOX_TEST_TIMEOUT=%q: %v", v, err) + } + env.timeout = d + } + env.logLevel = envInt(t, "PROXMOX_TEST_LOG_LEVEL", defaultIntegrationLogLvl) + return env +} + +// envInt reads a non-negative integer variable, def when it is unset. +func envInt(t *testing.T, name string, def int) int { + t.Helper() + v := os.Getenv(name) + if v == "" { + return def + } + n, err := strconv.Atoi(v) + if err != nil || n < 0 { + t.Fatalf("%s=%q: want a non-negative integer", name, v) + } + return n +} + +// providerConfig is what the operator would load for this host: one node, +// one template, one machine type. +func (e integrationEnv) providerConfig() provider.ProviderConfig { + return provider.ProviderConfig{ + Name: integrationProvider, + Type: "proxmox", + Proxmox: &provider.ProxmoxConfig{ + URL: e.url, + TokenFile: e.tokenFile, + CAFile: e.caFile, + InsecureSkipTLSVerify: e.insecure, + Nodes: []string{e.node}, + ISOStorage: e.isoStorage, + Storage: e.storage, + Bridge: e.bridge, + VLANTag: e.vlanTag, + VMIDRange: &provider.ProxmoxVMIDRange{Min: e.vmidMin, Max: e.vmidMax}, + Templates: map[string]provider.ProxmoxTemplate{integrationImage: {VMID: e.templateVMID}}, + MachineTypes: map[string]provider.ProxmoxMachineType{ + integrationMachine: {Cores: 1, MemoryMiB: 1024, DiskGiB: integrationDiskGiB}, + }, + }, + } +} + +// integrationLog routes the provider's logr output into t.Log through funcr +// at the chosen verbosity, keeps every line for assertions, and strips the +// key/value pairs that are constant for the run (stated once up front) so +// each line stays short. Lines arriving after the test ended are dropped: +// t.Log panics then. +type integrationLog struct { + t *testing.T + + mu sync.Mutex + drop []string + lines []string + done bool +} + +func newIntegrationLog(t *testing.T, level int, drop ...string) (*integrationLog, logr.Logger) { + t.Helper() + sink := &integrationLog{t: t, drop: drop} + t.Cleanup(func() { + sink.mu.Lock() + defer sink.mu.Unlock() + sink.done = true + }) + log := funcr.New(func(prefix, args string) { sink.write(prefix, args) }, funcr.Options{Verbosity: level}) + return sink, log +} + +// dropValue adds a rendered pair (`"key"="value"`) to strip from later lines. +func (l *integrationLog) dropValue(key, value string) { + l.mu.Lock() + defer l.mu.Unlock() + l.drop = append(l.drop, fmt.Sprintf("%q=%q", key, value)) +} + +func (l *integrationLog) write(prefix, args string) { + l.mu.Lock() + defer l.mu.Unlock() + l.lines = append(l.lines, args) + if l.done { + return + } + for _, d := range l.drop { + args = strings.ReplaceAll(args, " "+d, "") + } + l.t.Logf("%s %s %s", time.Now().Format("15:04:05.000"), prefix, args) +} + +func (l *integrationLog) matching(substr string) []string { + l.mu.Lock() + defer l.mu.Unlock() + var out []string + for _, line := range l.lines { + if strings.Contains(line, substr) { + out = append(out, line) + } + } + return out +} + +// newIntegrationProvider builds the provider the way New does, with the +// span recorder's tracer on the transport, so the HTTP spans land in the +// same recorder as provider.WithTracing's provider.* spans. The bare client +// is returned too, for the test's own read-only inspection of the host. +func newIntegrationProvider(t *testing.T, pc provider.ProviderConfig, tp *sdktrace.TracerProvider) (*Provider, *httpAPI) { + t.Helper() + cfg := withDefaults(*pc.Proxmox) + client, err := newHTTPAPI(cfg, tracing.WithTracerProvider(tp)) + if err != nil { + t.Fatalf("newHTTPAPI error = %v", err) + } + return newProvider(pc.Name, cfg, observe(client, pc.Name)), client +} + +// randomUID is a UUID-shaped string, so the run's VM never collides with a +// previous run's leftovers or with a real Proxy's. +func randomUID(t *testing.T) string { + t.Helper() + var b [16]byte + if _, err := rand.Read(b[:]); err != nil { + t.Fatalf("reading random bytes: %v", err) + } + h := hex.EncodeToString(b[:]) + return h[0:8] + "-" + h[8:12] + "-" + h[12:16] + "-" + h[16:20] + "-" + h[20:32] +} + +// stepDone logs one lifecycle step with its duration; the log is the record +// of how long a clone or a boot took on this host. +func stepDone(t *testing.T, step string, start time.Time) { + t.Helper() + t.Logf("step %s: %s", step, time.Since(start).Round(time.Millisecond)) +} + +// waitUntil polls cond every interval until it is true or ctx ends. +func waitUntil(ctx context.Context, t *testing.T, interval time.Duration, what string, cond func() bool) { + t.Helper() + ticker := time.NewTicker(interval) + defer ticker.Stop() + for !cond() { + select { + case <-ticker.C: + case <-ctx.Done(): + t.Fatalf("gave up waiting for %s: %v", what, ctx.Err()) + } + } +} + +// qemuVMs lists the cluster's QEMU VMs by VMID. +func qemuVMs(ctx context.Context, t *testing.T, client *httpAPI) map[int]vmResource { + t.Helper() + rs, err := client.ClusterResources(ctx) + if err != nil { + t.Fatalf("listing cluster resources: %v", err) + } + out := make(map[int]vmResource, len(rs)) + for _, r := range rs { + if r.Type == "qemu" { + out[r.VMID] = r + } + } + return out +} + +// vmidByName finds the VM named name, as a crashed Create may have left it. +func vmidByName(ctx context.Context, t *testing.T, client *httpAPI, name string) (int, bool) { + t.Helper() + for vmid, r := range qemuVMs(ctx, t, client) { + if r.Name == name { + return vmid, true + } + } + return 0, false +} + +// seedOnStorage reports whether the ISO storage on node holds name's seed. +// It is one extra read-only GET of /nodes/{node}/storage/{storage}/content +// (captured as storage_content_iso.json): the seam has no method to list +// storage content, and gets none for a test. +func seedOnStorage(ctx context.Context, t *testing.T, client *httpAPI, node, storage, name string) bool { + t.Helper() + var content []struct { + VolID string `json:"volid"` + } + path := apiPath("nodes", node, "storage", storage, "content") + if err := client.do(ctx, http.MethodGet, path, url.Values{"content": {"iso"}}, nil, &content); err != nil { + t.Fatalf("listing iso content of %s on %s: %v", storage, node, err) + } + want := seedVolID(storage, name) + for _, c := range content { + if c.VolID == want { + return true + } + } + return false +} + +// checkNode runs the health check and requires the one node Ready, with the +// details an admin reads. +func checkNode(ctx context.Context, t *testing.T, built provider.Provider, env integrationEnv) { + t.Helper() + checker, ok := built.(provider.Checker) + if !ok { + t.Fatalf("New() returned %T, which is not a provider.Checker", built) + } + targets := checker.Check(ctx) + if len(targets) != 1 { + t.Fatalf("Check() returned %d targets, want 1: %+v", len(targets), targets) + } + ts := targets[0] + if !ts.Ready || ts.Reason != provider.ReasonReachable { + t.Fatalf("Check() = ready %v reason %q message %q, want Ready/%s (details %v)", ts.Ready, ts.Reason, ts.Message, provider.ReasonReachable, ts.Details) + } + for _, key := range []string{"pveVersion", "kernel", "tokenID", "isoStorageAvailGiB", "templates"} { + if ts.Details[key] == "" { + t.Errorf("Check() details lack %q: %v", key, ts.Details) + } + } + if ts.Details["templates"] != "1/1" { + t.Errorf("Check() templates = %q, want 1/1", ts.Details["templates"]) + } + t.Logf("node %s (%s): pve %s, kernel %s, token %s, %s free on %s, templates %s, latency %s", + ts.Name, ts.Address, ts.Details["pveVersion"], ts.Details["kernel"], ts.Details["tokenID"], + ts.Details["isoStorageAvailGiB"]+" GiB", env.isoStorage, ts.Details["templates"], ts.Latency.Round(time.Millisecond)) +} + +func TestProxmox_lifecycle(t *testing.T) { + env := loadIntegrationEnv(t) + pc := env.providerConfig() + sink, log := newIntegrationLog(t, env.logLevel, + fmt.Sprintf("%q=%q", "provider", integrationProvider), + fmt.Sprintf("%q=%q", "node", env.node)) + ctx, cancel := context.WithTimeout(logf.IntoContext(context.Background(), log), env.timeout) + defer cancel() + + t.Logf("provider=%s url=%s node=%s template=%d isoStorage=%s storage=%q bridge=%q vlanTag=%d vmids %d..%d budget=%s log level %d; provider, node and name are left off every provider log line", + integrationProvider, env.url, env.node, env.templateVMID, env.isoStorage, env.storage, env.bridge, env.vlanTag, + env.vmidMin, env.vmidMax, env.timeout, env.logLevel) + if env.insecure { + t.Log("PROXMOX_TEST_INSECURE is set: the API's certificate is NOT verified for this run") + } + + // 1. New returns without a request. + start := time.Now() + built, err := New(ctx, pc) + elapsed := time.Since(start) + if err != nil { + t.Fatalf("New() error = %v", err) + } + if elapsed > 100*time.Millisecond { + t.Errorf("New() took %s, want under 100ms: it must not make a request", elapsed) + } + if lines := sink.matching(`"msg"="proxmox api call"`); len(lines) > 0 { + t.Errorf("New() called the API before any operation: %q", lines) + } + stepDone(t, "1 New", start) + + // 2. Check reports the node Ready with details. + start = time.Now() + checkNode(ctx, t, built, env) + stepDone(t, "2 Check", start) + + // 3. Every later call goes through the tracing decorator, with the + // transport's HTTP spans recorded alongside. + sr := tracetest.NewSpanRecorder() + tp := sdktrace.NewTracerProvider(sdktrace.WithSpanProcessor(sr)) + t.Cleanup(func() { _ = tp.Shutdown(context.Background()) }) + p, client := newIntegrationProvider(t, pc, tp) + traced := provider.WithTracing(pc.Name, p, tracing.WithTracerProvider(tp)) + + // 4. Create. + uid := randomUID(t) + name := provider.NameFromUID(types.UID(uid)) + sink.dropValue("name", name) + req := provider.CreateRequest{ + Name: name, + UID: uid, + Namespace: "integration", + ProxyName: "proxmox-lifecycle", + Placement: provider.Placement{Zone: env.node, MachineType: integrationMachine, Image: integrationImage}, + CloudInit: integrationCloudInit, + Port: integrationPort, + } + t.Logf("VM name %s, uid %s", name, uid) + + deleted := false + t.Cleanup(func() { + if deleted { + return + } + cctx, ccancel := context.WithTimeout(logf.IntoContext(context.Background(), log), cleanupBudget) + defer ccancel() + vmid, found := vmidByName(cctx, t, client, name) + if env.keep { + msg := "PROXMOX_TEST_KEEP is set: the VM and its seed stay. Clean up as root on " + env.node + " with:\n" + if found { + msg += fmt.Sprintf(" qm stop %d\n qm destroy %d --purge\n", vmid, vmid) + } + t.Log(msg + " pvesm free " + seedVolID(env.isoStorage, name)) + return + } + if !found { + // No VM, but a seed may be left: Delete of an absent VM still + // deletes it. + vmid = vmidFor(uid, env.vmidMin, env.vmidMax, 0) + } + if err := p.Delete(cctx, formatProviderID(vmid, name)); err != nil { + t.Errorf("cleanup Delete of VM %d error = %v; remove it and %s by hand", vmid, err, seedVolID(env.isoStorage, name)) + } + }) + + start = time.Now() + id, err := traced.Create(ctx, req) + if err != nil { + t.Fatalf("Create() error = %v", err) + } + vmid, gotName, err := parseProviderID(id) + if err != nil || gotName != name { + t.Fatalf("Create() = %q, want vms//%s (parse error %v)", id, name, err) + } + if vmid < env.vmidMin || vmid > env.vmidMax { + t.Errorf("Create() VMID %d is outside %d..%d", vmid, env.vmidMin, env.vmidMax) + } + sink.dropValue("vmid", strconv.Itoa(vmid)) + createReturned := time.Now() + t.Logf("VM %d, id %s", vmid, id) + stepDone(t, "4 Create", start) + + // 5. Create again: same ID, nothing new in the cluster. + start = time.Now() + before := len(qemuVMs(ctx, t, client)) + again, err := traced.Create(ctx, req) + if err != nil { + t.Fatalf("second Create() error = %v", err) + } + if again != id { + t.Errorf("second Create() = %q, want %q", again, id) + } + if after := len(qemuVMs(ctx, t, client)); after != before { + t.Errorf("second Create() changed the cluster's VM count from %d to %d", before, after) + } + stepDone(t, "5 Create again", start) + + // 6. Get reaches Running with an IPv4, and reads back what Create wrote. + start = time.Now() + var inst *provider.Instance + waitUntil(ctx, t, getPollInterval, "the VM to be Running with an IPv4", func() bool { + var err error + inst, err = traced.Get(ctx, id) + if errors.Is(err, provider.ErrTransient) { + t.Logf("Get(%s) is Transient while the guest boots: %v", id, err) + return false + } + if err != nil { + t.Fatalf("Get(%s) error = %v", id, err) + } + return inst.State == provider.StateRunning && net.ParseIP(inst.IP).To4() != nil + }) + assertMarkReadBack(ctx, t, client, env.node, vmid, inst, uid) + t.Logf("VM running at %s, %s after Create returned (createdAt %s)", inst.IP, time.Since(createReturned).Round(time.Millisecond), inst.CreatedAt.Format(time.RFC3339)) + stepDone(t, "6 Get until Running", start) + + // 7. ListByTag contains it. + start = time.Now() + list, err := traced.ListByTag(ctx) + if err != nil { + t.Fatalf("ListByTag() error = %v", err) + } + assertListed(t, list, inst) + stepDone(t, "7 ListByTag", start) + + // 8. Reachability, logged only: this machine may not route to the VM's + // network. + start = time.Now() + addr := net.JoinHostPort(inst.IP, "22") + if conn, err := net.DialTimeout("tcp", addr, 3*time.Second); err != nil { + t.Logf("TCP connect to %s failed (informational; the test host may not route to the VM's network): %v", addr, err) + } else { + _ = conn.Close() + t.Logf("TCP connect to %s succeeded", addr) + } + stepDone(t, "8 TCP connect", start) + + if env.keep { + t.Log("PROXMOX_TEST_KEEP is set: skipping Delete and the leftovers check") + } else { + // 9. Delete, Get is NotFound, Delete again. + start = time.Now() + if err := traced.Delete(ctx, id); err != nil { + t.Fatalf("Delete() error = %v", err) + } + deleted = true + if _, err := traced.Get(ctx, id); !errors.Is(err, provider.ErrNotFound) { + t.Errorf("Get() after Delete error = %v, want ErrNotFound", err) + } + if err := traced.Delete(ctx, id); err != nil { + t.Errorf("second Delete() error = %v, want nil", err) + } + stepDone(t, "9 Delete, Get, Delete", start) + + // 10. No VM with the name and no seed remain. + start = time.Now() + if left, ok := vmidByName(ctx, t, client, name); ok { + t.Errorf("VM %d named %s is still in the cluster after Delete", left, name) + } + if seedOnStorage(ctx, t, client, env.node, env.isoStorage, name) { + t.Errorf("%s is still on %s after Delete", seedVolID(env.isoStorage, name), env.node) + } + stepDone(t, "10 leftovers", start) + } + + // 11. Spans: HTTP spans are children of the decorator's. + assertLifecycleSpans(t, sr.Ended()) +} + +// assertMarkReadBack checks that Get's UID and CreatedAt are the mark +// Create wrote into the VM's description, read back from its config, and +// that CreatedAt is recent. +func assertMarkReadBack(ctx context.Context, t *testing.T, client *httpAPI, node string, vmid int, inst *provider.Instance, uid string) { + t.Helper() + cfg, err := client.VMConfig(ctx, node, vmid) + if err != nil { + t.Fatalf("reading the config of VM %d: %v", vmid, err) + } + written := parseDescription(cfg["description"]) + if inst.UID != uid || written.UID != uid { + t.Errorf("Get() UID = %q, the VM's mark says %q, want %q", inst.UID, written.UID, uid) + } + if !inst.CreatedAt.Equal(written.CreatedAt) || inst.CreatedAt.IsZero() { + t.Errorf("Get() CreatedAt = %s, the VM's mark says %s; want them equal and set", inst.CreatedAt, written.CreatedAt) + } else if skew := time.Since(inst.CreatedAt).Abs(); skew > 5*time.Minute { + t.Errorf("Get() CreatedAt = %s, %s from now; want within a few minutes", inst.CreatedAt, skew) + } +} + +// assertListed checks that list holds inst's ID with its UID and +// CreatedAt. +func assertListed(t *testing.T, list []provider.Instance, inst *provider.Instance) { + t.Helper() + for _, li := range list { + if li.ID != inst.ID { + continue + } + if li.UID != inst.UID || !li.CreatedAt.Equal(inst.CreatedAt) { + t.Errorf("ListByTag() entry = %+v, want UID %q and CreatedAt %s", li, inst.UID, inst.CreatedAt) + } + return + } + t.Errorf("ListByTag() = %d instances, none with ID %s", len(list), inst.ID) +} + +// assertLifecycleSpans checks that at least one provider.create span has an +// HTTP child, and that no attribute of any span carries the cloud-init text. +func assertLifecycleSpans(t *testing.T, spans []sdktrace.ReadOnlySpan) { + t.Helper() + satisfied := false + for _, create := range spans { + if create.Name() != "provider.create" { + continue + } + parent := create.SpanContext().SpanID() + children := 0 + for _, s := range spans { + if s.Parent().SpanID() == parent && strings.HasPrefix(s.Name(), "HTTP ") { + children++ + } + } + t.Logf("provider.create span %s: %d HTTP children", parent, children) + if children > 0 { + satisfied = true + } + } + if !satisfied { + names := make([]string, len(spans)) + for i, s := range spans { + names[i] = s.Name() + } + t.Errorf("no provider.create span has an HTTP child; spans: %v", names) + } + for _, s := range spans { + for _, a := range s.Attributes() { + v := a.Value.Emit() + if strings.Contains(v, integrationCloudInit) || strings.Contains(v, "#cloud-config") { + t.Errorf("span %s attribute %s carries the cloud-init text", s.Name(), a.Key) + } + } + } +} + +// TestProxmoxCheck_realHost runs the health check against the host and +// creates nothing. +func TestProxmoxCheck_realHost(t *testing.T) { + env := loadIntegrationEnv(t) + pc := env.providerConfig() + _, log := newIntegrationLog(t, env.logLevel, + fmt.Sprintf("%q=%q", "provider", integrationProvider), + fmt.Sprintf("%q=%q", "node", env.node)) + ctx, cancel := context.WithTimeout(logf.IntoContext(context.Background(), log), env.timeout) + defer cancel() + if env.insecure { + t.Log("PROXMOX_TEST_INSECURE is set: the API's certificate is NOT verified for this run") + } + built, err := New(ctx, pc) + if err != nil { + t.Fatalf("New() error = %v", err) + } + checkNode(ctx, t, built, env) +} diff --git a/internal/provider/proxmox/proxmox_test.go b/internal/provider/proxmox/proxmox_test.go index dd47efd..0fe034c 100644 --- a/internal/provider/proxmox/proxmox_test.go +++ b/internal/provider/proxmox/proxmox_test.go @@ -230,7 +230,7 @@ func TestLockName_givesUpWithContextAsTransient(t *testing.T) { other() ctx, cancel := context.WithTimeout(context.Background(), 20*time.Millisecond) defer cancel() - _, err = p.lockName(ctx, "proxy-abc") + within(t, 5*time.Second, func() { _, err = p.lockName(ctx, "proxy-abc") }) if !errors.Is(err, provider.ErrTransient) || !errors.Is(err, context.DeadlineExceeded) { t.Errorf("lockName on a held name error = %v, want ErrTransient wrapping DeadlineExceeded", err) } diff --git a/internal/provider/proxmox/task_test.go b/internal/provider/proxmox/task_test.go index a5c3b0b..c9b683e 100644 --- a/internal/provider/proxmox/task_test.go +++ b/internal/provider/proxmox/task_test.go @@ -110,7 +110,8 @@ func TestWaitTask_table(t *testing.T) { cancel() } start := time.Now() - err := fx.p.waitTask(ctx, fx.p.logger(ctx), upid, budget) + var err error + within(t, 10*time.Second, func() { err = fx.p.waitTask(ctx, fx.p.logger(ctx), upid, budget) }) if elapsed := time.Since(start); elapsed > 5*time.Second { t.Errorf("waitTask took %v", elapsed) } @@ -219,7 +220,11 @@ func TestWaitUnlocked_table(t *testing.T) { ctx, cancel = context.WithCancel(ctx) cancel() } - cfg, err := fx.p.waitUnlocked(ctx, fakeNode, testVMID, budget) + var ( + cfg vmConfig + err error + ) + within(t, 10*time.Second, func() { cfg, err = fx.p.waitUnlocked(ctx, fakeNode, testVMID, budget) }) if tc.wantErr == nil { if err != nil || cfg["name"] != testName || cfg["lock"] != tc.wantLock { t.Errorf("waitUnlocked = %v, %v; want the config with lock %q", cfg, err, tc.wantLock) From 7f2073f94bab93ba878254d31ff4a27ff4013225 Mon Sep 17 00:00:00 2001 From: kacerr Date: Wed, 23 Sep 2026 21:14:18 +0000 Subject: [PATCH 12/13] proxmox: deployment files, sample, user guide and doc rows The optional proxmox-token Secret (token, ca.pem) mounted at /etc/proxy-operator/proxmox, make proxmox-token-secret with content checks that never print the token, a commented proxmox-home entry in both providers configs, and config/samples/proxy_proxmox.yaml. TestLoadConfig_proxmoxSampleEntry loads the commented entry from both files, and a second test ties the sample Proxy's placement to it. docs/proxmox-provider.md is the user guide: prerequisites and privileges, cluster setup, config reference, placement, what lands on Proxmox, logging, tracing, timeouts, health check, error classes, known gaps and testing. providers-existing.md gains the proxmox column and section; architecture.md, the README, providers.md and CLAUDE.md gain their rows. Co-Authored-By: Claude Opus 5.5 (1M context) --- CLAUDE.md | 3 + Makefile | 17 + README.md | 1 + config/manager/manager.yaml | 17 + config/manager/providers_config.yaml | 28 + config/samples/kustomization.yaml | 1 + config/samples/providers-config.yaml | 28 + config/samples/proxy_proxmox.yaml | 46 ++ docs/architecture.md | 67 +- .../2026-09-18-1502-proxmox-provider.md | 179 ++++- docs/providers-existing.md | 320 +++++++-- docs/providers.md | 3 +- docs/proxmox-provider.md | 646 ++++++++++++++++++ internal/provider/config_proxmox_test.go | 118 ++++ 14 files changed, 1407 insertions(+), 67 deletions(-) create mode 100644 config/samples/proxy_proxmox.yaml create mode 100644 docs/proxmox-provider.md diff --git a/CLAUDE.md b/CLAUDE.md index 1026f73..32263e1 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -226,3 +226,6 @@ A lightweight `check` job (`go vet` / `go build` / `go test -short`) gates the b - `+kubebuilder:rbac` markers are package-level. Keep them in their own comment block, never inside a type's or function's doc comment: controller-gen silently ignores them there. After `make manifests`, check `git diff config/rbac/role.yaml`. +- `make test-integration-proxmox` runs the proxmox provider's integration tests against + a real Proxmox VE host. They skip unless `PROXMOX_TEST_URL`, `_TOKEN_FILE`, `_NODE`, + `_TEMPLATE_VMID` and `_ISO_STORAGE` are set; see docs/testing.md for the full env block. diff --git a/Makefile b/Makefile index d5fd541..356027d 100644 --- a/Makefile +++ b/Makefile @@ -200,6 +200,23 @@ kvm-ssh-secret: ## Create or update the kvm-ssh Secret from KVM_SSH_KEY (private --from-file=known_hosts="$(KVM_KNOWN_HOSTS)" \ --dry-run=client -o yaml | "$(KUBECTL)" apply -f - +PROXMOX_TOKEN_FILE ?= $(HOME)/.config/proxmox-token +PROXMOX_CA_FILE ?= $(HOME)/.config/proxmox-ca.pem +PROXMOX_NAMESPACE ?= egress-proxies-operator-system + +.PHONY: proxmox-token-secret +proxmox-token-secret: ## Create or update the proxmox-token Secret from PROXMOX_TOKEN_FILE (user@realm!tokenid=secret) and PROXMOX_CA_FILE. + @test -f "$(PROXMOX_TOKEN_FILE)" || { echo "PROXMOX_TOKEN_FILE=$(PROXMOX_TOKEN_FILE) not found"; exit 1; } + @test -f "$(PROXMOX_CA_FILE)" || { echo "PROXMOX_CA_FILE=$(PROXMOX_CA_FILE) not found"; exit 1; } + @test "$$(grep -cv '^[[:space:]]*$$' "$(PROXMOX_TOKEN_FILE)")" = 1 && \ + grep -qxE '[[:space:]]*[^@[:space:]]+@[^![:space:]]+![^=[:space:]]+=[^[:space:]]+[[:space:]]*' "$(PROXMOX_TOKEN_FILE)" || \ + { echo "PROXMOX_TOKEN_FILE=$(PROXMOX_TOKEN_FILE) is not one line of user@realm!tokenid=secret"; exit 1; } + @grep -q 'BEGIN CERTIFICATE' "$(PROXMOX_CA_FILE)" || { echo "PROXMOX_CA_FILE=$(PROXMOX_CA_FILE) holds no PEM certificate"; exit 1; } + "$(KUBECTL)" -n $(PROXMOX_NAMESPACE) create secret generic proxmox-token \ + --from-file=token="$(PROXMOX_TOKEN_FILE)" \ + --from-file=ca.pem="$(PROXMOX_CA_FILE)" \ + --dry-run=client -o yaml | "$(KUBECTL)" apply -f - + .PHONY: deploy deploy: manifests kustomize ## Deploy controller to the K8s cluster specified in ~/.kube/config. cd config/manager && "$(KUSTOMIZE)" edit set image controller=${IMG} diff --git a/README.md b/README.md index 87d9d1a..80cbc65 100644 --- a/README.md +++ b/README.md @@ -192,6 +192,7 @@ can be two differently-configured instances of the same `gcp` type (see | `gcp` | A Compute Engine VM with an ephemeral external IP | `project` (required), `network`, `networkTag`, `diskSizeGb` | The real egress fleet. Deliberately uses only four API calls (Insert / Get / Delete / AggregatedList), all fire-and-forget: `Create` returns as soon as the operation is submitted and the reconciler discovers progress by polling `Get`. Auth is Application Default Credentials — workload identity in-cluster, `gcloud` ADC locally; no key-file plumbing. | | `kvm` | A libvirt VM on one of your own hypervisors, reached over SSH | `sshKeyFile`, `knownHostsFile` (unless `insecureIgnoreHostKey`), `machineTypes`, `hosts` (required); `images`, `diskSizeGb`, `vendorData`, `installGuestAgent` | Home-lab or on-prem egress. The base image is downloaded into each host's pool on first use; `Create` waits for the VM to start. See [docs/kvm-provider.md](docs/kvm-provider.md). | | `kvm2` | The same: a libvirt VM on one of your own hypervisors, reached over SSH | the same block shape as `kvm`, under `kvm2:` | The second-generation libvirt provider, built from the provider contract. Adds a child span per libvirt RPC, a pool-space precheck reported as quota (the reconciler waits instead of failing), SSH credentials re-read on every dial, and image downloads and health checks on connections of their own. It does not adopt `kvm`'s VMs. See [docs/kvm2-provider.md](docs/kvm2-provider.md). | +| `proxmox` | A VM on a Proxmox VE cluster: a full clone of a template you prepare, reached through the Proxmox REST API with an API token | `url`, `tokenFile`, `nodes`, `isoStorage`, `templates`, `machineTypes` (required); `caFile` or `insecureSkipTLSVerify`, `storage`, `bridge`, `vlanTag`, `vmidRange`, `requestTimeoutSeconds`, `vendorData`, `installGuestAgent` | Home-lab or on-prem egress without SSH. Cloud-init goes on a seed ISO uploaded through the API; the address comes from the QEMU guest agent, so the proxy network needs a route to a package mirror unless the template ships the agent. `Create` waits for the VM to run. The token is read on every request, from the optional `proxmox-token` Secret (`make proxmox-token-secret`). Built from the provider contract; not yet run against a real host end to end. See [docs/proxmox-provider.md](docs/proxmox-provider.md). | ### Adding a provider diff --git a/config/manager/manager.yaml b/config/manager/manager.yaml index c334b6d..3abe901 100644 --- a/config/manager/manager.yaml +++ b/config/manager/manager.yaml @@ -166,6 +166,18 @@ spec: - name: kvm-ssh mountPath: /etc/proxy-operator/kvm readOnly: true + # API token and CA for the proxmox provider (tokenFile / caFile in + # the providers config point here). Optional: the mount is empty when + # the Secret does not exist, and only proxmox entries need it. The + # token file is read on every request; the CA once, on first use. + # Create it with: + # kubectl -n egress-proxies-operator-system create secret generic proxmox-token \ + # --from-file=token= \ + # --from-file=ca.pem= + # or `make proxmox-token-secret`. + - name: proxmox-token + mountPath: /etc/proxy-operator/proxmox + readOnly: true # Scratch space: the root filesystem is read-only, and the kvm # provider builds each VM's cloud-init seed ISO through a temp # directory (a few hundred KB, deleted right after upload). @@ -180,6 +192,11 @@ spec: secret: secretName: kvm-ssh optional: true + # The proxmox provider's token and CA; see the mount above. + - name: proxmox-token + secret: + secretName: proxmox-token + optional: true - name: tmp emptyDir: sizeLimit: 64Mi diff --git a/config/manager/providers_config.yaml b/config/manager/providers_config.yaml index b3ae1c5..95beb55 100644 --- a/config/manager/providers_config.yaml +++ b/config/manager/providers_config.yaml @@ -84,4 +84,32 @@ data: bridge: br0 # socket: /var/run/libvirt/libvirt-sock # pool: default + # proxmox clones a prepared template on a Proxmox VE cluster through its + # REST API and seeds it with a NoCloud ISO uploaded to isoStorage; it needs + # no SSH. Both files come from the optional proxmox-token Secret mounted by + # config/manager/manager.yaml. The url's host must be a name or address in + # the node certificate. See docs/proxmox-provider.md. + # - name: proxmox-home + # type: proxmox + # proxmox: + # url: https://192.168.0.8:8006 + # tokenFile: /etc/proxy-operator/proxmox/token + # caFile: /etc/proxy-operator/proxmox/ca.pem + # # insecureSkipTLSVerify: true # instead of caFile; logged as a warning + # nodes: [pv1-1] # placement.zone names a node; optional with one node + # isoStorage: proxy-seeds # iso content; the token needs Datastore.Allocate here + # # storage: local-lvm # clone target; default is the template's disk storage + # # bridge: vmbr0 # replaces the cloned net0's bridge + # vlanTag: 4 # set on the cloned net0, MAC kept + # # vmidRange: {min: 100000, max: 199999} + # # requestTimeoutSeconds: 30 + # # installGuestAgent: false + # # vendorData: | + # # #cloud-config + # # ... + # templates: # placement.image names a key; add keys, never repoint one + # debian-13: {vmid: 9001} + # machineTypes: # placement.machineType names a key here + # small: {cores: 1, memoryMiB: 1024} + # medium: {cores: 2, memoryMiB: 2048, diskGiB: 10} diff --git a/config/samples/kustomization.yaml b/config/samples/kustomization.yaml index 518981b..37bab9d 100644 --- a/config/samples/kustomization.yaml +++ b/config/samples/kustomization.yaml @@ -9,5 +9,6 @@ resources: - proxy_gcp.yaml - proxy_kvm.yaml - proxy_kvm2.yaml +- proxy_proxmox.yaml - proxy_external.yaml # +kubebuilder:scaffold:manifestskustomizesamples diff --git a/config/samples/providers-config.yaml b/config/samples/providers-config.yaml index 4dc272d..0ea9918 100644 --- a/config/samples/providers-config.yaml +++ b/config/samples/providers-config.yaml @@ -88,3 +88,31 @@ providers: # bridge: br0 # # socket: /var/run/libvirt/libvirt-sock # # pool: default + # proxmox clones a prepared template on a Proxmox VE cluster through its + # REST API and seeds it with a NoCloud ISO uploaded to isoStorage; it needs + # no SSH. Both files come from the optional proxmox-token Secret mounted by + # config/manager/manager.yaml. The url's host must be a name or address in + # the node certificate. See docs/proxmox-provider.md. + # - name: proxmox-home + # type: proxmox + # proxmox: + # url: https://192.168.0.8:8006 + # tokenFile: /etc/proxy-operator/proxmox/token + # caFile: /etc/proxy-operator/proxmox/ca.pem + # # insecureSkipTLSVerify: true # instead of caFile; logged as a warning + # nodes: [pv1-1] # placement.zone names a node; optional with one node + # isoStorage: proxy-seeds # iso content; the token needs Datastore.Allocate here + # # storage: local-lvm # clone target; default is the template's disk storage + # # bridge: vmbr0 # replaces the cloned net0's bridge + # vlanTag: 4 # set on the cloned net0, MAC kept + # # vmidRange: {min: 100000, max: 199999} + # # requestTimeoutSeconds: 30 + # # installGuestAgent: false + # # vendorData: | + # # #cloud-config + # # ... + # templates: # placement.image names a key; add keys, never repoint one + # debian-13: {vmid: 9001} + # machineTypes: # placement.machineType names a key here + # small: {cores: 1, memoryMiB: 1024} + # medium: {cores: 2, memoryMiB: 2048, diskGiB: 10} diff --git a/config/samples/proxy_proxmox.yaml b/config/samples/proxy_proxmox.yaml new file mode 100644 index 0000000..7e9b361 --- /dev/null +++ b/config/samples/proxy_proxmox.yaml @@ -0,0 +1,46 @@ +# A Managed proxy backed by a Proxmox VE cluster through the proxmox +# provider: the operator full-clones a prepared template on the named node, +# attaches a NoCloud seed ISO carrying this cloud-init, and starts the VM on +# the configured network (here VLAN 4 on vmbr0, IP from that network's +# DHCP), where cloud-init installs Squid. Requires a providers-config entry +# named "proxmox-home" (see providers-config.yaml) and the proxmox-token +# Secret (docs/proxmox-provider.md). +# +# placement.zone is a node name from proxmox.nodes (optional when only one +# node is configured); machineType and image are keys in the provider's +# machineTypes and templates maps. The address comes from the guest agent, +# which a vendor-data script installs at first boot, so the VM's network +# needs a route to a package mirror. +apiVersion: crawl.example.com/v1alpha1 +kind: Proxy +metadata: + name: proxy-proxmox-sample +spec: + mode: Managed + provider: proxmox-home + placement: + zone: pv1-1 + machineType: small + image: debian-13 + port: 3128 + cloudInit: + inline: | + #cloud-config + packages: + - squid + write_files: + - path: /etc/squid/conf.d/proxy-operator.conf + content: | + http_port 3128 + http_access allow all + via off + forwarded_for off + runcmd: + - systemctl restart squid + # Optional: log in with `ssh debian@` (the Debian image's default + # user) to look around; `qm terminal ` on the node works too. + # ssh_authorized_keys: + # - ssh-ed25519 AAAA... you@laptop + attributes: + geo: home + purpose: crawl diff --git a/docs/architecture.md b/docs/architecture.md index 03d458f..b8f3ece 100644 --- a/docs/architecture.md +++ b/docs/architecture.md @@ -11,6 +11,7 @@ | gcp provider | `internal/provider/gcp` | library | — | Compute Engine VMs, four API calls, fire-and-forget ops | | kvm provider | `internal/provider/kvm` | library | — | VMs on libvirt hosts over SSH-forwarded sockets; image cache, overlay disks, NoCloud seeds | | kvm2 provider | `internal/provider/kvm2` | library | — | the same backend, built clean-room from the provider contract as a second type: per-RPC child spans, pool-space quota, per-call budgets, its own metadata namespace, credentials re-read on every dial | +| proxmox provider | `internal/provider/proxmox` | library | — | VMs on a Proxmox VE cluster through its REST API: full clones of a template, a NoCloud seed uploaded through the API, the VMID derived from the Proxy UID, every task waited on, HTTP child spans | | Shared provider helpers | `internal/provider/nocloud`, `internal/provider/keyedlock` | library | — | the NoCloud seed ISO and vendor-data builder, and one-slot semaphores by key; used by kvm2 and proxmox | | Health engine | `internal/health` | Runnable | yes | through-the-proxy probes, thresholds, transition events | | Lease store | `internal/lease` | Runnable (expiry sweep) | no | in-memory leases + cooldowns, single mutex | @@ -204,6 +205,37 @@ outlives its ctx or budget kills the pooled connection (the only way to unblock go-libvirt), so other calls in flight on that host fail once with Transient and retry; the next call redials. Each RPC is a child span kvm2. under provider.*, plus kvm2.dial and kvm2.download. + +proxmox provider (internal/provider/proxmox/) - the Proxmox VE REST API, one URL per cluster +prov.Create ──► placement checked against the catalogue (Permanent before any request) + GET /cluster/resources: a VM with the name → ownership check ─┐ every step + else: template config checked (not a template, cloud-init drive, │ idempotent, + no net0 or boot disk it needs = Permanent) │ under a + VMID = hash(UID) in vmidRange, probing on; a taken VMID → │ per-name lock + ownership check: wait out a lock, adopt our name + mark + │ shared with + UID, probe past another name, Permanent otherwise │ Delete; each + POST .../clone (full, description = the mark) → wait task │ task's UPID + upload seed ISO to isoStorage (overwrites) → wait task │ polled every + PUT .../config: cores, memory, agent=1, tag, ide2 = seed, │ 1s to its end + net0 bridge/tag when configured │ + resize if diskGiB is larger → wait task │ + start → wait task → poll status/current until running (60s) ─┘ +prov.Get ──► /cluster/resources (absent = NotFound; lock or node unknown = Transient) + + config (name and mark must match, else NotFound: VMID reused) + + status/current (paused and hibernated = Stopped) + + agent network-get-interfaces (first routable IPv4; agent not + up yet = Provisioning) +prov.Delete ──► resources + config: gone, not ours or unmarked = leave it; + hibernated = Permanent; stop → destroy (purge) → delete the seed + on the VM's node (or on every node when no VM of ours was left) +prov.ListByTag ─► one /cluster/resources for the whole cluster, one config read per + candidate (tag or proxy- name); the description's mark decides +Check ──► per node: /version, node status, isoStorage status, templates in + /cluster/resources +providerID = vms//: no node, because VMs migrate; the node is looked +up in /cluster/resources on every call. Token read from its file on every +request. Each request is an HTTP child span from the parent-gated +transport wrapper; a cancelled request ends only itself. ``` The reconciler never watches provider-side resources (Pods or GCP VMs). @@ -388,6 +420,12 @@ incoming trace context): from a span decorator over the seam, applied once in the dialer, so no RPC is unspanned. The `libvirt call` log line of an RPC is written inside its span. The `kvm` provider has no child spans. +- proxmox child spans: one `HTTP ` client span per API request, + from `tracing.RestConfigWrapper()` on the provider's own HTTP transport, + under `provider.*` or `providerhealth.check`. Parent-gated like the + Kubernetes API spans, so a request without a parent starts none. The + provider's logging decorator starts no span; its `proxmox api call` line + carries the operation span's IDs. - Kubernetes API calls become client child spans via a wrapped `rest.Config` transport — gated on an existing parent span, so informer list/watch long-polls and leader-election renewals never create root @@ -413,12 +451,13 @@ the slog handler's ctx. Answers "can this cluster reach each provider's backend right now?" without waiting for a Proxy to fail. A provider opts in by implementing `provider.Checker`: `Check(ctx) []TargetStatus` returns one verdict per -target and never fails as a whole. All four providers do: +target and never fails as a whole. All five providers do: | Provider | Target | What one check does | |---|---|---| | kvm | each `kvm.hosts` entry | over the pooled SSH+libvirt connection: libvirt version and hostname, then the configured pool must exist and be running; reports pool capacity and free space ([details](kvm-provider.md#health-check)) | | kvm2 | each `kvm2.hosts` entry | on a fresh SSH+libvirt connection dialled for the check and closed after it: libvirt version and hostname, then the configured pool must exist and be running; `details` carry `user`, `pool`, `hostname`, `libvirtVersion`, `poolState`, `poolCapacityGiB` and `poolAvailableGiB` ([details](kvm2-provider.md#health-check)) | +| proxmox | each `proxmox.nodes` entry | four requests on the shared HTTP client: API version, node status, the `isoStorage` status (it must be active and hold `iso` content), and the cluster's resources (every configured template must exist and be a template); `details` carry `isoStorage`, `tokenID`, `pveVersion`, `kernel`, `isoStorageAvailGiB` and `templates` ([details](proxmox-provider.md#health-check)) | | kubernetes | `apiserver` | lists at most one managed Pod cluster-wide: the verb and scope `ListByTag` and GC rely on | | gcp | the project | one page of an aggregated instance list with `maxResults=1`: credentials, Compute API enabled, `compute.instances.list` | @@ -704,3 +743,29 @@ they read as choices, not accidents. Chronological by build step. the Secret needs no restart; nothing is cached. The constructor only checks that the files exist, to warn early without failing the provider set. +- **proxmox derives the VMID from the Proxy UID and checks ownership + before using a taken one.** Proxmox addresses VMs by VMID, and names are + not unique, so the name cannot be the key. A hash of the UID placed in + `vmidRange` makes a retry land on the same VMID; a taken one is waited + on while locked, then adopted when it has our name, mark and UID, + probed past when it has another name, and refused otherwise. The wait + is what stops a retry after a crash mid-clone from cloning a duplicate: + a clone in progress has no name yet, so the by-name lookup misses it. +- **proxmox writes its mark into the VM's description**, in the clone + call itself, so no VM of ours exists unmarked. Tags cannot be passed to + the clone, and the clone's own creation time is copied from the + template. The `proxy_operator` tag is set afterwards for the Proxmox UI + and to narrow GC's candidates; the mark decides. +- **proxmox delivers cloud-init as a seed ISO uploaded through the API**, + not as Proxmox's own cloud-init drive or a `cicustom` snippet. The + upload endpoint does not accept snippets, and the operator has no shell + on the nodes; the seed code is kvm2's, extracted into + `internal/provider/nocloud`. Templates must therefore carry no Proxmox + cloud-init drive, and seeds of VMs destroyed outside the operator are a + documented GC blind spot. +- **proxmox resolves a VM's node on every call** from + `/cluster/resources`, and its providerID names none, because VMs migrate + between nodes. The cost is one extra request per operation; the gain is + that a migrated VM stays addressable and the listing, which includes the + guests of an unreachable node as `status: unknown`, never mistakes a + down node for a deleted VM. diff --git a/docs/plans-executions/2026-09-18-1502-proxmox-provider.md b/docs/plans-executions/2026-09-18-1502-proxmox-provider.md index 21e77f4..aca7c8d 100644 --- a/docs/plans-executions/2026-09-18-1502-proxmox-provider.md +++ b/docs/plans-executions/2026-09-18-1502-proxmox-provider.md @@ -13,7 +13,7 @@ Pairs with [docs/plans/2026-09-18-1502-proxmox-provider.md](../plans/2026-09-18- - [x] Phase 6 — Create - [x] Phase 7 — Delete, Checker, CRD comment - [x] Phase 8 — Contract and observability sweep, integration test, testing.md -- [ ] Phase 9 — Deployment and documentation +- [x] Phase 9 — Deployment and documentation - [ ] Phase 10 — Review, real-host validation, changelog, MR ## Phase 0 — Plan file, branch, host recon, fixtures @@ -1121,3 +1121,180 @@ Worth noting: `go test -short ./...` run of this phase. - `.claude/settings.json` stays out of the commit. Nothing in this phase talked to the host. + +## Phase 9 — Deployment and documentation + +Deployment: the optional `proxmox-token` Secret (keys `token` and `ca.pem`) +is mounted read-only at `/etc/proxy-operator/proxmox` in +`config/manager/manager.yaml`, next to `kvm-ssh` and commented the same way. +The Makefile gains `proxmox-token-secret`. Both providers configs gain a +commented `proxmox-home` entry after kvm2's. `config/samples/proxy_proxmox.yaml` +is listed in the samples kustomization. There is no RBAC change and no +`hack/providers-dev.yaml` entry. Docs: `docs/proxmox-provider.md` follows +kvm2's guide heading for heading and has no `—`. The rows and sections +named below went into `providers-existing.md`, `architecture.md`, the +README, `providers.md` and `CLAUDE.md`. `docs/testing.md` and the +integration Makefile target were done in Phase 8 and were not touched; the +guide's Testing section links to them. + +Where the spec and the plan's Phase 9 text disagree, the spec won. The URL +is `https://192.168.0.8:8006`, not 192.168.2.8, which is not in the node +certificate. The Secret has two keys, and the target takes +`PROXMOX_CA_FILE` as well as `PROXMOX_TOKEN_FILE`. + +Judgment calls: + +- **The target also checks both files' content.** The token file must be + exactly one non-blank line matching + `@!=`. That is the shape `readToken` + accepts: `TrimSpace`, then an anchored regexp. The check uses `grep -c` + and `grep -qx`, so the secret is never printed. The CA check mirrors + `kvm-ssh-secret`'s `PRIVATE KEY` check: the file must contain + `BEGIN CERTIFICATE`. With `insecureSkipTLSVerify` the guide says to + create the Secret by hand with the `token` key only. +- **The sample-entry test covers both files.** `commentedEntry` finds the + `# - name: proxmox-home` line and takes its indent. It keeps the lines + that continue at `# ` and strips `# ` at that indent. The same + helper then works for the ConfigMap, whose entry sits six spaces deep + inside a `|` block. So `TestLoadConfig_proxmoxSampleEntry` has a + `sample` subtest and a `configmap` subtest. It checks every field the + spec names, and that the optional fields stay commented out. +- **A second test goes beyond the plan:** + `TestLoadConfig_proxmoxSampleProxyMatchesSampleEntry`. The sample Proxy's + provider, zone, machine type and image must resolve in the sample + entry. kvm has exactly this bug as a known gap (`proxy_kvm.yaml` places + on `nas1`). +- **The sample entry also has `medium: {cores: 2, memoryMiB: 2048, + diskGiB: 10}`**, next to the `small` the spec asks for, so `diskGiB` + appears somewhere. The optional fields are commented out with a short + note each, because no live proxmox entry exists to explain them. The + kvm2 entry can rely on kvm's. +- **In the ConfigMap the proxmox entry is commented out**, although kvm2's + is live there. It needs a host and a token. +- **providers-existing.md needed more than the column and the section.** + The intro now says five providers and links the guide. In the shared + gaps, six sentences would otherwise be false, and each now names + proxmox where it applies: adoption verified in two of the five, Create + blocking in all three VM providers, no Terminated, the list policy, the + collector gap, and the "kvm2 does this" in proposed features. The new + section is §8, so the shared gaps and proposed features move to §9 and + §10. No doc links those sections by anchor (grep). +- **architecture.md §11 now says "All five providers do".** The sentence + counts Checkers, and proxmox has one. The four Decisions bullets are + entry 23's: the VMID from the UID with the ownership check, the mark in + the description, the seed uploaded through the API, and the node + resolved per call. +- **The guide lists the ten design limits first, then four more.** They + are the hibernated-VM latch, the one-request seed upload, the open-proxy + sample (kvm2's guide lists it too), and "not yet proven end to end" with + the three answers never captured. The Phase 6 to 8 guide items are + each in the section where a reader looks for them. The config write + replacing `agent` and `tags` is in *What lands on Proxmox*, and so are + the seed cleanup, `qm unlock` and the 403 that makes Delete Permanent. + The certificate rule is in *Cluster-side setup*. CA caching versus + token rotation is in both. +- **The guide resumes a hibernated VM with `qm start`, not + `qm resume`.** A VM suspended to disk resumes from its state file on + its next start; `qm resume` is for a VM paused in RAM. +- **The guide's `pvesm add dir proxy-seeds --path /var/lib/proxy-seeds + --content iso` is the standard form, not a recorded command.** Phase 0 + says only that the user created `proxy-seeds` as a `dir` storage with + ISO content; the path is illustrative. + +The token-file check, run without touching a cluster: `KUBECTL=echo` +prints only the apply stage, and `KUBECTL=true` runs nothing. Scratch +files covered one good line, two lines, a bare secret, blank lines around +a good line, a non-PEM CA and a missing file. The real files on this +workstation pass: + +```bash +make -n proxmox-token-secret +make -s proxmox-token-secret PROXMOX_TOKEN_FILE=$D/ PROXMOX_CA_FILE=$D/ca KUBECTL=echo +# good, ws: "apply -f -"; two, bad: "... is not one line of user@realm!tokenid=secret"; noca: "... holds no PEM certificate" +make -s proxmox-token-secret KUBECTL=true && echo "real files pass" # real files pass +``` + +The mutation check for the sample tests edited one line per run, +asserting a single match. It ran the two tests, restored the file from a +copy, and at the end `sha256sum -c` confirmed the three files. All five +mutations failed the tests: `vlanTag: 5000` and `isoStorag:` in the +sample, `192.168.2.8` and `type: kvm2` in the ConfigMap, and +`image: debian-12` in the sample Proxy. `type: kvm2` was rerun after lint +removed the test's own type comparison. It still fails, because +`LoadConfig` refuses a proxmox block under another type. + +The link sweep is a short script. It takes the docs as arguments, so it +can be rerun after any later edit. It skips fenced code and inline code +spans. For each relative link it checks that the file exists, and that +the `#anchor` matches a GitHub-style slug of a heading in the target file +(or the same file). Before the real run it was checked on a scratch copy +of the guide with five links added. It reported exactly the three bad +ones: a missing file, a missing anchor in `testing.md`, and a missing +same-file anchor. It checked 78 relative links in the six edited docs, +plus this log's, and found 0 broken: + +```bash +cat > doclinks.py <<'PY' +import os, re, sys +def slug(h, seen): + s = re.sub(r"[^\w\- ]", "", h.strip().lower()).replace(" ", "-") + n = seen.get(s, 0); seen[s] = n + 1 + return s if n == 0 else f"{s}-{n}" +def prose(path): + out, fence = [], False + for line in open(path, encoding="utf-8"): + if line.lstrip().startswith("```"): fence = not fence; continue + if not fence: out.append(line) + return out +def anchors(path): + seen = {} + return {slug(re.sub(r"`", "", l.lstrip("#").strip()), seen) for l in prose(path) if re.match(r"#{1,6} ", l)} +bad = 0 +for doc in sys.argv[1:]: + for n, line in enumerate(prose(doc), 1): + for target in re.findall(r"\]\(([^)\s]+)\)", re.sub(r"`[^`]*`", "", line)): + if re.match(r"[a-z]+:", target): continue + f, _, a = target.partition("#") + p = os.path.normpath(os.path.join(os.path.dirname(doc), f)) if f else doc + if not os.path.exists(p): print(f"{doc}: missing file {target}"); bad += 1 + elif a and p.endswith(".md") and a not in anchors(p): print(f"{doc}: missing anchor {target}"); bad += 1 +print(f"{bad} broken"); sys.exit(1 if bad else 0) +PY +python3 doclinks.py docs/proxmox-provider.md docs/providers-existing.md docs/architecture.md \ + README.md docs/providers.md CLAUDE.md docs/plans-executions/2026-09-18-1502-proxmox-provider.md # 0 broken +``` + +Verification: + +```bash +bin/kustomize build config/default >/dev/null && bin/kustomize build config/samples >/dev/null # ok +go test -count=1 -run 'TestLoadConfig_proxmoxSample' -v ./internal/provider/ # 3 PASS +grep -c '—' docs/proxmox-provider.md # 0 +go build ./... && go vet ./... && go vet -tags=integration ./... && go test -short -count=1 ./... # all ok +bin/golangci-lint run --disable staticcheck --max-same-issues 0 --max-issues-per-linter 0 ./internal/provider/ +git status --short config/ # the four edited files named above, plus proxy_proxmox.yaml +``` + +Worth noting: + +- Lint on `internal/provider` is at its baseline of three goconst hits. + The first version of the new test pushed it to five: its + `p.Type != "proxmox"` made `"proxmox"` a three-time literal, flagged in + both `config.go` and the test. That comparison added nothing, so it + went. `internal/provider/proxmox` was not touched. +- The `tmp` volume comment in `manager.yaml` still says only the kvm + provider stages seeds there. kvm2 and proxmox do too. It was already + stale for kvm2, so it was left out of this phase's diff. +- The architecture §5 block's right-hand bracket was aligned by + character, not byte (`→` is three bytes), so every line puts `│` in + column 83. +- Nothing in this phase talked to the Proxmox host or a cluster. + `.claude/settings.json` stays out of the commit. The parked kvm2 items + (S5.1, and the 200 ms `TestCheck_hostsCheckedConcurrently`) did not + bite: `go test -short ./...` passed. +- Next is Phase 10: the first real run of `TestProxmox_lifecycle` and + `TestProxmoxCheck_realHost` on pv1-1, with the env block in + `docs/testing.md`. The three answers never captured go with it, and the + guide's "Not yet proven end to end" gap and the README's matching + clause come out once they pass. Then the CHANGELOG entry after the user + confirms, and the MR. diff --git a/docs/providers-existing.md b/docs/providers-existing.md index 80cb104..4ea4851 100644 --- a/docs/providers-existing.md +++ b/docs/providers-existing.md @@ -1,17 +1,20 @@ # The existing providers This is the appendix to [provider-contract.md](provider-contract.md): how the -four providers that exist today (kubernetes, gcp, kvm, kvm2) meet the -contract, which profile each one chose on every axis, and what is still +five providers that exist today (kubernetes, gcp, kvm, kvm2, proxmox) meet +the contract, which profile each one chose on every axis, and what is still missing. It is reference material for whoever writes the next provider, not a user guide; the user guides are [kvm-provider.md](kvm-provider.md), [kvm2-provider.md](kvm2-provider.md), +[proxmox-provider.md](proxmox-provider.md), [gcp-vm-validation.md](gcp-vm-validation.md) and [gcp-in-specific-project.md](gcp-in-specific-project.md). The README's -"Implemented providers" table stays the short public summary. kvm2 is the -one provider built from the contract itself (clean-room, from -[kvm2-provider-spec.md](kvm2-provider-spec.md), without reading kvm's code), -so its column is also a record of what the contract produced. +"Implemented providers" table stays the short public summary. kvm2 and +proxmox are the two providers built from the contract itself (kvm2 +clean-room from [kvm2-provider-spec.md](kvm2-provider-spec.md) without +reading kvm's code, proxmox from +[proxmox-provider-spec.md](proxmox-provider-spec.md)), so their columns are +also a record of what the contract produced. The per-provider call flows are drawn in [architecture.md §5](architecture.md#5-what-provider-calls-do-in-the-outside-world), @@ -27,59 +30,59 @@ Code references are a file path plus a function or symbol name. One row per axis of contract §4, in the same order, with the profile name and the symbol that shows it. -| Axis | kubernetes | gcp | kvm | kvm2 | -|---|---|---|---|---| -| `readiness` | immediate: a Pod is scheduled within seconds (`stateFromPod`) | provisioning-window: minutes of boot (`mapState`) | provisioning-window: boot plus, on first use, an image download (`imageCache.ensure`) | provisioning-window: boot plus, on first use, an image download (`ensureImage`) | -| `create-return` | submit: returns after `client.Create` | submit: `Insert` is never awaited, `Operation.Wait` is not called | **blocking**: waits through `DomainCreate` (`ensureRunning`), and the first image fetch runs inside (`imageCache.fetch`) | **blocking**: waits through `DomainCreate` and a 15-second poll for running (`ensureRunning`, `waitRunning`), and the first image fetch runs inside (`fetchImage`) | -| `delete-return` | returns-early: `client.Delete`, Pod terminates on its own | returns-early: `Delete` submits an operation each time | waits: destroy, undefine and volume removal complete before return (`removeDomain`, `removeVolumes`) | waits: destroy, undefine and volume removal complete before return (`removeDomain`, `removeVolumes`) | -| `ip-source` | assigned-at-create: `PodIP` (`instanceFromPod`) | assigned-at-create: the external NAT IP of the first NIC (`toInstance`) | discovered: ARP table first, then the guest agent, first routable IPv4 (`address`, `firstIPv4`) | discovered: the guest agent first, then the ARP table, first routable IPv4 (`addressSources`, `address`, `firstIPv4`) | -| `gc-mark` | native-labels: Pod labels (`buildPod`) | native-labels: instance labels (`buildInsertRequest`) | written-metadata: a namespaced `` element written in `DomainDefineXML` (`metadata.go`) | written-metadata: an element in kvm2's own namespace written in `DomainDefineXML` (`metadataURI`, `newInstanceMetadata`) | -| `created-at` | backend-timestamp: `CreationTimestamp` | backend-timestamp: `CreationTimestamp`, parse error gives zero | written-at-create: `createdAt` in the metadata element (`newInstanceMetadata`) | written-at-create: `createdAt` in the metadata element, parse error gives zero (`newInstanceMetadata`, `createdAt`) | -| `list-partial` | n/a: one call, all namespaces | omit-unreachable: `ReturnPartialSuccess` on `AggregatedList` | fail-whole per host, one unreadable domain skipped (`ListByTag`, `managedInstance`) | fail-whole per host, one unreadable domain skipped unless ctx ended; `ID`, `UID` and `CreatedAt` from the name and the mark only, no state or address RPC (`ListByTag`) | -| `quota` | absent: a quota 403 is Permanent (`classify`) | detected: 429, and 403 with reason `quotaExceeded` or `rateLimitExceeded` (`classify`, `hasReason`) | absent (`classify`) | detected: a pool-space precheck before any volume is created (`checkPoolSpace`, `neededBytes`) | -| `not-found` | status-404: `apierrors.IsNotFound` | status-404: `isNotFound` | error-code: `ErrNoDomain`, `ErrNoStorageVol` (`hasCode`) | error-code: `ErrNoDomain`, `ErrNoStorageVol`, Transient inside Create (`classifyLibvirt`) | -| `primary-key` | name-is-key: `namespace/name` | name-is-key: `zones//instances/` | name-is-key: `hosts//domains/` | name-is-key: `hosts//domains/`, with a UID guard on adoption (`create`) | -| `provisioning-state` | native: `PodPending` | native: `PROVISIONING`, `STAGING`, `REPAIRING` | create-blocks: a domain is defined and started in one `Create`; `nostate` maps to Provisioning | create-blocks: defined, started and polled to running in one `Create`; `nostate` maps to Provisioning (`mapState`) | -| `async-ops` | none | task-handle, but the handle is discarded: an insert that fails after acceptance is never read | none | none | -| `secondary-resources` | none | cascade: `AutoDelete` on the boot disk (`buildInsertRequest`) | explicit: overlay and seed removed by name (`removeVolumes`); base image kept | explicit: overlay and seed removed by name (`removeVolumes`); base image kept | -| `credentials` | workload-identity: in-cluster config or kubeconfig (`NewWithTransportWrapper`) | environment: Application Default Credentials | mounted-files: SSH key and known_hosts from the optional `kvm-ssh` Secret, read lazily (`newLazySSH`) | mounted-files: the same `kvm-ssh` Secret, read on every dial and never cached (`loadCredentials`) | -| `client-kind` | vendor-sdk: controller-runtime `client.Client`, no seam interface | vendor-sdk: `cloud.google.com/go/compute` behind `instancesAPI` | rpc-over-ssh: `go-libvirt` over an SSH-forwarded unix socket, behind `libvirtAPI` (`conn.go`) | rpc-over-ssh: `go-libvirt` over an SSH-forwarded unix socket, behind the ctx-first `backend` seam (`backend.go`) | -| `timeouts` | ctx-only: no `rest.Config.Timeout` is set | ctx-only: SDK defaults, with SDK retries on Get and List | per-call for the dial only (`defaultHandshakeTimeout` in `dialSSH`); RPCs are ctx-only and a cancelled `call` closes the connection | per-call: a budget around the dial, every RPC, the uploads and the download (`defaultBudgets`, `call`); a cancelled or overlong call kills the pooled connection | -| `logging` | none | tiered: V(1) outcomes and `logAPIError`, V(2) detail, V(5) wire log (`wirelog.go`) | tiered: V(1) outcomes and one `libvirt call` line per RPC (`trace.go`), V(2) XML and addresses; no wire tier | tiered: V(1) outcomes and one `libvirt call` line per RPC (`observe.go`), V(2) XML and addresses; no wire tier | -| `child-spans` | transport-wrapper: `tracing.RestConfigWrapper()` on its own `rest.Config`, parent-gated | sdk-instrumentation: the SDK adds `otelhttp` itself; not parent-gated, but every call site has a parent | **none**: `trace.go` is a log decorator, not spans | span-decorator: `kvm2.` per RPC (`observe`), plus `kvm2.dial` (`libvirtDialer`) and `kvm2.download` (`fetchImage`) | -| `backend-test` | cluster-e2e: kind (`test/e2e`) | manual-runbook: [gcp-vm-validation.md](gcp-vm-validation.md) | integration-tag: `kvm_integration_test.go`, `make test-integration-kvm` | integration-tag: `kvm2_integration_test.go`, `make test-integration-kvm2` | +| Axis | kubernetes | gcp | kvm | kvm2 | proxmox | +|---|---|---|---|---|---| +| `readiness` | immediate: a Pod is scheduled within seconds (`stateFromPod`) | provisioning-window: minutes of boot (`mapState`) | provisioning-window: boot plus, on first use, an image download (`imageCache.ensure`) | provisioning-window: boot plus, on first use, an image download (`ensureImage`) | provisioning-window: boot plus the guest agent's install at first boot, about 50 s to an address on the test host (`agentAddress`) | +| `create-return` | submit: returns after `client.Create` | submit: `Insert` is never awaited, `Operation.Wait` is not called | **blocking**: waits through `DomainCreate` (`ensureRunning`), and the first image fetch runs inside (`imageCache.fetch`) | **blocking**: waits through `DomainCreate` and a 15-second poll for running (`ensureRunning`, `waitRunning`), and the first image fetch runs inside (`fetchImage`) | **blocking**: waits through the clone, upload, resize and start tasks and a 60-second poll for running (`provision`, `waitRunning`) | +| `delete-return` | returns-early: `client.Delete`, Pod terminates on its own | returns-early: `Delete` submits an operation each time | waits: destroy, undefine and volume removal complete before return (`removeDomain`, `removeVolumes`) | waits: destroy, undefine and volume removal complete before return (`removeDomain`, `removeVolumes`) | waits: the stop, destroy and seed delete tasks complete before return (`destroy`, `deleteSeed`) | +| `ip-source` | assigned-at-create: `PodIP` (`instanceFromPod`) | assigned-at-create: the external NAT IP of the first NIC (`toInstance`) | discovered: ARP table first, then the guest agent, first routable IPv4 (`address`, `firstIPv4`) | discovered: the guest agent first, then the ARP table, first routable IPv4 (`addressSources`, `address`, `firstIPv4`) | discovered: the guest agent only, first routable IPv4 in the agent's order (`agentAddress`, `firstIPv4`) | +| `gc-mark` | native-labels: Pod labels (`buildPod`) | native-labels: instance labels (`buildInsertRequest`) | written-metadata: a namespaced `` element written in `DomainDefineXML` (`metadata.go`) | written-metadata: an element in kvm2's own namespace written in `DomainDefineXML` (`metadataURI`, `newInstanceMetadata`) | written-metadata: three `key=value` lines in the VM's description, passed in the clone call itself (`formatDescription`, `parseDescription`) | +| `created-at` | backend-timestamp: `CreationTimestamp` | backend-timestamp: `CreationTimestamp`, parse error gives zero | written-at-create: `createdAt` in the metadata element (`newInstanceMetadata`) | written-at-create: `createdAt` in the metadata element, parse error gives zero (`newInstanceMetadata`, `createdAt`) | written-at-create: `proxy-operator-created-at` in the description, parse error gives zero (`parseDescription`) | +| `list-partial` | n/a: one call, all namespaces | omit-unreachable: `ReturnPartialSuccess` on `AggregatedList` | fail-whole per host, one unreadable domain skipped (`ListByTag`, `managedInstance`) | fail-whole per host, one unreadable domain skipped unless ctx ended; `ID`, `UID` and `CreatedAt` from the name and the mark only, no state or address RPC (`ListByTag`) | fail-whole for the one cluster-wide `/cluster/resources` call, one unreadable VM skipped unless ctx ended; `ID`, `UID` and `CreatedAt` from the description only, no status or agent call (`ListByTag`) | +| `quota` | absent: a quota 403 is Permanent (`classify`) | detected: 429, and 403 with reason `quotaExceeded` or `rateLimitExceeded` (`classify`, `hasReason`) | absent (`classify`) | detected: a pool-space precheck before any volume is created (`checkPoolSpace`, `neededBytes`) | absent: thin storage overprovisions, so no out-of-space answer exists to match; free space goes to the Checker's details (`classify`) | +| `not-found` | status-404: `apierrors.IsNotFound` | status-404: `isNotFound` | error-code: `ErrNoDomain`, `ErrNoStorageVol` (`hasCode`) | error-code: `ErrNoDomain`, `ErrNoStorageVol`, Transient inside Create (`classifyLibvirt`) | error-code: the 500 `Configuration file '…' does not exist`, plus list-and-miss on `/cluster/resources`; Transient inside Create (`isNotExist`, `classifyAPI`) | +| `primary-key` | name-is-key: `namespace/name` | name-is-key: `zones//instances/` | name-is-key: `hosts//domains/` | name-is-key: `hosts//domains/`, with a UID guard on adoption (`create`) | **allocated-id**: `vms//`, the VMID from an FNV-1a hash of the UID with forward probing, each taken VMID put through an ownership check with a UID guard (`vmidFor`, `allocate`, `inspect`) | +| `provisioning-state` | native: `PodPending` | native: `PROVISIONING`, `STAGING`, `REPAIRING` | create-blocks: a domain is defined and started in one `Create`; `nostate` maps to Provisioning | create-blocks: defined, started and polled to running in one `Create`; `nostate` maps to Provisioning (`mapState`) | create-blocks: cloned, configured, started and polled to running in one `Create`; `qmpstatus: prelaunch` maps to Provisioning (`mapState`) | +| `async-ops` | none | task-handle, but the handle is discarded: an insert that fails after acceptance is never read | none | none | task-handle and backend-lock: every clone, upload, resize, start, stop, destroy and volume delete returns a UPID polled to its end within the call (`waitTask`); a VM's lock is waited on in Create and Transient in Get and Delete (`waitUnlocked`) | +| `secondary-resources` | none | cascade: `AutoDelete` on the boot disk (`buildInsertRequest`) | explicit: overlay and seed removed by name (`removeVolumes`); base image kept | explicit: overlay and seed removed by name (`removeVolumes`); base image kept | explicit: the seed ISO deleted by name (`deleteSeed`, `deleteSeedEverywhere`); the disks go with the destroy's `purge=1&destroy-unreferenced-disks=1` | +| `credentials` | workload-identity: in-cluster config or kubeconfig (`NewWithTransportWrapper`) | environment: Application Default Credentials | mounted-files: SSH key and known_hosts from the optional `kvm-ssh` Secret, read lazily (`newLazySSH`) | mounted-files: the same `kvm-ssh` Secret, read on every dial and never cached (`loadCredentials`) | mounted-files: the token from the optional `proxmox-token` Secret, read on every request (`readToken`); the CA from the same Secret, loaded once and cached (`httpClient`) | +| `client-kind` | vendor-sdk: controller-runtime `client.Client`, no seam interface | vendor-sdk: `cloud.google.com/go/compute` behind `instancesAPI` | rpc-over-ssh: `go-libvirt` over an SSH-forwarded unix socket, behind `libvirtAPI` (`conn.go`) | rpc-over-ssh: `go-libvirt` over an SSH-forwarded unix socket, behind the ctx-first `backend` seam (`backend.go`) | thin-client: `net/http` and `encoding/json` behind the ctx-first `api` seam, one method per endpoint (`api.go`, `httpAPI`) | +| `timeouts` | ctx-only: no `rest.Config.Timeout` is set | ctx-only: SDK defaults, with SDK retries on Get and List | per-call for the dial only (`defaultHandshakeTimeout` in `dialSSH`); RPCs are ctx-only and a cancelled `call` closes the connection | per-call: a budget around the dial, every RPC, the uploads and the download (`defaultBudgets`, `call`); a cancelled or overlong call kills the pooled connection | per-call: dial and TLS handshake 10 s, each request `requestTimeoutSeconds`, a budget per task wait (`defaultBudgets`); a cancelled request ends only itself | +| `logging` | none | tiered: V(1) outcomes and `logAPIError`, V(2) detail, V(5) wire log (`wirelog.go`) | tiered: V(1) outcomes and one `libvirt call` line per RPC (`trace.go`), V(2) XML and addresses; no wire tier | tiered: V(1) outcomes and one `libvirt call` line per RPC (`observe.go`), V(2) XML and addresses; no wire tier | tiered: V(1) outcomes and one `proxmox api call` line per request (`observe.go`), V(2) parameters, config summaries and addresses; no wire tier | +| `child-spans` | transport-wrapper: `tracing.RestConfigWrapper()` on its own `rest.Config`, parent-gated | sdk-instrumentation: the SDK adds `otelhttp` itself; not parent-gated, but every call site has a parent | **none**: `trace.go` is a log decorator, not spans | span-decorator: `kvm2.` per RPC (`observe`), plus `kvm2.dial` (`libvirtDialer`) and `kvm2.download` (`fetchImage`) | transport-wrapper: `tracing.RestConfigWrapper()` on the client's transport, parent-gated, one `HTTP ` span per request (`newHTTPAPI`) | +| `backend-test` | cluster-e2e: kind (`test/e2e`) | manual-runbook: [gcp-vm-validation.md](gcp-vm-validation.md) | integration-tag: `kvm_integration_test.go`, `make test-integration-kvm` | integration-tag: `kvm2_integration_test.go`, `make test-integration-kvm2` | integration-tag: `proxmox_integration_test.go`, `make test-integration-proxmox`; compiled, not yet run against a host | ## 2. Other differences Rows that are not profile axes but that a new provider author compares against. -| Feature | kubernetes | gcp | kvm | kvm2 | -|---|---|---|---|---| -| Create verifies the adopted instance's UID | no | no | no | **yes**: a domain without the kvm2 mark or with a different non-empty UID is Permanent; a marked one with a matching or empty UID is adopted (`create`) | -| Placement used | none | Zone, MachineType, Image | Zone (host), MachineType, Image | Zone (host), MachineType, Image | -| CloudInit delivered | **no**: ignored | yes: `user-data` metadata | yes: NoCloud seed ISO built in-process (`buildSeedISO`) | yes: NoCloud seed ISO built in-process (`nocloud.BuildSeedISO`) | -| Port used | yes: Squid config and containerPort (`squidConf`) | no: firewall is manual | no | no | -| States produced | Provisioning, Running, Terminated | all four | Provisioning, Running, Stopped | Provisioning, Running, Stopped | -| Deleting instance still reports its state | yes: a Pod being deleted reports its phase | n/a | n/a | n/a: Delete waits, and a concurrent Get sees the domain until it is undefined | -| Error.Provider is the configured name | **no**: hardcoded `"kubernetes"` | yes | yes | yes (`wrapErr`) | -| Checker target | `apiserver` | the project | each host, concurrently | each host, concurrently, on a fresh connection | -| Checker Details | none | none | `user`, `pool`, `hostname`, `libvirtVersion`, `poolState`, `poolCapacity`, `poolAvailable` | `user`, `pool`, `hostname`, `libvirtVersion`, `poolState`, `poolCapacityGiB`, `poolAvailableGiB`, filled in as far as the check got | -| Credential rotation without restart | n/a | via ADC | no: cached after first successful load | yes: both files read on every dial (`loadCredentials`) | -| Connection handling | client-go | SDK client, never closed | one pooled SSH plus libvirt connection per host, 15s keepalive (`hostPool`) | one pooled SSH plus libvirt connection per host, 15s keepalive (`hostTable`); dedicated connections for the image upload and the check (`dialFresh`) | -| Unit-test double | controller-runtime fake client | `fakeAPI` behind `instancesAPI`, plus `httptest` for the real SDK client in `check_test.go` | `fakeAPI` behind `libvirtAPI`, plus an in-process `x/crypto/ssh` server in `conn_test.go` | `fakeBackend` behind `backend` (`fake_test.go`), plus an in-process `x/crypto/ssh` server (`sshserver_test.go`) and go-libvirt's own mock for the adapter | -| Proxy software | generated Squid config (`pod.go`) | from `cloudInit`; the image must run cloud-init | from `cloudInit`, plus a vendor-data script that installs the guest agent (`seed.go`) | from `cloudInit`, plus a vendor-data script that installs the guest agent (`nocloud.GuestAgentScript`) | -| IP reachable from the operator | Pod network, so in-cluster only (not from `make run-dev` on a laptop) | external IP, once a firewall rule allows the cluster's egress IP | LAN address on the host's bridge; the operator needs a route to that LAN | LAN address on the host's bridge; the operator needs a route to that LAN | -| Distinct egress IP per proxy | no: the cluster's egress | yes: ephemeral external IP | whatever the host's LAN routes through; not per VM by itself | whatever the host's LAN routes through; not per VM by itself | +| Feature | kubernetes | gcp | kvm | kvm2 | proxmox | +|---|---|---|---|---|---| +| Create verifies the adopted instance's UID | no | no | no | **yes**: a domain without the kvm2 mark or with a different non-empty UID is Permanent; a marked one with a matching or empty UID is adopted (`create`) | **yes**: a VM with the proxy's name and no mark, or with a different non-empty UID, is Permanent; a marked one with a matching or empty UID is adopted (`inspect`) | +| Placement used | none | Zone, MachineType, Image | Zone (host), MachineType, Image | Zone (host), MachineType, Image | Zone (node), MachineType, Image (a template key) | +| CloudInit delivered | **no**: ignored | yes: `user-data` metadata | yes: NoCloud seed ISO built in-process (`buildSeedISO`) | yes: NoCloud seed ISO built in-process (`nocloud.BuildSeedISO`) | yes: NoCloud seed ISO built in-process (`nocloud.BuildSeedISO`) and uploaded through the API (`uploadSeed`) | +| Port used | yes: Squid config and containerPort (`squidConf`) | no: firewall is manual | no | no | no | +| States produced | Provisioning, Running, Terminated | all four | Provisioning, Running, Stopped | Provisioning, Running, Stopped | Provisioning, Running, Stopped | +| Deleting instance still reports its state | yes: a Pod being deleted reports its phase | n/a | n/a | n/a: Delete waits, and a concurrent Get sees the domain until it is undefined | n/a: Delete waits, and a concurrent Get sees the VM (Stopped once stopped) until the destroy removes its config | +| Error.Provider is the configured name | **no**: hardcoded `"kubernetes"` | yes | yes | yes (`wrapErr`) | yes (`wrapErr`) | +| Checker target | `apiserver` | the project | each host, concurrently | each host, concurrently, on a fresh connection | each node, concurrently, on the shared HTTP client | +| Checker Details | none | none | `user`, `pool`, `hostname`, `libvirtVersion`, `poolState`, `poolCapacity`, `poolAvailable` | `user`, `pool`, `hostname`, `libvirtVersion`, `poolState`, `poolCapacityGiB`, `poolAvailableGiB`, filled in as far as the check got | `isoStorage`, `tokenID`, `pveVersion`, `kernel`, `isoStorageAvailGiB`, `templates`, filled in as far as the check got | +| Credential rotation without restart | n/a | via ADC | no: cached after first successful load | yes: both files read on every dial (`loadCredentials`) | the token yes, read on every request (`readToken`); the CA no, cached after first use | +| Connection handling | client-go | SDK client, never closed | one pooled SSH plus libvirt connection per host, 15s keepalive (`hostPool`) | one pooled SSH plus libvirt connection per host, 15s keepalive (`hostTable`); dedicated connections for the image upload and the check (`dialFresh`) | one `net/http` client, 4 idle connections per host, HTTP/1.1 only; a cancelled request ends only itself | +| Unit-test double | controller-runtime fake client | `fakeAPI` behind `instancesAPI`, plus `httptest` for the real SDK client in `check_test.go` | `fakeAPI` behind `libvirtAPI`, plus an in-process `x/crypto/ssh` server in `conn_test.go` | `fakeBackend` behind `backend` (`fake_test.go`), plus an in-process `x/crypto/ssh` server (`sshserver_test.go`) and go-libvirt's own mock for the adapter | `fakeAPI` behind `api` (`fake_test.go`), plus `httptest.NewTLSServer` replaying the Phase 0 captures for the real client (`client_test.go`) | +| Proxy software | generated Squid config (`pod.go`) | from `cloudInit`; the image must run cloud-init | from `cloudInit`, plus a vendor-data script that installs the guest agent (`seed.go`) | from `cloudInit`, plus a vendor-data script that installs the guest agent (`nocloud.GuestAgentScript`) | from `cloudInit`, plus a vendor-data script that installs the guest agent (`nocloud.GuestAgentScript`) | +| IP reachable from the operator | Pod network, so in-cluster only (not from `make run-dev` on a laptop) | external IP, once a firewall rule allows the cluster's egress IP | LAN address on the host's bridge; the operator needs a route to that LAN | LAN address on the host's bridge; the operator needs a route to that LAN | address on the proxy network (bridge or VLAN from the config); the operator needs a route to it | +| Distinct egress IP per proxy | no: the cluster's egress | yes: ephemeral external IP | whatever the host's LAN routes through; not per VM by itself | whatever the host's LAN routes through; not per VM by itself | whatever the proxy network routes through; not per VM by itself | ## 3. Observability status -| Layer | kubernetes | gcp | kvm | kvm2 | -|---|---|---|---|---| -| 1. Operation spans and metrics | yes: decorators | yes: decorators | yes: decorators | yes: decorators | -| 2. Child spans per backend call | yes: `tracing.RestConfigWrapper()`, parent-gated (`cmd/main.go`, `NewWithTransportWrapper`) | yes: the SDK's own `otelhttp`; not parent-gated | **no** | yes: `kvm2.` from the `observe` decorator, `kvm2.dial`, `kvm2.download`; always under the caller's span, since every call carries the caller's ctx | -| 3. Logs carry `traceID` and `spanID` | n/a: the package does not log | yes, including the V(5) wire log | yes: every `libvirt call` line at V(1) | yes: every `libvirt call` line at V(1), written inside its RPC's span | -| Tested against a real collector | yes: the kind e2e checks `provider.create` and `provider.delete` in Tempo (`test/e2e/tracing_test.go`), operation spans only | no | no | no; the integration test asserts the parent-child topology in an in-memory recorder (`assertLifecycleSpans`) | +| Layer | kubernetes | gcp | kvm | kvm2 | proxmox | +|---|---|---|---|---|---| +| 1. Operation spans and metrics | yes: decorators | yes: decorators | yes: decorators | yes: decorators | yes: decorators | +| 2. Child spans per backend call | yes: `tracing.RestConfigWrapper()`, parent-gated (`cmd/main.go`, `NewWithTransportWrapper`) | yes: the SDK's own `otelhttp`; not parent-gated | **no** | yes: `kvm2.` from the `observe` decorator, `kvm2.dial`, `kvm2.download`; always under the caller's span, since every call carries the caller's ctx | yes: `tracing.RestConfigWrapper()` on the transport, parent-gated, one `HTTP ` per request | +| 3. Logs carry `traceID` and `spanID` | n/a: the package does not log | yes, including the V(5) wire log | yes: every `libvirt call` line at V(1) | yes: every `libvirt call` line at V(1), written inside its RPC's span | yes: every `proxmox api call` line at V(1), with the operation span's `spanID` | +| Tested against a real collector | yes: the kind e2e checks `provider.create` and `provider.delete` in Tempo (`test/e2e/tracing_test.go`), operation spans only | no | no | no; the integration test asserts the parent-child topology in an in-memory recorder (`assertLifecycleSpans`) | no; `TestTracing_httpSpansUnderOperationSpan` asserts the parent-child topology in an in-memory recorder | ## 4. kubernetes (`internal/provider/kubernetes`) @@ -528,20 +531,209 @@ same libvirt. its backing image or size. - The sample cloud-init installs Squid with `http_access allow all`. -## 8. Gaps shared by the existing providers +## 8. proxmox (`internal/provider/proxmox`) -- **Adoption is not verified in three of the four.** kubernetes, gcp and +The full user guide is [proxmox-provider.md](proxmox-provider.md) and the +design it was built from is +[proxmox-provider-spec.md](proxmox-provider-spec.md); this section is the +author's view. It is the second provider built from the contract, and the +first on a REST API of the hypervisor's own: no SSH, no libvirt. Every +error text it matches was captured from a PVE 9.2.20 host before the code +was written. + +- **Package and seam.** `proxmox.go` (`Provider`, `New`, `withDefaults`, + `node`, `wrapErr`, the per-name lock, the budgets), `create.go`, + `get.go`, `delete.go`, `list.go` (the four methods), `api.go` (the + seam), `client.go` (the production adapter `httpAPI`, the token and CA + loading), `observe.go` (the logging decorator), `task.go` (task, lock + and running waits), `seed.go`, `mark.go`, `id.go`, `state.go`, + `errors.go`, `check.go`; the seed ISO comes from the shared + `internal/provider/nocloud` and the name lock from + `internal/provider/keyedlock`. The seam is `api`: one method per + endpoint, sixteen in all, each with `ctx` first. `New` wraps `httpAPI` in + `observe` once, so provider code never holds an undecorated seam; + `newProvider` is the test constructor. +- **Config.** `ProviderConfig.Proxmox`, the `ProxmoxConfig` type in + `internal/provider/config_proxmox.go`, validated by `validateProxmox` + (the URL shape, node names and storage ids by regexp because they end + up in request paths, `vlanTag`, `vmidRange`, the two catalogues). + `isoStorage` has no default on purpose: the token needs + `Datastore.Allocate` there. Defaults (`vmidRange` 100000 to 199999, + `requestTimeoutSeconds` 30, `installGuestAgent` true) are filled by + `withDefaults`. `New` never makes a request; it warns when the token + file does not exist, or when `insecureSkipTLSVerify` is on. +- **ID.** `formatProviderID(vmid, name)` gives `vms//`, with no + node, because VMs migrate; the node is resolved from + `/cluster/resources` on every call (`resolve`). `parseProviderID` + accepts exactly three segments, the literal `vms`, a decimal VMID + without leading zeros between 100 and 999999999 and a non-empty name, + else `ErrPermanent`. The name is there so `Get` and `Delete` can tell + our VM from another that later got the same VMID. +- **Create.** `placement` checks `Zone` (`node`; optional with one node), + `MachineType` and `Image` (both catalogue keys) before any request, and + logs the resolved parameters with `cloudInitBytes` at V(2). Then, under + the per-name lock, `create`: one `ClusterResources` read; + `adoptByName` (two VMs with the name are Permanent, one goes to + `inspect`); with none, `template` reads the template's config and + refuses a non-template, a cloud-init drive, a missing `net0` when the + config rewrites it, and an unsized boot disk when `diskGiB` is set, all + before any clone; `allocate` probes `vmidFor(uid, attempt)` up to 32 + times, cloning onto a free VMID and running `inspect` on a taken one or + on the clone's `config file already exists` answer. `inspect` waits out + a lock (`waitUnlocked`; a clone in progress is also nameless), then + adopts our name with our mark and a matching or empty UID, probes past + another name, and is Permanent for our name without the mark, with + another UID, or hibernated. An adopted VM gets one `VMStatus`: running + returns at once. Otherwise `provision`: `uploadSeed` (rebuilt and + overwritten every attempt), `SetVMConfig` (`configParams`: cores, + memory, `agent=1`, the tag, `ide2`, and `net0` through `rewriteNet0` + when a bridge or VLAN is configured), the resize task when + `resizeTarget` says the boot disk is smaller than `diskGiB`, the start + task, and `waitRunning` (every second up to 60 s, Transient on expiry + or on a stop). `Create` returns only when the VM runs. +- **Get and state.** `get` checks in order: absent from the resources is + NotFound; a lock other than `suspended` is Transient (checked before the + name, because a mid-clone entry has none); `status: unknown` is + Transient; the config's `does not exist` is NotFound, a config lock + Transient; another name or no mark is NotFound; hibernated is Stopped + without further calls. Then `VMStatus` and `mapState(status, + qmpstatus, ip)`: + + | `status` | `qmpstatus` | agent | Provider state | + |---|---|---|---| + | running | running | an IPv4 | Running | + | running | running | 200 without an IPv4, or `QEMU guest agent is not running` | Provisioning | + | running | running | `No QEMU guest agent configured` | Stopped | + | running | prelaunch | not asked | Provisioning | + | running | paused, suspended, io-error, anything else | not asked | Stopped | + | stopped, anything else | any | not asked | Stopped | + + Any other agent answer, including `VM is not running`, is + Transient. `UID` and `CreatedAt` are read from the description on every + call. A running `Get` is four calls: resources, config, status, agent. +- **Delete.** Under the per-name lock, `destroy`: `resolve`; a lock other + than `suspended` or `status: unknown` is Transient; the config's `does + not exist`, another name or no mark leave the VM alone; our VM + hibernated is Permanent, because the destroy refuses it and the lock + never clears. Otherwise stop (unless stopped) and destroy with + `purge=1&destroy-unreferenced-disks=1`, each a task waited on through + `runTask`, where `does not exist` means gone. Then `deleteSeed` on the + VM's node, where a failure fails the call (a 403 for + `Datastore.Allocate` is Permanent), or, with no VM of ours destroyed, + `deleteSeedEverywhere` over every configured node, where a node that + fails is logged and skipped unless ctx ended. A missing volume still + ends in a successful task, so no message is matched. Nothing is left + when nil is returned. +- **ListByTag.** One `ClusterResources` covers every node, unreachable + ones included (their guests are listed with `status: unknown`). + `isCandidate` keeps QEMU VMs that are not templates and carry the tag + `proxy_operator` or a name starting `proxy-`; a nameless entry (a clone + in progress) is not a candidate. One `VMConfig` per candidate; the mark + in the description decides, and the `Instance` is `ID`, `UID` and + `CreatedAt` only. A candidate whose config cannot be read is skipped at + V(1), unless ctx ended, which fails the call; the resources call failing + fails the whole call. +- **Errors.** `classify(op, err)`: an error already carrying a class keeps + it; then `apiError`s by `classifyAPI` (every 4xx Permanent; the 500 + `does not exist` NotFound, Transient when `op == "create"`; every other + status Transient, including the lock, the taken VMID and pveproxy's 595 + and 596); TLS verification failures (`tls.CertificateVerificationError` + and the three `x509` types) Permanent; everything else Transient. The + 500 matchers (`isNotExist`, `isLocked`, `isAlreadyExists`, the three + agent ones) are anchored to the 500 status and each is backed by a + captured response. `permanentf` and `transientf` classify at the + source: placement, the template checks, the ownership check, a + malformed token file or CA file (Permanent); a missing token or CA file, + an expired wait, a failed task (Transient). Nothing is Quota. +- **Check.** `Check` runs `checkNode` per node concurrently on the shared + client. `probeNode` reads the token file for `tokenID` (no request when + it is unusable), then `Version`, `NodeStatus`, `checkStorage` (a 500 + whatever its text, inactive, or no `iso` content is `StorageNotFound`) + and `checkTemplates` (each catalogue VMID exists and is a template, + else `TemplateNotFound`). `checkNode` answers `Timeout` whenever the + round's ctx ended; otherwise `checkReason`: the two failures above, + `errCredentials` `CredentialsUnavailable`, `errCAFile` or a TLS + verification failure `CertificateInvalid`, 401 `Unauthorized`, 403 + `Forbidden`, 595 and 596 `Unreachable`, any other status `APIError`, a + `net.OpError` `Unreachable` (before the deadline, so a dial out of its + own budget is a node that did not answer), a deadline or cancellation + `Timeout`, a cut connection `Unreachable`. +- **Connections.** One `http.Client`, built on first use once the CA + loads (`httpClient`), shared by every call and the Checker: dial and + TLS handshake 10 s each, `Client.Timeout` `requestTimeoutSeconds`, 4 + idle connections per host kept 90 s, HTTP/2 disabled. A cancelled + request ends only itself. The token file is read by `readToken` on + every request. +- **Logging.** Logger from the context, named `proxmox`, with `provider`, + and `vmid`, `name` and `node` on lines about an instance. V(1): one line + per operation outcome (`proxmox vm ensured running`, `proxmox vm + fetched`, `proxmox vm deleted` with `destroyed`, `proxmox vms listed`, + `proxmox node checked`), one `proxmox api call` line per seam call from + `observe` (call, identifiers, UPID, duration, `httpStatus` and `error`), + `proxmox task failed`, and the no-op lines. V(2): request parameters, + config summaries, agent addresses, the resolved Create parameters. + Cloud-init, the token, the CA and the upload body are never logged. +- **Tracing.** No span of its own: `newHTTPAPI` wraps the transport in + `tracing.RestConfigWrapper(opts...)`, so each request is an + `HTTP ` client span under the caller's span, and none without + one. `observe` only logs, so nothing is counted twice. +- **Tests.** Pure pieces (`id_test.go`, `mark_test.go`, `state_test.go`, + `seed_test.go`, `errors_test.go` with the classify table over the + captures, `api_test.go`); `client_test.go` (the real client over + `httptest.NewTLSServer`: the token never in errors or logs, x509 + failures Permanent, cancellation, the child span); `create_test.go`, + `get_test.go`, `delete_test.go`, `list_test.go`, `task_test.go`, + `proxmox_test.go`, `check_test.go`, `observe_test.go` (every seam + method logs exactly one line, by reflection) and `provider_test.go` + (the contract §11 sweep, named test by test in the spec's entry 21). + The double is `fakeAPI` in `fake_test.go`: VMs, tasks and ISOs in + memory, a clone that leaves a nameless locked entry until its task + ends, per-method errors and blocking, a call log; its error texts are + checked against the captures. Real backend: + `proxmox_integration_test.go` behind `//go:build integration`, driven + by `PROXMOX_TEST_*` variables and `make test-integration-proxmox`, + described in + [testing.md](testing.md#the-proxmox-integration-test-real-proxmox-ve-host); + it compiles but has not run against a host yet. +- **Deployment.** The optional `proxmox-token` Secret (keys `token` and + `ca.pem`) mounted at `/etc/proxy-operator/proxmox` + (`config/manager/manager.yaml`), `make proxmox-token-secret`, a + commented `proxmox-home` entry in both providers configs (loaded by + `TestLoadConfig_proxmoxSampleEntry`), `config/samples/proxy_proxmox.yaml`, + the `proxmox` type in the `status.type` comment of + `api/v1alpha1/providerhealth_types.go`. No RBAC, and no + `hack/providers-dev.yaml` entry, since it needs credentials. +- **Known gaps** (see also [proxmox-provider.md § Known gaps](proxmox-provider.md#known-gaps)). + - `Create` blocks until the VM runs. + - A clone lock that never clears makes every Create for that Proxy wait + out the 10-minute clone budget; `qm unlock` by hand. + - GC blind spot: a seed ISO whose VM was destroyed outside the operator + is invisible to GC, and a seed on node-local storage stays behind + when its VM migrated. + - The VMID range is shared with humans; a hand-made VM with a proxy's + name is Permanent. + - Cross-node clones (`target`) need shared storage and are untested. + - A hibernated VM of ours latches its Proxy in Create and in Delete. + - The CA is cached after first use; the token is not. + - Never run against a real host yet; three answers were never captured + (the storage status for a missing storage, a destroy on a hibernated + VM, a stop on a paused VM). + +## 9. Gaps shared by the existing providers + +- **Adoption is not verified in three of the five.** kubernetes, gcp and kvm adopt any existing instance with the right name without checking - `LabelUID`; kvm2 returns `ErrPermanent` when the UID differs. -- **The Create comment does not match kvm and kvm2.** The interface comment - in `internal/provider/provider.go` says `Create` returns as soon as the - request is submitted; both libvirt providers block until the domain has - started. + `LabelUID`; kvm2 and proxmox return `ErrPermanent` when the UID differs. +- **The Create comment does not match kvm, kvm2 and proxmox.** The + interface comment in `internal/provider/provider.go` says `Create` + returns as soon as the request is submitted; the three VM providers on + your own hypervisors block until the VM has started. - **`Stopped` and `Terminated` are produced inconsistently.** kubernetes - never produces Stopped, kvm and kvm2 never produce Terminated, and the - reconciler treats them the same. + never produces Stopped, kvm, kvm2 and proxmox never produce Terminated, + and the reconciler treats them the same. - **Different `ListByTag` partial-failure policies.** gcp leaves failed - parts out, kvm and kvm2 fail the whole list. Both are safe. + parts out, kvm and kvm2 fail the whole list, and proxmox fails it when + its one cluster-wide listing fails. Both are safe. - **Stale `mock` comments** in `internal/provider/provider.go`, `internal/provider/registry/registry.go` and `api/v1alpha1/proxy_types.go` (and so the generated CRD's `spec.provider` description). @@ -555,10 +747,10 @@ same libvirt. log lines. kvm2 has them, but only an in-memory recorder checks their topology. Only kubernetes is tested against a real collector, and that test checks only the operation spans; nothing checks that gcp's SDK spans, - the kubernetes API spans or kvm2's spans nest under `provider.*` in a - real collector. + the kubernetes API spans, kvm2's spans or proxmox's HTTP spans nest under + `provider.*` in a real collector. -## 9. Proposed features +## 10. Proposed features Listed only. Nothing below is designed or implemented, except where a bullet says kvm2 already does it. @@ -576,7 +768,7 @@ bullet says kvm2 already does it. backend, so a new provider gets the contract tests for free. - **Verify the UID when `Create` adopts an instance.** Return `ErrPermanent` on a mismatch instead of adopting an instance that belongs to someone else. - kvm2 does this; the other three do not. + kvm2 and proxmox do this; the other three do not. - **Report asynchronous create failures:** gcp keeps the operation name, and `Get` reports the failure instead of NotFound. - **Capacity or quota signals** for kvm, such as pool free space and host diff --git a/docs/providers.md b/docs/providers.md index c8431d9..8a96803 100644 --- a/docs/providers.md +++ b/docs/providers.md @@ -14,12 +14,13 @@ the existing ones happen to work. open. Copy it to `docs/-provider-spec.md`, fill it in, and hand it to the implementer with the other two files. - **[providers-existing.md](providers-existing.md)**: how kubernetes, gcp, - kvm and kvm2 meet the contract, a table mapping each of them to every + kvm, kvm2 and proxmox meet the contract, a table mapping each of them to every profile, their per-provider reference notes, shared gaps and proposed features. Per-provider user guides: [kvm-provider.md](kvm-provider.md), [kvm2-provider.md](kvm2-provider.md), +[proxmox-provider.md](proxmox-provider.md), [gcp-vm-validation.md](gcp-vm-validation.md) and [gcp-in-specific-project.md](gcp-in-specific-project.md). diff --git a/docs/proxmox-provider.md b/docs/proxmox-provider.md new file mode 100644 index 0000000..e152118 --- /dev/null +++ b/docs/proxmox-provider.md @@ -0,0 +1,646 @@ +# The proxmox provider + +Runs Managed proxies as VMs on a Proxmox VE cluster. It is the third +VM-style provider and the second built from +[provider-contract.md](provider-contract.md), from the design in +[proxmox-provider-spec.md](proxmox-provider-spec.md). The operator talks to +the cluster's REST API only (`https://:8006/api2/json`) with an API +token. It needs no SSH, no shell on the nodes, and no library beyond Go's +standard `net/http`. + +Each VM is a **full clone of a template** you prepare once. The operator +builds the Proxy's cloud-init into a NoCloud seed ISO in-process, uploads +it to a storage on the VM's node, attaches it as a CD-ROM, sets the size, +starts the VM and learns its address from the QEMU guest agent. Everything +here was proven against **Proxmox VE 9.2.20** (QEMU 11.0) on the single +node `pv1-1`; every error text the provider matches was captured from that +host. + +## Proxmox prerequisites + +Once per cluster, as root on a node: + +1. **Proxmox VE 9.x with the API reachable** from the operator's pods on + port 8006. The operator's pods and the crawler clients must also route to + the network the proxies land on (see step 5). +2. **A template without a Proxmox cloud-init drive.** The provider brings + its own seed on `ide2`; a Proxmox cloud-init drive would compete with it + and the Proxy's user-data would be ignored. Debian 13's generic cloud + image boots UEFI (under BIOS it hangs in GRUB on a headless VM), so the + template does too: + + ```sh + wget https://cloud.debian.org/images/cloud/trixie/latest/debian-13-generic-amd64.qcow2 + qm create 9001 --name debian-13-template --memory 2048 --cores 2 --ostype l26 \ + --machine q35 --bios ovmf --efidisk0 local-lvm:1,efitype=4m,pre-enrolled-keys=0 \ + --scsihw virtio-scsi-pci --net0 virtio,bridge=vmbr0 \ + --serial0 socket --vga serial0 --agent 1 + qm set 9001 --scsi0 local-lvm:0,import-from=/root/debian-13-generic-amd64.qcow2 + qm set 9001 --boot order=scsi0 + qm template 9001 + ``` + + The template must boot from its first disk, run cloud-init with the + NoCloud datasource (the Debian and Ubuntu cloud images do), and either + ship `qemu-guest-agent` or be able to install it at first boot. Before + the first clone for a Proxy, `Create` reads the template's config and + refuses it with a permanent error when: + + - it is not a template (`qm template` was not run); + - any disk slot holds a Proxmox cloud-init drive (a value containing + `cloudinit`), such as the one `qm set --ide2 local-lvm:cloudinit` + adds for Proxmox's own cloud-init support; + - it has no `net0` while the config sets `bridge` or `vlanTag`; + - the machine type sets `diskGiB` and the template has no boot disk, or + one without a readable `size=`. + + These are checked on the template because the clone copies its config: + a VM cloned from a bad template could not be fixed by fixing the + template. The template is checked only when a VM is about to be cloned; + a VM that already exists no longer depends on it. The health check sees + only whether each template exists and is a template. +3. **A storage for the seed ISOs**, holding `iso` content. A dedicated + directory storage is best, because the token needs `Datastore.Allocate` + on it to delete seeds, and on a dedicated storage that privilege cannot + touch backups or other ISOs: + + ```sh + pvesm add dir proxy-seeds --path /var/lib/proxy-seeds --content iso + ``` + + A seed is tens of KiB. Proxmox stages uploads under `/var/tmp`, not the + node's tmpfs `/tmp`. On a cluster, a storage that is not shared keeps + each seed on the node the VM was created on (see *Known gaps*). +4. **A user, two roles and a token.** `VM.Monitor` no longer exists in PVE + 9 and `pveum` rejects it; the agent read needs `VM.GuestAgent.Audit` + alone. `VM.Config.Cloudinit` is not needed for the seed on `ide2`. + `SDN.Use` on `/` is what lets the config write put a VLAN tag on + `vmbr0`: + + ```sh + pveum user add proxy-operator@pve --comment "egress-proxies-operator" + pveum role add ProxyOperator -privs "VM.Allocate VM.Clone VM.Audit VM.Config.CDROM VM.Config.CPU VM.Config.Memory VM.Config.Disk VM.Config.Network VM.Config.Options VM.PowerMgmt VM.GuestAgent.Audit Datastore.Audit Datastore.AllocateSpace Datastore.AllocateTemplate Sys.Audit SDN.Use" + pveum aclmod / -user proxy-operator@pve -role ProxyOperator + pveum role add ProxyOperatorSeeds -privs "Datastore.Allocate Datastore.AllocateTemplate Datastore.AllocateSpace Datastore.Audit" + pveum aclmod /storage/proxy-seeds -user proxy-operator@pve -role ProxyOperatorSeeds + pveum user token add proxy-operator@pve operator --privsep 0 + ``` + + The second role repeats three privileges of the first on purpose: an + ACL on a more specific path **replaces** what the user inherits from + `/` for that path, so a role with `Datastore.Allocate` alone would take + the upload's `Datastore.AllocateTemplate` away. The role on `/` covers + clone, config, resize, start, stop, the agent read, the upload and the + destroy; what it lacks is `Datastore.Allocate`, without which deleting + a seed is a 403 (`Permission check failed (/storage/, + Datastore.Allocate)`). With `--privsep 0` the token carries the user's + privileges. + + The token's secret is printed once. Save it on your workstation as one + line, `proxy-operator@pve!operator=`, in + `~/.config/proxmox-token` with mode 0600. Copy the node's CA, + `/etc/pve/pve-root-ca.pem`, to `~/.config/proxmox-ca.pem`. +5. **A proxy network with DHCP and a route to the internet.** Each VM + keeps the template's one NIC, optionally moved to another bridge or + VLAN by the config, and takes its address from that network's DHCP. The + network needs internet twice over: the vendor-data script installs + `qemu-guest-agent` from the distro mirrors at first boot, and the agent + is the only way the operator learns the VM's address; and an egress + proxy needs egress anyway. Baking `qemu-guest-agent` into the template + and setting `installGuestAgent: false` removes the mirror from IP + discovery. The operator's pods and the clients must route to that + network: the health engine probes `http://:` through the + proxy, and clients get the same address from the discovery API. The + provider does not use `spec.port` and configures no Proxmox firewall. +6. **A reserved VMID range.** VMIDs are allocated from `vmidRange` + (default 100000 to 199999). Keep hand-made VMs out of it: one there + costs a probe, and one named like a proxy is refused (see *What lands + on Proxmox*). + +Check from the operator's point of view before wiring it in. This sends +the token through a file descriptor, so it never reaches the command line: + +```sh +pve() { + curl -s --cacert ~/.config/proxmox-ca.pem \ + -H @<(printf 'Authorization: PVEAPIToken=%s\n' "$(tr -d '[:space:]' < ~/.config/proxmox-token)") \ + "https://192.168.0.8:8006/api2/json/$1"; echo +} +pve version # the token works, TLS verifies +pve nodes/pv1-1/storage/proxy-seeds/status # active, content includes iso +pve 'cluster/resources?type=vm' # the template is listed with "template":1 +``` + +## Cluster-side setup + +1. Create the Secret. It holds two keys, `token` (the one-line token file) + and `ca.pem` (the node's CA): + + ```sh + make proxmox-token-secret + # or by hand: + kubectl -n egress-proxies-operator-system create secret generic proxmox-token \ + --from-file=token=$HOME/.config/proxmox-token \ + --from-file=ca.pem=$HOME/.config/proxmox-ca.pem + ``` + + The make target runs that command as an apply, so it also updates an + existing Secret. Override the paths with + `PROXMOX_TOKEN_FILE=... PROXMOX_CA_FILE=...`. Before it applies, it + checks that both files exist, that the token file is exactly one line + of `@!=` (without printing it), and that + the CA file holds a PEM certificate. With `insecureSkipTLSVerify` + instead of a CA, create the Secret by hand with the `token` key only. + + `config/manager/manager.yaml` mounts the Secret (optional, read-only) + at `/etc/proxy-operator/proxmox`, so the files are + `/etc/proxy-operator/proxmox/token` and + `/etc/proxy-operator/proxmox/ca.pem`. The deployment starts fine + without it, even with a proxmox provider configured: `New` makes no + request, only checks that the token file exists, and logs a `WARNING:` + line when it does not. The provider's proxies then fail transiently and + recover once the Secret lands; the health check reports + `CredentialsUnavailable` meanwhile. + + **Rotating the token needs no restart.** The token file is read on + every request, so the next request after the kubelet refreshes the + mount uses the new token. **Rotating the CA does**: it is loaded on + first use and cached for the life of the process. A missing CA file is + retried until it lands; one without a certificate is a permanent error. + +2. Add a provider entry to the providers config + (`config/manager/providers_config.yaml` in-cluster; the full sample, + commented out, is in `config/samples/providers-config.yaml`): + + ```yaml + - name: proxmox-home + type: proxmox + proxmox: + url: https://192.168.0.8:8006 + tokenFile: /etc/proxy-operator/proxmox/token + caFile: /etc/proxy-operator/proxmox/ca.pem + nodes: [pv1-1] + isoStorage: proxy-seeds + vlanTag: 4 + templates: + debian-13: {vmid: 9001} + machineTypes: + small: {cores: 1, memoryMiB: 1024} + ``` + + **The URL's host must be in the node certificate.** The node's + self-signed certificate lists its own addresses and names, and TLS + verification checks the URL's host against them. On `pv1-1` it lists + `192.168.0.8`, `pv1-1` and `pv1-1.home.lab`, so + `https://192.168.2.8:8006` reaches the same node and still fails + verification (`CertificateInvalid` in the health check, a permanent + error on every call). See what yours lists with: + + ```sh + openssl s_client -connect 192.168.0.8:8006 /dev/null | openssl x509 -noout -ext subjectAltName + ``` + + If the address the operator must use is not there, either give the + node a certificate that names it (a custom certificate in the node's + Certificates panel) and put that certificate's CA in `caFile`, or set + `insecureSkipTLSVerify: true`, which `New` logs as a `WARNING:` line. + + `nodes` holds node names as Proxmox knows them, not addresses: the one + URL reaches the whole cluster. The providers config is read once per + process, so adding a node, a template or a machine type needs a manager + restart. + +3. Apply a Proxy (`config/samples/proxy_proxmox.yaml`) and watch it: + + ```sh + kubectl get px -w + # NAME MODE PROVIDER PHASE IP HEALTHY + # proxy-proxmox-sample Managed proxmox-home Provisioning + # proxy-proxmox-sample Managed proxmox-home Running 192.168.4.147 + # proxy-proxmox-sample Managed proxmox-home Ready 192.168.4.147 True + ``` + + (Illustrative.) `Create` returns once the VM runs, a few seconds after + the clone; the address follows when the guest agent is up, about 50 + seconds after the start on `pv1-1` with the agent installed at first + boot. + +## Config reference + +The block is `proxmox:` and its Go type is `ProxmoxConfig` in +`internal/provider/config_proxmox.go`, validated at load by +`validateProxmox`; unknown keys are rejected. Defaults are applied by +`withDefaults` in `internal/provider/proxmox/proxmox.go`. + +| Field | Required | Default | Meaning | +|---|---|---|---| +| `url` | yes | | `https://:8006`, with no path (the client appends `/api2/json`). The host must be in the node certificate | +| `tokenFile` | yes | | file holding one line, `@!=`, read on every request and sent as `Authorization: PVEAPIToken=` | +| `caFile` | | system roots | PEM bundle trusted for the API's certificate, typically the node's `/etc/pve/pve-root-ca.pem`. Loaded on first use and cached | +| `insecureSkipTLSVerify` | | `false` | skip certificate verification; `New` logs a `WARNING:` line when it is on | +| `nodes` | yes, at least one | | node names `placement.zone` selects from; unique, lowercase letters, digits and dashes. They end up in request paths and as the clone's `target` | +| `isoStorage` | yes | | storage id the seeds are uploaded to and deleted from. Must hold `iso` content; the token needs `Datastore.Allocate` on it. No default on purpose | +| `storage` | | the template's disk storage | storage id the full clone's disks go to | +| `bridge` | | as cloned | replaces `bridge=` in the cloned `net0` | +| `vlanTag` | | as cloned | 1 to 4094; replaces or appends `tag=` in the cloned `net0` | +| `vmidRange.min`, `vmidRange.max` | | `100000`, `199999` | where VMIDs are allocated; at least 100, at most 999999999, min below max | +| `templates..vmid` | at least one template | | the key names the template in `placement.image`; the VMID is at least 100 | +| `machineTypes.` | at least one | | `{cores, memoryMiB, diskGiB}` size profile. `cores` and `memoryMiB` above zero. `diskGiB` is optional: when larger than the cloned boot disk the disk is grown to it; it is never shrunk | +| `requestTimeoutSeconds` | | `30` | bound on one API request, response included. The seed upload is one request, so a multi-megabyte `vendorData` needs more | +| `vendorData` | | | extra cloud-init vendor-data placed on every seed next to the Proxy's user-data; a MIME multipart part beside the agent script when `installGuestAgent` is on. Cloud-init drops any top-level key of a vendor-data cloud-config that the user-data also defines | +| `installGuestAgent` | | `true` | add the vendor-data script that installs and starts `qemu-guest-agent`. Turn it off for templates that ship the agent | + +## Placement + +All three catalogue fields are checked before any request. An unknown +value is `ErrPermanent`, which the reconciler latches as `Failed` with the +message below until the spec changes. + +| Proxy field | Meaning for proxmox | Error when unknown | +|---|---|---| +| `placement.zone` | a node from `nodes`. May be omitted when exactly one node is configured; otherwise required | `placement.zone "x" is not in proxmox.nodes`; with several nodes and no zone, `placement.zone is required when proxmox.nodes lists more than one node` | +| `placement.machineType` | a key in `machineTypes` | `placement.machineType "x" is not in proxmox.machineTypes` | +| `placement.image` | a key in `templates` | `placement.image "x" is not in proxmox.templates` | +| `placement.region` | ignored. It is still part of the spec hash, so changing it replaces the VM and does nothing else | | + +The messages come from `placement` in `internal/provider/proxmox/create.go` +and `node` in `proxmox.go`. A key that exists but names a VMID that is +missing or not a template fails later, before any clone: +`placement.image "x": template VM 9001 does not exist` or `... is not a +template`. + +**Catalogue edits affect only new VMs.** The spec hash covers the +placement values, not what the config maps them to, so repointing +`templates.debian-13.vmid` at another template changes the VMs created +afterwards and leaves the running ones alone. Add a new key rather than +repointing one in use, so the Proxies that name a key keep a meaning you +can reason about. A template may be edited, replaced or deleted without +affecting existing VMs: they are full clones and share nothing with it. + +## What lands on Proxmox + +For a Proxy whose deterministic name is `proxy-abc…`: + +| Object | Name | Lifetime | +|---|---|---| +| VM | name `proxy-abc…`, a VMID from `vmidRange` | with the Proxy | +| its disks | the template's disks, fully copied onto `storage` (or the template's storage) | destroyed with the VM (`purge=1&destroy-unreferenced-disks=1`) | +| seed ISO | `:iso/proxy-abc…-seed.iso` on the VM's node | with the Proxy | + +**The VMID** is derived from the Proxy's UID, so a retried Create lands on +the same number: an FNV-1a hash of the UID placed in `vmidRange`, moving +forward one at a time on collision, up to 32 probes (`vmidFor` in +`id.go`). Before allocating, `Create` looks the name up in the cluster's +resources. A VMID that is taken, or a VM that already has the name, goes +through an ownership check (`inspect` in `create.go`): a VM with a lock +(a clone still running, which also has no name yet) is waited on; then + +- another name: someone else's VM, so the next VMID is probed; +- our name and our mark with our UID or an empty one: **adopted**, and + Create carries on from where the previous attempt stopped; +- our name without the mark, or with another UID: **permanent error** + naming the VMID and both UIDs. The operator never takes over a VM it did + not create; +- our name, our mark, hibernated (`lock: suspended`): **permanent error**, + `resume or destroy it in Proxmox` (see *Known gaps*). + +Two VMs with the proxy's name are a permanent error naming every VMID; +remove all but one. + +**The providerID** is `vms//`. It names no node, because VMs +migrate: the node is looked up in `GET /cluster/resources` on every call. +`Get` and `Delete` check that the VM at that VMID still has that name and +the mark. Another name means ours is gone and the VMID was reused: `Get` +answers NotFound, `Delete` succeeds without touching it. + +**The mark** is three lines in the VM's description (Notes in the UI), +passed in the clone call itself, so a VM is never unmarked: + +``` +proxy-operator-managed=true +proxy-operator-uid=0f3a… +proxy-operator-created-at=2026-09-23T17:42:11Z +``` + +That is how orphan GC recognises its own VMs and how a repeat `Create` +knows whether a VM of the same name belongs to this Proxy. Lines an admin +adds to the description are kept and ignored. The clone's own +`meta: ctime=` is useless for this: Proxmox copies it from the template. + +**The config write.** After the clone and the seed upload, one +`PUT /nodes/{node}/qemu/{vmid}/config` sets: + +| Key | Value | +|---|---| +| `cores`, `memory` | from the machine type | +| `agent` | `1` | +| `tags` | `proxy_operator` | +| `ide2` | `:iso/-seed.iso,media=cdrom` | +| `net0` | only when `bridge` or `vlanTag` is set: the cloned value with `bridge=` and `tag=` replaced or appended, the model, MAC and every other option kept | + +The write **replaces the template's `agent` option and tags**: a +template's `agent: 1,fstrim_cloned_disks=1` becomes `agent: 1`, and its +tags give way to `proxy_operator`. The tag is for filtering in the +Proxmox UI and narrows GC's candidates; the mark decides. Proxmox appends +`ide2` to the boot order on its own (`order=scsi0;ide2`), which is +harmless because the boot disk stays first. When the machine type's +`diskGiB` is larger than the boot disk (the first disk in `boot: order=` +that is not a CD-ROM), a resize task grows it; cloud-init's growpart +expands the root filesystem at first boot. + +**The seed** (`nocloud.BuildSeedISO`, shared with kvm2 in +`internal/provider/nocloud`) is an ISO 9660 image labelled `cidata`. +`user-data` is the Proxy's `cloudInit` **byte for byte**; the operator +never parses or re-renders it. `meta-data` holds `instance-id: ` and +`local-hostname: `. `vendor-data` carries the guest-agent script +(a shell script, not a cloud-config, so a Proxy's own `packages:` cannot +shadow it), joined with `vendorData` as a MIME multipart when both are +set. It is staged under `/tmp` (the manager's emptyDir), built in memory +and uploaded with `POST /nodes/{node}/storage/{isoStorage}/upload`. An +upload overwrites a file of the same name, so it is rebuilt and +re-uploaded on every Create attempt with no delete first: a previous +attempt may have been cut off. Only the API is used; the storage upload +endpoint does not accept `snippets`, so Proxmox's own `cicustom` would +need shell access to the node. + +**What a retried Create finds.** Every step is safe to repeat. After a +crash mid-clone, the retry lands on the same VMID, waits for the clone's +lock, and adopts the VM. A stopped VM of ours gets the seed, the config +and the resize again (the upload overwrites, the config is a replace, the +resize is skipped once large enough), then the start. A running one is +returned at once. + +**A clone lock that never clears.** A node that crashes mid-clone can +leave the VM locked with `lock: clone` for good. Every Create for that +Proxy then waits out the 10 minute clone budget and returns a transient +error, and the reconciler keeps retrying rather than latching. Clear it +on the node: + +```sh +qm unlock # then let the next Create adopt and finish it +qm destroy --purge # or remove it and let Create clone again +``` + +**Seeds without a VM.** A VM destroyed outside the operator leaves its +seed ISO, which GC cannot see (it lists VMs). Seeds are named +`proxy-*-seed.iso` on `isoStorage`; any without a matching VM may be +removed by hand: + +```sh +pvesm list proxy-seeds --content iso +pvesm free proxy-seeds:iso/proxy-abc…-seed.iso +``` + +**Delete** runs under a per-name lock shared with Create: stop (a paused +VM too), destroy with `purge=1&destroy-unreferenced-disks=1`, then delete +the seed on the VM's node. Each is a task that is waited on. A VM that is +already gone, not ours, or unmarked is left alone, and the seed is then +looked for on every configured node, where a node that fails is logged +and skipped. On the VM's own node a failed seed delete fails the call: +**without `Datastore.Allocate` on `isoStorage` the 403 makes Delete +permanent**, and the Proxy's deletion stays stuck with that message until +the privilege is granted. When Delete returns nil, nothing of the Proxy +is left. + +## Logging + +The provider logs through controller-runtime's logger, named `proxmox`, +with `provider` (the configured name) on every line, and `vmid`, `name` +and `node` on every line about an instance. Every line inside a traced +operation also carries `traceID` and `spanID`. + +- At startup (always on): `WARNING: token file is not readable; every + call fails until it is` with `path` and `reason`, and `WARNING: + insecureSkipTLSVerify is set`. +- `--zap-log-level=debug` (logr V(1)): + - one line per operation outcome: `proxmox vm ensured running`, + `proxmox vm adopted` with `status` and `qmpstatus`, + `proxmox vm fetched` with `state`, `ip`, `uid`, `status` and + `qmpstatus`, `proxmox vm not found` with `reason`, + `proxmox vm deleted` with `destroyed` (false when no VM of ours was + left to destroy), `proxmox vms listed` with `vms`, `candidates`, + `managed` and `skipped` counts, and `proxmox node checked` with + `ready`, `reason` and `latency`; + - **one `proxmox api call` line per API request**, written by the + decorator in `internal/provider/proxmox/observe.go`: `call` (the seam + method, for example `Clone`), the identifiers it was given (`node`, + `vmid`, `storage`), `upid` for a task, `newid` and `name` on a clone, + `filename` on an upload, `volid` on a volume delete, `duration`, and + on failure `httpStatus` and `error`. A running `Get` is four calls; a + fresh Create is a dozen or more, depending on how many polls each + task takes; + - `proxmox task failed` with `upid` and `exitstatus`, and + `proxmox task finished with warnings`; + - the no-op lines: `proxmox vm already gone`, + `proxmox vm at this vmid is not ours, left alone` with `vmName`, + `proxmox vm carries no proxy-operator mark, left alone`, + `proxmox seed delete failed on a node, skipped`, + `skipping unreadable proxmox vm`, and the probe's + `proxmox vmid taken by another vm, probing on`. +- `--zap-log-level=2`: `proxmox create resolved` with the resolved + parameters (node, machine type, cores, memory, disk, template VMID, + storages, bridge, VLAN tag) and `cloudInitBytes`; the `proxmox api + call` lines also carry the config write's parameters, a summary of each + config read, the clone's description, storage and target, the resize + size, VM and task status, the agent's addresses and the upload's byte + count; `proxmox guest agent answered`; `proxmox listed instance` per + listed VM. +- Never logged at any level: the cloud-init content, the token, the CA + content and the upload body. There is no raw wire tier. + +## Tracing + +With an OTLP exporter configured ([architecture.md +§10](architecture.md#10-tracing-internaltracing)), every API request is a +child span of the operation span the decorators open (`provider.create`, +`provider.get`, `provider.delete`, `provider.list`) or of the monitor's +`providerhealth.check`: + +| Span | Around | Attributes | +|---|---|---| +| `HTTP GET`, `HTTP POST`, `HTTP PUT`, `HTTP DELETE` | one API request, from `otelhttp` on the client's transport | otelhttp's standard HTTP client attributes: method, URL, server address and port, response status | + +The transport is `tracing.RestConfigWrapper()`, the same parent-gated +wrapper the kubernetes provider uses, so a request without a parent span +starts none. The logging decorator starts no span of its own, so nothing +is counted twice; its `proxmox api call` line carries the operation +span's `spanID`, not the HTTP span's. The token travels in a header, +which otelhttp does not record, and no attribute carries cloud-init. + +## Timeouts + +No caller puts a deadline on the context, so every request and every wait +carries its own budget (`defaultBudgets` in +`internal/provider/proxmox/proxmox.go`, and the transport in `client.go`): + +| Bound | Budget | +|---|---| +| TCP connect | 10 s | +| TLS handshake | 10 s | +| one request, response included | `requestTimeoutSeconds`, default 30 s | +| clone task, and the wait for a VM's lock in the ownership check | 10 m | +| upload task, resize task, start task, stop task | 2 m each | +| destroy task | 5 m | +| seed delete task | 1 m | +| wait for `running` after the start | 60 s | +| task and status poll interval | 1 s | + +A task wait that runs out is a transient error naming the UPID; the task +keeps running on Proxmox, and the next attempt finds its result through +the ownership check or the lock. A task that ends with any `exitstatus` +other than `OK` or `WARNINGS: ` is transient too, with Proxmox's text. + +Requests share one HTTP client and its idle connections (4 per host, +closed after 90 s idle). A cancelled or overlong request ends only +itself: nothing else in flight is disturbed, and there is no connection +to kill as there is for kvm2. HTTP/2 is disabled, because pveproxy speaks +HTTP/1.1 only. Create and Delete for the same name are serialised on a +per-name lock, so the reconciler's Delete and GC's Delete never race on +stop and destroy; Get and ListByTag take no lock. + +## Health check + +The operator checks every configured node every +`--provider-check-interval` (default 1m) and publishes the result in the +`ProviderHealth` object named after the provider +(`kubectl get ph proxmox-home -o yaml`) and in the +`proxy_operator_provider_target_up{provider,target}` gauge. One target per +`nodes` entry, named after the node with the URL as its address, in +config order, all checked concurrently. + +Each check (`checkNode` in `internal/provider/proxmox/check.go`) first +reads the token file for its id, then makes four requests on the shared +client: `GET /version`, `GET /nodes/{node}/status`, +`GET /nodes/{node}/storage/{isoStorage}/status` (the storage must exist, +be active and hold `iso` content), and `GET /cluster/resources?type=vm` +(every configured template must exist and be a template). A check that +runs out of time cancels only its own requests. + +Every target's `details` carry `isoStorage`. As facts arrive they gain +`tokenID` (the part of the token before `=`, never the secret), +`pveVersion`, `kernel`, `isoStorageAvailGiB` and `templates` +(`/`), so a target that fails late still says what the +node answered before. There is no quota signal; the storage's free space +is here instead. + +| Reason | Meaning | Look at | +|---|---|---| +| `Reachable` | ready | | +| `CredentialsUnavailable` | the token file is missing or not `@!=`; no request was made | the `proxmox-token` Secret; fixed in the cluster, not in Proxmox | +| `CertificateInvalid` | TLS verification failed, or `caFile` is missing or holds no certificate | `url` (its host must be in the node certificate) or `caFile`; not the node | +| `Unreachable` | the TCP connect or TLS handshake failed for a network reason, the connection was cut mid-response, or pveproxy answered 595 or 596 (it cannot reach the node behind it) | routing and firewalls between the cluster and `url`; `pveproxy` running; the node up | +| `Unauthorized` | 401: the token id or secret is wrong | the token file; `pveum user token list proxy-operator@pve` | +| `Forbidden` | 403; the message names the path and the privilege | the roles in *Proxmox prerequisites* | +| `Timeout` | the monitor's `--provider-check-timeout` (default 15s) ran out, whichever request was running, or a request ran out of `requestTimeoutSeconds` while the round still had time | a slow or hung pveproxy, a saturated link | +| `StorageNotFound` | `isoStorage` answered with a 500 on that node (it does not exist, is not enabled there, or is offline), is not active, or holds no `iso` content; the message carries Proxmox's text | `pvesm status`; the storage's nodes and content types | +| `TemplateNotFound` | a configured template VMID does not exist or is not a template | `qm list`; the `templates` catalogue | +| `APIError` | any other failure | the message carries Proxmox's own text | + +The check does not look inside the templates: a cloud-init drive, a +missing `net0` or an unreadable disk size surface only when a Create is +about to clone. + +## Error classes + +Every error is classified once, in `classify` in +`internal/provider/proxmox/errors.go`, with errors that carry a class from +their source (validation, the token and CA files, the ownership check) +keeping it. The reconciler latches `ErrPermanent` as `Failed` until the +spec changes, recreates on `ErrNotFound` from `Get`, and retries +everything else with backoff. The provider never returns +`ErrQuotaExceeded`: the thin `local-lvm` storage overprovisions, so no +out-of-space answer could be produced to match, and capacity failures are +transient. + +| Class | What lands there | +|---|---| +| `ErrPermanent` | placement values not in the catalogue; a template VMID that is missing or not a template, has a cloud-init drive, or lacks `net0` or a sized boot disk when the config needs them; a malformed providerID; any 4xx (401 `Authentication failed!` or `No ticket`: the token; 403 `Permission check failed (, )`: a missing privilege, including `Datastore.Allocate` for the seed delete; 400 `Parameter verification failed.` with the field errors); a TLS verification failure; a token file that exists but is not `@!=`; a CA file with no certificate; a VM with the proxy's name that carries no mark or another UID; two VMs with the proxy's name; our VM hibernated (`lock: suspended`), in both Create and Delete | +| `ErrNotFound` | from `Get`: the VMID absent from the cluster's resources, the 500 `Configuration file 'nodes//qemu-server/.conf' does not exist`, the VM at the VMID now named differently or unmarked. Inside `Create` the `does not exist` answer is `ErrTransient`, because the contract forbids NotFound from Create; `Delete` treats all of these as done | +| `ErrTransient` | everything else: a missing token or CA file (the Secret has not landed yet); network errors and cancellation; 500 `VM is locked ()` and any lock seen in the resources or config; a VM whose node does not answer (`status: unknown`); the agent's answers that `Get` cannot place, such as `VM is not running`; any other 5xx, including pveproxy's 595 and 596; a task that fails or runs out of its budget; no free VMID after 32 probes; the wait for running expiring, or a VM that stops right after its start | + +`Get` turns two agent answers into states rather than errors: +`QEMU guest agent is not running` is Provisioning (booting, or the agent +install has not finished), and `No QEMU guest agent configured` is +Stopped (someone removed `agent: 1`; the reconciler replaces the VM). + +Error strings name VMIDs, nodes, storages, UPIDs and Proxmox's own text. +They never contain the token, the CA content or cloud-init. + +## Known gaps + +- **Create blocks** until the VM runs, holding one of the three reconcile + workers for its duration. On `pv1-1` that is about five seconds plus the + clone; large templates on slow storage take longer. Other Proxies keep + reconciling on the remaining workers. +- **A clone lock that never clears** (a node crash mid-clone) makes every + Create for that Proxy wait out the 10 minute clone budget and return a + transient error. Clear it with `qm unlock ` or destroy the VM (see + *What lands on Proxmox*). +- **Seed ISOs of VMs destroyed outside the operator are invisible to + GC.** Remove `proxy-*-seed.iso` files without a matching VM by hand. + On a storage that is not shared, a VM migrated to another node leaves + its seed on the original node; Delete then looks on the VM's current + node, finds nothing, and the seed stays. Shared ISO storage avoids it. +- **The VMID range is shared with humans.** A VM created by hand inside + it costs a probe; a hand-made VM named like a proxy is refused as + `ErrPermanent`. Reserve the range. +- **Cross-node clones need shared storage and are untested.** When the + zone's node differs from the template's, the clone is sent with + `target`, which Proxmox allows only when the template's disks are on + shared storage. The host this was built on has one node. +- **A VM whose agent was switched off is replaced**, not repaired: with + `agent: 1` removed it can never report an address, so `Get` reports it + Stopped. +- **The proxy network needs internet** for the agent install, unless the + template ships the agent and `installGuestAgent` is `false`. Without it + the VM boots, gets an address, and stays `Provisioning` forever, + because the agent is the only address source. Check + `systemctl status qemu-guest-agent` and + `/var/log/cloud-init-output.log` on the guest (`qm terminal ` on + the node, or the console in the UI). +- **The CA is cached** after first use; rotating it needs a manager + restart. The token rotates freely. +- **A node that stops answering** makes `Get` and `Delete` for its VMs + transient until it returns, so deleting such a Proxy waits. Proxmox + lists an unreachable node's guests with `status: unknown` rather than + dropping them, so they are never mistaken for gone. That is Proxmox's + documented behaviour; a single-node host could not show it. +- **Contract limits that apply unchanged** ([provider-contract.md + §13](provider-contract.md)): the providers config is read once per + process; GC has no installation identity, so two operators sharing a + Proxmox cluster and a VMID range delete each other's VMs; and a Proxy + naming an unconfigured provider latches until its spec changes. + +Beyond the ten limits accepted in the design: + +- **A hibernated VM latches its Proxy.** A VM of ours with + `lock: suspended` (hibernated to disk) makes both Create and Delete fail + permanently with `resume or destroy it in Proxmox`: the lock never + clears on its own, and the destroy refuses a locked VM. `Get` reports it + Stopped. Resume it by starting it (`qm start ` restores the + hibernated state) or destroy it on the node, then touch the Proxy's + spec, or let the pending Delete retry. +- **The seed upload is one request**, bounded by `requestTimeoutSeconds`. + A seed is tens of KiB, but a multi-megabyte `vendorData` would need a + larger timeout. +- **The sample cloud-init is an open proxy.** + `config/samples/proxy_proxmox.yaml` installs Squid with + `http_access allow all`; restrict it before exposing the proxy network + to anything but the cluster. +- **Not yet proven end to end.** The unit tests run over captured + responses from `pv1-1` and an in-memory fake; the integration tests + below compile but have not run against a host yet. Three answers were + never captured: the storage status for a missing storage, a destroy on + a hibernated VM, and a stop on a paused VM. + +## Testing + +Unit tests run over an in-memory fake of the API seam, and the HTTP +client over a TLS test server replaying the responses captured from +`pv1-1` (`internal/provider/proxmox/testdata/`). The optional integration +test against a real host is described in +[testing.md](testing.md#the-proxmox-integration-test-real-proxmox-ve-host), +which also lists the variables it needs; run it with +`make test-integration-proxmox`. `TestProxmoxCheck_realHost` runs the +health check against the same host and creates nothing: +`go test -tags=integration -count=1 -v -run TestProxmoxCheck ./internal/provider/proxmox/`. diff --git a/internal/provider/config_proxmox_test.go b/internal/provider/config_proxmox_test.go index d623928..8585fdd 100644 --- a/internal/provider/config_proxmox_test.go +++ b/internal/provider/config_proxmox_test.go @@ -1,8 +1,12 @@ package provider import ( + "os" + "slices" "strings" "testing" + + "sigs.k8s.io/yaml" ) const validProxmoxYAML = ` @@ -130,3 +134,117 @@ machineTypes: {small: {cores: 1, memoryMiB: 1024}} }) } } + +// commentedEntry cuts the commented-out providers entry named name out of a +// config file and strips its "# " at the entry's own indent, so a sample +// that is shipped commented out is still loaded by a test. +func commentedEntry(t *testing.T, path, name string) string { + t.Helper() + data, err := os.ReadFile(path) + if err != nil { + t.Fatalf("reading %s: %v", path, err) + } + var out []string + var prefix string + for line := range strings.SplitSeq(string(data), "\n") { + if prefix == "" { + if strings.TrimSpace(line) == "# - name: "+name { + prefix = line[:strings.Index(line, "#")] + out = append(out, prefix+"- name: "+name) + } + continue + } + if !strings.HasPrefix(line, prefix+"# ") { + break + } + out = append(out, prefix+strings.TrimPrefix(line, prefix+"# ")) + } + if prefix == "" { + t.Fatalf("%s has no commented entry %q", path, name) + } + return strings.Join(out, "\n") + "\n" +} + +func TestLoadConfig_proxmoxSampleEntry(t *testing.T) { + t.Parallel() + for name, path := range map[string]string{ + "sample": "../../config/samples/providers-config.yaml", + "configmap": "../../config/manager/providers_config.yaml", + } { + t.Run(name, func(t *testing.T) { + t.Parallel() + cfg, err := LoadConfig([]byte("providers:\n" + commentedEntry(t, path, "proxmox-home"))) + if err != nil { + t.Fatalf("LoadConfig(proxmox-home from %s) error = %v, want nil", path, err) + } + p := cfg.Providers[0] + // LoadConfig already refused a proxmox block under another type. + if p.Name != "proxmox-home" || p.Proxmox == nil { + t.Fatalf("provider %q has proxmox block %v, want proxmox-home with one", p.Name, p.Proxmox) + } + c := p.Proxmox + if c.URL != "https://192.168.0.8:8006" { + t.Errorf("url = %q, want https://192.168.0.8:8006 (the address in the node certificate)", c.URL) + } + if c.TokenFile != "/etc/proxy-operator/proxmox/token" || c.CAFile != "/etc/proxy-operator/proxmox/ca.pem" { + t.Errorf("tokenFile, caFile = %q, %q, want the proxmox-token Secret's mount paths", c.TokenFile, c.CAFile) + } + if c.InsecureSkipTLSVerify { + t.Error("insecureSkipTLSVerify = true, want it left commented out") + } + if len(c.Nodes) != 1 || c.Nodes[0] != "pv1-1" { + t.Errorf("nodes = %v, want [pv1-1]", c.Nodes) + } + if c.ISOStorage != "proxy-seeds" || c.Storage != "" || c.Bridge != "" || c.VLANTag != 4 { + t.Errorf("isoStorage, storage, bridge, vlanTag = %q, %q, %q, %d, want proxy-seeds, \"\", \"\", 4", c.ISOStorage, c.Storage, c.Bridge, c.VLANTag) + } + if len(c.Templates) != 1 || c.Templates["debian-13"].VMID != 9001 { + t.Errorf("templates = %v, want {debian-13: {vmid: 9001}}", c.Templates) + } + if mt, ok := c.MachineTypes["small"]; !ok || mt.Cores != 1 || mt.MemoryMiB != 1024 || mt.DiskGiB != 0 { + t.Errorf("machineTypes[small] = %+v (present %v), want {1 1024 0}", mt, ok) + } + if c.VMIDRange != nil || c.RequestTimeoutSeconds != 0 || c.InstallGuestAgent != nil || c.VendorData != "" { + t.Errorf("optional fields set: %+v, want vmidRange, requestTimeoutSeconds, installGuestAgent and vendorData left commented out", c) + } + }) + } +} + +func TestLoadConfig_proxmoxSampleProxyMatchesSampleEntry(t *testing.T) { + t.Parallel() + cfg, err := LoadConfig([]byte("providers:\n" + commentedEntry(t, "../../config/samples/providers-config.yaml", "proxmox-home"))) + if err != nil { + t.Fatalf("LoadConfig() error = %v, want nil", err) + } + data, err := os.ReadFile("../../config/samples/proxy_proxmox.yaml") + if err != nil { + t.Fatalf("reading the sample Proxy: %v", err) + } + var px struct { + Spec struct { + Provider string `json:"provider"` + Placement struct { + Zone string `json:"zone"` + MachineType string `json:"machineType"` + Image string `json:"image"` + } `json:"placement"` + } `json:"spec"` + } + if err := yaml.Unmarshal(data, &px); err != nil { + t.Fatalf("parsing the sample Proxy: %v", err) + } + c := cfg.Providers[0].Proxmox + if px.Spec.Provider != cfg.Providers[0].Name { + t.Errorf("spec.provider = %q, want %q", px.Spec.Provider, cfg.Providers[0].Name) + } + if !slices.Contains(c.Nodes, px.Spec.Placement.Zone) { + t.Errorf("placement.zone = %q, not in nodes %v", px.Spec.Placement.Zone, c.Nodes) + } + if _, ok := c.MachineTypes[px.Spec.Placement.MachineType]; !ok { + t.Errorf("placement.machineType = %q, not in machineTypes", px.Spec.Placement.MachineType) + } + if _, ok := c.Templates[px.Spec.Placement.Image]; !ok { + t.Errorf("placement.image = %q, not in templates", px.Spec.Placement.Image) + } +} From 3d725078947256cf9ba3e4e242f78dcdb41103d7 Mon Sep 17 00:00:00 2001 From: kacerr Date: Thu, 24 Sep 2026 10:28:28 +0000 Subject: [PATCH 13/13] proxmox: review fixes, real-host validation record, changelog Phase 10. Two review passes (go-operator-reviewer, /code-review high) over the branch, recorded in docs/reviews/2026-09-23-proxmox-provider-review/: 9 findings fixed, 3 accepted as documented limits, 7 rejected with a reason. - Delete removes the seed between the stop and the destroy, so a failed seed delete leaves the VM for the next Delete instead of leaking the ISO; a Permanent refusal on the every-node path fails the call. - Create skips a probe VMID whose listed name is someone else's without waiting on its lock. - The three answers Phase 0 never captured are fixtures now: a missing storage is a 400 (the Checker reports StorageNotFound), a hibernated VM is destroyable (Delete destroys ours), a paused VM stops. - TestProxmox_lifecycle and TestProxmoxCheck_realHost passed on pv1-1 (PVE 9.2.20); the docs record the run and drop "not yet run". Co-Authored-By: Claude Opus 5.5 (1M context) --- CHANGELOG.md | 19 ++ README.md | 2 +- docs/architecture.md | 5 +- .../2026-09-18-1502-proxmox-provider.md | 143 ++++++++- docs/providers-existing.md | 40 +-- docs/proxmox-provider-spec.md | 126 +++++--- docs/proxmox-provider.md | 73 +++-- .../REPORT.md | 286 ++++++++++++++++++ docs/testing.md | 47 ++- internal/provider/proxmox/check.go | 18 +- internal/provider/proxmox/check_test.go | 16 +- internal/provider/proxmox/create.go | 9 +- internal/provider/proxmox/create_test.go | 20 +- internal/provider/proxmox/delete.go | 63 ++-- internal/provider/proxmox/delete_test.go | 82 +++-- internal/provider/proxmox/errors.go | 21 +- internal/provider/proxmox/errors_test.go | 8 +- internal/provider/proxmox/fake_test.go | 41 ++- internal/provider/proxmox/get.go | 4 +- internal/provider/proxmox/get_test.go | 6 +- internal/provider/proxmox/provider_test.go | 9 +- internal/provider/proxmox/state.go | 15 +- .../testdata/destroy_hibernated_response.json | 5 + .../status_current_after_stop_paused.json | 5 + .../testdata/stop_paused_response.json | 5 + .../testdata/storage_status_missing.json | 5 + .../task_status_destroy_hibernated.json | 5 + .../testdata/task_status_stop_paused.json | 5 + 28 files changed, 882 insertions(+), 201 deletions(-) create mode 100644 docs/reviews/2026-09-23-proxmox-provider-review/REPORT.md create mode 100644 internal/provider/proxmox/testdata/destroy_hibernated_response.json create mode 100644 internal/provider/proxmox/testdata/status_current_after_stop_paused.json create mode 100644 internal/provider/proxmox/testdata/stop_paused_response.json create mode 100644 internal/provider/proxmox/testdata/storage_status_missing.json create mode 100644 internal/provider/proxmox/testdata/task_status_destroy_hibernated.json create mode 100644 internal/provider/proxmox/testdata/task_status_stop_paused.json diff --git a/CHANGELOG.md b/CHANGELOG.md index 6ca1b6a..3857a63 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -1,5 +1,24 @@ # Changelog +## 2026-09-24 10:28 UTC — proxmox provider: reviewed, fixed, proven on a real Proxmox VE 9.2 host + +The `proxmox` provider (VMs cloned from a template on Proxmox VE through its REST API) +finished its last phase: a two-pass review, a real-host run on pv1-1, and captures of the +three API answers the design had only guessed. + +- Delete now removes the seed ISO **before** destroying the VM. Before, a seed delete that + failed after the destroy was never retried (the next Get is NotFound), so the ISO leaked. + A 403 for `Datastore.Allocate` now fails Delete on every path instead of being skipped. +- Create skips a probe VMID held by someone else's VM at once, instead of waiting up to 10 + minutes on that VM's lock on every retry. +- The captures corrected two assumptions: a missing ISO storage is a 400 (the health check + now says `StorageNotFound`, not `APIError`), and a hibernated VM can be destroyed (Delete + destroys ours instead of latching the Proxy). +- `TestProxmox_lifecycle` and `TestProxmoxCheck_realHost` passed on PVE 9.2.20: Create 6.2 s, + an address 50 s later. A kind pass took a sample Proxy to Ready and back. +- Review record: `docs/reviews/2026-09-23-proxmox-provider-review/REPORT.md`. Timings: + `docs/testing.md`. Code: `internal/provider/proxmox/{delete,create,check,errors}.go`. + ## 2026-09-18 07:52 UTC — docs/operating.md, starting with the libvirt-RPC log query New home for day-to-day helpers against a running operator, seeded with the `jq` query diff --git a/README.md b/README.md index 80cbc65..4e43dbf 100644 --- a/README.md +++ b/README.md @@ -192,7 +192,7 @@ can be two differently-configured instances of the same `gcp` type (see | `gcp` | A Compute Engine VM with an ephemeral external IP | `project` (required), `network`, `networkTag`, `diskSizeGb` | The real egress fleet. Deliberately uses only four API calls (Insert / Get / Delete / AggregatedList), all fire-and-forget: `Create` returns as soon as the operation is submitted and the reconciler discovers progress by polling `Get`. Auth is Application Default Credentials — workload identity in-cluster, `gcloud` ADC locally; no key-file plumbing. | | `kvm` | A libvirt VM on one of your own hypervisors, reached over SSH | `sshKeyFile`, `knownHostsFile` (unless `insecureIgnoreHostKey`), `machineTypes`, `hosts` (required); `images`, `diskSizeGb`, `vendorData`, `installGuestAgent` | Home-lab or on-prem egress. The base image is downloaded into each host's pool on first use; `Create` waits for the VM to start. See [docs/kvm-provider.md](docs/kvm-provider.md). | | `kvm2` | The same: a libvirt VM on one of your own hypervisors, reached over SSH | the same block shape as `kvm`, under `kvm2:` | The second-generation libvirt provider, built from the provider contract. Adds a child span per libvirt RPC, a pool-space precheck reported as quota (the reconciler waits instead of failing), SSH credentials re-read on every dial, and image downloads and health checks on connections of their own. It does not adopt `kvm`'s VMs. See [docs/kvm2-provider.md](docs/kvm2-provider.md). | -| `proxmox` | A VM on a Proxmox VE cluster: a full clone of a template you prepare, reached through the Proxmox REST API with an API token | `url`, `tokenFile`, `nodes`, `isoStorage`, `templates`, `machineTypes` (required); `caFile` or `insecureSkipTLSVerify`, `storage`, `bridge`, `vlanTag`, `vmidRange`, `requestTimeoutSeconds`, `vendorData`, `installGuestAgent` | Home-lab or on-prem egress without SSH. Cloud-init goes on a seed ISO uploaded through the API; the address comes from the QEMU guest agent, so the proxy network needs a route to a package mirror unless the template ships the agent. `Create` waits for the VM to run. The token is read on every request, from the optional `proxmox-token` Secret (`make proxmox-token-secret`). Built from the provider contract; not yet run against a real host end to end. See [docs/proxmox-provider.md](docs/proxmox-provider.md). | +| `proxmox` | A VM on a Proxmox VE cluster: a full clone of a template you prepare, reached through the Proxmox REST API with an API token | `url`, `tokenFile`, `nodes`, `isoStorage`, `templates`, `machineTypes` (required); `caFile` or `insecureSkipTLSVerify`, `storage`, `bridge`, `vlanTag`, `vmidRange`, `requestTimeoutSeconds`, `vendorData`, `installGuestAgent` | Home-lab or on-prem egress without SSH. Cloud-init goes on a seed ISO uploaded through the API; the address comes from the QEMU guest agent, so the proxy network needs a route to a package mirror unless the template ships the agent. `Create` waits for the VM to run. The token is read on every request, from the optional `proxmox-token` Secret (`make proxmox-token-secret`). Built from the provider contract and run end to end against Proxmox VE 9.2.20. See [docs/proxmox-provider.md](docs/proxmox-provider.md). | ### Adding a provider diff --git a/docs/architecture.md b/docs/architecture.md index b8f3ece..6251bcb 100644 --- a/docs/architecture.md +++ b/docs/architecture.md @@ -226,8 +226,9 @@ prov.Get ──► /cluster/resources (absent = NotFound; lock or node unknow + agent network-get-interfaces (first routable IPv4; agent not up yet = Provisioning) prov.Delete ──► resources + config: gone, not ours or unmarked = leave it; - hibernated = Permanent; stop → destroy (purge) → delete the seed - on the VM's node (or on every node when no VM of ours was left) + stop (unless stopped or hibernated) → delete the seed on the + VM's node → destroy (purge); with no VM of ours left, the seed + is deleted on every node prov.ListByTag ─► one /cluster/resources for the whole cluster, one config read per candidate (tag or proxy- name); the description's mark decides Check ──► per node: /version, node status, isoStorage status, templates in diff --git a/docs/plans-executions/2026-09-18-1502-proxmox-provider.md b/docs/plans-executions/2026-09-18-1502-proxmox-provider.md index aca7c8d..482aa94 100644 --- a/docs/plans-executions/2026-09-18-1502-proxmox-provider.md +++ b/docs/plans-executions/2026-09-18-1502-proxmox-provider.md @@ -14,7 +14,7 @@ Pairs with [docs/plans/2026-09-18-1502-proxmox-provider.md](../plans/2026-09-18- - [x] Phase 7 — Delete, Checker, CRD comment - [x] Phase 8 — Contract and observability sweep, integration test, testing.md - [x] Phase 9 — Deployment and documentation -- [ ] Phase 10 — Review, real-host validation, changelog, MR +- [x] Phase 10 — Review, real-host validation, changelog, MR ## Phase 0 — Plan file, branch, host recon, fixtures @@ -1298,3 +1298,144 @@ Worth noting: guide's "Not yet proven end to end" gap and the README's matching clause come out once they pass. Then the CHANGELOG entry after the user confirms, and the MR. + +## Phase 10 — Review, real-host validation, changelog, MR + +Where the plan's Phase 10 text and the spec disagreed, the spec won, as in +every phase since Phase 1. The host is pv1-1 at `https://192.168.0.8:8006`, +not 192.168.2.8, the template is 9001 and the ISO storage `proxy-seeds`. + +**Review.** Two read-only passes over `git diff main...HEAD`: the +`go-operator-reviewer` agent, briefed with the spec and the contract +(P1 to P6, S1 to S3), then `/code-review high` on the branch (CR1 to +CR10). Of the 19 findings (two found by both passes), 9 were fixed, 3 +accepted as documented limits and 7 rejected with a reason, all recorded in +[docs/reviews/2026-09-23-proxmox-provider-review/REPORT.md](../reviews/2026-09-23-proxmox-provider-review/REPORT.md). +The one that mattered was CR1/P1. Delete destroyed the VM and then deleted +the seed, so a failed seed delete left a seed nothing would ever remove: +the next `Get` is NotFound, the finalizer goes, and GC lists only VMs. +Delete now removes the seed between the stop and the destroy, so the VM +`Get` looks at goes last. A 403 on the every-node path now fails instead +of being skipped. P2 (CR3) was the other code change: a probe VMID whose +listed name is someone else's is skipped at once, instead of waiting up to +the clone budget on its lock on every retry. + +**Lint.** `make lint` still panics inside staticcheck on Go 1.27, so, as +in Phases 8 and 9: + +```bash +bin/golangci-lint run --disable staticcheck --max-same-issues 0 --max-issues-per-linter 0 ./internal/provider/proxmox/... ./internal/provider/ +bin/golangci-lint run --build-tags integration --disable staticcheck --max-same-issues 0 --max-issues-per-linter 0 ./internal/provider/proxmox/... ./internal/provider/ +``` + +Each run was diffed against the pre-fix baseline by file and message, +with line numbers and occurrence counts stripped. Three new goconst hits +came in along the way, and all three were fixed rather than accepted. The +P2 test's lock value made `"backup"`, then `"migrate"`, a repeated +literal; it now uses `"snapshot-delete"`. The fake's missing-storage error +repeated `"Parameter verification failed."`; it is now built from the raw +captured body. Final counts: goconst 224 (231 with the tag), logcheck 12, +modernize 2, unparam 3, gocyclo 1. + +**Real host.** The user ran the host checklist on pv1-1 and then: + +```bash +export PROXMOX_TEST_URL=https://192.168.0.8:8006 +export PROXMOX_TEST_TOKEN_FILE=$HOME/.config/proxmox-token +export PROXMOX_TEST_CA_FILE=$HOME/.config/proxmox-ca.pem +export PROXMOX_TEST_NODE=pv1-1 +export PROXMOX_TEST_TEMPLATE_VMID=9001 +export PROXMOX_TEST_ISO_STORAGE=proxy-seeds +export PROXMOX_TEST_VLAN_TAG=4 +make test-integration-proxmox +# --- PASS: TestProxmox_lifecycle (59.68s) +# --- PASS: TestProxmoxCheck_realHost (0.13s) +``` + +It ran the review fixes, uncommitted, on the first try. Clone task about +3 s, Create 6.2 s, address 50.2 s after Create returned, Delete plus Get +plus a second Delete 3.1 s. The Delete deleted the seed while the stopped +VM's `ide2` still referenced it (imgdel `OK`), then destroyed the VM, and +step 10 found neither left. That settles the one assumption behind the +reorder that the fake could not prove. The run confirmed P6 as well: the +first `Get`, about 6 s after the clone, met `status: unknown` in the +listing and answered Transient, blaming the node, which was up. The +message now names both causes. + +**The three answers Phase 0 never captured.** Captured with a scratch +helper that passes the token to curl through a process substitution and +writes the Phase 0 fixture shape. Its first offline test caught a bug: +Python's text mode folds `\r\n`, so the header split found nothing, and +the file is now opened with `newline=""`. The calls were: + +```bash +pvecap storage_status_missing GET /nodes/pv1-1/storage/proxy-missing/status +pve POST /nodes/pv1-1/qemu/9001/clone -d newid=199999 -d name=proxy-reconaaaaaaaaaaa -d full=1 +pve POST /nodes/pv1-1/qemu/199999/status/start +pve POST /nodes/pv1-1/qemu/199999/status/suspend # pause +pvecap stop_paused_response POST /nodes/pv1-1/qemu/199999/status/stop +pvecap task_status_stop_paused GET /nodes/pv1-1/tasks//status +pvecap status_current_after_stop_paused GET /nodes/pv1-1/qemu/199999/status/current +pve POST /nodes/pv1-1/qemu/199999/status/start +pve POST /nodes/pv1-1/qemu/199999/status/suspend -d todisk=1 # hibernate: stopped, lock suspended +pvecap err_destroy_hibernated DELETE '/nodes/pv1-1/qemu/199999?purge=1&destroy-unreferenced-disks=1' +pvecap task_status_destroy_hibernated GET /nodes/pv1-1/tasks//status +``` + +Two of the three contradicted the code: + +- A missing storage is a **400** with the field error + `storage: No such storage.`, not a 500, so the Checker reported + `APIError`. `isNoSuchStorage` matches that exact field error, and a 500 + is still taken as missing (not captured). The matcher matrix caught a + first version that fired on any `storage` field error, which a clone + onto a storage without image support also has. +- The destroy **accepts a hibernated VM**. Its task ended `OK`, removing + both disks and the `vm-199999-state-suspend-…` volume. Delete refused + such a VM of ours as Permanent, believing the destroy would, so it now + destroys it like a stopped one. Create's adoption still refuses one it + finds before it was ever recorded. The capture was renamed + `destroy_hibernated_response.json`, because the client test globs + `err_*.json` as errors. +- A stop on a paused VM ends `OK` with the VM stopped, as assumed. + +The destroy also cleaned up the scratch VM, and a listing of local-lvm +content for VMID 199999 came back empty. + +**Kind pass.** The user built and loaded the image, deployed, applied a +scratch providers config with only `kubernetes` and `proxmox-home`, ran +the sample Proxy to Ready, ran a `curl -x` through it, deleted it and +checked pv1-1 for leftovers. Everything passed. The first attempt at the +override went to a ConfigMap named `providers-config`. `config/default`'s +`namePrefix` makes the mounted one `egress-proxies-operator-providers-config`, +so the pod kept the shipped config: no `proxmox-home` ProviderHealth, and +`kvm-home` and `kvm2-home` failing on the missing SSH key. The corrected +command replaced the prefixed ConfigMap and deleted the stray one. The +Secret keeps its plain name (`proxmox-token`), since kustomize renames +only resources it builds. `make deploy`'s image edit to +`config/manager/kustomization.yaml` was reverted, not committed. + +Worth noting: + +- **Behaviour changes from Phase 9.** Delete's order is now stop, seed, + destroy. A hibernated VM of ours is destroyed rather than latched. The + Checker reports a missing ISO storage as `StorageNotFound`. A + foreign-named VM on a probe VMID is skipped without a config read. +- **Accepted, not fixed** (guide Known gaps, spec entry 24): + - P3: Create adopts a leftover stopped VM whatever the current + `placement` says. + - P4: a retry can clone a second VM in a narrow crash window. +- **Still unproven:** + - A storage that exists but cannot be used on a node. + - Every cross-node path. The host has one node. +- **Flakes.** kvm's `TestKeepalive_keepsHealthyClientOpen` failed once + under `-race` ("unexpected packet in response to channel open") and + passed 5 of 5 on rerun. This branch does not touch `internal/provider/kvm`. + The parked kvm2 items (S5.1, the 200 ms `TestCheck_hostsCheckedConcurrently`) + did not bite, though the reviewer agent saw the latter fail once. +- **Docs.** The "not yet run" wording is gone from the README, the + guide, `testing.md`, spec entry 21 and `providers-existing.md`. + `testing.md` gained a timings table and the `PROXMOX_TEST_VLAN_TAG` + export line. The link sweep script from the Phase 9 entry found 0 + broken links over the seven edited docs, and the guide has no `—`. +- `.claude/settings.json` stays out of the commit. diff --git a/docs/providers-existing.md b/docs/providers-existing.md index 4ea4851..b0121ad 100644 --- a/docs/providers-existing.md +++ b/docs/providers-existing.md @@ -34,7 +34,7 @@ and the symbol that shows it. |---|---|---|---|---|---| | `readiness` | immediate: a Pod is scheduled within seconds (`stateFromPod`) | provisioning-window: minutes of boot (`mapState`) | provisioning-window: boot plus, on first use, an image download (`imageCache.ensure`) | provisioning-window: boot plus, on first use, an image download (`ensureImage`) | provisioning-window: boot plus the guest agent's install at first boot, about 50 s to an address on the test host (`agentAddress`) | | `create-return` | submit: returns after `client.Create` | submit: `Insert` is never awaited, `Operation.Wait` is not called | **blocking**: waits through `DomainCreate` (`ensureRunning`), and the first image fetch runs inside (`imageCache.fetch`) | **blocking**: waits through `DomainCreate` and a 15-second poll for running (`ensureRunning`, `waitRunning`), and the first image fetch runs inside (`fetchImage`) | **blocking**: waits through the clone, upload, resize and start tasks and a 60-second poll for running (`provision`, `waitRunning`) | -| `delete-return` | returns-early: `client.Delete`, Pod terminates on its own | returns-early: `Delete` submits an operation each time | waits: destroy, undefine and volume removal complete before return (`removeDomain`, `removeVolumes`) | waits: destroy, undefine and volume removal complete before return (`removeDomain`, `removeVolumes`) | waits: the stop, destroy and seed delete tasks complete before return (`destroy`, `deleteSeed`) | +| `delete-return` | returns-early: `client.Delete`, Pod terminates on its own | returns-early: `Delete` submits an operation each time | waits: destroy, undefine and volume removal complete before return (`removeDomain`, `removeVolumes`) | waits: destroy, undefine and volume removal complete before return (`removeDomain`, `removeVolumes`) | waits: the stop, seed delete and destroy tasks complete before return (`destroy`, `deleteSeed`) | | `ip-source` | assigned-at-create: `PodIP` (`instanceFromPod`) | assigned-at-create: the external NAT IP of the first NIC (`toInstance`) | discovered: ARP table first, then the guest agent, first routable IPv4 (`address`, `firstIPv4`) | discovered: the guest agent first, then the ARP table, first routable IPv4 (`addressSources`, `address`, `firstIPv4`) | discovered: the guest agent only, first routable IPv4 in the agent's order (`agentAddress`, `firstIPv4`) | | `gc-mark` | native-labels: Pod labels (`buildPod`) | native-labels: instance labels (`buildInsertRequest`) | written-metadata: a namespaced `` element written in `DomainDefineXML` (`metadata.go`) | written-metadata: an element in kvm2's own namespace written in `DomainDefineXML` (`metadataURI`, `newInstanceMetadata`) | written-metadata: three `key=value` lines in the VM's description, passed in the clone call itself (`formatDescription`, `parseDescription`) | | `created-at` | backend-timestamp: `CreationTimestamp` | backend-timestamp: `CreationTimestamp`, parse error gives zero | written-at-create: `createdAt` in the metadata element (`newInstanceMetadata`) | written-at-create: `createdAt` in the metadata element, parse error gives zero (`newInstanceMetadata`, `createdAt`) | written-at-create: `proxy-operator-created-at` in the description, parse error gives zero (`parseDescription`) | @@ -50,7 +50,7 @@ and the symbol that shows it. | `timeouts` | ctx-only: no `rest.Config.Timeout` is set | ctx-only: SDK defaults, with SDK retries on Get and List | per-call for the dial only (`defaultHandshakeTimeout` in `dialSSH`); RPCs are ctx-only and a cancelled `call` closes the connection | per-call: a budget around the dial, every RPC, the uploads and the download (`defaultBudgets`, `call`); a cancelled or overlong call kills the pooled connection | per-call: dial and TLS handshake 10 s, each request `requestTimeoutSeconds`, a budget per task wait (`defaultBudgets`); a cancelled request ends only itself | | `logging` | none | tiered: V(1) outcomes and `logAPIError`, V(2) detail, V(5) wire log (`wirelog.go`) | tiered: V(1) outcomes and one `libvirt call` line per RPC (`trace.go`), V(2) XML and addresses; no wire tier | tiered: V(1) outcomes and one `libvirt call` line per RPC (`observe.go`), V(2) XML and addresses; no wire tier | tiered: V(1) outcomes and one `proxmox api call` line per request (`observe.go`), V(2) parameters, config summaries and addresses; no wire tier | | `child-spans` | transport-wrapper: `tracing.RestConfigWrapper()` on its own `rest.Config`, parent-gated | sdk-instrumentation: the SDK adds `otelhttp` itself; not parent-gated, but every call site has a parent | **none**: `trace.go` is a log decorator, not spans | span-decorator: `kvm2.` per RPC (`observe`), plus `kvm2.dial` (`libvirtDialer`) and `kvm2.download` (`fetchImage`) | transport-wrapper: `tracing.RestConfigWrapper()` on the client's transport, parent-gated, one `HTTP ` span per request (`newHTTPAPI`) | -| `backend-test` | cluster-e2e: kind (`test/e2e`) | manual-runbook: [gcp-vm-validation.md](gcp-vm-validation.md) | integration-tag: `kvm_integration_test.go`, `make test-integration-kvm` | integration-tag: `kvm2_integration_test.go`, `make test-integration-kvm2` | integration-tag: `proxmox_integration_test.go`, `make test-integration-proxmox`; compiled, not yet run against a host | +| `backend-test` | cluster-e2e: kind (`test/e2e`) | manual-runbook: [gcp-vm-validation.md](gcp-vm-validation.md) | integration-tag: `kvm_integration_test.go`, `make test-integration-kvm` | integration-tag: `kvm2_integration_test.go`, `make test-integration-kvm2` | integration-tag: `proxmox_integration_test.go`, `make test-integration-proxmox`; passed on a PVE 9.2.20 host | ## 2. Other differences @@ -614,16 +614,18 @@ was written. - **Delete.** Under the per-name lock, `destroy`: `resolve`; a lock other than `suspended` or `status: unknown` is Transient; the config's `does not exist`, another name or no mark leave the VM alone; our VM - hibernated is Permanent, because the destroy refuses it and the lock - never clears. Otherwise stop (unless stopped) and destroy with - `purge=1&destroy-unreferenced-disks=1`, each a task waited on through - `runTask`, where `does not exist` means gone. Then `deleteSeed` on the - VM's node, where a failure fails the call (a 403 for - `Datastore.Allocate` is Permanent), or, with no VM of ours destroyed, - `deleteSeedEverywhere` over every configured node, where a node that - fails is logged and skipped unless ctx ended. A missing volume still - ends in a successful task, so no message is matched. Nothing is left - when nil is returned. + hibernated is destroyed like a stopped one, since the destroy takes it. + Otherwise stop (unless stopped), `deleteSeed` on the + VM's node, where a failure fails the call and leaves the VM for the + next Delete (a 403 for `Datastore.Allocate` is Permanent), then destroy + with `purge=1&destroy-unreferenced-disks=1`. Stop and destroy are tasks + waited on through `runTask`, where `does not exist` means gone. The + seed goes first because a VM already destroyed is NotFound to `Get`, + and nothing would call Delete again. With no VM of ours, + `deleteSeedEverywhere` tries every configured node, where a node that + fails is logged and skipped unless ctx ended or the failure is + Permanent. A missing volume still ends in a successful task, so no + message is matched. Nothing is left when nil is returned. - **ListByTag.** One `ClusterResources` covers every node, unreachable ones included (their guests are listed with `status: unknown`). `isCandidate` keeps QEMU VMs that are not templates and carry the tag @@ -639,9 +641,9 @@ was written. status Transient, including the lock, the taken VMID and pveproxy's 595 and 596); TLS verification failures (`tls.CertificateVerificationError` and the three `x509` types) Permanent; everything else Transient. The - 500 matchers (`isNotExist`, `isLocked`, `isAlreadyExists`, the three - agent ones) are anchored to the 500 status and each is backed by a - captured response. `permanentf` and `transientf` classify at the + 500 matchers (`isNotExist`, `isAlreadyExists`, the two agent ones) + are anchored to the 500 status and each is backed by a captured + response. `permanentf` and `transientf` classify at the source: placement, the template checks, the ownership check, a malformed token file or CA file (Permanent); a missing token or CA file, an expired wait, a failed task (Transient). Nothing is Quota. @@ -694,7 +696,7 @@ was written. by `PROXMOX_TEST_*` variables and `make test-integration-proxmox`, described in [testing.md](testing.md#the-proxmox-integration-test-real-proxmox-ve-host); - it compiles but has not run against a host yet. + it passed against PVE 9.2.20 on 2026-09-24. - **Deployment.** The optional `proxmox-token` Secret (keys `token` and `ca.pem`) mounted at `/etc/proxy-operator/proxmox` (`config/manager/manager.yaml`), `make proxmox-token-secret`, a @@ -713,11 +715,9 @@ was written. - The VMID range is shared with humans; a hand-made VM with a proxy's name is Permanent. - Cross-node clones (`target`) need shared storage and are untested. - - A hibernated VM of ours latches its Proxy in Create and in Delete. + - A hibernated VM of ours latches its Proxy when Create's adoption + finds it; a recorded one is replaced. - The CA is cached after first use; the token is not. - - Never run against a real host yet; three answers were never captured - (the storage status for a missing storage, a destroy on a hibernated - VM, a stop on a paused VM). ## 9. Gaps shared by the existing providers diff --git a/docs/proxmox-provider-spec.md b/docs/proxmox-provider-spec.md index 7af56d7..cac3451 100644 --- a/docs/proxmox-provider-spec.md +++ b/docs/proxmox-provider-spec.md @@ -77,7 +77,11 @@ lands on the same number: `vmidFor(uid, attempt)` is clears by itself. Ownership is judged at once, as `Get` does (entry 9): a foreign hibernated VM is probed past, and ours is `ErrPermanent` (resume or destroy it), like kvm2's paused domain. Container and - template entries at a VMID are foreign without a config read. + template entries at a VMID are foreign without a config read, and so + is a VM whose listed name is not `req.Name`, whatever its lock: only a + clone in progress is nameless, and waiting on another VM's `backup` + lock would hold Create for the whole clone budget on every retry + (review finding P2). The wait in step 3 is what closes the duplicate-VM bug Phase 0 found: after a crash mid-clone, the retry's by-name lookup misses the nameless VM, but @@ -284,7 +288,7 @@ and starts it; the reconciler has no ID for it to `Get`. | VMID absent | `/cluster/resources` | `ErrNotFound` | | `lock` set, other than `suspended` | resources entry (a clone in progress also has no name) | `ErrTransient` | | `lock: suspended` (hibernated to disk) | resources entry | Stopped | -| `status: unknown` (node not reachable) | resources entry | `ErrTransient` | +| `status: unknown` (node not reachable, or a VM seconds old the listing has not reported yet: the first `Get` after Create met this on pv1-1) | resources entry | `ErrTransient` | | config `does not exist` | `GET …/config` | `ErrNotFound` (deleted between the two calls) | | config `lock` set | config | `ErrTransient` | | `name` differs from the ID's name | config | `ErrNotFound`, logged: the VMID was reused | @@ -382,34 +386,48 @@ the seed ISO; the VM's disks are removed by the destroy call itself, with 1. Parse the ID (`ErrPermanent` if malformed). 2. `/cluster/resources`. VMID absent, or held by a container: skip to - step 7, trying every configured node because the VM's node is no longer + step 6, trying every configured node because the VM's node is no longer known. 3. `lock` set, other than `suspended`, or `status: unknown`: `ErrTransient`. A clone or another task still running finishes first; the reconciler calls Delete again after `DeletionPoll`. -4. Read the config. `does not exist`: skip to step 7 on every node. A +4. Read the config. `does not exist`: skip to step 6 on every node. A different name, or no mark: the VM is not ours; log it at V(1) and skip - to step 7 on every node without touching it. Our VM with a `lock` other + to step 6 on every node without touching it. Our VM with a `lock` other than `suspended` (taken since the listing): `ErrTransient`. Our VM with - `lock: suspended` (hibernated): `ErrPermanent` asking the admin to resume - or destroy it, the same answer Create's ownership check gives. The - destroy refuses a locked VM, and that lock never clears on its own, so - Transient would retry forever; Permanent puts the admin action in the - Proxy's status, where `reconcileManaged` latches it and the finalizer - path keeps retrying it visibly. A hibernated VM that is not ours is left - alone like any other. + `lock: suspended` (hibernated) is destroyed like a stopped one: the + listing shows it `stopped`, so step 5 is skipped, and the destroy + accepts it and removes its suspend state volume with the disks + (`destroy_hibernated_response.json`, `task_status_destroy_hibernated.json`, + captured in Phase 10). Until then Delete refused it as `ErrPermanent` + on the belief that the destroy would. A hibernated VM that is not ours + is left alone like any other. 5. Status (from the listing) other than `stopped`: `POST …/status/stop`, wait (stop budget). Stop works on a paused VM too, which the listing - shows as `running`. `does not exist` means the VM is gone. -6. `DELETE /nodes/{node}/qemu/{vmid}?purge=1&destroy-unreferenced-disks=1`, - wait (destroy budget). `does not exist` is success. -7. `DELETE /nodes/{node}/storage/{isoStorage}/content/:iso/-seed.iso`, - wait (seed-delete budget), on the VM's node when steps 5 and 6 ran on a - VM of ours, and on every configured node in config order otherwise. A - missing volume is still a successful task, so no message is matched. On - the every-node path a node whose request or task fails is logged at V(1) - and skipped, unless ctx ended, which fails the call; on the VM's node a - failure fails the call (a 403 for `Datastore.Allocate` is Permanent). + shows as `running`: the task ends `OK` and the VM is `stopped` + (`stop_paused_response.json`, `task_status_stop_paused.json`, + `status_current_after_stop_paused.json`, captured in Phase 10). + `does not exist` means the VM is gone. +6. `DELETE /nodes/{node}/storage/{isoStorage}/content/:iso/-seed.iso`, + wait (seed-delete budget), on the VM's node when step 5 was reached on + a VM of ours, and on every configured node in config order otherwise + (then step 7 is skipped). A missing volume is still a successful task, + so no message is matched. The seed goes **before** the destroy: once + the VM is gone `Get` is `ErrNotFound`, the reconciler releases the + finalizer and GC no longer lists it, so a seed left by a failed step + after the destroy would never be deleted (review finding CR1). On the + VM's node a failure fails the call and leaves the VM stopped for the + next Delete. On the every-node path a node whose request or task fails + is logged at V(1) and skipped, unless ctx ended or the failure is + `ErrPermanent` (a 403 for `Datastore.Allocate`, which every node + would give), which fail the call, so the missing privilege is visible + on both paths (review finding P1). +7. `DELETE /nodes/{node}/qemu/{vmid}?purge=1&destroy-unreferenced-disks=1`, + wait (destroy budget). `does not exist` is success. The destroy leaves + the seed alone (Phase 0 deleted it after the destroy), which is why + step 6 exists. That Proxmox deletes an ISO a stopped VM's `ide2` still + references was not in the Phase 0 captures; the Phase 10 real-host run + proved it (imgdel `OK` on the stopped VM's seed, then the destroy). 8. Return nil. Everything is gone when Delete returns nil, so `Get` afterwards is @@ -474,6 +492,7 @@ per-parameter map. One `classify(op string, err error)`: | 401 `Authentication failed!`, 401 `No ticket` | Permanent | the token or its secret is wrong; fixed in the Secret | | 403 `Permission check failed (, )` | Permanent | the token lacks a privilege; the message names it | | 400 `Parameter verification failed.` with `errors` | Permanent | the request is wrong; the message includes the field errors | +| 400 as above with `storage: No such storage.` | Permanent from a call; the Checker's `StorageNotFound` (entry 16) | the storage does not exist in the cluster | | other 4xx | Permanent | a request Proxmox will never accept | | 500 `Configuration file '…' does not exist` | NotFound (Transient when `op == "create"`) | the VM is gone | | 500 `VM is locked ()` | Transient | a task holds the VM | @@ -490,17 +509,24 @@ per-parameter map. One `classify(op string, err error)`: Every matched string is backed by a captured response under `testdata/`; the table test loads them rather than repeating the text. Matching is on -fixed substrings anchored to the 500 status: `does not exist`, -`VM is locked`, `config file already exists`, `QEMU guest agent is not -running`, `No QEMU guest agent configured`, and for the stopped VM the whole -message against `^VM [0-9]+ is not running$`. The agent messages are tested -in that order, because the last is a suffix of the first. +fixed strings anchored to the 500 status: `does not exist`, +`config file already exists`, and the whole messages +`QEMU guest agent is not running` and `No QEMU guest agent configured`, +plus, on a 400, the `storage` field error `No such storage.` (whole; a +clone onto a storage without image support has another `storage` error). +`VM is locked` and the stopped VM's `VM is not running` are not +matched by the provider: both are Transient through the default branch. +The tests keep a matcher for each, with the whole stopped-VM message +against `^VM [0-9]+ is not running$` (its tail is a suffix of the +agent-not-running answer), only to hold the fake's wording to the +captures (review finding CR5). **Evidence:** F1 (error text in the reason phrase and the body; 401 has no body); F2, F4, F5, F7 (the 500 messages); F6 (no out-of-space response could be produced on thin storage); F22 (both 401 texts); fixtures `err_forbidden.json`, `err_bad_param.json`, `err_tag_invalid.json`, -`err_upload_bad_name.json`; the Phase 0 TLS finding (`192.168.2.8` not in +`err_upload_bad_name.json`, and from Phase 10 `storage_status_missing.json` +(a missing storage is a 400, not the 500 the spec had assumed); the Phase 0 TLS finding (`192.168.2.8` not in the node certificate); contract §5. ## 14. Credentials and TLS (`credentials`) @@ -611,11 +637,15 @@ an admin: - `CredentialsUnavailable`: the token file is missing or malformed. Fix the Secret, not Proxmox. - `StorageNotFound`: `isoStorage` is missing, inactive, or has no `iso` - content. Fix the storage on the node. Any 500 from the storage status - call counts as missing, whatever its wording (it does not exist, is not - enabled on that node, or is offline; no missing-storage response was - captured, so no text is matched), and the message carries Proxmox's - text. A 401, 403, 595 or 596 there keeps its shared reason. + content. Fix the storage on the node. A storage the cluster does not + know answers `400 Parameter verification failed.` with the field error + `storage: No such storage.` (`storage_status_missing.json`, captured in + Phase 10); that exact field error is matched, and the message carries + it. Any 500 from the storage status call counts as missing too, + whatever its wording: Proxmox knows the storage but cannot use it on + that node (not enabled there, or offline). That answer was not + captured, so no text is matched. Any other 4xx, and a 401, 403, 595 or + 596, keeps its shared reason. - `TemplateNotFound`: a configured template VMID does not exist or is not a template. Fix the template or the catalogue. @@ -783,8 +813,8 @@ carries the cloud-init text. Contract §11 list, with the test for each item. Every name below exists (Phase 8 checked each with a grep for `func (`). The two integration -tests compile under `-tags=integration` but have not run against a host -yet; Phase 10 runs them. +tests passed on pv1-1 on 2026-09-24 (timings in +[testing.md](testing.md#the-proxmox-integration-test-real-proxmox-ve-host)). - [x] repeat `Create`: `TestCreate_repeatReturnsSameIDAndCreatesNothing` - [x] `Get` after `Delete`: `TestGet_afterDeleteReturnsNotFound` @@ -802,7 +832,7 @@ yet; Phase 10 runs them. - [x] Checker reasons, expired `ctx`, stable order: `TestCheck_reasonsTable`, `TestCheck_expiredContextReturnsPromptly`, `TestCheck_oneEntryPerNodeInConfigOrder` - [x] sentinel tests for `CloudInit` and the credential: `TestCreate_cloudInitSentinelNeverLoggedOrInErrors`, `TestClient_tokenNeverInErrorsOrLogs`, and `TestLogging_v0SilentV1OutcomeV2Detail` over a whole lifecycle at every level - [x] child spans under the operation span: `TestTracing_httpSpansUnderOperationSpan`, `TestClient_childSpanUnderParent` -- [x] real-backend lifecycle test: `TestProxmox_lifecycle` and `TestProxmoxCheck_realHost` in `proxmox_integration_test.go` (compiled, not yet run) +- [x] real-backend lifecycle test: `TestProxmox_lifecycle` and `TestProxmoxCheck_realHost` in `proxmox_integration_test.go`, passed on pv1-1 Tests that cover the Phase 0 corrections specifically, in addition to the plan's per-phase lists: @@ -904,4 +934,28 @@ user confirms. VMID range delete each other's VMs), an unconfigured provider latches the Proxy. +Accepted after the Phase 10 review +([docs/reviews/2026-09-23-proxmox-provider-review/](reviews/2026-09-23-proxmox-provider-review/REPORT.md), +findings P3 and P4), and listed in the guide's Known gaps: + +- **A Create that failed permanently after its clone keeps its VM**, and + the next Create adopts it by name even when `placement` has changed. + Destroy the VM before changing `placement` on such a Proxy. +- **A retry can clone a second VM** when a Create is cut off while + cloning onto a later probe VMID, the VM on the earlier one is removed, + and the retry comes before the clone ends. The next Create names both + VMIDs as `ErrPermanent`. + +Changed by the Phase 10 captures of the three answers Phase 0 never saw: + +- **A hibernated VM of ours** latches only Create's adoption + (`ErrPermanent`, resume or destroy it; entry 2). `Get` reports it + Stopped, so the reconciler replaces it, and Delete destroys it, since + the destroy takes a hibernated VM (entry 11). Before the capture, Delete + refused it as well. +- **A missing `isoStorage`** is a 400 with `storage: No such storage.`, + which the Checker reports as `StorageNotFound` (entry 16); a missing + storage at upload time is Permanent like any 4xx. +- **A paused VM** stops like a running one (entry 11, step 5), as assumed. + **Evidence:** contract §13; entries 2, 7, 10, 11, 14 above. diff --git a/docs/proxmox-provider.md b/docs/proxmox-provider.md index e152118..03b969b 100644 --- a/docs/proxmox-provider.md +++ b/docs/proxmox-provider.md @@ -164,8 +164,14 @@ pve 'cluster/resources?type=vm' # the template is listed with "temp **Rotating the token needs no restart.** The token file is read on every request, so the next request after the kubelet refreshes the - mount uses the new token. **Rotating the CA does**: it is loaded on - first use and cached for the life of the process. A missing CA file is + mount uses the new token. Keep the old token valid until then: a 401 + is a permanent error, so a Proxy whose request lands in the gap + latches until its spec changes. Add the new token + (`pveum user token add`), update the Secret, wait a couple of minutes + for the mount to refresh, and only then remove the old token. A token + created with `--expire` needs the same rotation before it expires. + **Rotating the CA does** need a restart: it is loaded on first use and + cached for the life of the process. A missing CA file is retried until it lands; one without a certificate is a permanent error. 2. Add a provider entry to the providers config @@ -302,7 +308,9 @@ through an ownership check (`inspect` in `create.go`): a VM with a lock naming the VMID and both UIDs. The operator never takes over a VM it did not create; - our name, our mark, hibernated (`lock: suspended`): **permanent error**, - `resume or destroy it in Proxmox` (see *Known gaps*). + `resume or destroy it in Proxmox` (see *Known gaps*). This only happens + when Create has not recorded the VM yet; a recorded VM that gets + hibernated is reported Stopped and replaced. Two VMs with the proxy's name are a permanent error naming every VMID; remove all but one. @@ -392,15 +400,18 @@ pvesm free proxy-seeds:iso/proxy-abc…-seed.iso ``` **Delete** runs under a per-name lock shared with Create: stop (a paused -VM too), destroy with `purge=1&destroy-unreferenced-disks=1`, then delete -the seed on the VM's node. Each is a task that is waited on. A VM that is -already gone, not ours, or unmarked is left alone, and the seed is then -looked for on every configured node, where a node that fails is logged -and skipped. On the VM's own node a failed seed delete fails the call: -**without `Datastore.Allocate` on `isoStorage` the 403 makes Delete -permanent**, and the Proxy's deletion stays stuck with that message until -the privilege is granted. When Delete returns nil, nothing of the Proxy -is left. +VM too), delete the seed on the VM's node, then destroy with +`purge=1&destroy-unreferenced-disks=1`. Each is a task that is waited on. +The seed goes before the VM because once the VM is gone nothing calls +Delete again; a failed seed delete leaves the VM stopped, and the next +Delete finishes both. A VM that is already gone, not ours, or unmarked is +left alone, and the seed is then looked for on every configured node, +where a node that fails is logged and skipped. A refusal every node would +give fails the call instead: **without `Datastore.Allocate` on +`isoStorage` the 403 makes Delete permanent** on either path, and the +Proxy's deletion stays stuck with that message (a replaced Proxy stays +failed, its VM stopped) until the privilege is granted. When Delete +returns nil, nothing of the Proxy is left. ## Logging @@ -554,9 +565,9 @@ transient. | Class | What lands there | |---|---| -| `ErrPermanent` | placement values not in the catalogue; a template VMID that is missing or not a template, has a cloud-init drive, or lacks `net0` or a sized boot disk when the config needs them; a malformed providerID; any 4xx (401 `Authentication failed!` or `No ticket`: the token; 403 `Permission check failed (, )`: a missing privilege, including `Datastore.Allocate` for the seed delete; 400 `Parameter verification failed.` with the field errors); a TLS verification failure; a token file that exists but is not `@!=`; a CA file with no certificate; a VM with the proxy's name that carries no mark or another UID; two VMs with the proxy's name; our VM hibernated (`lock: suspended`), in both Create and Delete | +| `ErrPermanent` | placement values not in the catalogue; a template VMID that is missing or not a template, has a cloud-init drive, or lacks `net0` or a sized boot disk when the config needs them; a malformed providerID; any 4xx (401 `Authentication failed!` or `No ticket`: the token; 403 `Permission check failed (, )`: a missing privilege, including `Datastore.Allocate` for the seed delete; 400 `Parameter verification failed.` with the field errors); a TLS verification failure; a token file that exists but is not `@!=`; a CA file with no certificate; a VM with the proxy's name that carries no mark or another UID; two VMs with the proxy's name; our VM hibernated (`lock: suspended`) found by Create's adoption | | `ErrNotFound` | from `Get`: the VMID absent from the cluster's resources, the 500 `Configuration file 'nodes//qemu-server/.conf' does not exist`, the VM at the VMID now named differently or unmarked. Inside `Create` the `does not exist` answer is `ErrTransient`, because the contract forbids NotFound from Create; `Delete` treats all of these as done | -| `ErrTransient` | everything else: a missing token or CA file (the Secret has not landed yet); network errors and cancellation; 500 `VM is locked ()` and any lock seen in the resources or config; a VM whose node does not answer (`status: unknown`); the agent's answers that `Get` cannot place, such as `VM is not running`; any other 5xx, including pveproxy's 595 and 596; a task that fails or runs out of its budget; no free VMID after 32 probes; the wait for running expiring, or a VM that stops right after its start | +| `ErrTransient` | everything else: a missing token or CA file (the Secret has not landed yet); network errors and cancellation; 500 `VM is locked ()` and any lock seen in the resources or config; a VM whose node does not answer, or one a few seconds old the listing has not reported yet (`status: unknown`); the agent's answers that `Get` cannot place, such as `VM is not running`; any other 5xx, including pveproxy's 595 and 596; a task that fails or runs out of its budget; no free VMID after 32 probes; the wait for running expiring, or a VM that stops right after its start | `Get` turns two agent answers into states rather than errors: `QEMU guest agent is not running` is Provisioning (booting, or the agent @@ -613,13 +624,26 @@ They never contain the token, the CA content or cloud-init. Beyond the ten limits accepted in the design: -- **A hibernated VM latches its Proxy.** A VM of ours with - `lock: suspended` (hibernated to disk) makes both Create and Delete fail - permanently with `resume or destroy it in Proxmox`: the lock never - clears on its own, and the destroy refuses a locked VM. `Get` reports it - Stopped. Resume it by starting it (`qm start ` restores the - hibernated state) or destroy it on the node, then touch the Proxy's - spec, or let the pending Delete retry. +- **A hibernated VM is replaced, or latches a Create.** `Get` reports a + VM of ours with `lock: suspended` (hibernated to disk) as Stopped, so + the reconciler deletes it (the destroy removes its suspend state volume + too) and creates a new one. Only a Create that finds such a VM before it + was ever recorded, by name, fails permanently with + `resume or destroy it in Proxmox`: resume it by starting it + (`qm start ` restores the hibernated state) or destroy it on the + node, then touch the Proxy's spec. +- **A Create that failed permanently after its clone keeps its VM.** + The stopped VM is adopted by name on the next Create, even when the + Proxy's `placement` has changed since: the Proxy then runs the old + template on the old node. After changing `placement` on such a Proxy, + destroy the VM on the node (`qm destroy --purge`) first, or + delete and recreate the Proxy. +- **A retry can clone a second VM** in one rare sequence: a Create is + cut off while cloning onto a later probe VMID (the first was taken by + someone else's VM), that other VM is removed, and the retry comes + while the clone is still running, so its VM is still nameless. The + retry clones onto the now free first VMID. The next Create then fails + permanently with `VMs … are all named …`; destroy either VM. - **The seed upload is one request**, bounded by `requestTimeoutSeconds`. A seed is tens of KiB, but a multi-megabyte `vendorData` would need a larger timeout. @@ -627,11 +651,6 @@ Beyond the ten limits accepted in the design: `config/samples/proxy_proxmox.yaml` installs Squid with `http_access allow all`; restrict it before exposing the proxy network to anything but the cluster. -- **Not yet proven end to end.** The unit tests run over captured - responses from `pv1-1` and an in-memory fake; the integration tests - below compile but have not run against a host yet. Three answers were - never captured: the storage status for a missing storage, a destroy on - a hibernated VM, and a stop on a paused VM. ## Testing @@ -644,3 +663,5 @@ which also lists the variables it needs; run it with `make test-integration-proxmox`. `TestProxmoxCheck_realHost` runs the health check against the same host and creates nothing: `go test -tags=integration -count=1 -v -run TestProxmoxCheck ./internal/provider/proxmox/`. +Both passed on `pv1-1` (PVE 9.2.20) on 2026-09-24: Create took 6.2 s, +and the guest reported an address 50 s after Create returned. diff --git a/docs/reviews/2026-09-23-proxmox-provider-review/REPORT.md b/docs/reviews/2026-09-23-proxmox-provider-review/REPORT.md new file mode 100644 index 0000000..b59b3f4 --- /dev/null +++ b/docs/reviews/2026-09-23-proxmox-provider-review/REPORT.md @@ -0,0 +1,286 @@ +# proxmox provider review: final report + +Branch `feat/proxmox-provider` at `7f2073f` (Phases 0 to 9), diffed against +`main` (`2167819`): 123 files, 49 of them Go. The core is +`internal/provider/proxmox/`; also in scope were the `nocloud` and +`keyedlock` extraction from kvm2 (`0a6c862`), the config block in +`internal/provider/config*.go`, registration and the deploy files. + +**Inputs.** [docs/proxmox-provider-spec.md](../../proxmox-provider-spec.md) +(the spec the code implements) and +[docs/provider-contract.md](../../provider-contract.md) (the MUST/SHOULD +rules). A disagreement with either counts as a finding. + +**Reviewers.** Two independent passes, both read-only: + +1. The `go-operator-reviewer` agent (`.claude/agents/go-operator-reviewer.md`), + briefed with the spec and the contract, findings **P1 to P6** and + simplifications **S1 to S3**. +2. `/code-review high` on the branch, findings **CR1 to CR10**. + +Where both passes found the same thing, the finding is listed once with both +IDs. Every finding is **fixed** (with a regression test when it is code), +**accepted** (real, not fixed here, documented as a known limit), or +**rejected** (with the reason). + +**Toolchain.** `make test` green before and after the fixes. `make lint` +panics inside staticcheck on the local Go 1.27 toolchain, as it did during +kvm2, so lint ran without it, over the proxmox package and +`internal/provider`, with and without the integration tag: + +```bash +bin/golangci-lint run --disable staticcheck --max-same-issues 0 --max-issues-per-linter 0 ./internal/provider/proxmox/... ./internal/provider/ +bin/golangci-lint run --build-tags integration --disable staticcheck --max-same-issues 0 --max-issues-per-linter 0 ./internal/provider/proxmox/... ./internal/provider/ +``` + +Before the fixes both runs were at the Phase 8 and 9 baseline: logcheck 12, +modernize (mapsloop) 2, unparam 3 in `fake_test.go`, gocyclo 1 +(`TestAPITypes_decodeCaptures`), goconst on repeated strings, and the three +goconst hits in `internal/provider`. After the fixes the counts are the same +except goconst, one lower (224, 231 with the tag). No hit is new. The first +version of the P2 test did add one: a third `"backup"` literal in the +package's tests, then `"migrate"`. It now uses `"snapshot-delete"`. + +## Summary + +| ID | Severity | Finding | Disposition | +|---|---|---|---| +| CR1 + P1 | medium-high | A seed delete that fails after the destroy leaks the seed for good, and the class of the error depends on the attempt | **fixed** | +| CR2 | medium | A 403 on the seed delete latches a replaced Proxy whose VM is already gone | **fixed** (same change as CR1) | +| CR3 + P2 | medium | The ownership check waits out any lock, even on a VM whose listed name is someone else's | **fixed** | +| P3 | low-medium | Create adopts a leftover stopped VM whatever the current placement says | **accepted**, documented | +| P4 | low | A crash mid-clone plus a freed earlier VMID can yield a second VM | **accepted**, documented | +| P5 | low | 401/403 are Permanent, which bites during a token rotation | **fixed** in the guide | +| P6 | low | Get and Delete trust the listing's `status`, which lags | **accepted**, confirmed on the host; error message fixed | +| CR5 | low | `isLocked` and `isVMNotRunning` are called only from tests | **fixed** | +| CR8 | low | `TestPermanentfAndTransientf` breaks the `Test_` rule | **fixed** | +| CR9 | low | `get` repeats `mapState`'s "guest running" rule inline | **fixed** | +| CR4 + S2 | low | The Checker repeats the cluster-wide `/version` and `/cluster/resources` per node | **rejected** | +| CR6 | low | `lockName`, `permanentf` and `transientf` are copies of kvm2's | **rejected** | +| CR7 | low | `readToken`, the CA load and seed staging do file I/O without a ctx | **rejected** | +| CR10 | low | `New` runs `withDefaults` twice | **rejected** | +| S1 | simplification | The `observe.go` decorator could be one log line in `send` | **rejected** | +| S3 | simplification | `nocloud` exports `GuestAgentScript` and `SeedLabel` for tests only | **rejected** | + +## Fixed + +### CR1 + P1: the seed outlives a Delete that fails after the destroy + +`internal/provider/proxmox/delete.go`. Delete stopped the VM, destroyed it, +then deleted the seed. If the seed delete failed (a 595, a task over its +1 minute budget, a 403), Delete returned an error, but the VM was already +gone. The reconciler's next `Get` was `ErrNotFound`, so `reconcileDelete` +released the finalizer without calling Delete again, and GC lists only VMs. +The seed stayed on `isoStorage` for good. That breaks contract §2.3 ("MUST +remove everything Create made ... GC sees only what ListByTag returns"). A +destroy task that outlived its budget but finished in the background had +the same effect. + +P1 adds the second half. On the finalizer path the next Delete found the +VMID gone and took `deleteSeedEverywhere`, which logged the same 403 at V(1) +and returned nil. The seed leaked with nothing visible. + +**Fix.** The seed now goes between the stop and the destroy, so the VM, +which is what `Get` looks at, is the last thing removed: a failed seed +delete leaves the VM stopped, `Get` keeps finding it, and the next Delete +finishes both. `deleteSeedEverywhere` now fails on an `ErrPermanent` +answer (a 401 or 403 every node would give) instead of skipping it. It +still skips a node that is down or lacks the storage. `runTask`'s "gone" +result lost its last reader and was dropped. + +That Proxmox deletes an ISO a stopped VM's `ide2` still references is the +one assumption the fake cannot prove. From memory of the PVE source, not +checked against it: the content delete runs `vdisk_free` without looking +for references to `iso` volumes, and the destroy skips CD-ROM drives. +Phase 0 only ever deleted the seed after the destroy. **The Phase 10 +real-host run proved it** on 2026-09-24: the imgdel task on the stopped +VM's seed ended `OK`, the destroy followed, and `TestProxmox_lifecycle` +step 10 found neither the VM nor the seed. + +**Tests.** `TestDelete_seedDeleteFailureOnVMNodeFails` (403, 596 and a +failed task on the VM's node) now asserts that no Destroy was sent, that +`Get` still finds the VM, that a second Delete gives the same class, and +that a Delete after the fault clears removes both. It failed on the old +code with `Destroy calls = 1` and `Get = instance not found`. +`TestDelete_seedDeleteNodeFailureSkippedWhenNodeUnknown/a refused privilege +fails the call` covers the every-node 403. Updated for the new order: +`TestDelete_stoppedVMSkipsStop`, `TestDelete_runningVMStoppedFirst`, +`TestDelete_seedDeleteIsTaskAndIdempotent`, +`TestDelete_destroyTaskFailureIsTransient` (the seed is now gone and the VM +kept) and `TestDelete_goneMidwayIsSuccess` (a VM gone at the stop still +gets a Destroy call, which answers `does not exist`). +`TestLogging_noOpLinesAtV1`'s skip case now injects a node-level 500, +since a 403 no longer skips. + +Docs: spec entry 11 (steps 5 to 7 reordered, the Permanent exception), +the guide's Delete paragraph, `providers-existing.md`'s Delete bullet and +its `delete-return` cell. + +### CR2: a 403 on the seed delete latched a Proxy with no VM + +Same code, same fix. On the replace path (`reconcileManaged` sees a Stopped +VM and calls Delete), the old order destroyed the VM, got a 403 on the +seed, and returned Permanent. `providerFailure` latched it, so the Proxy +sat Failed pointing at a VM that no longer existed. The next attempt would +have returned nil. Now the 403 comes before the destroy: the VM stays +stopped, and every attempt gives the same Permanent answer naming +`Datastore.Allocate` until the privilege is granted. The guide says so. + +### CR3 + P2: the lock wait on a VM that is plainly not ours + +`internal/provider/proxmox/create.go`, `allocate`. Any taken qemu VMID went +through `inspect`, which first waits for the VM's lock to clear (clone +budget, 10 minutes) and only then compares names. A VM on one of the +Proxy's probe VMIDs, listed with another name and holding a long lock (a +`vzdump` backup, or a stale lock a failed backup left behind), made every +Create wait 10 minutes and return Transient. `vmidFor` is deterministic, +so every retry landed on the same VMID. Only a clone in progress is +nameless (F8), so a listed name other than ours settles the question. + +**Fix.** `allocate` sends a taken entry to `inspect` only when its listed +name is empty or ours; any other name probes on without a config read. +Spec entry 2's step 3 says so. + +**Tests.** `TestCreate_takenVMIDInspectedBeforeProbing` gains "foreign VM +under a lock probes on without waiting" (a `snapshot-delete` lock with a +2 s clone budget; the old code spent the 2 s and failed Transient), and +"foreign VM probes on without a read" now expects one config read (the +template's). `TestCreate_vmidRangeExhaustedIsTransient` expects one +VMConfig call instead of five. + +### P5: token rotation order + +`docs/proxmox-provider.md`, cluster-side setup. A 401 is Permanent (the +same convention as GCP), so a request made with a revoked token before the +kubelet refreshes the Secret latches that Proxy. The guide now gives the +order: add the new token, update the Secret, wait for the mount, then +remove the old one, and the same before an `--expire` date. + +### CR5: matchers used only by tests + +`isLocked` and `isVMNotRunning` in `errors.go` had no production caller. +Both answers are Transient through `classify`'s default branch. They moved +to `fake_test.go`, where `TestFakeAPI_errorsMatchCaptures`, +`TestAPIError_matchers` and the fake's own checks still use them to hold the +fake's wording to the captures. Spec entry 13's matching sentence and +`providers-existing.md` now name only the matchers the provider uses. + +### CR8: test name + +`TestPermanentfAndTransientf` became `TestPermanentf_wrapsClassWithMessage` +and `TestTransientf_wrapsClassWithMessage`. kvm2 has the same name; it is +out of scope here. + +### CR9: one rule, one place + +`get` asked `status == "running" && qmpstatus in {running, ""}` inline to +decide whether to query the agent, the same rule `mapState` applies. Both +now call `guestRunning(status, qmpStatus)` in `state.go`, so a change to it +cannot leave the two out of step. `TestMapState_*` and the Get tests cover +it unchanged (`TestMapState_table`, `TestMapState_fromCaptures`). + +## Accepted + +### P3: adoption ignores a placement change + +`create.go:94-118`. A Create that fails Permanent after the clone (a 403 on +the upload, say) leaves a stopped VM. If the user then changes +`placement.image` or `zone`, the next Create adopts that VM by name, and +the Proxy runs the old template on the old node. The fix the reviewer +proposed (record the template in the mark, re-clone a never-started VM +whose template differs) is a new mark field and a destroy path inside +Create. That is too much for this MR for a two-step, user-driven sequence. +Documented in the guide's Known gaps and spec entry 24: destroy the VM +before changing `placement` on such a Proxy. + +### P4: a second VM after a crash mid-clone + +The retry misses a nameless in-progress clone on a later probe VMID when +the VM on the earlier VMID was removed meanwhile, and clones again. It +takes a crash or cut-off during a clone of a few seconds, a foreign VM on +the first probe VMID, and that VM's removal in the same window. The +proposed fix (inspect every nameless `lock: clone` entry in the range +first) would make every Create wait on every other Proxy's clone. The +outcome is visible, not silent: the next Create fails Permanent naming +both VMIDs. Documented in the guide and spec entry 24. + +### P6: the listing's `status` may lag + +`get.go:51`, `delete.go:60`, `delete.go:90`. This is the reviewer's +inference, not observed: `/cluster/resources` comes from pvestatd, which +may lag a state change by seconds. Both cases heal on retry: `Get` +answering Transient for `unknown` is polled through, and a destroy of a VM +that started behind the listing's back fails its task as Transient and the +next Delete stops it first. + +**Confirmed on pv1-1.** The first `Get` came 32 ms after the second Create, +about 6 s after the clone, and was Transient: the listing still showed the +VM as `status: unknown`. The next poll 5 s later saw it running. The +behaviour is right, but the message blamed the node ("its node pv1-1 is not +answering"), which was up. Get and Delete now say "its node … is not +answering, or the VM is too new to be reported yet". +`TestGet_unknownStatusIsTransient` asserts the wording, and spec entry 9 +and the guide's error table name both causes. The Delete side (a stop +skipped because the listing lags) was not met: the run's Delete came a +minute after the start. + +## Rejected + +- **CR4 + S2, Checker repeats cluster-wide calls per node.** Spec entry 16 + makes each node's target self-contained, so a failure is reported + against the node whose round saw it. The cost is two extra requests per + extra node per round, and the configured cluster + has one node. Worth revisiting with a multi-node config, not before. +- **CR6, helpers copied from kvm2.** `permanentf`/`transientf` are one + line each; `lockName` is five and wraps `keyedlock.Acquire`'s error in + `provider.ErrTransient`. Moving that into `keyedlock` would couple a + generic lock package to the provider sentinels, and putting the helpers + in `internal/provider` would touch every kvm2 call site in a proxmox MR. +- **CR7, file I/O without a ctx.** `readToken` and the CA load read a + mounted Secret with `os.ReadFile`, which cannot be cancelled; a ctx + parameter would be unused. kvm2's `os.ReadFile` of the SSH key is the + precedent. Seed staging is `nocloud`'s, shared with kvm2 and unchanged by + the extraction. +- **CR10, `withDefaults` twice in `New`.** Startup only, idempotent, a few + small maps. `newProvider` applies it because the tests build providers + through it; defaulting once would need a second constructor shape. +- **S1, replace the `observe.go` decorator with one log line in `send`.** + The decorator is spec entries 17 and 20's `span-decorator` answer: each + seam call gets a child span and a V(1)/V(2) line with node, VMID and + UPID, which `send` does not know. It mirrors kvm2's decorator, and the + reflection test guarantees every seam method is covered. Swapping the + design in the review phase is out of proportion. +- **S3, `nocloud` exports used only by tests.** `SeedLabel` is read by + both kvm2's and proxmox's Create tests to recognise a real seed, and + `GuestAgentScript` by proxmox's `TestBuildSeed_table`, which tests + proxmox's own mapping of its config fields onto the vendor-data, not + `nocloud`'s composition. The exports are the price of cross-package + tests. + +## Checked and fine + +From the reviewer agent's report, verified in code: + +- The kvm2 extraction is a pure move. kvm2 still wraps every seed-build + error as Permanent, as before, and `keyedlock` is byte-identical to the + code it replaced. +- Delete stops or destroys only after the listing shows a qemu entry at + the VMID and the config carries the name and the mark; a reused VMID or + an unmarked VM is left alone. +- A missing VM inside Create is Transient, never NotFound; locks and + `status: unknown` are Transient, never NotFound. +- Create resumes at every step: the VMID comes from the UID, a nameless + locked entry is waited on and adopted, upload/config/resize are + repeatable, and the mark is written in the clone call. +- Secrets: `apiError` carries status and message only, the token never + reaches a `url.Error`, the Checker keeps only the token id, and the + cloud-init text appears only as `cloudInitBytes`. +- Every seam call takes the ctx; `pollUntil` honours ctx and budget; + `keyedlock.Acquire` gives up with ctx. +- Spec entries 1, 3, 4, 9, 11, 12, 13, 15 and 16 match the code (P2 was the + one disagreement, and the spec shared the flaw). +- Every §11 test name in spec entry 21 exists. + +The agent also saw kvm2's `TestCheck_hostsCheckedConcurrently` fail once +under `-race` (199.9 ms against its 200 ms floor). That file is unchanged +on this branch; the flake is parked, as before. diff --git a/docs/testing.md b/docs/testing.md index 5e29127..7f0c22e 100644 --- a/docs/testing.md +++ b/docs/testing.md @@ -71,9 +71,9 @@ manual runs too. | `internal/provider/keyedlock` | `keyedlock_test.go` | the per-name lock kvm2 and proxmox share: the entry is removed once uncontended, waiters share one lock, a waiter gives up with its ctx | pure | | `internal/provider/proxmox` | `id_test.go`, `mark_test.go`, `state_test.go`, `seed_test.go`, `errors_test.go`, `api_test.go` | pure pieces: providerID round-trip and malformed forms, the VMID derived from the UID (deterministic, in range, probing forward and wrapping), the description mark (keys, parsed from the captured clone, admin lines kept), state mapping and the IPv4 filter from captures, seed names and building (empty name Permanent without its payload, staging failure Transient); the `classify` table and the 500-message matchers against every captured error; the API types decoded from the captures | pure, over the responses Phase 0 captured from PVE 9.2.20 (`testdata/`) | | | `client_test.go` | the thin HTTP client: the token read from its file on every request (missing Transient, unparsable Permanent), `apiError` from status, reason phrase and body, the data envelope, method, path and parameters of every seam call, the clone and destroy query shapes, the volid's slash escaped, HTTP/1.1 only, request timeout and cancel Transient and the client unaffected, a child span under a parent and none without, the token never in errors or logs, the CA file trusted and loaded lazily, an x509 failure Permanent, insecure mode, the upload streamed as multipart | `httptest.NewTLSServer` replaying the captures byte for byte | -| | `create_test.go`, `get_test.go`, `delete_test.go`, `list_test.go`, `task_test.go`, `proxmox_test.go`, `provider_test.go`, `fake_test.go` | Create: resume after every partial step and after a crash mid-clone, the VMID probe and ownership check, the template's shape refused before any clone, resize as a task, the seed overwritten without a delete, the net0 bridge and VLAN rewrite keeping the MAC, returns only when running, cloud-init sentinel never logged; Get: lock checked before name, the agent's messages, paused from `status/current`, hibernated, a reused VMID, four calls when running; Delete: every already-gone and not-ours case, the seed deleted on the VM's node or on every node, hibernated Permanent, the per-name lock; ListByTag: candidates by tag or name prefix, the mark decides, an unreadable VM skipped unless ctx ended; task and lock waits; `New` makes no request; the contract sweep: every method returns within a second of a cancel (blocked on its first seam call, and mid-operation), every error wrapped with the configured name, op and id, logging tiers V0/V1/V2 with the no-op lines | in-memory `fakeAPI` behind the `api` seam, its error texts checked against the captures | +| | `create_test.go`, `get_test.go`, `delete_test.go`, `list_test.go`, `task_test.go`, `proxmox_test.go`, `provider_test.go`, `fake_test.go` | Create: resume after every partial step and after a crash mid-clone, the VMID probe and ownership check, the template's shape refused before any clone, resize as a task, the seed overwritten without a delete, the net0 bridge and VLAN rewrite keeping the MAC, returns only when running, cloud-init sentinel never logged; Get: lock checked before name, the agent's messages, paused from `status/current`, hibernated, a reused VMID, four calls when running; Delete: every already-gone and not-ours case, the seed deleted before the destroy on the VM's node or on every node, a hibernated VM destroyed, the per-name lock; ListByTag: candidates by tag or name prefix, the mark decides, an unreadable VM skipped unless ctx ended; task and lock waits; `New` makes no request; the contract sweep: every method returns within a second of a cancel (blocked on its first seam call, and mid-operation), every error wrapped with the configured name, op and id, logging tiers V0/V1/V2 with the no-op lines | in-memory `fakeAPI` behind the `api` seam, its error texts checked against the captures | | | `observe_test.go`, `check_test.go`, `provider_test.go` | the decorator: every seam method logs exactly one line (guarded by reflection), starts no span, identifiers at V1, detail at V2, upload body never logged, error fields; a Get through `provider.WithTracing` over the real client makes four `HTTP GET` spans, all children of `provider.get` and none carrying the token; health check: one entry per node in config order, nodes checked concurrently (a barrier, not a timer), expired ctx returns promptly, the reasons table at V2 with a sentinel secret, the transport reasons from real `net/http` and `crypto/tls` errors, detail keys | `fakeAPI` + `funcr` capture + span recorder; the real client over `httptest` for the tracing and transport-reason rows | -| | `proxmox_integration_test.go` (`-tags=integration`) | one real VM: `New` without a request, `Check` Ready with details, Create, idempotent re-Create with an unchanged VM count, Running with an IPv4 and the mark read back, ListByTag, Delete, NotFound, no VM and no seed left, HTTP spans under `provider.create`; `TestProxmoxCheck_realHost` runs the health check against the host and creates nothing. **Compiles, not yet run against a host** | a **real Proxmox VE host** — see below | +| | `proxmox_integration_test.go` (`-tags=integration`) | one real VM: `New` without a request, `Check` Ready with details, Create, idempotent re-Create with an unchanged VM count, Running with an IPv4 and the mark read back, ListByTag, Delete, NotFound, no VM and no seed left, HTTP spans under `provider.create`; `TestProxmoxCheck_realHost` runs the health check against the host and creates nothing. **Passed on pv1-1 (PVE 9.2.20), 2026-09-24**: Create 6.2 s, address 50 s later, whole run 61 s | a **real Proxmox VE host** — see below | | `internal/health` | `probe_test.go` | probes through a **real CONNECT-capable proxy stub** to a real TLS server: tunnel success, refused CONNECT, wrong status, dead proxy, plain-http forward | `httptest` + hijacked bidirectional tunnel | | | `engine_test.go` | thresholds, first-verdict, latency suppression matrix, dropped-event retry, stale-probe UID guard, tick scheduling/seeding/pruning, `Start` end-to-end | fake `client.Reader` + injected `probeFn` | | `internal/lease` | `store_test.go` | capacity, `MaxLeases=0`, all three selection tie-breaks, target-scoped vs global cooldowns, expiry via fake clock, report-on-retained-lease, idempotent release, **40 concurrent acquires never exceeding capacity** | injectable clock; the concurrency case is why `-race` matters | @@ -117,19 +117,21 @@ envtest suite the scaffold generated. `httptest` replaying responses captured from PVE 9.2.20. Whether a clone, config write, seed upload and start yield a guest that boots and reports an address is provable only on a real host (the proxmox - integration test below). Two kinds of path have met only the fake: + integration test below, which has passed on pv1-1). Two kinds of path + have met only the fake: - **the cross-node paths**: a clone that targets another node, the seed deleted on every node when the VM's node is unknown, a Check round over several nodes. The test host is a single node, so the integration test cannot reach them either ([proxmox-provider-spec.md](proxmox-provider-spec.md) entry 24); - - **answers never captured**: what the storage status call returns for a - missing or disabled storage (every 500 there is taken as - `StorageNotFound`), what destroy says for a hibernated VM, and a stop - on a paused VM. The provider does not depend on their wording: any - 500 from the storage status is `StorageNotFound`, a hibernated VM is - refused before destroy is called, and a paused VM gets the same stop - as a running one. What the fake answers there is its own guess. + - **an answer never captured**: the storage status for a storage the + cluster knows but cannot use on a node (disabled there, or offline). + Every 500 there is taken as `StorageNotFound`, and what the fake + answers is its own guess. The three answers Phase 0 missed were + captured on pv1-1 on 2026-09-24: a missing storage is a 400 with + `storage: No such storage.`, a hibernated VM is destroyed without + complaint (its suspend state volume with it), and a stop on a paused + VM ends `OK` with the VM stopped. ## The KVM integration test (real hypervisor) @@ -303,6 +305,7 @@ export PROXMOX_TEST_CA_FILE=$HOME/.config/proxmox-ca.pem # the node's pve-roo export PROXMOX_TEST_NODE=pv1-1 export PROXMOX_TEST_TEMPLATE_VMID=9001 export PROXMOX_TEST_ISO_STORAGE=proxy-seeds # holds iso content +export PROXMOX_TEST_VLAN_TAG=4 # template 9001's net0 is untagged; proxies live on VLAN 4 make test-integration-proxmox ``` @@ -381,10 +384,26 @@ variables; to run only this test: go test -tags=integration -count=1 -v -run TestProxmoxCheck ./internal/provider/proxmox/ ``` -**Neither test has run against a host yet.** Both compile -(`go vet -tags=integration ./...`); Phase 10 of the plan runs them on -pv1-1. Like the kvm tests they are not part of `make test` and not run in -CI. +**Last run: 2026-09-24 on pv1-1** (Proxmox VE 9.2.20, kernel +7.0.14-17-pve, template 9001, `proxy-seeds`, VLAN tag 4). Both tests +passed; `go test` reported 60.9 s for the package. + +| Step | Time | +|---|---| +| `Check` | 141 ms (125 ms in `TestProxmoxCheck_realHost`) | +| clone task, submit to done | about 3 s (seen done at the third 1 s poll) | +| `Create`, clone to started VM | 6.2 s | +| second `Create` (adoption, nothing created) | 32 ms | +| Create's return to the first address | 50.2 s (the agent install from the internet) | +| `Delete` (stop, seed delete, destroy), `Get`, second `Delete` | 3.1 s | +| `TestProxmox_lifecycle` | 59.7 s | + +The first `Get`, 32 ms after the second Create, was Transient: the +listing still showed the new VM with `status: unknown`. The next poll 5 s +later saw it running. The Delete deleted the seed while the stopped VM's +`ide2` still referenced it, then destroyed the VM, and step 10 found +neither left. Like the kvm tests they are not part of `make test` and not +run in CI. ## The scaffolded `make test-e2e` suite diff --git a/internal/provider/proxmox/check.go b/internal/provider/proxmox/check.go index a4b7836..bb0e62a 100644 --- a/internal/provider/proxmox/check.go +++ b/internal/provider/proxmox/check.go @@ -111,16 +111,22 @@ func (p *Provider) probeNode(ctx context.Context, node string, details map[strin return p.checkTemplates(ctx, details) } -// checkStorage reads isoStorage on the node. A 500 means Proxmox cannot -// use that storage on this node (it does not exist, is not enabled here, -// or is offline), whatever its wording; the message carries Proxmox's -// text. +// checkStorage reads isoStorage on the node. A storage the cluster does +// not know is a 400 on the storage parameter (captured). A 500 is taken +// the same way, whatever its wording: Proxmox cannot use that storage on +// this node (not enabled here, or offline; not captured). The message +// carries Proxmox's text. func (p *Provider) checkStorage(ctx context.Context, node string, details map[string]string) error { storage := p.cfg.ISOStorage st, err := p.api.StorageStatus(ctx, node, storage) var ae *apiError - if errors.As(err, &ae) && ae.Status == http.StatusInternalServerError { - return &checkFailure{reason: reasonStorageNotFound, msg: fmt.Sprintf("storage %s on node %s: %s", storage, node, ae.Message)} + if errors.As(err, &ae) { + switch { + case isNoSuchStorage(ae): + return &checkFailure{reason: reasonStorageNotFound, msg: fmt.Sprintf("storage %s on node %s: %s", storage, node, ae.Errors["storage"])} + case ae.Status == http.StatusInternalServerError: + return &checkFailure{reason: reasonStorageNotFound, msg: fmt.Sprintf("storage %s on node %s: %s", storage, node, ae.Message)} + } } if err != nil { return fmt.Errorf("reading storage %s on node %s: %w", storage, node, err) diff --git a/internal/provider/proxmox/check_test.go b/internal/provider/proxmox/check_test.go index 1e3bcbe..2984501 100644 --- a/internal/provider/proxmox/check_test.go +++ b/internal/provider/proxmox/check_test.go @@ -230,7 +230,21 @@ func TestCheck_reasonsTable(t *testing.T) { fx.fake.mu.Unlock() }, wantReason: reasonStorageNotFound, - wantMessage: "storage proxy-seeds on node pv1-1: fake: storage 'proxy-seeds' does not exist", + wantMessage: "storage proxy-seeds on node pv1-1: No such storage.", + }, + { + // Not captured: a storage Proxmox knows but cannot use here. + name: "iso storage unusable on the node", + prep: func(fx *fixture) { + fx.fake.failWith("StorageStatus", fakeErr(500, "fake: storage 'proxy-seeds' is not online")) + }, + wantReason: reasonStorageNotFound, + wantMessage: "storage proxy-seeds on node pv1-1: fake: storage 'proxy-seeds' is not online", + }, + { + name: "another bad parameter is not a missing storage", + prep: func(fx *fixture) { fx.fake.failWith("StorageStatus", loadAPIError(fx.t, "err_bad_param")) }, + wantReason: provider.ReasonAPIError, }, { name: "iso storage inactive", diff --git a/internal/provider/proxmox/create.go b/internal/provider/proxmox/create.go index df4c7ae..ad6d458 100644 --- a/internal/provider/proxmox/create.go +++ b/internal/provider/proxmox/create.go @@ -198,9 +198,10 @@ func (p *Provider) template(ctx context.Context, rs []vmResource, image string, } // allocate is the probe of spec entry 2: the attempt-th VMID for the UID -// is cloned onto when the listing shows it free and put through the +// is cloned onto when the listing shows it free, and put through the // ownership check when it is taken, or when the clone finds it taken after -// all (F4). The boolean reports an adoption rather than a fresh clone. +// all (F4), by a VM that is nameless or carries our name. The boolean +// reports an adoption rather than a fresh clone. func (p *Provider) allocate(ctx context.Context, log logr.Logger, rs []vmResource, node string, src vmResource, req provider.CreateRequest) (ownedVM, bool, error) { taken := make(map[int]vmResource, len(rs)) for _, r := range rs { @@ -225,7 +226,9 @@ func (p *Provider) allocate(ctx context.Context, log logr.Logger, rs []vmResourc continue } } - if r.Type == "qemu" && r.Template == 0 { + // A listed name other than ours is someone else's VM whatever its + // lock: only a clone in progress is nameless. + if r.Type == "qemu" && r.Template == 0 && (r.Name == "" || r.Name == req.Name) { vm, ours, err := p.inspect(ctx, r, req) if err != nil || ours { return vm, ours, err diff --git a/internal/provider/proxmox/create_test.go b/internal/provider/proxmox/create_test.go index cce31f4..6b4e7dc 100644 --- a/internal/provider/proxmox/create_test.go +++ b/internal/provider/proxmox/create_test.go @@ -248,10 +248,22 @@ func TestCreate_takenVMIDInspectedBeforeProbing(t *testing.T) { wantConfigReads int }{ { - name: "foreign VM probes on after its config is read", + name: "foreign VM probes on without a read", setup: func(fx *fixture) { fx.fake.addVM(slot(0), "someone-else", "running") }, wantAttempt: 1, - wantConfigReads: 2, + wantConfigReads: 1, + }, + { + // Review finding P2: another VM's lock (a backup, a snapshot) + // can outlast the clone budget, and every retry lands on the + // same VMID. + name: "foreign VM under a lock probes on without waiting", + setup: func(fx *fixture) { + fx.fake.addVM(slot(0), "someone-else", "running").lock = "snapshot-delete" + fx.p.budgets.clone = 2 * time.Second + }, + wantAttempt: 1, + wantConfigReads: 1, }, { name: "container probes on without a read", @@ -581,8 +593,8 @@ func TestCreate_vmidRangeExhaustedIsTransient(t *testing.T) { if got := fx.fake.callCount("Clone"); got != 0 { t.Errorf("Clone calls = %d, want 0", got) } - if got := fx.fake.callCount("VMConfig"); got != 5 { - t.Errorf("VMConfig calls = %d, want the template and each of the 4 VMIDs once", got) + if got := fx.fake.callCount("VMConfig"); got != 1 { + t.Errorf("VMConfig calls = %d, want only the template's: the listed names are foreign", got) } } diff --git a/internal/provider/proxmox/delete.go b/internal/provider/proxmox/delete.go index 309324e..94600d9 100644 --- a/internal/provider/proxmox/delete.go +++ b/internal/provider/proxmox/delete.go @@ -2,14 +2,17 @@ package proxmox import ( "context" + "errors" "fmt" "time" "github.com/go-logr/logr" + + "gitea.home.hrajfrisbee.cz/kacerr/egress-proxies-operator/internal/provider" ) -// Delete is spec entry 11: under the per-name lock, stop and destroy the VM -// when it is ours, then delete its seed. Everything already gone is +// Delete is spec entry 11: under the per-name lock, stop the VM when it is +// ours, delete its seed, then destroy it. Everything already gone is // success, and nothing of ours is left when nil is returned, so Get // afterwards is NotFound. func (p *Provider) Delete(ctx context.Context, id string) error { @@ -29,19 +32,17 @@ func (p *Provider) Delete(ctx context.Context, id string) error { return p.wrapErr("delete", id, err) } if node == "" { - err = p.deleteSeedEverywhere(ctx, log, name) + if err := p.deleteSeedEverywhere(ctx, log, name); err != nil { + return p.wrapErr("delete", id, err) + } } else { log = log.WithValues("node", node) - err = p.deleteSeed(ctx, log, node, name) - } - if err != nil { - return p.wrapErr("delete", id, err) } log.V(1).Info("proxmox vm deleted", "destroyed", node != "") return nil } -// destroy is steps 2 to 6. It returns the node of the VM it destroyed, or +// destroy is steps 2 to 7. It returns the node of the VM it destroyed, or // "" when there was no VM of ours to destroy: then the seed's node is not // known, and the seed is looked for on every node. func (p *Provider) destroy(ctx context.Context, log logr.Logger, vmid int, name string) (string, error) { @@ -58,7 +59,7 @@ func (p *Provider) destroy(ctx context.Context, log logr.Logger, vmid int, name return "", transientf("VM %d is locked (%s)", vmid, r.Lock) } if r.Status == "unknown" { - return "", transientf("VM %d has status unknown: its node %s is not answering", vmid, r.Node) + return "", transientf("VM %d has status unknown: its node %s is not answering, or the VM is too new to be reported yet", vmid, r.Node) } cfg, err := p.api.VMConfig(ctx, r.Node, vmid) if isNotExist(err) { @@ -76,47 +77,47 @@ func (p *Provider) destroy(ctx context.Context, log logr.Logger, vmid int, name log.V(1).Info("proxmox vm carries no proxy-operator mark, left alone") return "", nil } - switch lock := cfg["lock"]; lock { - case "": - case lockSuspended: - // Destroy refuses a locked VM and the lock never clears on its own, - // so only an admin can move this on, as in Create's ownership check. - return "", permanentf("VM %d named %s is hibernated (lock suspended); resume or destroy it in Proxmox", vmid, name) - default: + // A hibernated VM is stopped with lock suspended, and the destroy + // takes it, state volume included (captured on pv1-1). + if lock := cfg["lock"]; lock != "" && lock != lockSuspended { return "", transientf("VM %d is locked (%s)", vmid, lock) } // The listing shows a paused VM as running, and Stop ends it too. if r.Status != "stopped" { - gone, err := p.runTask(ctx, log, "stopping", vmid, p.budgets.stop, func() (string, error) { return p.api.Stop(ctx, r.Node, vmid) }) - if err != nil || gone { - return r.Node, err + if err := p.runTask(ctx, log, "stopping", vmid, p.budgets.stop, func() (string, error) { return p.api.Stop(ctx, r.Node, vmid) }); err != nil { + return "", err } } - if _, err := p.runTask(ctx, log, "destroying", vmid, p.budgets.destroy, func() (string, error) { return p.api.Destroy(ctx, r.Node, vmid) }); err != nil { + // The seed goes while the VM is still there: once it is gone, Get is + // NotFound and nothing calls Delete again to finish the job. + if err := p.deleteSeed(ctx, log, r.Node, name); err != nil { + return "", err + } + if err := p.runTask(ctx, log, "destroying", vmid, p.budgets.destroy, func() (string, error) { return p.api.Destroy(ctx, r.Node, vmid) }); err != nil { return "", err } return r.Node, nil } // runTask submits a task on a VM and waits for it. A VM that is gone by the -// time the task is submitted reports gone: it is what the task was for. -func (p *Provider) runTask(ctx context.Context, log logr.Logger, what string, vmid int, budget time.Duration, submit func() (string, error)) (bool, error) { +// time the task is submitted is success: it is what the task was for. +func (p *Provider) runTask(ctx context.Context, log logr.Logger, what string, vmid int, budget time.Duration, submit func() (string, error)) error { upid, err := submit() if isNotExist(err) { log.V(1).Info("proxmox vm already gone") - return true, nil + return nil } if err != nil { - return false, fmt.Errorf("%s VM %d: %w", what, vmid, err) + return fmt.Errorf("%s VM %d: %w", what, vmid, err) } if err := p.waitTask(ctx, log, upid, budget); err != nil { - return false, fmt.Errorf("%s VM %d: %w", what, vmid, err) + return fmt.Errorf("%s VM %d: %w", what, vmid, err) } - return false, nil + return nil } -// deleteSeed is step 7 on one node. A missing volume still ends in a +// deleteSeed is step 6 on one node. A missing volume still ends in a // successful task (F13), so nothing is matched. func (p *Provider) deleteSeed(ctx context.Context, log logr.Logger, node, name string) error { volid := seedVolID(p.cfg.ISOStorage, name) @@ -130,17 +131,19 @@ func (p *Provider) deleteSeed(ctx context.Context, log logr.Logger, node, name s return nil } -// deleteSeedEverywhere is step 7 when the VM's node is not known: every +// deleteSeedEverywhere is step 6 when the VM's node is not known: every // configured node is tried, and one that fails is logged and skipped, so a // node that is down or lacks the storage cannot hold the Delete up. An -// ended ctx fails the call instead. +// ended ctx fails the call instead, and so does a Permanent refusal such as +// a 403 for Datastore.Allocate, which every node would give and which +// would otherwise leave the seed behind unseen. func (p *Provider) deleteSeedEverywhere(ctx context.Context, log logr.Logger, name string) error { for _, node := range p.cfg.Nodes { err := p.deleteSeed(ctx, log, node, name) if err == nil { continue } - if ctx.Err() != nil { + if ctx.Err() != nil || errors.Is(classify("delete", err), provider.ErrPermanent) { return err } log.V(1).Info("proxmox seed delete failed on a node, skipped", "node", node, "reason", err.Error()) diff --git a/internal/provider/proxmox/delete_test.go b/internal/provider/proxmox/delete_test.go index b73d936..0b4c817 100644 --- a/internal/provider/proxmox/delete_test.go +++ b/internal/provider/proxmox/delete_test.go @@ -203,6 +203,14 @@ func TestDelete_unmarkedLeavesVMAlone(t *testing.T) { // wait for the stop fails. func TestDelete_runningVMStoppedFirst(t *testing.T) { t.Parallel() + // pv1-1 took a stop on a paused VM as a task that ended OK with the VM + // stopped (Phase 10 captures), which is what the fake does for all three. + accepted := readCapture(t, "stop_paused_response") + done := decodeData[taskStatus](t, "task_status_stop_paused") + after := decodeData[vmStatus](t, "status_current_after_stop_paused") + if accepted.Status != http.StatusOK || done.Type != "qmstop" || done.ExitStatus != "OK" || after.Status != "stopped" || after.QMPStatus != "stopped" { + t.Fatalf("captures = %d, %+v, %+v; want the stop accepted, OK, and the VM stopped", accepted.Status, done, after) + } for _, qmp := range []string{"running", "paused", "prelaunch"} { t.Run(qmp, func(t *testing.T) { t.Parallel() @@ -215,8 +223,8 @@ func TestDelete_runningVMStoppedFirst(t *testing.T) { if stop < 0 || destroy < stop { t.Fatalf("calls = %v, want Stop before Destroy", log) } - if n := callsBefore(log[stop:], "TaskStatus", "Destroy"); n != fx.fake.taskPolls+1 { - t.Errorf("stop task polled %d times before Destroy, want %d", n, fx.fake.taskPolls+1) + if n := callsBefore(log[stop:], "TaskStatus", "DeleteVolume"); n != fx.fake.taskPolls+1 { + t.Errorf("stop task polled %d times before the seed delete, want %d", n, fx.fake.taskPolls+1) } if _, ok := fx.fake.vmSnapshot(testVMID); ok { t.Error("VM still exists") @@ -230,7 +238,7 @@ func TestDelete_stoppedVMSkipsStop(t *testing.T) { fx := newFixture(t, 1) fx.addOurVM(testVMID, testName, "stopped") fx.delete() - want := []string{"ClusterResources", "VMConfig", "Destroy", "TaskStatus", "DeleteVolume", "TaskStatus"} + want := []string{"ClusterResources", "VMConfig", "DeleteVolume", "TaskStatus", "Destroy", "TaskStatus"} if got := fx.fake.callLog(); !slices.Equal(got, want) { t.Errorf("calls = %v, want %v", got, want) } @@ -291,26 +299,32 @@ func TestDelete_unknownStatusIsTransient(t *testing.T) { } } -// TestDelete_hibernatedVMIsPermanentIfOurs matches Create's ownership -// check: Destroy refuses the suspended lock, which never clears on its own, -// so an admin must act. A hibernated VM that is not ours is left alone. -func TestDelete_hibernatedVMIsPermanentIfOurs(t *testing.T) { +// TestDelete_hibernatedVMIsDestroyedIfOurs: on pv1-1 the destroy took a +// hibernated VM, suspend state volume included, and its task ended OK. +// Delete used to refuse it as Permanent on the belief that the destroy +// would. A hibernated VM that is not ours is still left alone. +func TestDelete_hibernatedVMIsDestroyedIfOurs(t *testing.T) { t.Parallel() t.Run("ours", func(t *testing.T) { t.Parallel() + accepted := readCapture(t, "destroy_hibernated_response") + done := decodeData[taskStatus](t, "task_status_destroy_hibernated") + if accepted.Status != http.StatusOK || done.Type != "qmdestroy" || done.ExitStatus != "OK" { + t.Fatalf("captures = %d, %+v; want the destroy accepted and its task OK", accepted.Status, done) + } fx := newFixture(t, 1) fx.addOurVM(testVMID, testName, "stopped").lock = lockSuspended fx.putSeed() - if msg := fx.deleteFails(provider.ErrPermanent); !strings.Contains(msg, "resume or destroy it in Proxmox") { - t.Errorf("error = %s, want the admin action", msg) + fx.delete() + want := []string{"ClusterResources", "VMConfig", "DeleteVolume", "TaskStatus", "Destroy", "TaskStatus"} + if got := fx.fake.callLog()[2:]; !slices.Equal(got, want) { + t.Errorf("calls = %v, want %v: no stop, since the VM is not running", got, want) } - for _, method := range []string{"Stop", "Destroy", "DeleteVolume"} { - if n := fx.fake.callCount(method); n != 0 { - t.Errorf("%s calls = %d, want 0", method, n) - } + if _, ok := fx.fake.vmSnapshot(testVMID); ok { + t.Error("hibernated VM still exists") } - if _, err := fx.fake.Destroy(fx.ctx, fakeNode, testVMID); !isLocked(err) { - t.Errorf("fake Destroy on the hibernated VM error = %v, want the lock answer", err) + if fx.seedPresent() { + t.Error("seed still exists") } }) t.Run("someone else's", func(t *testing.T) { @@ -367,7 +381,7 @@ func TestDelete_seedDeleteIsTaskAndIdempotent(t *testing.T) { for log[last] != "DeleteVolume" { last-- } - if n := callsBefore(log[last:], "TaskStatus", ""); n != fx.fake.taskPolls+1 { + if n := callsBefore(log[last:], "TaskStatus", "Destroy"); n != fx.fake.taskPolls+1 { t.Errorf("seed delete task polled %d times, want %d: %v", n, fx.fake.taskPolls+1, log) } if fx.seedPresent() { @@ -377,6 +391,9 @@ func TestDelete_seedDeleteIsTaskAndIdempotent(t *testing.T) { } } +// TestDelete_seedDeleteFailureOnVMNodeFails: the VM outlives a failed seed +// delete, so Get keeps finding it and the next Delete finishes the job with +// the same class of answer (review findings CR1, CR2, P1). func TestDelete_seedDeleteFailureOnVMNodeFails(t *testing.T) { t.Parallel() tests := []struct { @@ -403,9 +420,22 @@ func TestDelete_seedDeleteFailureOnVMNodeFails(t *testing.T) { fx := newFixture(t, 1) fx.addOurVM(testVMID, testName, "stopped") tc.setup(fx) + fx.putSeed() fx.deleteFails(tc.class) + if n := fx.fake.callCount("Destroy"); n != 0 { + t.Errorf("Destroy calls = %d, want 0 while the seed is still there", n) + } + if _, err := fx.p.Get(fx.ctx, fx.id()); errors.Is(err, provider.ErrNotFound) { + t.Fatalf("Get = %v, want the VM still found so Delete runs again", err) + } + fx.deleteFails(tc.class) + fx.fake.clearHooks() + fx.delete() if _, ok := fx.fake.vmSnapshot(testVMID); ok { - t.Error("VM still exists: the destroy should have run before the seed delete") + t.Error("VM still exists after the retry") + } + if fx.seedPresent() { + t.Error("seed still exists after the retry") } }) } @@ -436,6 +466,18 @@ func TestDelete_seedDeleteNodeFailureSkippedWhenNodeUnknown(t *testing.T) { t.Error("seed still exists after pv1-2's delete") } }) + t.Run("a refused privilege fails the call", func(t *testing.T) { + t.Parallel() + fx := twoNodeFixture(t) + fx.putSeed() + fx.fake.failWith("DeleteVolume", loadAPIError(t, "err_forbidden_volume_delete")) + if msg := fx.deleteFails(provider.ErrPermanent); !strings.Contains(msg, "Datastore.Allocate") { + t.Errorf("error = %s, want the missing privilege named", msg) + } + if n := fx.fake.callCount("DeleteVolume"); n != 1 { + t.Errorf("DeleteVolume calls = %d, want 1: every node would refuse the same", n) + } + }) t.Run("ended ctx fails the call", func(t *testing.T) { t.Parallel() fx := twoNodeFixture(t) @@ -465,8 +507,8 @@ func TestDelete_destroyTaskFailureIsTransient(t *testing.T) { if _, ok := fx.fake.vmSnapshot(testVMID); !ok { t.Error("VM gone after a failed destroy task") } - if !fx.seedPresent() { - t.Error("seed deleted although the VM is still there") + if fx.seedPresent() { + t.Error("seed still exists: it goes before the destroy") } if lines := fx.logs.matching(`"msg"="proxmox task failed"`); len(lines) != 1 { t.Errorf("task failed lines = %v, want one", lines) @@ -485,7 +527,7 @@ func TestDelete_goneMidwayIsSuccess(t *testing.T) { wantNodes []string }{ {name: "before the config read", status: "stopped", method: "VMConfig", wantNodes: []string{fakeNode, "pv1-2"}}, - {name: "before the stop", status: "running", method: "Stop", wantNodes: []string{fakeNode}}, + {name: "before the stop", status: "running", method: "Stop", wantDestroy: 1, wantNodes: []string{fakeNode}}, {name: "before the destroy", status: "stopped", method: "Destroy", wantDestroy: 1, wantNodes: []string{fakeNode}}, } for _, tc := range tests { diff --git a/internal/provider/proxmox/errors.go b/internal/provider/proxmox/errors.go index 3dbf074..4921aeb 100644 --- a/internal/provider/proxmox/errors.go +++ b/internal/provider/proxmox/errors.go @@ -76,14 +76,13 @@ func newAPIError(status int, reason string, body []byte) *apiError { // under testdata/. var ( vmConfigMissing = regexp.MustCompile(`^Configuration file '[^']*' does not exist$`) - vmNotRunning = regexp.MustCompile(`^VM [0-9]+ is not running$`) ) const ( - msgLocked = "VM is locked" msgAlreadyExists = "config file already exists" msgAgentNotRunning = "QEMU guest agent is not running" msgAgentNotConfigured = "No QEMU guest agent configured" + msgNoSuchStorage = "No such storage." ) // server500 returns the message of err when it is, or wraps, an apiError @@ -102,10 +101,12 @@ func isNotExist(err error) bool { return ok && vmConfigMissing.MatchString(msg) } -// isLocked reports a VM held by a running task ("VM is locked (clone)"). -func isLocked(err error) bool { - msg, ok := server500(err) - return ok && strings.HasPrefix(msg, msgLocked) +// isNoSuchStorage reports a storage the cluster does not know: a 400 whose +// storage field error is exactly msgNoSuchStorage. Other storage field +// errors, such as a clone onto a storage without image support, are not. +func isNoSuchStorage(err error) bool { + var ae *apiError + return errors.As(err, &ae) && ae.Status == http.StatusBadRequest && ae.Errors["storage"] == msgNoSuchStorage } // isAlreadyExists reports a clone onto a VMID that is taken. @@ -127,14 +128,6 @@ func isAgentNotConfigured(err error) bool { return ok && msg == msgAgentNotConfigured } -// isVMNotRunning reports an agent call on a VM that is stopped. It is -// matched on the whole message because " is not running" alone is also a -// suffix of the agent-not-running answer. -func isVMNotRunning(err error) bool { - msg, ok := server500(err) - return ok && vmNotRunning.MatchString(msg) -} - // classify maps any error reaching the provider's boundary to one of the // four contract sentinels, per spec entry 13. op is the contract operation: // a vanished VM inside Create is Transient, because the contract forbids diff --git a/internal/provider/proxmox/errors_test.go b/internal/provider/proxmox/errors_test.go index f3e8fc3..4a28d02 100644 --- a/internal/provider/proxmox/errors_test.go +++ b/internal/provider/proxmox/errors_test.go @@ -93,6 +93,7 @@ func TestAPIError_matchers(t *testing.T) { "isAlreadyExists": isAlreadyExists, "isAgentNotRunning": isAgentNotRunning, "isAgentNotConfigured": isAgentNotConfigured, + "isNoSuchStorage": isNoSuchStorage, "isVMNotRunning": isVMNotRunning, } tests := []struct { @@ -104,6 +105,7 @@ func TestAPIError_matchers(t *testing.T) { {"err_destroy_missing", "isNotExist"}, {"err_config_while_clone_locked", "isLocked"}, {"err_newid_in_use", "isAlreadyExists"}, + {"storage_status_missing", "isNoSuchStorage"}, {"err_agent_not_running", "isAgentNotRunning"}, {"err_agent_vm_stopped", "isAgentNotConfigured"}, {"err_agent_vm_stopped_agent_enabled", "isVMNotRunning"}, @@ -184,12 +186,16 @@ func TestAPIError_errorString(t *testing.T) { } } -func TestPermanentfAndTransientf(t *testing.T) { +func TestPermanentf_wrapsClassWithMessage(t *testing.T) { t.Parallel() perm := permanentf("node %q missing", "x") if !errors.Is(perm, provider.ErrPermanent) || perm.Error() != `provider: permanent error: node "x" missing` { t.Errorf("permanentf() = %q, want ErrPermanent with the message", perm) } +} + +func TestTransientf_wrapsClassWithMessage(t *testing.T) { + t.Parallel() trans := transientf("retry %d", 3) if !errors.Is(trans, provider.ErrTransient) || trans.Error() != "provider: transient error: retry 3" { t.Errorf("transientf() = %q, want ErrTransient with the message", trans) diff --git a/internal/provider/proxmox/fake_test.go b/internal/provider/proxmox/fake_test.go index 594f569..927c2af 100644 --- a/internal/provider/proxmox/fake_test.go +++ b/internal/provider/proxmox/fake_test.go @@ -6,6 +6,7 @@ import ( "fmt" "io" "maps" + "regexp" "slices" "strconv" "strings" @@ -140,6 +141,13 @@ func (f *fakeAPI) hookOn(method string, fn func() error) { f.hooks[method] = fn } +// clearHooks drops every hook, so the faults they injected stop. +func (f *fakeAPI) clearHooks() { + f.mu.Lock() + defer f.mu.Unlock() + clear(f.hooks) +} + func (f *fakeAPI) blockOn(method string) chan struct{} { f.mu.Lock() defer f.mu.Unlock() @@ -191,8 +199,30 @@ func errVMMissing(node string, vmid int) *apiError { return fakeErr(500, fmt.Sprintf("Configuration file 'nodes/%s/qemu-server/%d.conf' does not exist", node, vmid)) } +// errNoSuchStorage is the answer for a storage the cluster does not know +// (Phase 10 capture storage_status_missing). +func errNoSuchStorage() *apiError { + return newAPIError(400, "", []byte(`{"message":"Parameter verification failed.\n","data":null,"errors":{"storage":"No such storage."}}`)) +} + func errLocked(lock string) *apiError { return fakeErr(500, "VM is locked ("+lock+")") } +// isLocked and isVMNotRunning recognise two answers the provider leaves to +// classify's Transient default. They live here only to hold the fake's +// wording to the captures. The stopped-VM message is matched whole because +// " is not running" is also a suffix of the agent-not-running answer. +var vmNotRunning = regexp.MustCompile(`^VM [0-9]+ is not running$`) + +func isLocked(err error) bool { + msg, ok := server500(err) + return ok && strings.HasPrefix(msg, "VM is locked") +} + +func isVMNotRunning(err error) bool { + msg, ok := server500(err) + return ok && vmNotRunning.MatchString(msg) +} + func errNewIDInUse(vmid int) *apiError { return fakeErr(500, fmt.Sprintf("unable to create VM %d: config file already exists", vmid)) } @@ -272,7 +302,7 @@ func (f *fakeAPI) StorageStatus(ctx context.Context, node, storage string) (stor defer f.mu.Unlock() s, ok := f.storages[storage] if !ok { - return storageStatus{}, fakeErr(500, "fake: storage '"+storage+"' does not exist") + return storageStatus{}, errNoSuchStorage() } return s, nil } @@ -505,7 +535,9 @@ func (f *fakeAPI) Destroy(ctx context.Context, node string, vmid int) (string, e if err != nil { return "", err } - if vm.lock != "" { + // A hibernated VM is destroyed like a stopped one (Phase 10 captures + // destroy_hibernated_response and task_status_destroy_hibernated). + if vm.lock != "" && vm.lock != lockSuspended { return "", errLocked(vm.lock) } if vm.status == "running" { @@ -666,6 +698,7 @@ func TestFakeAPI_errorsMatchCaptures(t *testing.T) { {capture: "err_destroy_missing", fake: errVMMissing("pv1-1", 199999), matcher: isNotExist}, {capture: "err_config_while_clone_locked", fake: errLocked("clone"), matcher: isLocked}, {capture: "err_newid_in_use", fake: errNewIDInUse(199999), matcher: isAlreadyExists}, + {capture: "storage_status_missing", fake: errNoSuchStorage(), matcher: isNoSuchStorage}, {capture: "err_agent_vm_stopped_agent_enabled", fake: errVMNotRunning(199999), matcher: isVMNotRunning}, {capture: "err_agent_not_running", fake: fakeErr(500, msgAgentNotRunning), matcher: isAgentNotRunning}, {capture: "err_agent_vm_stopped", fake: fakeErr(500, msgAgentNotConfigured), matcher: isAgentNotConfigured}, @@ -674,8 +707,8 @@ func TestFakeAPI_errorsMatchCaptures(t *testing.T) { t.Run(tc.capture, func(t *testing.T) { t.Parallel() want := loadAPIError(t, tc.capture) - if tc.fake.Status != want.Status || tc.fake.Message != want.Message { - t.Errorf("fake error = %d %q, capture = %d %q", tc.fake.Status, tc.fake.Message, want.Status, want.Message) + if tc.fake.Status != want.Status || tc.fake.Message != want.Message || !maps.Equal(tc.fake.Errors, want.Errors) { + t.Errorf("fake error = %d %q %v, capture = %d %q %v", tc.fake.Status, tc.fake.Message, tc.fake.Errors, want.Status, want.Message, want.Errors) } if !tc.matcher(tc.fake) { t.Errorf("matcher does not accept the fake's %q", tc.fake.Message) diff --git a/internal/provider/proxmox/get.go b/internal/provider/proxmox/get.go index b0a2379..0beb2b7 100644 --- a/internal/provider/proxmox/get.go +++ b/internal/provider/proxmox/get.go @@ -49,7 +49,7 @@ func (p *Provider) get(ctx context.Context, log logr.Logger, vmid int, name stri return nil, transientf("VM %d is locked (%s)", vmid, r.Lock) } if r.Status == "unknown" { - return nil, transientf("VM %d has status unknown: its node %s is not answering", vmid, r.Node) + return nil, transientf("VM %d has status unknown: its node %s is not answering, or the VM is too new to be reported yet", vmid, r.Node) } cfg, err := p.api.VMConfig(ctx, r.Node, vmid) if err != nil { @@ -78,7 +78,7 @@ func (p *Provider) get(ctx context.Context, log logr.Logger, vmid int, name stri if err != nil { return nil, fmt.Errorf("reading status of VM %d: %w", vmid, err) } - if st.Status == "running" && (st.QMPStatus == "running" || st.QMPStatus == "") { + if guestRunning(st.Status, st.QMPStatus) { ip, state, err := p.agentAddress(ctx, log, r.Node, vmid) if err != nil { return nil, err diff --git a/internal/provider/proxmox/get_test.go b/internal/provider/proxmox/get_test.go index 39d672d..7b0f7c5 100644 --- a/internal/provider/proxmox/get_test.go +++ b/internal/provider/proxmox/get_test.go @@ -341,8 +341,10 @@ func TestGet_unknownStatusIsTransient(t *testing.T) { fx := newFixture(t, 1) fx.addOurVM(testVMID, testName, "running").status = "unknown" msg := assertGetFails(t, fx, provider.ErrTransient) - if !strings.Contains(msg, "pv1-1") { - t.Errorf("error = %s, want it to name the node", msg) + // On pv1-1 the first Get after Create met this with the node up: the + // listing had not reported the new VM yet (Phase 10 run, finding P6). + if !strings.Contains(msg, "pv1-1") || !strings.Contains(msg, "too new to be reported yet") { + t.Errorf("error = %s, want it to name the node and both causes", msg) } if n := fx.fake.callCount("VMConfig"); n != 0 { t.Errorf("VMConfig calls = %d, want 0 while the node is unknown", n) diff --git a/internal/provider/proxmox/provider_test.go b/internal/provider/proxmox/provider_test.go index 0ed7dff..82c6ea7 100644 --- a/internal/provider/proxmox/provider_test.go +++ b/internal/provider/proxmox/provider_test.go @@ -310,13 +310,6 @@ func TestProvider_everyErrorIsWrappedWithConfiguredName(t *testing.T) { op: "delete", id: "vms/150042", class: provider.ErrPermanent, }, {name: "delete waiting on the name lock", setup: holdNameLock, call: del, op: "delete", id: "self", class: provider.ErrTransient}, - { - name: "delete hibernated vm", - setup: func(fx *fixture, _ context.CancelFunc) { - fx.addOurVM(testVMID, testName, "stopped").lock = lockSuspended - }, - call: del, op: "delete", id: "self", class: provider.ErrPermanent, - }, { name: "delete destroy task fails", setup: func(fx *fixture, _ context.CancelFunc) { @@ -579,7 +572,7 @@ func TestLogging_noOpLinesAtV1(t *testing.T) { name: "seed delete skipped on each node", setup: func(fx *fixture) { fx.p.cfg.Nodes = []string{fakeNode, "pv1-2"} - fx.fake.failWith("DeleteVolume", loadAPIError(fx.t, "err_forbidden_volume_delete")) + fx.fake.failWith("DeleteVolume", fakeErr(500, "fake: storage 'proxy-seeds' is not available on this node")) }, call: del, msg: "proxmox seed delete failed on a node, skipped", want: 2, }, diff --git a/internal/provider/proxmox/state.go b/internal/provider/proxmox/state.go index 4807d7a..dae4aef 100644 --- a/internal/provider/proxmox/state.go +++ b/internal/provider/proxmox/state.go @@ -13,22 +13,25 @@ import ( // is Stopped, which the reconciler answers by replacing the VM. An empty // qmpstatus is taken as running: the field is missing, not a state. func mapState(status, qmpStatus, ip string) provider.InstanceState { - if status != "running" { - return provider.StateStopped - } - switch qmpStatus { - case "running", "": + switch { + case guestRunning(status, qmpStatus): if ip == "" { return provider.StateProvisioning } return provider.StateRunning - case "prelaunch": + case status == "running" && qmpStatus == "prelaunch": return provider.StateProvisioning default: return provider.StateStopped } } +// guestRunning reports a VM whose guest runs, the only state in which its +// agent can answer and mapState can call it Running. +func guestRunning(status, qmpStatus string) bool { + return status == "running" && (qmpStatus == "running" || qmpStatus == "") +} + // firstIPv4 picks the first usable IPv4 from the guest agent's interfaces, // in the agent's order: not on the loopback interface, not loopback, // link-local or unspecified. Empty means no address yet. diff --git a/internal/provider/proxmox/testdata/destroy_hibernated_response.json b/internal/provider/proxmox/testdata/destroy_hibernated_response.json new file mode 100644 index 0000000..2a8f0f3 --- /dev/null +++ b/internal/provider/proxmox/testdata/destroy_hibernated_response.json @@ -0,0 +1,5 @@ +{ + "status": 200, + "reason": "OK", + "body": "{\"data\":\"UPID:pv1-1:0014F167:030AA126:6AB4DB09:qmdestroy:199999:proxy-operator@pve!operator:\"}" +} diff --git a/internal/provider/proxmox/testdata/status_current_after_stop_paused.json b/internal/provider/proxmox/testdata/status_current_after_stop_paused.json new file mode 100644 index 0000000..cc159b9 --- /dev/null +++ b/internal/provider/proxmox/testdata/status_current_after_stop_paused.json @@ -0,0 +1,5 @@ +{ + "status": 200, + "reason": "OK", + "body": "{\"data\":{\"name\":\"proxy-reconaaaaaaaaaaa\",\"agent\":1,\"mem\":0,\"maxmem\":2147483648,\"memhost\":0,\"serial\":1,\"maxdisk\":3221225472,\"vmid\":199999,\"ha\":{\"managed\":0},\"status\":\"stopped\",\"qmpstatus\":\"stopped\",\"netin\":0,\"uptime\":0,\"netout\":0,\"disk\":0,\"cpus\":2,\"cpu\":0,\"clipboard\":null}}" +} diff --git a/internal/provider/proxmox/testdata/stop_paused_response.json b/internal/provider/proxmox/testdata/stop_paused_response.json new file mode 100644 index 0000000..8630fbc --- /dev/null +++ b/internal/provider/proxmox/testdata/stop_paused_response.json @@ -0,0 +1,5 @@ +{ + "status": 200, + "reason": "OK", + "body": "{\"data\":\"UPID:pv1-1:0014F097:030A9433:6AB4DAE8:qmstop:199999:proxy-operator@pve!operator:\"}" +} diff --git a/internal/provider/proxmox/testdata/storage_status_missing.json b/internal/provider/proxmox/testdata/storage_status_missing.json new file mode 100644 index 0000000..279979b --- /dev/null +++ b/internal/provider/proxmox/testdata/storage_status_missing.json @@ -0,0 +1,5 @@ +{ + "status": 400, + "reason": "Parameter verification failed.", + "body": "{\"message\":\"Parameter verification failed.\\n\",\"data\":null,\"errors\":{\"storage\":\"No such storage.\"}}" +} diff --git a/internal/provider/proxmox/testdata/task_status_destroy_hibernated.json b/internal/provider/proxmox/testdata/task_status_destroy_hibernated.json new file mode 100644 index 0000000..75bb4a0 --- /dev/null +++ b/internal/provider/proxmox/testdata/task_status_destroy_hibernated.json @@ -0,0 +1,5 @@ +{ + "status": 200, + "reason": "OK", + "body": "{\"data\":{\"status\":\"stopped\",\"exitstatus\":\"OK\",\"upid\":\"UPID:pv1-1:0014F167:030AA126:6AB4DB09:qmdestroy:199999:proxy-operator@pve!operator:\",\"tokenid\":\"operator\",\"id\":\"199999\",\"starttime\":1790237449,\"type\":\"qmdestroy\",\"node\":\"pv1-1\",\"user\":\"proxy-operator@pve\",\"pstart\":51028262,\"pid\":1372519}}" +} diff --git a/internal/provider/proxmox/testdata/task_status_stop_paused.json b/internal/provider/proxmox/testdata/task_status_stop_paused.json new file mode 100644 index 0000000..03ad756 --- /dev/null +++ b/internal/provider/proxmox/testdata/task_status_stop_paused.json @@ -0,0 +1,5 @@ +{ + "status": 200, + "reason": "OK", + "body": "{\"data\":{\"upid\":\"UPID:pv1-1:0014F097:030A9433:6AB4DAE8:qmstop:199999:proxy-operator@pve!operator:\",\"status\":\"stopped\",\"exitstatus\":\"OK\",\"node\":\"pv1-1\",\"user\":\"proxy-operator@pve\",\"pstart\":51024947,\"pid\":1372311,\"id\":\"199999\",\"tokenid\":\"operator\",\"starttime\":1790237416,\"type\":\"qmstop\"}}" +}