egress-proxies-operator/docs/plans-executions/2026-09-18-1502-proxmox-provider.md

# Execution: proxmox, a Proxmox VE provider built from the provider contract

Pairs with [docs/plans/2026-09-18-1502-proxmox-provider.md](../plans/2026-09-18-1502-proxmox-provider.md).

## Status

- [x] Phase 0 — Plan file, branch, host recon, fixtures
- [x] Phase 1 — The spec (`docs/proxmox-provider-spec.md`)
- [x] Phase 2 — Config block, registration, package skeleton
- [x] Phase 3 — Extract `nocloud` and `keyedlock` from kvm2
- [x] Phase 4 — Seam, HTTP client, decorator, fake
- [x] Phase 5 — Get, ListByTag, task helpers
- [x] Phase 6 — Create
- [x] Phase 7 — Delete, Checker, CRD comment
- [x] Phase 8 — Contract and observability sweep, integration test, testing.md
- [x] Phase 9 — Deployment and documentation
- [x] Phase 10 — Review, real-host validation, changelog, MR

## Phase 0 — Plan file, branch, host recon, fixtures

The plan was committed alone on `main` on 2026-09-18 (`708989c`, then
`2167819` retargeting it at PVE 9.2). `origin/main` had not moved in the
meantime, so the branch was cut straight from local `main`:

```bash
git fetch -q origin && git rev-list --left-right --count origin/main...main   # 0 2
git switch -c feat/proxmox-provider
```

The host is **pv1-1**, a single node (no cluster peers), `pve-manager/9.2.20`,
kernel `7.0.14-17-pve`, QEMU 11.0. The user prepared the token
(`~/.config/proxmox-token`, privsep off) and the CA (`~/.config/proxmox-ca.pem`)
before the session.

**TLS: 192.168.2.8 is not in the node certificate.** The first probe failed
with curl exit 60. The CA is right (`openssl s_client -CAfile` verifies the
chain), but the leaf's SANs are `127.0.0.1, ::1, localhost, 192.168.0.8,
pv1-1, pv1-1.home.lab`, and neither name resolves on the workstation.
192.168.0.8 and 192.168.2.8 present the same certificate fingerprint: the
node sits on `vmbr0` (192.168.0.8/24, untagged) with VLAN interfaces
`vmbr0.2` (192.168.2.8/24) and `vmbr0.4` (192.168.4.8/24). All recon kept
full verification by connecting to 192.168.2.8 while verifying the
certificate as `pv1-1.home.lab`:

```bash
curl -s --resolve pv1-1.home.lab:8006:192.168.2.8 --cacert ~/.config/proxmox-ca.pem \
  -H @<(printf 'Authorization: PVEAPIToken=%s\n' "$(tr -d '[:space:]' < ~/.config/proxmox-token)") \
  https://pv1-1.home.lab:8006/api2/json/version
# {"data":{"release":"9.2","version":"9.2.20","repoid":"49318c671b82f31e"}}
```

Every capture went through a scratch helper wrapping exactly that call. It
passes the token through a file descriptor so it never reaches argv or
output, and writes `{"status","reason","body"}` to
`internal/provider/proxmox/testdata/<name>.json`, refusing to save a body
that contains `PVEAPIToken`.

A real NoCloud seed was used instead of the plan's 1 KiB placeholder, so the
agent could be installed and F21 captured from a live guest. No ISO tool is
installed locally, so a throwaway module in the scratchpad reused the
repository's `github.com/kdomanski/iso9660 v0.4.0` from the module cache and
copied kvm2's `guestAgentScript` verbatim:

```bash
GOFLAGS=-mod=mod GOPROXY=off GOSUMDB=off go run . proxy-reconaaaaaaaaaaa proxy-reconaaaaaaaaaaa-seed.iso
# 45056 bytes, ISO 9660 'cidata'
```

Write experiments ran against VMID 199999 (`proxy-reconaaaaaaaaaaa`), a
clone of template 9001. Sequence: validation error; clone; resources, config
write and start while the clone lock was held; status and agent while
running bare; stop; config read; clone onto the taken VMID; tag tests;
upload twice plus a bad name; attach `ide2`, set cores and memory; three
resizes; start and agent poll; suspend and resume; a clone onto a storage
without image support; stop; destroy with `purge=1&destroy-unreferenced-disks=1`;
destroy again; ISO delete.

### Facts F1 to F23 (cited as Evidence by the spec)

- **F1.** Errors carry the text twice: in the HTTP reason phrase and as the
  body's `message`, which ends in `\n`. A 400 adds `errors`, a map from
  parameter to message. A 401 has the reason phrase only and an empty body.
  The client should use the trimmed body `message` and fall back to the
  reason phrase. Headers are ordinary CRLF.
- **F2.** Missing VM, identical for `GET config`, `GET status/current` and
  `DELETE`: `500 Configuration file 'nodes/pv1-1/qemu-server/999999.conf' does not exist`.
- **F3.** A start issued while the VM is clone-locked is **accepted** (200
  with a UPID). The task log reads `trying to acquire lock... OK` and the VM
  starts once the clone finishes. Setting a lock through the API was not
  tried.
- **F4.** Clone onto a taken VMID, synchronous:
  `500 unable to create VM 199999: config file already exists`.
- **F5.** Agent enabled but not running: `500 QEMU guest agent is not running`.
  Agent not enabled in the config: `500 No QEMU guest agent configured`.
  Stopped VM with `agent: 1`: `500 VM 199999 is not running`.
- **F6.** Not produced. `local-lvm` is thin and overprovisions, so
  `quota: absent`.
- **F7.** Config write while clone-locked: `500 VM is locked (clone)`.
- **F8.** `/cluster/resources?type=vm` entries carry `vmid, name, node,
  status, type, template, tags, maxcpu, maxmem, maxdisk, uptime` and more;
  `lock` appears only when set. Status values seen: `running`, `stopped`,
  and `unknown` during a clone. **Mid-clone the entry has no `name`**: only
  `type, vmid, id, node, lock: clone, status: unknown`.
- **F9.** Hyphenated tags are **accepted** and stored as given. Uppercase is
  folded (`Proxy_Operator` → `proxy_operator`), the separator is `;`, and
  characters such as a space or `!` give
  `400 … tags: invalid format - invalid characters in tag`. The datacenter
  has no `tag-style` or `user-tag-access` set.
- **F10.** `description` on clone is accepted and survives verbatim,
  newlines included. The 22-character `proxy-<base32>` name is accepted.
- **F11.** The clone's `meta` is copied from the template
  (`creation-qemu=11.0.3,ctime=1790145676`, identical to 9001), so it says
  nothing about when the clone was made. `written-at-create` is required.
- **F12.** Upload: multipart with field `content=iso` and the file part named
  `filename`, carrying the target file name. Returns a UPID for an
  `imgcopy` task. A name without `.iso` gives
  `400 … filename: wrong file extension`. **Re-uploading an existing name
  silently overwrites it** (the task log shows a plain `cp`).
  `Datastore.AllocateTemplate` suffices.
- **F13.** Deleting a volume needs `Datastore.Allocate` on
  `/storage/<id>`; without it the call gives
  `403 Permission check failed (/storage/local, Datastore.Allocate)`. With it
  (on `proxy-seeds`), the delete **runs as an `imgdel` task** and returns a
  UPID. Deleting a volume that no longer exists is also accepted and its
  task still ends `OK`; the log reads `disk image '…' does not exist`, then
  `Removed volume`. Seed cleanup is idempotent with no message matching.
- **F14.** Stopped: `status: stopped, qmpstatus: stopped`. Paused:
  `status: running, qmpstatus: paused` in `status/current`, while
  `/cluster/resources` shows plain `running`.
- **F15.** A clone onto a storage without image support is rejected up
  front (`400 … storage 'local' does not support vm images`) and leaves no
  VM. A clone failing mid-task was not observed.
- **F16.** Task status fields: `status` (`running`, `stopped`), `exitstatus`
  (`OK` or the error text), `type, id, node, user, tokenid, upid, starttime,
  pid, pstart`. A full clone of the 3 GiB template onto `local-lvm` finished
  within about two seconds.
- **F17.** The planned role, granted on `/`, covered clone, config, resize,
  start, stop, suspend, agent read, upload and destroy. **Missing:
  `Datastore.Allocate` for deleting the seed ISO.** `VM.Config.Cloudinit` is
  not needed for the `ide2` write. `SDN.Use` was held but never exercised,
  because no call rewrote `net0`.
- **F18.** HTTP/1.1 only: `curl --http2` negotiates `HTTP/1.1`.
- **F19.** Template 9001 (`debian-13-template`): `scsi0` on `local-lvm`,
  `boot: order=scsi0`, `ide2` free, `agent: 1`, OVMF, q35, `net0` untagged on
  `vmbr0`. **Attaching `ide2` makes Proxmox append it to the boot order**
  (`order=scsi0;ide2`). Template 9000 (Ubuntu) is unsuitable: it has a
  Proxmox cloud-init drive on `ide2` and VLAN tag 4 on `net0`.
- **F20.** Resize **runs as a task** (returns a UPID). The same size is an
  `OK` no-op; shrinking fails inside the task with `exitstatus`
  `shrinking disks is not supported`.
- **F21.** Agent response: `result[]` of `{name, ip-addresses[{ip-address,
  ip-address-type}]}`. The live guest reported `lo` and `ens18` with
  `192.168.0.150`, 50 seconds after start, via DHCP on untagged `vmbr0`.
  This proves the NoCloud-seed-on-`ide2` route end to end on this host.
- **F22.** A wrong secret and an unknown token id both give
  `401 Authentication failed!`; no credentials gives `401 No ticket`.
- **F23.** Uploads are staged in `/var/tmp/pveupload-*` and then copied to
  `/var/lib/vz/template/iso/`, so the tmpfs `/tmp` is not involved.

### Corrections to the plan that Phase 1 must fold into the spec

1. **Check `lock` before the name.** A mid-clone VM has no name in
   `/cluster/resources` (F8), so name-first logic turns a clone in progress
   into NotFound. Get, Delete and Create's lookup must treat a locked or
   nameless entry at our VMID as in-progress (Transient or wait), never
   absent.
2. **Inspect a taken VMID before probing forward.** After a crash mid-clone,
   the retry's name lookup misses the nameless VM, the clone onto the
   deterministic VMID fails with F4, and forward probing would clone a
   duplicate. On F4, read that VMID: `lock: clone` means wait for it, and our
   UID in the description means adopt it. Probe forward only for a VM that
   is demonstrably someone else's.
3. **A start queues behind a lock** instead of failing (F3).
4. **Resize is asynchronous** (F20); wait on its task.
5. **Drop delete-before-upload** for the seed: upload overwrites (F12).
   **Seed deletion is a task** (F13): wait on it. A repeat delete succeeds,
   so Delete needs no absent-volume matcher.
6. **Get reads `status/current`**, not only `/cluster/resources`, to tell
   paused from running (F14).
7. **Tag rationale**: hyphens are accepted on 9.2.20 (F9). `proxy_operator`
   stays, since it is valid everywhere, but the claim that hyphens are
   rejected is removed.
8. **The boot-disk logic must skip `ide2`**, which Proxmox adds to the boot
   order (F19). The plan's rule already excludes it; add a test with
   `order=scsi0;ide2`.
9. **Three agent messages, not one** (F5): `QEMU guest agent is not running`
   (booting, or the install failed), `No QEMU guest agent configured` (our
   `agent=1` was removed), and `VM <vmid> is not running` (stopped between
   the status read and the agent call). The spec decides each class; none
   may become NotFound.
10. **Internet on the proxy network is a hard prerequisite**, twice over: the
    vendor-data agent install needs a package mirror, and the proxy itself
    needs egress. The guide says so, and recommends baking
    `qemu-guest-agent` into the template with `installGuestAgent: false` to
    drop the mirror dependency from IP discovery.

### Decisions made with the user in Phase 0

- **Seeds go to a dedicated directory storage**, `proxy-seeds`, with ISO
  content only. The token gets `Datastore.Allocate` there and nowhere else,
  so it can delete seeds but never backups or other ISOs on `local`. The
  sample config's `isoStorage` becomes `proxy-seeds`. The guide must say that
  an ACL on a more specific path replaces the inherited one for that user,
  so the role granted on `/storage/proxy-seeds` must repeat
  `Datastore.AllocateTemplate`, `AllocateSpace` and `Audit` alongside
  `Allocate`.
- **Proxies land on a VLAN** set in config. `ProxmoxConfig` gains an
  optional `vlanTag` (1 to 4094), applied when Create rewrites `net0`
  together with the optional bridge override, keeping the MAC. Unset leaves
  `net0` as cloned. This is the first call that exercises `SDN.Use` (F17).
- **The API URL is `https://192.168.0.8:8006`**, the address in the node
  certificate, so `caFile` verifies with no provider change. No
  `tlsServerName` option. The operator's cluster must be able to reach
  192.168.0.x.

### Seed storage and VLAN 4

The user created `proxy-seeds` (`dir`, ISO content only), granted the token
`Datastore.Allocate`, `AllocateSpace`, `AllocateTemplate` and `Audit` on
`/storage/proxy-seeds` through a role `ProxyOperatorSeeds`, and freed the
leftover recon ISO on `local`. An upload to `proxy-seeds` and two deletes of
a throwaway ISO there produced `content_delete_response.json`,
`content_delete_again_response.json` and `task_status_volume_missing.json`.

The user gave the VLAN as "vlan 4, bridge: vmbr0.4@vmbr0". That is the
host's own VLAN interface name as `ip link` prints it; a VM NIC cannot
attach to it because it is not a bridge. On the VLAN-aware `vmbr0` the
Proxmox form is `bridge=vmbr0,tag=4`, which is what template 9000 uses, so
the provider's `vlanTag: 4` appends `tag=4` to the cloned `net0`.

A second clone of 9001 at VMID 199999 got its seed from `proxy-seeds` and
`net0` rewritten from `virtio=BC:24:11:EE:48:83,bridge=vmbr0` to
`virtio=BC:24:11:EE:48:83,bridge=vmbr0,tag=4`. **The write succeeded, so the
token's `SDN.Use` on `/` covers a VLAN tag on `vmbr0`**
(`set_config_vlan_response.json`).

**The first VLAN 4 attempt never produced an agent** in seven minutes of
polling (every 13 seconds, 33 polls). The only signal was the agent call, so
the cause could not be seen from the API. A debug seed added a `recon` user
(user-data only, with a new instance id so cloud-init re-ran), and the user
logged in on the console: DHCP worked, but VLAN 4 had no route to the
internet because of a routing misconfiguration on the Proxmox host, so the
vendor-data `apt-get install qemu-guest-agent` could not run. The user fixed
the routing.

**Rerun from scratch after the fix**, same sequence the provider will use
(destroy and seed delete first, then a fresh clone and seed, `instance-id`
equal to the name):

| Step | Elapsed |
|---|---|
| clone task done | 4 s |
| seed upload task done | 4 s |
| config (`ide2`, `net0` with `tag=4`, tag) | 4 s |
| start task done | 5 s |
| agent reports `192.168.4.147` | 47 s after start |

The guest was reachable from the workstation (ping, TCP 22), as the contract
requires of a proxy address. Stopping it gave the F5 stopped-VM fixture;
destroy and seed delete then left the host with only its original VMs 100,
9000 and 9001, no ISOs on `local` or `proxy-seeds`, and no `local-lvm`
volumes of 199999. The capture is `agent_network_get_interfaces_vlan.json`.

### Worth noting

- The token file is mode 664; 600 would be tighter.
- Fixtures: 48 files, all valid JSON, none containing `PVEAPIToken`.
  Several names differ from the plan's list because what they captured
  differed from what was expected: `start_while_clone_locked_response.json`
  (the start succeeded), `task_status_stopped_error.json` (from the shrink,
  not a start), `upload_overwrite_response.json` (success, not an error),
  `resize_same_size_response.json` and `resize_shrink_response.json`
  (UPIDs), `err_forbidden_volume_delete.json` (the 403 from F13).
  `err_no_space.json` was not produced (F6).
- The host is back to its original state; see the table above.
- The scratch seed builder (`seedgen` in the session scratchpad) took the
  user-data file and instance id as arguments by the end, for the debug
  login. It is not part of the repository; Phase 3's `nocloud` package
  replaces it.
- Time to IP was about 50 s on both networks, with the agent installed at
  first boot. Budgets in the plan (running wait 60 s for the VM state, the
  agent discovered by `Get` polling) are consistent with this.

## Phase 1 — The spec

Wrote [docs/proxmox-provider-spec.md](../proxmox-provider-spec.md), all 24
entries of the template, from the plan, the contract and the Phase 0 facts.
The ten Phase 0 corrections are folded in, and the plan now opens with a
note that the spec wins where the two disagree.

Judgment calls made here, each recorded in its entry:

- **Create's allocation loop** (entry 2) runs an ownership check on any
  taken VMID: wait out a lock or a nameless entry, then adopt when the name
  and UID are ours, skip when the name differs, refuse as Permanent when the
  name is ours but the mark is not. This is the fix for the duplicate-VM bug.
- **Get's check order** (entry 9): lock, then name and mark from the config,
  then `status/current`, then the agent. `lock: suspended` alone maps to
  Stopped, since a hibernated VM never resumes by itself.
- **Agent messages** (entry 7): `QEMU guest agent is not running` is
  Provisioning, `VM <vmid> is not running` is Transient, and
  `No QEMU guest agent configured` is **Stopped**, so the reconciler replaces
  a VM that can never report an address instead of latching the Proxy.
- **Templates are checked before cloning** (entry 4): not a template, or a
  Proxmox cloud-init drive in any slot, is Permanent. This makes the "no
  cloud-init drive" prerequisite enforceable instead of a silent failure.
- **`isoStorage` is required**, with no default (entry 15), because the token
  must hold `Datastore.Allocate` on it.
- **TLS verification failures are Permanent**, with a Checker reason
  `CertificateInvalid` (entries 13, 16), since the fix is the URL or CA file.
- **The CA is loaded lazily and cached**; the token is read per request
  (entry 14).

Checks, as the plan's Verify clause asks:

```bash
grep -c '^## [0-9]*\. ' docs/proxmox-provider-spec.md     # 24
grep -n '^\*\*Answer:\*\*$' docs/proxmox-provider-spec.md # nothing
grep -c '—' docs/proxmox-provider-spec.md                 # 0
```

plus a script comparing every `` `<axis>`: `<profile>` `` pair against the
contract's §4.20 table (no mismatches, every axis answered) and a list of
the F numbers cited (all 23).

Worth noting: the first pass stated two things as fact that recon could not
show and was corrected before commit. That an unreachable node's guests stay
listed as `unknown` is Proxmox's documented design, unverifiable on a single
node; entry 13 now says so. And the matcher for `VM <vmid> is not running`
is a whole-message regexp, because ` is not running` alone also matches
`QEMU guest agent is not running`.

## Phase 2 — Config block, registration, package skeleton

Added the `proxmox:` block (`internal/provider/config_proxmox.go`), wired it
into `config.go` (the field, `setBlocks`, the allow-list and the dispatch:
three `"proxmox"` sites) and registered `"proxmox": proxmox.New` in
`cmd/main.go`, so the two type lists changed in one commit. The package
`internal/provider/proxmox` now holds the seam interface and its data types
(`api.go`), `id.go`, `mark.go`, `state.go`, `errors.go` and the constructor
(`proxmox.go`). The four contract methods are stubs that answer
`ErrTransient`, so a Proxy on a configured `proxmox` entry retries instead of
latching until Phases 5 to 7 land. `New` passes a nil `api` until Phase 4
builds the HTTP client.

Every test that depends on Proxmox's wording or JSON shape reads the Phase 0
captures through `fixtures_test.go` (`readCapture`, `loadAPIError`,
`decodeData`), so no message string is repeated in a test. The one exception
is `TestClassify_table`'s synthetic rows (a 404, 502, 595, 596, a storage
message), for statuses recon never produced.

Deviations from the plan's Phase 2 list, both making it smaller:

- **No temporary `locks.go`.** Only Create and Delete take the per-name lock,
  and they arrive after Phase 3 extracts `internal/provider/keyedlock`, so
  copying kvm2's lock now would only be deleted next phase.
- **No `provider.Checker` assertion yet.** `cmd/main.go` type-asserts the
  Checker and would call a stub; the assertion comes with the real `Check`
  in Phase 7.

Added beyond the plan: `api_test.go` decodes the captured responses into the
seam's types, so a misspelled JSON tag fails now rather than in Phase 5, and
`TestParseDescription_fromClone` reads the mark back from the description of
the VM cloned in Phase 0.

A mutation check confirmed the suite can fail. Matching `VM <vmid> is not
running` as a plain suffix (the first spec draft's mistake) failed
`TestAPIError_matchers` on the agent-not-running capture; treating a paused
VM as running failed `TestMapState_table` and `TestMapState_fromCaptures`.
Both mutations were reverted and the files compared identical to backups.

```bash
gofmt -w cmd/main.go            # realigned the constructor map for the longer key
grep -c '"proxmox"' internal/provider/config.go                     # 3
go build ./... && go vet ./...
go test -race -count=1 ./internal/provider/... ./cmd/...            # all ok
go test -short -count=1 ./...                                       # all ok
go test -count=1 -v ./internal/provider/proxmox/ | grep -c '^=== RUN' # 152
```

Worth noting: the `validProxmoxYAML` constant is the shape Phase 9's commented
sample entry should take, with the user's values (`192.168.0.8`, `pv1-1`,
`proxy-seeds`, `vlanTag: 4`, template 9001).

## Phase 3 — Extract `nocloud` and `keyedlock` from kvm2

kvm2's `seed.go` and `seed_test.go` moved to `internal/provider/nocloud`
with `git mv`, so history follows them. The package exports `BuildSeedISO`,
`BuildVendorData`, `GuestAgentScript`, `SeedLabel` and a new `ErrEmptyName`
sentinel in place of kvm2's `permanentf`. kvm2's `keyedLocks` became
`internal/provider/keyedlock` (`Locks`, `New`, `Acquire`), and its three
`TestKeyedLocks_*` tests moved with it under their old names. kvm2 now
imports both packages, and its `create.go` still wraps every seed-build error
with `permanentf`, so kvm2's behaviour is unchanged. proxmox gained the
`nameLocks` field and `lockName`.

The test counts match the plan's Verify clause: kvm2 dropped from 465 to 447
`=== RUN` entries, which is the 12 moved tests plus the 6 subtests of
`TestSeedISO_userDataByteForByte`. `nocloud` (15) and `keyedlock` (3) carry
exactly those 18.

```bash
git mv internal/provider/kvm2/seed.go internal/provider/nocloud/seed.go
git mv internal/provider/kvm2/seed_test.go internal/provider/nocloud/seed_test.go
go test -count=1 -v ./internal/provider/kvm2/ | grep -c '^=== RUN'   # 465 before, 447 after
grep -rn 'iso9660' internal/provider/kvm2/                          # nothing
go build ./... && go vet ./... && go vet -tags=integration ./...
go test -race -count=1 ./internal/provider/...                      # all ok, run three times
go test -short -count=1 ./...                                       # all ok
```

**Pulled forward from Phase 6 so proxmox imports `nocloud` now**, as the
"Done when" line requires: `seed.go` with `seedISOName`, `seedVolID` and
`buildSeed`, the pure part of the seed step. `uploadSeed` still belongs to
Phase 6, which calls `buildSeed`. The new tests are `TestSeedNames_table`,
`TestBuildSeed_table` (4 rows),
`TestBuildSeed_emptyNameIsPermanentWithoutPayload`,
`TestBuildSeed_stagingFailureIsTransient` and
`TestLockName_givesUpWithContextAsTransient`. proxmox went from 152 to 161
test entries.

**proxmox does not copy kvm2's open review finding S5.1**
(`docs/reviews/2026-09-17-kvm2-provider-review/sweep.md`). In kvm2, a
staging failure such as a missing `/tmp` or a full emptyDir becomes
Permanent, and that latches the Proxy. `buildSeed` makes only
`ErrEmptyName` Permanent and wraps any other error with `%w`, so
`classify` reads it as Transient. The sentinel makes the same three-line
fix possible in kvm2, but this phase leaves kvm2's behaviour unchanged.

Worth noting:

- The older `kvm` provider has its own `seed.go` with a different
  signature (`buildSeedISO` returns `[]byte`). The plan scopes the
  extraction to kvm2, so `kvm` is untouched.
- `GuestAgentScript` now says the operator reads the address "from the
  hypervisor" instead of "through libvirt", in both the Go comment and the
  script's comment line inside the guest. The script's commands are
  unchanged.
- The docs now name the new packages: `kvm2-provider.md`,
  `kvm2-provider-spec.md` entry 19, the kvm2 columns and section of
  `providers-existing.md`, plus a "Shared provider helpers" row in the
  `architecture.md` component table. `docs/reviews/` keeps the old names,
  because it records the code as it was reviewed.
- `TestCheck_hostsCheckedConcurrently` in kvm2 failed once under the full
  `-race` suite (latency 199.96 ms against a 200 ms floor). It passed 20
  runs out of 20 in isolation, with and without this phase's changes. The
  test starts its 200 ms release timers before `Check` starts its clock, so
  under load the measured latency can come in just under 200 ms. The flake
  predates this phase, and the test is not touched here.

## Phase 4 — Seam, HTTP client, decorator, fake

`client.go` holds `httpAPI`, the thin `net/http` adapter for all sixteen
seam methods. It reads the token file on every request, turns any non-2xx
answer into `*apiError` from the body's `message` or the reason phrase
(F1), and wraps its transport in `tracing.RestConfigWrapper`. `observe.go`
is the logging decorator: one V(1) `proxmox api call` line per call,
detail at V(2), and no span, because the transport already traces each
request. `New` now builds `observe(newHTTPAPI(...))` instead of passing a
nil `api`, still without touching the network or the token file.
`fake_test.go` holds `fakeAPI`, which Phases 5 to 7 build on.

Where this phase departs from the plan's Phase 4 text:

- **The `http.Client` is built on first use, not in `newHTTPAPI`.** Spec
  entry 14 wins over the plan: the CA is loaded lazily, a missing CA file
  is Transient and retried until the file lands, one without a certificate
  is Permanent, and the pool is cached only once it loads. A transport's
  `RootCAs` cannot change after first use, so the whole client waits for
  the CA (`TestClient_caFileLoadedLazily`).
- **The upload is not an `io.Pipe` with a goroutine.** The multipart
  framing (the `content=iso` field, the `filename` part header and the
  closing boundary) is rendered into two small buffers, and the body is
  `io.MultiReader(head, io.LimitReader(r, size), tail)` with an exact
  `Content-Length`. The seed is still streamed and never held by the
  client. There is also no goroutine to leak, and the request carries a
  length, as Phase 0's `curl -F` upload did. A chunked upload to pveproxy
  was never tried. `TestClient_uploadStreamsMultipart` proves the
  streaming with a reader that stalls after its first 256 KiB until the
  server has seen file bytes.
- **The fake's `DeleteVolume` tolerates a missing volume**, as spec
  correction 5 and F13 say, instead of the plan's error.
- **Added:** secret redaction in `apiError` (contract §7), because
  otherwise a server that echoes the header would put the secret into
  `Proxy` status; an `errCredentials` sentinel on token-file errors, so
  Phase 7's Checker can report `CredentialsUnavailable`; and the V(1)
  identifiers `newid` and `name` on a clone, `filename` on an upload and
  `volid` on a volume delete. Spec entry 20 now lists them.

**The volid's `/` travels as `%2F`.** Neither the plan nor the spec says
how `{volid}` is encoded in the content-delete path. The Phase 0 session's
transcript shows every delete went through a helper encoding the whole
volid:

```bash
enc() { python3 -c "import urllib.parse,sys; print(urllib.parse.quote(sys.argv[1], safe=''))" "$1"; }
curl ... -X DELETE ".../storage/proxy-seeds/content/$(enc 'proxy-seeds:iso/proxy-reconaaaaaaaaaaa-seed.iso')"
```

So `apiPath` runs `url.PathEscape` over each segment
(`TestClient_volIDSlashIsEscaped`).

`net/http` cannot send a custom reason phrase, so the client tests replay
every capture byte for byte through a hijacked connection (`serveCapture`).
`TestClient_apiErrorFromStatusReasonAndBody` globs `testdata/err_*.json`
and skips `err_tag_hyphen.json`, which is a 200 despite its name: hyphenated
tags were accepted (F9).

Tests beyond the plan's list: `TestClient_methodPathAndParams` (the method,
path, query and form of every seam method, with a reflection check that no
method lacks a row), `TestClient_cloneOmitsEmptyStorageAndTarget`,
`TestClient_volIDSlashIsEscaped`, `TestClient_caFileLoadedLazily`,
`TestClient_x509FailureIsPermanent` (from spec entry 21),
`TestDecodeConfig_scalarsAsStrings`, `TestReadToken_trimsTrailingNewline`,
`TestObserve_startsNoSpan`, `TestObserve_identifiersAtV1`,
`TestNew_wiresObservedHTTPClient`, and six `TestFakeAPI_*` tests.
`TestFakeAPI_errorsMatchCaptures` compares the fake's error texts with the
captures, so every test that goes through the fake exercises the real
wording. Texts that recon never produced start with `fake:`.

A mutation check confirmed the new tests can fail: dropping `redact`,
buffering the upload with `io.ReadAll`, and logging V(2) detail at V(1)
each failed their test. Removing only the empty `TLSNextProto` map did
**not** fail `TestClient_http1Only`. A transport with a custom
`TLSClientConfig` and `DialContext` never offers h2 unless
`ForceAttemptHTTP2` is set, so the map is a second guard. With both
guards removed the test failed with `client spoke HTTP/2`. Every mutation
was reverted, and the files were compared identical to backups.

```bash
go test -race -count=2 ./internal/provider/proxmox/          # ok, run three times
grep -n 'ReadAll' internal/provider/proxmox/client.go        # only the 64 KiB error-body read
grep -n 'context.Background()' internal/provider/proxmox/*.go | grep -v _test.go   # nothing
go build ./... && go vet ./... && go vet -tags=integration ./...
go test -short -count=1 ./...                                # all ok
go test -count=1 -v ./internal/provider/proxmox/ | grep -c '^=== RUN'   # 161 before, 289 after
```

Worth noting:

- `bin/golangci-lint run ./internal/provider/proxmox/...` panics inside
  staticcheck's IR builder (`unexpected expr: *ast.KeyValueExpr` in the
  Go 1.27 standard library's `poll` package). This is a toolchain problem,
  not this code. With `--disable staticcheck` the new files' findings are
  goconst on form keys (kvm2 carries 50 of the same) and two unparam hits
  on the fake's `addTemplate` and `resourceByID`, which Phases 5 and 6 will
  call with other VMIDs. The mapsloop and gocyclo findings in `proxmox.go`
  and `api_test.go` date from Phase 2.
- Nothing in this phase talked to the host. The client has only met
  `httptest` servers replaying the captures. The first real request is
  Phase 10's integration run, which is where the `Content-Length` upload
  and the `%2F` volid get proven against pveproxy.
- The fake refuses to destroy a running VM with an uncaptured
  `fake: VM <vmid> is running - destroy failed`, so Phase 7's
  stop-before-destroy order is enforced by the fake even though the real
  text is unknown.

## Phase 5 — Get, ListByTag, task helpers

`task.go` holds `parseUPID`, a shared `pollUntil` loop, `waitTask` and
`waitUnlocked`. `get.go` holds `Get`, the guest-agent read and `resolve`
(one resources call, QEMU entries only, which Delete will reuse). `list.go`
holds `ListByTag` and its candidate rule. The Get and ListByTag stubs are
gone, and `TestStubs_answerTransient` now covers only Create and Delete.

The spec overrides the plan in the places the Phase 4 notes listed. Get
checks in entry 9's order, so a lock or `status: unknown` in the resources
entry is Transient before the name is ever read, and a running Get makes
four calls (`TestGet_runningMakesFourCalls`). The three agent messages
follow entry 7. The poll interval is `budgets.poll`, and a failed task logs
a V(1) `proxmox task failed` line with `upid` and `exitstatus`.

Judgment calls, most of them spec gaps:

- **`WARNINGS: <n>` counts as a successful task.** Proxmox reports a task
  that finished but logged warnings with that exitstatus and treats it as
  finished, not failed. Under the spec's old "anything but `OK` is
  Transient" rule, a start that logged a warning would make Create retry
  forever. Recon never produced one, so this row relies on how Proxmox is
  known to behave, not on a capture. The spec's entry 10 text and entry 13
  table now say so.
- **A `suspended` lock still goes through the name and mark checks.** The
  entry 9 table puts "lock suspended is Stopped" before the config rows.
  Taken literally, a hibernated VM that reused our VMID would be reported
  as ours and Stopped, and Delete would then skip it as foreign in a loop.
  Get therefore reads the config, which it needs anyway for UID and
  CreatedAt, applies the name and mark checks, and only then answers
  Stopped. It skips `status/current` and the agent. Entry 9 now says this.
- **`TestGet_twoVMsWithOurNameIsPermanent` became
  `TestGet_otherVMWithOurNameIsIgnored`.** Spec entries 2 and 9 address Get
  by VMID, and only Create refuses a duplicate name. Get answering
  Permanent would latch a working Proxy as Failed because of a VM someone
  else created.
- **`TestGet_unmanagedDescriptionHasEmptyUID` tests a mark with no `uid`
  line.** The plan's name read literally ("unmanaged") would contradict
  "no mark is NotFound".
- **Agent errors outside the entry 7 table are forced to Transient**, even
  a 403 or a `does not exist`. Otherwise `classify` would make them
  Permanent or NotFound, and the spec says "anything else: Transient".
- **`waitUnlocked` returns the unlocked config**, which saves Create's
  ownership check one read. When the VM disappears while it waits (a
  failed clone), it returns the does-not-exist error and lets Create
  decide.

Pulled forward and left behind:

- **`provider_test.go` now exists** with `newFixture`, `testBudgets`,
  `addOurVM` and `assertProviderError`, because every Get and ListByTag
  test needs them. The plan created this file in Phase 8. Phase 8 adds the
  contract sweep to it.
- **`waitRunning` is left to Phase 6.** The plan's package layout puts it
  in `task.go`, but Phase 5 lists no test for it and only Create uses it.
  It should be built on `pollUntil`.
- **The fake gained two things.** `fakeVM.kind` makes an entry listed as
  `lxc` invisible to the qemu endpoints. `hookOn` is a general per-method
  hook, and `failWith` is now built on it. Tests use `hookOn` (through
  `mutateOn` in `get_test.go`) to change a VM between two of Get's calls.

A mutation check confirmed the new tests can fail. Each of these failed a
test: classifying agent errors instead of forcing Transient; dropping the
config lock check; dropping the `suspended` special case; ignoring an ended
ctx in ListByTag; treating `WARNINGS` as failure; and not mapping
`No QEMU guest agent configured` to Stopped. Removing the resources-entry
lock and `unknown` checks together failed only
`TestGet_unknownStatusIsTransient`. The fake reports a lock in the config
as well, so the config check catches every lock row, including a mid-clone
VM. The resources check stays because the spec orders it first and
because it spares a config read. Every mutation was reverted, and the
files were compared identical to backups.

```bash
go test -race -count=2 ./internal/provider/proxmox/                     # ok
grep -n 'not implemented' internal/provider/proxmox/*.go | grep -v _test # Create and Delete only
go build ./... && go vet ./... && go vet -tags=integration ./...
go test -race -count=1 ./internal/provider/... ./cmd/...                 # all ok
go test -short -count=1 ./...                                            # all ok
go test -count=1 -v ./internal/provider/proxmox/ | grep -c '^=== RUN'   # 289 before, 393 after
```

Lint ran with staticcheck disabled, because staticcheck panics on
Go 1.27:

```bash
bin/golangci-lint run --disable staticcheck ./internal/provider/proxmox/...
```

Worth noting:

- The lint findings in the new files are goconst on test strings and
  logcheck on `get`, `agentAddress` and `waitTask` taking both a ctx and a
  logger. That shape copies kvm2, which has five of the same findings. The
  errcheck hits on `assertGetFails` were fixed by returning the error text.
  The two unparam hits and the mapsloop hits predate this phase.
- `waitUnlocked` waits on any lock, `suspended` included. Phase 6's
  ownership check should decide whether a hibernated VM at our VMID is
  worth waiting out the clone budget for, or should be refused at once.
- Nothing in this phase talked to the host. Every Proxmox message a test
  relies on comes from `testdata/` or from fake errors that
  `TestFakeAPI_errorsMatchCaptures` ties to the captures.

## Phase 6 — Create

`create.go` holds `Create` and its steps: placement, the by-name lookup,
the template check, the VMID probe with its ownership check (`inspect`),
the clone, and `provision` (seed, config, resize, start). `configParams`,
`rewriteNet0`, `resizeTarget`, `bootDisk` and `diskBytes` are pure helpers
with their own table tests. `seed.go` gains `uploadSeed`, which uploads
over any existing file with no delete first (F12). `task.go` gains
`waitRunning`, built on `pollUntil` with `budgets.running`. It finishes
only at `running`/`running`, and a `stopped` status after the start is
Transient at once. The Create stub is gone, and `TestStubs_answerTransient`
now covers only Delete.

The spec won over the plan wherever the Phase 6 notes said it would. The
plan's allocation loop is replaced by entry 2: a taken VMID and a clone
answering F4 both go through the ownership check before any probing, and a
locked or nameless entry is waited on with `waitUnlocked` and the clone
budget. The seed is uploaded without a delete. The resize is a task. `net0`
changes only when `bridge` or `vlanTag` is set, and `tag=` is replaced or
appended in place.

Judgment calls, with spec entries 2, 4 and 10 updated to match:

- **The Phase 5 open question: a `suspended` lock is not waited on.**
  `waitUnlocked` now returns at once when it sees `suspended`, because a
  hibernated VM is not a task and would otherwise burn the 10-minute clone
  budget on every attempt. The ownership check then judges the VM the way
  `Get` does. A foreign hibernated VM is probed past. Ours is Permanent
  ("resume or destroy it"), following kvm2's precedent for a paused or
  pmsuspended domain in Create. Transient was the alternative: it would
  self-heal once an admin acts, but it would retry indefinitely on
  something only an admin can fix. `TestWaitUnlocked_table` has a new row,
  and `TestCreate_hibernatedVMIsNotWaitedOn` covers both cases.
- **The template is checked only when a clone is needed.** Spec entry 10
  read the template's config in step 2, before the by-name lookup. A retry
  for a VM that already exists would then fail Permanent if the template
  had been deleted since, which entry 4 explicitly allows. The template
  check now runs after the by-name lookup misses, so a repeat Create costs
  three calls: resources, config and status
  (`TestCreate_adoptionDoesNotNeedTheTemplate`).
- **The template's shape is checked before the clone.** No `net0` while
  `bridge` or `vlanTag` is set, or no boot disk (or none with a readable
  `size=`) while `diskGiB` is set, is Permanent. Found after the clone, the
  VM would already carry the bad config, so fixing the template could not
  help that Proxy's next attempt. `template()` runs the same
  `configParams` and `resizeTarget` over the template's config that
  `provision` later runs over the clone's.
- **An empty `req.Name` is Permanent at placement.** A nameless request
  would match nameless mid-clone entries in the by-name lookup.
- **Adopted VMs get one `status/current` read** rather than trusting the
  listing's status, which may be stale after a lock wait. A running VM,
  whatever its `qmpstatus`, returns its ID at once, as entry 10 says. `Get`
  then reports prelaunch as Provisioning and paused as Stopped.
- **After the clone task, Create reads the config through
  `waitUnlocked`.** That read supplies the `net0` and boot disk the later
  steps need, and it turns a clone that ended but kept its lock (F15) into
  a Transient lock wait rather than a refused config write.
- **An F4 whose VMID then resolves to no QEMU VM probes on.** Either the
  racer's clone failed and removed it, or a container took it.
- **Errors carry no ID** (`wrapErr("create", "", …)`), since the VMID is
  known only after allocation. The messages name the VMID wherever one
  exists.

Test names: every plan and entry 21 name exists except
`TestCreate_resizeOnlyWhenLarger`. It is covered by the spec's pair:
`TestCreate_resizeWaitsForTask` for the larger case and
`TestCreate_resizeSkippedWhenNotLarger` for equal, smaller and unset. Tests
beyond both lists: `TestCreate_vmThatStopsAfterStartIsTransientAtOnce`,
`TestCreate_hibernatedVMIsNotWaitedOn`,
`TestCreate_templateShapeRefusedBeforeClone`,
`TestCreate_adoptionDoesNotNeedTheTemplate`,
`TestCreate_cloneTargetAndStorage` (read from the decorator's V(2) clone
line), `TestCreate_adoptedVMStaysOnItsNode`, `TestCreate_outcomeLine`,
`TestCreate_quietAtV0`, `TestRewriteNet0_table`, `TestBootDisk_capturedClone`
and `TestDiskBytes_table`. The only new test helpers are the small ones in
`create_test.go` (`slot`, `newCreateFixture`, `req`/`create`/`createFails`,
`assertRunning`, `callsBefore`, `clearLockOnVMConfig`). Everything else
reuses the Phase 5 fixture and fake. Two tests check config-write contents
without a recording wrapper. `TestCreate_markWrittenInCloneCall` reads the
VM at the start of `UploadISO`, which is after the clone and before the
config write. `TestCreate_noBridgeOverrideLeavesNet0` changes `net0` at the
start of `SetVMConfig`, so a write that carried `net0` would put the old
value back.

A mutation check applied 18 mutations, and each failed at least one test.
They were: no lock wait in the ownership check; taken VMIDs never
inspected; the F4 race skipping the ownership check; no cloud-init drive
check; upload or resize task not waited on; resize at equal size; a
CD-ROM counted as a boot disk; prelaunch counted as running; the suspended
lock waited on; `net0` always written; `bridge=` appended instead of
replaced; duplicate names ignored; a running VM re-provisioned; `target`
always set; a foreign UID adopted; the tag dropped; probes not capped by
the range. The last one was first **missed**. A 4-VMID range with 32
probes wraps, re-inspects each VM eight times and still ends Transient, so
`TestCreate_vmidRangeExhaustedIsTransient` now pins five config reads.
Five mutations did not compile on the first try and were redone. Every
mutation was reverted, and the files were compared identical to backups.

```bash
go test -race -count=2 ./internal/provider/proxmox/                          # ok
grep -n 'CloudInit' internal/provider/proxmox/create.go internal/provider/proxmox/seed.go
#   seed.go: the pass-through into nocloud.BuildSeedISO; create.go: len(req.CloudInit)
grep -n 'not implemented' internal/provider/proxmox/*.go | grep -v _test      # Delete only
grep -n 'context.Background()' internal/provider/proxmox/*.go | grep -v _test.go   # nothing
go build ./... && go vet ./... && go vet -tags=integration ./...
go test -short -count=1 ./...                                                 # all ok but the parked kvm2 flake, below
go test -count=1 -v ./internal/provider/proxmox/ | grep -c '^=== RUN'         # 393 before, 519 after
bin/golangci-lint run --disable staticcheck ./internal/provider/proxmox/...
```

Worth noting:

- `go test -short ./...` failed once on the parked kvm2 flake
  `TestCheck_hostsCheckedConcurrently` (measured 199.98 ms against a
  200 ms floor). Three reruns passed. This phase does not touch kvm2.
- The new lint findings follow the baseline's shape. There are logcheck
  hits on `create`, `allocate` and `clone` (ctx plus logger, as in kvm2),
  and goconst on test strings. One more unparam than the notes list:
  `(*fakeAPI).iso`'s `storage` always receives `fakeISOStorage`, because
  this phase is its first caller. `addTemplate`'s hit changed from a
  constant VMID to an unused result.
- The config write sets `agent=1` and `tags=proxy_operator` as values. A
  template's agent options (such as `fstrim_cloned_disks=1`) and its own
  tags are replaced, not merged. That is what entry 10 specifies, and the
  guide should say so.
- In the fake, a cross-node clone puts the config on the target at once.
  Real Proxmox may move it at the end of the task, in which case a lock
  wait on the target node would see `does not exist` and the next attempt
  would find the VM. This stays untested, like every cross-node path
  (entry 24).
- Nothing in this phase talked to the host.

## Phase 7 — Delete, Checker, CRD comment

`delete.go` holds `Delete`: parse, the per-name lock, `destroy` (resolve,
the lock and status checks, config, ownership, stop, destroy) and the seed
delete. `check.go` holds `Check`, which fans out one `checkNode` per node
with `wg.Go` and returns the verdicts in config order, plus `checkReason`
and the four provider reasons. The Delete stub and
`TestStubs_answerTransient` are gone, and `grep -rn 'not implemented'
internal/provider/proxmox/` is empty. All four contract methods and
`Check` are real. The `status.type` comment in
`api/v1alpha1/providerhealth_types.go` now ends `kvm2 or proxmox`, and
`make manifests` changed only that description line in the ProviderHealth
CRD.

The spec won over the plan as the Phase 7 notes said. An absent VM still
gets its seed deleted, on every configured node. A task waits on the seed
delete, and nothing matches F13's absent-volume message. A paused VM gets
the stop too. The Checker's reasons include `CertificateInvalid` and
`CredentialsUnavailable`, and pveproxy's 595 and 596 are `Unreachable`.

Open questions decided (spec entries 11 and 16 updated):

- **Our hibernated VM (`lock: suspended`) is Permanent in Delete:** "resume
  or destroy it in Proxmox", the same answer Create's ownership check
  gives. Proxmox's destroy refuses a locked VM (the fake answers with F7's
  `VM is locked (…)` text; the suspended variant was never captured), and
  that lock never clears on its own. Transient would retry forever. A
  Permanent from Delete in `reconcileManaged`'s Stopped branch goes through
  `providerFailure` and latches the Proxy with the admin action in its
  status. On the finalizer path `deletionFailure` keeps retrying, and the
  retries can be seen. Nothing captured shows what a hibernated VM does on
  a stop or start through the API, so there was no evidence for a
  self-healing path (for example, start to resume, then stop and destroy).
  A hibernated VM that is not ours is left alone. The resources-entry lock
  check skips `suspended`, so ownership is judged from the config, as in
  Get.
- **The Checker reads `p.cfg.TokenFile` itself, through the client's
  `readToken`**, before its first request, and keeps only the part before
  `=`. A seam method would break the rule of one method per endpoint, and
  both `TestClient_methodPathAndParams` and
  `TestObserve_everySeamMethodLogsOneLine` enumerate the interface. An
  unusable file answers `CredentialsUnavailable` without a request.
  `readToken`'s errors never carry the content. Every row of
  `TestCheck_reasonsTable` runs at V(2) over the decorator with a sentinel
  secret and asserts that the secret is in no message, detail or log line.

Other judgment calls:

- **The seed is deleted on the VM's node only when Delete destroyed a VM
  of ours; otherwise on every node.** Entry 11 said "skip to step 7" for
  a VM that is foreign, unmarked, or gone by the time the config is read,
  without naming a node. In each of those cases our VM's node is unknown:
  the node in the listing belongs to someone else's VM, or to one that
  just vanished. A container at the VMID takes the same path.
- **Stop's `does not exist` means the VM is gone**, the same as Destroy's.
  The seed is then deleted on its node, and Destroy is not called.
- **`errCAFile` joins `errCredentials` in `client.go`.** A missing CA file
  or one without a certificate is `CertificateInvalid` ("fix `caFile`").
  Without the sentinel it would be `APIError`. Its class is unchanged
  (Transient or Permanent, as entry 14 says).
- **Any 500 from the storage status call is `StorageNotFound`**, carrying
  Proxmox's text. The plan matched a 500 "does not exist", but no response
  for a missing storage was captured. Matching an invented string would go
  against the fixture discipline, and every 500 there (missing, not
  enabled on that node, offline) is fixed on the storage anyway. A 401,
  403, 595 or 596 keeps its shared reason.
- **`checkReason` puts `*net.OpError` before the deadline check**, so a
  dial that runs out of its own 10 s budget is `Unreachable`, not
  `Timeout`. A request timing out on its own while the round still runs
  is `Timeout`. The round's ctx ending is `Timeout` whatever the error
  says, and that guard is in `checkNode`, as in kvm2.
- **The outcome line `proxmox vm deleted` carries `destroyed`**, which is
  false when there was no VM of ours left to destroy. It has no `node` in
  that case. Entry 20 lists the new V(1) no-op lines.
- `templates` counts catalogue entries, not unique VMIDs.

Test names. These plan names changed with the behaviour:
`TestDelete_absentIsNoOp` → `TestDelete_absentStillDeletesSeedOnEveryNode`,
`TestDelete_foreignNameIsNoOp` → `TestDelete_foreignNameLeavesVMAlone`,
`TestDelete_unmarkedIsNoOp` → `TestDelete_unmarkedLeavesVMAlone`, and
`TestDelete_toleratesMissingISO` merged into entry 21's
`TestDelete_seedDeleteIsTaskAndIdempotent`, which runs with the seed both
present and absent and ties the absent case to
`task_status_volume_missing.json`. Every other plan and entry 21 name
exists. `TestDelete_usesPurgeAndDestroyUnreferencedDisks` runs Delete over
the real `httpAPI` against the Phase 0 destroy captures
(`cluster_resources_during_destroy`, `config_during_destroy`,
`destroy_response`, `content_delete_response`,
`task_status_volume_missing`). `TestGet_duringDeleteStillReturnsInstance`
holds the destroy task in its first poll and first checks that the
mid-destroy captures show the VM listed, stopped and without a lock.
Tests beyond both lists: `TestDelete_containerAtVMIDIsLeftAlone`,
`TestDelete_hibernatedVMIsPermanentIfOurs`,
`TestDelete_seedDeleteFailureOnVMNodeFails`,
`TestDelete_seedDeleteNodeFailureSkippedWhenNodeUnknown`,
`TestDelete_goneMidwayIsSuccess`, `TestDelete_resourcesFailureIsTransient`,
`TestDelete_waitsOnTheNameLock` (a held name lock means no call at all,
which is what makes a Delete racing a Create wait),
`TestDelete_cancelledMidWaitReturnsPromptly`, `TestDelete_outcomeLine` and
`TestCheck_reasonsOverHTTPClient`, which takes the transport reasons from
real net/http and crypto/tls errors: unknown CA, hostname, missing CA file,
a replayed 401, 596, a closed server and a cut response. The fake gained
`destroyFailsWith`. New test helpers are only the small ones in
`delete_test.go` (`putSeed`, `seedPresent`, `delete`/`deleteFails`,
`seedDeleteNodes` read from the decorator's lines, `twoNodeFixture`) and
`check_test.go` (`newCheckFixture`, `assertNoSecret`, `setStorage`).
`TestCheck_nodesCheckedConcurrently` uses a barrier, not a timer: every
node's `Version` waits until all have arrived, so checking the nodes one
at a time fails the verdicts deterministically and does not measure
elapsed time. That avoids the kind of flake kvm2's 200 ms test has.

The mutation check applied 34 mutations: 17 in `delete.go`, 16 in
`check.go` and 1 in `client.go`. Every one compiled and every one failed at
least one test. Three did not compile on the first try (an unused `upid`
or `tokenID`) and were redone. The first run of "never stop" did fail
tests, but it also **hung** `TestDelete_concurrentSameNameSafe`, whose
wait for the first `Stop` call could not end. The wait is now bounded by
the first Delete returning, and the rerun fails fast. One reasons row
("dial cut short by the round's deadline") was added before the run,
because no existing row would have caught dropping `checkNode`'s ctx
guard. Every mutation was reverted by a script that compared each file's
hash with the original.

```bash
go test -race -count=2 ./internal/provider/proxmox/ ./internal/providerhealth/   # ok
make manifests && git diff --stat config/    # crawl.example.com_providerhealths.yaml | 2 +-
grep -rn 'not implemented' internal/provider/proxmox/                            # nothing
go build ./... && go vet ./... && go vet -tags=integration ./...
go test -short -count=1 ./...                                                     # all ok
go test -count=1 -v ./internal/provider/proxmox/ | grep -c '^=== RUN'            # 519 before, 649 after
bin/golangci-lint run --disable staticcheck --max-same-issues 0 --max-issues-per-linter 0 ./internal/provider/proxmox/...
```

Worth noting:

- Lint defaults hide repeats: with `--max-same-issues 0` the logcheck hits
  are 4 in `create.go`, 4 in the new `delete.go`, and 2, 1 and 1 in
  `get.go`, `seed.go` and `task.go`, all the ctx-plus-logger shape.
  Nothing else is new except goconst on test strings. A prealloc hit in
  `delete_test.go` was fixed.
- `TestCreate_concurrentSameNameSerialised` has the same unbounded
  `for callCount("Clone") == 0` wait that hung the Delete test under a
  mutation. It is left for the Phase 8 sweep, because this phase does not
  touch Create.
- Never seen on the host, for Phase 10: what the storage status call
  answers for a missing or disabled storage, what destroy says for a
  hibernated VM, and a stop on a paused VM. The try-every-node seed delete
  and the multi-node Checker have only met the fake (entry 24).
- Nothing in this phase talked to the host.

## Phase 8 — Contract and observability sweep, integration test, testing.md

The sweep went into the existing `provider_test.go` and reuses the earlier
phases' helpers. The new helpers are `cancelWhileBlocked`, `within`,
`newLoggingFixture` with `lifecycle`/`sentinelReq`, `assertNeverLogged`,
`assertEveryV1Line` and `msgLine`.

- `TestProvider_cancelledContextReturnsPromptly` blocks Create, Get,
  Delete and ListByTag on `ClusterResources`, and Check on `Version`,
  with `blockOn`. It asserts that the blocked call was the first in the
  call log, then cancels. Each must return within 1 s, as a Transient
  that still carries `context.Canceled` (Check: one target, `Timeout`).
  Two more rows block mid-operation: Create on `UploadISO` and Delete on
  `Stop`.
- `TestProvider_everyErrorIsWrappedWithConfiguredName` has 20 rows, one
  or more behind every `wrapErr` call site: placement, the name lock with
  an ended ctx, resources, a 403 on the clone, a failed start task, the
  malformed, missing, locked and 401 Get, the hibernated VM, a failed
  destroy task, a 403 on the seed delete, and the list with a ctx that
  ends on a candidate. Each row also checks `Error.ID` (empty for
  Create) and the `provider proxmox-home: <op>` prefix.
- `TestLogging_v0SilentV1OutcomeV2Detail` runs one whole lifecycle over
  the observed fake with a token file: Create with a sentinel in
  CloudInit, a Get that finds the captured address, ListByTag, Delete,
  Delete of the gone VM, and Check. It runs at V(0), V(1) and V(2). V(0)
  logs nothing. V(1) has each outcome line once, with its keys, and two
  `proxmox vm deleted` lines: `destroyed=true` with node, then
  `destroyed=false` without. Every line has the `proxmox` logger, the
  provider and level 1, and no V(2) key. V(2) has the resolved line with
  `cloudInitBytes`, agent addresses, the listed instance, and the
  clone's description. The sentinel, `#cloud-config` and the token
  secret appear at no level.
- `TestLogging_noOpLinesAtV1` (a name beyond the plan) covers entry 20's
  no-op lines. Each carries provider and vmid: VM already gone, foreign,
  unmarked, seed delete skipped on each of two nodes, unreadable
  candidate.
- `TestTracing_httpSpansUnderOperationSpan` routes captures by path, as
  `TestDelete_usesPurgeAndDestroyUnreferencedDisks` does. The captures
  are `cluster_resources_during_destroy`, `config_during_destroy`,
  `status_current_running` and `agent_network_get_interfaces`. A Get
  runs over the real `httpAPI`, wrapped in `observe` and
  `provider.WithTracing`, with the recorder's tracer passed through
  `newHTTPAPI`'s opts. It asserts Running at 192.168.0.150, and exactly
  four spans named `HTTP GET` (otelhttp v0.70's
  `defaultTransportFormatter`), each a child of `provider.get` in its
  trace. No attribute on any span carries the secret or `PVEAPIToken`.

Judgment calls:

- **"Instance lines carry vmid and name" is checked on the provider's
  lines, not the decorator's.** Entry 20 gives `proxmox api call` the
  fields `call`, `node`, `vmid`, `storage`, `upid` and `duration`, with
  `name` only on a clone. So `assertEveryV1Line` exempts that line and
  the two lines that are not about one instance (`vms listed`, `node
  checked`), and requires vmid and name on all the others.
- **The integration test compares `CreatedAt` with the mark Create
  wrote.** kvm2 only checks that it is non-zero. Entry 21 says "equal
  what Create wrote", so the test reads the VM's config through the
  bare client and parses its description.
- **Get's poll tolerates Transient.** kvm2's poll fails on any error.
  Entry 7 makes every agent answer the provider cannot place Transient,
  and a booting guest gives some. Those answers are logged and polled
  through; any other class fails.
- **The integration machine type has `diskGiB: 4`.** The 3 GiB recon
  template is then grown by the resize task on the real host. A larger
  template skips the resize.
- **The seed check after Delete is one extra read-only GET in the test**:
  `client.do` on `/nodes/{node}/storage/{storage}/content?content=iso`,
  decoded the way `storage_content_iso.json` is shaped. No seam method
  was added. The VM count and the by-name lookup use the existing
  `ClusterResources`.
- **The cleanup finds the VM by name**, so a Create that failed halfway
  is also removed. With `PROXMOX_TEST_KEEP` it prints `qm stop <vmid>`
  before `qm destroy <vmid> --purge`, because destroy refuses a running
  VM. It then prints `pvesm free <isoStorage>:iso/<name>-seed.iso`.

**Unbounded waits.** `TestCreate_concurrentSameNameSerialised` now uses
the `first` channel shape of `TestDelete_concurrentSameNameSafe`. The grep
found no other sleep loop; `client_test.go`'s `for {` reads multipart
parts until EOF. The mutation check found a different shape of the same
hang: tests that call a blocking operation directly on the test goroutine,
where the lock or `blockOn` is released only after the call returns. Under
"lockName ignores ctx", "Check ignores ctx" and "pollUntil ignores ctx"
the package ran into its 60 s timeout on
`TestDelete_waitsOnTheNameLock`, `TestLockName_givesUpWithContextAsTransient`,
`TestCheck_expiredContextReturnsPromptly`, two `TestCheck_reasonsTable`
rows, `TestCreate_cancelledMidWaitReturnsPromptly`,
`TestDelete_cancelledMidWaitReturnsPromptly`, and the ended-ctx rows of
`TestWaitTask_table` and `TestWaitUnlocked_table`. Each call now runs in
`within(t, 5s or 10s, …)`, which fails the test by name. After the fix
the same mutations fail those tests in seconds.

The mutation check ran 14 mutations on production code. The script copied
each file, applied the edit (asserting that it matched exactly once), and
ran `go test -c` before counting a mutation. It then ran
`go test -count=1 -timeout=60s`, restored the file from the copy, and
asserted the sha256 matched. Afterwards `sha256sum -c` over the eight
files hashed up front was OK. `task.go`, added for M14, was covered by the
script's own assertion. `git status` showed no production file changed.
Results:

- Caught by the new tests:
  - Get drops the span context before its requests →
    `TestTracing_httpSpansUnderOperationSpan`.
  - Cloud-init logged at V(2) → `TestLogging_…` and
    `TestCreate_cloudInitSentinel…`.
  - Delete's destroy path skips `wrapErr` → the wrap test and 6 Delete
    tests.
  - ListByTag's ended-ctx path skips `wrapErr` → the wrap test's `list
    ended on a candidate` row.
  - Delete's resolve with `context.Background()` → the cancel test's
    `delete` row.
  - `lockName` with `context.Background()` → the wrap test's two lock
    rows.
  - Check's probe with `context.Background()` → the cancel test's
    `check` row.
  - Delete's logger without the name, `destroyed` always true, `vms
    listed` at V(0), the clone description at V(1), and the list skip
    line without vmid → `TestLogging_*`.
- Caught only by the older tests: `pollUntil` without its `ctx.Done()`
  case → the four waits tests above. The first attempt deleted the case
  and did not compile (the `provider` import became unused). The rerun
  replaced it with a nil channel.
- Survived, and equivalent: `newHTTPAPI` without the tracer-provider
  option. The transport starts a span only under a valid parent. With no
  `WithTracerProvider`, otelhttp v0.70 takes the tracer from
  `trace.SpanFromContext(r.Context()).TracerProvider()`
  (`transport.go:99`), so the child always lands in the parent's
  provider. No test can tell the two apart.

`Makefile`: `test-integration-proxmox` next to the kvm2 target, with a
`##` help line naming the five required variables. `docs/testing.md`
gained:

- rows for `nocloud`, `keyedlock` and five for `proxmox` (both packages
  were missing);
- a "Live Proxmox, by default" bullet under "deliberately do NOT cover",
  naming the cross-node paths and the three answers never captured;
- a "The proxmox integration test" section: variables, the eleven
  assertions in order, the KEEP cleanup, the cost, and that it has not
  run yet.

In spec entry 21 every §11 box is ticked. The list's intro says the two
integration tests compile but have not run against a host. The cancel and
sentinel items name the new coverage.

```bash
# the names in entry 21, each checked before ticking: 42 of 42 exist
awk '/^## 21\. Tests/,/^## 22\./' docs/proxmox-provider-spec.md | grep -o '`Test[A-Za-z0-9_]*`' | tr -d '`' | sort -u \
  | while read n; do grep -rqE "^func $n\(" internal/provider/proxmox/ && echo "ok $n" || echo "MISSING $n"; done
go test -race -count=2 ./internal/provider/proxmox/                        # ok
go vet -tags=integration ./... && go vet -tags=e2e ./...                    # ok
make -n test-integration-proxmox
# go test -race -tags=integration -count=1 -v -timeout 30m -run TestProxmox ./internal/provider/proxmox/
go test -tags=integration -count=1 -v -run TestProxmox ./internal/provider/proxmox/   # both SKIP: PROXMOX_TEST_URL is not set
go build ./... && go test -short -count=1 ./...                             # all ok
go test -count=1 -v ./internal/provider/proxmox/ | grep -c '^=== RUN'      # 649 before, 688 after
bin/golangci-lint run --disable staticcheck --max-same-issues 0 --max-issues-per-linter 0 ./internal/provider/proxmox/...
bin/golangci-lint run --build-tags integration --disable staticcheck --max-same-issues 0 --max-issues-per-linter 0 ./internal/provider/proxmox/...
```

Worth noting:

- Lint is at the baseline:
  - the logcheck hits (4/4/2/1/1), mapsloop ×2 in `proxmox.go`, unparam
    ×3 in `fake_test.go`, gocyclo in `api_test.go`, and goconst on test
    strings.
  - Two new gocyclo hits were removed, not accepted.
    `TestLogging_v0SilentV1OutcomeV2Detail` was at 32; the per-line loop
    became `assertEveryV1Line`. `TestProxmox_lifecycle` was at 40; steps
    6 and 7 became `assertMarkReadBack` and `assertListed`. kvm2's
    lifecycle test is still at 39.
- The integration tests only compile so far. The first real run is Phase
  10's. That run meets the Datastore.Allocate point too: a token without
  it on the ISO storage fails step 9 as Permanent. Phase 0 granted it on
  `proxy-seeds`, and it is already a Phase 9 guide item.
- The parked items did not bite: kvm2's S5.1 and its 200 ms
  `TestCheck_hostsCheckedConcurrently` passed in every
  `go test -short ./...` run of this phase.
- `.claude/settings.json` stays out of the commit. Nothing in this phase
  talked to the host.

## Phase 9 — Deployment and documentation

Deployment: the optional `proxmox-token` Secret (keys `token` and `ca.pem`)
is mounted read-only at `/etc/proxy-operator/proxmox` in
`config/manager/manager.yaml`, next to `kvm-ssh` and commented the same way.
The Makefile gains `proxmox-token-secret`. Both providers configs gain a
commented `proxmox-home` entry after kvm2's. `config/samples/proxy_proxmox.yaml`
is listed in the samples kustomization. There is no RBAC change and no
`hack/providers-dev.yaml` entry. Docs: `docs/proxmox-provider.md` follows
kvm2's guide heading for heading and has no `—`. The rows and sections
named below went into `providers-existing.md`, `architecture.md`, the
README, `providers.md` and `CLAUDE.md`. `docs/testing.md` and the
integration Makefile target were done in Phase 8 and were not touched; the
guide's Testing section links to them.

Where the spec and the plan's Phase 9 text disagree, the spec won. The URL
is `https://192.168.0.8:8006`, not 192.168.2.8, which is not in the node
certificate. The Secret has two keys, and the target takes
`PROXMOX_CA_FILE` as well as `PROXMOX_TOKEN_FILE`.

Judgment calls:

- **The target also checks both files' content.** The token file must be
  exactly one non-blank line matching
  `<user>@<realm>!<tokenid>=<secret>`. That is the shape `readToken`
  accepts: `TrimSpace`, then an anchored regexp. The check uses `grep -c`
  and `grep -qx`, so the secret is never printed. The CA check mirrors
  `kvm-ssh-secret`'s `PRIVATE KEY` check: the file must contain
  `BEGIN CERTIFICATE`. With `insecureSkipTLSVerify` the guide says to
  create the Secret by hand with the `token` key only.
- **The sample-entry test covers both files.** `commentedEntry` finds the
  `# - name: proxmox-home` line and takes its indent. It keeps the lines
  that continue at `<indent>#   ` and strips `# ` at that indent. The same
  helper then works for the ConfigMap, whose entry sits six spaces deep
  inside a `|` block. So `TestLoadConfig_proxmoxSampleEntry` has a
  `sample` subtest and a `configmap` subtest. It checks every field the
  spec names, and that the optional fields stay commented out.
- **A second test goes beyond the plan:**
  `TestLoadConfig_proxmoxSampleProxyMatchesSampleEntry`. The sample Proxy's
  provider, zone, machine type and image must resolve in the sample
  entry. kvm has exactly this bug as a known gap (`proxy_kvm.yaml` places
  on `nas1`).
- **The sample entry also has `medium: {cores: 2, memoryMiB: 2048,
  diskGiB: 10}`**, next to the `small` the spec asks for, so `diskGiB`
  appears somewhere. The optional fields are commented out with a short
  note each, because no live proxmox entry exists to explain them. The
  kvm2 entry can rely on kvm's.
- **In the ConfigMap the proxmox entry is commented out**, although kvm2's
  is live there. It needs a host and a token.
- **providers-existing.md needed more than the column and the section.**
  The intro now says five providers and links the guide. In the shared
  gaps, six sentences would otherwise be false, and each now names
  proxmox where it applies: adoption verified in two of the five, Create
  blocking in all three VM providers, no Terminated, the list policy, the
  collector gap, and the "kvm2 does this" in proposed features. The new
  section is §8, so the shared gaps and proposed features move to §9 and
  §10. No doc links those sections by anchor (grep).
- **architecture.md §11 now says "All five providers do".** The sentence
  counts Checkers, and proxmox has one. The four Decisions bullets are
  entry 23's: the VMID from the UID with the ownership check, the mark in
  the description, the seed uploaded through the API, and the node
  resolved per call.
- **The guide lists the ten design limits first, then four more.** They
  are the hibernated-VM latch, the one-request seed upload, the open-proxy
  sample (kvm2's guide lists it too), and "not yet proven end to end" with
  the three answers never captured. The Phase 6 to 8 guide items are
  each in the section where a reader looks for them. The config write
  replacing `agent` and `tags` is in *What lands on Proxmox*, and so are
  the seed cleanup, `qm unlock` and the 403 that makes Delete Permanent.
  The certificate rule is in *Cluster-side setup*. CA caching versus
  token rotation is in both.
- **The guide resumes a hibernated VM with `qm start`, not
  `qm resume`.** A VM suspended to disk resumes from its state file on
  its next start; `qm resume` is for a VM paused in RAM.
- **The guide's `pvesm add dir proxy-seeds --path /var/lib/proxy-seeds
  --content iso` is the standard form, not a recorded command.** Phase 0
  says only that the user created `proxy-seeds` as a `dir` storage with
  ISO content; the path is illustrative.

The token-file check, run without touching a cluster: `KUBECTL=echo`
prints only the apply stage, and `KUBECTL=true` runs nothing. Scratch
files covered one good line, two lines, a bare secret, blank lines around
a good line, a non-PEM CA and a missing file. The real files on this
workstation pass:

```bash
make -n proxmox-token-secret
make -s proxmox-token-secret PROXMOX_TOKEN_FILE=$D/<case> PROXMOX_CA_FILE=$D/ca KUBECTL=echo
# good, ws: "apply -f -"; two, bad: "... is not one line of user@realm!tokenid=secret"; noca: "... holds no PEM certificate"
make -s proxmox-token-secret KUBECTL=true && echo "real files pass"   # real files pass
```

The mutation check for the sample tests edited one line per run,
asserting a single match. It ran the two tests, restored the file from a
copy, and at the end `sha256sum -c` confirmed the three files. All five
mutations failed the tests: `vlanTag: 5000` and `isoStorag:` in the
sample, `192.168.2.8` and `type: kvm2` in the ConfigMap, and
`image: debian-12` in the sample Proxy. `type: kvm2` was rerun after lint
removed the test's own type comparison. It still fails, because
`LoadConfig` refuses a proxmox block under another type.

The link sweep is a short script. It takes the docs as arguments, so it
can be rerun after any later edit. It skips fenced code and inline code
spans. For each relative link it checks that the file exists, and that
the `#anchor` matches a GitHub-style slug of a heading in the target file
(or the same file). Before the real run it was checked on a scratch copy
of the guide with five links added. It reported exactly the three bad
ones: a missing file, a missing anchor in `testing.md`, and a missing
same-file anchor. It checked 78 relative links in the six edited docs,
plus this log's, and found 0 broken:

```bash
cat > doclinks.py <<'PY'
import os, re, sys
def slug(h, seen):
    s = re.sub(r"[^\w\- ]", "", h.strip().lower()).replace(" ", "-")
    n = seen.get(s, 0); seen[s] = n + 1
    return s if n == 0 else f"{s}-{n}"
def prose(path):
    out, fence = [], False
    for line in open(path, encoding="utf-8"):
        if line.lstrip().startswith("```"): fence = not fence; continue
        if not fence: out.append(line)
    return out
def anchors(path):
    seen = {}
    return {slug(re.sub(r"`", "", l.lstrip("#").strip()), seen) for l in prose(path) if re.match(r"#{1,6} ", l)}
bad = 0
for doc in sys.argv[1:]:
    for n, line in enumerate(prose(doc), 1):
        for target in re.findall(r"\]\(([^)\s]+)\)", re.sub(r"`[^`]*`", "", line)):
            if re.match(r"[a-z]+:", target): continue
            f, _, a = target.partition("#")
            p = os.path.normpath(os.path.join(os.path.dirname(doc), f)) if f else doc
            if not os.path.exists(p): print(f"{doc}: missing file {target}"); bad += 1
            elif a and p.endswith(".md") and a not in anchors(p): print(f"{doc}: missing anchor {target}"); bad += 1
print(f"{bad} broken"); sys.exit(1 if bad else 0)
PY
python3 doclinks.py docs/proxmox-provider.md docs/providers-existing.md docs/architecture.md \
  README.md docs/providers.md CLAUDE.md docs/plans-executions/2026-09-18-1502-proxmox-provider.md   # 0 broken
```

Verification:

```bash
bin/kustomize build config/default >/dev/null && bin/kustomize build config/samples >/dev/null   # ok
go test -count=1 -run 'TestLoadConfig_proxmoxSample' -v ./internal/provider/                     # 3 PASS
grep -c '—' docs/proxmox-provider.md                                                             # 0
go build ./... && go vet ./... && go vet -tags=integration ./... && go test -short -count=1 ./...   # all ok
bin/golangci-lint run --disable staticcheck --max-same-issues 0 --max-issues-per-linter 0 ./internal/provider/
git status --short config/   # the four edited files named above, plus proxy_proxmox.yaml
```

Worth noting:

- Lint on `internal/provider` is at its baseline of three goconst hits.
  The first version of the new test pushed it to five: its
  `p.Type != "proxmox"` made `"proxmox"` a three-time literal, flagged in
  both `config.go` and the test. That comparison added nothing, so it
  went. `internal/provider/proxmox` was not touched.
- The `tmp` volume comment in `manager.yaml` still says only the kvm
  provider stages seeds there. kvm2 and proxmox do too. It was already
  stale for kvm2, so it was left out of this phase's diff.
- The architecture §5 block's right-hand bracket was aligned by
  character, not byte (`→` is three bytes), so every line puts `│` in
  column 83.
- Nothing in this phase talked to the Proxmox host or a cluster.
  `.claude/settings.json` stays out of the commit. The parked kvm2 items
  (S5.1, and the 200 ms `TestCheck_hostsCheckedConcurrently`) did not
  bite: `go test -short ./...` passed.
- Next is Phase 10: the first real run of `TestProxmox_lifecycle` and
  `TestProxmoxCheck_realHost` on pv1-1, with the env block in
  `docs/testing.md`. The three answers never captured go with it, and the
  guide's "Not yet proven end to end" gap and the README's matching
  clause come out once they pass. Then the CHANGELOG entry after the user
  confirms, and the MR.

## Phase 10 — Review, real-host validation, changelog, MR

Where the plan's Phase 10 text and the spec disagreed, the spec won, as in
every phase since Phase 1. The host is pv1-1 at `https://192.168.0.8:8006`,
not 192.168.2.8, the template is 9001 and the ISO storage `proxy-seeds`.

**Review.** Two read-only passes over `git diff main...HEAD`: the
`go-operator-reviewer` agent, briefed with the spec and the contract
(P1 to P6, S1 to S3), then `/code-review high` on the branch (CR1 to
CR10). Of the 19 findings (two found by both passes), 9 were fixed, 3
accepted as documented limits and 7 rejected with a reason, all recorded in
[docs/reviews/2026-09-23-proxmox-provider-review/REPORT.md](../reviews/2026-09-23-proxmox-provider-review/REPORT.md).
The one that mattered was CR1/P1. Delete destroyed the VM and then deleted
the seed, so a failed seed delete left a seed nothing would ever remove:
the next `Get` is NotFound, the finalizer goes, and GC lists only VMs.
Delete now removes the seed between the stop and the destroy, so the VM
`Get` looks at goes last. A 403 on the every-node path now fails instead
of being skipped. P2 (CR3) was the other code change: a probe VMID whose
listed name is someone else's is skipped at once, instead of waiting up to
the clone budget on its lock on every retry.

**Lint.** `make lint` still panics inside staticcheck on Go 1.27, so, as
in Phases 8 and 9:

```bash
bin/golangci-lint run --disable staticcheck --max-same-issues 0 --max-issues-per-linter 0 ./internal/provider/proxmox/... ./internal/provider/
bin/golangci-lint run --build-tags integration --disable staticcheck --max-same-issues 0 --max-issues-per-linter 0 ./internal/provider/proxmox/... ./internal/provider/
```

Each run was diffed against the pre-fix baseline by file and message,
with line numbers and occurrence counts stripped. Three new goconst hits
came in along the way, and all three were fixed rather than accepted. The
P2 test's lock value made `"backup"`, then `"migrate"`, a repeated
literal; it now uses `"snapshot-delete"`. The fake's missing-storage error
repeated `"Parameter verification failed."`; it is now built from the raw
captured body. Final counts: goconst 224 (231 with the tag), logcheck 12,
modernize 2, unparam 3, gocyclo 1.

**Real host.** The user ran the host checklist on pv1-1 and then:

```bash
export PROXMOX_TEST_URL=https://192.168.0.8:8006
export PROXMOX_TEST_TOKEN_FILE=$HOME/.config/proxmox-token
export PROXMOX_TEST_CA_FILE=$HOME/.config/proxmox-ca.pem
export PROXMOX_TEST_NODE=pv1-1
export PROXMOX_TEST_TEMPLATE_VMID=9001
export PROXMOX_TEST_ISO_STORAGE=proxy-seeds
export PROXMOX_TEST_VLAN_TAG=4
make test-integration-proxmox
# --- PASS: TestProxmox_lifecycle (59.68s)
# --- PASS: TestProxmoxCheck_realHost (0.13s)
```

It ran the review fixes, uncommitted, on the first try. Clone task about
3 s, Create 6.2 s, address 50.2 s after Create returned, Delete plus Get
plus a second Delete 3.1 s. The Delete deleted the seed while the stopped
VM's `ide2` still referenced it (imgdel `OK`), then destroyed the VM, and
step 10 found neither left. That settles the one assumption behind the
reorder that the fake could not prove. The run confirmed P6 as well: the
first `Get`, about 6 s after the clone, met `status: unknown` in the
listing and answered Transient, blaming the node, which was up. The
message now names both causes.

**The three answers Phase 0 never captured.** Captured with a scratch
helper that passes the token to curl through a process substitution and
writes the Phase 0 fixture shape. Its first offline test caught a bug:
Python's text mode folds `\r\n`, so the header split found nothing, and
the file is now opened with `newline=""`. The calls were:

```bash
pvecap storage_status_missing GET /nodes/pv1-1/storage/proxy-missing/status
pve POST /nodes/pv1-1/qemu/9001/clone -d newid=199999 -d name=proxy-reconaaaaaaaaaaa -d full=1
pve POST /nodes/pv1-1/qemu/199999/status/start
pve POST /nodes/pv1-1/qemu/199999/status/suspend                 # pause
pvecap stop_paused_response POST /nodes/pv1-1/qemu/199999/status/stop
pvecap task_status_stop_paused GET /nodes/pv1-1/tasks/<upid>/status
pvecap status_current_after_stop_paused GET /nodes/pv1-1/qemu/199999/status/current
pve POST /nodes/pv1-1/qemu/199999/status/start
pve POST /nodes/pv1-1/qemu/199999/status/suspend -d todisk=1     # hibernate: stopped, lock suspended
pvecap err_destroy_hibernated DELETE '/nodes/pv1-1/qemu/199999?purge=1&destroy-unreferenced-disks=1'
pvecap task_status_destroy_hibernated GET /nodes/pv1-1/tasks/<upid>/status
```

Two of the three contradicted the code:

- A missing storage is a **400** with the field error
  `storage: No such storage.`, not a 500, so the Checker reported
  `APIError`. `isNoSuchStorage` matches that exact field error, and a 500
  is still taken as missing (not captured). The matcher matrix caught a
  first version that fired on any `storage` field error, which a clone
  onto a storage without image support also has.
- The destroy **accepts a hibernated VM**. Its task ended `OK`, removing
  both disks and the `vm-199999-state-suspend-…` volume. Delete refused
  such a VM of ours as Permanent, believing the destroy would, so it now
  destroys it like a stopped one. Create's adoption still refuses one it
  finds before it was ever recorded. The capture was renamed
  `destroy_hibernated_response.json`, because the client test globs
  `err_*.json` as errors.
- A stop on a paused VM ends `OK` with the VM stopped, as assumed.

The destroy also cleaned up the scratch VM, and a listing of local-lvm
content for VMID 199999 came back empty.

**Kind pass.** The user built and loaded the image, deployed, applied a
scratch providers config with only `kubernetes` and `proxmox-home`, ran
the sample Proxy to Ready, ran a `curl -x` through it, deleted it and
checked pv1-1 for leftovers. Everything passed. The first attempt at the
override went to a ConfigMap named `providers-config`. `config/default`'s
`namePrefix` makes the mounted one `egress-proxies-operator-providers-config`,
so the pod kept the shipped config: no `proxmox-home` ProviderHealth, and
`kvm-home` and `kvm2-home` failing on the missing SSH key. The corrected
command replaced the prefixed ConfigMap and deleted the stray one. The
Secret keeps its plain name (`proxmox-token`), since kustomize renames
only resources it builds. `make deploy`'s image edit to
`config/manager/kustomization.yaml` was reverted, not committed.

Worth noting:

- **Behaviour changes from Phase 9.** Delete's order is now stop, seed,
  destroy. A hibernated VM of ours is destroyed rather than latched. The
  Checker reports a missing ISO storage as `StorageNotFound`. A
  foreign-named VM on a probe VMID is skipped without a config read.
- **Accepted, not fixed** (guide Known gaps, spec entry 24):
  - P3: Create adopts a leftover stopped VM whatever the current
    `placement` says.
  - P4: a retry can clone a second VM in a narrow crash window.
- **Still unproven:**
  - A storage that exists but cannot be used on a node.
  - Every cross-node path. The host has one node.
- **Flakes.** kvm's `TestKeepalive_keepsHealthyClientOpen` failed once
  under `-race` ("unexpected packet in response to channel open") and
  passed 5 of 5 on rerun. This branch does not touch `internal/provider/kvm`.
  The parked kvm2 items (S5.1, the 200 ms `TestCheck_hostsCheckedConcurrently`)
  did not bite, though the reviewer agent saw the latter fail once.
- **Docs.** The "not yet run" wording is gone from the README, the
  guide, `testing.md`, spec entry 21 and `providers-existing.md`.
  `testing.md` gained a timings table and the `PROXMOX_TEST_VLAN_TAG`
  export line. The link sweep script from the Phase 9 entry found 0
  broken links over the seven edited docs, and the guide has no `—`.
- `.claude/settings.json` stays out of the commit.