proxmox: contract and observability tests, integration test, testing docs

The contract sweep in provider_test.go: every method and Check returns
within a second of a cancel, every wrapErr path carries the configured
name, op and id, the V0/V1/V2 logging tiers and the no-op lines, and a
Get over the real client whose HTTP spans are children of provider.get.
proxmox_integration_test.go (-tags=integration) holds TestProxmox_lifecycle
and TestProxmoxCheck_realHost; both compile and skip without
PROXMOX_TEST_*, and have not run against a host yet. make
test-integration-proxmox runs them. docs/testing.md gains the proxmox,
nocloud and keyedlock rows, the Live Proxmox gaps and the integration
section; spec entry 21's section 11 list is ticked.

Waits that could hang the package under a mutation are bounded: the
concurrent-Create loop and eight direct blocking calls in older tests.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>

docs/testing.md

@@ -67,6 +67,13 @@ manual runs too.
 | | `image_test.go`, `create_test.go`, `get_test.go`, `delete_test.go`, `list_test.go`, `kvm2_test.go`, `provider_test.go` | image cache (fetch with the marker written last, cached, marker missing or without a digest, sha256 change or mismatch, 4xx/5xx, no Content-Length, short read, concurrent callers fetch once, cancel mid-stream, streamed without buffering, upload on a dedicated connection) and the pool-space quota precheck; Create resume after every partial step, returns only when running, adoption guard on UID, seed always rebuilt, cloud-init sentinel never logged; Get state and address mapping incl. agent-before-ARP; every Delete already-gone case, NVRAM flag, per-name lock; ListByTag across hosts, one down fails the whole call, unreadable domain skipped; `New` never dials and warns on missing files; every error wrapped with the configured name; logging tiers V0/V1/V2 | `httptest` image server + in-memory `fakeBackend` behind the `backend` seam |
 | | `observe_test.go`, `check_test.go` | every seam method traced and logged exactly once (guarded by reflection), child spans under the parent, detail only at V2, upload bodies never logged, error fields; health check: one entry per host in config order, hosts checked concurrently, expired ctx returns promptly, never touches the pooled connection, the reason table, detail keys | `fakeBackend` + `funcr` capture + span recorder |
 | | `kvm2_integration_test.go` (`-tags=integration`) | one real VM: `New` without a dial, `Check` Ready with details, Create, idempotent re-Create with an unchanged domain count, Running with IP and the metadata round-trip, ListByTag, Delete, NotFound, pool state afterwards, `kvm2.*` spans under `provider.create`; `TestKVM2Check_realHost` runs the health check against the host and creates nothing | a **real hypervisor** — see below |
+| `internal/provider/nocloud` | `seed_test.go` | the NoCloud seed kvm2 and proxmox share: the ISO reopened (files, label, user-data byte for byte, temp dir removed, an empty vendor-data still a file, an empty name refused) and the vendor-data rules (a shell script, not cloud-config; multipart with configured extra vendor-data; the agent install switched off) | pure |
+| `internal/provider/keyedlock` | `keyedlock_test.go` | the per-name lock kvm2 and proxmox share: the entry is removed once uncontended, waiters share one lock, a waiter gives up with its ctx | pure |
+| `internal/provider/proxmox` | `id_test.go`, `mark_test.go`, `state_test.go`, `seed_test.go`, `errors_test.go`, `api_test.go` | pure pieces: providerID round-trip and malformed forms, the VMID derived from the UID (deterministic, in range, probing forward and wrapping), the description mark (keys, parsed from the captured clone, admin lines kept), state mapping and the IPv4 filter from captures, seed names and building (empty name Permanent without its payload, staging failure Transient); the `classify` table and the 500-message matchers against every captured error; the API types decoded from the captures | pure, over the responses Phase 0 captured from PVE 9.2.20 (`testdata/`) |
+| | `client_test.go` | the thin HTTP client: the token read from its file on every request (missing Transient, unparsable Permanent), `apiError` from status, reason phrase and body, the data envelope, method, path and parameters of every seam call, the clone and destroy query shapes, the volid's slash escaped, HTTP/1.1 only, request timeout and cancel Transient and the client unaffected, a child span under a parent and none without, the token never in errors or logs, the CA file trusted and loaded lazily, an x509 failure Permanent, insecure mode, the upload streamed as multipart | `httptest.NewTLSServer` replaying the captures byte for byte |
+| | `create_test.go`, `get_test.go`, `delete_test.go`, `list_test.go`, `task_test.go`, `proxmox_test.go`, `provider_test.go`, `fake_test.go` | Create: resume after every partial step and after a crash mid-clone, the VMID probe and ownership check, the template's shape refused before any clone, resize as a task, the seed overwritten without a delete, the net0 bridge and VLAN rewrite keeping the MAC, returns only when running, cloud-init sentinel never logged; Get: lock checked before name, the agent's messages, paused from `status/current`, hibernated, a reused VMID, four calls when running; Delete: every already-gone and not-ours case, the seed deleted on the VM's node or on every node, hibernated Permanent, the per-name lock; ListByTag: candidates by tag or name prefix, the mark decides, an unreadable VM skipped unless ctx ended; task and lock waits; `New` makes no request; the contract sweep: every method returns within a second of a cancel (blocked on its first seam call, and mid-operation), every error wrapped with the configured name, op and id, logging tiers V0/V1/V2 with the no-op lines | in-memory `fakeAPI` behind the `api` seam, its error texts checked against the captures |
+| | `observe_test.go`, `check_test.go`, `provider_test.go` | the decorator: every seam method logs exactly one line (guarded by reflection), starts no span, identifiers at V1, detail at V2, upload body never logged, error fields; a Get through `provider.WithTracing` over the real client makes four `HTTP GET` spans, all children of `provider.get` and none carrying the token; health check: one entry per node in config order, nodes checked concurrently (a barrier, not a timer), expired ctx returns promptly, the reasons table at V2 with a sentinel secret, the transport reasons from real `net/http` and `crypto/tls` errors, detail keys | `fakeAPI` + `funcr` capture + span recorder; the real client over `httptest` for the tracing and transport-reason rows |
+| | `proxmox_integration_test.go` (`-tags=integration`) | one real VM: `New` without a request, `Check` Ready with details, Create, idempotent re-Create with an unchanged VM count, Running with an IPv4 and the mark read back, ListByTag, Delete, NotFound, no VM and no seed left, HTTP spans under `provider.create`; `TestProxmoxCheck_realHost` runs the health check against the host and creates nothing. **Compiles, not yet run against a host** | a **real Proxmox VE host** — see below |
 | `internal/health` | `probe_test.go` | probes through a **real CONNECT-capable proxy stub** to a real TLS server: tunnel success, refused CONNECT, wrong status, dead proxy, plain-http forward | `httptest` + hijacked bidirectional tunnel |
 | | `engine_test.go` | thresholds, first-verdict, latency suppression matrix, dropped-event retry, stale-probe UID guard, tick scheduling/seeding/pruning, `Start` end-to-end | fake `client.Reader` + injected `probeFn` |
 | `internal/lease` | `store_test.go` | capacity, `MaxLeases=0`, all three selection tie-breaks, target-scoped vs global cooldowns, expiry via fake clock, report-on-retained-lease, idempotent release, **40 concurrent acquires never exceeding capacity** | injectable clock; the concurrency case is why `-race` matters |
@@ -105,6 +112,25 @@ envtest suite the scaffold generated.
   boots and reports an address are provable only on a real host — that is
   the opt-in integration test below (one per provider).
 
+- **Live Proxmox, by default**: the proxmox unit tests run over an
+  in-memory fake of the 16 seam calls, and the HTTP client over
+  `httptest` replaying responses captured from PVE 9.2.20. Whether a
+  clone, config write, seed upload and start yield a guest that boots and
+  reports an address is provable only on a real host (the proxmox
+  integration test below). Two kinds of path have met only the fake:
+  - **the cross-node paths**: a clone that targets another node, the seed
+    deleted on every node when the VM's node is unknown, a Check round over
+    several nodes. The test host is a single node, so the integration test
+    cannot reach them either
+    ([proxmox-provider-spec.md](proxmox-provider-spec.md) entry 24);
+  - **answers never captured**: what the storage status call returns for a
+    missing or disabled storage (every 500 there is taken as
+    `StorageNotFound`), what destroy says for a hibernated VM, and a stop
+    on a paused VM. The provider does not depend on their wording: any
+    500 from the storage status is `StorageNotFound`, a hibernated VM is
+    refused before destroy is called, and a paused VM gets the same stop
+    as a running one. What the fake answers there is its own guess.
+
 ## The KVM integration test (real hypervisor)
 
 `internal/provider/kvm/kvm_integration_test.go` compiles only under
@@ -256,6 +282,110 @@ go test -tags=integration -count=1 -v -run TestKVM2Check ./internal/provider/kvm
 The test is not part of `make test` and not run in CI. Keep it compiling
 with `go vet -tags=integration ./...` when the provider's internals move.
 
+## The proxmox integration test (real Proxmox VE host)
+
+`internal/provider/proxmox/proxmox_integration_test.go` compiles only
+under `-tags=integration` and skips itself unless the environment names a
+host. It deviates from the testcontainers convention for the same reason
+as the kvm tests: Proxmox VE is a hypervisor, not something a container
+can stand in for. It needs a node with a cloud-init template without a
+Proxmox cloud-init drive and with the guest agent enabled, and a token
+with the privileges of [proxmox-provider-spec.md](proxmox-provider-spec.md)
+entry 14: the role on `/` from the plan's "Before Phase 0" section
+([2026-09-18-1502-proxmox-provider.md](plans/2026-09-18-1502-proxmox-provider.md)),
+plus `Datastore.Allocate` on the ISO storage. A passing run leaves
+nothing behind: its VM and its seed ISO are gone.
+
+```sh
+export PROXMOX_TEST_URL=https://192.168.0.8:8006           # host must be in the node certificate
+export PROXMOX_TEST_TOKEN_FILE=$HOME/.config/proxmox-token  # one line: user@realm!tokenid=secret
+export PROXMOX_TEST_CA_FILE=$HOME/.config/proxmox-ca.pem    # the node's pve-root-ca.pem
+export PROXMOX_TEST_NODE=pv1-1
+export PROXMOX_TEST_TEMPLATE_VMID=9001
+export PROXMOX_TEST_ISO_STORAGE=proxy-seeds                 # holds iso content
+make test-integration-proxmox
+```
+
+Required: `PROXMOX_TEST_URL`, `PROXMOX_TEST_TOKEN_FILE`,
+`PROXMOX_TEST_NODE`, `PROXMOX_TEST_TEMPLATE_VMID` and
+`PROXMOX_TEST_ISO_STORAGE`; the test skips when any is missing. Optional:
+`PROXMOX_TEST_CA_FILE` (empty trusts the system roots),
+`PROXMOX_TEST_INSECURE` (`true` skips certificate verification; the log
+says so), `PROXMOX_TEST_STORAGE` (the clone's target storage; empty keeps
+the template's), `PROXMOX_TEST_BRIDGE` and `PROXMOX_TEST_VLAN_TAG` (the
+cloned `net0`'s bridge and tag; empty keeps the template's),
+`PROXMOX_TEST_VMID_MIN` and `PROXMOX_TEST_VMID_MAX` (default
+`100000`..`199999`), `PROXMOX_TEST_TIMEOUT` (default `20m`, the budget for
+the whole lifecycle), `PROXMOX_TEST_KEEP` to leave the VM behind, and
+`PROXMOX_TEST_LOG_LEVEL` (default `1`): at `1` every request is one
+`proxmox api call` line with the call, node, VMID, UPID and duration,
+plus the per-operation outcome lines; `2` adds request parameters, config
+summaries and agent addresses; `0` leaves only `t.Log`. Provider, node,
+name and VMID are stated once and left off every provider line. Without
+`Datastore.Allocate` on the ISO storage the seed delete at the end of
+Delete is a 403, and Delete fails as Permanent.
+
+What one run asserts, in order, each step logged with its duration:
+
+1. `New` returns within 100 ms and makes no request (no `proxmox api
+   call` line before the first operation).
+2. `Check` reports the node `Ready` with reason `Reachable` and the
+   `pveVersion`, `kernel`, `tokenID`, `isoStorageAvailGiB` and
+   `templates` (`1/1`) details, which are logged.
+3. Every later call goes through `provider.WithTracing`, with an
+   in-memory span recorder on the transport as well.
+4. `Create` returns `vms/<vmid>/<name>` with the VMID in range. The
+   machine type is 1 core, 1024 MiB and a 4 GiB disk, so a 3 GiB template
+   is grown by the resize task.
+5. A second `Create` returns the same ID and the cluster's QEMU VM count
+   is unchanged.
+6. `Get`, polled every 5 s, reaches `Running` with an IPv4 within the
+   budget (a Transient answer while the guest boots is logged and polled
+   through). Its `UID` and `CreatedAt` equal the mark in the VM's
+   description, read back from its config, and `CreatedAt` is within a
+   few minutes of now. The time from Create's return to the first address
+   is logged.
+7. `ListByTag` contains the ID with that UID and CreatedAt.
+8. A TCP connect to port 22 on the guest, logged only: the machine
+   running the test may not route to the VM's network.
+9. `Delete` returns nil, `Get` is `ErrNotFound`, a second `Delete` is a
+   no-op.
+10. No VM with the name is left in the cluster's resources, and no
+    `<isoStorage>:iso/<name>-seed.iso` in the storage's iso content. The
+    content is read with one extra read-only GET of
+    `/nodes/{node}/storage/{storage}/content?content=iso` made by the test
+    itself (`storage_content_iso.json` is its captured shape): the seam
+    has no method to list storage content, and does not get one for a
+    test.
+11. Among the recorded spans, a `provider.create` span has at least one
+    `HTTP` child, and no span attribute carries the cloud-init text.
+
+With `PROXMOX_TEST_KEEP` steps 9 and 10 are skipped, and at the end the
+log prints what to run as root on the node to clean up:
+`qm stop <vmid>`, `qm destroy <vmid> --purge` and
+`pvesm free <isoStorage>:iso/<name>-seed.iso`. Otherwise a `t.Cleanup`
+runs the Delete even when an earlier assertion failed. It finds the VM by
+name, so a Create that failed halfway is cleaned up too.
+
+Cost: one full clone of the template (about 2 s for the 3 GiB template on
+pv1-1), one boot to an address (about 50 s), and a few seconds of
+destroy; a run takes a couple of minutes. The Makefile target's 30 m
+timeout covers the 20 m budget plus the cleanup.
+
+`TestProxmoxCheck_realHost` runs the provider health check against the
+same host. It must find the node reachable, the ISO storage usable and
+the template present, and it creates nothing. It needs the same
+variables; to run only this test:
+
+```sh
+go test -tags=integration -count=1 -v -run TestProxmoxCheck ./internal/provider/proxmox/
+```
+
+**Neither test has run against a host yet.** Both compile
+(`go vet -tags=integration ./...`); Phase 10 of the plan runs them on
+pv1-1. Like the kvm tests they are not part of `make test` and not run in
+CI.
+
 ## The scaffolded `make test-e2e` suite
 
 `test/e2e/` is the kubebuilder-generated smoke suite (build image → kind