proxmox: contract and observability tests, integration test, testing docs

The contract sweep in provider_test.go: every method and Check returns
within a second of a cancel, every wrapErr path carries the configured
name, op and id, the V0/V1/V2 logging tiers and the no-op lines, and a
Get over the real client whose HTTP spans are children of provider.get.
proxmox_integration_test.go (-tags=integration) holds TestProxmox_lifecycle
and TestProxmoxCheck_realHost; both compile and skip without
PROXMOX_TEST_*, and have not run against a host yet. make
test-integration-proxmox runs them. docs/testing.md gains the proxmox,
nocloud and keyedlock rows, the Live Proxmox gaps and the integration
section; spec entry 21's section 11 list is ticked.

Waits that could hang the package under a mutation are bounded: the
concurrent-Create loop and eight direct blocking calls in older tests.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>

docs/plans-executions/2026-09-18-1502-proxmox-provider.md

@@ -12,7 +12,7 @@ Pairs with [docs/plans/2026-09-18-1502-proxmox-provider.md](../plans/2026-09-18-
 - [x] Phase 5 — Get, ListByTag, task helpers
 - [x] Phase 6 — Create
 - [x] Phase 7 — Delete, Checker, CRD comment
-- [ ] Phase 8 — Contract and observability sweep, integration test, testing.md
+- [x] Phase 8 — Contract and observability sweep, integration test, testing.md
 - [ ] Phase 9 — Deployment and documentation
 - [ ] Phase 10 — Review, real-host validation, changelog, MR
 
@@ -939,3 +939,185 @@ Worth noting:
   hibernated VM, and a stop on a paused VM. The try-every-node seed delete
   and the multi-node Checker have only met the fake (entry 24).
 - Nothing in this phase talked to the host.
+
+## Phase 8 — Contract and observability sweep, integration test, testing.md
+
+The sweep went into the existing `provider_test.go` and reuses the earlier
+phases' helpers. The new helpers are `cancelWhileBlocked`, `within`,
+`newLoggingFixture` with `lifecycle`/`sentinelReq`, `assertNeverLogged`,
+`assertEveryV1Line` and `msgLine`.
+
+- `TestProvider_cancelledContextReturnsPromptly` blocks Create, Get,
+  Delete and ListByTag on `ClusterResources`, and Check on `Version`,
+  with `blockOn`. It asserts that the blocked call was the first in the
+  call log, then cancels. Each must return within 1 s, as a Transient
+  that still carries `context.Canceled` (Check: one target, `Timeout`).
+  Two more rows block mid-operation: Create on `UploadISO` and Delete on
+  `Stop`.
+- `TestProvider_everyErrorIsWrappedWithConfiguredName` has 20 rows, one
+  or more behind every `wrapErr` call site: placement, the name lock with
+  an ended ctx, resources, a 403 on the clone, a failed start task, the
+  malformed, missing, locked and 401 Get, the hibernated VM, a failed
+  destroy task, a 403 on the seed delete, and the list with a ctx that
+  ends on a candidate. Each row also checks `Error.ID` (empty for
+  Create) and the `provider proxmox-home: <op>` prefix.
+- `TestLogging_v0SilentV1OutcomeV2Detail` runs one whole lifecycle over
+  the observed fake with a token file: Create with a sentinel in
+  CloudInit, a Get that finds the captured address, ListByTag, Delete,
+  Delete of the gone VM, and Check. It runs at V(0), V(1) and V(2). V(0)
+  logs nothing. V(1) has each outcome line once, with its keys, and two
+  `proxmox vm deleted` lines: `destroyed=true` with node, then
+  `destroyed=false` without. Every line has the `proxmox` logger, the
+  provider and level 1, and no V(2) key. V(2) has the resolved line with
+  `cloudInitBytes`, agent addresses, the listed instance, and the
+  clone's description. The sentinel, `#cloud-config` and the token
+  secret appear at no level.
+- `TestLogging_noOpLinesAtV1` (a name beyond the plan) covers entry 20's
+  no-op lines. Each carries provider and vmid: VM already gone, foreign,
+  unmarked, seed delete skipped on each of two nodes, unreadable
+  candidate.
+- `TestTracing_httpSpansUnderOperationSpan` routes captures by path, as
+  `TestDelete_usesPurgeAndDestroyUnreferencedDisks` does. The captures
+  are `cluster_resources_during_destroy`, `config_during_destroy`,
+  `status_current_running` and `agent_network_get_interfaces`. A Get
+  runs over the real `httpAPI`, wrapped in `observe` and
+  `provider.WithTracing`, with the recorder's tracer passed through
+  `newHTTPAPI`'s opts. It asserts Running at 192.168.0.150, and exactly
+  four spans named `HTTP GET` (otelhttp v0.70's
+  `defaultTransportFormatter`), each a child of `provider.get` in its
+  trace. No attribute on any span carries the secret or `PVEAPIToken`.
+
+Judgment calls:
+
+- **"Instance lines carry vmid and name" is checked on the provider's
+  lines, not the decorator's.** Entry 20 gives `proxmox api call` the
+  fields `call`, `node`, `vmid`, `storage`, `upid` and `duration`, with
+  `name` only on a clone. So `assertEveryV1Line` exempts that line and
+  the two lines that are not about one instance (`vms listed`, `node
+  checked`), and requires vmid and name on all the others.
+- **The integration test compares `CreatedAt` with the mark Create
+  wrote.** kvm2 only checks that it is non-zero. Entry 21 says "equal
+  what Create wrote", so the test reads the VM's config through the
+  bare client and parses its description.
+- **Get's poll tolerates Transient.** kvm2's poll fails on any error.
+  Entry 7 makes every agent answer the provider cannot place Transient,
+  and a booting guest gives some. Those answers are logged and polled
+  through; any other class fails.
+- **The integration machine type has `diskGiB: 4`.** The 3 GiB recon
+  template is then grown by the resize task on the real host. A larger
+  template skips the resize.
+- **The seed check after Delete is one extra read-only GET in the test**:
+  `client.do` on `/nodes/{node}/storage/{storage}/content?content=iso`,
+  decoded the way `storage_content_iso.json` is shaped. No seam method
+  was added. The VM count and the by-name lookup use the existing
+  `ClusterResources`.
+- **The cleanup finds the VM by name**, so a Create that failed halfway
+  is also removed. With `PROXMOX_TEST_KEEP` it prints `qm stop <vmid>`
+  before `qm destroy <vmid> --purge`, because destroy refuses a running
+  VM. It then prints `pvesm free <isoStorage>:iso/<name>-seed.iso`.
+
+**Unbounded waits.** `TestCreate_concurrentSameNameSerialised` now uses
+the `first` channel shape of `TestDelete_concurrentSameNameSafe`. The grep
+found no other sleep loop; `client_test.go`'s `for {` reads multipart
+parts until EOF. The mutation check found a different shape of the same
+hang: tests that call a blocking operation directly on the test goroutine,
+where the lock or `blockOn` is released only after the call returns. Under
+"lockName ignores ctx", "Check ignores ctx" and "pollUntil ignores ctx"
+the package ran into its 60 s timeout on
+`TestDelete_waitsOnTheNameLock`, `TestLockName_givesUpWithContextAsTransient`,
+`TestCheck_expiredContextReturnsPromptly`, two `TestCheck_reasonsTable`
+rows, `TestCreate_cancelledMidWaitReturnsPromptly`,
+`TestDelete_cancelledMidWaitReturnsPromptly`, and the ended-ctx rows of
+`TestWaitTask_table` and `TestWaitUnlocked_table`. Each call now runs in
+`within(t, 5s or 10s, …)`, which fails the test by name. After the fix
+the same mutations fail those tests in seconds.
+
+The mutation check ran 14 mutations on production code. The script copied
+each file, applied the edit (asserting that it matched exactly once), and
+ran `go test -c` before counting a mutation. It then ran
+`go test -count=1 -timeout=60s`, restored the file from the copy, and
+asserted the sha256 matched. Afterwards `sha256sum -c` over the eight
+files hashed up front was OK. `task.go`, added for M14, was covered by the
+script's own assertion. `git status` showed no production file changed.
+Results:
+
+- Caught by the new tests:
+  - Get drops the span context before its requests →
+    `TestTracing_httpSpansUnderOperationSpan`.
+  - Cloud-init logged at V(2) → `TestLogging_…` and
+    `TestCreate_cloudInitSentinel…`.
+  - Delete's destroy path skips `wrapErr` → the wrap test and 6 Delete
+    tests.
+  - ListByTag's ended-ctx path skips `wrapErr` → the wrap test's `list
+    ended on a candidate` row.
+  - Delete's resolve with `context.Background()` → the cancel test's
+    `delete` row.
+  - `lockName` with `context.Background()` → the wrap test's two lock
+    rows.
+  - Check's probe with `context.Background()` → the cancel test's
+    `check` row.
+  - Delete's logger without the name, `destroyed` always true, `vms
+    listed` at V(0), the clone description at V(1), and the list skip
+    line without vmid → `TestLogging_*`.
+- Caught only by the older tests: `pollUntil` without its `ctx.Done()`
+  case → the four waits tests above. The first attempt deleted the case
+  and did not compile (the `provider` import became unused). The rerun
+  replaced it with a nil channel.
+- Survived, and equivalent: `newHTTPAPI` without the tracer-provider
+  option. The transport starts a span only under a valid parent. With no
+  `WithTracerProvider`, otelhttp v0.70 takes the tracer from
+  `trace.SpanFromContext(r.Context()).TracerProvider()`
+  (`transport.go:99`), so the child always lands in the parent's
+  provider. No test can tell the two apart.
+
+`Makefile`: `test-integration-proxmox` next to the kvm2 target, with a
+`##` help line naming the five required variables. `docs/testing.md`
+gained:
+
+- rows for `nocloud`, `keyedlock` and five for `proxmox` (both packages
+  were missing);
+- a "Live Proxmox, by default" bullet under "deliberately do NOT cover",
+  naming the cross-node paths and the three answers never captured;
+- a "The proxmox integration test" section: variables, the eleven
+  assertions in order, the KEEP cleanup, the cost, and that it has not
+  run yet.
+
+In spec entry 21 every §11 box is ticked. The list's intro says the two
+integration tests compile but have not run against a host. The cancel and
+sentinel items name the new coverage.
+
+```bash
+# the names in entry 21, each checked before ticking: 42 of 42 exist
+awk '/^## 21\. Tests/,/^## 22\./' docs/proxmox-provider-spec.md | grep -o '`Test[A-Za-z0-9_]*`' | tr -d '`' | sort -u \
+  | while read n; do grep -rqE "^func $n\(" internal/provider/proxmox/ && echo "ok $n" || echo "MISSING $n"; done
+go test -race -count=2 ./internal/provider/proxmox/                        # ok
+go vet -tags=integration ./... && go vet -tags=e2e ./...                    # ok
+make -n test-integration-proxmox
+# go test -race -tags=integration -count=1 -v -timeout 30m -run TestProxmox ./internal/provider/proxmox/
+go test -tags=integration -count=1 -v -run TestProxmox ./internal/provider/proxmox/   # both SKIP: PROXMOX_TEST_URL is not set
+go build ./... && go test -short -count=1 ./...                             # all ok
+go test -count=1 -v ./internal/provider/proxmox/ | grep -c '^=== RUN'      # 649 before, 688 after
+bin/golangci-lint run --disable staticcheck --max-same-issues 0 --max-issues-per-linter 0 ./internal/provider/proxmox/...
+bin/golangci-lint run --build-tags integration --disable staticcheck --max-same-issues 0 --max-issues-per-linter 0 ./internal/provider/proxmox/...
+```
+
+Worth noting:
+
+- Lint is at the baseline:
+  - the logcheck hits (4/4/2/1/1), mapsloop ×2 in `proxmox.go`, unparam
+    ×3 in `fake_test.go`, gocyclo in `api_test.go`, and goconst on test
+    strings.
+  - Two new gocyclo hits were removed, not accepted.
+    `TestLogging_v0SilentV1OutcomeV2Detail` was at 32; the per-line loop
+    became `assertEveryV1Line`. `TestProxmox_lifecycle` was at 40; steps
+    6 and 7 became `assertMarkReadBack` and `assertListed`. kvm2's
+    lifecycle test is still at 39.
+- The integration tests only compile so far. The first real run is Phase
+  10's. That run meets the Datastore.Allocate point too: a token without
+  it on the ISO storage fails step 9 as Permanent. Phase 0 granted it on
+  `proxy-seeds`, and it is already a Phase 9 guide item.
+- The parked items did not bite: kvm2's S5.1 and its 200 ms
+  `TestCheck_hostsCheckedConcurrently` passed in every
+  `go test -short ./...` run of this phase.
+- `.claude/settings.json` stays out of the commit. Nothing in this phase
+  talked to the host.