proxmox: review fixes, real-host validation record, changelog

Phase 10. Two review passes (go-operator-reviewer, /code-review high)
over the branch, recorded in docs/reviews/2026-09-23-proxmox-provider-review/:
9 findings fixed, 3 accepted as documented limits, 7 rejected with a
reason.

- Delete removes the seed between the stop and the destroy, so a failed
  seed delete leaves the VM for the next Delete instead of leaking the
  ISO; a Permanent refusal on the every-node path fails the call.
- Create skips a probe VMID whose listed name is someone else's without
  waiting on its lock.
- The three answers Phase 0 never captured are fixtures now: a missing
  storage is a 400 (the Checker reports StorageNotFound), a hibernated
  VM is destroyable (Delete destroys ours), a paused VM stops.
- TestProxmox_lifecycle and TestProxmoxCheck_realHost passed on pv1-1
  (PVE 9.2.20); the docs record the run and drop "not yet run".

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>

docs/proxmox-provider-spec.md

@@ -77,7 +77,11 @@ lands on the same number: `vmidFor(uid, attempt)` is
    clears by itself. Ownership is judged at once, as `Get` does (entry 9):
    a foreign hibernated VM is probed past, and ours is `ErrPermanent`
    (resume or destroy it), like kvm2's paused domain. Container and
-   template entries at a VMID are foreign without a config read.
+   template entries at a VMID are foreign without a config read, and so
+   is a VM whose listed name is not `req.Name`, whatever its lock: only a
+   clone in progress is nameless, and waiting on another VM's `backup`
+   lock would hold Create for the whole clone budget on every retry
+   (review finding P2).
 
 The wait in step 3 is what closes the duplicate-VM bug Phase 0 found: after
 a crash mid-clone, the retry's by-name lookup misses the nameless VM, but
@@ -284,7 +288,7 @@ and starts it; the reconciler has no ID for it to `Get`.
 | VMID absent | `/cluster/resources` | `ErrNotFound` |
 | `lock` set, other than `suspended` | resources entry (a clone in progress also has no name) | `ErrTransient` |
 | `lock: suspended` (hibernated to disk) | resources entry | Stopped |
-| `status: unknown` (node not reachable) | resources entry | `ErrTransient` |
+| `status: unknown` (node not reachable, or a VM seconds old the listing has not reported yet: the first `Get` after Create met this on pv1-1) | resources entry | `ErrTransient` |
 | config `does not exist` | `GET …/config` | `ErrNotFound` (deleted between the two calls) |
 | config `lock` set | config | `ErrTransient` |
 | `name` differs from the ID's name | config | `ErrNotFound`, logged: the VMID was reused |
@@ -382,34 +386,48 @@ the seed ISO; the VM's disks are removed by the destroy call itself, with
 
 1. Parse the ID (`ErrPermanent` if malformed).
 2. `/cluster/resources`. VMID absent, or held by a container: skip to
-   step 7, trying every configured node because the VM's node is no longer
+   step 6, trying every configured node because the VM's node is no longer
    known.
 3. `lock` set, other than `suspended`, or `status: unknown`: `ErrTransient`.
    A clone or another task still running finishes first; the reconciler
    calls Delete again after `DeletionPoll`.
-4. Read the config. `does not exist`: skip to step 7 on every node. A
+4. Read the config. `does not exist`: skip to step 6 on every node. A
    different name, or no mark: the VM is not ours; log it at V(1) and skip
-   to step 7 on every node without touching it. Our VM with a `lock` other
+   to step 6 on every node without touching it. Our VM with a `lock` other
    than `suspended` (taken since the listing): `ErrTransient`. Our VM with
-   `lock: suspended` (hibernated): `ErrPermanent` asking the admin to resume
-   or destroy it, the same answer Create's ownership check gives. The
-   destroy refuses a locked VM, and that lock never clears on its own, so
-   Transient would retry forever; Permanent puts the admin action in the
-   Proxy's status, where `reconcileManaged` latches it and the finalizer
-   path keeps retrying it visibly. A hibernated VM that is not ours is left
-   alone like any other.
+   `lock: suspended` (hibernated) is destroyed like a stopped one: the
+   listing shows it `stopped`, so step 5 is skipped, and the destroy
+   accepts it and removes its suspend state volume with the disks
+   (`destroy_hibernated_response.json`, `task_status_destroy_hibernated.json`,
+   captured in Phase 10). Until then Delete refused it as `ErrPermanent`
+   on the belief that the destroy would. A hibernated VM that is not ours
+   is left alone like any other.
 5. Status (from the listing) other than `stopped`: `POST …/status/stop`,
    wait (stop budget). Stop works on a paused VM too, which the listing
-   shows as `running`. `does not exist` means the VM is gone.
-6. `DELETE /nodes/{node}/qemu/{vmid}?purge=1&destroy-unreferenced-disks=1`,
-   wait (destroy budget). `does not exist` is success.
-7. `DELETE /nodes/{node}/storage/{isoStorage}/content/<isoStorage>:iso/<name>-seed.iso`,
-   wait (seed-delete budget), on the VM's node when steps 5 and 6 ran on a
-   VM of ours, and on every configured node in config order otherwise. A
-   missing volume is still a successful task, so no message is matched. On
-   the every-node path a node whose request or task fails is logged at V(1)
-   and skipped, unless ctx ended, which fails the call; on the VM's node a
-   failure fails the call (a 403 for `Datastore.Allocate` is Permanent).
+   shows as `running`: the task ends `OK` and the VM is `stopped`
+   (`stop_paused_response.json`, `task_status_stop_paused.json`,
+   `status_current_after_stop_paused.json`, captured in Phase 10).
+   `does not exist` means the VM is gone.
+6. `DELETE /nodes/{node}/storage/{isoStorage}/content/<isoStorage>:iso/<name>-seed.iso`,
+   wait (seed-delete budget), on the VM's node when step 5 was reached on
+   a VM of ours, and on every configured node in config order otherwise
+   (then step 7 is skipped). A missing volume is still a successful task,
+   so no message is matched. The seed goes **before** the destroy: once
+   the VM is gone `Get` is `ErrNotFound`, the reconciler releases the
+   finalizer and GC no longer lists it, so a seed left by a failed step
+   after the destroy would never be deleted (review finding CR1). On the
+   VM's node a failure fails the call and leaves the VM stopped for the
+   next Delete. On the every-node path a node whose request or task fails
+   is logged at V(1) and skipped, unless ctx ended or the failure is
+   `ErrPermanent` (a 403 for `Datastore.Allocate`, which every node
+   would give), which fail the call, so the missing privilege is visible
+   on both paths (review finding P1).
+7. `DELETE /nodes/{node}/qemu/{vmid}?purge=1&destroy-unreferenced-disks=1`,
+   wait (destroy budget). `does not exist` is success. The destroy leaves
+   the seed alone (Phase 0 deleted it after the destroy), which is why
+   step 6 exists. That Proxmox deletes an ISO a stopped VM's `ide2` still
+   references was not in the Phase 0 captures; the Phase 10 real-host run
+   proved it (imgdel `OK` on the stopped VM's seed, then the destroy).
 8. Return nil.
 
 Everything is gone when Delete returns nil, so `Get` afterwards is
@@ -474,6 +492,7 @@ per-parameter map. One `classify(op string, err error)`:
 | 401 `Authentication failed!`, 401 `No ticket` | Permanent | the token or its secret is wrong; fixed in the Secret |
 | 403 `Permission check failed (<path>, <privilege>)` | Permanent | the token lacks a privilege; the message names it |
 | 400 `Parameter verification failed.` with `errors` | Permanent | the request is wrong; the message includes the field errors |
+| 400 as above with `storage: No such storage.` | Permanent from a call; the Checker's `StorageNotFound` (entry 16) | the storage does not exist in the cluster |
 | other 4xx | Permanent | a request Proxmox will never accept |
 | 500 `Configuration file '…' does not exist` | NotFound (Transient when `op == "create"`) | the VM is gone |
 | 500 `VM is locked (<lock>)` | Transient | a task holds the VM |
@@ -490,17 +509,24 @@ per-parameter map. One `classify(op string, err error)`:
 
 Every matched string is backed by a captured response under `testdata/`;
 the table test loads them rather than repeating the text. Matching is on
-fixed substrings anchored to the 500 status: `does not exist`,
-`VM is locked`, `config file already exists`, `QEMU guest agent is not
-running`, `No QEMU guest agent configured`, and for the stopped VM the whole
-message against `^VM [0-9]+ is not running$`. The agent messages are tested
-in that order, because the last is a suffix of the first.
+fixed strings anchored to the 500 status: `does not exist`,
+`config file already exists`, and the whole messages
+`QEMU guest agent is not running` and `No QEMU guest agent configured`,
+plus, on a 400, the `storage` field error `No such storage.` (whole; a
+clone onto a storage without image support has another `storage` error).
+`VM is locked` and the stopped VM's `VM <vmid> is not running` are not
+matched by the provider: both are Transient through the default branch.
+The tests keep a matcher for each, with the whole stopped-VM message
+against `^VM [0-9]+ is not running$` (its tail is a suffix of the
+agent-not-running answer), only to hold the fake's wording to the
+captures (review finding CR5).
 
 **Evidence:** F1 (error text in the reason phrase and the body; 401 has no
 body); F2, F4, F5, F7 (the 500 messages); F6 (no out-of-space response
 could be produced on thin storage); F22 (both 401 texts); fixtures
 `err_forbidden.json`, `err_bad_param.json`, `err_tag_invalid.json`,
-`err_upload_bad_name.json`; the Phase 0 TLS finding (`192.168.2.8` not in
+`err_upload_bad_name.json`, and from Phase 10 `storage_status_missing.json`
+(a missing storage is a 400, not the 500 the spec had assumed); the Phase 0 TLS finding (`192.168.2.8` not in
 the node certificate); contract §5.
 
 ## 14. Credentials and TLS (`credentials`)
@@ -611,11 +637,15 @@ an admin:
 - `CredentialsUnavailable`: the token file is missing or malformed. Fix the
   Secret, not Proxmox.
 - `StorageNotFound`: `isoStorage` is missing, inactive, or has no `iso`
-  content. Fix the storage on the node. Any 500 from the storage status
-  call counts as missing, whatever its wording (it does not exist, is not
-  enabled on that node, or is offline; no missing-storage response was
-  captured, so no text is matched), and the message carries Proxmox's
-  text. A 401, 403, 595 or 596 there keeps its shared reason.
+  content. Fix the storage on the node. A storage the cluster does not
+  know answers `400 Parameter verification failed.` with the field error
+  `storage: No such storage.` (`storage_status_missing.json`, captured in
+  Phase 10); that exact field error is matched, and the message carries
+  it. Any 500 from the storage status call counts as missing too,
+  whatever its wording: Proxmox knows the storage but cannot use it on
+  that node (not enabled there, or offline). That answer was not
+  captured, so no text is matched. Any other 4xx, and a 401, 403, 595 or
+  596, keeps its shared reason.
 - `TemplateNotFound`: a configured template VMID does not exist or is not a
   template. Fix the template or the catalogue.
 
@@ -783,8 +813,8 @@ carries the cloud-init text.
 
 Contract §11 list, with the test for each item. Every name below exists
 (Phase 8 checked each with a grep for `func <name>(`). The two integration
-tests compile under `-tags=integration` but have not run against a host
-yet; Phase 10 runs them.
+tests passed on pv1-1 on 2026-09-24 (timings in
+[testing.md](testing.md#the-proxmox-integration-test-real-proxmox-ve-host)).
 
 - [x] repeat `Create`: `TestCreate_repeatReturnsSameIDAndCreatesNothing`
 - [x] `Get` after `Delete`: `TestGet_afterDeleteReturnsNotFound`
@@ -802,7 +832,7 @@ yet; Phase 10 runs them.
 - [x] Checker reasons, expired `ctx`, stable order: `TestCheck_reasonsTable`, `TestCheck_expiredContextReturnsPromptly`, `TestCheck_oneEntryPerNodeInConfigOrder`
 - [x] sentinel tests for `CloudInit` and the credential: `TestCreate_cloudInitSentinelNeverLoggedOrInErrors`, `TestClient_tokenNeverInErrorsOrLogs`, and `TestLogging_v0SilentV1OutcomeV2Detail` over a whole lifecycle at every level
 - [x] child spans under the operation span: `TestTracing_httpSpansUnderOperationSpan`, `TestClient_childSpanUnderParent`
-- [x] real-backend lifecycle test: `TestProxmox_lifecycle` and `TestProxmoxCheck_realHost` in `proxmox_integration_test.go` (compiled, not yet run)
+- [x] real-backend lifecycle test: `TestProxmox_lifecycle` and `TestProxmoxCheck_realHost` in `proxmox_integration_test.go`, passed on pv1-1
 
 Tests that cover the Phase 0 corrections specifically, in addition to the
 plan's per-phase lists:
@@ -904,4 +934,28 @@ user confirms.
   VMID range delete each other's VMs), an unconfigured provider latches the
   Proxy.
 
+Accepted after the Phase 10 review
+([docs/reviews/2026-09-23-proxmox-provider-review/](reviews/2026-09-23-proxmox-provider-review/REPORT.md),
+findings P3 and P4), and listed in the guide's Known gaps:
+
+- **A Create that failed permanently after its clone keeps its VM**, and
+  the next Create adopts it by name even when `placement` has changed.
+  Destroy the VM before changing `placement` on such a Proxy.
+- **A retry can clone a second VM** when a Create is cut off while
+  cloning onto a later probe VMID, the VM on the earlier one is removed,
+  and the retry comes before the clone ends. The next Create names both
+  VMIDs as `ErrPermanent`.
+
+Changed by the Phase 10 captures of the three answers Phase 0 never saw:
+
+- **A hibernated VM of ours** latches only Create's adoption
+  (`ErrPermanent`, resume or destroy it; entry 2). `Get` reports it
+  Stopped, so the reconciler replaces it, and Delete destroys it, since
+  the destroy takes a hibernated VM (entry 11). Before the capture, Delete
+  refused it as well.
+- **A missing `isoStorage`** is a 400 with `storage: No such storage.`,
+  which the Checker reports as `StorageNotFound` (entry 16); a missing
+  storage at upload time is Permanent like any 4xx.
+- **A paused VM** stops like a running one (entry 11, step 5), as assumed.
+
 **Evidence:** contract §13; entries 2, 7, 10, 11, 14 above.