kvm2: apply the milestone review

Shorter start wait with immediate verdicts for dead or paused domains,
ListByTag without state or address RPCs, Timeout for an expired check
round, no adoption of unmarked domains, reference-counted keyed locks,
dead options removed, HEAD fallback on any non-2xx.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>

docs/kvm2-provider-spec.md

@@ -41,9 +41,12 @@ ID.
 **Answer:** `name-is-key`. libvirt addresses domains by name
 (`DomainLookupByName`); `req.Name` is used verbatim as the domain name and as
 the base of the per-VM volume names. Create looks the name up first and adopts
-an existing domain only if its metadata `uid` is empty or equals `req.UID`; a
-different non-empty `uid` returns `ErrPermanent` naming both UIDs. This closes
-the contract's "adoption is not verified" limit for this provider.
+an existing domain only if it carries the kvm2 mark (`managed` true, entry 8)
+and its metadata `uid` is empty or equals `req.UID`. A domain without the
+mark returns `ErrPermanent` saying it was not created by this provider: kvm2
+never starts or takes over a VM someone else defined under that name. A
+different non-empty `uid` returns `ErrPermanent` naming both UIDs. This
+closes the contract's "adoption is not verified" limit for this provider.
 
 **Evidence:** `DomainLookupByName(Name string) (Domain, error)` in the
 go-libvirt module; contract §2.1 idempotency and §13.
@@ -201,15 +204,24 @@ libvirt has no locks or tasks; every RPC is synchronous.
 **Answer:** `create-return`: `blocking`. `async-ops`: `none`. Create returns
 when the domain is running, holding one reconcile worker for the duration,
 including the first image download on a host (minutes on a cold pool; the
-guide says so). The wait after `DomainCreate` is capped at 2 minutes; every
-RPC has its own deadline (entry 18).
+guide says so). The wait after `DomainCreate` is capped at 15 seconds:
+`DomainCreate` is synchronous in libvirt, so the first `DomainGetState`
+normally answers running and the cap only covers state settling. Two rules
+keep the wait from pinning a worker: a state after the start that is neither
+active nor `nostate` (shutoff, crashed, paused, pmsuspended, shutdown) is
+`ErrTransient` at once, since polling will not bring that guest up; and a
+domain found paused or pmsuspended before the start is `ErrPermanent`
+(`cannot be started by the operator; resume or destroy it on the host`), an
+admin's doing that `DomainCreate` would refuse and the reconciler must not
+retry. Every RPC has its own deadline (entry 18).
 
 Steps, each idempotent, in order, all under the per-name lock (entry 19):
 
 1. Validate placement (entry 3). No backend call yet.
 2. `DomainLookupByName`. Found: adoption guard (entry 2), then start it if it
-   is not running (`DomainCreate`; `ErrOperationInvalid` means already
-   running), return the ID.
+   is not running (paused or pmsuspended is `ErrPermanent`; `DomainCreate`;
+   `ErrOperationInvalid` means already running), poll as in step 9, return
+   the ID.
 3. Pool space precheck (entry 13, quota).
 4. Ensure the base image (entry 4): marker present and matching means done.
 5. Overlay volume `<name>.qcow2`, qcow2 backed by the base file with
@@ -220,7 +232,9 @@ Steps, each idempotent, in order, all under the per-name lock (entry 19):
 7. `StorageVolGetPath` for the overlay and the seed.
 8. `DomainDefineXML` with file-type disks (entry 15's AppArmor note), the
    metadata element, firmware per image.
-9. `DomainCreate`, then poll `DomainGetState` until running.
+9. `DomainCreate`, then poll `DomainGetState` until running: `nostate` is
+   polled again, any other inactive state is `ErrTransient` at once, and the
+   15-second cap is `ErrTransient` too.
 10. Return `hosts/<host>/domains/<name>`.
 
 A retry after a crash finds whatever the previous attempt got to: an existing
@@ -267,13 +281,16 @@ contract §2.3.
 **Answer:** `fail-whole`, per host, with single unreadable domains skipped.
 Scope: every host in `hosts`, in config order, `ConnectListAllDomains` with
 active and inactive domains. For each domain, read the metadata element; keep
-those with `managed` true, then read state and addresses for the `Instance`.
-A domain whose metadata or state cannot be read (mid teardown, corrupt
-element) is skipped with a V(1) line; a host that cannot be dialled or whose
-listing fails fails the whole call, so GC skips this provider for the sweep
-rather than act on a partial view. Filtering is client-side; libvirt has no
-label index. Cost: one `DomainGetMetadata` per domain per sweep, about half a
-millisecond each over the tunnel.
+those with `managed` true and build the `Instance` from the domain name and
+the element alone: `ID`, `UID` and `CreatedAt`, which is all GC reads
+(contract §2.4). `State` and `IP` are left empty; no state or address RPC is
+made. A domain whose metadata cannot be read (mid teardown, corrupt element)
+is skipped with a V(1) line, unless ctx ended, which fails the call; a host
+that cannot be dialled or whose listing fails fails the whole call, so GC
+skips this provider for the sweep rather than act on a partial view.
+Filtering is client-side; libvirt has no label index. Cost: exactly one
+`DomainGetMetadata` per domain per sweep, about half a millisecond each over
+the tunnel.
 
 **Evidence:** execution log run 6 (per-domain metadata cost); contract §2.4
 and §4.7.
@@ -390,9 +407,11 @@ pooled connection, so a check that runs out of time closes only its own
 connection and cannot disrupt a reconcile. Cost: one SSH handshake and four
 cheap RPCs per host per round.
 
-Reasons: `Reachable`; `Timeout` (ctx ended before the check finished);
-`Unreachable` (TCP connect or the SSH transport failed, or the link died
-mid-check); `Unauthorized` (SSH authentication refused, host key unknown or
+Reasons: `Reachable`; `Timeout` (the round's ctx ended before the check
+finished, whichever stage was running, the dial included, or an RPC's own
+budget ran out once the connection was up); `Unreachable` (TCP connect or the
+SSH transport failed, including on the dial's own 10 s budget while the round
+still had time, or the link died mid-check); `Unauthorized` (SSH authentication refused, host key unknown or
 changed); `Forbidden` (`ErrAccessDenied` or `ErrAuthFailed` from libvirt: the
 socket refused the user); `APIError` (any other libvirt error);
 `CredentialsUnavailable` (the key or known_hosts file is missing or
@@ -458,7 +477,7 @@ goroutine and waits on the result, `ctx.Done()` and a timer:
 | `DomainCreate` | 90 s |
 | `StorageVolUpload` (seed) | 60 s |
 | `StorageVolUpload` (image), HTTP download | ctx, capped at 20 m |
-| wait for running after `DomainCreate` | 2 m |
+| wait for running after `DomainCreate` | 15 s (entry 10) |
 | HTTP client for images | no client timeout; the per-request ctx carries the cap |
 
 When a call is abandoned (ctx cancelled or budget exceeded), the adapter
@@ -479,10 +498,14 @@ every 15 s) marks the connection dead on failure.
 **Answer:** shared state is the host connection table (one pooled connection
 per host, dialled lazily, under a mutex with a per-host dial mutex so
 concurrent callers dial once and share the result), the per-`(host, volume)`
-download locks, and a keyed mutex per instance name. Create and Delete for the
-same name run under that name's lock, so two Deletes (GC and the reconciler)
-never race on destroy and undefine, and a Delete cannot interleave with a
-Create's steps. Get and ListByTag take no name lock. go-libvirt multiplexes
+download locks, and a lock per instance name. Both lock sets are one instance
+each of `keyedLocks`: a one-slot semaphore per key, so a waiter gives up when
+its ctx ends, reference-counted so an entry is removed from the map once no
+caller holds or waits for it and the map does not grow with every name ever
+seen, while two waiters on one key always share one semaphore. Create and
+Delete for the same name run under that name's lock, so two Deletes (GC and
+the reconciler) never race on destroy and undefine, and a Delete cannot
+interleave with a Create's steps. Get and ListByTag take no name lock. go-libvirt multiplexes
 RPCs on one connection, so calls for different names proceed concurrently
 on the shared connection. Three reconciles, a GC sweep and a check round at
 once are safe; the check uses its own connections.
@@ -598,7 +621,7 @@ entry after the user confirms.
 
 ## 24. Known limits accepted at design time
 
-**Answer:** eight limits, each with its workaround:
+**Answer:** nine limits, each with its workaround:
 
 - **Create blocks**, including the first image download per host, holding one
   of three reconcile workers. Workaround: warm the cache with one Proxy per
@@ -615,6 +638,11 @@ entry after the user confirms.
   isolated on its own connection.
 - **`Content-Length` is required** for image downloads; a chunked mirror is
   rejected as Permanent.
+- **A paused or pmsuspended domain is not started.** A repeat `Create` for
+  a domain in that state is `ErrPermanent` (entry 10), so the Proxy latches
+  as Failed until its spec changes, rather than retrying a `DomainCreate`
+  that libvirt refuses. Resume the domain on the host (`virsh resume`) and
+  touch the spec, or delete the Proxy.
 - **No v1 log-format compatibility**: the `libvirt call` lines have the same
   keys where the same facts exist, but dashboards keyed on `kvm` logger names
   need `kvm2`.