docs/architecture.md

@@ -172,7 +172,8 @@ closes the SSH connection to unblock it.
 
 kvm2 provider (internal/provider/kvm2/) - the same transport, a second type built from the contract
 prov.Create ──► placement checked against the catalogue (Permanent before any dial)
-                DomainLookupByName: exists → uid guard (foreign uid = Permanent),  ─┐ every step
+                DomainLookupByName: exists → mark+uid guard (no mark or foreign   ─┐ every step
+                  uid = Permanent; paused = Permanent),                            │
                   start if needed, wait for running, done                          │ idempotent,
                 else: StoragePoolGetInfo precheck: image (if uncached) + diskSizeGb │ under a
                         + 16 MiB must fit, else ErrQuotaExceeded (reconciler       │ per-name lock
@@ -184,7 +185,7 @@ prov.Create ──► placement checked against the catalogue (Permanent before
                       seed ISO (always rebuilt)                                    │
                       DomainDefineXML (<metadata> in kvm2's own namespace:         │
                         managed, uid, createdAt) + DomainCreate, poll until        │
-                        running (2m cap)                                          ─┘
+                        running (15s cap; a domain that dies at once = Transient) ─┘
 prov.Get    ──► DomainLookupByName + DomainGetMetadata + DomainGetState
                   + DomainInterfaceAddresses (guest agent, then ARP; first routable IPv4)
                   running without IP = Provisioning; nostate = Provisioning;

docs/kvm2-provider-spec.md

@@ -41,9 +41,12 @@ ID.
 **Answer:** `name-is-key`. libvirt addresses domains by name
 (`DomainLookupByName`); `req.Name` is used verbatim as the domain name and as
 the base of the per-VM volume names. Create looks the name up first and adopts
-an existing domain only if its metadata `uid` is empty or equals `req.UID`; a
+an existing domain only if it carries the kvm2 mark (`managed` true, entry 8)
-different non-empty `uid` returns `ErrPermanent` naming both UIDs. This closes
+and its metadata `uid` is empty or equals `req.UID`. A domain without the
-the contract's "adoption is not verified" limit for this provider.
+mark returns `ErrPermanent` saying it was not created by this provider: kvm2
+never starts or takes over a VM someone else defined under that name. A
+different non-empty `uid` returns `ErrPermanent` naming both UIDs. This
+closes the contract's "adoption is not verified" limit for this provider.
 
 **Evidence:** `DomainLookupByName(Name string) (Domain, error)` in the
 go-libvirt module; contract §2.1 idempotency and §13.
@@ -201,15 +204,24 @@ libvirt has no locks or tasks; every RPC is synchronous.
 **Answer:** `create-return`: `blocking`. `async-ops`: `none`. Create returns
 when the domain is running, holding one reconcile worker for the duration,
 including the first image download on a host (minutes on a cold pool; the
-guide says so). The wait after `DomainCreate` is capped at 2 minutes; every
+guide says so). The wait after `DomainCreate` is capped at 15 seconds:
-RPC has its own deadline (entry 18).
+`DomainCreate` is synchronous in libvirt, so the first `DomainGetState`
+normally answers running and the cap only covers state settling. Two rules
+keep the wait from pinning a worker: a state after the start that is neither
+active nor `nostate` (shutoff, crashed, paused, pmsuspended, shutdown) is
+`ErrTransient` at once, since polling will not bring that guest up; and a
+domain found paused or pmsuspended before the start is `ErrPermanent`
+(`cannot be started by the operator; resume or destroy it on the host`), an
+admin's doing that `DomainCreate` would refuse and the reconciler must not
+retry. Every RPC has its own deadline (entry 18).
 
 Steps, each idempotent, in order, all under the per-name lock (entry 19):
 
 1. Validate placement (entry 3). No backend call yet.
 2. `DomainLookupByName`. Found: adoption guard (entry 2), then start it if it
-   is not running (`DomainCreate`; `ErrOperationInvalid` means already
+   is not running (paused or pmsuspended is `ErrPermanent`; `DomainCreate`;
-   running), return the ID.
+   `ErrOperationInvalid` means already running), poll as in step 9, return
+   the ID.
 3. Pool space precheck (entry 13, quota).
 4. Ensure the base image (entry 4): marker present and matching means done.
 5. Overlay volume `<name>.qcow2`, qcow2 backed by the base file with
@@ -220,7 +232,9 @@ Steps, each idempotent, in order, all under the per-name lock (entry 19):
 7. `StorageVolGetPath` for the overlay and the seed.
 8. `DomainDefineXML` with file-type disks (entry 15's AppArmor note), the
    metadata element, firmware per image.
-9. `DomainCreate`, then poll `DomainGetState` until running.
+9. `DomainCreate`, then poll `DomainGetState` until running: `nostate` is
+   polled again, any other inactive state is `ErrTransient` at once, and the
+   15-second cap is `ErrTransient` too.
 10. Return `hosts/<host>/domains/<name>`.
 
 A retry after a crash finds whatever the previous attempt got to: an existing
@@ -267,13 +281,16 @@ contract §2.3.
 **Answer:** `fail-whole`, per host, with single unreadable domains skipped.
 Scope: every host in `hosts`, in config order, `ConnectListAllDomains` with
 active and inactive domains. For each domain, read the metadata element; keep
-those with `managed` true, then read state and addresses for the `Instance`.
+those with `managed` true and build the `Instance` from the domain name and
-A domain whose metadata or state cannot be read (mid teardown, corrupt
+the element alone: `ID`, `UID` and `CreatedAt`, which is all GC reads
-element) is skipped with a V(1) line; a host that cannot be dialled or whose
+(contract §2.4). `State` and `IP` are left empty; no state or address RPC is
-listing fails fails the whole call, so GC skips this provider for the sweep
+made. A domain whose metadata cannot be read (mid teardown, corrupt element)
-rather than act on a partial view. Filtering is client-side; libvirt has no
+is skipped with a V(1) line, unless ctx ended, which fails the call; a host
-label index. Cost: one `DomainGetMetadata` per domain per sweep, about half a
+that cannot be dialled or whose listing fails fails the whole call, so GC
-millisecond each over the tunnel.
+skips this provider for the sweep rather than act on a partial view.
+Filtering is client-side; libvirt has no label index. Cost: exactly one
+`DomainGetMetadata` per domain per sweep, about half a millisecond each over
+the tunnel.
 
 **Evidence:** execution log run 6 (per-domain metadata cost); contract §2.4
 and §4.7.
@@ -390,9 +407,11 @@ pooled connection, so a check that runs out of time closes only its own
 connection and cannot disrupt a reconcile. Cost: one SSH handshake and four
 cheap RPCs per host per round.
 
-Reasons: `Reachable`; `Timeout` (ctx ended before the check finished);
+Reasons: `Reachable`; `Timeout` (the round's ctx ended before the check
-`Unreachable` (TCP connect or the SSH transport failed, or the link died
+finished, whichever stage was running, the dial included, or an RPC's own
-mid-check); `Unauthorized` (SSH authentication refused, host key unknown or
+budget ran out once the connection was up); `Unreachable` (TCP connect or the
+SSH transport failed, including on the dial's own 10 s budget while the round
+still had time, or the link died mid-check); `Unauthorized` (SSH authentication refused, host key unknown or
 changed); `Forbidden` (`ErrAccessDenied` or `ErrAuthFailed` from libvirt: the
 socket refused the user); `APIError` (any other libvirt error);
 `CredentialsUnavailable` (the key or known_hosts file is missing or
@@ -458,7 +477,7 @@ goroutine and waits on the result, `ctx.Done()` and a timer:
 | `DomainCreate` | 90 s |
 | `StorageVolUpload` (seed) | 60 s |
 | `StorageVolUpload` (image), HTTP download | ctx, capped at 20 m |
-| wait for running after `DomainCreate` | 2 m |
+| wait for running after `DomainCreate` | 15 s (entry 10) |
 | HTTP client for images | no client timeout; the per-request ctx carries the cap |
 
 When a call is abandoned (ctx cancelled or budget exceeded), the adapter
@@ -479,10 +498,14 @@ every 15 s) marks the connection dead on failure.
 **Answer:** shared state is the host connection table (one pooled connection
 per host, dialled lazily, under a mutex with a per-host dial mutex so
 concurrent callers dial once and share the result), the per-`(host, volume)`
-download locks, and a keyed mutex per instance name. Create and Delete for the
+download locks, and a lock per instance name. Both lock sets are one instance
-same name run under that name's lock, so two Deletes (GC and the reconciler)
+each of `keyedLocks`: a one-slot semaphore per key, so a waiter gives up when
-never race on destroy and undefine, and a Delete cannot interleave with a
+its ctx ends, reference-counted so an entry is removed from the map once no
-Create's steps. Get and ListByTag take no name lock. go-libvirt multiplexes
+caller holds or waits for it and the map does not grow with every name ever
+seen, while two waiters on one key always share one semaphore. Create and
+Delete for the same name run under that name's lock, so two Deletes (GC and
+the reconciler) never race on destroy and undefine, and a Delete cannot
+interleave with a Create's steps. Get and ListByTag take no name lock. go-libvirt multiplexes
 RPCs on one connection, so calls for different names proceed concurrently
 on the shared connection. Three reconciles, a GC sweep and a check round at
 once are safe; the check uses its own connections.
@@ -598,7 +621,7 @@ entry after the user confirms.
 
 ## 24. Known limits accepted at design time
 
-**Answer:** eight limits, each with its workaround:
+**Answer:** nine limits, each with its workaround:
 
 - **Create blocks**, including the first image download per host, holding one
   of three reconcile workers. Workaround: warm the cache with one Proxy per
@@ -615,6 +638,11 @@ entry after the user confirms.
   isolated on its own connection.
 - **`Content-Length` is required** for image downloads; a chunked mirror is
   rejected as Permanent.
+- **A paused or pmsuspended domain is not started.** A repeat `Create` for
+  a domain in that state is `ErrPermanent` (entry 10), so the Proxy latches
+  as Failed until its spec changes, rather than retrying a `DomainCreate`
+  that libvirt refuses. Resume the domain on the host (`virsh resume`) and
+  touch the spec, or delete the Proxy.
 - **No v1 log-format compatibility**: the `libvirt call` lines have the same
   keys where the same facts exist, but dashboards keyed on `kvm` logger names
   need `kvm2`.

docs/kvm2-provider.md

@@ -235,10 +235,12 @@ Its `<metadata>` carries one element in kvm2's own namespace,
 
 That is how orphan GC recognises its own VMs, since libvirt has no
 labels, and how a repeat `Create` knows whether an existing domain of the
-same name belongs to this Proxy: a different non-empty `uid` is a
+same name belongs to this Proxy: a domain without the element is a
-permanent error naming both UIDs; an empty one is adopted. The namespace
+permanent error (`carries no kvm2 mark; it was not created by this
-is kvm2's own, so the `kvm` provider's domains are invisible to it (see
+provider`), a different non-empty `uid` is a permanent error naming both
-*Migrating from kvm*).
+UIDs, and a marked domain with a matching or empty `uid` is adopted. The
+namespace is kvm2's own, so the `kvm` provider's domains are invisible to
+it (see *Migrating from kvm*).
 
 The seed (`buildSeedISO` in `seed.go`) is an ISO 9660 image labelled
 `cidata` with three files. `user-data` is the Proxy's `cloudInit`, **byte
@@ -321,8 +323,10 @@ Before creating any volume, `Create` reads `StoragePoolGetInfo` and
 compares the pool's free space with what it is about to commit
 (`checkPoolSpace` and `neededBytes` in `image.go`): the image's
 `Content-Length` when the base is not cached yet (found with a `HEAD`
-request, or a `GET` whose body is closed at once when the server refuses
+request, or a `GET` whose body is closed at once when `HEAD` answers
-`HEAD`), plus `diskSizeGb` in bytes for the overlay, plus 16 MiB for the
+anything but 2xx, since mirrors refuse it with 405, 403 or 501 alike; the
+`GET`'s status is then what counts), plus `diskSizeGb` in bytes for the
+overlay, plus 16 MiB for the
 seed. Below that it returns `ErrQuotaExceeded` with a message naming the
 pool, the bytes available and the bytes needed. The reconciler answers
 quota with a wait, not a failure: the Proxy shows `Provisioned=False`
@@ -347,9 +351,12 @@ by changing `type: kvm` to `type: kvm2` and `kvm:` to `kvm2:`. The
 
 What does not port is the VMs. kvm2 marks its domains in its own metadata
 namespace and neither lists nor adopts domains the `kvm` provider made:
-`ListByTag` skips them, so orphan GC never sees them, and a Proxy's
+`ListByTag` skips them, so orphan GC never sees them, and `Create` refuses
-`spec.provider` is immutable, so an existing Proxy cannot be moved to the
+a domain of the same name that lacks the kvm2 mark with a permanent error
-new entry anyway. Do it in this order:
+rather than start it (a Proxy recreated under the new entry gets the same
+deterministic name, so this is what happens when step 1 below is skipped).
+A Proxy's `spec.provider` is immutable, so an existing Proxy cannot be
+moved to the new entry anyway. Do it in this order:
 
 1. Delete the Proxies on the `kvm` entry and wait for them to go; the
    `kvm` provider removes their domains and volumes.
@@ -437,7 +444,7 @@ applied by `call`):
 | `DomainCreate` | 90 s |
 | `StorageVolUpload` of at most 16 MiB (the seed, the marker) | 60 s |
 | `StorageVolUpload` of an image, `StorageVolDownload`, and the whole image fetch from HTTP request to marker | 20 m; the caller's ctx may end it earlier |
-| wait for running after `DomainCreate` | 2 m, polled every 500 ms (`defaultRunningWait` and `defaultRunningPoll` in `create.go`) |
+| wait for running after `DomainCreate` | 15 s, polled every 500 ms (`defaultRunningWait` and `defaultRunningPoll` in `create.go`); `DomainCreate` is synchronous, so the first poll normally answers running, and a domain that is shut off, crashed or paused right after the start fails at once rather than waiting out the cap |
 | the `HEAD` request of the pool-space precheck | 30 s |
 | the courtesy `ConnectClose` when a connection is closed | 5 s |
 
@@ -499,10 +506,10 @@ A failing target says where the check broke (`checkReason` in `check.go`):
 |---|---|---|
 | `Reachable` | ready | |
 | `CredentialsUnavailable` | the key or known_hosts file is missing or does not parse | the `kvm-ssh` Secret (see *Cluster-side setup*); fixed in the cluster, not on the host |
-| `Unreachable` | the TCP connect failed or ran out of the 10 s dial budget, the SSH transport failed for a reason other than authentication, the forwarded libvirt socket was refused, or the link died mid-check | DNS, Tailscale or a firewall between the cluster and `address`; `libvirtd` running and the `socket` path |
+| `Unreachable` | the TCP connect failed or ran out of the 10 s dial budget while the round still had time, the SSH transport failed for a reason other than authentication, the forwarded libvirt socket was refused, or the link died mid-check | DNS, Tailscale or a firewall between the cluster and `address`; `libvirtd` running and the `socket` path |
 | `Unauthorized` | the SSH server rejected the key, or the host key is missing from known_hosts or has changed | `user`'s `authorized_keys`; the known_hosts entry (re-run `ssh-keyscan` and update the Secret) |
 | `Forbidden` | SSH worked, but libvirt refused the user (`ErrAccessDenied` or `ErrAuthFailed`) | `user` in the `libvirt` group |
-| `Timeout` | the connection was up, but the monitor's `--provider-check-timeout` (default 15s) or an RPC's budget ran out before the check finished | a hung `libvirtd` or a saturated link |
+| `Timeout` | the monitor's `--provider-check-timeout` (default 15s) ran out before the check finished, whichever stage was running, the dial included; or the connection was up and an RPC's own budget ran out | a hung `libvirtd`, a saturated link, or a round timeout shorter than the dial takes |
 | `PoolNotFound` | there is no pool named `pool` | `virsh pool-list --all` |
 | `PoolNotActive` | the pool exists but is not running | `virsh pool-start <pool>` and `virsh pool-autostart <pool>` |
 | `APIError` | any other libvirt error | the message carries libvirt's own text |
@@ -518,10 +525,10 @@ waits 5 minutes on `ErrQuotaExceeded`, recreates on `ErrNotFound` from
 
 | Class | What lands there |
 |---|---|
-| `ErrPermanent` | placement values not in the catalogue; a malformed providerID; a host no longer in the config (from `Get`; `Delete` treats it as done); libvirt `ErrXMLError`, `ErrNoSupport`, `ErrInvalidArg`, `ErrAuthFailed`, `ErrAccessDenied` and `ErrNoStoragePool`; a pool that is not running; SSH authentication refused (`unable to authenticate`); a host key missing from known_hosts or changed (`knownhosts.KeyError`); a key or known_hosts file that exists but does not parse; a domain of the same name owned by another UID; corrupt metadata on a domain; an image URL answering 4xx, a response without `Content-Length` or with length 0, a sha256 that does not match the configured one |
+| `ErrPermanent` | placement values not in the catalogue; a malformed providerID; a host no longer in the config (from `Get`; `Delete` treats it as done); libvirt `ErrXMLError`, `ErrNoSupport`, `ErrInvalidArg`, `ErrAuthFailed`, `ErrAccessDenied` and `ErrNoStoragePool`; a pool that is not running; SSH authentication refused (`unable to authenticate`); a host key missing from known_hosts or changed (`knownhosts.KeyError`); a key or known_hosts file that exists but does not parse; a domain of the same name owned by another UID or carrying no kvm2 mark; a domain of the same name that is paused or pmsuspended; corrupt metadata on a domain; an image URL answering 4xx, a response without `Content-Length` or with length 0, a sha256 that does not match the configured one |
 | `ErrQuotaExceeded` | the pool-space precheck |
 | `ErrNotFound` | libvirt `ErrNoDomain` or `ErrNoStorageVol` from `Get`, `Delete` or `ListByTag`; inside `Create` the same codes are `ErrTransient`, because the contract forbids NotFound from Create |
-| `ErrTransient` | everything else: a missing key or known_hosts file (the Secret has not landed yet), TCP and network errors, context cancellation, an abandoned or budget-exceeded call, a connection killed by another caller, libvirt `ErrOperationInvalid` and every unrecognised libvirt code, an image URL answering 429 or 5xx or ending the body early, the wait for running expiring |
+| `ErrTransient` | everything else: a missing key or known_hosts file (the Secret has not landed yet), TCP and network errors, context cancellation, an abandoned or budget-exceeded call, a connection killed by another caller, libvirt `ErrOperationInvalid` and every unrecognised libvirt code, an image URL answering 429 or 5xx or ending the body early, the wait for running expiring or the domain going shut off or crashed right after its start |
 
 Error strings name paths, hosts, volumes and libvirt's own text. They
 never contain key bytes, the known_hosts content or cloud-init.
@@ -556,12 +563,12 @@ never contain key bytes, the known_hosts content or cloud-init.
 - **`Content-Length` is required** for image downloads; a mirror that
   only chunks is rejected permanently. Point the catalogue at one that
   sends the length.
-- **A paused or pmsuspended domain cannot be adopted.** A repeat `Create`
+- **A paused or pmsuspended domain is not started.** A repeat `Create`
-  for a domain in that state calls `DomainCreate`, which libvirt refuses
+  for a domain in that state fails permanently (`is DomainPaused and
-  as "already running"; the provider then waits for the domain to report
+  cannot be started by the operator; resume or destroy it on the host`),
-  running, the 2-minute wait expires, and the call fails transiently on
+  so the Proxy shows `Failed` until its spec changes; the operator never
-  every retry. Resume the domain on the host (`virsh resume <name>`) or
+  resumes a VM an admin paused. Resume it on the host
-  delete the Proxy.
+  (`virsh resume <name>`) and touch the spec, or delete the Proxy.
 - **Parser text in credential errors.** A key or known_hosts file that
   does not parse fails with the parser's own message, which can quote a
   PEM header type or a malformed known_hosts token. Key bytes are never

docs/plans-executions/2026-09-16-1333-kvm2-provider.md

@@ -12,7 +12,7 @@ Plan: [docs/plans/2026-09-16-1333-kvm2-provider.md](../plans/2026-09-16-1333-kvm
 - [x] Step 7 — Checker and the CRD comment
 - [x] Step 8 — Deployment files, Makefile, integration test
 - [x] Step 9 — Documentation
-- [ ] Step 10 — Milestone review
+- [x] Step 10 — Milestone review
 - [ ] Step 11 — Real-host validation, changelog, MR
 
 ## Step 0 — Plan file and branch
@@ -338,3 +338,45 @@ resolves. The one added em-dash is the `Leader-elected` cell of the new
 components-table row, where the table's own convention wins.
 
 Attestation: "files read outside the allowed set: none".
+
+## Step 10 — Milestone review
+
+2026-09-16 21:19. The `go-operator-reviewer` agent reviewed the package
+against the contract and the spec (v1 code off limits to it as well) and
+returned seven findings, all accepted and applied by a fresh fix agent in
+one commit with regression tests:
+
+1. `waitRunning`'s two-minute cap only ever elapsed in the bad cases and
+   pinned a reconcile worker; now 15 s, a domain that dies right after
+   start is Transient at once, and a paused or pmsuspended domain is
+   Permanent before `DomainCreate` (an admin's doing).
+2. `ListByTag` read state and addresses that GC never uses, one agent RPC
+   per active domain per sweep; it now builds instances from the name and
+   the metadata only (`TestListByTag_doesNotReadStateOrAddresses`).
+3. A health round that ran out of time was blamed on the host
+   (`Unreachable`); `checkHost` now reports `Timeout` whenever the round's
+   ctx ended, whichever dial stage was running, and the promptness test
+   uses the production error shape.
+4. `Create` adopted an unmarked domain and never wrote the mark, so GC could
+   not see it; an unmarked domain under the name is now Permanent
+   (`TestCreate_adoptsOnlyMarkedDomains`).
+5. `hostTable.closeAll` had no production caller and `withBudgets` could not
+   work (the dialer captured the defaults first); both removed.
+6. The per-name and per-image lock maps grew without bound; replaced by a
+   reference-counted `keyedLocks` whose entries vanish when uncontended.
+7. `imageSizeIfUncached` falls back to GET on any non-2xx HEAD, and two
+   Delete no-op lines moved from V(0) to V(1).
+
+The spec (entries 2, 10, 12, 16, 18, 19, 24), the guide, the appendix and
+the architecture flow were updated in the same commit. The reviewer also
+listed what it checked and found correct: the error boundary, the state
+rules, Delete's idempotency and the replacement window, ListByTag's
+partial-failure policy, the abandon policy and double-close guards,
+resource handling on every path, and secret hygiene in the format strings.
+
+Lint: `bin/golangci-lint` still panics on this toolchain (Go 1.27 against
+a staticcheck built for 1.26), reproducible on untouched packages, so
+`make lint` could not gate this step; `go vet` with every build tag is
+clean. Tests: `go test -race -count=3 ./internal/provider/kvm2/` (168
+top-level tests) and `go test -short ./...` pass; `make test` with envtest
+runs next.

docs/providers-existing.md

@@ -30,12 +30,12 @@ and the symbol that shows it.
 | Axis | kubernetes | gcp | kvm | kvm2 |
 |---|---|---|---|---|
 | `readiness` | immediate: a Pod is scheduled within seconds (`stateFromPod`) | provisioning-window: minutes of boot (`mapState`) | provisioning-window: boot plus, on first use, an image download (`imageCache.ensure`) | provisioning-window: boot plus, on first use, an image download (`ensureImage`) |
-| `create-return` | submit: returns after `client.Create` | submit: `Insert` is never awaited, `Operation.Wait` is not called | **blocking**: waits through `DomainCreate` (`ensureRunning`), and the first image fetch runs inside (`imageCache.fetch`) | **blocking**: waits through `DomainCreate` and a 2-minute poll for running (`ensureRunning`, `waitRunning`), and the first image fetch runs inside (`fetchImage`) |
+| `create-return` | submit: returns after `client.Create` | submit: `Insert` is never awaited, `Operation.Wait` is not called | **blocking**: waits through `DomainCreate` (`ensureRunning`), and the first image fetch runs inside (`imageCache.fetch`) | **blocking**: waits through `DomainCreate` and a 15-second poll for running (`ensureRunning`, `waitRunning`), and the first image fetch runs inside (`fetchImage`) |
 | `delete-return` | returns-early: `client.Delete`, Pod terminates on its own | returns-early: `Delete` submits an operation each time | waits: destroy, undefine and volume removal complete before return (`removeDomain`, `removeVolumes`) | waits: destroy, undefine and volume removal complete before return (`removeDomain`, `removeVolumes`) |
 | `ip-source` | assigned-at-create: `PodIP` (`instanceFromPod`) | assigned-at-create: the external NAT IP of the first NIC (`toInstance`) | discovered: ARP table first, then the guest agent, first routable IPv4 (`address`, `firstIPv4`) | discovered: the guest agent first, then the ARP table, first routable IPv4 (`addressSources`, `address`, `firstIPv4`) |
 | `gc-mark` | native-labels: Pod labels (`buildPod`) | native-labels: instance labels (`buildInsertRequest`) | written-metadata: a namespaced `<metadata>` element written in `DomainDefineXML` (`metadata.go`) | written-metadata: an element in kvm2's own namespace written in `DomainDefineXML` (`metadataURI`, `newInstanceMetadata`) |
 | `created-at` | backend-timestamp: `CreationTimestamp` | backend-timestamp: `CreationTimestamp`, parse error gives zero | written-at-create: `createdAt` in the metadata element (`newInstanceMetadata`) | written-at-create: `createdAt` in the metadata element, parse error gives zero (`newInstanceMetadata`, `createdAt`) |
-| `list-partial` | n/a: one call, all namespaces | omit-unreachable: `ReturnPartialSuccess` on `AggregatedList` | fail-whole per host, one unreadable domain skipped (`ListByTag`, `managedInstance`) | fail-whole per host, one unreadable domain skipped unless ctx ended (`ListByTag`) |
+| `list-partial` | n/a: one call, all namespaces | omit-unreachable: `ReturnPartialSuccess` on `AggregatedList` | fail-whole per host, one unreadable domain skipped (`ListByTag`, `managedInstance`) | fail-whole per host, one unreadable domain skipped unless ctx ended; `ID`, `UID` and `CreatedAt` from the name and the mark only, no state or address RPC (`ListByTag`) |
 | `quota` | absent: a quota 403 is Permanent (`classify`) | detected: 429, and 403 with reason `quotaExceeded` or `rateLimitExceeded` (`classify`, `hasReason`) | absent (`classify`) | detected: a pool-space precheck before any volume is created (`checkPoolSpace`, `neededBytes`) |
 | `not-found` | status-404: `apierrors.IsNotFound` | status-404: `isNotFound` | error-code: `ErrNoDomain`, `ErrNoStorageVol` (`hasCode`) | error-code: `ErrNoDomain`, `ErrNoStorageVol`, Transient inside Create (`classifyLibvirt`) |
 | `primary-key` | name-is-key: `namespace/name` | name-is-key: `zones/<zone>/instances/<name>` | name-is-key: `hosts/<host>/domains/<name>` | name-is-key: `hosts/<host>/domains/<name>`, with a UID guard on adoption (`create`) |
@@ -56,7 +56,7 @@ against.
 
 | Feature | kubernetes | gcp | kvm | kvm2 |
 |---|---|---|---|---|
-| Create verifies the adopted instance's UID | no | no | no | **yes**: a different non-empty UID in the metadata is Permanent, an empty one is adopted (`create`) |
+| Create verifies the adopted instance's UID | no | no | no | **yes**: a domain without the kvm2 mark or with a different non-empty UID is Permanent; a marked one with a matching or empty UID is adopted (`create`) |
 | Placement used | none | Zone, MachineType, Image | Zone (host), MachineType, Image | Zone (host), MachineType, Image |
 | CloudInit delivered | **no**: ignored | yes: `user-data` metadata | yes: NoCloud seed ISO built in-process (`buildSeedISO`) | yes: NoCloud seed ISO built in-process (`buildSeedISO`) |
 | Port used | yes: Squid config and containerPort (`squidConf`) | no: firewall is manual | no | no |
@@ -372,9 +372,10 @@ same libvirt.
   logs the resolved parameters with `cloudInitBytes` at V(2). Then, under
   the per-name lock (`lockName`) and on the pooled connection
   (`hostTable.get`), `create`: `DomainLookupByName`; found means the
-  adoption guard (a non-empty metadata UID that differs from `req.UID` is
+  adoption guard (no kvm2 mark, or a non-empty metadata UID that differs
-  Permanent) and `ensureRunning`. Otherwise `StoragePoolLookupByName`,
+  from `req.UID`, is Permanent) and `ensureRunning`. Otherwise
-  `imageSizeIfUncached` (marker lookup, then `HEAD` or a closed `GET`) and
+  `StoragePoolLookupByName`, `imageSizeIfUncached` (marker lookup, then
+  `HEAD`, or a closed `GET` when `HEAD` is not 2xx) and
   `checkPoolSpace` with `neededBytes` (image if uncached, plus
   `diskSizeGb`, plus 16 MiB), then `ensureImage` (`cachedImage` on the
   pooled connection, `fetchImage` streaming into `uploadImage` on a
@@ -382,10 +383,11 @@ same libvirt.
   `ensureOverlay` (`ErrStorageVolExist` means reuse), `writeSeed` (delete,
   build, upload; always rebuilt), two `StorageVolGetPath`, `buildDomainXML`
   with file-type disks and the metadata element, `DomainDefineXML`,
-  `ensureRunning` (`DomainGetState`; `DomainCreate` with
+  `ensureRunning` (`DomainGetState`, where paused or pmsuspended is
-  `ErrOperationInvalid` as already running; `waitRunning` polls every
+  Permanent; `DomainCreate` with `ErrOperationInvalid` as already running;
-  500 ms for up to 2 minutes, Transient on expiry). `Create` returns only
+  `waitRunning` polls every 500 ms for up to 15 seconds, Transient on
-  when the domain is running.
+  expiry and at once for any inactive state other than `nostate`).
+  `Create` returns only when the domain is running.
 - **Get and state.** `mapState(state, ip)`:
 
   | libvirt state | IP | Provider state |
@@ -398,8 +400,8 @@ same libvirt.
   `get`: `DomainLookupByName` (only `ErrNoDomain` reaches NotFound),
   `readMetadata` (`DomainGetMetadata` with `metadataURI`;
   `ErrNoDomainMetadata` means unmanaged with an empty UID; an element that
-  does not parse is Permanent), `inspect` (`DomainGetState`, then for an
+  does not parse is Permanent), then `DomainGetState` and, for an active
-  active domain `address`). `addressSources` asks the guest agent first,
+  domain, `address`. `addressSources` asks the guest agent first,
   then ARP; a source that fails is skipped at V(2), only ctx ending is an
   error, so a cancelled Get is Transient and not a false Provisioning.
   `firstIPv4` drops loopback, link-local and unspecified addresses and an
@@ -413,9 +415,12 @@ same libvirt.
   when nil is returned.
 - **ListByTag.** `ConnectListAllDomains` (active and inactive) on every
   host in config order; `readMetadata` and `managed` keep kvm2's own
-  domains; `inspect` fills state and address. A domain whose metadata or
+  domains, and the `Instance` is `ID`, `UID` and `CreatedAt` from the name
-  state cannot be read is skipped at V(1), unless ctx ended, which fails
+  and the element, with `State` and `IP` empty: GC reads nothing else
-  the call. A host that cannot be dialled or listed fails the whole call.
+  (contract §2.4), so no state or address RPC is made. A domain whose
+  metadata cannot be read is skipped at V(1), unless ctx ended, which
+  fails the call. A host that cannot be dialled or listed fails the whole
+  call.
 - **Errors.** `classify(op, err)`: an error already carrying a class keeps
   it; libvirt codes through `classifyLibvirt` (`ErrNoDomain` and
   `ErrNoStorageVol` NotFound, Transient when `op == "create"`;
@@ -436,14 +441,16 @@ same libvirt.
   `ConnectGetHostname`, `StoragePoolLookupByName`, `StoragePoolGetInfo`;
   a pool that is not running is `poolNotActiveError`. Details are filled
   as facts arrive (`user` and `pool` always; `libvirtVersion`, `hostname`,
-  `poolState`, `poolCapacityGiB`, `poolAvailableGiB`). `checkReason`: a
+  `poolState`, `poolCapacityGiB`, `poolAvailableGiB`). `checkHost` answers
-  `dialError` is judged by its stage (`credentials` gives
+  `Timeout` whenever the round's ctx ended, whatever stage was running and
-  `CredentialsUnavailable`; `ssh` with a `knownhosts.KeyError` or the auth
+  whatever shape the error has; otherwise `checkReason`: a `dialError` is
-  text gives `Unauthorized`; any other stage, including a dial that ran
+  judged by its stage (`credentials` gives `CredentialsUnavailable`; `ssh`
-  out of time, gives `Unreachable`); then `context.DeadlineExceeded` or
+  with a `knownhosts.KeyError` or the auth text gives `Unauthorized`; any
-  `Canceled` `Timeout`, `ErrAccessDenied` or `ErrAuthFailed` `Forbidden`,
+  other stage, including a dial that ran out of its own budget, gives
-  `ErrNoStoragePool` `PoolNotFound`, any other libvirt code `APIError`,
+  `Unreachable`); then `context.DeadlineExceeded` or `Canceled` `Timeout`,
-  anything else `Unreachable`; `PoolNotActive` from the pool state.
+  `ErrAccessDenied` or `ErrAuthFailed` `Forbidden`, `ErrNoStoragePool`
+  `PoolNotFound`, any other libvirt code `APIError`, anything else
+  `Unreachable`; `PoolNotActive` from the pool state.
 - **Connections.** `hostTable` keeps one pooled connection per host,
   dialled lazily under a per-host one-slot semaphore so waiting ends with
   the caller's ctx; a dead backend (`liveness`) is closed and redialled on
@@ -510,8 +517,9 @@ same libvirt.
   - Cancelling one operation cuts off the others on that host once: a
     cancelled `call` kills the pooled connection.
   - `Content-Length` is required for image downloads.
-  - A paused or pmsuspended domain cannot be adopted by a repeat `Create`:
+  - A paused or pmsuspended domain is not started by a repeat `Create`:
-    `DomainCreate` refuses, `waitRunning` expires, Transient on every retry.
+    `ensureRunning` returns Permanent, so the Proxy latches until its spec
+    changes; the admin resumes the VM on the host.
   - Credential parse errors carry the parser's own text, which can quote a
     PEM header type or a malformed known_hosts token (never key bytes).
   - No reused-overlay check: an existing overlay is reused without checking

internal/provider/kvm2/check.go

@@ -62,9 +62,15 @@ func (p *Provider) checkHost(ctx context.Context, host provider.KVMHost) provide
 		ts.Reason = provider.ReasonReachable
 	default:
 		var inactive *poolNotActiveError
-		if errors.As(err, &inactive) {
+		switch {
+		case ctx.Err() != nil:
+			// The round ran out of time, whichever stage was running: the
+			// error's shape (a dial that saw the ctx end, an abandoned
+			// call) says nothing the admin should act on.
+			ts.Reason = provider.ReasonTimeout
+		case errors.As(err, &inactive):
 			ts.Reason = reasonPoolNotActive
-		} else {
+		default:
 			ts.Reason = checkReason(err)
 		}
 		ts.Message = err.Error()
@@ -121,10 +127,11 @@ func (e *poolNotActiveError) Error() string {
 	return fmt.Sprintf("storage pool %q is %s", e.pool, e.state)
 }
 
-// checkReason maps a failed probe to a reason. A dial failure is judged by
+// checkReason maps a failed probe to a reason once checkHost has ruled out
-// its stage, so a TCP connect or an SSH handshake that ran out of time is
+// the round's ctx ending. A dial failure is judged by its stage, so a TCP
-// Unreachable, not Timeout: the host did not answer. Timeout is for ctx
+// connect or an SSH handshake that ran out of the dial budget on its own is
-// ending or a budget expiring once the connection was up.
+// Unreachable, not Timeout: the host did not answer. Timeout is for a
+// budget expiring once the connection was up.
 func checkReason(err error) string {
 	var de *dialError
 	if errors.As(err, &de) {

internal/provider/kvm2/check_test.go

@@ -33,7 +33,9 @@ func checkContext() (context.Context, *logCapture) {
 }
 
 // checkDialer hands out a distinct fake per host name, fails the hosts in
-// errs, counts every dial, and, with block set, waits for ctx to end.
+// errs, counts every dial, and, with block set, waits for ctx to end and
+// then fails the way dialSSHSocket does when the TCP connect sees the ctx
+// end: a dialError at the tcp stage wrapping the ctx's error.
 type checkDialer struct {
 	backends map[string]*fakeBackend
 	errs     map[string]error
@@ -46,7 +48,7 @@ func (d *checkDialer) dial(ctx context.Context, host provider.KVMHost) (backend,
 	d.dials.Add(1)
 	if d.block {
 		<-ctx.Done()
-		return nil, fmt.Errorf("%w: dialing %s: %w", provider.ErrTransient, host.Name, ctx.Err())
+		return nil, &dialError{stage: stageTCP, host: host.Name, err: fmt.Errorf("dial tcp %s: %w", host.Address, ctx.Err())}
 	}
 	if err := d.errs[host.Name]; err != nil {
 		return nil, err
@@ -140,13 +142,18 @@ func TestCheck_hostsCheckedConcurrently(t *testing.T) {
 	}
 }
 
+// TestCheck_expiredContextReturnsPromptly drives the production shape of a
+// round that ran out of time during the dial: the dialer fails at the tcp
+// stage wrapping context.DeadlineExceeded, which checkReason alone would
+// call Unreachable; the verdict must still be Timeout because the round's
+// ctx ended.
 func TestCheck_expiredContextReturnsPromptly(t *testing.T) {
 	t.Parallel()
 	cfg := twoHosts()
 	p := newProvider(testProvider, cfg, &checkDialer{block: true})
 	ctx, _ := checkContext()
-	ctx, cancel := context.WithCancel(ctx)
+	ctx, cancel := context.WithTimeout(ctx, 0)
-	cancel()
+	defer cancel()
 	start := time.Now()
 	got := p.Check(ctx)
 	if elapsed := time.Since(start); elapsed > time.Second {
@@ -159,8 +166,8 @@ func TestCheck_expiredContextReturnsPromptly(t *testing.T) {
 		if ts.Ready || ts.Reason != provider.ReasonTimeout {
 			t.Errorf("target %s = ready %v reason %s, want not ready, Timeout", ts.Name, ts.Ready, ts.Reason)
 		}
-		if ts.Message == "" {
+		if !strings.Contains(ts.Message, context.DeadlineExceeded.Error()) || !strings.Contains(ts.Message, string(stageTCP)) {
-			t.Errorf("target %s has no message", ts.Name)
+			t.Errorf("target %s message = %q, want the tcp stage and the deadline named", ts.Name, ts.Message)
 		}
 	}
 }
@@ -217,7 +224,7 @@ func TestCheck_reasonsTable(t *testing.T) {
 			wantReason: provider.ReasonUnreachable,
 		},
 		{
-			name: "tcp timed out",
+			name: "tcp timed out on the dial budget but ctx is fine",
 			dialErr: func(*testing.T) error {
 				return &dialError{stage: stageTCP, host: testHost, err: fmt.Errorf("dial tcp 192.0.2.1:22: %w", context.DeadlineExceeded)}
 			},
@@ -248,7 +255,7 @@ func TestCheck_reasonsTable(t *testing.T) {
 			wantReason: provider.ReasonUnreachable,
 		},
 		{
-			name:       "ssh handshake timed out",
+			name:       "ssh handshake timed out on the dial budget but ctx is fine",
 			dialErr:    func(*testing.T) error { return sshHandshake(context.DeadlineExceeded) },
 			wantReason: provider.ReasonUnreachable,
 		},

internal/provider/kvm2/create.go

@@ -15,9 +15,11 @@ import (
 
 // The wait after DomainCreate (spec entry 18): Create returns only once the
 // domain is running, so Get never sees a domain kvm2 defined but did not
-// start (spec entry 9).
+// start (spec entry 9). DomainCreate is synchronous, so the first read
+// normally succeeds; the cap only covers state settling and must not pin a
+// reconcile worker for long.
 const (
-	defaultRunningWait = 2 * time.Minute
+	defaultRunningWait = 15 * time.Second
 	defaultRunningPoll = 500 * time.Millisecond
 )
 
@@ -75,6 +77,9 @@ func (p *Provider) create(ctx context.Context, log logr.Logger, be backend, host
 		if err != nil {
 			return err
 		}
+		if !meta.managed() {
+			return permanentf("domain %s exists but carries no kvm2 mark; it was not created by this provider", req.Name)
+		}
 		if meta.UID != "" && meta.UID != req.UID {
 			return permanentf("domain %s belongs to proxy uid %s, not %s", req.Name, meta.UID, req.UID)
 		}
@@ -211,14 +216,19 @@ func (p *Provider) writeSeed(ctx context.Context, log logr.Logger, be backend, p
 
 // ensureRunning starts the domain unless it is active already, then waits
 // until libvirt reports it running. ErrOperationInvalid from DomainCreate
-// means another caller started it between the state read and the call.
+// means another caller started it between the state read and the call. A
+// paused or pmsuspended domain is an admin's doing and DomainCreate would
+// refuse it; the verdict is Permanent so the reconciler does not retry it.
 func (p *Provider) ensureRunning(ctx context.Context, log logr.Logger, be backend, dom libvirt.Domain) error {
 	state, _, err := be.DomainGetState(ctx, dom, 0)
 	if err != nil {
 		return fmt.Errorf("reading state of domain %s: %w", dom.Name, err)
 	}
-	if isActive(libvirt.DomainState(state)) {
+	switch s := libvirt.DomainState(state); {
+	case isActive(s):
 		return nil
+	case s == libvirt.DomainPaused || s == libvirt.DomainPmsuspended:
+		return permanentf("domain %s is %s and cannot be started by the operator; resume or destroy it on the host", dom.Name, s)
 	}
 	switch err := be.DomainCreate(ctx, dom); {
 	case err == nil:
@@ -233,7 +243,9 @@ func (p *Provider) ensureRunning(ctx context.Context, log logr.Logger, be backen
 
 // waitRunning polls DomainGetState until the domain is active, the cap
 // passes (Transient: the next Create attempt finds the domain defined and
-// tries again) or ctx ends.
+// tries again) or ctx ends. Only nostate is worth another poll; a domain
+// that is shut off, crashed or paused right after its start is not going
+// to come up on its own, so that is Transient at once.
 func (p *Provider) waitRunning(ctx context.Context, be backend, dom libvirt.Domain) error {
 	deadline := time.NewTimer(p.runningWait)
 	defer deadline.Stop()
@@ -244,8 +256,11 @@ func (p *Provider) waitRunning(ctx context.Context, be backend, dom libvirt.Doma
 		if err != nil {
 			return fmt.Errorf("reading state of domain %s: %w", dom.Name, err)
 		}
-		if isActive(libvirt.DomainState(state)) {
+		switch s := libvirt.DomainState(state); {
+		case isActive(s):
 			return nil
+		case s != libvirt.DomainNostate:
+			return transientf("domain %s went %s right after start", dom.Name, s)
 		}
 		select {
 		case <-ticker.C:

internal/provider/kvm2/create_test.go

@@ -182,6 +182,64 @@ func TestCreate_returnsOnlyWhenRunning(t *testing.T) {
 	})
 }
 
+func TestCreate_domainThatDiesAfterStartIsTransientAtOnce(t *testing.T) {
+	t.Parallel()
+	for _, state := range []libvirt.DomainState{libvirt.DomainCrashed, libvirt.DomainShutoff, libvirt.DomainPaused} {
+		t.Run(state.String(), func(t *testing.T) {
+			t.Parallel()
+			// The cap is well over the bound asserted below, so a poll that
+			// ran to the cap would fail the test.
+			fx := newFixture(t, 1, withRunningWait(time.Millisecond, 10*time.Second))
+			fx.fake.stateAfterStart = state
+			start := time.Now()
+			id, err := fx.p.Create(fx.ctx, fx.req(testName))
+			elapsed := time.Since(start)
+			if id != "" {
+				t.Errorf("Create = %q, want no ID", id)
+			}
+			assertProviderError(t, err, "create", provider.ErrTransient)
+			if !strings.Contains(err.Error(), "right after start") {
+				t.Errorf("error = %v, want the state after start named", err)
+			}
+			if elapsed > time.Second {
+				t.Errorf("Create took %v, want it to give up at once, not poll to the cap", elapsed)
+			}
+			if got := fx.fake.callCount("DomainCreate"); got != 1 {
+				t.Errorf("DomainCreate calls = %d, want 1", got)
+			}
+		})
+	}
+}
+
+func TestCreate_pausedDomainIsPermanent(t *testing.T) {
+	t.Parallel()
+	for _, state := range []libvirt.DomainState{libvirt.DomainPaused, libvirt.DomainPmsuspended} {
+		t.Run(state.String(), func(t *testing.T) {
+			t.Parallel()
+			fx := newFixture(t, 1)
+			fx.fake.addDomain(testName, state, managedMetadata(testUID, time.Now()))
+			id, err := fx.p.Create(fx.ctx, fx.req(testName))
+			if id != "" {
+				t.Errorf("Create = %q, want no ID", id)
+			}
+			assertProviderError(t, err, "create", provider.ErrPermanent)
+			if !strings.Contains(err.Error(), "cannot be started by the operator") {
+				t.Errorf("error = %v, want the admin action named", err)
+			}
+			for _, rpc := range []string{"DomainCreate", "DomainDefineXML", "DomainDestroy"} {
+				if got := fx.fake.callCount(rpc); got != 0 {
+					t.Errorf("%s calls = %d, want 0", rpc, got)
+				}
+			}
+			fx.withDomain(testName, func(d *fakeDomain) {
+				if d.state != state {
+					t.Errorf("domain state = %s, want it left %s", d.state, state)
+				}
+			})
+		})
+	}
+}
+
 func TestCreate_rejectsUnknownPlacement(t *testing.T) {
 	t.Parallel()
 	tests := []struct {
@@ -248,24 +306,42 @@ func TestCreate_adoptionWithForeignUIDIsPermanent(t *testing.T) {
 	}
 }
 
-func TestCreate_adoptsMatchingOrEmptyUID(t *testing.T) {
+func TestCreate_adoptsOnlyMarkedDomains(t *testing.T) {
 	t.Parallel()
 	tests := []struct {
-		name     string
+		name      string
-		metadata string
+		metadata  string
+		wantAdopt bool
 	}{
-		{name: "matching uid", metadata: managedMetadata(testUID, time.Now())},
+		{name: "matching uid", metadata: managedMetadata(testUID, time.Now()), wantAdopt: true},
-		{name: "empty uid", metadata: managedMetadata("", time.Now())},
+		{name: "managed, empty uid", metadata: managedMetadata("", time.Now()), wantAdopt: true},
 		{name: "no metadata element", metadata: ""},
+		{name: "element without the managed mark", metadata: `<instance><uid>` + testUID + `</uid></instance>`},
+		{name: "managed false", metadata: `<instance><managed>false</managed><uid>` + testUID + `</uid></instance>`},
 	}
 	for _, tc := range tests {
 		t.Run(tc.name, func(t *testing.T) {
 			t.Parallel()
 			fx := newFixture(t, 1)
 			fx.fake.addDomain(testName, libvirt.DomainRunning, tc.metadata)
-			fx.create(testName)
+			id, err := fx.p.Create(fx.ctx, fx.req(testName))
-			if got := fx.fake.callCount("DomainDefineXML"); got != 0 {
+			if tc.wantAdopt {
-				t.Errorf("DomainDefineXML calls = %d, want 0 (adopted)", got)
+				if err != nil || id != formatProviderID(testHost, testName) {
+					t.Fatalf("Create = %q, %v; want the existing domain adopted", id, err)
+				}
+			} else {
+				if id != "" {
+					t.Errorf("Create = %q, want no ID", id)
+				}
+				assertProviderError(t, err, "create", provider.ErrPermanent)
+				if !strings.Contains(err.Error(), "carries no kvm2 mark") {
+					t.Errorf("error = %v, want the missing mark named", err)
+				}
+			}
+			for _, rpc := range []string{"DomainDefineXML", "DomainCreate", "StorageVolCreateXML", "DomainDestroy"} {
+				if got := fx.fake.callCount(rpc); got != 0 {
+					t.Errorf("%s calls = %d, want 0", rpc, got)
+				}
 			}
 		})
 	}

internal/provider/kvm2/delete.go

@@ -26,7 +26,7 @@ func (p *Provider) Delete(ctx context.Context, providerID string) error {
 	log := p.logger(ctx).WithValues("host", hostName, "name", name)
 	host, err := p.host(hostName)
 	if err != nil {
-		log.Info("host is not configured; delete has nothing to clean up", "id", providerID)
+		log.V(1).Info("host is not configured; delete has nothing to clean up", "id", providerID)
 		return nil
 	}
 	unlock, err := p.lockName(ctx, name)
@@ -82,7 +82,7 @@ func removeDomain(ctx context.Context, log logr.Logger, be backend, name string)
 func removeVolumes(ctx context.Context, log logr.Logger, be backend, poolName, name string) error {
 	pool, err := be.StoragePoolLookupByName(ctx, poolName)
 	if hasCode(err, libvirt.ErrNoStoragePool) {
-		log.Info("pool not found, no volumes to delete", "pool", poolName)
+		log.V(1).Info("pool not found, no volumes to delete", "pool", poolName)
 		return nil
 	}
 	if err != nil {

internal/provider/kvm2/delete_test.go

@@ -128,7 +128,7 @@ func TestDelete_toleratesMissingVolumesAndPool(t *testing.T) {
 	})
 	t.Run("pool gone", func(t *testing.T) {
 		t.Parallel()
-		fx := newFixture(t, 0)
+		fx := newFixture(t, 1)
 		fx.p.cfg.Hosts[0].Pool = "retired"
 		fx.fake.addDomain(testName, libvirt.DomainShutoff, managedMetadata(testUID, time.Now()))
 		if err := fx.p.Delete(fx.ctx, formatProviderID(testHost, testName)); err != nil {
@@ -138,7 +138,7 @@ func TestDelete_toleratesMissingVolumesAndPool(t *testing.T) {
 			t.Error("domain still defined after Delete")
 		}
 		if got := fx.logs.matching("pool not found, no volumes to delete"); len(got) != 1 || !strings.Contains(got[0], `"pool"="retired"`) {
-			t.Errorf("'pool not found' Info line = %v", got)
+			t.Errorf("'pool not found' V(1) line = %v", got)
 		}
 	})
 	t.Run("nothing at all", func(t *testing.T) {
@@ -152,7 +152,7 @@ func TestDelete_toleratesMissingVolumesAndPool(t *testing.T) {
 
 func TestDelete_unconfiguredHostIsNoOp(t *testing.T) {
 	t.Parallel()
-	fx := newFixture(t, 0)
+	fx := newFixture(t, 1)
 	id := formatProviderID("nas9", testName)
 	if err := fx.p.Delete(fx.ctx, id); err != nil {
 		t.Fatalf("Delete() error = %v, want nil", err)
@@ -164,7 +164,21 @@ func TestDelete_unconfiguredHostIsNoOp(t *testing.T) {
 		t.Errorf("backend calls = %v, want none", got)
 	}
 	if got := fx.logs.matching("host is not configured; delete has nothing to clean up"); len(got) != 1 || !strings.Contains(got[0], `"host"="nas9"`) {
-		t.Errorf("Info line = %v, want one naming the host", got)
+		t.Errorf("V(1) line = %v, want one naming the host", got)
+	}
+}
+
+func TestDelete_noOpLinesAreSilentAtV0(t *testing.T) {
+	t.Parallel()
+	fx := newFixture(t, 0)
+	fx.p.cfg.Hosts[0].Pool = "retired"
+	for _, id := range []string{formatProviderID("nas9", testName), formatProviderID(testHost, testName)} {
+		if err := fx.p.Delete(fx.ctx, id); err != nil {
+			t.Fatalf("Delete(%s) error = %v, want nil", id, err)
+		}
+	}
+	if lines := fx.logs.all(); len(lines) != 0 {
+		t.Errorf("Delete logged %d lines at V(0), want none:\n%s", len(lines), strings.Join(lines, "\n"))
 	}
 }
 

internal/provider/kvm2/fake_test.go

@@ -36,10 +36,13 @@ type fakeBackend struct {
 	// sha256 in fixed chunks instead of holding the bytes, for tests that
 	// push more through the fake than they want in memory.
 	discardUploads bool
-	// startDelayPolls, when set, makes DomainCreate leave the domain shut
+	// startDelayPolls, when set, makes DomainCreate leave the domain in
-	// off; it turns running on the startDelayPolls-th DomainGetState after
+	// nostate; it turns running on the startDelayPolls-th DomainGetState
-	// that, as a real host that is still booting QEMU would.
+	// after that, as a host whose QEMU is still settling would.
 	startDelayPolls int
+	// stateAfterStart, when not nostate, is what DomainCreate leaves the
+	// domain in instead of running, as a guest that dies at once would.
+	stateAfterStart libvirt.DomainState
 	// undefineFlags records the flags of the last DomainUndefineFlags per
 	// domain name.
 	undefineFlags map[string]libvirt.DomainUndefineFlagsValues
@@ -291,9 +294,14 @@ func (f *fakeBackend) DomainCreate(ctx context.Context, dom libvirt.Domain) erro
 	if isActive(d.state) {
 		return fakeErr(libvirt.ErrOperationInvalid, "Requested operation is not valid: domain is already running")
 	}
-	if f.startDelayPolls > 0 {
+	switch {
+	case f.startDelayPolls > 0:
+		d.state = libvirt.DomainNostate
 		d.pendingStart = f.startDelayPolls
 		return nil
+	case f.stateAfterStart != libvirt.DomainNostate:
+		d.state = f.stateAfterStart
+		return nil
 	}
 	d.state = libvirt.DomainRunning
 	d.id = int32(len(f.domains))

internal/provider/kvm2/get.go

@@ -42,6 +42,8 @@ func (p *Provider) Get(ctx context.Context, providerID string) (*provider.Instan
 	return inst, nil
 }
 
+// get builds the Instance from the domain's metadata, its state and, for an
+// active domain, its discovered address.
 func get(ctx context.Context, log logr.Logger, be backend, hostName, name string) (*provider.Instance, error) {
 	dom, err := be.DomainLookupByName(ctx, name)
 	if err != nil {
@@ -51,11 +53,24 @@ func get(ctx context.Context, log logr.Logger, be backend, hostName, name string
 	if err != nil {
 		return nil, err
 	}
-	inst, err := inspect(ctx, log, be, hostName, dom, meta)
+	state, _, err := be.DomainGetState(ctx, dom, 0)
 	if err != nil {
-		return nil, err
+		return nil, fmt.Errorf("reading state of domain %s: %w", dom.Name, err)
 	}
-	return &inst, nil
+	s := libvirt.DomainState(state)
+	var ip string
+	if isActive(s) {
+		if ip, err = address(ctx, log, be, dom); err != nil {
+			return nil, err
+		}
+	}
+	return &provider.Instance{
+		ID:        formatProviderID(hostName, dom.Name),
+		IP:        ip,
+		State:     mapState(s, ip),
+		UID:       meta.UID,
+		CreatedAt: meta.createdAt(),
+	}, nil
 }
 
 // readMetadata returns the kvm2 mark of a domain, or the zero value (not
@@ -72,29 +87,6 @@ func readMetadata(ctx context.Context, be backend, dom libvirt.Domain) (instance
 	return parseInstanceMetadata(raw)
 }
 
-// inspect builds the Instance from the backend's state and, for an active
-// domain, its discovered address.
-func inspect(ctx context.Context, log logr.Logger, be backend, hostName string, dom libvirt.Domain, meta instanceMetadata) (provider.Instance, error) {
-	state, _, err := be.DomainGetState(ctx, dom, 0)
-	if err != nil {
-		return provider.Instance{}, fmt.Errorf("reading state of domain %s: %w", dom.Name, err)
-	}
-	s := libvirt.DomainState(state)
-	var ip string
-	if isActive(s) {
-		if ip, err = address(ctx, log, be, dom); err != nil {
-			return provider.Instance{}, err
-		}
-	}
-	return provider.Instance{
-		ID:        formatProviderID(hostName, dom.Name),
-		IP:        ip,
-		State:     mapState(s, ip),
-		UID:       meta.UID,
-		CreatedAt: meta.createdAt(),
-	}, nil
-}
-
 // addressSources is the discovery order of spec entry 7: the guest agent
 // answers where the host's ARP table never sees guest traffic.
 var addressSources = []struct {

internal/provider/kvm2/hosts.go

@@ -2,7 +2,6 @@ package kvm2
 
 import (
 	"context"
-	"errors"
 	"fmt"
 	"sync"
 	"time"
@@ -93,26 +92,6 @@ func (t *hostTable) dialFresh(ctx context.Context, host provider.KVMHost) (backe
 	return t.dialer.dial(ctx, host)
 }
 
-// closeAll closes every pooled connection, waiting for in-flight dials.
-func (t *hostTable) closeAll() error {
-	t.mu.Lock()
-	entries := make([]*hostEntry, 0, len(t.hosts))
-	for _, e := range t.hosts {
-		entries = append(entries, e)
-	}
-	t.mu.Unlock()
-	var errs []error
-	for _, e := range entries {
-		e.sem <- struct{}{}
-		if e.backend != nil {
-			errs = append(errs, e.backend.Close())
-			e.backend = nil
-		}
-		<-e.sem
-	}
-	return errors.Join(errs...)
-}
-
 // libvirtDialer is the production dialer: SSH transport, go-libvirt over
 // the forwarded socket, the connect handshake, then the observe decorator,
 // so provider code never holds an undecorated backend.

internal/provider/kvm2/hosts_test.go

@@ -176,23 +176,6 @@ func TestHosts_dialErrorIsNotCached(t *testing.T) {
 	}
 }
 
-func TestHosts_closeAll(t *testing.T) {
-	t.Parallel()
-	table := newHostTable(&stubDialer{fresh: func() backend { return newFakeBackend() }})
-	a, _ := table.get(context.Background(), hostA)
-	b, _ := table.get(context.Background(), hostB)
-	if err := table.closeAll(); err != nil {
-		t.Fatalf("closeAll error = %v", err)
-	}
-	if a.(*fakeBackend).closed != 1 || b.(*fakeBackend).closed != 1 {
-		t.Errorf("closed counts = %d, %d, want 1, 1", a.(*fakeBackend).closed, b.(*fakeBackend).closed)
-	}
-	again, _ := table.get(context.Background(), hostA)
-	if again == a {
-		t.Error("get after closeAll returned the closed backend")
-	}
-}
-
 // perHostDialer routes each host to its own stub.
 type perHostDialer map[string]*stubDialer
 

internal/provider/kvm2/image.go

@@ -66,24 +66,15 @@ func resolveImage(cfg provider.KVMConfig, image string) (imageSource, error) {
 }
 
 // lockImage takes the per-(host, volume) lock of spec entry 19, or gives up
-// with ctx. The lock is a one-slot channel rather than a mutex so a caller
+// with ctx: a caller whose ctx ends while another caller's download runs
-// whose ctx ends while another caller's download runs returns promptly
+// returns promptly (contract §1.5) instead of waiting out a 20-minute
-// (contract §1.5) instead of waiting out a 20-minute transfer.
+// transfer.
 func (p *Provider) lockImage(ctx context.Context, hostName, volume string) (func(), error) {
-	key := hostName + "/" + volume
+	release, err := p.imageLocks.acquire(ctx, hostName+"/"+volume)
-	p.imageMu.Lock()
+	if err != nil {
-	sem, ok := p.imageLocks[key]
+		return nil, fmt.Errorf("%w: waiting for image %s on host %s: %w", provider.ErrTransient, volume, hostName, err)
-	if !ok {
-		sem = make(chan struct{}, 1)
-		p.imageLocks[key] = sem
-	}
-	p.imageMu.Unlock()
-	select {
-	case sem <- struct{}{}:
-		return func() { <-sem }, nil
-	case <-ctx.Done():
-		return nil, fmt.Errorf("%w: waiting for image %s on host %s: %w", provider.ErrTransient, volume, hostName, ctx.Err())
 	}
+	return release, nil
 }
 
 // ensureImage makes the base image present and complete in the host's pool
@@ -208,7 +199,7 @@ func deleteVolumeIfPresent(ctx context.Context, be backend, pool libvirt.Storage
 // finding no marker, and removing it now frees the space at once.
 func (p *Provider) fetchImage(ctx context.Context, host provider.KVMHost, pool libvirt.StoragePool, be backend, src imageSource) (libvirt.StorageVol, error) {
 	name := src.volumeName()
-	ctx, cancel := context.WithTimeout(ctx, p.budgets.transfer)
+	ctx, cancel := context.WithTimeout(ctx, defaultBudgets.transfer)
 	defer cancel()
 	ctx, span := tracing.StartSpan(ctx, tracing.Tracer(p.tracing...), "kvm2.download",
 		trace.WithSpanKind(trace.SpanKindClient),
@@ -337,7 +328,7 @@ func (c *countingReader) state() (int64, error) {
 // budget bounds it on its own. Failure is logged, not returned: the next
 // attempt finds the volume without a marker and deletes it then.
 func (p *Provider) discardVolume(ctx context.Context, log logr.Logger, be backend, pool libvirt.StoragePool, name string) {
-	ctx, cancel := context.WithTimeout(context.WithoutCancel(ctx), p.budgets.rpc)
+	ctx, cancel := context.WithTimeout(context.WithoutCancel(ctx), defaultBudgets.rpc)
 	defer cancel()
 	if err := deleteVolumeIfPresent(ctx, be, pool, name); err != nil {
 		log.V(1).Info("incomplete base volume left behind; the next attempt removes it", "error", err.Error())
@@ -421,8 +412,10 @@ func urlHost(rawURL string) string {
 // imageSizeIfUncached returns the image's Content-Length when the base is
 // not cached on the host, and 0 when the marker is present, for the quota
 // precheck of spec entry 13. It never downloads: HEAD first, and a GET whose
-// body is closed at once when the server does not allow HEAD. A marker is
+// body is closed at once when HEAD answers anything but 2xx, since servers
-// taken at face value here; ensureImage decides whether it still matches.
+// refuse HEAD with 405, 403 or 501 alike; the GET's status is what gets
+// classified. A marker is taken at face value here; ensureImage decides
+// whether it still matches.
 func (p *Provider) imageSizeIfUncached(ctx context.Context, be backend, pool libvirt.StoragePool, src imageSource) (int64, error) {
 	_, err := be.StorageVolLookupByName(ctx, pool, src.markerName())
 	switch {
@@ -431,14 +424,14 @@ func (p *Provider) imageSizeIfUncached(ctx context.Context, be backend, pool lib
 	case !hasCode(err, libvirt.ErrNoStorageVol):
 		return 0, fmt.Errorf("looking up marker %s: %w", src.markerName(), err)
 	}
-	ctx, cancel := context.WithTimeout(ctx, p.budgets.rpc)
+	ctx, cancel := context.WithTimeout(ctx, defaultBudgets.rpc)
 	defer cancel()
 	resp, err := p.do(ctx, http.MethodHead, src.URL)
 	if err != nil {
 		return 0, err
 	}
 	_ = resp.Body.Close()
-	if resp.StatusCode == http.StatusMethodNotAllowed {
+	if resp.StatusCode < 200 || resp.StatusCode >= 300 {
 		if resp, err = p.do(ctx, http.MethodGet, src.URL); err != nil {
 			return 0, err
 		}

internal/provider/kvm2/image_test.go

@@ -152,6 +152,17 @@ func serveStatus(code int) http.HandlerFunc {
 	return func(w http.ResponseWriter, _ *http.Request) { http.Error(w, http.StatusText(code), code) }
 }
 
+// headFails answers HEAD with code and hands every other method to next.
+func headFails(code int, next http.HandlerFunc) http.HandlerFunc {
+	return func(w http.ResponseWriter, r *http.Request) {
+		if r.Method == http.MethodHead {
+			http.Error(w, http.StatusText(code), code)
+			return
+		}
+		next(w, r)
+	}
+}
+
 func TestResolveImage_table(t *testing.T) {
 	t.Parallel()
 	cfg := withDefaults(provider.KVMConfig{Images: map[string]provider.KVMImage{
@@ -883,17 +894,30 @@ func TestImageSizeIfUncached_headThenGetFallback(t *testing.T) {
 			wantMethods: []string{http.MethodHead},
 		},
 		{
-			name: "HEAD not allowed falls back to GET with the body closed",
+			name:        "HEAD not allowed falls back to GET with the body closed",
-			handler: func(w http.ResponseWriter, r *http.Request) {
+			handler:     headFails(http.StatusMethodNotAllowed, serveBytes(body)),
-				if r.Method == http.MethodHead {
-					http.Error(w, "nope", http.StatusMethodNotAllowed)
-					return
-				}
-				serveBytes(body)(w, r)
-			},
 			wantSize:    int64(len(body)),
 			wantMethods: []string{http.MethodHead, http.MethodGet},
 		},
+		{
+			name:        "HEAD forbidden falls back to GET",
+			handler:     headFails(http.StatusForbidden, serveBytes(body)),
+			wantSize:    int64(len(body)),
+			wantMethods: []string{http.MethodHead, http.MethodGet},
+		},
+		{
+			name:        "HEAD not implemented falls back to GET",
+			handler:     headFails(http.StatusNotImplemented, serveBytes(body)),
+			wantSize:    int64(len(body)),
+			wantMethods: []string{http.MethodHead, http.MethodGet},
+		},
+		{
+			name:        "HEAD refused and GET 404 is permanent from the GET",
+			handler:     headFails(http.StatusMethodNotAllowed, serveStatus(http.StatusNotFound)),
+			wantMethods: []string{http.MethodHead, http.MethodGet},
+			wantClass:   provider.ErrPermanent,
+			wantErr:     "404",
+		},
 		{
 			name:        "marker present means nothing to download",
 			handler:     serveBytes(body),
@@ -902,16 +926,16 @@ func TestImageSizeIfUncached_headThenGetFallback(t *testing.T) {
 			wantMethods: nil,
 		},
 		{
-			name:        "404 is permanent",
+			name:        "404 on both is permanent",
 			handler:     serveStatus(http.StatusNotFound),
-			wantMethods: []string{http.MethodHead},
+			wantMethods: []string{http.MethodHead, http.MethodGet},
 			wantClass:   provider.ErrPermanent,
 			wantErr:     "404",
 		},
 		{
-			name:        "503 is transient",
+			name:        "503 on both is transient",
 			handler:     serveStatus(http.StatusServiceUnavailable),
-			wantMethods: []string{http.MethodHead},
+			wantMethods: []string{http.MethodHead, http.MethodGet},
 			wantClass:   provider.ErrTransient,
 		},
 		{

internal/provider/kvm2/kvm2.go

@@ -33,7 +33,6 @@ type Provider struct {
 	now     func() time.Time
 	hosts   *hostTable
 	http    *http.Client
-	budgets budgets
 	tracing []tracing.Option
 
 	// runningWait caps the wait for a started domain to report running,
@@ -42,28 +41,23 @@ type Provider struct {
 	runningPoll time.Duration
 
 	// imageLocks serialises image fetches per (host, volume), spec entry 19.
-	imageMu    sync.Mutex
+	imageLocks *keyedLocks
-	imageLocks map[string]chan struct{}
 
 	// nameLocks serialises Create and Delete per instance name, spec entry
 	// 19, so two Deletes never race on destroy and undefine and a Delete
 	// cannot interleave with a Create's steps.
-	nameMu    sync.Mutex
+	nameLocks *keyedLocks
-	nameLocks map[string]chan struct{}
 }
 
 // providerOption adjusts what newProvider builds; tests inject an HTTP
-// client bound to a test server, a span recorder and shorter budgets.
+// client bound to a test server, a span recorder and a shorter running
+// wait.
 type providerOption func(*Provider)
 
 func withHTTPClient(c *http.Client) providerOption {
 	return func(p *Provider) { p.http = c }
 }
 
-func withBudgets(b budgets) providerOption {
-	return func(p *Provider) { p.budgets = b }
-}
-
 func withTracing(opts ...tracing.Option) providerOption {
 	return func(p *Provider) { p.tracing = opts }
 }
@@ -104,7 +98,7 @@ func New(ctx context.Context, pc provider.ProviderConfig) (provider.Provider, er
 // newProvider is the constructor tests use: d hands out the backends, the
 // production one being newLibvirtDialer. Without options the image
 // downloads use http.DefaultClient (system CA roots, no client timeout: the
-// per-request ctx carries the transfer budget) and the default budgets.
+// per-request ctx carries the transfer budget).
 func newProvider(name string, cfg provider.KVMConfig, d dialer, opts ...providerOption) *Provider {
 	p := &Provider{
 		name:        name,
@@ -112,11 +106,10 @@ func newProvider(name string, cfg provider.KVMConfig, d dialer, opts ...provider
 		now:         time.Now,
 		hosts:       newHostTable(d),
 		http:        http.DefaultClient,
-		budgets:     defaultBudgets,
 		runningWait: defaultRunningWait,
 		runningPoll: defaultRunningPoll,
-		imageLocks:  make(map[string]chan struct{}),
+		imageLocks:  newKeyedLocks(),
-		nameLocks:   make(map[string]chan struct{}),
+		nameLocks:   newKeyedLocks(),
 	}
 	for _, opt := range opts {
 		opt(p)
@@ -187,21 +180,64 @@ func (p *Provider) wrapErr(op, id string, err error) error {
 	return provider.Wrap(classify(op, err), op, p.name, id, err)
 }
 
-// lockName takes the per-name lock of spec entry 19, or gives up with ctx.
+// lockName takes the per-name lock of spec entry 19, or gives up with ctx,
-// A one-slot channel rather than a mutex, so a caller whose ctx ends while
+// so a caller whose ctx ends while another caller's Create is downloading
-// another caller's Create is downloading an image returns promptly.
+// an image returns promptly.
 func (p *Provider) lockName(ctx context.Context, name string) (func(), error) {
-	p.nameMu.Lock()
+	release, err := p.nameLocks.acquire(ctx, name)
-	sem, ok := p.nameLocks[name]
+	if err != nil {
-	if !ok {
+		return nil, fmt.Errorf("%w: waiting for the lock on %s: %w", provider.ErrTransient, name, err)
-		sem = make(chan struct{}, 1)
-		p.nameLocks[name] = sem
 	}
-	p.nameMu.Unlock()
+	return release, nil
+}
+
+// keyedLocks is a set of one-slot semaphores by key (spec entry 19): a
+// semaphore rather than a mutex so a waiter can give up with its ctx. Each
+// entry counts the callers holding or waiting for it and is dropped from
+// the map at zero, so the map does not grow with every name ever seen while
+// two waiters on one key still always share one semaphore.
+type keyedLocks struct {
+	mu    sync.Mutex
+	locks map[string]*keyedLock
+}
+
+type keyedLock struct {
+	sem  chan struct{}
+	refs int
+}
+
+func newKeyedLocks() *keyedLocks {
+	return &keyedLocks{locks: make(map[string]*keyedLock)}
+}
+
+// acquire takes the lock for key, or gives up with ctx's error. The
+// returned func releases it and must be called exactly once.
+func (k *keyedLocks) acquire(ctx context.Context, key string) (func(), error) {
+	k.mu.Lock()
+	l, ok := k.locks[key]
+	if !ok {
+		l = &keyedLock{sem: make(chan struct{}, 1)}
+		k.locks[key] = l
+	}
+	l.refs++
+	k.mu.Unlock()
 	select {
-	case sem <- struct{}{}:
+	case l.sem <- struct{}{}:
-		return func() { <-sem }, nil
+		return func() {
+			<-l.sem
+			k.unref(key, l)
+		}, nil
 	case <-ctx.Done():
-		return nil, fmt.Errorf("%w: waiting for the lock on %s: %w", provider.ErrTransient, name, ctx.Err())
+		k.unref(key, l)
+		return nil, ctx.Err()
+	}
+}
+
+func (k *keyedLocks) unref(key string, l *keyedLock) {
+	k.mu.Lock()
+	defer k.mu.Unlock()
+	l.refs--
+	if l.refs == 0 {
+		delete(k.locks, key)
 	}
 }

internal/provider/kvm2/kvm2_test.go

@@ -5,6 +5,7 @@ import (
 	"errors"
 	"strings"
 	"sync"
+	"sync/atomic"
 	"testing"
 	"time"
 
@@ -254,6 +255,86 @@ func TestWithDefaults_doesNotAliasInput(t *testing.T) {
 	}
 }
 
+func lockCount(k *keyedLocks) int {
+	k.mu.Lock()
+	defer k.mu.Unlock()
+	return len(k.locks)
+}
+
+func TestKeyedLocks_entryRemovedWhenUncontended(t *testing.T) {
+	t.Parallel()
+	k := newKeyedLocks()
+	release, err := k.acquire(context.Background(), "a")
+	if err != nil {
+		t.Fatalf("acquire error = %v", err)
+	}
+	if lockCount(k) != 1 {
+		t.Errorf("entries while held = %d, want 1", lockCount(k))
+	}
+	release()
+	if lockCount(k) != 0 {
+		t.Errorf("entries after release = %d, want 0", lockCount(k))
+	}
+	if _, err := k.acquire(context.Background(), "a"); err != nil {
+		t.Fatalf("acquire after release error = %v", err)
+	}
+}
+
+func TestKeyedLocks_waitersShareOneLock(t *testing.T) {
+	t.Parallel()
+	k := newKeyedLocks()
+	const n = 20
+	var inside, overlaps atomic.Int32
+	var wg sync.WaitGroup
+	for range n {
+		wg.Go(func() {
+			release, err := k.acquire(context.Background(), "shared")
+			if err != nil {
+				t.Errorf("acquire error = %v", err)
+				return
+			}
+			if inside.Add(1) > 1 {
+				overlaps.Add(1)
+			}
+			time.Sleep(time.Millisecond)
+			inside.Add(-1)
+			release()
+		})
+	}
+	wg.Wait()
+	if overlaps.Load() != 0 {
+		t.Errorf("%d holders overlapped on one key", overlaps.Load())
+	}
+	if lockCount(k) != 0 {
+		t.Errorf("entries after every holder released = %d, want 0", lockCount(k))
+	}
+}
+
+func TestKeyedLocks_waiterGivesUpWithContext(t *testing.T) {
+	t.Parallel()
+	k := newKeyedLocks()
+	release, err := k.acquire(context.Background(), "a")
+	if err != nil {
+		t.Fatalf("acquire error = %v", err)
+	}
+	ctx, cancel := context.WithTimeout(context.Background(), 20*time.Millisecond)
+	defer cancel()
+	start := time.Now()
+	if _, err := k.acquire(ctx, "a"); !errors.Is(err, context.DeadlineExceeded) {
+		t.Fatalf("waiter error = %v, want DeadlineExceeded", err)
+	}
+	if time.Since(start) > time.Second {
+		t.Errorf("waiter took %v, want to give up with its ctx", time.Since(start))
+	}
+	if lockCount(k) != 1 {
+		t.Errorf("entries with the holder still in = %d, want 1: the waiter's exit must not drop the holder's entry", lockCount(k))
+	}
+	release()
+	if lockCount(k) != 0 {
+		t.Errorf("entries after the holder released = %d, want 0", lockCount(k))
+	}
+}
+
 func TestWrapErr_carriesProviderNameOpAndID(t *testing.T) {
 	t.Parallel()
 	p := newProvider("kvm2-home", twoHosts(), &stubDialer{})

internal/provider/kvm2/list.go

@@ -10,10 +10,13 @@ import (
 )
 
 // ListByTag is spec entry 12: every domain on every configured host, in
-// config order, filtered client-side on the kvm2 mark. A host that cannot
+// config order, filtered client-side on the kvm2 mark. GC uses only ID, UID
-// be reached or listed fails the whole call (fail-whole); a single domain
+// and CreatedAt (contract §2.4), all of which the name and the metadata
-// that cannot be read is skipped, unless ctx ended, which fails the call
+// element give, so no state or address RPC is made and State and IP stay
-// rather than pass off an empty view as the truth.
+// empty. A host that cannot be reached or listed fails the whole call
+// (fail-whole); a single domain whose metadata cannot be read is skipped,
+// unless ctx ended, which fails the call rather than pass off an empty view
+// as the truth.
 func (p *Provider) ListByTag(ctx context.Context) ([]provider.Instance, error) {
 	log := p.logger(ctx)
 	var out []provider.Instance
@@ -43,15 +46,8 @@ func (p *Provider) ListByTag(ctx context.Context) ([]provider.Instance, error) {
 			if !meta.managed() {
 				continue
 			}
-			inst, err := inspect(ctx, dlog, be, host.Name, dom, meta)
+			inst := provider.Instance{ID: formatProviderID(host.Name, dom.Name), UID: meta.UID, CreatedAt: meta.createdAt()}
-			if err != nil {
+			dlog.V(2).Info("kvm2 listed instance", "id", inst.ID, "uid", inst.UID, "createdAt", inst.CreatedAt)
-				if ctx.Err() != nil {
-					return nil, p.wrapErr("list", "", err)
-				}
-				dlog.V(1).Info("skipping domain that could not be inspected", "error", err.Error())
-				continue
-			}
-			dlog.V(2).Info("kvm2 listed instance", "id", inst.ID, "state", inst.State, "uid", inst.UID, "createdAt", inst.CreatedAt)
 			out = append(out, inst)
 		}
 	}

internal/provider/kvm2/list_test.go

@@ -72,8 +72,43 @@ func TestListByTag_acrossHostsInConfigOrder(t *testing.T) {
 	if got := listIDs(list); strings.Join(got, ",") != strings.Join(want, ",") {
 		t.Errorf("ListByTag = %v, want %v (config order, then name order)", got, want)
 	}
-	if list[0].State != provider.StateStopped || list[1].State != provider.StateProvisioning {
+	if list[0].UID != "uid-a" || list[1].UID != "uid-z" || list[2].UID != "uid-b" {
-		t.Errorf("states = %s, %s, want Stopped, Provisioning", list[0].State, list[1].State)
+		t.Errorf("uids = %s, %s, %s, want uid-a, uid-z, uid-b", list[0].UID, list[1].UID, list[2].UID)
+	}
+}
+
+func TestListByTag_doesNotReadStateOrAddresses(t *testing.T) {
+	t.Parallel()
+	fx := newFixture(t, 2)
+	id := fx.create(testName)
+	fx.setAddrs(testName, libvirt.DomainInterfaceAddressesSrcAgent, "10.0.0.5")
+	fx.fake.addDomain("proxy-off", libvirt.DomainShutoff, managedMetadata("uid-off", time.Now()))
+	before := len(fx.fake.callLog())
+
+	list, err := fx.p.ListByTag(fx.ctx)
+	if err != nil {
+		t.Fatalf("ListByTag() error = %v", err)
+	}
+	if len(list) != 2 || list[0].ID != id || list[1].ID != formatProviderID(testHost, "proxy-off") {
+		t.Fatalf("ListByTag = %v, want the two managed domains", listIDs(list))
+	}
+	for _, inst := range list {
+		if inst.State != "" || inst.IP != "" {
+			t.Errorf("listed %s = state %q ip %q, want both empty: GC does not use them", inst.ID, inst.State, inst.IP)
+		}
+		if inst.UID == "" || inst.CreatedAt.IsZero() {
+			t.Errorf("listed %s lacks uid or createdAt: %+v", inst.ID, inst)
+		}
+	}
+	for _, rpc := range fx.fake.callLog()[before:] {
+		switch rpc {
+		case "ConnectListAllDomains", "DomainGetMetadata":
+		default:
+			t.Errorf("ListByTag made %s; want only the listing and one metadata read per domain", rpc)
+		}
+	}
+	if got := fx.logs.matching("kvm2 listed instance"); len(got) != 2 || strings.Contains(got[0], `"state"=`) {
+		t.Errorf("listed instance lines = %v, want two without a state", got)
 	}
 }
 
@@ -126,12 +161,12 @@ func TestListByTag_unreadableDomainSkipped(t *testing.T) {
 			line: "skipping domain with unreadable metadata",
 		},
 		{
-			name: "state rpc fails for the first domain only",
+			name: "domain gone between listing and metadata read",
 			setup: func(fx *fixture) {
 				fx.fake.addDomain("proxy-bad", libvirt.DomainRunning, managedMetadata("uid-bad", time.Now()))
 				var calls int
 				fx.fake.mu.Lock()
-				fx.fake.hooks["DomainGetState"] = func() error {
+				fx.fake.hooks["DomainGetMetadata"] = func() error {
 					calls++
 					if calls == 1 {
 						return fakeErr(libvirt.ErrNoDomain, "Domain not found")
@@ -140,7 +175,7 @@ func TestListByTag_unreadableDomainSkipped(t *testing.T) {
 				}
 				fx.fake.mu.Unlock()
 			},
-			line: "skipping domain that could not be inspected",
+			line: "skipping domain with unreadable metadata",
 		},
 	}
 	for _, tc := range tests {
@@ -172,7 +207,7 @@ func TestListByTag_includesInactiveDomains(t *testing.T) {
 	if err != nil {
 		t.Fatalf("ListByTag() error = %v", err)
 	}
-	if len(list) != 1 || list[0].ID != id || list[0].State != provider.StateStopped {
+	if len(list) != 1 || list[0].ID != id || list[0].UID != testUID {
-		t.Errorf("ListByTag = %+v, want the shut-off domain as Stopped", list)
+		t.Errorf("ListByTag = %+v, want the shut-off domain listed with its uid", list)
 	}
 }