kvm2: user guide, appendix column and section, architecture and README entries

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>

docs/providers-existing.md

@@ -1,13 +1,17 @@
 # The existing providers
 
 This is the appendix to [provider-contract.md](provider-contract.md): how the
-three providers that exist today (kubernetes, gcp, kvm) meet the contract,
-which profile each one chose on every axis, and what is still missing. It is
-reference material for whoever writes the next provider, not a user guide;
-the user guides are [kvm-provider.md](kvm-provider.md),
+four providers that exist today (kubernetes, gcp, kvm, kvm2) meet the
+contract, which profile each one chose on every axis, and what is still
+missing. It is reference material for whoever writes the next provider, not
+a user guide; the user guides are [kvm-provider.md](kvm-provider.md),
+[kvm2-provider.md](kvm2-provider.md),
 [gcp-vm-validation.md](gcp-vm-validation.md) and
 [gcp-in-specific-project.md](gcp-in-specific-project.md). The README's
-"Implemented providers" table stays the short public summary.
+"Implemented providers" table stays the short public summary. kvm2 is the
+one provider built from the contract itself (clean-room, from
+[kvm2-provider-spec.md](kvm2-provider-spec.md), without reading kvm's code),
+so its column is also a record of what the contract produced.
 
 The per-provider call flows are drawn in
 [architecture.md §5](architecture.md#5-what-provider-calls-do-in-the-outside-world),
@@ -23,59 +27,59 @@ Code references are a file path plus a function or symbol name.
 One row per axis of contract §4, in the same order, with the profile name
 and the symbol that shows it.
 
-| Axis | kubernetes | gcp | kvm |
-|---|---|---|---|
-| `readiness` | immediate: a Pod is scheduled within seconds (`stateFromPod`) | provisioning-window: minutes of boot (`mapState`) | provisioning-window: boot plus, on first use, an image download (`imageCache.ensure`) |
-| `create-return` | submit: returns after `client.Create` | submit: `Insert` is never awaited, `Operation.Wait` is not called | **blocking**: waits through `DomainCreate` (`ensureRunning`), and the first image fetch runs inside (`imageCache.fetch`) |
-| `delete-return` | returns-early: `client.Delete`, Pod terminates on its own | returns-early: `Delete` submits an operation each time | waits: destroy, undefine and volume removal complete before return (`removeDomain`, `removeVolumes`) |
-| `ip-source` | assigned-at-create: `PodIP` (`instanceFromPod`) | assigned-at-create: the external NAT IP of the first NIC (`toInstance`) | discovered: ARP table first, then the guest agent, first routable IPv4 (`address`, `firstIPv4`) |
-| `gc-mark` | native-labels: Pod labels (`buildPod`) | native-labels: instance labels (`buildInsertRequest`) | written-metadata: a namespaced `<metadata>` element written in `DomainDefineXML` (`metadata.go`) |
-| `created-at` | backend-timestamp: `CreationTimestamp` | backend-timestamp: `CreationTimestamp`, parse error gives zero | written-at-create: `createdAt` in the metadata element (`newInstanceMetadata`) |
-| `list-partial` | n/a: one call, all namespaces | omit-unreachable: `ReturnPartialSuccess` on `AggregatedList` | fail-whole per host, one unreadable domain skipped (`ListByTag`, `managedInstance`) |
-| `quota` | absent: a quota 403 is Permanent (`classify`) | detected: 429, and 403 with reason `quotaExceeded` or `rateLimitExceeded` (`classify`, `hasReason`) | absent (`classify`) |
-| `not-found` | status-404: `apierrors.IsNotFound` | status-404: `isNotFound` | error-code: `ErrNoDomain`, `ErrNoStorageVol` (`hasCode`) |
-| `primary-key` | name-is-key: `namespace/name` | name-is-key: `zones/<zone>/instances/<name>` | name-is-key: `hosts/<host>/domains/<name>` |
-| `provisioning-state` | native: `PodPending` | native: `PROVISIONING`, `STAGING`, `REPAIRING` | create-blocks: a domain is defined and started in one `Create`; `nostate` maps to Provisioning |
-| `async-ops` | none | task-handle, but the handle is discarded: an insert that fails after acceptance is never read | none |
-| `secondary-resources` | none | cascade: `AutoDelete` on the boot disk (`buildInsertRequest`) | explicit: overlay and seed removed by name (`removeVolumes`); base image kept |
-| `credentials` | workload-identity: in-cluster config or kubeconfig (`NewWithTransportWrapper`) | environment: Application Default Credentials | mounted-files: SSH key and known_hosts from the optional `kvm-ssh` Secret, read lazily (`newLazySSH`) |
-| `client-kind` | vendor-sdk: controller-runtime `client.Client`, no seam interface | vendor-sdk: `cloud.google.com/go/compute` behind `instancesAPI` | rpc-over-ssh: `go-libvirt` over an SSH-forwarded unix socket, behind `libvirtAPI` (`conn.go`) |
-| `timeouts` | ctx-only: no `rest.Config.Timeout` is set | ctx-only: SDK defaults, with SDK retries on Get and List | per-call for the dial only (`defaultHandshakeTimeout` in `dialSSH`); RPCs are ctx-only and a cancelled `call` closes the connection |
-| `logging` | none | tiered: V(1) outcomes and `logAPIError`, V(2) detail, V(5) wire log (`wirelog.go`) | tiered: V(1) outcomes and one `libvirt call` line per RPC (`trace.go`), V(2) XML and addresses; no wire tier |
-| `child-spans` | transport-wrapper: `tracing.RestConfigWrapper()` on its own `rest.Config`, parent-gated | sdk-instrumentation: the SDK adds `otelhttp` itself; not parent-gated, but every call site has a parent | **none**: `trace.go` is a log decorator, not spans |
-| `backend-test` | cluster-e2e: kind (`test/e2e`) | manual-runbook: [gcp-vm-validation.md](gcp-vm-validation.md) | integration-tag: `kvm_integration_test.go`, `make test-integration-kvm` |
+| Axis | kubernetes | gcp | kvm | kvm2 |
+|---|---|---|---|---|
+| `readiness` | immediate: a Pod is scheduled within seconds (`stateFromPod`) | provisioning-window: minutes of boot (`mapState`) | provisioning-window: boot plus, on first use, an image download (`imageCache.ensure`) | provisioning-window: boot plus, on first use, an image download (`ensureImage`) |
+| `create-return` | submit: returns after `client.Create` | submit: `Insert` is never awaited, `Operation.Wait` is not called | **blocking**: waits through `DomainCreate` (`ensureRunning`), and the first image fetch runs inside (`imageCache.fetch`) | **blocking**: waits through `DomainCreate` and a 2-minute poll for running (`ensureRunning`, `waitRunning`), and the first image fetch runs inside (`fetchImage`) |
+| `delete-return` | returns-early: `client.Delete`, Pod terminates on its own | returns-early: `Delete` submits an operation each time | waits: destroy, undefine and volume removal complete before return (`removeDomain`, `removeVolumes`) | waits: destroy, undefine and volume removal complete before return (`removeDomain`, `removeVolumes`) |
+| `ip-source` | assigned-at-create: `PodIP` (`instanceFromPod`) | assigned-at-create: the external NAT IP of the first NIC (`toInstance`) | discovered: ARP table first, then the guest agent, first routable IPv4 (`address`, `firstIPv4`) | discovered: the guest agent first, then the ARP table, first routable IPv4 (`addressSources`, `address`, `firstIPv4`) |
+| `gc-mark` | native-labels: Pod labels (`buildPod`) | native-labels: instance labels (`buildInsertRequest`) | written-metadata: a namespaced `<metadata>` element written in `DomainDefineXML` (`metadata.go`) | written-metadata: an element in kvm2's own namespace written in `DomainDefineXML` (`metadataURI`, `newInstanceMetadata`) |
+| `created-at` | backend-timestamp: `CreationTimestamp` | backend-timestamp: `CreationTimestamp`, parse error gives zero | written-at-create: `createdAt` in the metadata element (`newInstanceMetadata`) | written-at-create: `createdAt` in the metadata element, parse error gives zero (`newInstanceMetadata`, `createdAt`) |
+| `list-partial` | n/a: one call, all namespaces | omit-unreachable: `ReturnPartialSuccess` on `AggregatedList` | fail-whole per host, one unreadable domain skipped (`ListByTag`, `managedInstance`) | fail-whole per host, one unreadable domain skipped unless ctx ended (`ListByTag`) |
+| `quota` | absent: a quota 403 is Permanent (`classify`) | detected: 429, and 403 with reason `quotaExceeded` or `rateLimitExceeded` (`classify`, `hasReason`) | absent (`classify`) | detected: a pool-space precheck before any volume is created (`checkPoolSpace`, `neededBytes`) |
+| `not-found` | status-404: `apierrors.IsNotFound` | status-404: `isNotFound` | error-code: `ErrNoDomain`, `ErrNoStorageVol` (`hasCode`) | error-code: `ErrNoDomain`, `ErrNoStorageVol`, Transient inside Create (`classifyLibvirt`) |
+| `primary-key` | name-is-key: `namespace/name` | name-is-key: `zones/<zone>/instances/<name>` | name-is-key: `hosts/<host>/domains/<name>` | name-is-key: `hosts/<host>/domains/<name>`, with a UID guard on adoption (`create`) |
+| `provisioning-state` | native: `PodPending` | native: `PROVISIONING`, `STAGING`, `REPAIRING` | create-blocks: a domain is defined and started in one `Create`; `nostate` maps to Provisioning | create-blocks: defined, started and polled to running in one `Create`; `nostate` maps to Provisioning (`mapState`) |
+| `async-ops` | none | task-handle, but the handle is discarded: an insert that fails after acceptance is never read | none | none |
+| `secondary-resources` | none | cascade: `AutoDelete` on the boot disk (`buildInsertRequest`) | explicit: overlay and seed removed by name (`removeVolumes`); base image kept | explicit: overlay and seed removed by name (`removeVolumes`); base image kept |
+| `credentials` | workload-identity: in-cluster config or kubeconfig (`NewWithTransportWrapper`) | environment: Application Default Credentials | mounted-files: SSH key and known_hosts from the optional `kvm-ssh` Secret, read lazily (`newLazySSH`) | mounted-files: the same `kvm-ssh` Secret, read on every dial and never cached (`loadCredentials`) |
+| `client-kind` | vendor-sdk: controller-runtime `client.Client`, no seam interface | vendor-sdk: `cloud.google.com/go/compute` behind `instancesAPI` | rpc-over-ssh: `go-libvirt` over an SSH-forwarded unix socket, behind `libvirtAPI` (`conn.go`) | rpc-over-ssh: `go-libvirt` over an SSH-forwarded unix socket, behind the ctx-first `backend` seam (`backend.go`) |
+| `timeouts` | ctx-only: no `rest.Config.Timeout` is set | ctx-only: SDK defaults, with SDK retries on Get and List | per-call for the dial only (`defaultHandshakeTimeout` in `dialSSH`); RPCs are ctx-only and a cancelled `call` closes the connection | per-call: a budget around the dial, every RPC, the uploads and the download (`defaultBudgets`, `call`); a cancelled or overlong call kills the pooled connection |
+| `logging` | none | tiered: V(1) outcomes and `logAPIError`, V(2) detail, V(5) wire log (`wirelog.go`) | tiered: V(1) outcomes and one `libvirt call` line per RPC (`trace.go`), V(2) XML and addresses; no wire tier | tiered: V(1) outcomes and one `libvirt call` line per RPC (`observe.go`), V(2) XML and addresses; no wire tier |
+| `child-spans` | transport-wrapper: `tracing.RestConfigWrapper()` on its own `rest.Config`, parent-gated | sdk-instrumentation: the SDK adds `otelhttp` itself; not parent-gated, but every call site has a parent | **none**: `trace.go` is a log decorator, not spans | span-decorator: `kvm2.<Method>` per RPC (`observe`), plus `kvm2.dial` (`libvirtDialer`) and `kvm2.download` (`fetchImage`) |
+| `backend-test` | cluster-e2e: kind (`test/e2e`) | manual-runbook: [gcp-vm-validation.md](gcp-vm-validation.md) | integration-tag: `kvm_integration_test.go`, `make test-integration-kvm` | integration-tag: `kvm2_integration_test.go`, `make test-integration-kvm2` |
 
 ## 2. Other differences
 
 Rows that are not profile axes but that a new provider author compares
 against.
 
-| Feature | kubernetes | gcp | kvm |
-|---|---|---|---|
-| Create verifies the adopted instance's UID | no | no | no |
-| Placement used | none | Zone, MachineType, Image | Zone (host), MachineType, Image |
-| CloudInit delivered | **no**: ignored | yes: `user-data` metadata | yes: NoCloud seed ISO built in-process (`buildSeedISO`) |
-| Port used | yes: Squid config and containerPort (`squidConf`) | no: firewall is manual | no |
-| States produced | Provisioning, Running, Terminated | all four | Provisioning, Running, Stopped |
-| Deleting instance still reports its state | yes: a Pod being deleted reports its phase | n/a | n/a |
-| Error.Provider is the configured name | **no**: hardcoded `"kubernetes"` | yes | yes |
-| Checker target | `apiserver` | the project | each host, concurrently |
-| Checker Details | none | none | `user`, `pool`, `hostname`, `libvirtVersion`, `poolState`, `poolCapacity`, `poolAvailable` |
-| Credential rotation without restart | n/a | via ADC | no: cached after first successful load |
-| Connection handling | client-go | SDK client, never closed | one pooled SSH plus libvirt connection per host, 15s keepalive (`hostPool`) |
-| Unit-test double | controller-runtime fake client | `fakeAPI` behind `instancesAPI`, plus `httptest` for the real SDK client in `check_test.go` | `fakeAPI` behind `libvirtAPI`, plus an in-process `x/crypto/ssh` server in `conn_test.go` |
-| Proxy software | generated Squid config (`pod.go`) | from `cloudInit`; the image must run cloud-init | from `cloudInit`, plus a vendor-data script that installs the guest agent (`seed.go`) |
-| IP reachable from the operator | Pod network, so in-cluster only (not from `make run-dev` on a laptop) | external IP, once a firewall rule allows the cluster's egress IP | LAN address on the host's bridge; the operator needs a route to that LAN |
-| Distinct egress IP per proxy | no: the cluster's egress | yes: ephemeral external IP | whatever the host's LAN routes through; not per VM by itself |
+| Feature | kubernetes | gcp | kvm | kvm2 |
+|---|---|---|---|---|
+| Create verifies the adopted instance's UID | no | no | no | **yes**: a different non-empty UID in the metadata is Permanent, an empty one is adopted (`create`) |
+| Placement used | none | Zone, MachineType, Image | Zone (host), MachineType, Image | Zone (host), MachineType, Image |
+| CloudInit delivered | **no**: ignored | yes: `user-data` metadata | yes: NoCloud seed ISO built in-process (`buildSeedISO`) | yes: NoCloud seed ISO built in-process (`buildSeedISO`) |
+| Port used | yes: Squid config and containerPort (`squidConf`) | no: firewall is manual | no | no |
+| States produced | Provisioning, Running, Terminated | all four | Provisioning, Running, Stopped | Provisioning, Running, Stopped |
+| Deleting instance still reports its state | yes: a Pod being deleted reports its phase | n/a | n/a | n/a: Delete waits, and a concurrent Get sees the domain until it is undefined |
+| Error.Provider is the configured name | **no**: hardcoded `"kubernetes"` | yes | yes | yes (`wrapErr`) |
+| Checker target | `apiserver` | the project | each host, concurrently | each host, concurrently, on a fresh connection |
+| Checker Details | none | none | `user`, `pool`, `hostname`, `libvirtVersion`, `poolState`, `poolCapacity`, `poolAvailable` | `user`, `pool`, `hostname`, `libvirtVersion`, `poolState`, `poolCapacityGiB`, `poolAvailableGiB`, filled in as far as the check got |
+| Credential rotation without restart | n/a | via ADC | no: cached after first successful load | yes: both files read on every dial (`loadCredentials`) |
+| Connection handling | client-go | SDK client, never closed | one pooled SSH plus libvirt connection per host, 15s keepalive (`hostPool`) | one pooled SSH plus libvirt connection per host, 15s keepalive (`hostTable`); dedicated connections for the image upload and the check (`dialFresh`) |
+| Unit-test double | controller-runtime fake client | `fakeAPI` behind `instancesAPI`, plus `httptest` for the real SDK client in `check_test.go` | `fakeAPI` behind `libvirtAPI`, plus an in-process `x/crypto/ssh` server in `conn_test.go` | `fakeBackend` behind `backend` (`fake_test.go`), plus an in-process `x/crypto/ssh` server (`sshserver_test.go`) and go-libvirt's own mock for the adapter |
+| Proxy software | generated Squid config (`pod.go`) | from `cloudInit`; the image must run cloud-init | from `cloudInit`, plus a vendor-data script that installs the guest agent (`seed.go`) | from `cloudInit`, plus a vendor-data script that installs the guest agent (`seed.go`) |
+| IP reachable from the operator | Pod network, so in-cluster only (not from `make run-dev` on a laptop) | external IP, once a firewall rule allows the cluster's egress IP | LAN address on the host's bridge; the operator needs a route to that LAN | LAN address on the host's bridge; the operator needs a route to that LAN |
+| Distinct egress IP per proxy | no: the cluster's egress | yes: ephemeral external IP | whatever the host's LAN routes through; not per VM by itself | whatever the host's LAN routes through; not per VM by itself |
 
 ## 3. Observability status
 
-| Layer | kubernetes | gcp | kvm |
-|---|---|---|---|
-| 1. Operation spans and metrics | yes: decorators | yes: decorators | yes: decorators |
-| 2. Child spans per backend call | yes: `tracing.RestConfigWrapper()`, parent-gated (`cmd/main.go`, `NewWithTransportWrapper`) | yes: the SDK's own `otelhttp`; not parent-gated | **no** |
-| 3. Logs carry `traceID` and `spanID` | n/a: the package does not log | yes, including the V(5) wire log | yes: every `libvirt call` line at V(1) |
-| Tested against a real collector | yes: the kind e2e checks `provider.create` and `provider.delete` in Tempo (`test/e2e/tracing_test.go`), operation spans only | no | no |
+| Layer | kubernetes | gcp | kvm | kvm2 |
+|---|---|---|---|---|
+| 1. Operation spans and metrics | yes: decorators | yes: decorators | yes: decorators | yes: decorators |
+| 2. Child spans per backend call | yes: `tracing.RestConfigWrapper()`, parent-gated (`cmd/main.go`, `NewWithTransportWrapper`) | yes: the SDK's own `otelhttp`; not parent-gated | **no** | yes: `kvm2.<Method>` from the `observe` decorator, `kvm2.dial`, `kvm2.download`; always under the caller's span, since every call carries the caller's ctx |
+| 3. Logs carry `traceID` and `spanID` | n/a: the package does not log | yes, including the V(5) wire log | yes: every `libvirt call` line at V(1) | yes: every `libvirt call` line at V(1), written inside its RPC's span |
+| Tested against a real collector | yes: the kind e2e checks `provider.create` and `provider.delete` in Tempo (`test/e2e/tracing_test.go`), operation spans only | no | no | no; the integration test asserts the parent-child topology in an in-memory recorder (`assertLifecycleSpans`) |
 
 ## 4. kubernetes (`internal/provider/kubernetes`)
 
@@ -329,18 +333,205 @@ the author's view.
     tries ARP first; it says `host` and `name` are on every log line, but
     the check and list paths lack them.
 
-## 7. Gaps shared by the existing providers
+## 7. kvm2 (`internal/provider/kvm2`)
 
-- **Adoption is not verified.** None of the three checks `LabelUID` when
-  `Create` finds an existing instance with the right name.
-- **The Create comment does not match kvm.** The interface comment in
-  `internal/provider/provider.go` says `Create` returns as soon as the
-  request is submitted; kvm blocks until the domain has started.
+The full user guide is [kvm2-provider.md](kvm2-provider.md) and the design
+it was built from is [kvm2-provider-spec.md](kvm2-provider-spec.md); this
+section is the author's view. Nothing in the package was taken from
+`internal/provider/kvm`; where the two agree it is because both follow the
+same libvirt.
+
+- **Package and seam.** `kvm2.go` (`Provider`, `New`, `withDefaults`,
+  `host`, `wrapErr`, the per-name lock), `create.go`, `get.go`,
+  `delete.go`, `list.go` (the four methods), `backend.go` (the seam, the
+  production adapter and the budgets), `ssh.go` (credentials, the forwarded
+  socket, keepalive), `hosts.go` (the pooled table and the production
+  dialer), `observe.go` (the span and log decorator), `image.go` (cache,
+  download, quota precheck), `seed.go`, `xml.go`, `volume.go`,
+  `metadata.go`, `id.go`, `state.go`, `errors.go`, `check.go`. The seam is
+  `backend`: the twenty go-libvirt calls the provider uses, each with `ctx`
+  first (go-libvirt's own methods take none and block), plus `Close`.
+  `libvirtBackend` adapts `*libvirt.Libvirt` and runs every call through
+  `call`, which applies the budget and the abandon policy. The `dialer`
+  seam separates the SSH transport: `newProvider` injects one for tests,
+  `newLibvirtDialer` is production. The `observe` decorator is applied
+  once, in `libvirtDialer.dial`, so provider code never holds an
+  undecorated backend.
+- **Config.** `ProviderConfig.KVM2`, the same `KVMConfig` type as kvm,
+  validated by `validateKVMBlock` with block name `kvm2` so messages read
+  `kvm2.hosts`. Defaults (`diskSizeGb` 10, socket, pool, image format and
+  firmware) are filled by `withDefaults`. `New` never dials; it checks that
+  the credential files exist and warns when one does not, or when
+  `insecureIgnoreHostKey` is on.
+- **ID.** `formatProviderID(host, name)` gives `hosts/<host>/domains/<name>`;
+  `parseProviderID` accepts exactly four segments with the literal
+  `hosts` and `domains` and non-empty host and name, else `ErrPermanent`.
+- **Create.** `Create` validates `Zone` (`host`; optional with one host),
+  `MachineType` (a catalogue key) and `Image` (`resolveImage`: a catalogue
+  key or an http(s) URL with qcow2 and uefi defaults) before any dial, and
+  logs the resolved parameters with `cloudInitBytes` at V(2). Then, under
+  the per-name lock (`lockName`) and on the pooled connection
+  (`hostTable.get`), `create`: `DomainLookupByName`; found means the
+  adoption guard (a non-empty metadata UID that differs from `req.UID` is
+  Permanent) and `ensureRunning`. Otherwise `StoragePoolLookupByName`,
+  `imageSizeIfUncached` (marker lookup, then `HEAD` or a closed `GET`) and
+  `checkPoolSpace` with `neededBytes` (image if uncached, plus
+  `diskSizeGb`, plus 16 MiB), then `ensureImage` (`cachedImage` on the
+  pooled connection, `fetchImage` streaming into `uploadImage` on a
+  `dialFresh` connection, `StoragePoolRefresh`, `writeMarker` last),
+  `ensureOverlay` (`ErrStorageVolExist` means reuse), `writeSeed` (delete,
+  build, upload; always rebuilt), two `StorageVolGetPath`, `buildDomainXML`
+  with file-type disks and the metadata element, `DomainDefineXML`,
+  `ensureRunning` (`DomainGetState`; `DomainCreate` with
+  `ErrOperationInvalid` as already running; `waitRunning` polls every
+  500 ms for up to 2 minutes, Transient on expiry). `Create` returns only
+  when the domain is running.
+- **Get and state.** `mapState(state, ip)`:
+
+  | libvirt state | IP | Provider state |
+  |---|---|---|
+  | nostate | any | Provisioning |
+  | running, blocked | empty | Provisioning |
+  | running, blocked | set | Running |
+  | paused, shutdown, shutoff, crashed, pmsuspended, anything else | any | Stopped |
+
+  `get`: `DomainLookupByName` (only `ErrNoDomain` reaches NotFound),
+  `readMetadata` (`DomainGetMetadata` with `metadataURI`;
+  `ErrNoDomainMetadata` means unmanaged with an empty UID; an element that
+  does not parse is Permanent), `inspect` (`DomainGetState`, then for an
+  active domain `address`). `addressSources` asks the guest agent first,
+  then ARP; a source that fails is skipped at V(2), only ctx ending is an
+  error, so a cancelled Get is Transient and not a false Provisioning.
+  `firstIPv4` drops loopback, link-local and unspecified addresses and an
+  address whose type tag and literal disagree. `UID` and `CreatedAt` are
+  read back on every call; an unparsable `createdAt` gives the zero time.
+- **Delete.** Under the per-name lock: `removeDomain` (`DomainDestroy`,
+  tolerating not running and gone; `DomainUndefineFlags` with NVRAM,
+  managed save and snapshot metadata) and `removeVolumes` (overlay and seed
+  by name through `deleteVolumeIfPresent`; a missing pool is a no-op). A
+  host no longer in the config is a logged no-op success. Nothing is left
+  when nil is returned.
+- **ListByTag.** `ConnectListAllDomains` (active and inactive) on every
+  host in config order; `readMetadata` and `managed` keep kvm2's own
+  domains; `inspect` fills state and address. A domain whose metadata or
+  state cannot be read is skipped at V(1), unless ctx ended, which fails
+  the call. A host that cannot be dialled or listed fails the whole call.
+- **Errors.** `classify(op, err)`: an error already carrying a class keeps
+  it; libvirt codes through `classifyLibvirt` (`ErrNoDomain` and
+  `ErrNoStorageVol` NotFound, Transient when `op == "create"`;
+  `ErrNoStoragePool`, `ErrXMLError`, `ErrNoSupport`, `ErrInvalidArg`,
+  `ErrAuthFailed`, `ErrAccessDenied` Permanent; the rest Transient);
+  `knownhosts.KeyError` and the `unable to authenticate` text from
+  `x/crypto/ssh` Permanent; everything else Transient, including a missing
+  credential file. `permanentf` and `transientf` classify at the source:
+  placement, XML builders, a parse failure of the key or known_hosts, an
+  image 4xx, a missing or zero `Content-Length`, a sha256 mismatch, a pool
+  that is not running (Permanent); an image 429 or 5xx, a short body, the
+  running wait expiring (Transient). `checkPoolSpace` is the only source of
+  `ErrQuotaExceeded`. `logAPIError` writes `kvm2 backend call failed` with
+  `libvirtErrorFields`.
+- **Check.** `Check` runs `checkHost` per host concurrently, each through
+  `probeHost` on a `dialFresh` connection that is closed when the probe
+  ends; the pooled table is never consulted. `ConnectGetLibVersion`,
+  `ConnectGetHostname`, `StoragePoolLookupByName`, `StoragePoolGetInfo`;
+  a pool that is not running is `poolNotActiveError`. Details are filled
+  as facts arrive (`user` and `pool` always; `libvirtVersion`, `hostname`,
+  `poolState`, `poolCapacityGiB`, `poolAvailableGiB`). `checkReason`: a
+  `dialError` is judged by its stage (`credentials` gives
+  `CredentialsUnavailable`; `ssh` with a `knownhosts.KeyError` or the auth
+  text gives `Unauthorized`; any other stage, including a dial that ran
+  out of time, gives `Unreachable`); then `context.DeadlineExceeded` or
+  `Canceled` `Timeout`, `ErrAccessDenied` or `ErrAuthFailed` `Forbidden`,
+  `ErrNoStoragePool` `PoolNotFound`, any other libvirt code `APIError`,
+  anything else `Unreachable`; `PoolNotActive` from the pool state.
+- **Connections.** `hostTable` keeps one pooled connection per host,
+  dialled lazily under a per-host one-slot semaphore so waiting ends with
+  the caller's ctx; a dead backend (`liveness`) is closed and redialled on
+  the next `get`. `dialSSHSocket` runs credentials, TCP, the SSH handshake
+  and the `direct-streamlocal` channel to the socket, each failure a
+  `dialError` with its stage; `libvirtDialer.connect` bounds the whole
+  sequence plus the libvirt `Connect` with the dial budget and starts the
+  `keepalive` (15 s, `keepalive@openssh.com`) and a watcher on go-libvirt's
+  `Disconnected`. `call` runs each blocking go-libvirt call in a goroutine
+  and waits on the result, ctx and a budget timer; abandoning a call
+  (`link.markDead`) closes the socket and the SSH client, returns Transient
+  wrapping the ctx or deadline error, and every other call waiting on that
+  link fails once with Transient. `loadCredentials` runs on every dial.
+- **Logging.** Logger from the context, named `kvm2`, with `provider`,
+  and `host` and `name` on lines about an instance. V(1): one line per
+  operation outcome (`kvm2 domain ensured running`, `kvm2 domain fetched`,
+  `kvm2 domain deleted`, `kvm2 domains listed`, `kvm2 host checked`), one
+  `libvirt call` line per RPC from `observe` (rpc name, identifiers,
+  duration, error fields), the connection events `libvirt host connected`
+  and `libvirt host connect failed` with `stage`, and `kvm2 backend call
+  failed`. V(2): XML, metadata, addresses, listed names, pool sizes, the
+  resolved Create parameters. Upload and download bodies, cloud-init and
+  key material are never logged.
+- **Tracing.** `observe.call` starts `kvm2.<Method>` per RPC with
+  `kvm2.host` and the identifiers (`kvm2.domain`, `kvm2.pool`,
+  `kvm2.volume`), sets `kvm2.libvirt_code` on failure and writes the
+  `libvirt call` line inside the span; `libvirtDialer.dial` starts
+  `kvm2.dial` (`kvm2.address`, `kvm2.user`); `fetchImage` starts
+  `kvm2.download` (`kvm2.url_host`, `kvm2.sha256_configured`,
+  `kvm2.bytes`). Both the decorator and the dialer take `tracing.Option`
+  so tests inject a recorder; `New` leaves them on the global provider.
+- **Tests.** Pure builders (`xml_test.go`, `volume_test.go`,
+  `seed_test.go`, `id_test.go`, `metadata_test.go`, `state_test.go`);
+  `errors_test.go` (the classify table with real SSH error strings and a
+  real `knownhosts.KeyError`); `backend_test.go` (the adapter over
+  go-libvirt's own mock through a `net.Pipe`: deadline kills the
+  connection, cancel unblocks, killed by another caller); `hosts_test.go`;
+  `ssh_test.go` over `sshserver_test.go` (every dial stage, host-key
+  policies, key material never in errors or logs); `image_test.go`
+  (marker rules, streaming asserted by counting bytes, dedicated upload
+  connection); `create_test.go`, `get_test.go`, `delete_test.go`,
+  `list_test.go`, `kvm2_test.go`, `provider_test.go` (the contract §11
+  list, named test by test in the spec's entry 21); `observe_test.go`
+  (every seam method spanned and logged exactly once, by reflection);
+  `check_test.go` (the reason table, never touches the pool). The double is
+  `fakeBackend` in `fake_test.go`, whose `twin` shares state under its own
+  call log so a test can tell the pooled connection from a dedicated one.
+  Real backend: `kvm2_integration_test.go` behind `//go:build integration`,
+  driven by `KVM2_TEST_*` variables and `make test-integration-kvm2`,
+  described in
+  [testing.md](testing.md#the-kvm2-integration-test-real-hypervisor).
+- **Deployment.** The `kvm-ssh` Secret and its mount are shared with kvm
+  (`config/manager/manager.yaml` says so), `make kvm-ssh-secret` is reused,
+  a commented `kvm2-home` entry in both providers configs,
+  `config/samples/proxy_kvm2.yaml` (zone `beelink`, matching the shipped
+  config), the `kvm2` type in the `status.type` comment of
+  `api/v1alpha1/providerhealth_types.go`. No RBAC.
+- **Known gaps** (see also [kvm2-provider.md § Known gaps](kvm2-provider.md#known-gaps)).
+  - `Create` blocks, including the first image download on each host.
+  - GC blind spots: volumes without a domain are invisible to GC, and so are
+    VMs on a host removed from the config, which `Delete` treats as done.
+  - No adoption of kvm's domains: a switched entry needs its Proxies
+    deleted first.
+  - Cancelling one operation cuts off the others on that host once: a
+    cancelled `call` kills the pooled connection.
+  - `Content-Length` is required for image downloads.
+  - A paused or pmsuspended domain cannot be adopted by a repeat `Create`:
+    `DomainCreate` refuses, `waitRunning` expires, Transient on every retry.
+  - Credential parse errors carry the parser's own text, which can quote a
+    PEM header type or a malformed known_hosts token (never key bytes).
+  - No reused-overlay check: an existing overlay is reused without checking
+    its backing image or size.
+  - The sample cloud-init installs Squid with `http_access allow all`.
+
+## 8. Gaps shared by the existing providers
+
+- **Adoption is not verified in three of the four.** kubernetes, gcp and
+  kvm adopt any existing instance with the right name without checking
+  `LabelUID`; kvm2 returns `ErrPermanent` when the UID differs.
+- **The Create comment does not match kvm and kvm2.** The interface comment
+  in `internal/provider/provider.go` says `Create` returns as soon as the
+  request is submitted; both libvirt providers block until the domain has
+  started.
 - **`Stopped` and `Terminated` are produced inconsistently.** kubernetes
-  never produces Stopped, kvm never produces Terminated, and the reconciler
-  treats them the same.
+  never produces Stopped, kvm and kvm2 never produce Terminated, and the
+  reconciler treats them the same.
 - **Different `ListByTag` partial-failure policies.** gcp leaves failed
-  parts out, kvm fails the whole list. Both are safe.
+  parts out, kvm and kvm2 fail the whole list. Both are safe.
 - **Stale `mock` comments** in `internal/provider/provider.go`,
   `internal/provider/registry/registry.go` and `api/v1alpha1/proxy_types.go`
   (and so the generated CRD's `spec.provider` description).
@@ -351,33 +542,41 @@ the author's view.
   is still open.
 - **Tracing.** kvm has no child spans, so a kvm trace shows `provider.create`
   and nothing under it; correlation works only through `traceID` on its V(1)
-  log lines. Only kubernetes is tested against a real collector, and that
-  test checks only the operation spans; nothing checks that gcp's SDK spans
-  or the kubernetes API spans nest under `provider.*`.
+  log lines. kvm2 has them, but only an in-memory recorder checks their
+  topology. Only kubernetes is tested against a real collector, and that
+  test checks only the operation spans; nothing checks that gcp's SDK spans,
+  the kubernetes API spans or kvm2's spans nest under `provider.*` in a
+  real collector.
 
-## 8. Proposed features
+## 9. Proposed features
 
-Listed only. Nothing below is designed or implemented.
+Listed only. Nothing below is designed or implemented, except where a
+bullet says kvm2 already does it.
 
 - **Child spans in kvm:** extend the `tracingAPI` decorator in
   `kvm/trace.go`, which already wraps every libvirt call, to also start a
   span per RPC, and add spans for the SSH dial and the image download. The
   seam's methods take no `ctx` today (`hostPool.call` hands `fn` only the
-  API), so the span context has to be plumbed in.
-- **Tempo e2e checks for child spans**, and for gcp and kvm through their
-  real backend runbooks.
+  API), so the span context has to be plumbed in. kvm2's `observe.go` over
+  its ctx-first seam is the shape this would take.
+- **Tempo e2e checks for child spans**, and for gcp, kvm and kvm2 through
+  their real backend runbooks.
 - **A shared conformance suite:** for example `internal/provider/providertest`,
   which runs the contract §11 tests against any provider plus a fake
   backend, so a new provider gets the contract tests for free.
 - **Verify the UID when `Create` adopts an instance.** Return `ErrPermanent`
   on a mismatch instead of adopting an instance that belongs to someone else.
+  kvm2 does this; the other three do not.
 - **Report asynchronous create failures:** gcp keeps the operation name, and
   `Get` reports the failure instead of NotFound.
 - **Capacity or quota signals** for kvm, such as pool free space and host
-  memory, mapped to `ErrQuotaExceeded`.
+  memory, mapped to `ErrQuotaExceeded`. kvm2 has the pool-space precheck;
+  host memory is still unchecked in both.
 - **GC for secondary resources:** mark volumes and similar resources, and
   let `ListByTag` or a second listing method cover them.
-- **Reload credentials and config without a restart.**
+- **Reload credentials and config without a restart.** kvm2 re-reads its
+  SSH material on every dial; the providers config is still read once per
+  process for every type.
 - **Checker `Details`** for kubernetes (server version) and gcp (quota
   headroom).
 - **Internal-IP mode** for gcp, and a firewall rule managed per provider.