2026-09-17 23:12:46 +02:00
6 changed files with 985 additions and 73 deletions
@@ -160,6 +160,10 @@ Managed proxies as VMs on libvirt/KVM hosts you own. Full guide:
   size (`placement.machineType`) and an image (`placement.image`) — see
   [config/samples/proxy_kvm.yaml](config/samples/proxy_kvm.yaml).
 The `kvm2` provider uses the same Secret and the same block shape (`type: kvm2`,
 block `kvm2:`), so the steps above apply to it unchanged; its guide is
 [docs/kvm2-provider.md](docs/kvm2-provider.md).
 Cloud-init from a Secret: the Secret **must** carry the label
 `crawl.example.com/cloud-init: "true"` — the operator's cache only holds
 labelled Secrets, so an unlabelled one is invisible (the Proxy reports
@@ -187,6 +191,7 @@ can be two differently-configured instances of the same `gcp` type (see
 | `kubernetes` | A real Squid pod (`ubuntu/squid:6.6-24.04_edge` by default) in the same cluster the operator runs in | `image` (optional) | Needs no cloud account — local dev, CI, and the kind quickstart. Pods share the cluster's egress IP, so it exercises the full lifecycle but not distinct egress paths. |
 | `gcp` | A Compute Engine VM with an ephemeral external IP | `project` (required), `network`, `networkTag`, `diskSizeGb` | The real egress fleet. Deliberately uses only four API calls (Insert / Get / Delete / AggregatedList), all fire-and-forget: `Create` returns as soon as the operation is submitted and the reconciler discovers progress by polling `Get`. Auth is Application Default Credentials — workload identity in-cluster, `gcloud` ADC locally; no key-file plumbing. |
 | `kvm` | A libvirt VM on one of your own hypervisors, reached over SSH | `sshKeyFile`, `knownHostsFile` (unless `insecureIgnoreHostKey`), `machineTypes`, `hosts` (required); `images`, `diskSizeGb`, `vendorData`, `installGuestAgent` | Home-lab or on-prem egress. The base image is downloaded into each host's pool on first use; `Create` waits for the VM to start. See [docs/kvm-provider.md](docs/kvm-provider.md). |
 | `kvm2` | The same: a libvirt VM on one of your own hypervisors, reached over SSH | the same block shape as `kvm`, under `kvm2:` | The second-generation libvirt provider, built from the provider contract. Adds a child span per libvirt RPC, a pool-space precheck reported as quota (the reconciler waits instead of failing), SSH credentials re-read on every dial, and image downloads and health checks on connections of their own. It does not adopt `kvm`'s VMs. See [docs/kvm2-provider.md](docs/kvm2-provider.md). |
 ### Adding a provider
@@ -10,6 +10,7 @@
 | kubernetes provider | `internal/provider/kubernetes` | library | — | real Squid pods in this cluster (local dev/CI) |
 | gcp provider | `internal/provider/gcp` | library | — | Compute Engine VMs, four API calls, fire-and-forget ops |
 | kvm provider | `internal/provider/kvm` | library | — | VMs on libvirt hosts over SSH-forwarded sockets; image cache, overlay disks, NoCloud seeds |
 | kvm2 provider | `internal/provider/kvm2` | library | — | the same backend, built clean-room from the provider contract as a second type: per-RPC child spans, pool-space quota, per-call budgets, its own metadata namespace, credentials re-read on every dial |
 | Health engine | `internal/health` | Runnable | yes | through-the-proxy probes, thresholds, transition events |
 | Lease store | `internal/lease` | Runnable (expiry sweep) | no | in-memory leases + cooldowns, single mutex |
 | Discovery API | `internal/discovery` | Runnable | no | HTTP list/lease/release/report on `:8090` |
@@ -168,6 +169,39 @@ providerID = hosts/<host>/domains/<name>; GC identity lives in the domain's
 <metadata> element (libvirt has no labels). All RPCs run under a context-
 bounded call: go-libvirt never times out on its own, so a cancelled reconcile
 closes the SSH connection to unblock it.
 kvm2 provider (internal/provider/kvm2/) - the same transport, a second type built from the contract
 prov.Create ──► placement checked against the catalogue (Permanent before any dial)
                DomainLookupByName: exists → uid guard (foreign uid = Permanent),  ─┐ every step
                  start if needed, wait for running, done                          │ idempotent,
                else: StoragePoolGetInfo precheck: image (if uncached) + diskSizeGb │ under a
                        + 16 MiB must fit, else ErrQuotaExceeded (reconciler       │ per-name lock
                        waits 5m)                                                  │ shared with
                      ensure base image: marker present = cached; otherwise GET    │ Delete
                        streamed into StorageVolUpload on a DEDICATED connection,  │
                        StoragePoolRefresh, marker written last                    │
                      overlay volume (exists = reuse)                              │
                      seed ISO (always rebuilt)                                    │
                      DomainDefineXML (<metadata> in kvm2's own namespace:         │
                        managed, uid, createdAt) + DomainCreate, poll until        │
                        running (2m cap)                                          ─┘
 prov.Get    ──► DomainLookupByName + DomainGetMetadata + DomainGetState
                  + DomainInterfaceAddresses (guest agent, then ARP; first routable IPv4)
                  running without IP = Provisioning; nostate = Provisioning;
                  only ErrNoDomain is NotFound
 prov.Delete ──► destroy (not running = fine) → undefine (nvram, managed save, snapshots)
                  → delete overlay + seed by name; unconfigured host = logged no-op
 prov.ListByTag ─► ConnectListAllDomains on every host, keep domains whose kvm2
                    <metadata> says managed=true; one host down fails the sweep,
                    one unreadable domain is skipped
 Check       ──► per host, on a FRESH connection: version, hostname, pool lookup,
                  pool info, close; never the pooled one
 providerID = hosts/<host>/domains/<name>. Every RPC has its own budget (dial 10s,
 RPC 30s, DomainCreate 90s, seed upload 60s, image transfer 20m): a call that
 outlives its ctx or budget kills the pooled connection (the only way to unblock
 go-libvirt), so other calls in flight on that host fail once with Transient and
 retry; the next call redials. Each RPC is a child span kvm2.<Method> under
 provider.*, plus kvm2.dial and kvm2.download.
 ```
 The reconciler never watches provider-side resources (Pods or GCP VMs).
@@ -343,6 +377,15 @@ incoming trace context):
 - `provider.create|get|delete|list` (client spans, `provider.WithTracing`
  decorator wired outermost around `WithMetrics`); the GCP SDK's own
  otelhttp transport contributes HTTP child spans automatically.
 - kvm2 child spans: under `provider.*`, one `kvm2.<Method>` client span per
  libvirt RPC (for example `kvm2.DomainDefineXML`, attributes `kvm2.host`,
  `kvm2.domain`, `kvm2.pool`, `kvm2.volume`, `kvm2.libvirt_code` on
  failure), `kvm2.dial` around the SSH handshake and libvirt connect, and
  `kvm2.download` around an image fetch; under `providerhealth.check`,
  `kvm2.dial` and the four `kvm2.<Method>` spans of a check. They come
  from a span decorator over the seam, applied once in the dialer, so no
  RPC is unspanned. The `libvirt call` log line of an RPC is written
  inside its span. The `kvm` provider has no child spans.
 - Kubernetes API calls become client child spans via a wrapped
  `rest.Config` transport — gated on an existing parent span, so informer
  list/watch long-polls and leader-election renewals never create root
@@ -368,11 +411,12 @@ the slog handler's ctx.
 Answers "can this cluster reach each provider's backend right now?"
 without waiting for a Proxy to fail. A provider opts in by implementing
 `provider.Checker`: `Check(ctx) []TargetStatus` returns one verdict per
-target and never fails as a whole. All three providers do:
+target and never fails as a whole. All four providers do:
 | Provider | Target | What one check does |
 |---|---|---|
 | kvm | each `kvm.hosts` entry | over the pooled SSH+libvirt connection: libvirt version and hostname, then the configured pool must exist and be running; reports pool capacity and free space ([details](kvm-provider.md#health-check)) |
 | kvm2 | each `kvm2.hosts` entry | on a fresh SSH+libvirt connection dialled for the check and closed after it: libvirt version and hostname, then the configured pool must exist and be running; `details` carry `user`, `pool`, `hostname`, `libvirtVersion`, `poolState`, `poolCapacityGiB` and `poolAvailableGiB` ([details](kvm2-provider.md#health-check)) |
 | kubernetes | `apiserver` | lists at most one managed Pod cluster-wide: the verb and scope `ListByTag` and GC rely on |
 | gcp | the project | one page of an aggregated instance list with `maxResults=1`: credentials, Compute API enabled, `compute.instances.list` |
@@ -613,3 +657,48 @@ they read as choices, not accidents. Chronological by build step.
 - **A provider name that is not a valid object name** gets metrics but no
  `ProviderHealth` object, with a warning at start. Tightening config
  validation instead would crash-loop existing deployments on upgrade.
 - **kvm2 is a second provider type built clean-room from the contract,**
  not a rewrite of `kvm` in place. Its spec
  (`docs/kvm2-provider-spec.md`) and code were written from
  [provider-contract.md](provider-contract.md), the kvm user guide and
  go-libvirt alone, never from `internal/provider/kvm/`, as a test of
  whether the contract is enough to build a provider. Both types ship,
  share the `KVMConfig` block shape and the `kvm-ssh` Secret, and may
  share a host.
 - **kvm2 image downloads run on a dedicated connection.** A cancelled or
  overlong call kills the connection it ran on (the only way to unblock
  go-libvirt), so a 435 MB upload on the pooled connection could die
  because a poll for another Proxy was cancelled. The transfer dials its
  own connection and closes it afterwards; the cheap checks and the marker
  stay on the pooled one.
 - **kvm2 health checks dial fresh connections**, the opposite of the kvm
  decision above. With credentials re-read on every dial a fresh
  connection does pick up a rotated Secret, and a check that runs out of
  time then closes only its own connection, so it cannot disrupt a
  reconcile. The cost is one sshd session per host per round.
 - **A cancelled kvm2 call kills the pooled connection.** Every RPC has a
  budget; when a call outlives its ctx or budget the adapter closes the
  socket and the SSH client, returns Transient, and marks the connection
  dead so the next call redials. Other calls in flight on that host fail
  once with Transient and their callers retry. Documented in the guide
  rather than avoided: the alternative, leaving a hung RPC blocked
  forever, pins a worker.
 - **kvm2 marks domains in its own metadata namespace**, so it never lists
  or adopts domains the `kvm` provider made. Switching an entry from `kvm`
  to `kvm2` means deleting its Proxies first; the two types can share a
  host without one collecting the other's VMs.
 - **kvm2 turns pool free space into the quota signal.** libvirt reports a
  full pool as an ordinary error, which would retry on the fast backoff.
  Before creating any volume, `Create` compares `StoragePoolGetInfo`'s
  available bytes with the image (if uncached) plus `diskSizeGb` plus 16
  MiB for the seed, and returns `ErrQuotaExceeded` below that, so the
  reconciler waits 5 minutes instead.
 - **kvm2 classifies SSH authentication and host-key failures as
  Permanent.** A key missing from `authorized_keys` or a host key missing
  from or different in known_hosts needs an admin; kvm retried these
  forever. A missing key or known_hosts file stays Transient, because the
  optional Secret may land later.
 - **kvm2 re-reads the SSH key and known_hosts on every dial**, so rotating
  the Secret needs no restart; nothing is cached. The constructor only
  checks that the files exist, to warn early without failing the provider
  set.
@@ -0,0 +1,597 @@
 # The kvm2 provider
 Runs Managed proxies as VMs on your own libvirt/KVM hypervisors. It is the
 second-generation libvirt provider, built clean-room from
 [provider-contract.md](provider-contract.md) and the design in
 [kvm2-provider-spec.md](kvm2-provider-spec.md), and it ships side by side
 with the first-generation `kvm` provider: both types can be configured at
 once, both take the same config block shape, and both read the same
 `kvm-ssh` Secret. What kvm2 adds over kvm: a child span per libvirt RPC,
 a pool-space precheck that maps to the reconciler's quota wait, SSH
 credentials re-read on every dial, image downloads and health checks on
 connections of their own, an adoption guard on the owning UID, and SSH
 authentication failures that stop retrying.
 Like kvm, the operator reaches each host over SSH, forwards the libvirt
 unix socket through that connection and speaks libvirt's own RPC over it,
 so the hosts need nothing beyond `sshd` and `libvirtd` and the operator
 image carries no libvirt libraries. VMs boot a stock distro cloud image
 that the operator downloads into the host's storage pool on first use,
 with the Proxy's cloud-init delivered on a NoCloud seed ISO built
 in-process.
 ## Hypervisor prerequisites
 Per host, once:
 1. **libvirt and KVM** (`libvirtd` running, `/dev/kvm` present).
 2. **An SSH user in the `libvirt` group**, so it may open
   `/var/run/libvirt/libvirt-sock`. The operator authenticates with a
   private key; nothing else on the host is touched. The socket is used
   as `qemu:///system`, so no further host permissions are needed.
 3. **A bridge to a network with DHCP** (for example `br0` bridged onto
   the LAN). Each VM gets one virtio NIC on it and takes its IP from that
   network's DHCP. The operator's pod network and the crawler clients
   must route to those IPs: the health engine probes `http://<ip>:<port>`
   through the proxy and clients get the same address from the discovery
   API. Nothing on the host filters the port; the provider does not use
   `spec.port` at all.
 4. **A storage pool** (default `default`, usually `/var/lib/libvirt/images`)
   with room for one base image per distro (a few hundred MB) plus the
   per-VM overlays (`diskSizeGb` each, thin-provisioned). The pool must be
   running: a stopped pool fails the health check with `PoolNotActive` and
   fails `Create` permanently.
 5. **UEFI firmware for guests**: the `ovmf` package on Debian/Ubuntu,
   `edk2-ovmf` on Fedora/RHEL. VMs boot UEFI by default with secure boot
   off (libvirt picks the OVMF binary itself); a domain start fails with
   "no firmware found" without it. Only `firmware: bios` images skip this.
 6. **Outbound internet from the VMs**, in practice: a vendor-data script
   installs `qemu-guest-agent` from the distro mirrors on first boot, and
   the agent is how the operator learns the VM's address. Without it the
   operator falls back to the host's ARP table, which only sees traffic
   addressed to the host itself, so on a plain bridge a VM without the
   agent stays `Provisioning`.
 Check from the operator's point of view before wiring it in:
 ```sh
 ssh novakj@beelink virsh -c qemu:///system list --all      # user can drive libvirt
 ssh novakj@beelink virsh -c qemu:///system pool-info default
 ssh novakj@beelink ip link show br0
 ```
 ## Cluster-side setup
 1. Create the SSH Secret from a key pair dedicated to the operator and a
   `known_hosts` file holding every host's key. It is the same `kvm-ssh`
   Secret the `kvm` provider uses; one Secret serves both types, and if
   you already have it there is nothing to do here.
   ```sh
   # once: a key pair for the operator; the .pub goes on every host
   ssh-keygen -t ed25519 -f ~/.ssh/proxy-operator_ed25519 -N '' -C proxy-operator
   ssh-copy-id -i ~/.ssh/proxy-operator_ed25519.pub novakj@beelink
   # once per host: its host key, under the address the *operator* will
   # dial (hosts[].address), which may differ from what your laptop uses
   ssh-keyscan -H beelink.tail8c69c.ts.net >> ~/.ssh/kvm_known_hosts
   # the Secret: the PRIVATE key (not the .pub) plus known_hosts
   kubectl -n egress-proxies-operator-system create secret generic kvm-ssh \
     --from-file=id_ed25519=$HOME/.ssh/proxy-operator_ed25519 \
     --from-file=known_hosts=$HOME/.ssh/kvm_known_hosts
   ```
   `make kvm-ssh-secret` runs that last command as an apply, so it also
   updates an existing Secret; override the paths with
   `KVM_SSH_KEY=... KVM_KNOWN_HOSTS=...`. Putting the `.pub` in the
   Secret is the classic mistake: the target's `kvm-ssh-secret` check
   refuses a file without `PRIVATE KEY` in it, and the provider itself
   would fail every call permanently with a "parsing ssh key file" error.
   Without `known_hosts`, set `insecureIgnoreHostKey: true` and accept
   that a MITM could hand you a fake hypervisor.
   `config/manager/manager.yaml` mounts the Secret (optional) at
   `/etc/proxy-operator/kvm` as two files, `id_ed25519` and
   `known_hosts`, which is what `sshKeyFile` and `knownHostsFile` in the
   provider config point at. The deployment starts fine without the
   Secret, even with a kvm2 provider configured: `New` only checks that
   the files exist and logs a `WARNING:` line when one does not; the
   provider's proxies fail transiently and recover once the Secret lands.
   **Rotating the key or a host key needs no restart.** Both files are
   read and parsed on every dial (`loadCredentials` in
   `internal/provider/kvm2/ssh.go`); nothing is cached. Re-run the
   command or the make target, and the next connection, health check
   included, uses the new material. Connections that are already open
   stay open until they drop.
 2. Add a provider entry to the providers config
   (`config/manager/providers_config.yaml` in-cluster; full sample in
   `config/samples/providers-config.yaml`). The type is `kvm2` and the
   block is `kvm2:`; its fields are those of the `kvm` block:
   ```yaml
   - name: kvm2-home
     type: kvm2
     kvm2:
       sshKeyFile: /etc/proxy-operator/kvm/id_ed25519
       knownHostsFile: /etc/proxy-operator/kvm/known_hosts
       images:
         debian-13:
           url: https://cloud.debian.org/images/cloud/trixie/latest/debian-13-generic-amd64.qcow2
       machineTypes:
         small: {vcpus: 1, memoryMiB: 1024}
       hosts:
         - name: beelink
           address: beelink.tail8c69c.ts.net:22
           user: novakj
           bridge: br0
   ```
   Validation messages name the block, so a mistake reads
   `kvm2.hosts[0]: bridge is required`, not `kvm.hosts`. The providers
   config is read once per process: adding a host or an image entry needs
   a manager restart; credentials do not.
 3. Apply a Proxy (`config/samples/proxy_kvm2.yaml`) and watch it:
   ```sh
   kubectl get px -w
   # NAME                MODE      PROVIDER    PHASE          IP             HEALTHY
   # proxy-kvm2-sample   Managed   kvm2-home   Provisioning
   # proxy-kvm2-sample   Managed   kvm2-home   Running        192.168.1.73
   # proxy-kvm2-sample   Managed   kvm2-home   Ready          192.168.1.73   True
   ```
   The first Proxy for a given image on a given host sits in
   `Provisioning` for the length of the image download; the operator log
   shows `fetching base image into host pool` with the URL and size and
   `base image cached` with the size, duration and digest.
 ## Config reference
 The block is `kvm2:` and its Go type is the shared `KVMConfig` in
 `internal/provider/config_kvm.go`, so the fields, defaults and validation
 are the same as for `kvm`. Defaults are applied by `withDefaults` in
 `internal/provider/kvm2/kvm2.go`.
 | Field | Required | Default | Meaning |
 |---|---|---|---|
 | `sshKeyFile` | yes | | private key used for every host; OpenSSH or PEM, parsed on every dial |
 | `knownHostsFile` | unless `insecureIgnoreHostKey` | | OpenSSH `known_hosts` to verify host keys against, read on every dial |
 | `insecureIgnoreHostKey` | | `false` | skip host-key verification entirely; `New` logs a `WARNING:` line when it is on |
 | `diskSizeGb` | | `10` | capacity of each VM's overlay disk; cloud-init's growpart expands the root fs into it. Also what the pool-space precheck reserves per VM |
 | `vendorData` | | | extra cloud-init vendor-data placed on every seed next to the Proxy's user-data. Beware that cloud-init drops any top-level key of a vendor-data **cloud-config** that the user-data also defines. Shipped as one part of a MIME multipart alongside the agent script when `installGuestAgent` is on |
 | `installGuestAgent` | | `true` | add the vendor-data script that installs and starts `qemu-guest-agent`. Turn off for images that ship it already or cannot reach a package mirror; IP discovery then depends on the host's ARP table |
 | `images.<key>` | | | the key names the image in `placement.image`. Letters, digits, dots, dashes and underscores only: it becomes the cached volume name `base-<key>.<format>` |
 | `images.<key>.url` | yes per entry | | http(s) URL of a cloud image (qcow2 or raw). The server must send `Content-Length` |
 | `images.<key>.sha256` | | | expected digest, 64 hex characters; verified after download, a change triggers a re-download |
 | `images.<key>.format` | | `qcow2` | `qcow2` or `raw` |
 | `images.<key>.firmware` | | `uefi` | `uefi` (OVMF, secure boot off) or `bios` (SeaBIOS) for images without an EFI system partition |
 | `machineTypes.<key>` | at least one | | `{vcpus, memoryMiB}` size profile, both greater than zero |
 | `hosts[].name` | yes | | the `placement.zone` value that selects this host; unique. Letters, digits, dots, dashes and underscores only: it becomes part of the providerID `hosts/<name>/domains/<vm>` |
 | `hosts[].address` | yes | | SSH endpoint, `host:port` |
 | `hosts[].user` | yes | | SSH user, member of the `libvirt` group |
 | `hosts[].socket` | | `/var/run/libvirt/libvirt-sock` | libvirt socket path on the host |
 | `hosts[].pool` | | `default` | storage pool for base images, overlays and seeds |
 | `hosts[].bridge` | yes | | host bridge for the VM's NIC |
 ## Placement
 All three fields are checked before any host is dialled. An unknown value
 is `ErrPermanent`, which the reconciler latches as `Failed` with the
 message below until the spec changes.
 | Proxy field | Meaning for kvm2 | Error when unknown |
 |---|---|---|
 | `placement.zone` | host name from `hosts`. May be omitted when exactly one host is configured; otherwise required | `placement.zone "x" is not in kvm2.hosts`; with several hosts and no zone, `placement.zone is required when kvm2.hosts lists more than one host` |
 | `placement.machineType` | key in `machineTypes` | `placement.machineType "x" is not in kvm2.machineTypes` |
 | `placement.image` | key in `images`, or an `http(s)://` URL used directly with the defaults (`qcow2`, `uefi`) | `placement.image "x" is neither a key in kvm2.images nor an http(s) URL` |
 | `placement.region` | ignored. It is still part of the spec hash, so changing it replaces the VM and does nothing else | |
 The messages come from `Create` in `internal/provider/kvm2/create.go`,
 `host` in `kvm2.go` and `resolveImage` in `image.go`. Fix the spec (which
 replaces the VM) or the config. Catalogue edits affect only new VMs: the
 spec hash covers the placement values, not what the config maps them to.
 ## What lands on the host
 For a Proxy whose deterministic name is `proxy-abc…`, on the chosen
 host's pool:
 | Object | Name | Lifetime |
 |---|---|---|
 | base image | `base-<key>.<format>` (for a URL image, `<key>` is the first 16 hex characters of the URL's sha256) | cached; survives Proxy deletion |
 | image marker | `base-<key>.<format>.sha256`, a 64-byte raw volume holding the digest | with the base image |
 | overlay disk | `proxy-abc….qcow2`, qcow2 backed by the base file, `diskSizeGb` capacity | with the Proxy |
 | seed ISO | `proxy-abc…-seed.iso`, raw | with the Proxy |
 | domain | `proxy-abc…` | with the Proxy |
 The names are built in `internal/provider/kvm2/volume.go`.
 The domain (`buildDomainXML` in `xml.go`) is a `q35` machine with a
 host-passthrough CPU, ACPI and APIC, memory in MiB and vcpus from the
 machine type, the overlay on `vda` (virtio) and the seed as a read-only
 SATA cdrom on `sda`, one virtio NIC on the bridge, a serial console on a
 pty, the guest-agent channel `org.qemu.guest_agent.0`, a virtio RNG fed
 from `/dev/urandom`, and no graphics. Both disks are `type="file"` with
 the paths `StorageVolGetPath` returned: AppArmor derives QEMU's profile
 from those paths, and volume-type disks fail with permission denied on
 stock Debian and Ubuntu hosts. UEFI images get `<os firmware="efi">` with
 the `secure-boot` feature disabled; BIOS images get neither.
 Its `<metadata>` carries one element in kvm2's own namespace,
 `https://gitea.home.hrajfrisbee.cz/kacerr/egress-proxies-operator/kvm2`
 (`metadataURI` in `metadata.go`):
 ```xml
 <instance xmlns="https://gitea.home.hrajfrisbee.cz/kacerr/egress-proxies-operator/kvm2">
  <managed>true</managed>
  <uid>0f3a…</uid>
  <createdAt>2026-09-16T17:42:11Z</createdAt>
 </instance>
 ```
 That is how orphan GC recognises its own VMs, since libvirt has no
 labels, and how a repeat `Create` knows whether an existing domain of the
 same name belongs to this Proxy: a different non-empty `uid` is a
 permanent error naming both UIDs; an empty one is adopted. The namespace
 is kvm2's own, so the `kvm` provider's domains are invisible to it (see
 *Migrating from kvm*).
 The seed (`buildSeedISO` in `seed.go`) is an ISO 9660 image labelled
 `cidata` with three files. `user-data` is the Proxy's `cloudInit`, **byte
 for byte**: any format cloud-init accepts works, including a Jinja
 template, a MIME archive, a shell script or a gzipped payload; the
 operator never parses or re-renders it. `meta-data` holds
 `instance-id: <name>` and `local-hostname: <name>`, so a reboot is never
 taken for a new instance. `vendor-data` carries the guest-agent
 bootstrap. The ISO is staged in a temp directory under `/tmp` (the
 manager's 64Mi emptyDir), assembled in memory (tens of KiB) and uploaded
 as a raw volume; the staging directory is removed before the call
 returns. The seed is rebuilt on every `Create` attempt, because a
 previous attempt may have been cut off mid-upload and a truncated seed
 would boot a VM without cloud-init.
 The guest agent rides along in `vendor-data` as a **shell script**
 (`guestAgentScript` in `seed.go`), not a cloud-config, on purpose:
 cloud-init merges a vendor-data cloud-config under the user-data with
 mergers that keep the user-data's value for every top-level key both
 define and never append lists, so a Proxy carrying its own `packages:`
 would silently lose the agent. Scripts bypass the merge. The script is
 idempotent, picks whichever of `apt-get`, `dnf`, `yum`, `zypper` or `apk`
 the image has, enables the service with `systemctl` or OpenRC, and logs
 into `/var/log/cloud-init-output.log`. With `vendorData` also configured,
 the two travel as separate parts of a MIME multipart (`text/plain` for
 your document, `text/x-shellscript` for the script), with the
 `MIME-Version` header first so cloud-init detects it. With
 `installGuestAgent: false` the file is your `vendorData` alone, or empty.
 ## The image cache
 The base image is downloaded **by the operator, through libvirt**, into
 the host's pool the first time a Proxy asks for it on that host. Nothing
 is pre-placed on the hypervisor and the operator needs no filesystem
 access there. The bytes stream from the URL through the operator into
 `StorageVolUpload` with a running sha256; nothing is buffered
 (`fetchImage` and `uploadImage` in `internal/provider/kvm2/image.go`).
 The upload runs on a **dedicated connection** to the host, dialled for
 the transfer and closed after it, not on the pooled connection the other
 operations share. A cancelled or overlong call kills the connection it
 ran on (see *Timeouts and connections*), and a poll for some other Proxy
 being cancelled must not cut off a 20-minute upload.
 The download needs a `Content-Length`: the volume's capacity is declared
 before the upload starts. The request sends `Accept-Encoding: identity`
 so a transparently gzipped body cannot hide the length. A response
 without it, or with length 0, is a permanent error naming the URL, and so
 is any 4xx status; 429 and 5xx are transient and retried.
 The marker volume is written **last**, on the pooled connection, after
 `StoragePoolRefresh` has made libvirt read the file's real format. Its
 presence is what marks a base complete: a crash mid-download leaves a
 base volume without a marker, which the next attempt deletes and redoes
 (`cachedImage`). A re-download also happens when the marker's content is
 not a 64-character hex digest, or when `sha256` is configured and differs
 from it. Otherwise the cached copy is used for as long as it exists, even
 if the URL's content changed upstream: a `latest` URL is pinned at the
 moment of first fetch, per host. Concurrent Creates for the same image
 on the same host queue on a per-(host, volume) lock; the first fetches,
 the rest find the marker.
 To evict manually:
 ```sh
 ssh novakj@beelink virsh -c qemu:///system vol-delete --pool default base-debian-13.qcow2.sha256
 ssh novakj@beelink virsh -c qemu:///system vol-delete --pool default base-debian-13.qcow2
 ```
 **Warning: do not change an image entry's `url` or `sha256`, and do not
 evict its base volume, while Proxies using that image exist on the
 host.** The overlays back onto the base file; deleting or replacing it
 corrupts every VM that uses it. Delete those Proxies first, or add the
 new image under a new key and move Proxies over by changing their
 `placement.image` (which replaces the VM).
 ## Quota: the pool-space precheck
 Before creating any volume, `Create` reads `StoragePoolGetInfo` and
 compares the pool's free space with what it is about to commit
 (`checkPoolSpace` and `neededBytes` in `image.go`): the image's
 `Content-Length` when the base is not cached yet (found with a `HEAD`
 request, or a `GET` whose body is closed at once when the server refuses
 `HEAD`), plus `diskSizeGb` in bytes for the overlay, plus 16 MiB for the
 seed. Below that it returns `ErrQuotaExceeded` with a message naming the
 pool, the bytes available and the bytes needed. The reconciler answers
 quota with a wait, not a failure: the Proxy shows `Provisioned=False`
 with reason `QuotaExceeded` and is retried after `QuotaRetry` (5
 minutes), off the error backoff curve and never `Failed`. Free up the
 pool or lower `diskSizeGb` and the next attempt goes through.
 The precheck is the only quota signal; libvirt itself reports a full pool
 as an ordinary error, which would be transient. The health check exposes
 the same number as `poolAvailableGiB`, so a pool that is filling up shows
 there before Creates start waiting.
 A pool that is not running fails the precheck permanently (`storage pool
 <pool> is not running`): start it with `virsh pool-start` and
 `virsh pool-autostart`.
 ## Migrating from `kvm`
 The `kvm2` block is the `kvm` block under another name, so an entry ports
 by changing `type: kvm` to `type: kvm2` and `kvm:` to `kvm2:`. The
 `kvm-ssh` Secret is shared; nothing changes on the hosts.
 What does not port is the VMs. kvm2 marks its domains in its own metadata
 namespace and neither lists nor adopts domains the `kvm` provider made:
 `ListByTag` skips them, so orphan GC never sees them, and a Proxy's
 `spec.provider` is immutable, so an existing Proxy cannot be moved to the
 new entry anyway. Do it in this order:
 1. Delete the Proxies on the `kvm` entry and wait for them to go; the
   `kvm` provider removes their domains and volumes.
 2. Change the entry (or add a second one under a new name and keep the
   old one for a while; both types may share a host).
 3. Restart the manager, then recreate the Proxies with the new
   `spec.provider`.
 A `kvm` domain left behind on a host is invisible to kvm2 forever; clean
 it up on the host with `virsh undefine --nvram <name>` and
 `virsh vol-delete` for its `<name>.qcow2` and `<name>-seed.iso`.
 ## Logging
 The provider logs through controller-runtime's logger, named `kvm2`, with
 `provider` (the configured name) on every line, and `host` and `name`
 (the domain) on every line about an instance. Every line inside a traced
 operation also carries `traceID` and `spanID`.
 - `--zap-log-level=debug` (logr V(1)):
  - one line per operation outcome: `kvm2 domain ensured running` with the
    `id`, `kvm2 domain fetched` with `state`, `ip` and `uid`,
    `kvm2 domain not found`, `kvm2 domain deleted`, `kvm2 domains listed`
    with `hosts` and `managed` counts, `kvm2 host checked` with `ready`,
    `reason` and `latency`; plus the steps inside a Create
    (`kvm2 domain already defined`, `overlay volume created`,
    `seed volume uploaded` with `bytes`, `domain defined` with `uid` and
    `firmware`, `domain started`, `fetching base image into host pool`,
    `base image cached`);
  - **one `libvirt call` line per libvirt RPC**, written by the decorator
    in `internal/provider/kvm2/observe.go` inside that RPC's span: `rpc`
    (the go-libvirt method name, for example `DomainLookupByName`), the
    identifiers it was given (`domain`, `pool`, `volume`), `duration`, and
    on failure `error` plus `libvirtCode` and `libvirtError` (the code's
    name, for example `ErrNoDomain`). A Get is four or five RPCs; a
    Create with a cached image is around twenty;
  - connection events from `internal/provider/kvm2/hosts.go`:
    `libvirt host connected` with `address`, `user` and `duration`, and
    `libvirt host connect failed` with the same plus `error` and `stage`
    (`credentials`, `tcp`, `ssh` or `socket`);
  - `kvm2 backend call failed` with `op` and the raw `error` once per
    failed operation, before the error is returned;
  - a line per domain that `ListByTag` skips, and a line when the image
    marker forces a refetch.
 - `--zap-log-level=2`: the `libvirt call` lines also carry the domain and
  volume `xml`, the `metadata` element, the interface `addresses`, the
  listed `domains`, the pool `state` and sizes and the volume `path`; the
  `domain defined` line carries the `xml`; `kvm2 create resolved` logs the
  resolved parameters (pool, bridge, machine type, sizes, image volume,
  firmware) and `cloudInitBytes`; `kvm2 listed instance` logs each listed
  domain.
 - Never logged at any level: the cloud-init content, the SSH key, the
  known_hosts content, and upload or download bodies. The seed and image
  uploads report `offset` and `length` only. There is no raw wire tier
  like the GCP provider's V(5); go-libvirt exposes no hook for it.
 ## Tracing
 With an OTLP exporter configured ([architecture.md
 §10](architecture.md#10-tracing-internaltracing)), every backend call is
 a child span of the operation span the decorators open
 (`provider.create`, `provider.get`, `provider.delete`, `provider.list`)
 or of the monitor's `providerhealth.check`:
 | Span | Around | Attributes |
 |---|---|---|
 | `kvm2.<Method>`, for example `kvm2.DomainDefineXML` | one libvirt RPC (`observe.go`) | `kvm2.host`, and the identifiers passed: `kvm2.domain`, `kvm2.pool`, `kvm2.volume`; on failure the status is Error and `kvm2.libvirt_code` carries the libvirt error number |
 | `kvm2.dial` | the SSH handshake and the libvirt connect (`hosts.go`) | `kvm2.host`, `kvm2.address`, `kvm2.user` |
 | `kvm2.download` | one image fetch, from the HTTP request to the marker (`image.go`) | `kvm2.host`, `kvm2.volume`, `kvm2.url_host`, `kvm2.sha256_configured`, and `kvm2.bytes` once the response arrived |
 All spans are client spans. No attribute carries cloud-init, key material
 or a URL beyond its host. The `libvirt call` log line for an RPC is
 written inside that RPC's span, so its `spanID` points at the span.
 ## Timeouts and connections
 No caller puts a deadline on the context, so every backend call carries
 its own budget (`defaultBudgets` in `internal/provider/kvm2/backend.go`,
 applied by `call`):
 | Call | Budget |
 |---|---|
 | TCP connect plus SSH handshake plus libvirt connect | 10 s |
 | ordinary RPC | 30 s |
 | `DomainCreate` | 90 s |
 | `StorageVolUpload` of at most 16 MiB (the seed, the marker) | 60 s |
 | `StorageVolUpload` of an image, `StorageVolDownload`, and the whole image fetch from HTTP request to marker | 20 m; the caller's ctx may end it earlier |
 | wait for running after `DomainCreate` | 2 m, polled every 500 ms (`defaultRunningWait` and `defaultRunningPoll` in `create.go`) |
 | the `HEAD` request of the pool-space precheck | 30 s |
 | the courtesy `ConnectClose` when a connection is closed | 5 s |
 go-libvirt's calls take no context and block, so the only way to abandon
 one is to close the connection under it. That is what a cancelled or
 overlong call does: the adapter kills the connection (socket and SSH
 client), returns `ErrTransient` wrapping the context or deadline error,
 and marks the connection dead so the next call redials. The consequence:
 **other RPCs in flight on that host's pooled connection fail once with
 `ErrTransient`** and their callers retry on a fresh connection. In the
 reconciler that is one requeue with backoff; in GC it is one skipped
 sweep.
 Two callers are kept off the pooled connection for that reason
 (`dialFresh` in `hosts.go`): the image upload, because a 435 MB transfer
 must not die because a poll for another Proxy was cancelled, and the
 health check, because a check that runs out of the monitor's timeout must
 close only its own connection.
 Otherwise there is one pooled SSH plus libvirt connection per host,
 dialled lazily on the first call that needs it and shared by every
 operation for that host (go-libvirt multiplexes RPCs on one connection).
 Concurrent callers wait for one dial and share the result; waiting ends
 with the caller's context. A `keepalive@openssh.com` request every 15 s
 marks the connection dead when it fails or gets no reply within another
 15 s, as does go-libvirt noticing the socket is gone; the next call
 redials. Create and Delete for the same domain name are serialised on a
 per-name lock, so two Deletes (GC and the reconciler) never race on
 destroy and undefine, and a Delete cannot interleave with a Create's
 steps; Get and ListByTag take no lock.
 ## Health check
 The operator checks every configured host every `--provider-check-interval`
 (default 1m) and publishes the result in the `ProviderHealth` object named
 after the provider (`kubectl get ph kvm2-home -o yaml`) and in the
 `proxy_operator_provider_target_up{provider,target}` gauge. One target per
 `hosts[]` entry, in config order, all checked concurrently.
 Each check dials a **fresh connection** to the host and closes it when
 done (`probeHost` in `internal/provider/kvm2/check.go`); the pooled
 connection is never touched, so a check that runs out of time cannot fail
 a reconcile's call in flight, and a fresh dial picks up a rotated Secret
 at once. The sequence is the SSH and libvirt handshakes, then four RPCs:
 `ConnectGetLibVersion`, `ConnectGetHostname`, `StoragePoolLookupByName`
 and `StoragePoolGetInfo` on the host's `pool`. The cost per host per
 round is one sshd session and those four calls.
 Every target's `details` carry `user` and `pool`. As facts arrive they
 gain `libvirtVersion` (decoded `major.minor.micro`), `hostname`,
 `poolState` (virsh's word: `running`, `inactive`, ...),
 `poolCapacityGiB` and `poolAvailableGiB`, so a target that fails late in
 the sequence still says what the host answered before. The free space is
 the number the quota precheck compares against.
 A failing target says where the check broke (`checkReason` in `check.go`):
 | Reason | Meaning | Look at |
 |---|---|---|
 | `Reachable` | ready | |
 | `CredentialsUnavailable` | the key or known_hosts file is missing or does not parse | the `kvm-ssh` Secret (see *Cluster-side setup*); fixed in the cluster, not on the host |
 | `Unreachable` | the TCP connect failed or ran out of the 10 s dial budget, the SSH transport failed for a reason other than authentication, the forwarded libvirt socket was refused, or the link died mid-check | DNS, Tailscale or a firewall between the cluster and `address`; `libvirtd` running and the `socket` path |
 | `Unauthorized` | the SSH server rejected the key, or the host key is missing from known_hosts or has changed | `user`'s `authorized_keys`; the known_hosts entry (re-run `ssh-keyscan` and update the Secret) |
 | `Forbidden` | SSH worked, but libvirt refused the user (`ErrAccessDenied` or `ErrAuthFailed`) | `user` in the `libvirt` group |
 | `Timeout` | the connection was up, but the monitor's `--provider-check-timeout` (default 15s) or an RPC's budget ran out before the check finished | a hung `libvirtd` or a saturated link |
 | `PoolNotFound` | there is no pool named `pool` | `virsh pool-list --all` |
 | `PoolNotActive` | the pool exists but is not running | `virsh pool-start <pool>` and `virsh pool-autostart <pool>` |
 | `APIError` | any other libvirt error | the message carries libvirt's own text |
 ## Error classes
 Every error is classified once, in `classify` in
 `internal/provider/kvm2/errors.go`, with errors that carry a class from
 their source (validation, the image fetch, the dial) keeping it. The
 reconciler latches `ErrPermanent` as `Failed` until the spec changes,
 waits 5 minutes on `ErrQuotaExceeded`, recreates on `ErrNotFound` from
 `Get`, and retries everything else with backoff.
 | Class | What lands there |
 |---|---|
 | `ErrPermanent` | placement values not in the catalogue; a malformed providerID; a host no longer in the config (from `Get`; `Delete` treats it as done); libvirt `ErrXMLError`, `ErrNoSupport`, `ErrInvalidArg`, `ErrAuthFailed`, `ErrAccessDenied` and `ErrNoStoragePool`; a pool that is not running; SSH authentication refused (`unable to authenticate`); a host key missing from known_hosts or changed (`knownhosts.KeyError`); a key or known_hosts file that exists but does not parse; a domain of the same name owned by another UID; corrupt metadata on a domain; an image URL answering 4xx, a response without `Content-Length` or with length 0, a sha256 that does not match the configured one |
 | `ErrQuotaExceeded` | the pool-space precheck |
 | `ErrNotFound` | libvirt `ErrNoDomain` or `ErrNoStorageVol` from `Get`, `Delete` or `ListByTag`; inside `Create` the same codes are `ErrTransient`, because the contract forbids NotFound from Create |
 | `ErrTransient` | everything else: a missing key or known_hosts file (the Secret has not landed yet), TCP and network errors, context cancellation, an abandoned or budget-exceeded call, a connection killed by another caller, libvirt `ErrOperationInvalid` and every unrecognised libvirt code, an image URL answering 429 or 5xx or ending the body early, the wait for running expiring |
 Error strings name paths, hosts, volumes and libvirt's own text. They
 never contain key bytes, the known_hosts content or cloud-init.
 ## Known gaps
 - **Create blocks**, including the first image download on each host,
  holding one of the three reconcile workers for its duration (LAN
  speed, a few hundred MB, once per image per host). Other Proxies keep
  reconciling on the remaining workers. Warm the cache with one Proxy per
  image per host before creating many.
 - **Volumes without a domain are invisible to GC.** A crash between
  creating the overlay or seed and defining the domain leaves volumes
  that `ListByTag` cannot see (it lists domains). The same Proxy's next
  Create reuses the overlay and rebuilds the seed, and its Delete removes
  both by name even with no domain, but a Proxy CR that is gone for good
  leaves them for `virsh vol-list` to find.
 - **A host removed from the config orphans its VMs.** `Delete` for a
  providerID naming an unconfigured host is a logged no-op success, so
  the Proxy finishes deleting instead of pinning its finalizer, but the
  domain and its volumes stay on the hypervisor and orphan GC cannot see
  them either (it only sweeps configured hosts). Delete the Proxies before
  retiring a host, or clean up afterwards with `virsh undefine --nvram`
  plus `virsh vol-delete`. `Get` fails permanently for such a providerID
  on purpose: a typo in `hosts` must not look like a missing VM and
  trigger a rebuild elsewhere.
 - **No adoption of `kvm` domains.** See *Migrating from kvm*.
 - **A cancelled call drops the host's pooled connection once**, so other
  RPCs in flight on that host fail once with `ErrTransient` and retry.
  The image upload and the health check are isolated on their own
  connections.
 - **`Content-Length` is required** for image downloads; a mirror that
  only chunks is rejected permanently. Point the catalogue at one that
  sends the length.
 - **A paused or pmsuspended domain cannot be adopted.** A repeat `Create`
  for a domain in that state calls `DomainCreate`, which libvirt refuses
  as "already running"; the provider then waits for the domain to report
  running, the 2-minute wait expires, and the call fails transiently on
  every retry. Resume the domain on the host (`virsh resume <name>`) or
  delete the Proxy.
 - **Parser text in credential errors.** A key or known_hosts file that
  does not parse fails with the parser's own message, which can quote a
  PEM header type or a malformed known_hosts token. Key bytes are never
  quoted.
 - **The sample cloud-init is an open proxy.**
  `config/samples/proxy_kvm2.yaml` installs Squid with
  `http_access allow all`; restrict it before exposing the bridge network
  to anything but the cluster.
 - **IP discovery needs the agent or guest traffic to the host.** A VM on
  a bridge with no DHCP never gets an address and stays `Provisioning`
  forever; check `virsh console <name>` on the host. The same symptom
  with a booted, addressed VM means the guest agent never came up: check
  `systemctl status qemu-guest-agent`, `cloud-init status --long` and the
  vendor script's output in `/var/log/cloud-init-output.log` (a VM without
  outbound internet cannot install it). A user-data that disables
  vendor-data (`vendor_data: {enabled: false}`) also removes the agent, by
  design. `virsh domifaddr <name> --source agent` and `--source arp` show
  what the operator sees, in that order.
 - **Debian 13 cloud images hang in GRUB under BIOS on a headless VM**
  ([Debian bug #1111240](https://bugs.debian.org/1111240)), which is why
  `firmware` defaults to `uefi`; the same image boots under UEFI with the
  serial console intact. Images that need `firmware: bios` are the
  exception.
 ## Testing
 Unit tests run over an in-memory fake of the libvirt seam and an
 in-process SSH server. The optional integration test against a real host
 is described in
 [testing.md](testing.md#the-kvm2-integration-test-real-hypervisor); run it
 with `make test-integration-kvm2`. `TestKVM2Check_realHost` runs the
 health check against the same host and creates nothing:
 `go test -tags=integration -count=1 -v -run TestKVM2Check ./internal/provider/kvm2/`.
@@ -11,7 +11,7 @@ Plan: [docs/plans/2026-09-16-1333-kvm2-provider.md](../plans/2026-09-16-1333-kvm
 - [x] Step 6 — The four contract methods
 - [x] Step 7 — Checker and the CRD comment
 - [x] Step 8 — Deployment files, Makefile, integration test
- [ ] Step 9 — Documentation
+- [x] Step 9 — Documentation
 - [ ] Step 10 — Milestone review
 - [ ] Step 11 — Real-host validation, changelog, MR
@@ -318,3 +318,23 @@ make -n test-integration-kvm2
 ```
 All clean. Attestation: "files read outside the allowed set: none".
 ## Step 9 — Documentation
 2026-09-16 19:54. Eighth subagent step: `docs/kvm2-provider.md` (sixteen
 sections modelled on the kvm guide), a `kvm2` column in the three tables of
 `docs/providers-existing.md` plus a kvm2 section and renumbering, the
 components row, §5 block, §10 span bullet, §11 row and eight Decisions
 bullets in `docs/architecture.md`, a README row and paragraph, and the
 guide in the `docs/providers.md` index. The agent verified each claim
 against the package rather than the spec and dropped a draft sentence
 claiming kvm and kvm2 share the base-image cache, because checking it would
 have meant reading v1's volume names.
 Checks re-run by the orchestrator: no em-dash in the guide, no line
 references, each of the 19 axis codes appears once in the appendix table
 with seven cells, every link and anchor across the seven touched documents
 resolves. The one added em-dash is the `Leader-elected` cell of the new
 components-table row, where the table's own convention wins.
 Attestation: "files read outside the allowed set: none".
@@ -1,13 +1,17 @@
 # The existing providers
 This is the appendix to [provider-contract.md](provider-contract.md): how the
-three providers that exist today (kubernetes, gcp, kvm) meet the contract,
+four providers that exist today (kubernetes, gcp, kvm, kvm2) meet the
-which profile each one chose on every axis, and what is still missing. It is
+contract, which profile each one chose on every axis, and what is still
-reference material for whoever writes the next provider, not a user guide;
+missing. It is reference material for whoever writes the next provider, not
-the user guides are [kvm-provider.md](kvm-provider.md),
+a user guide; the user guides are [kvm-provider.md](kvm-provider.md),
 [kvm2-provider.md](kvm2-provider.md),
 [gcp-vm-validation.md](gcp-vm-validation.md) and
 [gcp-in-specific-project.md](gcp-in-specific-project.md). The README's
-"Implemented providers" table stays the short public summary.
+"Implemented providers" table stays the short public summary. kvm2 is the
 one provider built from the contract itself (clean-room, from
 [kvm2-provider-spec.md](kvm2-provider-spec.md), without reading kvm's code),
 so its column is also a record of what the contract produced.
 The per-provider call flows are drawn in
 [architecture.md §5](architecture.md#5-what-provider-calls-do-in-the-outside-world),
@@ -23,59 +27,59 @@ Code references are a file path plus a function or symbol name.
 One row per axis of contract §4, in the same order, with the profile name
 and the symbol that shows it.
-| Axis | kubernetes | gcp | kvm |
+| Axis | kubernetes | gcp | kvm | kvm2 |
-|---|---|---|---|
+|---|---|---|---|---|
-| `readiness` | immediate: a Pod is scheduled within seconds (`stateFromPod`) | provisioning-window: minutes of boot (`mapState`) | provisioning-window: boot plus, on first use, an image download (`imageCache.ensure`) |
+| `readiness` | immediate: a Pod is scheduled within seconds (`stateFromPod`) | provisioning-window: minutes of boot (`mapState`) | provisioning-window: boot plus, on first use, an image download (`imageCache.ensure`) | provisioning-window: boot plus, on first use, an image download (`ensureImage`) |
-| `create-return` | submit: returns after `client.Create` | submit: `Insert` is never awaited, `Operation.Wait` is not called | **blocking**: waits through `DomainCreate` (`ensureRunning`), and the first image fetch runs inside (`imageCache.fetch`) |
+| `create-return` | submit: returns after `client.Create` | submit: `Insert` is never awaited, `Operation.Wait` is not called | **blocking**: waits through `DomainCreate` (`ensureRunning`), and the first image fetch runs inside (`imageCache.fetch`) | **blocking**: waits through `DomainCreate` and a 2-minute poll for running (`ensureRunning`, `waitRunning`), and the first image fetch runs inside (`fetchImage`) |
-| `delete-return` | returns-early: `client.Delete`, Pod terminates on its own | returns-early: `Delete` submits an operation each time | waits: destroy, undefine and volume removal complete before return (`removeDomain`, `removeVolumes`) |
+| `delete-return` | returns-early: `client.Delete`, Pod terminates on its own | returns-early: `Delete` submits an operation each time | waits: destroy, undefine and volume removal complete before return (`removeDomain`, `removeVolumes`) | waits: destroy, undefine and volume removal complete before return (`removeDomain`, `removeVolumes`) |
-| `ip-source` | assigned-at-create: `PodIP` (`instanceFromPod`) | assigned-at-create: the external NAT IP of the first NIC (`toInstance`) | discovered: ARP table first, then the guest agent, first routable IPv4 (`address`, `firstIPv4`) |
+| `ip-source` | assigned-at-create: `PodIP` (`instanceFromPod`) | assigned-at-create: the external NAT IP of the first NIC (`toInstance`) | discovered: ARP table first, then the guest agent, first routable IPv4 (`address`, `firstIPv4`) | discovered: the guest agent first, then the ARP table, first routable IPv4 (`addressSources`, `address`, `firstIPv4`) |
-| `gc-mark` | native-labels: Pod labels (`buildPod`) | native-labels: instance labels (`buildInsertRequest`) | written-metadata: a namespaced `<metadata>` element written in `DomainDefineXML` (`metadata.go`) |
+| `gc-mark` | native-labels: Pod labels (`buildPod`) | native-labels: instance labels (`buildInsertRequest`) | written-metadata: a namespaced `<metadata>` element written in `DomainDefineXML` (`metadata.go`) | written-metadata: an element in kvm2's own namespace written in `DomainDefineXML` (`metadataURI`, `newInstanceMetadata`) |
-| `created-at` | backend-timestamp: `CreationTimestamp` | backend-timestamp: `CreationTimestamp`, parse error gives zero | written-at-create: `createdAt` in the metadata element (`newInstanceMetadata`) |
+| `created-at` | backend-timestamp: `CreationTimestamp` | backend-timestamp: `CreationTimestamp`, parse error gives zero | written-at-create: `createdAt` in the metadata element (`newInstanceMetadata`) | written-at-create: `createdAt` in the metadata element, parse error gives zero (`newInstanceMetadata`, `createdAt`) |
-| `list-partial` | n/a: one call, all namespaces | omit-unreachable: `ReturnPartialSuccess` on `AggregatedList` | fail-whole per host, one unreadable domain skipped (`ListByTag`, `managedInstance`) |
+| `list-partial` | n/a: one call, all namespaces | omit-unreachable: `ReturnPartialSuccess` on `AggregatedList` | fail-whole per host, one unreadable domain skipped (`ListByTag`, `managedInstance`) | fail-whole per host, one unreadable domain skipped unless ctx ended (`ListByTag`) |
-| `quota` | absent: a quota 403 is Permanent (`classify`) | detected: 429, and 403 with reason `quotaExceeded` or `rateLimitExceeded` (`classify`, `hasReason`) | absent (`classify`) |
+| `quota` | absent: a quota 403 is Permanent (`classify`) | detected: 429, and 403 with reason `quotaExceeded` or `rateLimitExceeded` (`classify`, `hasReason`) | absent (`classify`) | detected: a pool-space precheck before any volume is created (`checkPoolSpace`, `neededBytes`) |
-| `not-found` | status-404: `apierrors.IsNotFound` | status-404: `isNotFound` | error-code: `ErrNoDomain`, `ErrNoStorageVol` (`hasCode`) |
+| `not-found` | status-404: `apierrors.IsNotFound` | status-404: `isNotFound` | error-code: `ErrNoDomain`, `ErrNoStorageVol` (`hasCode`) | error-code: `ErrNoDomain`, `ErrNoStorageVol`, Transient inside Create (`classifyLibvirt`) |
-| `primary-key` | name-is-key: `namespace/name` | name-is-key: `zones/<zone>/instances/<name>` | name-is-key: `hosts/<host>/domains/<name>` |
+| `primary-key` | name-is-key: `namespace/name` | name-is-key: `zones/<zone>/instances/<name>` | name-is-key: `hosts/<host>/domains/<name>` | name-is-key: `hosts/<host>/domains/<name>`, with a UID guard on adoption (`create`) |
-| `provisioning-state` | native: `PodPending` | native: `PROVISIONING`, `STAGING`, `REPAIRING` | create-blocks: a domain is defined and started in one `Create`; `nostate` maps to Provisioning |
+| `provisioning-state` | native: `PodPending` | native: `PROVISIONING`, `STAGING`, `REPAIRING` | create-blocks: a domain is defined and started in one `Create`; `nostate` maps to Provisioning | create-blocks: defined, started and polled to running in one `Create`; `nostate` maps to Provisioning (`mapState`) |
-| `async-ops` | none | task-handle, but the handle is discarded: an insert that fails after acceptance is never read | none |
+| `async-ops` | none | task-handle, but the handle is discarded: an insert that fails after acceptance is never read | none | none |
-| `secondary-resources` | none | cascade: `AutoDelete` on the boot disk (`buildInsertRequest`) | explicit: overlay and seed removed by name (`removeVolumes`); base image kept |
+| `secondary-resources` | none | cascade: `AutoDelete` on the boot disk (`buildInsertRequest`) | explicit: overlay and seed removed by name (`removeVolumes`); base image kept | explicit: overlay and seed removed by name (`removeVolumes`); base image kept |
-| `credentials` | workload-identity: in-cluster config or kubeconfig (`NewWithTransportWrapper`) | environment: Application Default Credentials | mounted-files: SSH key and known_hosts from the optional `kvm-ssh` Secret, read lazily (`newLazySSH`) |
+| `credentials` | workload-identity: in-cluster config or kubeconfig (`NewWithTransportWrapper`) | environment: Application Default Credentials | mounted-files: SSH key and known_hosts from the optional `kvm-ssh` Secret, read lazily (`newLazySSH`) | mounted-files: the same `kvm-ssh` Secret, read on every dial and never cached (`loadCredentials`) |
-| `client-kind` | vendor-sdk: controller-runtime `client.Client`, no seam interface | vendor-sdk: `cloud.google.com/go/compute` behind `instancesAPI` | rpc-over-ssh: `go-libvirt` over an SSH-forwarded unix socket, behind `libvirtAPI` (`conn.go`) |
+| `client-kind` | vendor-sdk: controller-runtime `client.Client`, no seam interface | vendor-sdk: `cloud.google.com/go/compute` behind `instancesAPI` | rpc-over-ssh: `go-libvirt` over an SSH-forwarded unix socket, behind `libvirtAPI` (`conn.go`) | rpc-over-ssh: `go-libvirt` over an SSH-forwarded unix socket, behind the ctx-first `backend` seam (`backend.go`) |
-| `timeouts` | ctx-only: no `rest.Config.Timeout` is set | ctx-only: SDK defaults, with SDK retries on Get and List | per-call for the dial only (`defaultHandshakeTimeout` in `dialSSH`); RPCs are ctx-only and a cancelled `call` closes the connection |
+| `timeouts` | ctx-only: no `rest.Config.Timeout` is set | ctx-only: SDK defaults, with SDK retries on Get and List | per-call for the dial only (`defaultHandshakeTimeout` in `dialSSH`); RPCs are ctx-only and a cancelled `call` closes the connection | per-call: a budget around the dial, every RPC, the uploads and the download (`defaultBudgets`, `call`); a cancelled or overlong call kills the pooled connection |
-| `logging` | none | tiered: V(1) outcomes and `logAPIError`, V(2) detail, V(5) wire log (`wirelog.go`) | tiered: V(1) outcomes and one `libvirt call` line per RPC (`trace.go`), V(2) XML and addresses; no wire tier |
+| `logging` | none | tiered: V(1) outcomes and `logAPIError`, V(2) detail, V(5) wire log (`wirelog.go`) | tiered: V(1) outcomes and one `libvirt call` line per RPC (`trace.go`), V(2) XML and addresses; no wire tier | tiered: V(1) outcomes and one `libvirt call` line per RPC (`observe.go`), V(2) XML and addresses; no wire tier |
-| `child-spans` | transport-wrapper: `tracing.RestConfigWrapper()` on its own `rest.Config`, parent-gated | sdk-instrumentation: the SDK adds `otelhttp` itself; not parent-gated, but every call site has a parent | **none**: `trace.go` is a log decorator, not spans |
+| `child-spans` | transport-wrapper: `tracing.RestConfigWrapper()` on its own `rest.Config`, parent-gated | sdk-instrumentation: the SDK adds `otelhttp` itself; not parent-gated, but every call site has a parent | **none**: `trace.go` is a log decorator, not spans | span-decorator: `kvm2.<Method>` per RPC (`observe`), plus `kvm2.dial` (`libvirtDialer`) and `kvm2.download` (`fetchImage`) |
-| `backend-test` | cluster-e2e: kind (`test/e2e`) | manual-runbook: [gcp-vm-validation.md](gcp-vm-validation.md) | integration-tag: `kvm_integration_test.go`, `make test-integration-kvm` |
+| `backend-test` | cluster-e2e: kind (`test/e2e`) | manual-runbook: [gcp-vm-validation.md](gcp-vm-validation.md) | integration-tag: `kvm_integration_test.go`, `make test-integration-kvm` | integration-tag: `kvm2_integration_test.go`, `make test-integration-kvm2` |
 ## 2. Other differences
 Rows that are not profile axes but that a new provider author compares
 against.
-| Feature | kubernetes | gcp | kvm |
+| Feature | kubernetes | gcp | kvm | kvm2 |
-|---|---|---|---|
+|---|---|---|---|---|
-| Create verifies the adopted instance's UID | no | no | no |
+| Create verifies the adopted instance's UID | no | no | no | **yes**: a different non-empty UID in the metadata is Permanent, an empty one is adopted (`create`) |
-| Placement used | none | Zone, MachineType, Image | Zone (host), MachineType, Image |
+| Placement used | none | Zone, MachineType, Image | Zone (host), MachineType, Image | Zone (host), MachineType, Image |
-| CloudInit delivered | **no**: ignored | yes: `user-data` metadata | yes: NoCloud seed ISO built in-process (`buildSeedISO`) |
+| CloudInit delivered | **no**: ignored | yes: `user-data` metadata | yes: NoCloud seed ISO built in-process (`buildSeedISO`) | yes: NoCloud seed ISO built in-process (`buildSeedISO`) |
-| Port used | yes: Squid config and containerPort (`squidConf`) | no: firewall is manual | no |
+| Port used | yes: Squid config and containerPort (`squidConf`) | no: firewall is manual | no | no |
-| States produced | Provisioning, Running, Terminated | all four | Provisioning, Running, Stopped |
+| States produced | Provisioning, Running, Terminated | all four | Provisioning, Running, Stopped | Provisioning, Running, Stopped |
-| Deleting instance still reports its state | yes: a Pod being deleted reports its phase | n/a | n/a |
+| Deleting instance still reports its state | yes: a Pod being deleted reports its phase | n/a | n/a | n/a: Delete waits, and a concurrent Get sees the domain until it is undefined |
-| Error.Provider is the configured name | **no**: hardcoded `"kubernetes"` | yes | yes |
+| Error.Provider is the configured name | **no**: hardcoded `"kubernetes"` | yes | yes | yes (`wrapErr`) |
-| Checker target | `apiserver` | the project | each host, concurrently |
+| Checker target | `apiserver` | the project | each host, concurrently | each host, concurrently, on a fresh connection |
-| Checker Details | none | none | `user`, `pool`, `hostname`, `libvirtVersion`, `poolState`, `poolCapacity`, `poolAvailable` |
+| Checker Details | none | none | `user`, `pool`, `hostname`, `libvirtVersion`, `poolState`, `poolCapacity`, `poolAvailable` | `user`, `pool`, `hostname`, `libvirtVersion`, `poolState`, `poolCapacityGiB`, `poolAvailableGiB`, filled in as far as the check got |
-| Credential rotation without restart | n/a | via ADC | no: cached after first successful load |
+| Credential rotation without restart | n/a | via ADC | no: cached after first successful load | yes: both files read on every dial (`loadCredentials`) |
-| Connection handling | client-go | SDK client, never closed | one pooled SSH plus libvirt connection per host, 15s keepalive (`hostPool`) |
+| Connection handling | client-go | SDK client, never closed | one pooled SSH plus libvirt connection per host, 15s keepalive (`hostPool`) | one pooled SSH plus libvirt connection per host, 15s keepalive (`hostTable`); dedicated connections for the image upload and the check (`dialFresh`) |
-| Unit-test double | controller-runtime fake client | `fakeAPI` behind `instancesAPI`, plus `httptest` for the real SDK client in `check_test.go` | `fakeAPI` behind `libvirtAPI`, plus an in-process `x/crypto/ssh` server in `conn_test.go` |
+| Unit-test double | controller-runtime fake client | `fakeAPI` behind `instancesAPI`, plus `httptest` for the real SDK client in `check_test.go` | `fakeAPI` behind `libvirtAPI`, plus an in-process `x/crypto/ssh` server in `conn_test.go` | `fakeBackend` behind `backend` (`fake_test.go`), plus an in-process `x/crypto/ssh` server (`sshserver_test.go`) and go-libvirt's own mock for the adapter |
-| Proxy software | generated Squid config (`pod.go`) | from `cloudInit`; the image must run cloud-init | from `cloudInit`, plus a vendor-data script that installs the guest agent (`seed.go`) |
+| Proxy software | generated Squid config (`pod.go`) | from `cloudInit`; the image must run cloud-init | from `cloudInit`, plus a vendor-data script that installs the guest agent (`seed.go`) | from `cloudInit`, plus a vendor-data script that installs the guest agent (`seed.go`) |
-| IP reachable from the operator | Pod network, so in-cluster only (not from `make run-dev` on a laptop) | external IP, once a firewall rule allows the cluster's egress IP | LAN address on the host's bridge; the operator needs a route to that LAN |
+| IP reachable from the operator | Pod network, so in-cluster only (not from `make run-dev` on a laptop) | external IP, once a firewall rule allows the cluster's egress IP | LAN address on the host's bridge; the operator needs a route to that LAN | LAN address on the host's bridge; the operator needs a route to that LAN |
-| Distinct egress IP per proxy | no: the cluster's egress | yes: ephemeral external IP | whatever the host's LAN routes through; not per VM by itself |
+| Distinct egress IP per proxy | no: the cluster's egress | yes: ephemeral external IP | whatever the host's LAN routes through; not per VM by itself | whatever the host's LAN routes through; not per VM by itself |
 ## 3. Observability status
-| Layer | kubernetes | gcp | kvm |
+| Layer | kubernetes | gcp | kvm | kvm2 |
-|---|---|---|---|
+|---|---|---|---|---|
-| 1. Operation spans and metrics | yes: decorators | yes: decorators | yes: decorators |
+| 1. Operation spans and metrics | yes: decorators | yes: decorators | yes: decorators | yes: decorators |
-| 2. Child spans per backend call | yes: `tracing.RestConfigWrapper()`, parent-gated (`cmd/main.go`, `NewWithTransportWrapper`) | yes: the SDK's own `otelhttp`; not parent-gated | **no** |
+| 2. Child spans per backend call | yes: `tracing.RestConfigWrapper()`, parent-gated (`cmd/main.go`, `NewWithTransportWrapper`) | yes: the SDK's own `otelhttp`; not parent-gated | **no** | yes: `kvm2.<Method>` from the `observe` decorator, `kvm2.dial`, `kvm2.download`; always under the caller's span, since every call carries the caller's ctx |
-| 3. Logs carry `traceID` and `spanID` | n/a: the package does not log | yes, including the V(5) wire log | yes: every `libvirt call` line at V(1) |
+| 3. Logs carry `traceID` and `spanID` | n/a: the package does not log | yes, including the V(5) wire log | yes: every `libvirt call` line at V(1) | yes: every `libvirt call` line at V(1), written inside its RPC's span |
-| Tested against a real collector | yes: the kind e2e checks `provider.create` and `provider.delete` in Tempo (`test/e2e/tracing_test.go`), operation spans only | no | no |
+| Tested against a real collector | yes: the kind e2e checks `provider.create` and `provider.delete` in Tempo (`test/e2e/tracing_test.go`), operation spans only | no | no | no; the integration test asserts the parent-child topology in an in-memory recorder (`assertLifecycleSpans`) |
 ## 4. kubernetes (`internal/provider/kubernetes`)
@@ -329,18 +333,205 @@ the author's view.
    tries ARP first; it says `host` and `name` are on every log line, but
    the check and list paths lack them.
-## 7. Gaps shared by the existing providers
+## 7. kvm2 (`internal/provider/kvm2`)
- **Adoption is not verified.** None of the three checks `LabelUID` when
+The full user guide is [kvm2-provider.md](kvm2-provider.md) and the design
-  `Create` finds an existing instance with the right name.
+it was built from is [kvm2-provider-spec.md](kvm2-provider-spec.md); this
- **The Create comment does not match kvm.** The interface comment in
+section is the author's view. Nothing in the package was taken from
-  `internal/provider/provider.go` says `Create` returns as soon as the
+`internal/provider/kvm`; where the two agree it is because both follow the
-  request is submitted; kvm blocks until the domain has started.
+same libvirt.
 - **Package and seam.** `kvm2.go` (`Provider`, `New`, `withDefaults`,
  `host`, `wrapErr`, the per-name lock), `create.go`, `get.go`,
  `delete.go`, `list.go` (the four methods), `backend.go` (the seam, the
  production adapter and the budgets), `ssh.go` (credentials, the forwarded
  socket, keepalive), `hosts.go` (the pooled table and the production
  dialer), `observe.go` (the span and log decorator), `image.go` (cache,
  download, quota precheck), `seed.go`, `xml.go`, `volume.go`,
  `metadata.go`, `id.go`, `state.go`, `errors.go`, `check.go`. The seam is
  `backend`: the twenty go-libvirt calls the provider uses, each with `ctx`
  first (go-libvirt's own methods take none and block), plus `Close`.
  `libvirtBackend` adapts `*libvirt.Libvirt` and runs every call through
  `call`, which applies the budget and the abandon policy. The `dialer`
  seam separates the SSH transport: `newProvider` injects one for tests,
  `newLibvirtDialer` is production. The `observe` decorator is applied
  once, in `libvirtDialer.dial`, so provider code never holds an
  undecorated backend.
 - **Config.** `ProviderConfig.KVM2`, the same `KVMConfig` type as kvm,
  validated by `validateKVMBlock` with block name `kvm2` so messages read
  `kvm2.hosts`. Defaults (`diskSizeGb` 10, socket, pool, image format and
  firmware) are filled by `withDefaults`. `New` never dials; it checks that
  the credential files exist and warns when one does not, or when
  `insecureIgnoreHostKey` is on.
 - **ID.** `formatProviderID(host, name)` gives `hosts/<host>/domains/<name>`;
  `parseProviderID` accepts exactly four segments with the literal
  `hosts` and `domains` and non-empty host and name, else `ErrPermanent`.
 - **Create.** `Create` validates `Zone` (`host`; optional with one host),
  `MachineType` (a catalogue key) and `Image` (`resolveImage`: a catalogue
  key or an http(s) URL with qcow2 and uefi defaults) before any dial, and
  logs the resolved parameters with `cloudInitBytes` at V(2). Then, under
  the per-name lock (`lockName`) and on the pooled connection
  (`hostTable.get`), `create`: `DomainLookupByName`; found means the
  adoption guard (a non-empty metadata UID that differs from `req.UID` is
  Permanent) and `ensureRunning`. Otherwise `StoragePoolLookupByName`,
  `imageSizeIfUncached` (marker lookup, then `HEAD` or a closed `GET`) and
  `checkPoolSpace` with `neededBytes` (image if uncached, plus
  `diskSizeGb`, plus 16 MiB), then `ensureImage` (`cachedImage` on the
  pooled connection, `fetchImage` streaming into `uploadImage` on a
  `dialFresh` connection, `StoragePoolRefresh`, `writeMarker` last),
  `ensureOverlay` (`ErrStorageVolExist` means reuse), `writeSeed` (delete,
  build, upload; always rebuilt), two `StorageVolGetPath`, `buildDomainXML`
  with file-type disks and the metadata element, `DomainDefineXML`,
  `ensureRunning` (`DomainGetState`; `DomainCreate` with
  `ErrOperationInvalid` as already running; `waitRunning` polls every
  500 ms for up to 2 minutes, Transient on expiry). `Create` returns only
  when the domain is running.
 - **Get and state.** `mapState(state, ip)`:
  | libvirt state | IP | Provider state |
  |---|---|---|
  | nostate | any | Provisioning |
  | running, blocked | empty | Provisioning |
  | running, blocked | set | Running |
  | paused, shutdown, shutoff, crashed, pmsuspended, anything else | any | Stopped |
  `get`: `DomainLookupByName` (only `ErrNoDomain` reaches NotFound),
  `readMetadata` (`DomainGetMetadata` with `metadataURI`;
  `ErrNoDomainMetadata` means unmanaged with an empty UID; an element that
  does not parse is Permanent), `inspect` (`DomainGetState`, then for an
  active domain `address`). `addressSources` asks the guest agent first,
  then ARP; a source that fails is skipped at V(2), only ctx ending is an
  error, so a cancelled Get is Transient and not a false Provisioning.
  `firstIPv4` drops loopback, link-local and unspecified addresses and an
  address whose type tag and literal disagree. `UID` and `CreatedAt` are
  read back on every call; an unparsable `createdAt` gives the zero time.
 - **Delete.** Under the per-name lock: `removeDomain` (`DomainDestroy`,
  tolerating not running and gone; `DomainUndefineFlags` with NVRAM,
  managed save and snapshot metadata) and `removeVolumes` (overlay and seed
  by name through `deleteVolumeIfPresent`; a missing pool is a no-op). A
  host no longer in the config is a logged no-op success. Nothing is left
  when nil is returned.
 - **ListByTag.** `ConnectListAllDomains` (active and inactive) on every
  host in config order; `readMetadata` and `managed` keep kvm2's own
  domains; `inspect` fills state and address. A domain whose metadata or
  state cannot be read is skipped at V(1), unless ctx ended, which fails
  the call. A host that cannot be dialled or listed fails the whole call.
 - **Errors.** `classify(op, err)`: an error already carrying a class keeps
  it; libvirt codes through `classifyLibvirt` (`ErrNoDomain` and
  `ErrNoStorageVol` NotFound, Transient when `op == "create"`;
  `ErrNoStoragePool`, `ErrXMLError`, `ErrNoSupport`, `ErrInvalidArg`,
  `ErrAuthFailed`, `ErrAccessDenied` Permanent; the rest Transient);
  `knownhosts.KeyError` and the `unable to authenticate` text from
  `x/crypto/ssh` Permanent; everything else Transient, including a missing
  credential file. `permanentf` and `transientf` classify at the source:
  placement, XML builders, a parse failure of the key or known_hosts, an
  image 4xx, a missing or zero `Content-Length`, a sha256 mismatch, a pool
  that is not running (Permanent); an image 429 or 5xx, a short body, the
  running wait expiring (Transient). `checkPoolSpace` is the only source of
  `ErrQuotaExceeded`. `logAPIError` writes `kvm2 backend call failed` with
  `libvirtErrorFields`.
 - **Check.** `Check` runs `checkHost` per host concurrently, each through
  `probeHost` on a `dialFresh` connection that is closed when the probe
  ends; the pooled table is never consulted. `ConnectGetLibVersion`,
  `ConnectGetHostname`, `StoragePoolLookupByName`, `StoragePoolGetInfo`;
  a pool that is not running is `poolNotActiveError`. Details are filled
  as facts arrive (`user` and `pool` always; `libvirtVersion`, `hostname`,
  `poolState`, `poolCapacityGiB`, `poolAvailableGiB`). `checkReason`: a
  `dialError` is judged by its stage (`credentials` gives
  `CredentialsUnavailable`; `ssh` with a `knownhosts.KeyError` or the auth
  text gives `Unauthorized`; any other stage, including a dial that ran
  out of time, gives `Unreachable`); then `context.DeadlineExceeded` or
  `Canceled` `Timeout`, `ErrAccessDenied` or `ErrAuthFailed` `Forbidden`,
  `ErrNoStoragePool` `PoolNotFound`, any other libvirt code `APIError`,
  anything else `Unreachable`; `PoolNotActive` from the pool state.
 - **Connections.** `hostTable` keeps one pooled connection per host,
  dialled lazily under a per-host one-slot semaphore so waiting ends with
  the caller's ctx; a dead backend (`liveness`) is closed and redialled on
  the next `get`. `dialSSHSocket` runs credentials, TCP, the SSH handshake
  and the `direct-streamlocal` channel to the socket, each failure a
  `dialError` with its stage; `libvirtDialer.connect` bounds the whole
  sequence plus the libvirt `Connect` with the dial budget and starts the
  `keepalive` (15 s, `keepalive@openssh.com`) and a watcher on go-libvirt's
  `Disconnected`. `call` runs each blocking go-libvirt call in a goroutine
  and waits on the result, ctx and a budget timer; abandoning a call
  (`link.markDead`) closes the socket and the SSH client, returns Transient
  wrapping the ctx or deadline error, and every other call waiting on that
  link fails once with Transient. `loadCredentials` runs on every dial.
 - **Logging.** Logger from the context, named `kvm2`, with `provider`,
  and `host` and `name` on lines about an instance. V(1): one line per
  operation outcome (`kvm2 domain ensured running`, `kvm2 domain fetched`,
  `kvm2 domain deleted`, `kvm2 domains listed`, `kvm2 host checked`), one
  `libvirt call` line per RPC from `observe` (rpc name, identifiers,
  duration, error fields), the connection events `libvirt host connected`
  and `libvirt host connect failed` with `stage`, and `kvm2 backend call
  failed`. V(2): XML, metadata, addresses, listed names, pool sizes, the
  resolved Create parameters. Upload and download bodies, cloud-init and
  key material are never logged.
 - **Tracing.** `observe.call` starts `kvm2.<Method>` per RPC with
  `kvm2.host` and the identifiers (`kvm2.domain`, `kvm2.pool`,
  `kvm2.volume`), sets `kvm2.libvirt_code` on failure and writes the
  `libvirt call` line inside the span; `libvirtDialer.dial` starts
  `kvm2.dial` (`kvm2.address`, `kvm2.user`); `fetchImage` starts
  `kvm2.download` (`kvm2.url_host`, `kvm2.sha256_configured`,
  `kvm2.bytes`). Both the decorator and the dialer take `tracing.Option`
  so tests inject a recorder; `New` leaves them on the global provider.
 - **Tests.** Pure builders (`xml_test.go`, `volume_test.go`,
  `seed_test.go`, `id_test.go`, `metadata_test.go`, `state_test.go`);
  `errors_test.go` (the classify table with real SSH error strings and a
  real `knownhosts.KeyError`); `backend_test.go` (the adapter over
  go-libvirt's own mock through a `net.Pipe`: deadline kills the
  connection, cancel unblocks, killed by another caller); `hosts_test.go`;
  `ssh_test.go` over `sshserver_test.go` (every dial stage, host-key
  policies, key material never in errors or logs); `image_test.go`
  (marker rules, streaming asserted by counting bytes, dedicated upload
  connection); `create_test.go`, `get_test.go`, `delete_test.go`,
  `list_test.go`, `kvm2_test.go`, `provider_test.go` (the contract §11
  list, named test by test in the spec's entry 21); `observe_test.go`
  (every seam method spanned and logged exactly once, by reflection);
  `check_test.go` (the reason table, never touches the pool). The double is
  `fakeBackend` in `fake_test.go`, whose `twin` shares state under its own
  call log so a test can tell the pooled connection from a dedicated one.
  Real backend: `kvm2_integration_test.go` behind `//go:build integration`,
  driven by `KVM2_TEST_*` variables and `make test-integration-kvm2`,
  described in
  [testing.md](testing.md#the-kvm2-integration-test-real-hypervisor).
 - **Deployment.** The `kvm-ssh` Secret and its mount are shared with kvm
  (`config/manager/manager.yaml` says so), `make kvm-ssh-secret` is reused,
  a commented `kvm2-home` entry in both providers configs,
  `config/samples/proxy_kvm2.yaml` (zone `beelink`, matching the shipped
  config), the `kvm2` type in the `status.type` comment of
  `api/v1alpha1/providerhealth_types.go`. No RBAC.
 - **Known gaps** (see also [kvm2-provider.md § Known gaps](kvm2-provider.md#known-gaps)).
  - `Create` blocks, including the first image download on each host.
  - GC blind spots: volumes without a domain are invisible to GC, and so are
    VMs on a host removed from the config, which `Delete` treats as done.
  - No adoption of kvm's domains: a switched entry needs its Proxies
    deleted first.
  - Cancelling one operation cuts off the others on that host once: a
    cancelled `call` kills the pooled connection.
  - `Content-Length` is required for image downloads.
  - A paused or pmsuspended domain cannot be adopted by a repeat `Create`:
    `DomainCreate` refuses, `waitRunning` expires, Transient on every retry.
  - Credential parse errors carry the parser's own text, which can quote a
    PEM header type or a malformed known_hosts token (never key bytes).
  - No reused-overlay check: an existing overlay is reused without checking
    its backing image or size.
  - The sample cloud-init installs Squid with `http_access allow all`.
 ## 8. Gaps shared by the existing providers
 - **Adoption is not verified in three of the four.** kubernetes, gcp and
  kvm adopt any existing instance with the right name without checking
  `LabelUID`; kvm2 returns `ErrPermanent` when the UID differs.
 - **The Create comment does not match kvm and kvm2.** The interface comment
  in `internal/provider/provider.go` says `Create` returns as soon as the
  request is submitted; both libvirt providers block until the domain has
  started.
 - **`Stopped` and `Terminated` are produced inconsistently.** kubernetes
-  never produces Stopped, kvm never produces Terminated, and the reconciler
+  never produces Stopped, kvm and kvm2 never produce Terminated, and the
-  treats them the same.
+  reconciler treats them the same.
 - **Different `ListByTag` partial-failure policies.** gcp leaves failed
-  parts out, kvm fails the whole list. Both are safe.
+  parts out, kvm and kvm2 fail the whole list. Both are safe.
 - **Stale `mock` comments** in `internal/provider/provider.go`,
  `internal/provider/registry/registry.go` and `api/v1alpha1/proxy_types.go`
  (and so the generated CRD's `spec.provider` description).
@@ -351,33 +542,41 @@ the author's view.
  is still open.
 - **Tracing.** kvm has no child spans, so a kvm trace shows `provider.create`
  and nothing under it; correlation works only through `traceID` on its V(1)
-  log lines. Only kubernetes is tested against a real collector, and that
+  log lines. kvm2 has them, but only an in-memory recorder checks their
-  test checks only the operation spans; nothing checks that gcp's SDK spans
+  topology. Only kubernetes is tested against a real collector, and that
-  or the kubernetes API spans nest under `provider.*`.
+  test checks only the operation spans; nothing checks that gcp's SDK spans,
  the kubernetes API spans or kvm2's spans nest under `provider.*` in a
  real collector.
-## 8. Proposed features
+## 9. Proposed features
-Listed only. Nothing below is designed or implemented.
+Listed only. Nothing below is designed or implemented, except where a
 bullet says kvm2 already does it.
 - **Child spans in kvm:** extend the `tracingAPI` decorator in
  `kvm/trace.go`, which already wraps every libvirt call, to also start a
  span per RPC, and add spans for the SSH dial and the image download. The
  seam's methods take no `ctx` today (`hostPool.call` hands `fn` only the
-  API), so the span context has to be plumbed in.
+  API), so the span context has to be plumbed in. kvm2's `observe.go` over
- **Tempo e2e checks for child spans**, and for gcp and kvm through their
+  its ctx-first seam is the shape this would take.
-  real backend runbooks.
+- **Tempo e2e checks for child spans**, and for gcp, kvm and kvm2 through
  their real backend runbooks.
 - **A shared conformance suite:** for example `internal/provider/providertest`,
  which runs the contract §11 tests against any provider plus a fake
  backend, so a new provider gets the contract tests for free.
 - **Verify the UID when `Create` adopts an instance.** Return `ErrPermanent`
  on a mismatch instead of adopting an instance that belongs to someone else.
  kvm2 does this; the other three do not.
 - **Report asynchronous create failures:** gcp keeps the operation name, and
  `Get` reports the failure instead of NotFound.
 - **Capacity or quota signals** for kvm, such as pool free space and host
-  memory, mapped to `ErrQuotaExceeded`.
+  memory, mapped to `ErrQuotaExceeded`. kvm2 has the pool-space precheck;
  host memory is still unchecked in both.
 - **GC for secondary resources:** mark volumes and similar resources, and
  let `ListByTag` or a second listing method cover them.
- **Reload credentials and config without a restart.**
+- **Reload credentials and config without a restart.** kvm2 re-reads its
  SSH material on every dial; the providers config is still read once per
  process for every type.
 - **Checker `Details`** for kubernetes (server version) and gcp (quota
  headroom).
 - **Internal-IP mode** for gcp, and a firewall rule managed per provider.
@@ -13,11 +13,13 @@ the existing ones happen to work.
  a new provider's design answers, one entry per decision the contract leaves
  open. Copy it to `docs/<type>-provider-spec.md`, fill it in, and hand it to
  the implementer with the other two files.
- **[providers-existing.md](providers-existing.md)**: how kubernetes, gcp and
+- **[providers-existing.md](providers-existing.md)**: how kubernetes, gcp,
-  kvm meet the contract, a table mapping each of them to every profile, their
+  kvm and kvm2 meet the contract, a table mapping each of them to every
-  per-provider reference notes, shared gaps and proposed features.
+  profile, their per-provider reference notes, shared gaps and proposed
  features.
 Per-provider user guides: [kvm-provider.md](kvm-provider.md),
 [kvm2-provider.md](kvm2-provider.md),
 [gcp-vm-validation.md](gcp-vm-validation.md) and
 [gcp-in-specific-project.md](gcp-in-specific-project.md).