egress-proxies-operator/docs/kvm2-provider.md

# The kvm2 provider

Runs Managed proxies as VMs on your own libvirt/KVM hypervisors. It is the
second-generation libvirt provider, built clean-room from
[provider-contract.md](provider-contract.md) and the design in
[kvm2-provider-spec.md](kvm2-provider-spec.md), and it ships side by side
with the first-generation `kvm` provider: both types can be configured at
once, both take the same config block shape, and both read the same
`kvm-ssh` Secret. What kvm2 adds over kvm: a child span per libvirt RPC,
a pool-space precheck that maps to the reconciler's quota wait, SSH
credentials re-read on every dial, image downloads and health checks on
connections of their own, an adoption guard on the owning UID, and SSH
authentication failures that stop retrying.

Like kvm, the operator reaches each host over SSH, forwards the libvirt
unix socket through that connection and speaks libvirt's own RPC over it,
so the hosts need nothing beyond `sshd` and `libvirtd` and the operator
image carries no libvirt libraries. VMs boot a stock distro cloud image
that the operator downloads into the host's storage pool on first use,
with the Proxy's cloud-init delivered on a NoCloud seed ISO built
in-process.

## Hypervisor prerequisites

Per host, once:

1. **libvirt and KVM** (`libvirtd` running, `/dev/kvm` present).
2. **An SSH user in the `libvirt` group**, so it may open
   `/var/run/libvirt/libvirt-sock`. The operator authenticates with a
   private key; nothing else on the host is touched. The socket is used
   as `qemu:///system`, so no further host permissions are needed.
3. **A bridge to a network with DHCP** (for example `br0` bridged onto
   the LAN). Each VM gets one virtio NIC on it and takes its IP from that
   network's DHCP. The operator's pod network and the crawler clients
   must route to those IPs: the health engine probes `http://<ip>:<port>`
   through the proxy and clients get the same address from the discovery
   API. Nothing on the host filters the port; the provider does not use
   `spec.port` at all.
4. **A storage pool** (default `default`, usually `/var/lib/libvirt/images`)
   with room for one base image per distro and provider entry (a few
   hundred MB each) plus the
   per-VM overlays (`diskSizeGb` each, thin-provisioned). The pool must be
   running: a stopped pool fails the health check with `PoolNotActive` and
   fails `Create` permanently.
5. **UEFI firmware for guests**: the `ovmf` package on Debian/Ubuntu,
   `edk2-ovmf` on Fedora/RHEL. VMs boot UEFI by default with secure boot
   off (libvirt picks the OVMF binary itself); a domain start fails with
   "no firmware found" without it. Only `firmware: bios` images skip this.
6. **Outbound internet from the VMs**, in practice: a vendor-data script
   installs `qemu-guest-agent` from the distro mirrors on first boot, and
   the agent is how the operator learns the VM's address. Without it the
   operator falls back to the host's ARP table, which only sees traffic
   addressed to the host itself, so on a plain bridge a VM without the
   agent stays `Provisioning`.

Check from the operator's point of view before wiring it in:

```sh
ssh novakj@beelink virsh -c qemu:///system list --all      # user can drive libvirt
ssh novakj@beelink virsh -c qemu:///system pool-info default
ssh novakj@beelink ip link show br0
```

## Cluster-side setup

1. Create the SSH Secret from a key pair dedicated to the operator and a
   `known_hosts` file holding every host's key. It is the same `kvm-ssh`
   Secret the `kvm` provider uses; one Secret serves both types, and if
   you already have it there is nothing to do here.

   ```sh
   # once: a key pair for the operator; the .pub goes on every host
   ssh-keygen -t ed25519 -f ~/.ssh/proxy-operator_ed25519 -N '' -C proxy-operator
   ssh-copy-id -i ~/.ssh/proxy-operator_ed25519.pub novakj@beelink

   # once per host: its host key, under the address the *operator* will
   # dial (hosts[].address), which may differ from what your laptop uses
   ssh-keyscan -H beelink.tail8c69c.ts.net >> ~/.ssh/kvm_known_hosts

   # the Secret: the PRIVATE key (not the .pub) plus known_hosts
   kubectl -n egress-proxies-operator-system create secret generic kvm-ssh \
     --from-file=id_ed25519=$HOME/.ssh/proxy-operator_ed25519 \
     --from-file=known_hosts=$HOME/.ssh/kvm_known_hosts
   ```

   `make kvm-ssh-secret` runs that last command as an apply, so it also
   updates an existing Secret; override the paths with
   `KVM_SSH_KEY=... KVM_KNOWN_HOSTS=...`. Putting the `.pub` in the
   Secret is the classic mistake: the target's `kvm-ssh-secret` check
   refuses a file without `PRIVATE KEY` in it, and the provider itself
   would fail every call permanently with a "parsing ssh key file" error.
   Without `known_hosts`, set `insecureIgnoreHostKey: true` and accept
   that a MITM could hand you a fake hypervisor.

   `config/manager/manager.yaml` mounts the Secret (optional) at
   `/etc/proxy-operator/kvm` as two files, `id_ed25519` and
   `known_hosts`, which is what `sshKeyFile` and `knownHostsFile` in the
   provider config point at. The deployment starts fine without the
   Secret, even with a kvm2 provider configured: `New` only checks that
   the files exist and logs a `WARNING:` line when one does not; the
   provider's proxies fail transiently and recover once the Secret lands.

   **Rotating the key or a host key needs no restart.** Both files are
   read and parsed on every dial (`loadCredentials` in
   `internal/provider/kvm2/ssh.go`); nothing is cached. Re-run the
   command or the make target, and the next connection, health check
   included, uses the new material. Connections that are already open
   stay open until they drop.

2. Add a provider entry to the providers config
   (`config/manager/providers_config.yaml` in-cluster; full sample in
   `config/samples/providers-config.yaml`). The type is `kvm2` and the
   block is `kvm2:`; its fields are those of the `kvm` block:

   ```yaml
   - name: kvm2-home
     type: kvm2
     kvm2:
       sshKeyFile: /etc/proxy-operator/kvm/id_ed25519
       knownHostsFile: /etc/proxy-operator/kvm/known_hosts
       images:
         debian-13:
           url: https://cloud.debian.org/images/cloud/trixie/latest/debian-13-generic-amd64.qcow2
       machineTypes:
         small: {vcpus: 1, memoryMiB: 1024}
       hosts:
         - name: beelink
           address: beelink.tail8c69c.ts.net:22
           user: novakj
           bridge: br0
   ```

   Validation messages name the block, so a mistake reads
   `kvm2.hosts[0]: bridge is required`, not `kvm.hosts`. The providers
   config is read once per process: adding a host or an image entry needs
   a manager restart; credentials do not.

3. Apply a Proxy (`config/samples/proxy_kvm2.yaml`) and watch it:

   ```sh
   kubectl get px -w
   # NAME                MODE      PROVIDER    PHASE          IP             HEALTHY
   # proxy-kvm2-sample   Managed   kvm2-home   Provisioning
   # proxy-kvm2-sample   Managed   kvm2-home   Running        192.168.1.73
   # proxy-kvm2-sample   Managed   kvm2-home   Ready          192.168.1.73   True
   ```

   The first Proxy for a given image on a given host sits in
   `Provisioning` for the length of the image download; the operator log
   shows `fetching base image into host pool` with the URL and size and
   `base image cached` with the size, duration and digest.

## Config reference

The block is `kvm2:` and its Go type is the shared `KVMConfig` in
`internal/provider/config_kvm.go`, so the fields, defaults and validation
are the same as for `kvm`. Defaults are applied by `withDefaults` in
`internal/provider/kvm2/kvm2.go`.

| Field | Required | Default | Meaning |
|---|---|---|---|
| `sshKeyFile` | yes | | private key used for every host; OpenSSH or PEM, parsed on every dial |
| `knownHostsFile` | unless `insecureIgnoreHostKey` | | OpenSSH `known_hosts` to verify host keys against, read on every dial |
| `insecureIgnoreHostKey` | | `false` | skip host-key verification entirely; `New` logs a `WARNING:` line when it is on |
| `diskSizeGb` | | `10` | capacity of each VM's overlay disk; cloud-init's growpart expands the root fs into it. Also what the pool-space precheck reserves per VM |
| `vendorData` | | | extra cloud-init vendor-data placed on every seed next to the Proxy's user-data. Beware that cloud-init drops any top-level key of a vendor-data **cloud-config** that the user-data also defines. Shipped as one part of a MIME multipart alongside the agent script when `installGuestAgent` is on |
| `installGuestAgent` | | `true` | add the vendor-data script that installs and starts `qemu-guest-agent`. Turn off for images that ship it already or cannot reach a package mirror; IP discovery then depends on the host's ARP table |
| `images.<key>` | | | the key names the image in `placement.image`. Letters, digits, dots, dashes and underscores only: it becomes the cached volume name `base-<key>.<format>` |
| `images.<key>.url` | yes per entry | | http(s) URL of a cloud image (qcow2 or raw). The server must send `Content-Length` |
| `images.<key>.sha256` | | | expected digest, 64 hex characters; verified after download, a change triggers a re-download |
| `images.<key>.format` | | `qcow2` | `qcow2` or `raw` |
| `images.<key>.firmware` | | `uefi` | `uefi` (OVMF, secure boot off) or `bios` (SeaBIOS) for images without an EFI system partition |
| `machineTypes.<key>` | at least one | | `{vcpus, memoryMiB}` size profile, both greater than zero |
| `hosts[].name` | yes | | the `placement.zone` value that selects this host; unique. Letters, digits, dots, dashes and underscores only: it becomes part of the providerID `hosts/<name>/domains/<vm>` |
| `hosts[].address` | yes | | SSH endpoint, `host:port` |
| `hosts[].user` | yes | | SSH user, member of the `libvirt` group |
| `hosts[].socket` | | `/var/run/libvirt/libvirt-sock` | libvirt socket path on the host |
| `hosts[].pool` | | `default` | storage pool for base images, overlays and seeds |
| `hosts[].bridge` | yes | | host bridge for the VM's NIC |

## Placement

All three fields are checked before any host is dialled. An unknown value
is `ErrPermanent`, which the reconciler latches as `Failed` with the
message below until the spec changes.

| Proxy field | Meaning for kvm2 | Error when unknown |
|---|---|---|
| `placement.zone` | host name from `hosts`. May be omitted when exactly one host is configured; otherwise required | `placement.zone "x" is not in kvm2.hosts`; with several hosts and no zone, `placement.zone is required when kvm2.hosts lists more than one host` |
| `placement.machineType` | key in `machineTypes` | `placement.machineType "x" is not in kvm2.machineTypes` |
| `placement.image` | key in `images`, or an `http(s)://` URL used directly with the defaults (`qcow2`, `uefi`) | `placement.image "x" is neither a key in kvm2.images nor an http(s) URL` |
| `placement.region` | ignored. It is still part of the spec hash, so changing it replaces the VM and does nothing else | |

The messages come from `Create` in `internal/provider/kvm2/create.go`,
`host` in `kvm2.go` and `resolveImage` in `image.go`. Fix the spec (which
replaces the VM) or the config. Catalogue edits affect only new VMs: the
spec hash covers the placement values, not what the config maps them to.

## What lands on the host

For a Proxy whose deterministic name is `proxy-abc…`, on the chosen
host's pool:

| Object | Name | Lifetime |
|---|---|---|
| base image | `base-<entry>-<key>.<format>`, where `<entry>` is the provider entry's name (for a URL image, `<key>` is the first 16 hex characters of the URL's sha256) | cached; survives Proxy deletion |
| image marker | `base-<entry>-<key>.<format>.sha256`, a 64-byte raw volume holding the digest | with the base image |
| overlay disk | `proxy-abc….qcow2`, qcow2 backed by the base file, `diskSizeGb` capacity | with the Proxy |
| seed ISO | `proxy-abc…-seed.iso`, raw | with the Proxy |
| domain | `proxy-abc…` | with the Proxy |

The names are built in `internal/provider/kvm2/volume.go`. The base image
carries the **provider entry's name** because the file is shared by every
Proxy on that host but must not be shared by two entries: the `kvm`
provider builds `base-<key>.<format>` in the same pool from the same
catalogue shape, and two kvm2 entries pointing one key at different URLs
would otherwise agree on the name and disagree on the contents. The cost
is one copy of each image per entry per host. For that reason the entry's
name must itself be a usable file name — letters, digits, dots, dashes and
underscores — which `New` checks at startup, since provider names are
otherwise only required to be non-empty and unique.

The domain (`buildDomainXML` in `xml.go`) is a `q35` machine with a
host-passthrough CPU, ACPI and APIC, memory in MiB and vcpus from the
machine type, the overlay on `vda` (virtio) and the seed as a read-only
SATA cdrom on `sda`, one virtio NIC on the bridge, a serial console on a
pty, the guest-agent channel `org.qemu.guest_agent.0`, a virtio RNG fed
from `/dev/urandom`, and no graphics. Both disks are `type="file"` with
the paths `StorageVolGetPath` returned: AppArmor derives QEMU's profile
from those paths, and volume-type disks fail with permission denied on
stock Debian and Ubuntu hosts. UEFI images get `<os firmware="efi">` with
the `secure-boot` feature disabled; BIOS images get neither.

Its `<metadata>` carries one element in kvm2's own namespace,
`https://gitea.home.hrajfrisbee.cz/kacerr/egress-proxies-operator/kvm2`
(`metadataURI` in `metadata.go`):

```xml
<instance xmlns="https://gitea.home.hrajfrisbee.cz/kacerr/egress-proxies-operator/kvm2">
  <managed>true</managed>
  <uid>0f3a…</uid>
  <createdAt>2026-09-16T17:42:11Z</createdAt>
</instance>
```

That is how orphan GC recognises its own VMs, since libvirt has no
labels, and how a repeat `Create` knows whether an existing domain of the
same name belongs to this Proxy: a domain without the element is a
permanent error (`carries no kvm2 mark; it was not created by this
provider`), a different non-empty `uid` is a permanent error naming both
UIDs, and a marked domain with a matching or empty `uid` is adopted. The
namespace is kvm2's own, so the `kvm` provider's domains are invisible to
it (see *Migrating from kvm*).

The seed (`nocloud.BuildSeedISO`, shared with the proxmox provider in
`internal/provider/nocloud`) is an ISO 9660 image labelled
`cidata` with three files. `user-data` is the Proxy's `cloudInit`, **byte
for byte**: any format cloud-init accepts works, including a Jinja
template, a MIME archive, a shell script or a gzipped payload; the
operator never parses or re-renders it. `meta-data` holds
`instance-id: <name>` and `local-hostname: <name>`, so a reboot is never
taken for a new instance. `vendor-data` carries the guest-agent
bootstrap. The ISO is staged in a temp directory under `/tmp` (the
manager's 64Mi emptyDir), assembled in memory (tens of KiB) and uploaded
as a raw volume; the staging directory is removed before the call
returns. The seed is rebuilt on every `Create` attempt, because a
previous attempt may have been cut off mid-upload and a truncated seed
would boot a VM without cloud-init.

The guest agent rides along in `vendor-data` as a **shell script**
(`nocloud.GuestAgentScript`), not a cloud-config, on purpose:
cloud-init merges a vendor-data cloud-config under the user-data with
mergers that keep the user-data's value for every top-level key both
define and never append lists, so a Proxy carrying its own `packages:`
would silently lose the agent. Scripts bypass the merge. The script is
idempotent, picks whichever of `apt-get`, `dnf`, `yum`, `zypper` or `apk`
the image has, enables the service with `systemctl` or OpenRC, and logs
into `/var/log/cloud-init-output.log`. With `vendorData` also configured,
the two travel as separate parts of a MIME multipart (`text/plain` for
your document, `text/x-shellscript` for the script), with the
`MIME-Version` header first so cloud-init detects it. With
`installGuestAgent: false` the file is your `vendorData` alone, or empty.

## The image cache

The base image is downloaded **by the operator, through libvirt**, into
the host's pool the first time a Proxy asks for it on that host. Nothing
is pre-placed on the hypervisor and the operator needs no filesystem
access there. The bytes stream from the URL through the operator into
`StorageVolUpload` with a running sha256; nothing is buffered
(`fetchImage` and `uploadImage` in `internal/provider/kvm2/image.go`).

The upload runs on a **dedicated connection** to the host, dialled for
the transfer and closed after it, not on the pooled connection the other
operations share. A cancelled or overlong call kills the connection it
ran on (see *Timeouts and connections*), and a poll for some other Proxy
being cancelled must not cut off a 20-minute upload.

A body that fails part way through is reported to the caller **out of
band**: the reader handed to `StorageVolUpload` turns any source error
into a clean end of stream and keeps the real error for afterwards, so
the short upload is caught by the length and digest checks rather than
by the stream. This is not tidiness. go-libvirt answers a reader error
by parking its stream sender forever on a channel it no longer reads, so
handing it one wedges the call and leaks the sender with its 4 MiB
buffer; see *Known gaps*. For the same reason the download carries its
own deadline, a minute inside the transfer budget, so a stalled mirror
ends through its body rather than through the killed socket the budget
would otherwise leave behind.

The download needs a `Content-Length`: the volume's capacity is declared
before the upload starts. The request sends `Accept-Encoding: identity`
so a transparently gzipped body cannot hide the length. A response
without it, or with length 0, is a permanent error naming the URL, and so
is any 4xx status; 429 and 5xx are transient and retried.

The marker volume is written **last**, on the pooled connection, after
`StoragePoolRefresh` has made libvirt read the file's real format. Its
presence is what marks a base complete: a crash mid-download leaves a
base volume without a marker, which the next attempt deletes and redoes
(`cachedImage`). A re-download also happens when the marker's content is
not a 64-character hex digest, or when `sha256` is configured and differs
from it. Otherwise the cached copy is used for as long as it exists, even
if the URL's content changed upstream: a `latest` URL is pinned at the
moment of first fetch, per host. Concurrent Creates for the same image
on the same host queue on a per-(host, volume) lock; the first fetches,
the rest find the marker.

To evict manually:

```sh
ssh novakj@beelink virsh -c qemu:///system vol-list --pool default | grep base-kvm2-home-
ssh novakj@beelink virsh -c qemu:///system vol-delete --pool default base-kvm2-home-debian-13.qcow2.sha256
ssh novakj@beelink virsh -c qemu:///system vol-delete --pool default base-kvm2-home-debian-13.qcow2
```

Marker first: a base without its marker is refetched, a marker without its
base is cleaned up, and neither leaves a Proxy pointing at nothing.

**Warning: do not evict a base volume while Proxies using that image exist
on the host.** The overlays back onto the base file, and libvirt will
unlink it under running domains without complaint; the guests keep running
on the open file descriptor and fail at their next start. Delete those
Proxies first.

Changing `sha256` on an entry whose base is already cached is **refused**,
permanently and by name, rather than resolved by replacing the file — that
replacement was the same corruption, just automated. Add the image under a
new key and move Proxies over by changing their `placement.image` (which
replaces the VM), or delete the Proxies and evict the base by hand.
Changing `url` alone does nothing at all: the cache is keyed by entry and
key, so the new URL is never fetched. Add a new key for that too.

## Quota: the pool-space precheck

Before creating any volume, `Create` reads `StoragePoolGetInfo` and
compares the pool's free space with what it is about to commit
(`checkPoolSpace` and `neededBytes` in `image.go`): the image's
`Content-Length` when the base is not cached yet (found with a `HEAD`
request, or a `GET` whose body is closed at once when `HEAD` answers
anything but 2xx, since mirrors refuse it with 405, 403 or 501 alike; the
`GET`'s status is then what counts), plus `diskSizeGb` in bytes for the
overlay, plus 16 MiB for the
seed. Below that it returns `ErrQuotaExceeded` with a message naming the
pool, the bytes available and the bytes needed. The reconciler answers
quota with a wait, not a failure: the Proxy shows `Provisioned=False`
with reason `QuotaExceeded` and is retried after `QuotaRetry` (5
minutes), off the error backoff curve and never `Failed`. Free up the
pool or lower `diskSizeGb` and the next attempt goes through.

The precheck is the only quota signal; libvirt itself reports a full pool
as an ordinary error, which would be transient. The health check exposes
the same number as `poolAvailableGiB`, so a pool that is filling up shows
there before Creates start waiting.

A pool that is not running fails the precheck permanently (`storage pool
<pool> is not running`): start it with `virsh pool-start` and
`virsh pool-autostart`.

## Migrating from `kvm`

The `kvm2` block is the `kvm` block under another name, so an entry ports
by changing `type: kvm` to `type: kvm2` and `kvm:` to `kvm2:`. The
`kvm-ssh` Secret is shared; nothing changes on the hosts.

What does not port is the VMs. kvm2 marks its domains in its own metadata
namespace and neither lists nor adopts domains the `kvm` provider made:
`ListByTag` skips them, so orphan GC never sees them, and `Create` refuses
a domain of the same name that lacks the kvm2 mark with a permanent error
rather than start it (a Proxy recreated under the new entry gets the same
deterministic name, so this is what happens when step 1 below is skipped).
A Proxy's `spec.provider` is immutable, so an existing Proxy cannot be
moved to the new entry anyway. Do it in this order:

1. Delete the Proxies on the `kvm` entry and wait for them to go; the
   `kvm` provider removes their domains and volumes.
2. Change the entry (or add a second one under a new name and keep the
   old one for a while; both types may share a host).
3. Restart the manager, then recreate the Proxies with the new
   `spec.provider`.

A `kvm` domain left behind on a host is invisible to kvm2 forever; clean
it up on the host with `virsh undefine --nvram <name>` and
`virsh vol-delete` for its `<name>.qcow2` and `<name>-seed.iso`.

The base image cache is **not** shared between the two types, and that is
deliberate: kvm names its copy `base-<key>.<format>` while kvm2 names its
own `base-<entry>-<key>.<format>`, so kvm2 downloads the image once more
per host rather than adopting a file another provider maintains. Nothing
needs deleting; the `kvm` base stays where it is and keeps serving the
overlays that back onto it. Budget the extra few hundred MB in the pool.

## Logging

The provider logs through controller-runtime's logger, named `kvm2`, with
`provider` (the configured name) on every line, and `host` and `name`
(the domain) on every line about an instance. Every line inside a traced
operation also carries `traceID` and `spanID`.

- `--zap-log-level=debug` (logr V(1)):
  - one line per operation outcome: `kvm2 domain ensured running` with the
    `id`, `kvm2 domain fetched` with `state`, `ip` and `uid`,
    `kvm2 domain not found`, `kvm2 domain deleted`, `kvm2 domains listed`
    with `hosts` and `managed` counts, `kvm2 host checked` with `ready`,
    `reason` and `latency`; plus the steps inside a Create
    (`kvm2 domain already defined`, `overlay volume created`,
    `seed volume uploaded` with `bytes`, `domain defined` with `uid` and
    `firmware`, `domain started`, `fetching base image into host pool`,
    `base image cached`);
  - **one `libvirt call` line per libvirt RPC**, written by the decorator
    in `internal/provider/kvm2/observe.go` inside that RPC's span: `rpc`
    (the go-libvirt method name, for example `DomainLookupByName`), the
    identifiers it was given (`domain`, `pool`, `volume`), `duration`, and
    on failure `error` plus `libvirtCode` and `libvirtError` (the code's
    name, for example `ErrNoDomain`). A Get is four or five RPCs; a
    Create with a cached image is around twenty;
  - connection events from `internal/provider/kvm2/hosts.go`:
    `libvirt host connected` with `address`, `user` and `duration`, and
    `libvirt host connect failed` with the same plus `error` and `stage`
    (`credentials`, `tcp`, `ssh` or `socket`);
  - `kvm2 backend call failed` with `op` and the raw `error` once per
    failed operation, before the error is returned;
  - a line per domain that `ListByTag` skips, and a line when the image
    marker forces a refetch.
- `--zap-log-level=2`: the `libvirt call` lines also carry the domain and
  volume `xml`, the `metadata` element, the interface `addresses`, the
  listed `domains`, the pool `state` and sizes and the volume `path`; the
  `domain defined` line carries the `xml`; `kvm2 create resolved` logs the
  resolved parameters (pool, bridge, machine type, sizes, image volume,
  firmware) and `cloudInitBytes`; `kvm2 listed instance` logs each listed
  domain.
- Never logged at any level: the cloud-init content, the SSH key, the
  known_hosts content, and upload or download bodies. The seed and image
  uploads report `offset` and `length` only. There is no raw wire tier
  like the GCP provider's V(5); go-libvirt exposes no hook for it.

## Tracing

With an OTLP exporter configured ([architecture.md
§10](architecture.md#10-tracing-internaltracing)), every backend call is
a child span of the operation span the decorators open
(`provider.create`, `provider.get`, `provider.delete`, `provider.list`)
or of the monitor's `providerhealth.check`:

| Span | Around | Attributes |
|---|---|---|
| `kvm2.<Method>`, for example `kvm2.DomainDefineXML` | one libvirt RPC (`observe.go`) | `kvm2.host`, and the identifiers passed: `kvm2.domain`, `kvm2.pool`, `kvm2.volume`; on failure the status is Error and `kvm2.libvirt_code` carries the libvirt error number |
| `kvm2.dial` | the SSH handshake and the libvirt connect (`hosts.go`) | `kvm2.host`, `kvm2.address`, `kvm2.user` |
| `kvm2.download` | one image fetch, from the HTTP request to the marker (`image.go`) | `kvm2.host`, `kvm2.volume`, `kvm2.url_host`, `kvm2.sha256_configured`, and `kvm2.bytes` once the response arrived |

All spans are client spans. No attribute carries cloud-init, key material
or a URL beyond its host. The `libvirt call` log line for an RPC is
written inside that RPC's span, so its `spanID` points at the span.

## Timeouts and connections

No caller puts a deadline on the context, so every backend call carries
its own budget (`defaultBudgets` in `internal/provider/kvm2/backend.go`,
applied by `call`):

| Call | Budget |
|---|---|
| TCP connect plus SSH handshake plus libvirt connect | 10 s |
| ordinary RPC | 30 s |
| `DomainCreate` | 90 s |
| `StorageVolUpload` of at most 16 MiB (the seed, the marker) | 60 s |
| `StorageVolUpload` of an image, `StorageVolDownload`, and the whole image fetch from HTTP request to marker | 20 m; the caller's ctx may end it earlier |
| wait for running after `DomainCreate` | 15 s, polled every 500 ms (`defaultRunningWait` and `defaultRunningPoll` in `create.go`); `DomainCreate` is synchronous, so the first poll normally answers running, and a domain that is shut off, crashed or paused right after the start fails at once rather than waiting out the cap |
| the `HEAD` request of the pool-space precheck | 30 s |
| the courtesy `ConnectClose` when a connection is closed | 5 s |

go-libvirt's calls take no context and block, so the only way to abandon
one is to close the connection under it. That is what a cancelled or
overlong call does: the adapter marks the connection dead so the next call
redials, kills it by closing the SSH client's TCP connection, which takes
the forwarded socket with it, and returns `ErrTransient` wrapping the
context or deadline error. It marks dead before it kills, and closes the
TCP connection rather than the forwarded channel, because both of those are
lock-free: closing the channel writes a close message under the channel's
write lock, so it would queue behind a transfer stuck in a TCP write to a
host that has gone away — for as long as the kernel retransmits, which is
the one case the kill exists to survive. The consequence:
**other RPCs in flight on that host's pooled connection fail once with
`ErrTransient`** and their callers retry on a fresh connection. In the
reconciler that is one requeue with backoff; in GC it is one skipped
sweep.

Two callers are kept off the pooled connection for that reason
(`dialFresh` in `hosts.go`): the image upload, because a 435 MB transfer
must not die because a poll for another Proxy was cancelled, and the
health check, because a check that runs out of the monitor's timeout must
close only its own connection.

Otherwise there is one pooled SSH plus libvirt connection per host,
dialled lazily on the first call that needs it and shared by every
operation for that host (go-libvirt multiplexes RPCs on one connection).
Concurrent callers wait for one dial and share the result; waiting ends
with the caller's context. A `keepalive@openssh.com` request every 15 s
marks the connection dead when it fails or gets no reply within another
15 s, as does go-libvirt noticing the socket is gone; the next call
redials. Create and Delete for the same domain name are serialised on a
per-name lock, so two Deletes (GC and the reconciler) never race on
destroy and undefine, and a Delete cannot interleave with a Create's
steps; Get and ListByTag take no lock.

## Health check

The operator checks every configured host every `--provider-check-interval`
(default 1m) and publishes the result in the `ProviderHealth` object named
after the provider (`kubectl get ph kvm2-home -o yaml`) and in the
`proxy_operator_provider_target_up{provider,target}` gauge. One target per
`hosts[]` entry, in config order, all checked concurrently.

Each check dials a **fresh connection** to the host and closes it when
done (`probeHost` in `internal/provider/kvm2/check.go`); the pooled
connection is never touched, so a check that runs out of time cannot fail
a reconcile's call in flight, and a fresh dial picks up a rotated Secret
at once. The sequence is the SSH and libvirt handshakes, then four RPCs:
`ConnectGetLibVersion`, `ConnectGetHostname`, `StoragePoolLookupByName`
and `StoragePoolGetInfo` on the host's `pool`. The cost per host per
round is one sshd session and those four calls.

Every target's `details` carry `user` and `pool`. As facts arrive they
gain `libvirtVersion` (decoded `major.minor.micro`), `hostname`,
`poolState` (virsh's word: `running`, `inactive`, ...),
`poolCapacityGiB` and `poolAvailableGiB`, so a target that fails late in
the sequence still says what the host answered before. The free space is
the number the quota precheck compares against.

A failing target says where the check broke (`checkReason` in `check.go`):

| Reason | Meaning | Look at |
|---|---|---|
| `Reachable` | ready | |
| `CredentialsUnavailable` | the key or known_hosts file is missing or does not parse | the `kvm-ssh` Secret (see *Cluster-side setup*); fixed in the cluster, not on the host |
| `Unreachable` | the TCP connect failed or ran out of the 10 s dial budget while the round still had time, the SSH transport failed for a reason other than authentication, the forwarded libvirt socket was refused, or the link died mid-check | DNS, Tailscale or a firewall between the cluster and `address`; `libvirtd` running and the `socket` path |
| `Unauthorized` | the SSH server rejected the key, or the host key is missing from known_hosts or has changed | `user`'s `authorized_keys`; the known_hosts entry (re-run `ssh-keyscan` and update the Secret) |
| `Forbidden` | SSH worked, but libvirt refused the user (`ErrAccessDenied` or `ErrAuthFailed`) | `user` in the `libvirt` group |
| `Timeout` | the monitor's `--provider-check-timeout` (default 15s) ran out before the check finished, whichever stage was running, the dial included; or the connection was up and an RPC's own budget ran out | a hung `libvirtd`, a saturated link, or a round timeout shorter than the dial takes |
| `PoolNotFound` | there is no pool named `pool` | `virsh pool-list --all` |
| `PoolNotActive` | the pool exists but is not running | `virsh pool-start <pool>` and `virsh pool-autostart <pool>` |
| `APIError` | any other libvirt error | the message carries libvirt's own text |

## Error classes

Every error is classified once, in `classify` in
`internal/provider/kvm2/errors.go`, with errors that carry a class from
their source (validation, the image fetch, the dial) keeping it. The
reconciler latches `ErrPermanent` as `Failed` until the spec changes,
waits 5 minutes on `ErrQuotaExceeded`, recreates on `ErrNotFound` from
`Get`, and retries everything else with backoff.

| Class | What lands there |
|---|---|
| `ErrPermanent` | placement values not in the catalogue; a malformed providerID; a host no longer in the config (from `Get`; `Delete` treats it as done); libvirt `ErrXMLError`, `ErrNoSupport`, `ErrInvalidArg`, `ErrAuthFailed`, `ErrAccessDenied` and `ErrNoStoragePool`; a pool that is not running; SSH authentication refused (`unable to authenticate`); a host key missing from known_hosts or changed (`knownhosts.KeyError`); a key or known_hosts file that exists but does not parse; a domain of the same name owned by another UID or carrying no kvm2 mark; a domain of the same name that is paused or pmsuspended; corrupt metadata on a domain; an image URL answering 4xx, a response without `Content-Length` or with length 0, a sha256 that does not match the configured one |
| `ErrQuotaExceeded` | the pool-space precheck |
| `ErrNotFound` | libvirt `ErrNoDomain` or `ErrNoStorageVol` from `Get`, `Delete` or `ListByTag`; inside `Create` the same codes are `ErrTransient`, because the contract forbids NotFound from Create |
| `ErrTransient` | everything else: a missing key or known_hosts file (the Secret has not landed yet), TCP and network errors, context cancellation, an abandoned or budget-exceeded call, a connection killed by another caller, libvirt `ErrOperationInvalid` and every unrecognised libvirt code, an image URL answering 429 or 5xx or ending the body early, every address source failing on an active domain, the wait for running expiring or the domain going shut off or crashed right after its start |

Error strings name paths, hosts, volumes and libvirt's own text. They
never contain key bytes, the known_hosts content or cloud-init.

## Known gaps

- **An upload that libvirtd itself fails leaks a goroutine, upstream.**
  go-libvirt's `requestStream` hands its stream sender an unbuffered
  channel and then, on the error path, returns without ever reading it:

  ```go
  abort  := make(chan bool)    // rpc.go
  outErr := make(chan error)
  go func() { outErr <- l.socket.SendStream(...) }()
  resp, err = l.processIncomingStream(c, in)
  if err != nil {
      abort <- true
      return resp, err          // outErr is never drained
  }
  ```

  Whenever the daemon errors mid-upload — a pool that fills, a volume
  removed underneath — the sender parks on `outErr` for good, taking its
  4 MiB buffer with it. Closing the socket does not help: both ends are
  blocked on channel sends, not on I/O. kvm2 avoids the two cases it
  controls (a failing body, and its own transfer deadline; see *The image
  cache*), so what is left needs a server-side failure. The fix is two
  characters upstream, `make(chan bool, 1)` and `make(chan error, 1)`; if
  it ever bites hard enough, a `replace` carrying that patch is the
  escape hatch.
- **Create blocks**, including the first image download on each host,
  holding one of the three reconcile workers for its duration (LAN
  speed, a few hundred MB, once per image per host). Other Proxies keep
  reconciling on the remaining workers. Warm the cache with one Proxy per
  image per host before creating many.
- **Volumes without a domain are invisible to GC.** A crash between
  creating the overlay or seed and defining the domain leaves volumes
  that `ListByTag` cannot see (it lists domains). The same Proxy's next
  Create reuses the overlay and rebuilds the seed, and its Delete removes
  both by name even with no domain, but a Proxy CR that is gone for good
  leaves them for `virsh vol-list` to find.
- **A host removed from the config orphans its VMs.** `Delete` for a
  providerID naming an unconfigured host is a logged no-op success, so
  the Proxy finishes deleting instead of pinning its finalizer, but the
  domain and its volumes stay on the hypervisor and orphan GC cannot see
  them either (it only sweeps configured hosts). Delete the Proxies before
  retiring a host, or clean up afterwards with `virsh undefine --nvram`
  plus `virsh vol-delete`. `Get` fails permanently for such a providerID
  on purpose: a typo in `hosts` must not look like a missing VM and
  trigger a rebuild elsewhere.
- **No adoption of `kvm` domains.** See *Migrating from kvm*.
- **A cancelled call drops the host's pooled connection once**, so other
  RPCs in flight on that host fail once with `ErrTransient` and retry.
  The image upload and the health check are isolated on their own
  connections.
- **`Content-Length` is required** for image downloads; a mirror that
  only chunks is rejected permanently. Point the catalogue at one that
  sends the length.
- **A paused or pmsuspended domain is not started.** A repeat `Create`
  for a domain in that state fails permanently (`is DomainPaused and
  cannot be started by the operator; resume or destroy it on the host`),
  so the Proxy shows `Failed` until its spec changes; the operator never
  resumes a VM an admin paused. Resume it on the host
  (`virsh resume <name>`) and touch the spec, or delete the Proxy.
- **Parser text in credential errors.** A key or known_hosts file that
  does not parse fails with the parser's own message, which can quote a
  PEM header type or a malformed known_hosts token. Key bytes are never
  quoted.
- **The sample cloud-init is an open proxy.**
  `config/samples/proxy_kvm2.yaml` installs Squid with
  `http_access allow all`; restrict it before exposing the bridge network
  to anything but the cluster.
- **IP discovery needs the agent or guest traffic to the host.** A VM on
  a bridge with no DHCP never gets an address and stays `Provisioning`
  forever; check `virsh console <name>` on the host. The same symptom
  with a booted, addressed VM means the guest agent never came up: check
  `systemctl status qemu-guest-agent`, `cloud-init status --long` and the
  vendor script's output in `/var/log/cloud-init-output.log` (a VM without
  outbound internet cannot install it). A user-data that disables
  vendor-data (`vendor_data: {enabled: false}`) also removes the agent, by
  design. A host where the ARP source *errors* rather than answering with
  nothing is the one case this reads differently: with no source left to
  answer, a booting VM retries with `ErrTransient` instead of resting in
  `Provisioning`. It still converges; the giveaway is two "no addresses
  from source" lines per poll at `-v 2`.
  `virsh domifaddr <name> --source agent` and `--source arp` show
  what the operator sees, in that order.
- **Debian 13 cloud images hang in GRUB under BIOS on a headless VM**
  ([Debian bug #1111240](https://bugs.debian.org/1111240)), which is why
  `firmware` defaults to `uefi`; the same image boots under UEFI with the
  serial console intact. Images that need `firmware: bios` are the
  exception.

## Testing

Unit tests run over an in-memory fake of the libvirt seam and an
in-process SSH server. The optional integration test against a real host
is described in
[testing.md](testing.md#the-kvm2-integration-test-real-hypervisor); run it
with `make test-integration-kvm2`. `TestKVM2Check_realHost` runs the
health check against the same host and creates nothing:
`go test -tags=integration -count=1 -v -run TestKVM2Check ./internal/provider/kvm2/`.