# The kvm2 provider Runs Managed proxies as VMs on your own libvirt/KVM hypervisors. It is the second-generation libvirt provider, built clean-room from [provider-contract.md](provider-contract.md) and the design in [kvm2-provider-spec.md](kvm2-provider-spec.md), and it ships side by side with the first-generation `kvm` provider: both types can be configured at once, both take the same config block shape, and both read the same `kvm-ssh` Secret. What kvm2 adds over kvm: a child span per libvirt RPC, a pool-space precheck that maps to the reconciler's quota wait, SSH credentials re-read on every dial, image downloads and health checks on connections of their own, an adoption guard on the owning UID, and SSH authentication failures that stop retrying. Like kvm, the operator reaches each host over SSH, forwards the libvirt unix socket through that connection and speaks libvirt's own RPC over it, so the hosts need nothing beyond `sshd` and `libvirtd` and the operator image carries no libvirt libraries. VMs boot a stock distro cloud image that the operator downloads into the host's storage pool on first use, with the Proxy's cloud-init delivered on a NoCloud seed ISO built in-process. ## Hypervisor prerequisites Per host, once: 1. **libvirt and KVM** (`libvirtd` running, `/dev/kvm` present). 2. **An SSH user in the `libvirt` group**, so it may open `/var/run/libvirt/libvirt-sock`. The operator authenticates with a private key; nothing else on the host is touched. The socket is used as `qemu:///system`, so no further host permissions are needed. 3. **A bridge to a network with DHCP** (for example `br0` bridged onto the LAN). Each VM gets one virtio NIC on it and takes its IP from that network's DHCP. The operator's pod network and the crawler clients must route to those IPs: the health engine probes `http://:` through the proxy and clients get the same address from the discovery API. Nothing on the host filters the port; the provider does not use `spec.port` at all. 4. **A storage pool** (default `default`, usually `/var/lib/libvirt/images`) with room for one base image per distro and provider entry (a few hundred MB each) plus the per-VM overlays (`diskSizeGb` each, thin-provisioned). The pool must be running: a stopped pool fails the health check with `PoolNotActive` and fails `Create` permanently. 5. **UEFI firmware for guests**: the `ovmf` package on Debian/Ubuntu, `edk2-ovmf` on Fedora/RHEL. VMs boot UEFI by default with secure boot off (libvirt picks the OVMF binary itself); a domain start fails with "no firmware found" without it. Only `firmware: bios` images skip this. 6. **Outbound internet from the VMs**, in practice: a vendor-data script installs `qemu-guest-agent` from the distro mirrors on first boot, and the agent is how the operator learns the VM's address. Without it the operator falls back to the host's ARP table, which only sees traffic addressed to the host itself, so on a plain bridge a VM without the agent stays `Provisioning`. Check from the operator's point of view before wiring it in: ```sh ssh novakj@beelink virsh -c qemu:///system list --all # user can drive libvirt ssh novakj@beelink virsh -c qemu:///system pool-info default ssh novakj@beelink ip link show br0 ``` ## Cluster-side setup 1. Create the SSH Secret from a key pair dedicated to the operator and a `known_hosts` file holding every host's key. It is the same `kvm-ssh` Secret the `kvm` provider uses; one Secret serves both types, and if you already have it there is nothing to do here. ```sh # once: a key pair for the operator; the .pub goes on every host ssh-keygen -t ed25519 -f ~/.ssh/proxy-operator_ed25519 -N '' -C proxy-operator ssh-copy-id -i ~/.ssh/proxy-operator_ed25519.pub novakj@beelink # once per host: its host key, under the address the *operator* will # dial (hosts[].address), which may differ from what your laptop uses ssh-keyscan -H beelink.tail8c69c.ts.net >> ~/.ssh/kvm_known_hosts # the Secret: the PRIVATE key (not the .pub) plus known_hosts kubectl -n egress-proxies-operator-system create secret generic kvm-ssh \ --from-file=id_ed25519=$HOME/.ssh/proxy-operator_ed25519 \ --from-file=known_hosts=$HOME/.ssh/kvm_known_hosts ``` `make kvm-ssh-secret` runs that last command as an apply, so it also updates an existing Secret; override the paths with `KVM_SSH_KEY=... KVM_KNOWN_HOSTS=...`. Putting the `.pub` in the Secret is the classic mistake: the target's `kvm-ssh-secret` check refuses a file without `PRIVATE KEY` in it, and the provider itself would fail every call permanently with a "parsing ssh key file" error. Without `known_hosts`, set `insecureIgnoreHostKey: true` and accept that a MITM could hand you a fake hypervisor. `config/manager/manager.yaml` mounts the Secret (optional) at `/etc/proxy-operator/kvm` as two files, `id_ed25519` and `known_hosts`, which is what `sshKeyFile` and `knownHostsFile` in the provider config point at. The deployment starts fine without the Secret, even with a kvm2 provider configured: `New` only checks that the files exist and logs a `WARNING:` line when one does not; the provider's proxies fail transiently and recover once the Secret lands. **Rotating the key or a host key needs no restart.** Both files are read and parsed on every dial (`loadCredentials` in `internal/provider/kvm2/ssh.go`); nothing is cached. Re-run the command or the make target, and the next connection, health check included, uses the new material. Connections that are already open stay open until they drop. 2. Add a provider entry to the providers config (`config/manager/providers_config.yaml` in-cluster; full sample in `config/samples/providers-config.yaml`). The type is `kvm2` and the block is `kvm2:`; its fields are those of the `kvm` block: ```yaml - name: kvm2-home type: kvm2 kvm2: sshKeyFile: /etc/proxy-operator/kvm/id_ed25519 knownHostsFile: /etc/proxy-operator/kvm/known_hosts images: debian-13: url: https://cloud.debian.org/images/cloud/trixie/latest/debian-13-generic-amd64.qcow2 machineTypes: small: {vcpus: 1, memoryMiB: 1024} hosts: - name: beelink address: beelink.tail8c69c.ts.net:22 user: novakj bridge: br0 ``` Validation messages name the block, so a mistake reads `kvm2.hosts[0]: bridge is required`, not `kvm.hosts`. The providers config is read once per process: adding a host or an image entry needs a manager restart; credentials do not. 3. Apply a Proxy (`config/samples/proxy_kvm2.yaml`) and watch it: ```sh kubectl get px -w # NAME MODE PROVIDER PHASE IP HEALTHY # proxy-kvm2-sample Managed kvm2-home Provisioning # proxy-kvm2-sample Managed kvm2-home Running 192.168.1.73 # proxy-kvm2-sample Managed kvm2-home Ready 192.168.1.73 True ``` The first Proxy for a given image on a given host sits in `Provisioning` for the length of the image download; the operator log shows `fetching base image into host pool` with the URL and size and `base image cached` with the size, duration and digest. ## Config reference The block is `kvm2:` and its Go type is the shared `KVMConfig` in `internal/provider/config_kvm.go`, so the fields, defaults and validation are the same as for `kvm`. Defaults are applied by `withDefaults` in `internal/provider/kvm2/kvm2.go`. | Field | Required | Default | Meaning | |---|---|---|---| | `sshKeyFile` | yes | | private key used for every host; OpenSSH or PEM, parsed on every dial | | `knownHostsFile` | unless `insecureIgnoreHostKey` | | OpenSSH `known_hosts` to verify host keys against, read on every dial | | `insecureIgnoreHostKey` | | `false` | skip host-key verification entirely; `New` logs a `WARNING:` line when it is on | | `diskSizeGb` | | `10` | capacity of each VM's overlay disk; cloud-init's growpart expands the root fs into it. Also what the pool-space precheck reserves per VM | | `vendorData` | | | extra cloud-init vendor-data placed on every seed next to the Proxy's user-data. Beware that cloud-init drops any top-level key of a vendor-data **cloud-config** that the user-data also defines. Shipped as one part of a MIME multipart alongside the agent script when `installGuestAgent` is on | | `installGuestAgent` | | `true` | add the vendor-data script that installs and starts `qemu-guest-agent`. Turn off for images that ship it already or cannot reach a package mirror; IP discovery then depends on the host's ARP table | | `images.` | | | the key names the image in `placement.image`. Letters, digits, dots, dashes and underscores only: it becomes the cached volume name `base-.` | | `images..url` | yes per entry | | http(s) URL of a cloud image (qcow2 or raw). The server must send `Content-Length` | | `images..sha256` | | | expected digest, 64 hex characters; verified after download, a change triggers a re-download | | `images..format` | | `qcow2` | `qcow2` or `raw` | | `images..firmware` | | `uefi` | `uefi` (OVMF, secure boot off) or `bios` (SeaBIOS) for images without an EFI system partition | | `machineTypes.` | at least one | | `{vcpus, memoryMiB}` size profile, both greater than zero | | `hosts[].name` | yes | | the `placement.zone` value that selects this host; unique. Letters, digits, dots, dashes and underscores only: it becomes part of the providerID `hosts//domains/` | | `hosts[].address` | yes | | SSH endpoint, `host:port` | | `hosts[].user` | yes | | SSH user, member of the `libvirt` group | | `hosts[].socket` | | `/var/run/libvirt/libvirt-sock` | libvirt socket path on the host | | `hosts[].pool` | | `default` | storage pool for base images, overlays and seeds | | `hosts[].bridge` | yes | | host bridge for the VM's NIC | ## Placement All three fields are checked before any host is dialled. An unknown value is `ErrPermanent`, which the reconciler latches as `Failed` with the message below until the spec changes. | Proxy field | Meaning for kvm2 | Error when unknown | |---|---|---| | `placement.zone` | host name from `hosts`. May be omitted when exactly one host is configured; otherwise required | `placement.zone "x" is not in kvm2.hosts`; with several hosts and no zone, `placement.zone is required when kvm2.hosts lists more than one host` | | `placement.machineType` | key in `machineTypes` | `placement.machineType "x" is not in kvm2.machineTypes` | | `placement.image` | key in `images`, or an `http(s)://` URL used directly with the defaults (`qcow2`, `uefi`) | `placement.image "x" is neither a key in kvm2.images nor an http(s) URL` | | `placement.region` | ignored. It is still part of the spec hash, so changing it replaces the VM and does nothing else | | The messages come from `Create` in `internal/provider/kvm2/create.go`, `host` in `kvm2.go` and `resolveImage` in `image.go`. Fix the spec (which replaces the VM) or the config. Catalogue edits affect only new VMs: the spec hash covers the placement values, not what the config maps them to. ## What lands on the host For a Proxy whose deterministic name is `proxy-abc…`, on the chosen host's pool: | Object | Name | Lifetime | |---|---|---| | base image | `base--.`, where `` is the provider entry's name (for a URL image, `` is the first 16 hex characters of the URL's sha256) | cached; survives Proxy deletion | | image marker | `base--..sha256`, a 64-byte raw volume holding the digest | with the base image | | overlay disk | `proxy-abc….qcow2`, qcow2 backed by the base file, `diskSizeGb` capacity | with the Proxy | | seed ISO | `proxy-abc…-seed.iso`, raw | with the Proxy | | domain | `proxy-abc…` | with the Proxy | The names are built in `internal/provider/kvm2/volume.go`. The base image carries the **provider entry's name** because the file is shared by every Proxy on that host but must not be shared by two entries: the `kvm` provider builds `base-.` in the same pool from the same catalogue shape, and two kvm2 entries pointing one key at different URLs would otherwise agree on the name and disagree on the contents. The cost is one copy of each image per entry per host. For that reason the entry's name must itself be a usable file name — letters, digits, dots, dashes and underscores — which `New` checks at startup, since provider names are otherwise only required to be non-empty and unique. The domain (`buildDomainXML` in `xml.go`) is a `q35` machine with a host-passthrough CPU, ACPI and APIC, memory in MiB and vcpus from the machine type, the overlay on `vda` (virtio) and the seed as a read-only SATA cdrom on `sda`, one virtio NIC on the bridge, a serial console on a pty, the guest-agent channel `org.qemu.guest_agent.0`, a virtio RNG fed from `/dev/urandom`, and no graphics. Both disks are `type="file"` with the paths `StorageVolGetPath` returned: AppArmor derives QEMU's profile from those paths, and volume-type disks fail with permission denied on stock Debian and Ubuntu hosts. UEFI images get `` with the `secure-boot` feature disabled; BIOS images get neither. Its `` carries one element in kvm2's own namespace, `https://gitea.home.hrajfrisbee.cz/kacerr/egress-proxies-operator/kvm2` (`metadataURI` in `metadata.go`): ```xml true 0f3a… 2026-09-16T17:42:11Z ``` That is how orphan GC recognises its own VMs, since libvirt has no labels, and how a repeat `Create` knows whether an existing domain of the same name belongs to this Proxy: a domain without the element is a permanent error (`carries no kvm2 mark; it was not created by this provider`), a different non-empty `uid` is a permanent error naming both UIDs, and a marked domain with a matching or empty `uid` is adopted. The namespace is kvm2's own, so the `kvm` provider's domains are invisible to it (see *Migrating from kvm*). The seed (`buildSeedISO` in `seed.go`) is an ISO 9660 image labelled `cidata` with three files. `user-data` is the Proxy's `cloudInit`, **byte for byte**: any format cloud-init accepts works, including a Jinja template, a MIME archive, a shell script or a gzipped payload; the operator never parses or re-renders it. `meta-data` holds `instance-id: ` and `local-hostname: `, so a reboot is never taken for a new instance. `vendor-data` carries the guest-agent bootstrap. The ISO is staged in a temp directory under `/tmp` (the manager's 64Mi emptyDir), assembled in memory (tens of KiB) and uploaded as a raw volume; the staging directory is removed before the call returns. The seed is rebuilt on every `Create` attempt, because a previous attempt may have been cut off mid-upload and a truncated seed would boot a VM without cloud-init. The guest agent rides along in `vendor-data` as a **shell script** (`guestAgentScript` in `seed.go`), not a cloud-config, on purpose: cloud-init merges a vendor-data cloud-config under the user-data with mergers that keep the user-data's value for every top-level key both define and never append lists, so a Proxy carrying its own `packages:` would silently lose the agent. Scripts bypass the merge. The script is idempotent, picks whichever of `apt-get`, `dnf`, `yum`, `zypper` or `apk` the image has, enables the service with `systemctl` or OpenRC, and logs into `/var/log/cloud-init-output.log`. With `vendorData` also configured, the two travel as separate parts of a MIME multipart (`text/plain` for your document, `text/x-shellscript` for the script), with the `MIME-Version` header first so cloud-init detects it. With `installGuestAgent: false` the file is your `vendorData` alone, or empty. ## The image cache The base image is downloaded **by the operator, through libvirt**, into the host's pool the first time a Proxy asks for it on that host. Nothing is pre-placed on the hypervisor and the operator needs no filesystem access there. The bytes stream from the URL through the operator into `StorageVolUpload` with a running sha256; nothing is buffered (`fetchImage` and `uploadImage` in `internal/provider/kvm2/image.go`). The upload runs on a **dedicated connection** to the host, dialled for the transfer and closed after it, not on the pooled connection the other operations share. A cancelled or overlong call kills the connection it ran on (see *Timeouts and connections*), and a poll for some other Proxy being cancelled must not cut off a 20-minute upload. A body that fails part way through is reported to the caller **out of band**: the reader handed to `StorageVolUpload` turns any source error into a clean end of stream and keeps the real error for afterwards, so the short upload is caught by the length and digest checks rather than by the stream. This is not tidiness. go-libvirt answers a reader error by parking its stream sender forever on a channel it no longer reads, so handing it one wedges the call and leaks the sender with its 4 MiB buffer; see *Known gaps*. For the same reason the download carries its own deadline, a minute inside the transfer budget, so a stalled mirror ends through its body rather than through the killed socket the budget would otherwise leave behind. The download needs a `Content-Length`: the volume's capacity is declared before the upload starts. The request sends `Accept-Encoding: identity` so a transparently gzipped body cannot hide the length. A response without it, or with length 0, is a permanent error naming the URL, and so is any 4xx status; 429 and 5xx are transient and retried. The marker volume is written **last**, on the pooled connection, after `StoragePoolRefresh` has made libvirt read the file's real format. Its presence is what marks a base complete: a crash mid-download leaves a base volume without a marker, which the next attempt deletes and redoes (`cachedImage`). A re-download also happens when the marker's content is not a 64-character hex digest, or when `sha256` is configured and differs from it. Otherwise the cached copy is used for as long as it exists, even if the URL's content changed upstream: a `latest` URL is pinned at the moment of first fetch, per host. Concurrent Creates for the same image on the same host queue on a per-(host, volume) lock; the first fetches, the rest find the marker. To evict manually: ```sh ssh novakj@beelink virsh -c qemu:///system vol-list --pool default | grep base-kvm2-home- ssh novakj@beelink virsh -c qemu:///system vol-delete --pool default base-kvm2-home-debian-13.qcow2.sha256 ssh novakj@beelink virsh -c qemu:///system vol-delete --pool default base-kvm2-home-debian-13.qcow2 ``` Marker first: a base without its marker is refetched, a marker without its base is cleaned up, and neither leaves a Proxy pointing at nothing. **Warning: do not evict a base volume while Proxies using that image exist on the host.** The overlays back onto the base file, and libvirt will unlink it under running domains without complaint; the guests keep running on the open file descriptor and fail at their next start. Delete those Proxies first. Changing `sha256` on an entry whose base is already cached is **refused**, permanently and by name, rather than resolved by replacing the file — that replacement was the same corruption, just automated. Add the image under a new key and move Proxies over by changing their `placement.image` (which replaces the VM), or delete the Proxies and evict the base by hand. Changing `url` alone does nothing at all: the cache is keyed by entry and key, so the new URL is never fetched. Add a new key for that too. ## Quota: the pool-space precheck Before creating any volume, `Create` reads `StoragePoolGetInfo` and compares the pool's free space with what it is about to commit (`checkPoolSpace` and `neededBytes` in `image.go`): the image's `Content-Length` when the base is not cached yet (found with a `HEAD` request, or a `GET` whose body is closed at once when `HEAD` answers anything but 2xx, since mirrors refuse it with 405, 403 or 501 alike; the `GET`'s status is then what counts), plus `diskSizeGb` in bytes for the overlay, plus 16 MiB for the seed. Below that it returns `ErrQuotaExceeded` with a message naming the pool, the bytes available and the bytes needed. The reconciler answers quota with a wait, not a failure: the Proxy shows `Provisioned=False` with reason `QuotaExceeded` and is retried after `QuotaRetry` (5 minutes), off the error backoff curve and never `Failed`. Free up the pool or lower `diskSizeGb` and the next attempt goes through. The precheck is the only quota signal; libvirt itself reports a full pool as an ordinary error, which would be transient. The health check exposes the same number as `poolAvailableGiB`, so a pool that is filling up shows there before Creates start waiting. A pool that is not running fails the precheck permanently (`storage pool is not running`): start it with `virsh pool-start` and `virsh pool-autostart`. ## Migrating from `kvm` The `kvm2` block is the `kvm` block under another name, so an entry ports by changing `type: kvm` to `type: kvm2` and `kvm:` to `kvm2:`. The `kvm-ssh` Secret is shared; nothing changes on the hosts. What does not port is the VMs. kvm2 marks its domains in its own metadata namespace and neither lists nor adopts domains the `kvm` provider made: `ListByTag` skips them, so orphan GC never sees them, and `Create` refuses a domain of the same name that lacks the kvm2 mark with a permanent error rather than start it (a Proxy recreated under the new entry gets the same deterministic name, so this is what happens when step 1 below is skipped). A Proxy's `spec.provider` is immutable, so an existing Proxy cannot be moved to the new entry anyway. Do it in this order: 1. Delete the Proxies on the `kvm` entry and wait for them to go; the `kvm` provider removes their domains and volumes. 2. Change the entry (or add a second one under a new name and keep the old one for a while; both types may share a host). 3. Restart the manager, then recreate the Proxies with the new `spec.provider`. A `kvm` domain left behind on a host is invisible to kvm2 forever; clean it up on the host with `virsh undefine --nvram ` and `virsh vol-delete` for its `.qcow2` and `-seed.iso`. The base image cache is **not** shared between the two types, and that is deliberate: kvm names its copy `base-.` while kvm2 names its own `base--.`, so kvm2 downloads the image once more per host rather than adopting a file another provider maintains. Nothing needs deleting; the `kvm` base stays where it is and keeps serving the overlays that back onto it. Budget the extra few hundred MB in the pool. ## Logging The provider logs through controller-runtime's logger, named `kvm2`, with `provider` (the configured name) on every line, and `host` and `name` (the domain) on every line about an instance. Every line inside a traced operation also carries `traceID` and `spanID`. - `--zap-log-level=debug` (logr V(1)): - one line per operation outcome: `kvm2 domain ensured running` with the `id`, `kvm2 domain fetched` with `state`, `ip` and `uid`, `kvm2 domain not found`, `kvm2 domain deleted`, `kvm2 domains listed` with `hosts` and `managed` counts, `kvm2 host checked` with `ready`, `reason` and `latency`; plus the steps inside a Create (`kvm2 domain already defined`, `overlay volume created`, `seed volume uploaded` with `bytes`, `domain defined` with `uid` and `firmware`, `domain started`, `fetching base image into host pool`, `base image cached`); - **one `libvirt call` line per libvirt RPC**, written by the decorator in `internal/provider/kvm2/observe.go` inside that RPC's span: `rpc` (the go-libvirt method name, for example `DomainLookupByName`), the identifiers it was given (`domain`, `pool`, `volume`), `duration`, and on failure `error` plus `libvirtCode` and `libvirtError` (the code's name, for example `ErrNoDomain`). A Get is four or five RPCs; a Create with a cached image is around twenty; - connection events from `internal/provider/kvm2/hosts.go`: `libvirt host connected` with `address`, `user` and `duration`, and `libvirt host connect failed` with the same plus `error` and `stage` (`credentials`, `tcp`, `ssh` or `socket`); - `kvm2 backend call failed` with `op` and the raw `error` once per failed operation, before the error is returned; - a line per domain that `ListByTag` skips, and a line when the image marker forces a refetch. - `--zap-log-level=2`: the `libvirt call` lines also carry the domain and volume `xml`, the `metadata` element, the interface `addresses`, the listed `domains`, the pool `state` and sizes and the volume `path`; the `domain defined` line carries the `xml`; `kvm2 create resolved` logs the resolved parameters (pool, bridge, machine type, sizes, image volume, firmware) and `cloudInitBytes`; `kvm2 listed instance` logs each listed domain. - Never logged at any level: the cloud-init content, the SSH key, the known_hosts content, and upload or download bodies. The seed and image uploads report `offset` and `length` only. There is no raw wire tier like the GCP provider's V(5); go-libvirt exposes no hook for it. ## Tracing With an OTLP exporter configured ([architecture.md §10](architecture.md#10-tracing-internaltracing)), every backend call is a child span of the operation span the decorators open (`provider.create`, `provider.get`, `provider.delete`, `provider.list`) or of the monitor's `providerhealth.check`: | Span | Around | Attributes | |---|---|---| | `kvm2.`, for example `kvm2.DomainDefineXML` | one libvirt RPC (`observe.go`) | `kvm2.host`, and the identifiers passed: `kvm2.domain`, `kvm2.pool`, `kvm2.volume`; on failure the status is Error and `kvm2.libvirt_code` carries the libvirt error number | | `kvm2.dial` | the SSH handshake and the libvirt connect (`hosts.go`) | `kvm2.host`, `kvm2.address`, `kvm2.user` | | `kvm2.download` | one image fetch, from the HTTP request to the marker (`image.go`) | `kvm2.host`, `kvm2.volume`, `kvm2.url_host`, `kvm2.sha256_configured`, and `kvm2.bytes` once the response arrived | All spans are client spans. No attribute carries cloud-init, key material or a URL beyond its host. The `libvirt call` log line for an RPC is written inside that RPC's span, so its `spanID` points at the span. ## Timeouts and connections No caller puts a deadline on the context, so every backend call carries its own budget (`defaultBudgets` in `internal/provider/kvm2/backend.go`, applied by `call`): | Call | Budget | |---|---| | TCP connect plus SSH handshake plus libvirt connect | 10 s | | ordinary RPC | 30 s | | `DomainCreate` | 90 s | | `StorageVolUpload` of at most 16 MiB (the seed, the marker) | 60 s | | `StorageVolUpload` of an image, `StorageVolDownload`, and the whole image fetch from HTTP request to marker | 20 m; the caller's ctx may end it earlier | | wait for running after `DomainCreate` | 15 s, polled every 500 ms (`defaultRunningWait` and `defaultRunningPoll` in `create.go`); `DomainCreate` is synchronous, so the first poll normally answers running, and a domain that is shut off, crashed or paused right after the start fails at once rather than waiting out the cap | | the `HEAD` request of the pool-space precheck | 30 s | | the courtesy `ConnectClose` when a connection is closed | 5 s | go-libvirt's calls take no context and block, so the only way to abandon one is to close the connection under it. That is what a cancelled or overlong call does: the adapter marks the connection dead so the next call redials, kills it by closing the SSH client's TCP connection, which takes the forwarded socket with it, and returns `ErrTransient` wrapping the context or deadline error. It marks dead before it kills, and closes the TCP connection rather than the forwarded channel, because both of those are lock-free: closing the channel writes a close message under the channel's write lock, so it would queue behind a transfer stuck in a TCP write to a host that has gone away — for as long as the kernel retransmits, which is the one case the kill exists to survive. The consequence: **other RPCs in flight on that host's pooled connection fail once with `ErrTransient`** and their callers retry on a fresh connection. In the reconciler that is one requeue with backoff; in GC it is one skipped sweep. Two callers are kept off the pooled connection for that reason (`dialFresh` in `hosts.go`): the image upload, because a 435 MB transfer must not die because a poll for another Proxy was cancelled, and the health check, because a check that runs out of the monitor's timeout must close only its own connection. Otherwise there is one pooled SSH plus libvirt connection per host, dialled lazily on the first call that needs it and shared by every operation for that host (go-libvirt multiplexes RPCs on one connection). Concurrent callers wait for one dial and share the result; waiting ends with the caller's context. A `keepalive@openssh.com` request every 15 s marks the connection dead when it fails or gets no reply within another 15 s, as does go-libvirt noticing the socket is gone; the next call redials. Create and Delete for the same domain name are serialised on a per-name lock, so two Deletes (GC and the reconciler) never race on destroy and undefine, and a Delete cannot interleave with a Create's steps; Get and ListByTag take no lock. ## Health check The operator checks every configured host every `--provider-check-interval` (default 1m) and publishes the result in the `ProviderHealth` object named after the provider (`kubectl get ph kvm2-home -o yaml`) and in the `proxy_operator_provider_target_up{provider,target}` gauge. One target per `hosts[]` entry, in config order, all checked concurrently. Each check dials a **fresh connection** to the host and closes it when done (`probeHost` in `internal/provider/kvm2/check.go`); the pooled connection is never touched, so a check that runs out of time cannot fail a reconcile's call in flight, and a fresh dial picks up a rotated Secret at once. The sequence is the SSH and libvirt handshakes, then four RPCs: `ConnectGetLibVersion`, `ConnectGetHostname`, `StoragePoolLookupByName` and `StoragePoolGetInfo` on the host's `pool`. The cost per host per round is one sshd session and those four calls. Every target's `details` carry `user` and `pool`. As facts arrive they gain `libvirtVersion` (decoded `major.minor.micro`), `hostname`, `poolState` (virsh's word: `running`, `inactive`, ...), `poolCapacityGiB` and `poolAvailableGiB`, so a target that fails late in the sequence still says what the host answered before. The free space is the number the quota precheck compares against. A failing target says where the check broke (`checkReason` in `check.go`): | Reason | Meaning | Look at | |---|---|---| | `Reachable` | ready | | | `CredentialsUnavailable` | the key or known_hosts file is missing or does not parse | the `kvm-ssh` Secret (see *Cluster-side setup*); fixed in the cluster, not on the host | | `Unreachable` | the TCP connect failed or ran out of the 10 s dial budget while the round still had time, the SSH transport failed for a reason other than authentication, the forwarded libvirt socket was refused, or the link died mid-check | DNS, Tailscale or a firewall between the cluster and `address`; `libvirtd` running and the `socket` path | | `Unauthorized` | the SSH server rejected the key, or the host key is missing from known_hosts or has changed | `user`'s `authorized_keys`; the known_hosts entry (re-run `ssh-keyscan` and update the Secret) | | `Forbidden` | SSH worked, but libvirt refused the user (`ErrAccessDenied` or `ErrAuthFailed`) | `user` in the `libvirt` group | | `Timeout` | the monitor's `--provider-check-timeout` (default 15s) ran out before the check finished, whichever stage was running, the dial included; or the connection was up and an RPC's own budget ran out | a hung `libvirtd`, a saturated link, or a round timeout shorter than the dial takes | | `PoolNotFound` | there is no pool named `pool` | `virsh pool-list --all` | | `PoolNotActive` | the pool exists but is not running | `virsh pool-start ` and `virsh pool-autostart ` | | `APIError` | any other libvirt error | the message carries libvirt's own text | ## Error classes Every error is classified once, in `classify` in `internal/provider/kvm2/errors.go`, with errors that carry a class from their source (validation, the image fetch, the dial) keeping it. The reconciler latches `ErrPermanent` as `Failed` until the spec changes, waits 5 minutes on `ErrQuotaExceeded`, recreates on `ErrNotFound` from `Get`, and retries everything else with backoff. | Class | What lands there | |---|---| | `ErrPermanent` | placement values not in the catalogue; a malformed providerID; a host no longer in the config (from `Get`; `Delete` treats it as done); libvirt `ErrXMLError`, `ErrNoSupport`, `ErrInvalidArg`, `ErrAuthFailed`, `ErrAccessDenied` and `ErrNoStoragePool`; a pool that is not running; SSH authentication refused (`unable to authenticate`); a host key missing from known_hosts or changed (`knownhosts.KeyError`); a key or known_hosts file that exists but does not parse; a domain of the same name owned by another UID or carrying no kvm2 mark; a domain of the same name that is paused or pmsuspended; corrupt metadata on a domain; an image URL answering 4xx, a response without `Content-Length` or with length 0, a sha256 that does not match the configured one | | `ErrQuotaExceeded` | the pool-space precheck | | `ErrNotFound` | libvirt `ErrNoDomain` or `ErrNoStorageVol` from `Get`, `Delete` or `ListByTag`; inside `Create` the same codes are `ErrTransient`, because the contract forbids NotFound from Create | | `ErrTransient` | everything else: a missing key or known_hosts file (the Secret has not landed yet), TCP and network errors, context cancellation, an abandoned or budget-exceeded call, a connection killed by another caller, libvirt `ErrOperationInvalid` and every unrecognised libvirt code, an image URL answering 429 or 5xx or ending the body early, every address source failing on an active domain, the wait for running expiring or the domain going shut off or crashed right after its start | Error strings name paths, hosts, volumes and libvirt's own text. They never contain key bytes, the known_hosts content or cloud-init. ## Known gaps - **An upload that libvirtd itself fails leaks a goroutine, upstream.** go-libvirt's `requestStream` hands its stream sender an unbuffered channel and then, on the error path, returns without ever reading it: ```go abort := make(chan bool) // rpc.go outErr := make(chan error) go func() { outErr <- l.socket.SendStream(...) }() resp, err = l.processIncomingStream(c, in) if err != nil { abort <- true return resp, err // outErr is never drained } ``` Whenever the daemon errors mid-upload — a pool that fills, a volume removed underneath — the sender parks on `outErr` for good, taking its 4 MiB buffer with it. Closing the socket does not help: both ends are blocked on channel sends, not on I/O. kvm2 avoids the two cases it controls (a failing body, and its own transfer deadline; see *The image cache*), so what is left needs a server-side failure. The fix is two characters upstream, `make(chan bool, 1)` and `make(chan error, 1)`; if it ever bites hard enough, a `replace` carrying that patch is the escape hatch. - **Create blocks**, including the first image download on each host, holding one of the three reconcile workers for its duration (LAN speed, a few hundred MB, once per image per host). Other Proxies keep reconciling on the remaining workers. Warm the cache with one Proxy per image per host before creating many. - **Volumes without a domain are invisible to GC.** A crash between creating the overlay or seed and defining the domain leaves volumes that `ListByTag` cannot see (it lists domains). The same Proxy's next Create reuses the overlay and rebuilds the seed, and its Delete removes both by name even with no domain, but a Proxy CR that is gone for good leaves them for `virsh vol-list` to find. - **A host removed from the config orphans its VMs.** `Delete` for a providerID naming an unconfigured host is a logged no-op success, so the Proxy finishes deleting instead of pinning its finalizer, but the domain and its volumes stay on the hypervisor and orphan GC cannot see them either (it only sweeps configured hosts). Delete the Proxies before retiring a host, or clean up afterwards with `virsh undefine --nvram` plus `virsh vol-delete`. `Get` fails permanently for such a providerID on purpose: a typo in `hosts` must not look like a missing VM and trigger a rebuild elsewhere. - **No adoption of `kvm` domains.** See *Migrating from kvm*. - **A cancelled call drops the host's pooled connection once**, so other RPCs in flight on that host fail once with `ErrTransient` and retry. The image upload and the health check are isolated on their own connections. - **`Content-Length` is required** for image downloads; a mirror that only chunks is rejected permanently. Point the catalogue at one that sends the length. - **A paused or pmsuspended domain is not started.** A repeat `Create` for a domain in that state fails permanently (`is DomainPaused and cannot be started by the operator; resume or destroy it on the host`), so the Proxy shows `Failed` until its spec changes; the operator never resumes a VM an admin paused. Resume it on the host (`virsh resume `) and touch the spec, or delete the Proxy. - **Parser text in credential errors.** A key or known_hosts file that does not parse fails with the parser's own message, which can quote a PEM header type or a malformed known_hosts token. Key bytes are never quoted. - **The sample cloud-init is an open proxy.** `config/samples/proxy_kvm2.yaml` installs Squid with `http_access allow all`; restrict it before exposing the bridge network to anything but the cluster. - **IP discovery needs the agent or guest traffic to the host.** A VM on a bridge with no DHCP never gets an address and stays `Provisioning` forever; check `virsh console ` on the host. The same symptom with a booted, addressed VM means the guest agent never came up: check `systemctl status qemu-guest-agent`, `cloud-init status --long` and the vendor script's output in `/var/log/cloud-init-output.log` (a VM without outbound internet cannot install it). A user-data that disables vendor-data (`vendor_data: {enabled: false}`) also removes the agent, by design. A host where the ARP source *errors* rather than answering with nothing is the one case this reads differently: with no source left to answer, a booting VM retries with `ErrTransient` instead of resting in `Provisioning`. It still converges; the giveaway is two "no addresses from source" lines per poll at `-v 2`. `virsh domifaddr --source agent` and `--source arp` show what the operator sees, in that order. - **Debian 13 cloud images hang in GRUB under BIOS on a headless VM** ([Debian bug #1111240](https://bugs.debian.org/1111240)), which is why `firmware` defaults to `uefi`; the same image boots under UEFI with the serial console intact. Images that need `firmware: bios` are the exception. ## Testing Unit tests run over an in-memory fake of the libvirt seam and an in-process SSH server. The optional integration test against a real host is described in [testing.md](testing.md#the-kvm2-integration-test-real-hypervisor); run it with `make test-integration-kvm2`. `TestKVM2Check_realHost` runs the health check against the same host and creates nothing: `go test -tags=integration -count=1 -v -run TestKVM2Check ./internal/provider/kvm2/`.