egress-proxies-operator/docs/kvm2-provider.md

The kvm2 provider

Runs Managed proxies as VMs on your own libvirt/KVM hypervisors. It is the second-generation libvirt provider, built clean-room from provider-contract.md and the design in kvm2-provider-spec.md, and it ships side by side with the first-generation kvm provider: both types can be configured at once, both take the same config block shape, and both read the same kvm-ssh Secret. What kvm2 adds over kvm: a child span per libvirt RPC, a pool-space precheck that maps to the reconciler's quota wait, SSH credentials re-read on every dial, image downloads and health checks on connections of their own, an adoption guard on the owning UID, and SSH authentication failures that stop retrying.

Like kvm, the operator reaches each host over SSH, forwards the libvirt unix socket through that connection and speaks libvirt's own RPC over it, so the hosts need nothing beyond sshd and libvirtd and the operator image carries no libvirt libraries. VMs boot a stock distro cloud image that the operator downloads into the host's storage pool on first use, with the Proxy's cloud-init delivered on a NoCloud seed ISO built in-process.

Hypervisor prerequisites

Per host, once:

1. libvirt and KVM (libvirtd running, /dev/kvm present).
2. An SSH user in the libvirt group, so it may open /var/run/libvirt/libvirt-sock. The operator authenticates with a private key; nothing else on the host is touched. The socket is used as qemu:///system, so no further host permissions are needed.
3. A bridge to a network with DHCP (for example br0 bridged onto the LAN). Each VM gets one virtio NIC on it and takes its IP from that network's DHCP. The operator's pod network and the crawler clients must route to those IPs: the health engine probes http://<ip>:<port> through the proxy and clients get the same address from the discovery API. Nothing on the host filters the port; the provider does not use spec.port at all.
4. A storage pool (default default, usually /var/lib/libvirt/images) with room for one base image per distro (a few hundred MB) plus the per-VM overlays (diskSizeGb each, thin-provisioned). The pool must be running: a stopped pool fails the health check with PoolNotActive and fails Create permanently.
5. UEFI firmware for guests: the ovmf package on Debian/Ubuntu, edk2-ovmf on Fedora/RHEL. VMs boot UEFI by default with secure boot off (libvirt picks the OVMF binary itself); a domain start fails with "no firmware found" without it. Only firmware: bios images skip this.
6. Outbound internet from the VMs, in practice: a vendor-data script installs qemu-guest-agent from the distro mirrors on first boot, and the agent is how the operator learns the VM's address. Without it the operator falls back to the host's ARP table, which only sees traffic addressed to the host itself, so on a plain bridge a VM without the agent stays Provisioning.

Check from the operator's point of view before wiring it in:

ssh novakj@beelink virsh -c qemu:///system list --all      # user can drive libvirt
ssh novakj@beelink virsh -c qemu:///system pool-info default
ssh novakj@beelink ip link show br0

Cluster-side setup

1. Create the SSH Secret from a key pair dedicated to the operator and a known_hosts file holding every host's key. It is the same kvm-ssh Secret the kvm provider uses; one Secret serves both types, and if you already have it there is nothing to do here.

   # once: a key pair for the operator; the .pub goes on every host
   ssh-keygen -t ed25519 -f ~/.ssh/proxy-operator_ed25519 -N '' -C proxy-operator
   ssh-copy-id -i ~/.ssh/proxy-operator_ed25519.pub novakj@beelink
   
   # once per host: its host key, under the address the *operator* will
   # dial (hosts[].address), which may differ from what your laptop uses
   ssh-keyscan -H beelink.tail8c69c.ts.net >> ~/.ssh/kvm_known_hosts
   
   # the Secret: the PRIVATE key (not the .pub) plus known_hosts
   kubectl -n egress-proxies-operator-system create secret generic kvm-ssh \
     --from-file=id_ed25519=$HOME/.ssh/proxy-operator_ed25519 \
     --from-file=known_hosts=$HOME/.ssh/kvm_known_hosts
   

   make kvm-ssh-secret runs that last command as an apply, so it also updates an existing Secret; override the paths with KVM_SSH_KEY=... KVM_KNOWN_HOSTS=.... Putting the .pub in the Secret is the classic mistake: the target's kvm-ssh-secret check refuses a file without PRIVATE KEY in it, and the provider itself would fail every call permanently with a "parsing ssh key file" error. Without known_hosts, set insecureIgnoreHostKey: true and accept that a MITM could hand you a fake hypervisor.

   config/manager/manager.yaml mounts the Secret (optional) at /etc/proxy-operator/kvm as two files, id_ed25519 and known_hosts, which is what sshKeyFile and knownHostsFile in the provider config point at. The deployment starts fine without the Secret, even with a kvm2 provider configured: New only checks that the files exist and logs a WARNING: line when one does not; the provider's proxies fail transiently and recover once the Secret lands.

   Rotating the key or a host key needs no restart. Both files are read and parsed on every dial (loadCredentials in internal/provider/kvm2/ssh.go); nothing is cached. Re-run the command or the make target, and the next connection, health check included, uses the new material. Connections that are already open stay open until they drop.

2. Add a provider entry to the providers config (config/manager/providers_config.yaml in-cluster; full sample in config/samples/providers-config.yaml). The type is kvm2 and the block is kvm2:; its fields are those of the kvm block:

   - name: kvm2-home
     type: kvm2
     kvm2:
       sshKeyFile: /etc/proxy-operator/kvm/id_ed25519
       knownHostsFile: /etc/proxy-operator/kvm/known_hosts
       images:
         debian-13:
           url: https://cloud.debian.org/images/cloud/trixie/latest/debian-13-generic-amd64.qcow2
       machineTypes:
         small: {vcpus: 1, memoryMiB: 1024}
       hosts:
         - name: beelink
           address: beelink.tail8c69c.ts.net:22
           user: novakj
           bridge: br0
   

   Validation messages name the block, so a mistake reads kvm2.hosts[0]: bridge is required, not kvm.hosts. The providers config is read once per process: adding a host or an image entry needs a manager restart; credentials do not.

3. Apply a Proxy (config/samples/proxy_kvm2.yaml) and watch it:

   kubectl get px -w
   # NAME                MODE      PROVIDER    PHASE          IP             HEALTHY
   # proxy-kvm2-sample   Managed   kvm2-home   Provisioning
   # proxy-kvm2-sample   Managed   kvm2-home   Running        192.168.1.73
   # proxy-kvm2-sample   Managed   kvm2-home   Ready          192.168.1.73   True
   

   The first Proxy for a given image on a given host sits in Provisioning for the length of the image download; the operator log shows fetching base image into host pool with the URL and size and base image cached with the size, duration and digest.

Config reference

The block is kvm2: and its Go type is the shared KVMConfig in internal/provider/config_kvm.go, so the fields, defaults and validation are the same as for kvm. Defaults are applied by withDefaults in internal/provider/kvm2/kvm2.go.

Field	Required	Default	Meaning
sshKeyFile	yes		private key used for every host; OpenSSH or PEM, parsed on every dial
knownHostsFile	unless insecureIgnoreHostKey		OpenSSH known_hosts to verify host keys against, read on every dial
insecureIgnoreHostKey		false	skip host-key verification entirely; New logs a WARNING: line when it is on
diskSizeGb		10	capacity of each VM's overlay disk; cloud-init's growpart expands the root fs into it. Also what the pool-space precheck reserves per VM
vendorData			extra cloud-init vendor-data placed on every seed next to the Proxy's user-data. Beware that cloud-init drops any top-level key of a vendor-data cloud-config that the user-data also defines. Shipped as one part of a MIME multipart alongside the agent script when installGuestAgent is on
installGuestAgent		true	add the vendor-data script that installs and starts qemu-guest-agent. Turn off for images that ship it already or cannot reach a package mirror; IP discovery then depends on the host's ARP table
images.<key>			the key names the image in placement.image. Letters, digits, dots, dashes and underscores only: it becomes the cached volume name base-<key>.<format>
images.<key>.url	yes per entry		http(s) URL of a cloud image (qcow2 or raw). The server must send Content-Length
images.<key>.sha256			expected digest, 64 hex characters; verified after download, a change triggers a re-download
images.<key>.format		qcow2	qcow2 or raw
images.<key>.firmware		uefi	uefi (OVMF, secure boot off) or bios (SeaBIOS) for images without an EFI system partition
machineTypes.<key>	at least one		{vcpus, memoryMiB} size profile, both greater than zero
hosts[].name	yes		the placement.zone value that selects this host; unique. Letters, digits, dots, dashes and underscores only: it becomes part of the providerID hosts/<name>/domains/<vm>
hosts[].address	yes		SSH endpoint, host:port
hosts[].user	yes		SSH user, member of the libvirt group
hosts[].socket		/var/run/libvirt/libvirt-sock	libvirt socket path on the host
hosts[].pool		default	storage pool for base images, overlays and seeds
hosts[].bridge	yes		host bridge for the VM's NIC

Placement

All three fields are checked before any host is dialled. An unknown value is ErrPermanent, which the reconciler latches as Failed with the message below until the spec changes.

Proxy field	Meaning for kvm2	Error when unknown
placement.zone	host name from hosts. May be omitted when exactly one host is configured; otherwise required	placement.zone "x" is not in kvm2.hosts; with several hosts and no zone, placement.zone is required when kvm2.hosts lists more than one host
placement.machineType	key in machineTypes	placement.machineType "x" is not in kvm2.machineTypes
placement.image	key in images, or an http(s):// URL used directly with the defaults (qcow2, uefi)	placement.image "x" is neither a key in kvm2.images nor an http(s) URL
placement.region	ignored. It is still part of the spec hash, so changing it replaces the VM and does nothing else

The messages come from Create in internal/provider/kvm2/create.go, host in kvm2.go and resolveImage in image.go. Fix the spec (which replaces the VM) or the config. Catalogue edits affect only new VMs: the spec hash covers the placement values, not what the config maps them to.

What lands on the host

For a Proxy whose deterministic name is proxy-abc…, on the chosen host's pool:

Object	Name	Lifetime
base image	base-<key>.<format> (for a URL image, <key> is the first 16 hex characters of the URL's sha256)	cached; survives Proxy deletion
image marker	base-<key>.<format>.sha256, a 64-byte raw volume holding the digest	with the base image
overlay disk	proxy-abc….qcow2, qcow2 backed by the base file, diskSizeGb capacity	with the Proxy
seed ISO	proxy-abc…-seed.iso, raw	with the Proxy
domain	proxy-abc…	with the Proxy

The names are built in internal/provider/kvm2/volume.go.

The domain (buildDomainXML in xml.go) is a q35 machine with a host-passthrough CPU, ACPI and APIC, memory in MiB and vcpus from the machine type, the overlay on vda (virtio) and the seed as a read-only SATA cdrom on sda, one virtio NIC on the bridge, a serial console on a pty, the guest-agent channel org.qemu.guest_agent.0, a virtio RNG fed from /dev/urandom, and no graphics. Both disks are type="file" with the paths StorageVolGetPath returned: AppArmor derives QEMU's profile from those paths, and volume-type disks fail with permission denied on stock Debian and Ubuntu hosts. UEFI images get <os firmware="efi"> with the secure-boot feature disabled; BIOS images get neither.

Its <metadata> carries one element in kvm2's own namespace, https://gitea.home.hrajfrisbee.cz/kacerr/egress-proxies-operator/kvm2 (metadataURI in metadata.go):

<instance xmlns="https://gitea.home.hrajfrisbee.cz/kacerr/egress-proxies-operator/kvm2">
  <managed>true</managed>
  <uid>0f3a…</uid>
  <createdAt>2026-09-16T17:42:11Z</createdAt>
</instance>

That is how orphan GC recognises its own VMs, since libvirt has no labels, and how a repeat Create knows whether an existing domain of the same name belongs to this Proxy: a domain without the element is a permanent error (carries no kvm2 mark; it was not created by this provider), a different non-empty uid is a permanent error naming both UIDs, and a marked domain with a matching or empty uid is adopted. The namespace is kvm2's own, so the kvm provider's domains are invisible to it (see Migrating from kvm).

The seed (buildSeedISO in seed.go) is an ISO 9660 image labelled cidata with three files. user-data is the Proxy's cloudInit, byte for byte: any format cloud-init accepts works, including a Jinja template, a MIME archive, a shell script or a gzipped payload; the operator never parses or re-renders it. meta-data holds instance-id: <name> and local-hostname: <name>, so a reboot is never taken for a new instance. vendor-data carries the guest-agent bootstrap. The ISO is staged in a temp directory under /tmp (the manager's 64Mi emptyDir), assembled in memory (tens of KiB) and uploaded as a raw volume; the staging directory is removed before the call returns. The seed is rebuilt on every Create attempt, because a previous attempt may have been cut off mid-upload and a truncated seed would boot a VM without cloud-init.

The guest agent rides along in vendor-data as a shell script (guestAgentScript in seed.go), not a cloud-config, on purpose: cloud-init merges a vendor-data cloud-config under the user-data with mergers that keep the user-data's value for every top-level key both define and never append lists, so a Proxy carrying its own packages: would silently lose the agent. Scripts bypass the merge. The script is idempotent, picks whichever of apt-get, dnf, yum, zypper or apk the image has, enables the service with systemctl or OpenRC, and logs into /var/log/cloud-init-output.log. With vendorData also configured, the two travel as separate parts of a MIME multipart (text/plain for your document, text/x-shellscript for the script), with the MIME-Version header first so cloud-init detects it. With installGuestAgent: false the file is your vendorData alone, or empty.

The image cache

The base image is downloaded by the operator, through libvirt, into the host's pool the first time a Proxy asks for it on that host. Nothing is pre-placed on the hypervisor and the operator needs no filesystem access there. The bytes stream from the URL through the operator into StorageVolUpload with a running sha256; nothing is buffered (fetchImage and uploadImage in internal/provider/kvm2/image.go).

The upload runs on a dedicated connection to the host, dialled for the transfer and closed after it, not on the pooled connection the other operations share. A cancelled or overlong call kills the connection it ran on (see Timeouts and connections), and a poll for some other Proxy being cancelled must not cut off a 20-minute upload.

A body that fails part way through is reported to the caller out of band: the reader handed to StorageVolUpload turns any source error into a clean end of stream and keeps the real error for afterwards, so the short upload is caught by the length and digest checks rather than by the stream. This is not tidiness. go-libvirt answers a reader error by parking its stream sender forever on a channel it no longer reads, so handing it one wedges the call and leaks the sender with its 4 MiB buffer; see Known gaps. For the same reason the download carries its own deadline, a minute inside the transfer budget, so a stalled mirror ends through its body rather than through the killed socket the budget would otherwise leave behind.

The download needs a Content-Length: the volume's capacity is declared before the upload starts. The request sends Accept-Encoding: identity so a transparently gzipped body cannot hide the length. A response without it, or with length 0, is a permanent error naming the URL, and so is any 4xx status; 429 and 5xx are transient and retried.

The marker volume is written last, on the pooled connection, after StoragePoolRefresh has made libvirt read the file's real format. Its presence is what marks a base complete: a crash mid-download leaves a base volume without a marker, which the next attempt deletes and redoes (cachedImage). A re-download also happens when the marker's content is not a 64-character hex digest, or when sha256 is configured and differs from it. Otherwise the cached copy is used for as long as it exists, even if the URL's content changed upstream: a latest URL is pinned at the moment of first fetch, per host. Concurrent Creates for the same image on the same host queue on a per-(host, volume) lock; the first fetches, the rest find the marker.

To evict manually:

ssh novakj@beelink virsh -c qemu:///system vol-delete --pool default base-debian-13.qcow2.sha256
ssh novakj@beelink virsh -c qemu:///system vol-delete --pool default base-debian-13.qcow2

Warning: do not change an image entry's url or sha256, and do not evict its base volume, while Proxies using that image exist on the host. The overlays back onto the base file; deleting or replacing it corrupts every VM that uses it. Delete those Proxies first, or add the new image under a new key and move Proxies over by changing their placement.image (which replaces the VM).

Quota: the pool-space precheck

Before creating any volume, Create reads StoragePoolGetInfo and compares the pool's free space with what it is about to commit (checkPoolSpace and neededBytes in image.go): the image's Content-Length when the base is not cached yet (found with a HEAD request, or a GET whose body is closed at once when HEAD answers anything but 2xx, since mirrors refuse it with 405, 403 or 501 alike; the GET's status is then what counts), plus diskSizeGb in bytes for the overlay, plus 16 MiB for the seed. Below that it returns ErrQuotaExceeded with a message naming the pool, the bytes available and the bytes needed. The reconciler answers quota with a wait, not a failure: the Proxy shows Provisioned=False with reason QuotaExceeded and is retried after QuotaRetry (5 minutes), off the error backoff curve and never Failed. Free up the pool or lower diskSizeGb and the next attempt goes through.

The precheck is the only quota signal; libvirt itself reports a full pool as an ordinary error, which would be transient. The health check exposes the same number as poolAvailableGiB, so a pool that is filling up shows there before Creates start waiting.

A pool that is not running fails the precheck permanently (storage pool <pool> is not running): start it with virsh pool-start and virsh pool-autostart.

Migrating from kvm

The kvm2 block is the kvm block under another name, so an entry ports by changing type: kvm to type: kvm2 and kvm: to kvm2:. The kvm-ssh Secret is shared; nothing changes on the hosts.

What does not port is the VMs. kvm2 marks its domains in its own metadata namespace and neither lists nor adopts domains the kvm provider made: ListByTag skips them, so orphan GC never sees them, and Create refuses a domain of the same name that lacks the kvm2 mark with a permanent error rather than start it (a Proxy recreated under the new entry gets the same deterministic name, so this is what happens when step 1 below is skipped). A Proxy's spec.provider is immutable, so an existing Proxy cannot be moved to the new entry anyway. Do it in this order:

1. Delete the Proxies on the kvm entry and wait for them to go; the kvm provider removes their domains and volumes.
2. Change the entry (or add a second one under a new name and keep the old one for a while; both types may share a host).
3. Restart the manager, then recreate the Proxies with the new spec.provider.

A kvm domain left behind on a host is invisible to kvm2 forever; clean it up on the host with virsh undefine --nvram <name> and virsh vol-delete for its <name>.qcow2 and <name>-seed.iso.

Logging

The provider logs through controller-runtime's logger, named kvm2, with provider (the configured name) on every line, and host and name (the domain) on every line about an instance. Every line inside a traced operation also carries traceID and spanID.

• --zap-log-level=debug (logr V(1)):
  • one line per operation outcome: kvm2 domain ensured running with the id, kvm2 domain fetched with state, ip and uid, kvm2 domain not found, kvm2 domain deleted, kvm2 domains listed with hosts and managed counts, kvm2 host checked with ready, reason and latency; plus the steps inside a Create (kvm2 domain already defined, overlay volume created, seed volume uploaded with bytes, domain defined with uid and firmware, domain started, fetching base image into host pool, base image cached);
  • one libvirt call line per libvirt RPC, written by the decorator in internal/provider/kvm2/observe.go inside that RPC's span: rpc (the go-libvirt method name, for example DomainLookupByName), the identifiers it was given (domain, pool, volume), duration, and on failure error plus libvirtCode and libvirtError (the code's name, for example ErrNoDomain). A Get is four or five RPCs; a Create with a cached image is around twenty;
  • connection events from internal/provider/kvm2/hosts.go: libvirt host connected with address, user and duration, and libvirt host connect failed with the same plus error and stage (credentials, tcp, ssh or socket);
  • kvm2 backend call failed with op and the raw error once per failed operation, before the error is returned;
  • a line per domain that ListByTag skips, and a line when the image marker forces a refetch.
• --zap-log-level=2: the libvirt call lines also carry the domain and volume xml, the metadata element, the interface addresses, the listed domains, the pool state and sizes and the volume path; the domain defined line carries the xml; kvm2 create resolved logs the resolved parameters (pool, bridge, machine type, sizes, image volume, firmware) and cloudInitBytes; kvm2 listed instance logs each listed domain.
• Never logged at any level: the cloud-init content, the SSH key, the known_hosts content, and upload or download bodies. The seed and image uploads report offset and length only. There is no raw wire tier like the GCP provider's V(5); go-libvirt exposes no hook for it.

Tracing

With an OTLP exporter configured (architecture.md §10), every backend call is a child span of the operation span the decorators open (provider.create, provider.get, provider.delete, provider.list) or of the monitor's providerhealth.check:

Span	Around	Attributes
kvm2.<Method>, for example kvm2.DomainDefineXML	one libvirt RPC (observe.go)	kvm2.host, and the identifiers passed: kvm2.domain, kvm2.pool, kvm2.volume; on failure the status is Error and kvm2.libvirt_code carries the libvirt error number
kvm2.dial	the SSH handshake and the libvirt connect (hosts.go)	kvm2.host, kvm2.address, kvm2.user
kvm2.download	one image fetch, from the HTTP request to the marker (image.go)	kvm2.host, kvm2.volume, kvm2.url_host, kvm2.sha256_configured, and kvm2.bytes once the response arrived

All spans are client spans. No attribute carries cloud-init, key material or a URL beyond its host. The libvirt call log line for an RPC is written inside that RPC's span, so its spanID points at the span.

Timeouts and connections

No caller puts a deadline on the context, so every backend call carries its own budget (defaultBudgets in internal/provider/kvm2/backend.go, applied by call):

Call	Budget
TCP connect plus SSH handshake plus libvirt connect	10 s
ordinary RPC	30 s
DomainCreate	90 s
StorageVolUpload of at most 16 MiB (the seed, the marker)	60 s
StorageVolUpload of an image, StorageVolDownload, and the whole image fetch from HTTP request to marker	20 m; the caller's ctx may end it earlier
wait for running after DomainCreate	15 s, polled every 500 ms (defaultRunningWait and defaultRunningPoll in create.go); DomainCreate is synchronous, so the first poll normally answers running, and a domain that is shut off, crashed or paused right after the start fails at once rather than waiting out the cap
the HEAD request of the pool-space precheck	30 s
the courtesy ConnectClose when a connection is closed	5 s

go-libvirt's calls take no context and block, so the only way to abandon one is to close the connection under it. That is what a cancelled or overlong call does: the adapter marks the connection dead so the next call redials, kills it by closing the SSH client's TCP connection, which takes the forwarded socket with it, and returns ErrTransient wrapping the context or deadline error. It marks dead before it kills, and closes the TCP connection rather than the forwarded channel, because both of those are lock-free: closing the channel writes a close message under the channel's write lock, so it would queue behind a transfer stuck in a TCP write to a host that has gone away — for as long as the kernel retransmits, which is the one case the kill exists to survive. The consequence: other RPCs in flight on that host's pooled connection fail once with ErrTransient and their callers retry on a fresh connection. In the reconciler that is one requeue with backoff; in GC it is one skipped sweep.

Two callers are kept off the pooled connection for that reason (dialFresh in hosts.go): the image upload, because a 435 MB transfer must not die because a poll for another Proxy was cancelled, and the health check, because a check that runs out of the monitor's timeout must close only its own connection.

Otherwise there is one pooled SSH plus libvirt connection per host, dialled lazily on the first call that needs it and shared by every operation for that host (go-libvirt multiplexes RPCs on one connection). Concurrent callers wait for one dial and share the result; waiting ends with the caller's context. A keepalive@openssh.com request every 15 s marks the connection dead when it fails or gets no reply within another 15 s, as does go-libvirt noticing the socket is gone; the next call redials. Create and Delete for the same domain name are serialised on a per-name lock, so two Deletes (GC and the reconciler) never race on destroy and undefine, and a Delete cannot interleave with a Create's steps; Get and ListByTag take no lock.

Health check

The operator checks every configured host every --provider-check-interval (default 1m) and publishes the result in the ProviderHealth object named after the provider (kubectl get ph kvm2-home -o yaml) and in the proxy_operator_provider_target_up{provider,target} gauge. One target per hosts[] entry, in config order, all checked concurrently.

Each check dials a fresh connection to the host and closes it when done (probeHost in internal/provider/kvm2/check.go); the pooled connection is never touched, so a check that runs out of time cannot fail a reconcile's call in flight, and a fresh dial picks up a rotated Secret at once. The sequence is the SSH and libvirt handshakes, then four RPCs: ConnectGetLibVersion, ConnectGetHostname, StoragePoolLookupByName and StoragePoolGetInfo on the host's pool. The cost per host per round is one sshd session and those four calls.

Every target's details carry user and pool. As facts arrive they gain libvirtVersion (decoded major.minor.micro), hostname, poolState (virsh's word: running, inactive, ...), poolCapacityGiB and poolAvailableGiB, so a target that fails late in the sequence still says what the host answered before. The free space is the number the quota precheck compares against.

A failing target says where the check broke (checkReason in check.go):

Reason	Meaning	Look at
Reachable	ready
CredentialsUnavailable	the key or known_hosts file is missing or does not parse	the kvm-ssh Secret (see Cluster-side setup); fixed in the cluster, not on the host
Unreachable	the TCP connect failed or ran out of the 10 s dial budget while the round still had time, the SSH transport failed for a reason other than authentication, the forwarded libvirt socket was refused, or the link died mid-check	DNS, Tailscale or a firewall between the cluster and address; libvirtd running and the socket path
Unauthorized	the SSH server rejected the key, or the host key is missing from known_hosts or has changed	user's authorized_keys; the known_hosts entry (re-run ssh-keyscan and update the Secret)
Forbidden	SSH worked, but libvirt refused the user (ErrAccessDenied or ErrAuthFailed)	user in the libvirt group
Timeout	the monitor's --provider-check-timeout (default 15s) ran out before the check finished, whichever stage was running, the dial included; or the connection was up and an RPC's own budget ran out	a hung libvirtd, a saturated link, or a round timeout shorter than the dial takes
PoolNotFound	there is no pool named pool	virsh pool-list --all
PoolNotActive	the pool exists but is not running	virsh pool-start <pool> and virsh pool-autostart <pool>
APIError	any other libvirt error	the message carries libvirt's own text

Error classes

Every error is classified once, in classify in internal/provider/kvm2/errors.go, with errors that carry a class from their source (validation, the image fetch, the dial) keeping it. The reconciler latches ErrPermanent as Failed until the spec changes, waits 5 minutes on ErrQuotaExceeded, recreates on ErrNotFound from Get, and retries everything else with backoff.

Class	What lands there
ErrPermanent	placement values not in the catalogue; a malformed providerID; a host no longer in the config (from Get; Delete treats it as done); libvirt ErrXMLError, ErrNoSupport, ErrInvalidArg, ErrAuthFailed, ErrAccessDenied and ErrNoStoragePool; a pool that is not running; SSH authentication refused (unable to authenticate); a host key missing from known_hosts or changed (knownhosts.KeyError); a key or known_hosts file that exists but does not parse; a domain of the same name owned by another UID or carrying no kvm2 mark; a domain of the same name that is paused or pmsuspended; corrupt metadata on a domain; an image URL answering 4xx, a response without Content-Length or with length 0, a sha256 that does not match the configured one
ErrQuotaExceeded	the pool-space precheck
ErrNotFound	libvirt ErrNoDomain or ErrNoStorageVol from Get, Delete or ListByTag; inside Create the same codes are ErrTransient, because the contract forbids NotFound from Create
ErrTransient	everything else: a missing key or known_hosts file (the Secret has not landed yet), TCP and network errors, context cancellation, an abandoned or budget-exceeded call, a connection killed by another caller, libvirt ErrOperationInvalid and every unrecognised libvirt code, an image URL answering 429 or 5xx or ending the body early, every address source failing on an active domain, the wait for running expiring or the domain going shut off or crashed right after its start

Error strings name paths, hosts, volumes and libvirt's own text. They never contain key bytes, the known_hosts content or cloud-init.

Known gaps

• An upload that libvirtd itself fails leaks a goroutine, upstream. go-libvirt's requestStream hands its stream sender an unbuffered channel and then, on the error path, returns without ever reading it:

  abort  := make(chan bool)    // rpc.go
  outErr := make(chan error)
  go func() { outErr <- l.socket.SendStream(...) }()
  resp, err = l.processIncomingStream(c, in)
  if err != nil {
      abort <- true
      return resp, err          // outErr is never drained
  }
  

  Whenever the daemon errors mid-upload — a pool that fills, a volume removed underneath — the sender parks on outErr for good, taking its 4 MiB buffer with it. Closing the socket does not help: both ends are blocked on channel sends, not on I/O. kvm2 avoids the two cases it controls (a failing body, and its own transfer deadline; see The image cache), so what is left needs a server-side failure. The fix is two characters upstream, make(chan bool, 1) and make(chan error, 1); if it ever bites hard enough, a replace carrying that patch is the escape hatch.

• Create blocks, including the first image download on each host, holding one of the three reconcile workers for its duration (LAN speed, a few hundred MB, once per image per host). Other Proxies keep reconciling on the remaining workers. Warm the cache with one Proxy per image per host before creating many.

• Volumes without a domain are invisible to GC. A crash between creating the overlay or seed and defining the domain leaves volumes that ListByTag cannot see (it lists domains). The same Proxy's next Create reuses the overlay and rebuilds the seed, and its Delete removes both by name even with no domain, but a Proxy CR that is gone for good leaves them for virsh vol-list to find.

• A host removed from the config orphans its VMs. Delete for a providerID naming an unconfigured host is a logged no-op success, so the Proxy finishes deleting instead of pinning its finalizer, but the domain and its volumes stay on the hypervisor and orphan GC cannot see them either (it only sweeps configured hosts). Delete the Proxies before retiring a host, or clean up afterwards with virsh undefine --nvram plus virsh vol-delete. Get fails permanently for such a providerID on purpose: a typo in hosts must not look like a missing VM and trigger a rebuild elsewhere.

• No adoption of kvm domains. See Migrating from kvm.

• A cancelled call drops the host's pooled connection once, so other RPCs in flight on that host fail once with ErrTransient and retry. The image upload and the health check are isolated on their own connections.

• Content-Length is required for image downloads; a mirror that only chunks is rejected permanently. Point the catalogue at one that sends the length.

• A paused or pmsuspended domain is not started. A repeat Create for a domain in that state fails permanently (is DomainPaused and cannot be started by the operator; resume or destroy it on the host), so the Proxy shows Failed until its spec changes; the operator never resumes a VM an admin paused. Resume it on the host (virsh resume <name>) and touch the spec, or delete the Proxy.

• Parser text in credential errors. A key or known_hosts file that does not parse fails with the parser's own message, which can quote a PEM header type or a malformed known_hosts token. Key bytes are never quoted.

• The sample cloud-init is an open proxy. config/samples/proxy_kvm2.yaml installs Squid with http_access allow all; restrict it before exposing the bridge network to anything but the cluster.

• IP discovery needs the agent or guest traffic to the host. A VM on a bridge with no DHCP never gets an address and stays Provisioning forever; check virsh console <name> on the host. The same symptom with a booted, addressed VM means the guest agent never came up: check systemctl status qemu-guest-agent, cloud-init status --long and the vendor script's output in /var/log/cloud-init-output.log (a VM without outbound internet cannot install it). A user-data that disables vendor-data (vendor_data: {enabled: false}) also removes the agent, by design. A host where the ARP source errors rather than answering with nothing is the one case this reads differently: with no source left to answer, a booting VM retries with ErrTransient instead of resting in Provisioning. It still converges; the giveaway is two "no addresses from source" lines per poll at -v 2. virsh domifaddr <name> --source agent and --source arp show what the operator sees, in that order.

• Debian 13 cloud images hang in GRUB under BIOS on a headless VM (Debian bug #1111240), which is why firmware defaults to uefi; the same image boots under UEFI with the serial console intact. Images that need firmware: bios are the exception.

Testing

Unit tests run over an in-memory fake of the libvirt seam and an in-process SSH server. The optional integration test against a real host is described in testing.md; run it with make test-integration-kvm2. TestKVM2Check_realHost runs the health check against the same host and creates nothing: go test -tags=integration -count=1 -v -run TestKVM2Check ./internal/provider/kvm2/.