egress-proxies-operator/docs/kvm-provider.md

The KVM provider

Runs Managed proxies as VMs on your own libvirt/KVM hypervisors. The operator reaches each host over SSH, forwards the libvirt unix socket through that connection, and talks libvirt's own RPC over it — so the hosts need nothing beyond sshd and libvirtd, and the operator image carries no libvirt libraries. VMs boot a stock distro cloud image that the operator downloads into the host's storage pool on first use, with the Proxy's cloud-init delivered on a NoCloud seed ISO built in-process.

Hypervisor prerequisites

Per host, once:

1. libvirt + KVM (libvirtd running, /dev/kvm present).
2. An SSH user in the libvirt group, so it may open /var/run/libvirt/libvirt-sock. The operator authenticates with a private key; nothing else on the host is touched.
3. A bridge to a network with DHCP (e.g. br0 bridged onto the LAN). Each VM gets one virtio NIC on it and takes its IP from that network's DHCP; the cluster must be able to reach those IPs for health probes and for crawlers to use the proxy.
4. A storage pool (default default, usually /var/lib/libvirt/images) with room for one base image per distro (a few hundred MB) plus the per-VM overlays (diskSizeGb each, thin-provisioned).
5. UEFI firmware for guests: the ovmf package on Debian/Ubuntu, edk2-ovmf on Fedora/RHEL. VMs boot UEFI by default (libvirt picks the OVMF binary itself); a domain start fails with "no firmware found" without it. Only firmware: bios images skip this.
6. Optionally, outbound internet from the VMs: a vendor-data script installs qemu-guest-agent from the distro mirrors on first boot. Without it the VM still works — the operator falls back to the host's ARP table to find the IP, which fills in once the guest sends traffic — but the ARP table only sees traffic to the host itself, so on a plain bridge the agent is in practice required.

Check from the operator's point of view before wiring it in:

ssh virt@nas1.home.lan virsh -c qemu:///system list --all   # user can drive libvirt
ssh virt@nas1.home.lan virsh -c qemu:///system pool-info default
ssh virt@nas1.home.lan ip link show br0

Cluster-side setup

1. Create the SSH Secret from a key pair dedicated to the operator and a known_hosts file holding every host's key. The worked example below uses ~/.ssh/proxy-operator_ed25519 and ~/.ssh/kvm_known_hosts, the same paths the integration test and make kvm-ssh-secret default to.

   # once: a key pair for the operator; the .pub goes on every host
   ssh-keygen -t ed25519 -f ~/.ssh/proxy-operator_ed25519 -N '' -C proxy-operator
   ssh-copy-id -i ~/.ssh/proxy-operator_ed25519.pub virt@nas1.home.lan
   
   # once per host: its host key, under the address the *operator* will
   # dial (hosts[].address), which may differ from what your laptop uses
   ssh-keyscan -H nas1.home.lan >> ~/.ssh/kvm_known_hosts
   
   # the Secret: the PRIVATE key (not the .pub) plus known_hosts
   kubectl -n egress-proxies-operator-system create secret generic kvm-ssh \
     --from-file=id_ed25519=$HOME/.ssh/proxy-operator_ed25519 \
     --from-file=known_hosts=$HOME/.ssh/kvm_known_hosts
   

   make kvm-ssh-secret runs that last command as an apply, so it also updates an existing Secret; override the paths with KVM_SSH_KEY=... KVM_KNOWN_HOSTS=.... Putting the .pub in the Secret is the classic mistake: the operator then fails at startup with a "not a private key" error. Without known_hosts, set insecureIgnoreHostKey: true and accept that a MITM could hand you a fake hypervisor.

   config/manager/manager.yaml mounts the Secret (optional) at /etc/proxy-operator/kvm as two files, id_ed25519 and known_hosts, which is what sshKeyFile and knownHostsFile in the provider config point at. The deployment starts fine without the Secret, even with a kvm provider configured: only that provider's proxies fail, and they recover once the Secret exists. Rotating the key: re-run the command (or the make target) and restart the manager; both files are read once, on the first connection, and cached.

2. Add a provider entry to the providers config (config/manager/providers_config.yaml in-cluster; full sample in config/samples/providers-config.yaml):

   - name: kvm-home
     type: kvm
     kvm:
       sshKeyFile: /etc/proxy-operator/kvm/id_ed25519
       knownHostsFile: /etc/proxy-operator/kvm/known_hosts
       images:
         debian-13:
           url: https://cloud.debian.org/images/cloud/trixie/latest/debian-13-generic-amd64.qcow2
       machineTypes:
         small: {vcpus: 1, memoryMiB: 1024}
       hosts:
         - name: nas1
           address: nas1.home.lan:22
           user: virt
           bridge: br0
   

   The operator checks both files at startup and logs a WARNING: line when either is unreadable, but it does not refuse to start: the provider set is built all-or-nothing, so exiting here would crash-loop the manager and take the kubernetes and gcp providers with it. The files are read for real on the first connection. A missing file is a transient failure, so proxies heal by themselves once the Secret lands; a key that does not parse is permanent and needs a fix.

3. Apply a Proxy (config/samples/proxy_kvm.yaml) and watch it:

   kubectl get px -w
   # NAME               MODE      PROVIDER   PHASE          IP             HEALTHY
   # proxy-kvm-sample   Managed   kvm-home   Provisioning
   # proxy-kvm-sample   Managed   kvm-home   Running        192.168.1.73
   # proxy-kvm-sample   Managed   kvm-home   Ready          192.168.1.73   True
   

   The first Proxy for a given image on a given host sits in Provisioning for the length of the image download; the operator log shows fetching base image into host pool and base image cached with the size and duration.

Config reference

Field	Required	Default	Meaning
sshKeyFile	yes		private key used for every host
knownHostsFile	unless insecureIgnoreHostKey		OpenSSH known_hosts to verify host keys against
insecureIgnoreHostKey		false	skip host-key verification entirely
diskSizeGb		10	capacity of each VM's overlay disk; cloud-init's growpart expands the root fs into it
vendorData			extra cloud-init vendor-data placed on every seed next to the Proxy's user-data. Beware that cloud-init drops any top-level key of a vendor-data cloud-config that the user-data also defines. Shipped as one part of a MIME multipart alongside the agent script when installGuestAgent is on
installGuestAgent		true	add the vendor-data script that installs and starts qemu-guest-agent. Turn off for images that ship it already or cannot reach a package mirror; IP discovery then depends on the host's ARP table
images.<key>			the key names the image in placement.image. Letters, digits, dots, dashes and underscores only — it becomes the cached volume name base-<key>.<format>
images.<key>.url	yes per entry		http(s) URL of a cloud image (qcow2 or raw)
images.<key>.sha256			expected digest; verified after download, a change triggers a re-download
images.<key>.format		qcow2	qcow2 or raw
images.<key>.firmware		uefi	uefi (OVMF, secure boot off) or bios (SeaBIOS) for images without an EFI system partition; Debian 13 hangs in GRUB under BIOS on a headless VM, see Known gaps
machineTypes.<key>	≥ 1		{vcpus, memoryMiB} size profile
hosts[].name	yes		the placement.zone value that selects this host; unique. Letters, digits, dots, dashes and underscores only — it becomes part of the providerID hosts/<name>/domains/<vm>
hosts[].address	yes		SSH endpoint, host:port
hosts[].user	yes		SSH user, member of the libvirt group
hosts[].socket		/var/run/libvirt/libvirt-sock	libvirt socket path on the host
hosts[].pool		default	storage pool for base images, overlays and seeds
hosts[].bridge	yes		host bridge for the VM's NIC

Placement

Proxy field	Meaning for kvm
placement.zone	host name from hosts. May be omitted when exactly one host is configured; otherwise required. Unknown → Failed
placement.machineType	key in machineTypes. Unknown → Failed
placement.image	key in images, or an http(s):// URL used directly. Anything else → Failed
placement.region	ignored

Failed here means the reconciler stops retrying and the message names the offending value; fix the spec (which replaces the VM) or the config.

What lands on the host

For a Proxy whose deterministic name is proxy-abc…, on the chosen host's pool:

Object	Name	Lifetime
base image	base-<key>.qcow2 (or base-<url-hash>.qcow2 for a URL)	cached; survives Proxy deletion
image marker	base-<key>.qcow2.sha256	with the base image
overlay disk	proxy-abc….qcow2, qcow2 backed by the base	with the Proxy
seed ISO	proxy-abc…-seed.iso	with the Proxy
domain	proxy-abc…	with the Proxy

The domain is a q35 machine with host-passthrough CPU, the overlay on virtio, the seed on a SATA cdrom, one virtio NIC on the bridge, a serial console, the guest-agent channel, a virtio RNG and no graphics. Its <metadata> carries the operator's element (managed flag, owning UID, creation time) — that is how orphan GC recognises its own VMs, since libvirt has no labels.

The seed's user-data is the Proxy's cloudInit, byte for byte. Any format cloud-init accepts works, including a Jinja template, a MIME archive, a shell script or a gzipped payload; the operator never parses or re-renders it.

The guest agent rides along in vendor-data, as a shell script rather than a cloud-config. That is the only shape cloud-init cannot drop: vendor-data cloud-config is merged under the user's own document with mergers that keep the user-data's value for every key both define and never append lists, so a Proxy carrying its own packages: silently lost the agent and sat in Provisioning forever. The strategy cannot be overridden from either document — merge_how in user-data governs only the first source, and merge_how in vendor-data is consumed and discarded before the document is stored. A vendor-data script skips all of it: cloud-init gives vendor-data its own handler chain, writes scripts to a separate directory and runs them from cc_scripts_vendor, after the user's packages are installed and before the user's runcmd. The script is idempotent, picks whichever of apt-get, dnf, yum, zypper or apk the image has, and logs into /var/log/cloud-init-output.log. Set installGuestAgent: false to leave it out. When vendorData is also configured, the two travel as separate parts of a MIME multipart vendor-data.

The image cache

The base image is downloaded by the operator, through libvirt, into the host's pool the first time a Proxy asks for it on that host — nothing is pre-placed on the hypervisor, and the operator needs no filesystem access there. The bytes stream from the URL through the operator into the pool with a running sha256; nothing is buffered. The marker volume is written last and holds the digest, so a crash mid-download leaves a base volume without a marker, which the next attempt deletes and redoes.

A re-download happens when: the marker is missing, or sha256 is configured and differs from the marker's content. Otherwise the cached copy is used for as long as it exists, even if the URL's content changed upstream (a latest URL is pinned at the moment of first fetch, per host).

To evict manually:

ssh virt@nas1 virsh -c qemu:///system vol-delete --pool default base-debian-13.qcow2.sha256
ssh virt@nas1 virsh -c qemu:///system vol-delete --pool default base-debian-13.qcow2

⚠️ Do not change an image entry's url or sha256, and do not evict its base volume, while Proxies using that image exist on the host. The overlays back onto the base file; deleting or replacing it corrupts every VM that uses it. Delete those Proxies first, or add the new image under a new key and move Proxies over by changing their placement.image (which replaces the VM).

Logging into a VM

• Put your key in the Proxy's cloud-init (ssh_authorized_keys) and ssh debian@<ip> (Debian) / ssh ubuntu@<ip> (Ubuntu).
• Or on the host: virsh console proxy-abc… attaches to the serial console; cloud-init's output scrolls by there on first boot.
• virsh domifaddr proxy-abc… --source arp / --source agent shows what the operator sees when it looks for the IP.

Logging

The provider logs through controller-runtime's logger, named kvm, with provider, host and name (the domain) on every line.

• --zap-log-level=debug (logr V(1)): one line per provider operation (domain defined, KVM domain ensured running, ...) and one libvirt call line per libvirt RPC, named kvm.libvirt, with rpc (the libvirt API function, e.g. virDomainLookupByName), the identifiers it was given (domain, pool, volume), curated results (state, path, interfaces), duration, and on failure error plus libvirtError (the code name, e.g. ErrNoDomain) and libvirtCode. Connection events (libvirt host connected, dial failures, libvirt connection dropped with a reason) log at the same level. A Create is about ten RPCs; a Get is four.
• --zap-log-level=2: the same lines also carry the domain and volume xml, the metadata element, the interface addresses and the domain names from a listing.
• Cloud-init is never logged at any level: the seed ISO upload reports only length and duration. There is no raw wire tier like the GCP provider's V(5); go-libvirt exposes no hook for it.

Health check

The operator checks every configured host every --provider-check-interval (default 1m). It publishes the result in two places: the ProviderHealth object named after the provider (kubectl get ph kvm-home -o yaml), and the proxy_operator_provider_target_up{provider,target} gauge.

All hosts are checked concurrently, each over the same pooled connection that Create and Get use:

1. The SSH and libvirt handshakes. These only happen on the first check, or after the link dropped.
2. virConnectGetLibVersion and virConnectGetHostname.
3. virStoragePoolLookupByName and virStoragePoolGetInfo on the host's pool.

A passing target's details carry hostname, libvirtVersion, pool, poolState, poolCapacity and poolAvailable (in GiB). The free space is the closest thing this provider has to a capacity signal. Every target also shows its SSH user. A failing target says where the check broke:

Reason	Meaning	Look at
SSHMaterialUnavailable	the key or known_hosts file is missing or does not parse	the kvm-ssh Secret (see Cluster-side setup)
Unreachable	the TCP connect failed, or the link died mid-check	DNS, Tailscale or a firewall between the cluster and address
SSHHandshakeFailed	the host key is missing from known_hosts or has changed, or user rejected the key	the known_hosts entry; user's authorized_keys
LibvirtConnectFailed	SSH worked, the libvirt socket did not	user in the libvirt group, libvirtd running, the socket path
Timeout	no answer within --provider-check-timeout (default 15s)	a hung libvirtd or a saturated link
PoolNotFound	there is no pool named pool	virsh pool-list --all
PoolNotActive	the pool exists but is not running	virsh pool-start <pool> and virsh pool-autostart <pool>
LibvirtError	any other libvirt error	the message carries libvirt's own text

Sharing the connection has two consequences:

• The leader now keeps one SSH connection per host open from startup, with a keepalive every 15s. Before, the first connection opened with the first KVM reconcile.
• A check that hits its timeout drops the host's connection, just as a cancelled reconcile already does. Calls in flight on a hung link then fail fast, and the next call redials.

Known gaps

• No quota or capacity detection. libvirt reports an out-of-space pool or an over-committed host as an ordinary error; the reconciler treats it as transient and retries with backoff rather than the slower quota backoff. The health check does report each pool's free space (poolAvailable), so a pool that is filling up shows before Creates start failing.
• A rotated kvm-ssh Secret needs a manager restart. The key and known_hosts files are read on first use and cached for the life of the process. Replacing them in the Secret therefore changes nothing for later connections, health checks included. After a host key change, the check keeps reporting SSHHandshakeFailed until the manager restarts.
• Orphan volumes without a domain are invisible to GC. A crash between creating the volumes and defining the domain leaves an overlay and seed that ListByTag cannot see (it lists domains). They are cleaned when the same Proxy reconciles again (Delete removes volumes by name even with no domain), but a Proxy CR that is gone for good leaves them for virsh vol-list to find.
• A host removed from the config takes its VMs with it. Delete for a providerID naming an unconfigured host is a no-op success, so the Proxy finishes deleting instead of pinning its finalizer, but the domain and its volumes stay on the hypervisor and orphan GC cannot see them either (it only sweeps configured hosts). Delete the Proxies before retiring a host, or clean up afterwards with virsh undefine --nvram plus virsh vol-delete. Get still fails permanently for such a providerID on purpose: a typo in hosts must not look like a missing VM and trigger a rebuild elsewhere.
• The first image fetch is synchronous. It holds a reconcile worker for the download (LAN speed, a few hundred MB, once per image per host). Other Proxies keep reconciling on the remaining workers.
• Replacing an image in use breaks running VMs — see the warning under The image cache.
• IP discovery needs either guest traffic or the agent. A VM on a bridge with no DHCP never gets an address and stays Provisioning forever; check virsh console. The same symptom with a booted, addressed VM means the guest agent never came up: check systemctl status qemu-guest-agent, cloud-init status --long and the vendor script's output in /var/log/cloud-init-output.log (a VM without outbound internet cannot install it). A user-data that disables vendor-data (vendor_data: {enabled: false}) also removes the agent, by design.
• Debian 13 cloud images hang in GRUB under BIOS on a headless VM (Debian bug #1111240, open, in grub2): trixie's image ships terminal_output gfxterm serial in its GRUB config, and with a serial console but no video device (exactly the domain this provider defines) GRUB prints "Booting" and never hands over to the kernel. Not host-specific; reproduced by the Debian cloud team. The same image boots under UEFI with the serial console intact, which is why firmware defaults to uefi. Debian 12 (bookworm) images are unaffected and boot either way. Images that need firmware: bios are the exception, not the rule.

Testing

Unit tests run over an in-memory fake of the libvirt API. The optional integration test against a real host is described in testing.md; its -v output includes the libvirt call trace above, at the verbosity KVM_TEST_LOG_LEVEL selects (default 1). TestKVMCheck_realHost runs the health check against the same host and creates nothing: go test -tags=integration -count=1 -v -run TestKVMCheck ./internal/provider/kvm/.