# The KVM provider Runs Managed proxies as VMs on your own libvirt/KVM hypervisors. The operator reaches each host over SSH, forwards the libvirt unix socket through that connection, and talks libvirt's own RPC over it — so the hosts need nothing beyond `sshd` and `libvirtd`, and the operator image carries no libvirt libraries. VMs boot a stock distro cloud image that the operator downloads into the host's storage pool on first use, with the Proxy's cloud-init delivered on a NoCloud seed ISO built in-process. ## Hypervisor prerequisites Per host, once: 1. **libvirt + KVM** (`libvirtd` running, `/dev/kvm` present). 2. **An SSH user in the `libvirt` group**, so it may open `/var/run/libvirt/libvirt-sock`. The operator authenticates with a private key; nothing else on the host is touched. 3. **A bridge to a network with DHCP** (e.g. `br0` bridged onto the LAN). Each VM gets one virtio NIC on it and takes its IP from that network's DHCP; the cluster must be able to reach those IPs for health probes and for crawlers to use the proxy. 4. **A storage pool** (default `default`, usually `/var/lib/libvirt/images`) with room for one base image per distro (a few hundred MB) plus the per-VM overlays (`diskSizeGb` each, thin-provisioned). 5. **UEFI firmware for guests**: the `ovmf` package on Debian/Ubuntu, `edk2-ovmf` on Fedora/RHEL. VMs boot UEFI by default (libvirt picks the OVMF binary itself); a domain start fails with "no firmware found" without it. Only `firmware: bios` images skip this. 6. Optionally, **outbound internet from the VMs**: the default vendor-data installs `qemu-guest-agent` from the distro mirrors on first boot. Without it the VM still works — the operator falls back to the host's ARP table to find the IP, which fills in once the guest sends traffic. Check from the operator's point of view before wiring it in: ```sh ssh virt@nas1.home.lan virsh -c qemu:///system list --all # user can drive libvirt ssh virt@nas1.home.lan virsh -c qemu:///system pool-info default ssh virt@nas1.home.lan ip link show br0 ``` ## Cluster-side setup 1. Create the SSH Secret. `known_hosts` must contain every host's key (`ssh-keyscan -H nas1.home.lan >> known_hosts`); without it, set `insecureIgnoreHostKey: true` and accept that a MITM could hand you a fake hypervisor. ```sh kubectl -n egress-proxies-operator-system create secret generic kvm-ssh \ --from-file=id_ed25519=$HOME/.ssh/proxy-operator_ed25519 \ --from-file=known_hosts=./known_hosts ``` `config/manager/manager.yaml` mounts it (optional) at `/etc/proxy-operator/kvm`; the deployment starts fine without it as long as no `kvm` provider is configured. 2. Add a provider entry to the providers config (`config/manager/providers_config.yaml` in-cluster; full sample in `config/samples/providers-config.yaml`): ```yaml - name: kvm-home type: kvm kvm: sshKeyFile: /etc/proxy-operator/kvm/id_ed25519 knownHostsFile: /etc/proxy-operator/kvm/known_hosts images: debian-13: url: https://cloud.debian.org/images/cloud/trixie/latest/debian-13-generic-amd64.qcow2 machineTypes: small: {vcpus: 1, memoryMiB: 1024} hosts: - name: nas1 address: nas1.home.lan:22 user: virt bridge: br0 ``` The operator reads the key and known_hosts at startup and refuses to start if either is missing — a bad path fails fast, not on the first Proxy. 3. Apply a Proxy (`config/samples/proxy_kvm.yaml`) and watch it: ```sh kubectl get px -w # NAME MODE PROVIDER PHASE IP HEALTHY # proxy-kvm-sample Managed kvm-home Provisioning # proxy-kvm-sample Managed kvm-home Running 192.168.1.73 # proxy-kvm-sample Managed kvm-home Ready 192.168.1.73 True ``` The first Proxy for a given image on a given host sits in `Provisioning` for the length of the image download; the operator log shows `fetching base image into host pool` and `base image cached` with the size and duration. ## Config reference | Field | Required | Default | Meaning | |---|---|---|---| | `sshKeyFile` | yes | | private key used for every host | | `knownHostsFile` | unless `insecureIgnoreHostKey` | | OpenSSH `known_hosts` to verify host keys against | | `insecureIgnoreHostKey` | | `false` | skip host-key verification entirely | | `diskSizeGb` | | `10` | capacity of each VM's overlay disk; cloud-init's growpart expands the root fs into it | | `vendorData` | | installs `qemu-guest-agent` | cloud-init vendor-data placed on every seed next to the Proxy's user-data | | `images..url` | yes per entry | | http(s) URL of a cloud image (qcow2 or raw) | | `images..sha256` | | | expected digest; verified after download, a change triggers a re-download | | `images..format` | | `qcow2` | `qcow2` or `raw` | | `images..firmware` | | `uefi` | `uefi` (OVMF, secure boot off) or `bios` (SeaBIOS) for images without an EFI system partition; Debian 13 hangs in GRUB under BIOS on a headless VM, see *Known gaps* | | `machineTypes.` | ≥ 1 | | `{vcpus, memoryMiB}` size profile | | `hosts[].name` | yes | | the `placement.zone` value that selects this host; unique | | `hosts[].address` | yes | | SSH endpoint, `host:port` | | `hosts[].user` | yes | | SSH user, member of the `libvirt` group | | `hosts[].socket` | | `/var/run/libvirt/libvirt-sock` | libvirt socket path on the host | | `hosts[].pool` | | `default` | storage pool for base images, overlays and seeds | | `hosts[].bridge` | yes | | host bridge for the VM's NIC | ## Placement | Proxy field | Meaning for kvm | |---|---| | `placement.zone` | host name from `hosts`. May be omitted when exactly one host is configured; otherwise required. Unknown → `Failed` | | `placement.machineType` | key in `machineTypes`. Unknown → `Failed` | | `placement.image` | key in `images`, or an `http(s)://` URL used directly. Anything else → `Failed` | | `placement.region` | ignored | `Failed` here means the reconciler stops retrying and the message names the offending value; fix the spec (which replaces the VM) or the config. ## What lands on the host For a Proxy whose deterministic name is `proxy-abc…`, on the chosen host's pool: | Object | Name | Lifetime | |---|---|---| | base image | `base-.qcow2` (or `base-.qcow2` for a URL) | cached; survives Proxy deletion | | image marker | `base-.qcow2.sha256` | with the base image | | overlay disk | `proxy-abc….qcow2`, qcow2 backed by the base | with the Proxy | | seed ISO | `proxy-abc…-seed.iso` | with the Proxy | | domain | `proxy-abc…` | with the Proxy | The domain is a q35 machine with host-passthrough CPU, the overlay on virtio, the seed on a SATA cdrom, one virtio NIC on the bridge, a serial console, the guest-agent channel, a virtio RNG and no graphics. Its `` carries the operator's element (managed flag, owning UID, creation time) — that is how orphan GC recognises its own VMs, since libvirt has no labels. ## The image cache The base image is downloaded **by the operator, through libvirt**, into the host's pool the first time a Proxy asks for it on that host — nothing is pre-placed on the hypervisor, and the operator needs no filesystem access there. The bytes stream from the URL through the operator into the pool with a running sha256; nothing is buffered. The marker volume is written last and holds the digest, so a crash mid-download leaves a base volume without a marker, which the next attempt deletes and redoes. A re-download happens when: the marker is missing, or `sha256` is configured and differs from the marker's content. Otherwise the cached copy is used for as long as it exists, even if the URL's content changed upstream (a `latest` URL is pinned at the moment of first fetch, per host). To evict manually: ```sh ssh virt@nas1 virsh -c qemu:///system vol-delete --pool default base-debian-13.qcow2.sha256 ssh virt@nas1 virsh -c qemu:///system vol-delete --pool default base-debian-13.qcow2 ``` ⚠️ **Do not change an image entry's `url` or `sha256`, and do not evict its base volume, while Proxies using that image exist on the host.** The overlays back onto the base file; deleting or replacing it corrupts every VM that uses it. Delete those Proxies first, or add the new image under a new key and move Proxies over by changing their `placement.image` (which replaces the VM). ## Logging into a VM - Put your key in the Proxy's cloud-init (`ssh_authorized_keys`) and `ssh debian@` (Debian) / `ssh ubuntu@` (Ubuntu). - Or on the host: `virsh console proxy-abc…` attaches to the serial console; cloud-init's output scrolls by there on first boot. - `virsh domifaddr proxy-abc… --source arp` / `--source agent` shows what the operator sees when it looks for the IP. ## Logging The provider logs through controller-runtime's logger, named `kvm`, with `provider`, `host` and `name` (the domain) on every line. - `--zap-log-level=debug` (logr V(1)): one line per provider operation (`domain defined`, `KVM domain ensured running`, ...) **and one `libvirt call` line per libvirt RPC**, named `kvm.libvirt`, with `rpc` (the libvirt API function, e.g. `virDomainLookupByName`), the identifiers it was given (`domain`, `pool`, `volume`), curated results (`state`, `path`, `interfaces`), `duration`, and on failure `error` plus `libvirtError` (the code name, e.g. `ErrNoDomain`) and `libvirtCode`. Connection events (`libvirt host connected`, dial failures, `libvirt connection dropped` with a reason) log at the same level. A Create is about ten RPCs; a Get is four. - `--zap-log-level=2`: the same lines also carry the domain and volume `xml`, the `metadata` element, the interface `addresses` and the domain `names` from a listing. - Cloud-init is never logged at any level: the seed ISO upload reports only `length` and `duration`. There is no raw wire tier like the GCP provider's V(5); go-libvirt exposes no hook for it. ## Known gaps - **No quota or capacity detection.** libvirt reports an out-of-space pool or an over-committed host as an ordinary error; the reconciler treats it as transient and retries with backoff rather than the slower quota backoff. - **Orphan volumes without a domain are invisible to GC.** A crash between creating the volumes and defining the domain leaves an overlay and seed that `ListByTag` cannot see (it lists domains). They are cleaned when the same Proxy reconciles again (Delete removes volumes by name even with no domain), but a Proxy CR that is gone for good leaves them for `virsh vol-list` to find. - **The first image fetch is synchronous.** It holds a reconcile worker for the download (LAN speed, a few hundred MB, once per image per host). Other Proxies keep reconciling on the remaining workers. - **Replacing an image in use breaks running VMs** — see the warning under *The image cache*. - **IP discovery needs either guest traffic or the agent.** A VM on a bridge with no DHCP never gets an address and stays `Provisioning` forever; check `virsh console`. - **Debian 13 cloud images hang in GRUB under BIOS on a headless VM** ([Debian bug #1111240](https://bugs.debian.org/1111240), open, in `grub2`): trixie's image ships `terminal_output gfxterm serial` in its GRUB config, and with a serial console but no video device (exactly the domain this provider defines) GRUB prints "Booting" and never hands over to the kernel. Not host-specific; reproduced by the Debian cloud team. The same image boots under UEFI with the serial console intact, which is why `firmware` defaults to `uefi`. Debian 12 (bookworm) images are unaffected and boot either way. Images that need `firmware: bios` are the exception, not the rule. ## Testing Unit tests run over an in-memory fake of the libvirt API. The optional integration test against a real host is described in [testing.md](testing.md#the-kvm-integration-test-real-hypervisor); its `-v` output includes the `libvirt call` trace above, at the verbosity `KVM_TEST_LOG_LEVEL` selects (default 1).