egress-proxies-operator/docs/kvm-provider.md

# The KVM provider

Runs Managed proxies as VMs on your own libvirt/KVM hypervisors. The
operator reaches each host over SSH, forwards the libvirt unix socket
through that connection, and talks libvirt's own RPC over it — so the
hosts need nothing beyond `sshd` and `libvirtd`, and the operator image
carries no libvirt libraries. VMs boot a stock distro cloud image that the
operator downloads into the host's storage pool on first use, with the
Proxy's cloud-init delivered on a NoCloud seed ISO built in-process.

## Hypervisor prerequisites

Per host, once:

1. **libvirt + KVM** (`libvirtd` running, `/dev/kvm` present).
2. **An SSH user in the `libvirt` group**, so it may open
   `/var/run/libvirt/libvirt-sock`. The operator authenticates with a
   private key; nothing else on the host is touched.
3. **A bridge to a network with DHCP** (e.g. `br0` bridged onto the LAN).
   Each VM gets one virtio NIC on it and takes its IP from that network's
   DHCP; the cluster must be able to reach those IPs for health probes
   and for crawlers to use the proxy.
4. **A storage pool** (default `default`, usually `/var/lib/libvirt/images`)
   with room for one base image per distro (a few hundred MB) plus the
   per-VM overlays (`diskSizeGb` each, thin-provisioned).
5. **UEFI firmware for guests**: the `ovmf` package on Debian/Ubuntu,
   `edk2-ovmf` on Fedora/RHEL. VMs boot UEFI by default (libvirt picks
   the OVMF binary itself); a domain start fails with "no firmware
   found" without it. Only `firmware: bios` images skip this.
6. Optionally, **outbound internet from the VMs**: a vendor-data script
   installs `qemu-guest-agent` from the distro mirrors on first boot.
   Without it the VM still works — the operator falls back to the host's
   ARP table to find the IP, which fills in once the guest sends traffic —
   but the ARP table only sees traffic to the host itself, so on a plain
   bridge the agent is in practice required.

Check from the operator's point of view before wiring it in:

```sh
ssh virt@nas1.home.lan virsh -c qemu:///system list --all   # user can drive libvirt
ssh virt@nas1.home.lan virsh -c qemu:///system pool-info default
ssh virt@nas1.home.lan ip link show br0
```

## Cluster-side setup

1. Create the SSH Secret from a key pair dedicated to the operator and a
   `known_hosts` file holding every host's key. The worked example below
   uses `~/.ssh/proxy-operator_ed25519` and `~/.ssh/kvm_known_hosts`,
   the same paths the integration test and `make kvm-ssh-secret` default
   to.

   ```sh
   # once: a key pair for the operator; the .pub goes on every host
   ssh-keygen -t ed25519 -f ~/.ssh/proxy-operator_ed25519 -N '' -C proxy-operator
   ssh-copy-id -i ~/.ssh/proxy-operator_ed25519.pub virt@nas1.home.lan

   # once per host: its host key, under the address the *operator* will
   # dial (hosts[].address), which may differ from what your laptop uses
   ssh-keyscan -H nas1.home.lan >> ~/.ssh/kvm_known_hosts

   # the Secret: the PRIVATE key (not the .pub) plus known_hosts
   kubectl -n egress-proxies-operator-system create secret generic kvm-ssh \
     --from-file=id_ed25519=$HOME/.ssh/proxy-operator_ed25519 \
     --from-file=known_hosts=$HOME/.ssh/kvm_known_hosts
   ```

   `make kvm-ssh-secret` runs that last command as an apply, so it also
   updates an existing Secret; override the paths with
   `KVM_SSH_KEY=... KVM_KNOWN_HOSTS=...`. Putting the `.pub` in the
   Secret is the classic mistake: the operator then fails at startup
   with a "not a private key" error. Without `known_hosts`, set
   `insecureIgnoreHostKey: true` and accept that a MITM could hand you a
   fake hypervisor.

   `config/manager/manager.yaml` mounts the Secret (optional) at
   `/etc/proxy-operator/kvm` as two files, `id_ed25519` and
   `known_hosts`, which is what `sshKeyFile` and `knownHostsFile` in the
   provider config point at. The deployment starts fine without the
   Secret, even with a `kvm` provider configured: only that provider's
   proxies fail, and they recover once the Secret exists. Rotating the
   key: re-run the command (or the make target) and restart the manager;
   both files are read once, on the first connection, and cached.

2. Add a provider entry to the providers config
   (`config/manager/providers_config.yaml` in-cluster; full sample in
   `config/samples/providers-config.yaml`):

   ```yaml
   - name: kvm-home
     type: kvm
     kvm:
       sshKeyFile: /etc/proxy-operator/kvm/id_ed25519
       knownHostsFile: /etc/proxy-operator/kvm/known_hosts
       images:
         debian-13:
           url: https://cloud.debian.org/images/cloud/trixie/latest/debian-13-generic-amd64.qcow2
       machineTypes:
         small: {vcpus: 1, memoryMiB: 1024}
       hosts:
         - name: nas1
           address: nas1.home.lan:22
           user: virt
           bridge: br0
   ```

   The operator checks both files at startup and logs a `WARNING:` line
   when either is unreadable, but it does not refuse to start: the
   provider set is built all-or-nothing, so exiting here would crash-loop
   the manager and take the `kubernetes` and `gcp` providers with it.
   The files are read for real on the first connection. A missing file is
   a transient failure, so proxies heal by themselves once the Secret
   lands; a key that does not parse is permanent and needs a fix.

3. Apply a Proxy (`config/samples/proxy_kvm.yaml`) and watch it:

   ```sh
   kubectl get px -w
   # NAME               MODE      PROVIDER   PHASE          IP             HEALTHY
   # proxy-kvm-sample   Managed   kvm-home   Provisioning
   # proxy-kvm-sample   Managed   kvm-home   Running        192.168.1.73
   # proxy-kvm-sample   Managed   kvm-home   Ready          192.168.1.73   True
   ```

   The first Proxy for a given image on a given host sits in
   `Provisioning` for the length of the image download; the operator log
   shows `fetching base image into host pool` and `base image cached`
   with the size and duration.

## Config reference

| Field | Required | Default | Meaning |
|---|---|---|---|
| `sshKeyFile` | yes | | private key used for every host |
| `knownHostsFile` | unless `insecureIgnoreHostKey` | | OpenSSH `known_hosts` to verify host keys against |
| `insecureIgnoreHostKey` | | `false` | skip host-key verification entirely |
| `diskSizeGb` | | `10` | capacity of each VM's overlay disk; cloud-init's growpart expands the root fs into it |
| `vendorData` | | | extra cloud-init vendor-data placed on every seed next to the Proxy's user-data. Beware that cloud-init drops any top-level key of a vendor-data **cloud-config** that the user-data also defines. Shipped as one part of a MIME multipart alongside the agent script when `installGuestAgent` is on |
| `installGuestAgent` | | `true` | add the vendor-data script that installs and starts `qemu-guest-agent`. Turn off for images that ship it already or cannot reach a package mirror; IP discovery then depends on the host's ARP table |
| `images.<key>` | | | the key names the image in `placement.image`. Letters, digits, dots, dashes and underscores only — it becomes the cached volume name `base-<key>.<format>` |
| `images.<key>.url` | yes per entry | | http(s) URL of a cloud image (qcow2 or raw) |
| `images.<key>.sha256` | | | expected digest; verified after download, a change triggers a re-download |
| `images.<key>.format` | | `qcow2` | `qcow2` or `raw` |
| `images.<key>.firmware` | | `uefi` | `uefi` (OVMF, secure boot off) or `bios` (SeaBIOS) for images without an EFI system partition; Debian 13 hangs in GRUB under BIOS on a headless VM, see *Known gaps* |
| `machineTypes.<key>` | ≥ 1 | | `{vcpus, memoryMiB}` size profile |
| `hosts[].name` | yes | | the `placement.zone` value that selects this host; unique. Letters, digits, dots, dashes and underscores only — it becomes part of the providerID `hosts/<name>/domains/<vm>` |
| `hosts[].address` | yes | | SSH endpoint, `host:port` |
| `hosts[].user` | yes | | SSH user, member of the `libvirt` group |
| `hosts[].socket` | | `/var/run/libvirt/libvirt-sock` | libvirt socket path on the host |
| `hosts[].pool` | | `default` | storage pool for base images, overlays and seeds |
| `hosts[].bridge` | yes | | host bridge for the VM's NIC |

## Placement

| Proxy field | Meaning for kvm |
|---|---|
| `placement.zone` | host name from `hosts`. May be omitted when exactly one host is configured; otherwise required. Unknown → `Failed` |
| `placement.machineType` | key in `machineTypes`. Unknown → `Failed` |
| `placement.image` | key in `images`, or an `http(s)://` URL used directly. Anything else → `Failed` |
| `placement.region` | ignored |

`Failed` here means the reconciler stops retrying and the message names
the offending value; fix the spec (which replaces the VM) or the config.

## What lands on the host

For a Proxy whose deterministic name is `proxy-abc…`, on the chosen
host's pool:

| Object | Name | Lifetime |
|---|---|---|
| base image | `base-<key>.qcow2` (or `base-<url-hash>.qcow2` for a URL) | cached; survives Proxy deletion |
| image marker | `base-<key>.qcow2.sha256` | with the base image |
| overlay disk | `proxy-abc….qcow2`, qcow2 backed by the base | with the Proxy |
| seed ISO | `proxy-abc…-seed.iso` | with the Proxy |
| domain | `proxy-abc…` | with the Proxy |

The domain is a q35 machine with host-passthrough CPU, the overlay on
virtio, the seed on a SATA cdrom, one virtio NIC on the bridge, a serial
console, the guest-agent channel, a virtio RNG and no graphics. Its
`<metadata>` carries the operator's element (managed flag, owning UID,
creation time) — that is how orphan GC recognises its own VMs, since
libvirt has no labels.

The seed's `user-data` is the Proxy's `cloudInit`, **byte for byte**. Any
format cloud-init accepts works, including a Jinja template, a MIME
archive, a shell script or a gzipped payload; the operator never parses or
re-renders it.

The guest agent rides along in `vendor-data`, as a **shell script** rather
than a cloud-config. That is the only shape cloud-init cannot drop:
vendor-data cloud-config is merged under the user's own document with
mergers that keep the user-data's value for every key both define and never
append lists, so a Proxy carrying its own `packages:` silently lost the
agent and sat in `Provisioning` forever. The strategy cannot be overridden
from either document — `merge_how` in user-data governs only the first
source, and `merge_how` in vendor-data is consumed and discarded before the
document is stored. A vendor-data script skips all of it: cloud-init gives
vendor-data its own handler chain, writes scripts to a separate directory
and runs them from `cc_scripts_vendor`, after the user's packages are
installed and before the user's `runcmd`. The script is idempotent, picks
whichever of `apt-get`, `dnf`, `yum`, `zypper` or `apk` the image has, and
logs into `/var/log/cloud-init-output.log`. Set `installGuestAgent: false`
to leave it out. When `vendorData` is also configured, the two travel as
separate parts of a MIME multipart vendor-data.

## The image cache

The base image is downloaded **by the operator, through libvirt**, into
the host's pool the first time a Proxy asks for it on that host — nothing
is pre-placed on the hypervisor, and the operator needs no filesystem
access there. The bytes stream from the URL through the operator into
the pool with a running sha256; nothing is buffered. The marker volume is
written last and holds the digest, so a crash mid-download leaves a base
volume without a marker, which the next attempt deletes and redoes.

A re-download happens when: the marker is missing, or `sha256` is
configured and differs from the marker's content. Otherwise the cached
copy is used for as long as it exists, even if the URL's content changed
upstream (a `latest` URL is pinned at the moment of first fetch, per
host).

To evict manually:

```sh
ssh virt@nas1 virsh -c qemu:///system vol-delete --pool default base-debian-13.qcow2.sha256
ssh virt@nas1 virsh -c qemu:///system vol-delete --pool default base-debian-13.qcow2
```

⚠️ **Do not change an image entry's `url` or `sha256`, and do not evict
its base volume, while Proxies using that image exist on the host.** The
overlays back onto the base file; deleting or replacing it corrupts every
VM that uses it. Delete those Proxies first, or add the new image under a
new key and move Proxies over by changing their `placement.image` (which
replaces the VM).

## Logging into a VM

- Put your key in the Proxy's cloud-init (`ssh_authorized_keys`) and
  `ssh debian@<ip>` (Debian) / `ssh ubuntu@<ip>` (Ubuntu).
- Or on the host: `virsh console proxy-abc…` attaches to the serial
  console; cloud-init's output scrolls by there on first boot.
- `virsh domifaddr proxy-abc… --source arp` / `--source agent` shows what
  the operator sees when it looks for the IP.

## Logging

The provider logs through controller-runtime's logger, named `kvm`, with
`provider`, `host` and `name` (the domain) on every line.

- `--zap-log-level=debug` (logr V(1)): one line per provider operation
  (`domain defined`, `KVM domain ensured running`, ...) **and one
  `libvirt call` line per libvirt RPC**, named `kvm.libvirt`, with
  `rpc` (the libvirt API function, e.g. `virDomainLookupByName`), the
  identifiers it was given (`domain`, `pool`, `volume`), curated results
  (`state`, `path`, `interfaces`), `duration`, and on failure `error`
  plus `libvirtError` (the code name, e.g. `ErrNoDomain`) and
  `libvirtCode`. Connection events (`libvirt host connected`, dial
  failures, `libvirt connection dropped` with a reason) log at the same
  level. A Create is about ten RPCs; a Get is four.
- `--zap-log-level=2`: the same lines also carry the domain and volume
  `xml`, the `metadata` element, the interface `addresses` and the
  domain `names` from a listing.
- Cloud-init is never logged at any level: the seed ISO upload reports
  only `length` and `duration`. There is no raw wire tier like the GCP
  provider's V(5); go-libvirt exposes no hook for it.

## Health check

The operator checks every configured host every `--provider-check-interval`
(default 1m). It publishes the result in two places: the `ProviderHealth`
object named after the provider (`kubectl get ph kvm-home -o yaml`), and the
`proxy_operator_provider_target_up{provider,target}` gauge.

All hosts are checked concurrently, each over the same pooled connection
that Create and Get use:

1. The SSH and libvirt handshakes. These only happen on the first check, or
   after the link dropped.
2. `virConnectGetLibVersion` and `virConnectGetHostname`.
3. `virStoragePoolLookupByName` and `virStoragePoolGetInfo` on the host's
   `pool`.

A passing target's `details` carry `hostname`, `libvirtVersion`, `pool`,
`poolState`, `poolCapacity` and `poolAvailable` (in GiB). The free space is
the closest thing this provider has to a capacity signal. Every target also
shows its SSH `user`. A failing target says where the check broke:

| Reason | Meaning | Look at |
|---|---|---|
| `SSHMaterialUnavailable` | the key or known_hosts file is missing or does not parse | the `kvm-ssh` Secret (see *Cluster-side setup*) |
| `Unreachable` | the TCP connect failed, or the link died mid-check | DNS, Tailscale or a firewall between the cluster and `address` |
| `SSHHandshakeFailed` | the host key is missing from known_hosts or has changed, or `user` rejected the key | the known_hosts entry; `user`'s `authorized_keys` |
| `LibvirtConnectFailed` | SSH worked, the libvirt socket did not | `user` in the `libvirt` group, `libvirtd` running, the `socket` path |
| `Timeout` | no answer within `--provider-check-timeout` (default 15s) | a hung `libvirtd` or a saturated link |
| `PoolNotFound` | there is no pool named `pool` | `virsh pool-list --all` |
| `PoolNotActive` | the pool exists but is not running | `virsh pool-start <pool>` and `virsh pool-autostart <pool>` |
| `LibvirtError` | any other libvirt error | the message carries libvirt's own text |

Sharing the connection has two consequences:

- The leader now keeps one SSH connection per host open from startup, with a
  keepalive every 15s. Before, the first connection opened with the first KVM
  reconcile.
- A check that hits its timeout drops the host's connection, just as a
  cancelled reconcile already does. Calls in flight on a hung link then fail
  fast, and the next call redials.

## Known gaps

- **No quota or capacity detection.** libvirt reports an out-of-space
  pool or an over-committed host as an ordinary error; the reconciler
  treats it as transient and retries with backoff rather than the slower
  quota backoff. The health check does report each pool's free space
  (`poolAvailable`), so a pool that is filling up shows before Creates
  start failing.
- **A rotated `kvm-ssh` Secret needs a manager restart.** The key and
  known_hosts files are read on first use and cached for the life of the
  process. Replacing them in the Secret therefore changes nothing for
  later connections, health checks included. After a host key change, the
  check keeps reporting `SSHHandshakeFailed` until the manager restarts.
- **Orphan volumes without a domain are invisible to GC.** A crash
  between creating the volumes and defining the domain leaves an overlay
  and seed that `ListByTag` cannot see (it lists domains). They are
  cleaned when the same Proxy reconciles again (Delete removes volumes by
  name even with no domain), but a Proxy CR that is gone for good leaves
  them for `virsh vol-list` to find.
- **A host removed from the config takes its VMs with it.** `Delete` for a
  providerID naming an unconfigured host is a no-op success, so the Proxy
  finishes deleting instead of pinning its finalizer, but the domain and its
  volumes stay on the hypervisor and orphan GC cannot see them either (it
  only sweeps configured hosts). Delete the Proxies before retiring a host,
  or clean up afterwards with `virsh undefine --nvram` plus `virsh vol-delete`.
  `Get` still fails permanently for such a providerID on purpose: a typo in
  `hosts` must not look like a missing VM and trigger a rebuild elsewhere.
- **The first image fetch is synchronous.** It holds a reconcile worker
  for the download (LAN speed, a few hundred MB, once per image per
  host). Other Proxies keep reconciling on the remaining workers.
- **Replacing an image in use breaks running VMs** — see the warning
  under *The image cache*.
- **IP discovery needs either guest traffic or the agent.** A VM on a
  bridge with no DHCP never gets an address and stays `Provisioning`
  forever; check `virsh console`. The same symptom with a booted, addressed
  VM means the guest agent never came up: check `systemctl status
  qemu-guest-agent`, `cloud-init status --long` and the vendor script's
  output in `/var/log/cloud-init-output.log` (a VM without outbound
  internet cannot install it). A user-data that disables vendor-data
  (`vendor_data: {enabled: false}`) also removes the agent, by design.
- **Debian 13 cloud images hang in GRUB under BIOS on a headless VM**
  ([Debian bug #1111240](https://bugs.debian.org/1111240), open, in
  `grub2`): trixie's image ships `terminal_output gfxterm serial` in its
  GRUB config, and with a serial console but no video device (exactly
  the domain this provider defines) GRUB prints "Booting" and never
  hands over to the kernel. Not host-specific; reproduced by the Debian
  cloud team. The same image boots under UEFI with the serial console
  intact, which is why `firmware` defaults to `uefi`. Debian 12
  (bookworm) images are unaffected and boot either way. Images that
  need `firmware: bios` are the exception, not the rule.

## Testing

Unit tests run over an in-memory fake of the libvirt API. The optional
integration test against a real host is described in
[testing.md](testing.md#the-kvm-integration-test-real-hypervisor); its
`-v` output includes the `libvirt call` trace above, at the verbosity
`KVM_TEST_LOG_LEVEL` selects (default 1). `TestKVMCheck_realHost` runs the
health check against the same host and creates nothing:
`go test -tags=integration -count=1 -v -run TestKVMCheck ./internal/provider/kvm/`.