egress-proxies-operator/docs/kvm-provider.md

# The KVM provider

Runs Managed proxies as VMs on your own libvirt/KVM hypervisors. The
operator reaches each host over SSH, forwards the libvirt unix socket
through that connection, and talks libvirt's own RPC over it — so the
hosts need nothing beyond `sshd` and `libvirtd`, and the operator image
carries no libvirt libraries. VMs boot a stock distro cloud image that the
operator downloads into the host's storage pool on first use, with the
Proxy's cloud-init delivered on a NoCloud seed ISO built in-process.

## Hypervisor prerequisites

Per host, once:

1. **libvirt + KVM** (`libvirtd` running, `/dev/kvm` present).
2. **An SSH user in the `libvirt` group**, so it may open
   `/var/run/libvirt/libvirt-sock`. The operator authenticates with a
   private key; nothing else on the host is touched.
3. **A bridge to a network with DHCP** (e.g. `br0` bridged onto the LAN).
   Each VM gets one virtio NIC on it and takes its IP from that network's
   DHCP; the cluster must be able to reach those IPs for health probes
   and for crawlers to use the proxy.
4. **A storage pool** (default `default`, usually `/var/lib/libvirt/images`)
   with room for one base image per distro (a few hundred MB) plus the
   per-VM overlays (`diskSizeGb` each, thin-provisioned).
5. **UEFI firmware for guests**: the `ovmf` package on Debian/Ubuntu,
   `edk2-ovmf` on Fedora/RHEL. VMs boot UEFI by default (libvirt picks
   the OVMF binary itself); a domain start fails with "no firmware
   found" without it. Only `firmware: bios` images skip this.
6. Optionally, **outbound internet from the VMs**: the operator adds
   `qemu-guest-agent` to every Proxy's cloud-config (`packages` plus a
   `runcmd` that enables it), so it is installed from the distro mirrors
   on first boot. Without it the VM still works — the operator falls back
   to the host's ARP table to find the IP, which fills in once the guest
   sends traffic — but note the ARP table only sees traffic to the host
   itself, so on a plain bridge the agent is in practice required.

Check from the operator's point of view before wiring it in:

```sh
ssh virt@nas1.home.lan virsh -c qemu:///system list --all   # user can drive libvirt
ssh virt@nas1.home.lan virsh -c qemu:///system pool-info default
ssh virt@nas1.home.lan ip link show br0
```

## Cluster-side setup

1. Create the SSH Secret from a key pair dedicated to the operator and a
   `known_hosts` file holding every host's key. The worked example below
   uses `~/.ssh/proxy-operator_ed25519` and `~/.ssh/kvm_known_hosts`,
   the same paths the integration test and `make kvm-ssh-secret` default
   to.

   ```sh
   # once: a key pair for the operator; the .pub goes on every host
   ssh-keygen -t ed25519 -f ~/.ssh/proxy-operator_ed25519 -N '' -C proxy-operator
   ssh-copy-id -i ~/.ssh/proxy-operator_ed25519.pub virt@nas1.home.lan

   # once per host: its host key, under the address the *operator* will
   # dial (hosts[].address), which may differ from what your laptop uses
   ssh-keyscan -H nas1.home.lan >> ~/.ssh/kvm_known_hosts

   # the Secret: the PRIVATE key (not the .pub) plus known_hosts
   kubectl -n egress-proxies-operator-system create secret generic kvm-ssh \
     --from-file=id_ed25519=$HOME/.ssh/proxy-operator_ed25519 \
     --from-file=known_hosts=$HOME/.ssh/kvm_known_hosts
   ```

   `make kvm-ssh-secret` runs that last command as an apply, so it also
   updates an existing Secret; override the paths with
   `KVM_SSH_KEY=... KVM_KNOWN_HOSTS=...`. Putting the `.pub` in the
   Secret is the classic mistake: the operator then fails at startup
   with a "not a private key" error. Without `known_hosts`, set
   `insecureIgnoreHostKey: true` and accept that a MITM could hand you a
   fake hypervisor.

   `config/manager/manager.yaml` mounts the Secret (optional) at
   `/etc/proxy-operator/kvm` as two files, `id_ed25519` and
   `known_hosts`, which is what `sshKeyFile` and `knownHostsFile` in the
   provider config point at. The deployment starts fine without the
   Secret as long as no `kvm` provider is configured. Rotating the key:
   re-run the command (or the make target) and restart the manager; it
   reads both files once at startup.

2. Add a provider entry to the providers config
   (`config/manager/providers_config.yaml` in-cluster; full sample in
   `config/samples/providers-config.yaml`):

   ```yaml
   - name: kvm-home
     type: kvm
     kvm:
       sshKeyFile: /etc/proxy-operator/kvm/id_ed25519
       knownHostsFile: /etc/proxy-operator/kvm/known_hosts
       images:
         debian-13:
           url: https://cloud.debian.org/images/cloud/trixie/latest/debian-13-generic-amd64.qcow2
       machineTypes:
         small: {vcpus: 1, memoryMiB: 1024}
       hosts:
         - name: nas1
           address: nas1.home.lan:22
           user: virt
           bridge: br0
   ```

   The operator reads the key and known_hosts at startup and refuses to
   start if either is missing — a bad path fails fast, not on the first
   Proxy.

3. Apply a Proxy (`config/samples/proxy_kvm.yaml`) and watch it:

   ```sh
   kubectl get px -w
   # NAME               MODE      PROVIDER   PHASE          IP             HEALTHY
   # proxy-kvm-sample   Managed   kvm-home   Provisioning
   # proxy-kvm-sample   Managed   kvm-home   Running        192.168.1.73
   # proxy-kvm-sample   Managed   kvm-home   Ready          192.168.1.73   True
   ```

   The first Proxy for a given image on a given host sits in
   `Provisioning` for the length of the image download; the operator log
   shows `fetching base image into host pool` and `base image cached`
   with the size and duration.

## Config reference

| Field | Required | Default | Meaning |
|---|---|---|---|
| `sshKeyFile` | yes | | private key used for every host |
| `knownHostsFile` | unless `insecureIgnoreHostKey` | | OpenSSH `known_hosts` to verify host keys against |
| `insecureIgnoreHostKey` | | `false` | skip host-key verification entirely |
| `diskSizeGb` | | `10` | capacity of each VM's overlay disk; cloud-init's growpart expands the root fs into it |
| `vendorData` | | | extra cloud-init vendor-data placed on every seed next to the Proxy's user-data; cloud-init drops any top-level key here that the user-data also defines. The guest-agent bootstrap is not configured here, see *What lands on the host* |
| `images.<key>.url` | yes per entry | | http(s) URL of a cloud image (qcow2 or raw) |
| `images.<key>.sha256` | | | expected digest; verified after download, a change triggers a re-download |
| `images.<key>.format` | | `qcow2` | `qcow2` or `raw` |
| `images.<key>.firmware` | | `uefi` | `uefi` (OVMF, secure boot off) or `bios` (SeaBIOS) for images without an EFI system partition; Debian 13 hangs in GRUB under BIOS on a headless VM, see *Known gaps* |
| `machineTypes.<key>` | ≥ 1 | | `{vcpus, memoryMiB}` size profile |
| `hosts[].name` | yes | | the `placement.zone` value that selects this host; unique |
| `hosts[].address` | yes | | SSH endpoint, `host:port` |
| `hosts[].user` | yes | | SSH user, member of the `libvirt` group |
| `hosts[].socket` | | `/var/run/libvirt/libvirt-sock` | libvirt socket path on the host |
| `hosts[].pool` | | `default` | storage pool for base images, overlays and seeds |
| `hosts[].bridge` | yes | | host bridge for the VM's NIC |

## Placement

| Proxy field | Meaning for kvm |
|---|---|
| `placement.zone` | host name from `hosts`. May be omitted when exactly one host is configured; otherwise required. Unknown → `Failed` |
| `placement.machineType` | key in `machineTypes`. Unknown → `Failed` |
| `placement.image` | key in `images`, or an `http(s)://` URL used directly. Anything else → `Failed` |
| `placement.region` | ignored |

`Failed` here means the reconciler stops retrying and the message names
the offending value; fix the spec (which replaces the VM) or the config.

## What lands on the host

For a Proxy whose deterministic name is `proxy-abc…`, on the chosen
host's pool:

| Object | Name | Lifetime |
|---|---|---|
| base image | `base-<key>.qcow2` (or `base-<url-hash>.qcow2` for a URL) | cached; survives Proxy deletion |
| image marker | `base-<key>.qcow2.sha256` | with the base image |
| overlay disk | `proxy-abc….qcow2`, qcow2 backed by the base | with the Proxy |
| seed ISO | `proxy-abc…-seed.iso` | with the Proxy |
| domain | `proxy-abc…` | with the Proxy |

The domain is a q35 machine with host-passthrough CPU, the overlay on
virtio, the seed on a SATA cdrom, one virtio NIC on the bridge, a serial
console, the guest-agent channel, a virtio RNG and no graphics. Its
`<metadata>` carries the operator's element (managed flag, owning UID,
creation time) — that is how orphan GC recognises its own VMs, since
libvirt has no labels.

The seed's `user-data` is the Proxy's `cloudInit` with the guest-agent
bootstrap **merged into the document**: `qemu-guest-agent` appended to
`packages` and `[systemctl, enable, --now, qemu-guest-agent]` to `runcmd`
(neither duplicated if already there; other keys untouched, though the
YAML is re-rendered). It is not shipped as vendor-data because cloud-init
merges vendor-data under user-data with its default mergers, which keep
the user-data's value for any key both define and never append lists — a
cloud-config with its own `packages:` would silently lose the agent and
the Proxy would sit in `Provisioning` forever. Only when the user-data is
not a cloud-config document (a shell script, `#include`, MIME archive)
does the bootstrap go to `vendor-data`, and then only if `vendorData` is
not configured. A cloud-config that does not parse as a YAML mapping, or
whose `packages`/`runcmd` is not a list, fails the Proxy permanently
rather than booting a VM without its config.

## The image cache

The base image is downloaded **by the operator, through libvirt**, into
the host's pool the first time a Proxy asks for it on that host — nothing
is pre-placed on the hypervisor, and the operator needs no filesystem
access there. The bytes stream from the URL through the operator into
the pool with a running sha256; nothing is buffered. The marker volume is
written last and holds the digest, so a crash mid-download leaves a base
volume without a marker, which the next attempt deletes and redoes.

A re-download happens when: the marker is missing, or `sha256` is
configured and differs from the marker's content. Otherwise the cached
copy is used for as long as it exists, even if the URL's content changed
upstream (a `latest` URL is pinned at the moment of first fetch, per
host).

To evict manually:

```sh
ssh virt@nas1 virsh -c qemu:///system vol-delete --pool default base-debian-13.qcow2.sha256
ssh virt@nas1 virsh -c qemu:///system vol-delete --pool default base-debian-13.qcow2
```

⚠️ **Do not change an image entry's `url` or `sha256`, and do not evict
its base volume, while Proxies using that image exist on the host.** The
overlays back onto the base file; deleting or replacing it corrupts every
VM that uses it. Delete those Proxies first, or add the new image under a
new key and move Proxies over by changing their `placement.image` (which
replaces the VM).

## Logging into a VM

- Put your key in the Proxy's cloud-init (`ssh_authorized_keys`) and
  `ssh debian@<ip>` (Debian) / `ssh ubuntu@<ip>` (Ubuntu).
- Or on the host: `virsh console proxy-abc…` attaches to the serial
  console; cloud-init's output scrolls by there on first boot.
- `virsh domifaddr proxy-abc… --source arp` / `--source agent` shows what
  the operator sees when it looks for the IP.

## Logging

The provider logs through controller-runtime's logger, named `kvm`, with
`provider`, `host` and `name` (the domain) on every line.

- `--zap-log-level=debug` (logr V(1)): one line per provider operation
  (`domain defined`, `KVM domain ensured running`, ...) **and one
  `libvirt call` line per libvirt RPC**, named `kvm.libvirt`, with
  `rpc` (the libvirt API function, e.g. `virDomainLookupByName`), the
  identifiers it was given (`domain`, `pool`, `volume`), curated results
  (`state`, `path`, `interfaces`), `duration`, and on failure `error`
  plus `libvirtError` (the code name, e.g. `ErrNoDomain`) and
  `libvirtCode`. Connection events (`libvirt host connected`, dial
  failures, `libvirt connection dropped` with a reason) log at the same
  level. A Create is about ten RPCs; a Get is four.
- `--zap-log-level=2`: the same lines also carry the domain and volume
  `xml`, the `metadata` element, the interface `addresses` and the
  domain `names` from a listing.
- Cloud-init is never logged at any level: the seed ISO upload reports
  only `length` and `duration`. There is no raw wire tier like the GCP
  provider's V(5); go-libvirt exposes no hook for it.

## Known gaps

- **No quota or capacity detection.** libvirt reports an out-of-space
  pool or an over-committed host as an ordinary error; the reconciler
  treats it as transient and retries with backoff rather than the slower
  quota backoff.
- **Orphan volumes without a domain are invisible to GC.** A crash
  between creating the volumes and defining the domain leaves an overlay
  and seed that `ListByTag` cannot see (it lists domains). They are
  cleaned when the same Proxy reconciles again (Delete removes volumes by
  name even with no domain), but a Proxy CR that is gone for good leaves
  them for `virsh vol-list` to find.
- **The first image fetch is synchronous.** It holds a reconcile worker
  for the download (LAN speed, a few hundred MB, once per image per
  host). Other Proxies keep reconciling on the remaining workers.
- **Replacing an image in use breaks running VMs** — see the warning
  under *The image cache*.
- **IP discovery needs either guest traffic or the agent.** A VM on a
  bridge with no DHCP never gets an address and stays `Provisioning`
  forever; check `virsh console`. The same symptom with a booted, addressed
  VM means the guest agent never came up: check `systemctl status
  qemu-guest-agent` and `cloud-init status --long` in the guest (a VM
  without outbound internet cannot install it).
- **Debian 13 cloud images hang in GRUB under BIOS on a headless VM**
  ([Debian bug #1111240](https://bugs.debian.org/1111240), open, in
  `grub2`): trixie's image ships `terminal_output gfxterm serial` in its
  GRUB config, and with a serial console but no video device (exactly
  the domain this provider defines) GRUB prints "Booting" and never
  hands over to the kernel. Not host-specific; reproduced by the Debian
  cloud team. The same image boots under UEFI with the serial console
  intact, which is why `firmware` defaults to `uefi`. Debian 12
  (bookworm) images are unaffected and boot either way. Images that
  need `firmware: bios` are the exception, not the rule.

## Testing

Unit tests run over an in-memory fake of the libvirt API. The optional
integration test against a real host is described in
[testing.md](testing.md#the-kvm-integration-test-real-hypervisor); its
`-v` output includes the `libvirt call` trace above, at the verbosity
`KVM_TEST_LOG_LEVEL` selects (default 1).