egress-proxies-operator/docs/kvm-provider.md

# The KVM provider

Runs Managed proxies as VMs on your own libvirt/KVM hypervisors. The
operator reaches each host over SSH, forwards the libvirt unix socket
through that connection, and talks libvirt's own RPC over it — so the
hosts need nothing beyond `sshd` and `libvirtd`, and the operator image
carries no libvirt libraries. VMs boot a stock distro cloud image that the
operator downloads into the host's storage pool on first use, with the
Proxy's cloud-init delivered on a NoCloud seed ISO built in-process.

## Hypervisor prerequisites

Per host, once:

1. **libvirt + KVM** (`libvirtd` running, `/dev/kvm` present).
2. **An SSH user in the `libvirt` group**, so it may open
   `/var/run/libvirt/libvirt-sock`. The operator authenticates with a
   private key; nothing else on the host is touched.
3. **A bridge to a network with DHCP** (e.g. `br0` bridged onto the LAN).
   Each VM gets one virtio NIC on it and takes its IP from that network's
   DHCP; the cluster must be able to reach those IPs for health probes
   and for crawlers to use the proxy.
4. **A storage pool** (default `default`, usually `/var/lib/libvirt/images`)
   with room for one base image per distro (a few hundred MB) plus the
   per-VM overlays (`diskSizeGb` each, thin-provisioned).
5. **UEFI firmware for guests**: the `ovmf` package on Debian/Ubuntu,
   `edk2-ovmf` on Fedora/RHEL. VMs boot UEFI by default (libvirt picks
   the OVMF binary itself); a domain start fails with "no firmware
   found" without it. Only `firmware: bios` images skip this.
6. Optionally, **outbound internet from the VMs**: the default
   vendor-data installs `qemu-guest-agent` from the distro mirrors on
   first boot. Without it the VM still works — the operator falls back to
   the host's ARP table to find the IP, which fills in once the guest
   sends traffic.

Check from the operator's point of view before wiring it in:

```sh
ssh virt@nas1.home.lan virsh -c qemu:///system list --all   # user can drive libvirt
ssh virt@nas1.home.lan virsh -c qemu:///system pool-info default
ssh virt@nas1.home.lan ip link show br0
```

## Cluster-side setup

1. Create the SSH Secret. `known_hosts` must contain every host's key
   (`ssh-keyscan -H nas1.home.lan >> known_hosts`); without it, set
   `insecureIgnoreHostKey: true` and accept that a MITM could hand you a
   fake hypervisor.

   ```sh
   kubectl -n egress-proxies-operator-system create secret generic kvm-ssh \
     --from-file=id_ed25519=$HOME/.ssh/proxy-operator_ed25519 \
     --from-file=known_hosts=./known_hosts
   ```

   `config/manager/manager.yaml` mounts it (optional) at
   `/etc/proxy-operator/kvm`; the deployment starts fine without it as
   long as no `kvm` provider is configured.

2. Add a provider entry to the providers config
   (`config/manager/providers_config.yaml` in-cluster; full sample in
   `config/samples/providers-config.yaml`):

   ```yaml
   - name: kvm-home
     type: kvm
     kvm:
       sshKeyFile: /etc/proxy-operator/kvm/id_ed25519
       knownHostsFile: /etc/proxy-operator/kvm/known_hosts
       images:
         debian-13:
           url: https://cloud.debian.org/images/cloud/trixie/latest/debian-13-generic-amd64.qcow2
       machineTypes:
         small: {vcpus: 1, memoryMiB: 1024}
       hosts:
         - name: nas1
           address: nas1.home.lan:22
           user: virt
           bridge: br0
   ```

   The operator reads the key and known_hosts at startup and refuses to
   start if either is missing — a bad path fails fast, not on the first
   Proxy.

3. Apply a Proxy (`config/samples/proxy_kvm.yaml`) and watch it:

   ```sh
   kubectl get px -w
   # NAME               MODE      PROVIDER   PHASE          IP             HEALTHY
   # proxy-kvm-sample   Managed   kvm-home   Provisioning
   # proxy-kvm-sample   Managed   kvm-home   Running        192.168.1.73
   # proxy-kvm-sample   Managed   kvm-home   Ready          192.168.1.73   True
   ```

   The first Proxy for a given image on a given host sits in
   `Provisioning` for the length of the image download; the operator log
   shows `fetching base image into host pool` and `base image cached`
   with the size and duration.

## Config reference

| Field | Required | Default | Meaning |
|---|---|---|---|
| `sshKeyFile` | yes | | private key used for every host |
| `knownHostsFile` | unless `insecureIgnoreHostKey` | | OpenSSH `known_hosts` to verify host keys against |
| `insecureIgnoreHostKey` | | `false` | skip host-key verification entirely |
| `diskSizeGb` | | `10` | capacity of each VM's overlay disk; cloud-init's growpart expands the root fs into it |
| `vendorData` | | installs `qemu-guest-agent` | cloud-init vendor-data placed on every seed next to the Proxy's user-data |
| `images.<key>.url` | yes per entry | | http(s) URL of a cloud image (qcow2 or raw) |
| `images.<key>.sha256` | | | expected digest; verified after download, a change triggers a re-download |
| `images.<key>.format` | | `qcow2` | `qcow2` or `raw` |
| `images.<key>.firmware` | | `uefi` | `uefi` (OVMF, secure boot off) or `bios` (SeaBIOS) for images without an EFI system partition; Debian 13 hangs in GRUB under BIOS on a headless VM, see *Known gaps* |
| `machineTypes.<key>` | ≥ 1 | | `{vcpus, memoryMiB}` size profile |
| `hosts[].name` | yes | | the `placement.zone` value that selects this host; unique |
| `hosts[].address` | yes | | SSH endpoint, `host:port` |
| `hosts[].user` | yes | | SSH user, member of the `libvirt` group |
| `hosts[].socket` | | `/var/run/libvirt/libvirt-sock` | libvirt socket path on the host |
| `hosts[].pool` | | `default` | storage pool for base images, overlays and seeds |
| `hosts[].bridge` | yes | | host bridge for the VM's NIC |

## Placement

| Proxy field | Meaning for kvm |
|---|---|
| `placement.zone` | host name from `hosts`. May be omitted when exactly one host is configured; otherwise required. Unknown → `Failed` |
| `placement.machineType` | key in `machineTypes`. Unknown → `Failed` |
| `placement.image` | key in `images`, or an `http(s)://` URL used directly. Anything else → `Failed` |
| `placement.region` | ignored |

`Failed` here means the reconciler stops retrying and the message names
the offending value; fix the spec (which replaces the VM) or the config.

## What lands on the host

For a Proxy whose deterministic name is `proxy-abc…`, on the chosen
host's pool:

| Object | Name | Lifetime |
|---|---|---|
| base image | `base-<key>.qcow2` (or `base-<url-hash>.qcow2` for a URL) | cached; survives Proxy deletion |
| image marker | `base-<key>.qcow2.sha256` | with the base image |
| overlay disk | `proxy-abc….qcow2`, qcow2 backed by the base | with the Proxy |
| seed ISO | `proxy-abc…-seed.iso` | with the Proxy |
| domain | `proxy-abc…` | with the Proxy |

The domain is a q35 machine with host-passthrough CPU, the overlay on
virtio, the seed on a SATA cdrom, one virtio NIC on the bridge, a serial
console, the guest-agent channel, a virtio RNG and no graphics. Its
`<metadata>` carries the operator's element (managed flag, owning UID,
creation time) — that is how orphan GC recognises its own VMs, since
libvirt has no labels.

## The image cache

The base image is downloaded **by the operator, through libvirt**, into
the host's pool the first time a Proxy asks for it on that host — nothing
is pre-placed on the hypervisor, and the operator needs no filesystem
access there. The bytes stream from the URL through the operator into
the pool with a running sha256; nothing is buffered. The marker volume is
written last and holds the digest, so a crash mid-download leaves a base
volume without a marker, which the next attempt deletes and redoes.

A re-download happens when: the marker is missing, or `sha256` is
configured and differs from the marker's content. Otherwise the cached
copy is used for as long as it exists, even if the URL's content changed
upstream (a `latest` URL is pinned at the moment of first fetch, per
host).

To evict manually:

```sh
ssh virt@nas1 virsh -c qemu:///system vol-delete --pool default base-debian-13.qcow2.sha256
ssh virt@nas1 virsh -c qemu:///system vol-delete --pool default base-debian-13.qcow2
```

⚠️ **Do not change an image entry's `url` or `sha256`, and do not evict
its base volume, while Proxies using that image exist on the host.** The
overlays back onto the base file; deleting or replacing it corrupts every
VM that uses it. Delete those Proxies first, or add the new image under a
new key and move Proxies over by changing their `placement.image` (which
replaces the VM).

## Logging into a VM

- Put your key in the Proxy's cloud-init (`ssh_authorized_keys`) and
  `ssh debian@<ip>` (Debian) / `ssh ubuntu@<ip>` (Ubuntu).
- Or on the host: `virsh console proxy-abc…` attaches to the serial
  console; cloud-init's output scrolls by there on first boot.
- `virsh domifaddr proxy-abc… --source arp` / `--source agent` shows what
  the operator sees when it looks for the IP.

## Logging

The provider logs through controller-runtime's logger, named `kvm`, with
`provider`, `host` and `name` (the domain) on every line.

- `--zap-log-level=debug` (logr V(1)): one line per provider operation
  (`domain defined`, `KVM domain ensured running`, ...) **and one
  `libvirt call` line per libvirt RPC**, named `kvm.libvirt`, with
  `rpc` (the libvirt API function, e.g. `virDomainLookupByName`), the
  identifiers it was given (`domain`, `pool`, `volume`), curated results
  (`state`, `path`, `interfaces`), `duration`, and on failure `error`
  plus `libvirtError` (the code name, e.g. `ErrNoDomain`) and
  `libvirtCode`. Connection events (`libvirt host connected`, dial
  failures, `libvirt connection dropped` with a reason) log at the same
  level. A Create is about ten RPCs; a Get is four.
- `--zap-log-level=2`: the same lines also carry the domain and volume
  `xml`, the `metadata` element, the interface `addresses` and the
  domain `names` from a listing.
- Cloud-init is never logged at any level: the seed ISO upload reports
  only `length` and `duration`. There is no raw wire tier like the GCP
  provider's V(5); go-libvirt exposes no hook for it.

## Known gaps

- **No quota or capacity detection.** libvirt reports an out-of-space
  pool or an over-committed host as an ordinary error; the reconciler
  treats it as transient and retries with backoff rather than the slower
  quota backoff.
- **Orphan volumes without a domain are invisible to GC.** A crash
  between creating the volumes and defining the domain leaves an overlay
  and seed that `ListByTag` cannot see (it lists domains). They are
  cleaned when the same Proxy reconciles again (Delete removes volumes by
  name even with no domain), but a Proxy CR that is gone for good leaves
  them for `virsh vol-list` to find.
- **The first image fetch is synchronous.** It holds a reconcile worker
  for the download (LAN speed, a few hundred MB, once per image per
  host). Other Proxies keep reconciling on the remaining workers.
- **Replacing an image in use breaks running VMs** — see the warning
  under *The image cache*.
- **IP discovery needs either guest traffic or the agent.** A VM on a
  bridge with no DHCP never gets an address and stays `Provisioning`
  forever; check `virsh console`.
- **Debian 13 cloud images hang in GRUB under BIOS on a headless VM**
  ([Debian bug #1111240](https://bugs.debian.org/1111240), open, in
  `grub2`): trixie's image ships `terminal_output gfxterm serial` in its
  GRUB config, and with a serial console but no video device (exactly
  the domain this provider defines) GRUB prints "Booting" and never
  hands over to the kernel. Not host-specific; reproduced by the Debian
  cloud team. The same image boots under UEFI with the serial console
  intact, which is why `firmware` defaults to `uefi`. Debian 12
  (bookworm) images are unaffected and boot either way. Images that
  need `firmware: bios` are the exception, not the rule.

## Testing

Unit tests run over an in-memory fake of the libvirt API. The optional
integration test against a real host is described in
[testing.md](testing.md#the-kvm-integration-test-real-hypervisor); its
`-v` output includes the `libvirt call` trace above, at the verbosity
`KVM_TEST_LOG_LEVEL` selects (default 1).