Wire the KVM provider: main.go, kvm-ssh Secret mount, samples, docs, CHANGELOG
Registers "kvm" in the provider builtins, mounts an optional kvm-ssh Secret at /etc/proxy-operator/kvm, adds a kvm-home providers-config sample and a proxy_kvm.yaml Proxy sample, documents the provider in docs/kvm-provider.md (prerequisites, config reference, image cache and its do-not-replace-in-use rule, known gaps), and updates the README and architecture provider sections and decision log. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
This commit is contained in:
Claude Fable 5.1
@@ -0,0 +1,214 @@
|
||||
# The KVM provider
|
||||
|
||||
Runs Managed proxies as VMs on your own libvirt/KVM hypervisors. The
|
||||
operator reaches each host over SSH, forwards the libvirt unix socket
|
||||
through that connection, and talks libvirt's own RPC over it — so the
|
||||
hosts need nothing beyond `sshd` and `libvirtd`, and the operator image
|
||||
carries no libvirt libraries. VMs boot a stock distro cloud image that the
|
||||
operator downloads into the host's storage pool on first use, with the
|
||||
Proxy's cloud-init delivered on a NoCloud seed ISO built in-process.
|
||||
|
||||
## Hypervisor prerequisites
|
||||
|
||||
Per host, once:
|
||||
|
||||
1. **libvirt + KVM** (`libvirtd` running, `/dev/kvm` present).
|
||||
2. **An SSH user in the `libvirt` group**, so it may open
|
||||
`/var/run/libvirt/libvirt-sock`. The operator authenticates with a
|
||||
private key; nothing else on the host is touched.
|
||||
3. **A bridge to a network with DHCP** (e.g. `br0` bridged onto the LAN).
|
||||
Each VM gets one virtio NIC on it and takes its IP from that network's
|
||||
DHCP; the cluster must be able to reach those IPs for health probes
|
||||
and for crawlers to use the proxy.
|
||||
4. **A storage pool** (default `default`, usually `/var/lib/libvirt/images`)
|
||||
with room for one base image per distro (a few hundred MB) plus the
|
||||
per-VM overlays (`diskSizeGb` each, thin-provisioned).
|
||||
5. Optionally, **outbound internet from the VMs**: the default
|
||||
vendor-data installs `qemu-guest-agent` from the distro mirrors on
|
||||
first boot. Without it the VM still works — the operator falls back to
|
||||
the host's ARP table to find the IP, which fills in once the guest
|
||||
sends traffic.
|
||||
|
||||
Check from the operator's point of view before wiring it in:
|
||||
|
||||
```sh
|
||||
ssh virt@nas1.home.lan virsh -c qemu:///system list --all # user can drive libvirt
|
||||
ssh virt@nas1.home.lan virsh -c qemu:///system pool-info default
|
||||
ssh virt@nas1.home.lan ip link show br0
|
||||
```
|
||||
|
||||
## Cluster-side setup
|
||||
|
||||
1. Create the SSH Secret. `known_hosts` must contain every host's key
|
||||
(`ssh-keyscan -H nas1.home.lan >> known_hosts`); without it, set
|
||||
`insecureIgnoreHostKey: true` and accept that a MITM could hand you a
|
||||
fake hypervisor.
|
||||
|
||||
```sh
|
||||
kubectl -n egress-proxies-operator-system create secret generic kvm-ssh \
|
||||
--from-file=id_ed25519=$HOME/.ssh/proxy-operator_ed25519 \
|
||||
--from-file=known_hosts=./known_hosts
|
||||
```
|
||||
|
||||
`config/manager/manager.yaml` mounts it (optional) at
|
||||
`/etc/proxy-operator/kvm`; the deployment starts fine without it as
|
||||
long as no `kvm` provider is configured.
|
||||
|
||||
2. Add a provider entry to the providers config
|
||||
(`config/manager/providers_config.yaml` in-cluster; full sample in
|
||||
`config/samples/providers-config.yaml`):
|
||||
|
||||
```yaml
|
||||
- name: kvm-home
|
||||
type: kvm
|
||||
kvm:
|
||||
sshKeyFile: /etc/proxy-operator/kvm/id_ed25519
|
||||
knownHostsFile: /etc/proxy-operator/kvm/known_hosts
|
||||
images:
|
||||
debian-13:
|
||||
url: https://cloud.debian.org/images/cloud/trixie/latest/debian-13-generic-amd64.qcow2
|
||||
machineTypes:
|
||||
small: {vcpus: 1, memoryMiB: 1024}
|
||||
hosts:
|
||||
- name: nas1
|
||||
address: nas1.home.lan:22
|
||||
user: virt
|
||||
bridge: br0
|
||||
```
|
||||
|
||||
The operator reads the key and known_hosts at startup and refuses to
|
||||
start if either is missing — a bad path fails fast, not on the first
|
||||
Proxy.
|
||||
|
||||
3. Apply a Proxy (`config/samples/proxy_kvm.yaml`) and watch it:
|
||||
|
||||
```sh
|
||||
kubectl get px -w
|
||||
# NAME MODE PROVIDER PHASE IP HEALTHY
|
||||
# proxy-kvm-sample Managed kvm-home Provisioning
|
||||
# proxy-kvm-sample Managed kvm-home Running 192.168.1.73
|
||||
# proxy-kvm-sample Managed kvm-home Ready 192.168.1.73 True
|
||||
```
|
||||
|
||||
The first Proxy for a given image on a given host sits in
|
||||
`Provisioning` for the length of the image download; the operator log
|
||||
shows `fetching base image into host pool` and `base image cached`
|
||||
with the size and duration.
|
||||
|
||||
## Config reference
|
||||
|
||||
| Field | Required | Default | Meaning |
|
||||
|---|---|---|---|
|
||||
| `sshKeyFile` | yes | | private key used for every host |
|
||||
| `knownHostsFile` | unless `insecureIgnoreHostKey` | | OpenSSH `known_hosts` to verify host keys against |
|
||||
| `insecureIgnoreHostKey` | | `false` | skip host-key verification entirely |
|
||||
| `diskSizeGb` | | `10` | capacity of each VM's overlay disk; cloud-init's growpart expands the root fs into it |
|
||||
| `vendorData` | | installs `qemu-guest-agent` | cloud-init vendor-data placed on every seed next to the Proxy's user-data |
|
||||
| `images.<key>.url` | yes per entry | | http(s) URL of a cloud image (qcow2 or raw) |
|
||||
| `images.<key>.sha256` | | | expected digest; verified after download, a change triggers a re-download |
|
||||
| `images.<key>.format` | | `qcow2` | `qcow2` or `raw` |
|
||||
| `machineTypes.<key>` | ≥ 1 | | `{vcpus, memoryMiB}` size profile |
|
||||
| `hosts[].name` | yes | | the `placement.zone` value that selects this host; unique |
|
||||
| `hosts[].address` | yes | | SSH endpoint, `host:port` |
|
||||
| `hosts[].user` | yes | | SSH user, member of the `libvirt` group |
|
||||
| `hosts[].socket` | | `/var/run/libvirt/libvirt-sock` | libvirt socket path on the host |
|
||||
| `hosts[].pool` | | `default` | storage pool for base images, overlays and seeds |
|
||||
| `hosts[].bridge` | yes | | host bridge for the VM's NIC |
|
||||
|
||||
## Placement
|
||||
|
||||
| Proxy field | Meaning for kvm |
|
||||
|---|---|
|
||||
| `placement.zone` | host name from `hosts`. May be omitted when exactly one host is configured; otherwise required. Unknown → `Failed` |
|
||||
| `placement.machineType` | key in `machineTypes`. Unknown → `Failed` |
|
||||
| `placement.image` | key in `images`, or an `http(s)://` URL used directly. Anything else → `Failed` |
|
||||
| `placement.region` | ignored |
|
||||
|
||||
`Failed` here means the reconciler stops retrying and the message names
|
||||
the offending value; fix the spec (which replaces the VM) or the config.
|
||||
|
||||
## What lands on the host
|
||||
|
||||
For a Proxy whose deterministic name is `proxy-abc…`, on the chosen
|
||||
host's pool:
|
||||
|
||||
| Object | Name | Lifetime |
|
||||
|---|---|---|
|
||||
| base image | `base-<key>.qcow2` (or `base-<url-hash>.qcow2` for a URL) | cached; survives Proxy deletion |
|
||||
| image marker | `base-<key>.qcow2.sha256` | with the base image |
|
||||
| overlay disk | `proxy-abc….qcow2`, qcow2 backed by the base | with the Proxy |
|
||||
| seed ISO | `proxy-abc…-seed.iso` | with the Proxy |
|
||||
| domain | `proxy-abc…` | with the Proxy |
|
||||
|
||||
The domain is a q35 machine with host-passthrough CPU, the overlay on
|
||||
virtio, the seed on a SATA cdrom, one virtio NIC on the bridge, a serial
|
||||
console, the guest-agent channel, a virtio RNG and no graphics. Its
|
||||
`<metadata>` carries the operator's element (managed flag, owning UID,
|
||||
creation time) — that is how orphan GC recognises its own VMs, since
|
||||
libvirt has no labels.
|
||||
|
||||
## The image cache
|
||||
|
||||
The base image is downloaded **by the operator, through libvirt**, into
|
||||
the host's pool the first time a Proxy asks for it on that host — nothing
|
||||
is pre-placed on the hypervisor, and the operator needs no filesystem
|
||||
access there. The bytes stream from the URL through the operator into
|
||||
the pool with a running sha256; nothing is buffered. The marker volume is
|
||||
written last and holds the digest, so a crash mid-download leaves a base
|
||||
volume without a marker, which the next attempt deletes and redoes.
|
||||
|
||||
A re-download happens when: the marker is missing, or `sha256` is
|
||||
configured and differs from the marker's content. Otherwise the cached
|
||||
copy is used for as long as it exists, even if the URL's content changed
|
||||
upstream (a `latest` URL is pinned at the moment of first fetch, per
|
||||
host).
|
||||
|
||||
To evict manually:
|
||||
|
||||
```sh
|
||||
ssh virt@nas1 virsh -c qemu:///system vol-delete --pool default base-debian-13.qcow2.sha256
|
||||
ssh virt@nas1 virsh -c qemu:///system vol-delete --pool default base-debian-13.qcow2
|
||||
```
|
||||
|
||||
⚠️ **Do not change an image entry's `url` or `sha256`, and do not evict
|
||||
its base volume, while Proxies using that image exist on the host.** The
|
||||
overlays back onto the base file; deleting or replacing it corrupts every
|
||||
VM that uses it. Delete those Proxies first, or add the new image under a
|
||||
new key and move Proxies over by changing their `placement.image` (which
|
||||
replaces the VM).
|
||||
|
||||
## Logging into a VM
|
||||
|
||||
- Put your key in the Proxy's cloud-init (`ssh_authorized_keys`) and
|
||||
`ssh debian@<ip>` (Debian) / `ssh ubuntu@<ip>` (Ubuntu).
|
||||
- Or on the host: `virsh console proxy-abc…` attaches to the serial
|
||||
console; cloud-init's output scrolls by there on first boot.
|
||||
- `virsh domifaddr proxy-abc… --source arp` / `--source agent` shows what
|
||||
the operator sees when it looks for the IP.
|
||||
|
||||
## Known gaps
|
||||
|
||||
- **No quota or capacity detection.** libvirt reports an out-of-space
|
||||
pool or an over-committed host as an ordinary error; the reconciler
|
||||
treats it as transient and retries with backoff rather than the slower
|
||||
quota backoff.
|
||||
- **Orphan volumes without a domain are invisible to GC.** A crash
|
||||
between creating the volumes and defining the domain leaves an overlay
|
||||
and seed that `ListByTag` cannot see (it lists domains). They are
|
||||
cleaned when the same Proxy reconciles again (Delete removes volumes by
|
||||
name even with no domain), but a Proxy CR that is gone for good leaves
|
||||
them for `virsh vol-list` to find.
|
||||
- **The first image fetch is synchronous.** It holds a reconcile worker
|
||||
for the download (LAN speed, a few hundred MB, once per image per
|
||||
host). Other Proxies keep reconciling on the remaining workers.
|
||||
- **Replacing an image in use breaks running VMs** — see the warning
|
||||
under *The image cache*.
|
||||
- **IP discovery needs either guest traffic or the agent.** A VM on a
|
||||
bridge with no DHCP never gets an address and stays `Provisioning`
|
||||
forever; check `virsh console`.
|
||||
|
||||
## Testing
|
||||
|
||||
Unit tests run over an in-memory fake of the libvirt API. The optional
|
||||
integration test against a real host is described in
|
||||
[testing.md](testing.md#the-kvm-integration-test-real-hypervisor).
|
||||
Reference in New Issue
Block a user