Wire the KVM provider: main.go, kvm-ssh Secret mount, samples, docs, CHANGELOG
Registers "kvm" in the provider builtins, mounts an optional kvm-ssh Secret at /etc/proxy-operator/kvm, adds a kvm-home providers-config sample and a proxy_kvm.yaml Proxy sample, documents the provider in docs/kvm-provider.md (prerequisites, config reference, image cache and its do-not-replace-in-use rule, known gaps), and updates the README and architecture provider sections and decision log. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
This commit is contained in:
Claude Fable 5.1
@@ -9,6 +9,7 @@
|
||||
| Provider contract | `internal/provider` | library | — | `Provider` interface, error taxonomy, deterministic naming, config, metrics decorator |
|
||||
| kubernetes provider | `internal/provider/kubernetes` | library | — | real Squid pods in this cluster (local dev/CI) |
|
||||
| gcp provider | `internal/provider/gcp` | library | — | Compute Engine VMs, four API calls, fire-and-forget ops |
|
||||
| kvm provider | `internal/provider/kvm` | library | — | VMs on libvirt hosts over SSH-forwarded sockets; image cache, overlay disks, NoCloud seeds |
|
||||
| Health engine | `internal/health` | Runnable | yes | through-the-proxy probes, thresholds, transition events |
|
||||
| Lease store | `internal/lease` | Runnable (expiry sweep) | no | in-memory leases + cooldowns, single mutex |
|
||||
| Discovery API | `internal/discovery` | Runnable | no | HTTP list/lease/release/report on `:8090` |
|
||||
@@ -147,6 +148,25 @@ prov.ListByTag ─► AggregatedList(label filter, ─┘ prov
|
||||
ReturnPartialSuccess: true)
|
||||
providerID = zones/<zone>/instances/<name> — zone-qualified, so Get/Delete
|
||||
stay correct even mid-replacement after a zone edit
|
||||
|
||||
kvm provider (internal/provider/kvm/) — libvirt RPC over an SSH-forwarded unix socket, per host
|
||||
prov.Create ──► DomainLookupByName: exists → start if needed, done ─┐ every step tolerates
|
||||
else: ensure base image in pool (download on first use) │ the previous attempt
|
||||
overlay volume (exists = reuse) │ having got that far;
|
||||
seed ISO (always rebuilt — a cut-off upload must │ the seed is the one
|
||||
never be trusted) │ thing redone
|
||||
DomainDefineXML + DomainCreate ─┘
|
||||
prov.Get ──► DomainGetState + DomainGetMetadata + DomainInterfaceAddresses
|
||||
(ARP, then guest agent; first routable IPv4)
|
||||
running without IP = Provisioning; nostate = Provisioning
|
||||
prov.Delete ──► destroy (not running = fine) → undefine → delete overlay + seed by name
|
||||
prov.ListByTag ─► ConnectListAllDomains on every host, keep domains whose
|
||||
<metadata> carries managed=true; one host down fails the sweep,
|
||||
one unreadable domain is skipped
|
||||
providerID = hosts/<host>/domains/<name>; GC identity lives in the domain's
|
||||
<metadata> element (libvirt has no labels). All RPCs run under a context-
|
||||
bounded call: go-libvirt never times out on its own, so a cancelled reconcile
|
||||
closes the SSH connection to unblock it.
|
||||
```
|
||||
|
||||
The reconciler never watches provider-side resources (Pods or GCP VMs).
|
||||
@@ -415,6 +435,30 @@ they read as choices, not accidents. Chronological by build step.
|
||||
- **Unknown GCP instance statuses map to `Stopped`:** the reconciler's
|
||||
answer to Stopped is delete-and-recreate, the always-safe move for
|
||||
cattle when the API grows a new state.
|
||||
- **KVM hosts are reached over SSH with the libvirt socket forwarded,**
|
||||
not libvirtd's TCP/TLS listener: nothing to configure or expose on the
|
||||
hypervisors beyond sshd, pure Go on the operator side (no cgo, no
|
||||
libvirt libraries in the image), and the hosts list can grow by a
|
||||
config edit because the providerID carries the host name.
|
||||
- **KVM base images are fetched by the operator into the host's pool**
|
||||
through libvirt's upload stream, rather than expected to be pre-placed:
|
||||
the operator then needs no filesystem access on the host and a new
|
||||
host is usable with zero manual staging. The cost is a synchronous
|
||||
first-time download inside Create.
|
||||
- **The KVM seed ISO is rebuilt on every Create attempt** before the
|
||||
domain exists, instead of "exists → skip": a reconcile cancelled mid
|
||||
upload leaves a truncated seed, and the compensating delete cannot run
|
||||
because the connection was already closed to unblock the RPC. Skipping
|
||||
it would boot a VM with no cloud-init and no retry would ever repair it.
|
||||
- **KVM `nostate` maps to `Provisioning`, not `Stopped`:** unlike GCP's
|
||||
unknown statuses, `nostate` means libvirt could not determine the
|
||||
state at all; answering that with delete-and-recreate is too drastic,
|
||||
so the next poll decides.
|
||||
- **A KVM domain GC cannot read is skipped, a host it cannot reach fails
|
||||
the sweep.** GC never deletes what it does not see, so leaving one
|
||||
domain out is safe; one corrupt domain blocking GC for every other one
|
||||
is not. A whole host missing from the view would hide many, so that
|
||||
still aborts.
|
||||
- **Kubernetes 403s classify as `ErrPermanent`** even though quota
|
||||
exhaustion also surfaces as 403 (indistinguishable from RBAC denial in
|
||||
`apierrors`): not hammering an API server that may never allow the
|
||||
|
||||
@@ -0,0 +1,214 @@
|
||||
# The KVM provider
|
||||
|
||||
Runs Managed proxies as VMs on your own libvirt/KVM hypervisors. The
|
||||
operator reaches each host over SSH, forwards the libvirt unix socket
|
||||
through that connection, and talks libvirt's own RPC over it — so the
|
||||
hosts need nothing beyond `sshd` and `libvirtd`, and the operator image
|
||||
carries no libvirt libraries. VMs boot a stock distro cloud image that the
|
||||
operator downloads into the host's storage pool on first use, with the
|
||||
Proxy's cloud-init delivered on a NoCloud seed ISO built in-process.
|
||||
|
||||
## Hypervisor prerequisites
|
||||
|
||||
Per host, once:
|
||||
|
||||
1. **libvirt + KVM** (`libvirtd` running, `/dev/kvm` present).
|
||||
2. **An SSH user in the `libvirt` group**, so it may open
|
||||
`/var/run/libvirt/libvirt-sock`. The operator authenticates with a
|
||||
private key; nothing else on the host is touched.
|
||||
3. **A bridge to a network with DHCP** (e.g. `br0` bridged onto the LAN).
|
||||
Each VM gets one virtio NIC on it and takes its IP from that network's
|
||||
DHCP; the cluster must be able to reach those IPs for health probes
|
||||
and for crawlers to use the proxy.
|
||||
4. **A storage pool** (default `default`, usually `/var/lib/libvirt/images`)
|
||||
with room for one base image per distro (a few hundred MB) plus the
|
||||
per-VM overlays (`diskSizeGb` each, thin-provisioned).
|
||||
5. Optionally, **outbound internet from the VMs**: the default
|
||||
vendor-data installs `qemu-guest-agent` from the distro mirrors on
|
||||
first boot. Without it the VM still works — the operator falls back to
|
||||
the host's ARP table to find the IP, which fills in once the guest
|
||||
sends traffic.
|
||||
|
||||
Check from the operator's point of view before wiring it in:
|
||||
|
||||
```sh
|
||||
ssh virt@nas1.home.lan virsh -c qemu:///system list --all # user can drive libvirt
|
||||
ssh virt@nas1.home.lan virsh -c qemu:///system pool-info default
|
||||
ssh virt@nas1.home.lan ip link show br0
|
||||
```
|
||||
|
||||
## Cluster-side setup
|
||||
|
||||
1. Create the SSH Secret. `known_hosts` must contain every host's key
|
||||
(`ssh-keyscan -H nas1.home.lan >> known_hosts`); without it, set
|
||||
`insecureIgnoreHostKey: true` and accept that a MITM could hand you a
|
||||
fake hypervisor.
|
||||
|
||||
```sh
|
||||
kubectl -n egress-proxies-operator-system create secret generic kvm-ssh \
|
||||
--from-file=id_ed25519=$HOME/.ssh/proxy-operator_ed25519 \
|
||||
--from-file=known_hosts=./known_hosts
|
||||
```
|
||||
|
||||
`config/manager/manager.yaml` mounts it (optional) at
|
||||
`/etc/proxy-operator/kvm`; the deployment starts fine without it as
|
||||
long as no `kvm` provider is configured.
|
||||
|
||||
2. Add a provider entry to the providers config
|
||||
(`config/manager/providers_config.yaml` in-cluster; full sample in
|
||||
`config/samples/providers-config.yaml`):
|
||||
|
||||
```yaml
|
||||
- name: kvm-home
|
||||
type: kvm
|
||||
kvm:
|
||||
sshKeyFile: /etc/proxy-operator/kvm/id_ed25519
|
||||
knownHostsFile: /etc/proxy-operator/kvm/known_hosts
|
||||
images:
|
||||
debian-13:
|
||||
url: https://cloud.debian.org/images/cloud/trixie/latest/debian-13-generic-amd64.qcow2
|
||||
machineTypes:
|
||||
small: {vcpus: 1, memoryMiB: 1024}
|
||||
hosts:
|
||||
- name: nas1
|
||||
address: nas1.home.lan:22
|
||||
user: virt
|
||||
bridge: br0
|
||||
```
|
||||
|
||||
The operator reads the key and known_hosts at startup and refuses to
|
||||
start if either is missing — a bad path fails fast, not on the first
|
||||
Proxy.
|
||||
|
||||
3. Apply a Proxy (`config/samples/proxy_kvm.yaml`) and watch it:
|
||||
|
||||
```sh
|
||||
kubectl get px -w
|
||||
# NAME MODE PROVIDER PHASE IP HEALTHY
|
||||
# proxy-kvm-sample Managed kvm-home Provisioning
|
||||
# proxy-kvm-sample Managed kvm-home Running 192.168.1.73
|
||||
# proxy-kvm-sample Managed kvm-home Ready 192.168.1.73 True
|
||||
```
|
||||
|
||||
The first Proxy for a given image on a given host sits in
|
||||
`Provisioning` for the length of the image download; the operator log
|
||||
shows `fetching base image into host pool` and `base image cached`
|
||||
with the size and duration.
|
||||
|
||||
## Config reference
|
||||
|
||||
| Field | Required | Default | Meaning |
|
||||
|---|---|---|---|
|
||||
| `sshKeyFile` | yes | | private key used for every host |
|
||||
| `knownHostsFile` | unless `insecureIgnoreHostKey` | | OpenSSH `known_hosts` to verify host keys against |
|
||||
| `insecureIgnoreHostKey` | | `false` | skip host-key verification entirely |
|
||||
| `diskSizeGb` | | `10` | capacity of each VM's overlay disk; cloud-init's growpart expands the root fs into it |
|
||||
| `vendorData` | | installs `qemu-guest-agent` | cloud-init vendor-data placed on every seed next to the Proxy's user-data |
|
||||
| `images.<key>.url` | yes per entry | | http(s) URL of a cloud image (qcow2 or raw) |
|
||||
| `images.<key>.sha256` | | | expected digest; verified after download, a change triggers a re-download |
|
||||
| `images.<key>.format` | | `qcow2` | `qcow2` or `raw` |
|
||||
| `machineTypes.<key>` | ≥ 1 | | `{vcpus, memoryMiB}` size profile |
|
||||
| `hosts[].name` | yes | | the `placement.zone` value that selects this host; unique |
|
||||
| `hosts[].address` | yes | | SSH endpoint, `host:port` |
|
||||
| `hosts[].user` | yes | | SSH user, member of the `libvirt` group |
|
||||
| `hosts[].socket` | | `/var/run/libvirt/libvirt-sock` | libvirt socket path on the host |
|
||||
| `hosts[].pool` | | `default` | storage pool for base images, overlays and seeds |
|
||||
| `hosts[].bridge` | yes | | host bridge for the VM's NIC |
|
||||
|
||||
## Placement
|
||||
|
||||
| Proxy field | Meaning for kvm |
|
||||
|---|---|
|
||||
| `placement.zone` | host name from `hosts`. May be omitted when exactly one host is configured; otherwise required. Unknown → `Failed` |
|
||||
| `placement.machineType` | key in `machineTypes`. Unknown → `Failed` |
|
||||
| `placement.image` | key in `images`, or an `http(s)://` URL used directly. Anything else → `Failed` |
|
||||
| `placement.region` | ignored |
|
||||
|
||||
`Failed` here means the reconciler stops retrying and the message names
|
||||
the offending value; fix the spec (which replaces the VM) or the config.
|
||||
|
||||
## What lands on the host
|
||||
|
||||
For a Proxy whose deterministic name is `proxy-abc…`, on the chosen
|
||||
host's pool:
|
||||
|
||||
| Object | Name | Lifetime |
|
||||
|---|---|---|
|
||||
| base image | `base-<key>.qcow2` (or `base-<url-hash>.qcow2` for a URL) | cached; survives Proxy deletion |
|
||||
| image marker | `base-<key>.qcow2.sha256` | with the base image |
|
||||
| overlay disk | `proxy-abc….qcow2`, qcow2 backed by the base | with the Proxy |
|
||||
| seed ISO | `proxy-abc…-seed.iso` | with the Proxy |
|
||||
| domain | `proxy-abc…` | with the Proxy |
|
||||
|
||||
The domain is a q35 machine with host-passthrough CPU, the overlay on
|
||||
virtio, the seed on a SATA cdrom, one virtio NIC on the bridge, a serial
|
||||
console, the guest-agent channel, a virtio RNG and no graphics. Its
|
||||
`<metadata>` carries the operator's element (managed flag, owning UID,
|
||||
creation time) — that is how orphan GC recognises its own VMs, since
|
||||
libvirt has no labels.
|
||||
|
||||
## The image cache
|
||||
|
||||
The base image is downloaded **by the operator, through libvirt**, into
|
||||
the host's pool the first time a Proxy asks for it on that host — nothing
|
||||
is pre-placed on the hypervisor, and the operator needs no filesystem
|
||||
access there. The bytes stream from the URL through the operator into
|
||||
the pool with a running sha256; nothing is buffered. The marker volume is
|
||||
written last and holds the digest, so a crash mid-download leaves a base
|
||||
volume without a marker, which the next attempt deletes and redoes.
|
||||
|
||||
A re-download happens when: the marker is missing, or `sha256` is
|
||||
configured and differs from the marker's content. Otherwise the cached
|
||||
copy is used for as long as it exists, even if the URL's content changed
|
||||
upstream (a `latest` URL is pinned at the moment of first fetch, per
|
||||
host).
|
||||
|
||||
To evict manually:
|
||||
|
||||
```sh
|
||||
ssh virt@nas1 virsh -c qemu:///system vol-delete --pool default base-debian-13.qcow2.sha256
|
||||
ssh virt@nas1 virsh -c qemu:///system vol-delete --pool default base-debian-13.qcow2
|
||||
```
|
||||
|
||||
⚠️ **Do not change an image entry's `url` or `sha256`, and do not evict
|
||||
its base volume, while Proxies using that image exist on the host.** The
|
||||
overlays back onto the base file; deleting or replacing it corrupts every
|
||||
VM that uses it. Delete those Proxies first, or add the new image under a
|
||||
new key and move Proxies over by changing their `placement.image` (which
|
||||
replaces the VM).
|
||||
|
||||
## Logging into a VM
|
||||
|
||||
- Put your key in the Proxy's cloud-init (`ssh_authorized_keys`) and
|
||||
`ssh debian@<ip>` (Debian) / `ssh ubuntu@<ip>` (Ubuntu).
|
||||
- Or on the host: `virsh console proxy-abc…` attaches to the serial
|
||||
console; cloud-init's output scrolls by there on first boot.
|
||||
- `virsh domifaddr proxy-abc… --source arp` / `--source agent` shows what
|
||||
the operator sees when it looks for the IP.
|
||||
|
||||
## Known gaps
|
||||
|
||||
- **No quota or capacity detection.** libvirt reports an out-of-space
|
||||
pool or an over-committed host as an ordinary error; the reconciler
|
||||
treats it as transient and retries with backoff rather than the slower
|
||||
quota backoff.
|
||||
- **Orphan volumes without a domain are invisible to GC.** A crash
|
||||
between creating the volumes and defining the domain leaves an overlay
|
||||
and seed that `ListByTag` cannot see (it lists domains). They are
|
||||
cleaned when the same Proxy reconciles again (Delete removes volumes by
|
||||
name even with no domain), but a Proxy CR that is gone for good leaves
|
||||
them for `virsh vol-list` to find.
|
||||
- **The first image fetch is synchronous.** It holds a reconcile worker
|
||||
for the download (LAN speed, a few hundred MB, once per image per
|
||||
host). Other Proxies keep reconciling on the remaining workers.
|
||||
- **Replacing an image in use breaks running VMs** — see the warning
|
||||
under *The image cache*.
|
||||
- **IP discovery needs either guest traffic or the agent.** A VM on a
|
||||
bridge with no DHCP never gets an address and stays `Provisioning`
|
||||
forever; check `virsh console`.
|
||||
|
||||
## Testing
|
||||
|
||||
Unit tests run over an in-memory fake of the libvirt API. The optional
|
||||
integration test against a real host is described in
|
||||
[testing.md](testing.md#the-kvm-integration-test-real-hypervisor).
|
||||
@@ -8,7 +8,7 @@ Plan: [2026-09-04-1404-kvm-provider.md](../plans/2026-09-04-1404-kvm-provider.md
|
||||
- [x] Step 3 — Transport + errors (`conn.go`, `errors.go`)
|
||||
- [x] Step 3b — Image cache (`image.go`)
|
||||
- [x] Step 4 — Provider methods (`kvm.go`) over a fake API
|
||||
- [ ] Step 5 — Wiring, manifests, samples, docs, CHANGELOG
|
||||
- [x] Step 5 — Wiring, manifests, samples, docs, CHANGELOG
|
||||
- [x] Step 6 — Integration test against a real hypervisor (written and compiled; first real run is part of Step 7)
|
||||
- [ ] Step 7 — Manual validation + MR
|
||||
|
||||
@@ -338,3 +338,54 @@ Step 3b "confirm on a real host" notes get answered. The test's config
|
||||
is built directly rather than through the provider package's validation
|
||||
(unexported), so a typo in an env var surfaces from `New` or from
|
||||
libvirt, not from a validation message.
|
||||
|
||||
## Step 5 — Wiring, manifests, samples, docs, CHANGELOG
|
||||
|
||||
Done after Step 6 (see that entry for why the order flipped). Everything
|
||||
the plan listed, plus a few things it did not:
|
||||
|
||||
- `cmd/main.go`: `"kvm": kvm.New` in the builtins map. No RBAC change —
|
||||
the provider talks to hypervisors, not the API server.
|
||||
- `config/manager/manager.yaml`: `kvm-ssh` Secret mounted read-only at
|
||||
`/etc/proxy-operator/kvm` with `optional: true`, so deployments without
|
||||
a kvm provider are unaffected. It nests under the providers-config
|
||||
mount at `/etc/proxy-operator`; kustomize renders it (`kubectl
|
||||
kustomize config/default` checked). The container runs as uid 65532
|
||||
and Secret files default to 0644, so the key is readable without an
|
||||
`fsGroup`; the `kubectl create secret` recipe is in a comment next to
|
||||
the mount, in `docs/kvm-provider.md` and in the README.
|
||||
- `config/samples/providers-config.yaml`: `kvm-home` block with Debian 13
|
||||
and Ubuntu 24.04 official image URLs, two machine types, one host, and
|
||||
every optional field shown commented out. Parsed through
|
||||
`provider.LoadConfigFile` in a throwaway test to make sure the sample
|
||||
validates. `config/samples/proxy_kvm.yaml` uses the GCP sample's squid
|
||||
cloud-init and shows `ssh_authorized_keys` commented out rather than
|
||||
set — a sample with a placeholder key would either fail cloud-init or
|
||||
invite pasting a real key into a committed file. Added to the samples
|
||||
kustomization; `config/manager/providers_config.yaml`'s comment now
|
||||
mentions kvm.
|
||||
- `docs/kvm-provider.md`: prerequisites with a three-line preflight
|
||||
(`virsh list`, `pool-info`, `ip link` over ssh), cluster setup, the full
|
||||
config reference table, placement table, a table of what lands on the
|
||||
host and its lifetime, how the image cache works and how to evict, the
|
||||
**warning not to change or evict an image while Proxies use it** (the
|
||||
reviewer's finding from Step 4, carried as a documented rule since the
|
||||
content-addressed-naming alternative was not chosen), how to log in,
|
||||
and the known gaps.
|
||||
- README: providers bullet and a "KVM setup" section mirroring the GCP
|
||||
one. `docs/architecture.md`: components row, a §5 block for the kvm
|
||||
call flow in the same style as the other two, and five decision-log
|
||||
entries (SSH transport, operator-fetched images, seed always rebuilt,
|
||||
`nostate` → Provisioning, skip-vs-fail in the GC sweep).
|
||||
- `CHANGELOG.md` entry at the top, stating explicitly that the provider
|
||||
is not yet validated on a real host.
|
||||
|
||||
Verification: `go build ./...`, `go vet ./...`, `go test -short` on
|
||||
`cmd` and `internal/provider/...`, and both kustomizations rendered.
|
||||
`hack/providers-dev.yaml` untouched, as planned.
|
||||
|
||||
Worth noting: the CRD has no rule requiring all three placement fields
|
||||
(that is GCP's own check), so the single-host "zone may be omitted"
|
||||
behaviour works without an API change. And with this step the branch is
|
||||
feature-complete but **unproven**: Step 7 runs the integration test and
|
||||
the end-to-end checklist against a real hypervisor before the MR.
|
||||
|
||||
Reference in New Issue
Block a user