diff --git a/docs/plans/2026-09-04-1404-kvm-provider.md b/docs/plans/2026-09-04-1404-kvm-provider.md new file mode 100644 index 0000000..029e612 --- /dev/null +++ b/docs/plans/2026-09-04-1404-kvm-provider.md @@ -0,0 +1,313 @@ +# Plan: KVM/libvirt provider + +**Created:** 2026-09-04 14:04 + +## Context + +The operator provisions egress proxies through named provider instances (today +`gcp` and `kubernetes`). The user has several KVM hypervisors on the home +network and wants proxy VMs created there, using stock Debian/Ubuntu cloud +images so the VM stays a normal machine they can customise through their own +cloud-init user-data. Longer term the operator may become a generic "run a VM +or container with a set of scripts" tool; the KVM provider should not +foreclose that. + +The provider contract (`internal/provider/provider.go`: `Create`/`Get`/`Delete`/ +`ListByTag`, keyed by `NameFromUID`, cloud-init passed verbatim, GC-by-tag) fits +libvirt with **no CRD or reconciler changes**. All work is a new provider +package plus config/wiring/docs, following the GCP provider's shape. + +Decisions taken with the user (2026-09-04): + +- **Transport:** SSH to each hypervisor, forwarding the libvirt unix socket + (`ssh.Client.Dial("unix", "/var/run/libvirt/libvirt-sock")` → `libvirt.New(conn)`). + Pure Go, no cgo, no libvirt libs in the image. Verified: `x/crypto/ssh` + `Client.Dial` has a `"unix"` case (`dialStreamLocal`), go-libvirt `New(net.Conn)`. +- **Topology:** one provider instance can span several hosts from day one + (config has a `hosts` list; providerID encodes the host), so adding hosts + later is a config edit with no providerID migration. With one host configured + `placement.zone` may be omitted. +- **Networking:** VM NIC on a host bridge to the LAN, IP from the router's + DHCP. IP discovery via `DomainInterfaceAddresses` (ARP source first, then + guest agent). Provider-injected `vendor-data` installs `qemu-guest-agent` + so agent discovery works on stock images without touching user-data. +- **Base image:** a stock Debian/Ubuntu cloud image (qcow2, cloud-init + preinstalled) that **the operator fetches itself** on first use per host: + `placement.image` names an entry in the provider's `images:` map (url + + optional sha256) or is an `https://` URL directly. The provider streams the + download through libvirt's `StorageVolUpload` into the host's pool, so + nothing is pre-placed and the operator needs no filesystem access on the + host. Per-VM disk is a qcow2 overlay with the cached base as backing store. +- **Seed:** NoCloud ISO (`user-data`, `meta-data`, `vendor-data`) built in + pure Go with `github.com/kdomanski/iso9660` (`WriteTo(w, "cidata")`) and + streamed to the pool with `StorageVolUpload(vol, io.Reader, ...)`. Nothing + to install on the hypervisors. + +## Design + +### Config (`internal/provider/config.go`) + +```yaml +- name: kvm-home + type: kvm + kvm: + sshKeyFile: /etc/proxy-operator/kvm/id_ed25519 # required + knownHostsFile: /etc/proxy-operator/kvm/known_hosts # required unless insecureIgnoreHostKey + insecureIgnoreHostKey: false + diskSizeGb: 10 # overlay capacity; cloud-init growpart expands root fs + vendorData: | # optional override; default installs qemu-guest-agent + #cloud-config + ... + images: # optional; placement.image names a key (or is an https URL) + debian-13: + url: https://cloud.debian.org/images/cloud/trixie/latest/debian-13-generic-amd64.qcow2 + sha256: "…" # optional; verified after download, re-fetch when it changes + format: qcow2 # default qcow2 + ubuntu-24.04: + url: https://cloud-images.ubuntu.com/noble/current/noble-server-cloudimg-amd64.img + machineTypes: # required, ≥1; placement.machineType names a key + small: {vcpus: 1, memoryMiB: 1024} + medium: {vcpus: 2, memoryMiB: 2048} + hosts: # required, ≥1, unique names + - name: nas1 # placement.zone value + address: nas1.home.lan:22 # required + user: virt # required; must be in the libvirt group + socket: /var/run/libvirt/libvirt-sock # default + pool: default # default "default" + bridge: br0 # required +``` + +Add `KVMConfig`, `KVMHost`, `KVMMachineType`, `KVMImage` structs; a `"kvm"` +case in `validate()` (cross-block checks like the others, required fields +above, unique host names, image URLs must be `http(s)`, sha256 if set must be +64 hex chars). Tests in `config_test.go` extend the existing table. + +### Placement mapping + +| Placement field | KVM meaning | +|---|---| +| `zone` | host name from `hosts`; empty → the single configured host, else `ErrPermanent` | +| `machineType` | key in `machineTypes`; unknown → `ErrPermanent` | +| `image` | key in `images:`, or an `https://` URL; anything else → `ErrPermanent`. Fetched into the host's pool on first use (see *Image cache*) | +| `region` | ignored | + +### providerID + +`hosts//domains/` with `formatProviderID`/`parseProviderID` +(same self-contained style as GCP's `zones//instances/`). + +### GC contract without labels + +Domain XML gets a `` child in a private namespace, e.g. +`xmlns="https://gitea.home.hrajfrisbee.cz/kacerr/egress-proxies-operator/kvm"`, +carrying `true…RFC3339`. +`Get` reads it via `DomainGetMetadata(dom, DomainMetadataElement, uri)`; +`ListByTag` does `ConnectListAllDomains` on every host and keeps domains where +that call succeeds and `managed == "true"`. `CreatedAt` comes from the metadata +(libvirt has no creation time). A host that is unreachable makes `ListByTag` +return an error (GC skips the sweep that round rather than acting on a partial view). + +### Connection management (`internal/provider/kvm/conn.go`) + +Per-host lazily dialed `*ssh.Client` + `*libvirt.Libvirt`, cached under a +mutex, re-dialed when `Disconnected()` is closed or a call fails with a +transport error. SSH auth: private key from `sshKeyFile`; host key check via +`knownhosts.New(knownHostsFile)` unless `insecureIgnoreHostKey`. Dial timeout +from `ctx`. The provider calls libvirt through a narrow interface: + +```go +type libvirtAPI interface { + DomainLookupByName(name string) (libvirt.Domain, error) + DomainDefineXML(xml string) (libvirt.Domain, error) + DomainCreate(d libvirt.Domain) error + DomainDestroy(d libvirt.Domain) error + DomainUndefineFlags(d libvirt.Domain, f libvirt.DomainUndefineFlagsValues) error + DomainGetState(d libvirt.Domain, flags uint32) (int32, int32, error) + DomainGetMetadata(d libvirt.Domain, typ int32, uri libvirt.OptString, f libvirt.DomainModificationImpact) (string, error) + DomainInterfaceAddresses(d libvirt.Domain, src uint32, flags uint32) ([]libvirt.DomainInterface, error) + ConnectListAllDomains(need int32, f libvirt.ConnectListAllDomainsFlags) ([]libvirt.Domain, uint32, error) + StoragePoolLookupByName(name string) (libvirt.StoragePool, error) + StorageVolLookupByName(p libvirt.StoragePool, name string) (libvirt.StorageVol, error) + StorageVolGetPath(v libvirt.StorageVol) (string, error) + StorageVolCreateXML(p libvirt.StoragePool, xml string, f libvirt.StorageVolCreateFlags) (libvirt.StorageVol, error) + StorageVolUpload(v libvirt.StorageVol, r io.Reader, off, n uint64, f libvirt.StorageVolUploadFlags) error + StorageVolDownload(v libvirt.StorageVol, w io.Writer, off, n uint64, f libvirt.StorageVolDownloadFlags) error + StorageVolDelete(v libvirt.StorageVol, f libvirt.StorageVolDeleteFlags) error + StoragePoolRefresh(p libvirt.StoragePool, flags uint32) error +} +``` + +(All signatures verified against go-libvirt `main` on 2026-09-04.) A +`hostDialer func(ctx, KVMHost) (libvirtAPI, io.Closer, error)` is the test seam; +`newWithDialer(cfg, dialer)` mirrors GCP's `newWithAPI`. + +### Image cache (`internal/provider/kvm/image.go`) + +`ensureImage(ctx, host, api, img) (basePath string, err error)` makes the base +image present in the host's pool, once per host per image: + +- Volume names: `base-.` for an `images:` key, or + `base-.` for a URL given directly in + `placement.image`. Marker volume `.sha256` (raw, 64 bytes) holds the + digest of the verified upload. +- **Ready check:** base volume exists *and* marker exists *and* (no `sha256` + configured, or marker content == configured `sha256`). Read the marker with + `StorageVolDownload`. Ready → return `StorageVolGetPath(base)`. +- **Not ready:** delete both volumes if present (partial upload from a crash, + or a changed checksum), then `GET url` with the ctx, require a + `Content-Length` (`ErrPermanent` otherwise), `StorageVolCreateXML` the base + with `Content-Length` and the configured + format, `StorageVolUpload(base, io.TeeReader(body, sha256), 0, 0, 0)` (length + 0 = until EOF, so nothing is buffered in the operator), then compare the + digest with the configured one (mismatch → delete base, `ErrPermanent`), + `StoragePoolRefresh` so libvirt re-reads the real qcow2 capacity, create the + marker volume and upload the digest last. +- HTTP 4xx → `ErrPermanent`; 5xx, network, or upload transport errors → + `ErrTransient` (partial volumes are cleaned on the next attempt). +- A per-`(host, image)` `sync.Mutex` map serialises concurrent Creates for the + same image on the same host; the second caller finds it ready. +- **Judgment call:** the fetch runs synchronously inside `Create`, so the first + Proxy for a given image on a given host holds a reconcile worker for the + download (LAN speed, a few hundred MB, once). This keeps `Create` idempotent + and needs no background state. Async fetch with a `Provisioning`-style + progress status is a possible later improvement, noted in the docs. + +### Create (idempotent, fire-and-forget after the image is cached) + +1. Resolve host, machine type, pool, and image source (`images:` key or URL; + else `ErrPermanent`). `ensureImage` → base path. +2. Overlay volume `.qcow2`: `StorageVolCreateXML` with + `diskSizeGb`, ``, + `base`. + `ErrStorageVolExist` → reuse. +3. Seed volume `-seed.iso`: build ISO in memory (`user-data` = req.CloudInit, + `meta-data` = `instance-id: \nlocal-hostname: \n`, `vendor-data` = + cfg or default), `StorageVolCreateXML` with exact capacity + ``, + then `StorageVolUpload`. Exists → skip upload (content is deterministic per spec hash; + spec changes recreate the whole instance anyway). +4. `DomainLookupByName(name)`; if `ErrNoDomain` → `DomainDefineXML(buildDomainXML(...))`. +5. `DomainGetState`; if not running → `DomainCreate`. Return providerID. + +`buildDomainXML` is a pure function over `encoding/xml` structs: `type="kvm"`, +q35 machine, `host-passthrough` CPU, vcpus/memory from the profile, virtio disk +(overlay) + sata cdrom (seed), `interface type="bridge"` with virtio model, +`org.qemu.guest_agent.0` virtio channel, serial console, virtio rng, no +graphics, and the `` block. Unit-tested field by field like +`buildInsertRequest`. + +### Get + +`parseProviderID` → host → `DomainLookupByName` (`ErrNoDomain` → `ErrNotFound`) +→ `DomainGetState` + `DomainGetMetadata` + `DomainInterfaceAddresses` +(ARP, then agent; first IPv4 wins; agent errors while the agent isn't up yet +are swallowed). Mapping: + +| libvirt state | IP | Instance.State | +|---|---|---| +| running | yes | Running | +| running | no | Provisioning | +| shutoff / crashed / paused / pmsuspended | – | Stopped | +| no domain | – | `ErrNotFound` | + +### Delete (idempotent) + +Destroy if running (`ErrOperationInvalid` on a stopped domain is fine) → +`DomainUndefineFlags(managed-save | snapshots-metadata | nvram)` → delete +`.qcow2` and `-seed.iso` (`ErrNoStorageVol` fine). Volumes are +deleted by name even when the domain is already gone, so a crash between +volume creation and define still gets cleaned up on the next reconcile. +Known gap (documented, not solved now): volumes orphaned with no domain *and* +no CR are invisible to `ListByTag`. + +### Errors (`internal/provider/kvm/errors.go`) + +`classify(err)` on `libvirt.Error.Code`: `ErrNoDomain`, `ErrNoStorageVol` → +`ErrNotFound`; `ErrNoStoragePool`, `ErrXMLError`, `ErrConfigUnsupported`, +`ErrNoSupport`, `ErrAuthFailed`, `ErrInvalidArg` → `ErrPermanent`; SSH dial / +host-key / transport errors and everything else → `ErrTransient`. Image +download: HTTP 4xx, missing Content-Length, checksum mismatch → `ErrPermanent`; +5xx and network errors → `ErrTransient`. Quota is not detectable from libvirt; +left as transient (revisit with a capacity check later). + +### Wiring & manifests + +- `cmd/main.go`: add `"kvm": kvm.New` to the builtins map. +- `config/manager/manager.yaml`: mount Secret `kvm-ssh` (optional) at + `/etc/proxy-operator/kvm` (keys `id_ed25519`, `known_hosts`); document the + `kubectl create secret generic kvm-ssh --from-file=...` command. +- `config/samples/providers-config.yaml`: add the `kvm-home` block with the + `images:` map (Debian 13 and Ubuntu 24.04 official cloud-image URLs); + `config/samples/proxy_kvm.yaml`: Proxy with `placement: {zone: nas1, + machineType: small, image: debian-13}` and the same squid cloud-init as the + GCP sample plus `ssh_authorized_keys`. +- `hack/providers-dev.yaml` stays kubernetes-only (needs no secrets). +- Deps: `github.com/digitalocean/go-libvirt` (latest pseudo-version), + `github.com/kdomanski/iso9660 v0.4.0`, promote `golang.org/x/crypto` to direct. + +### Docs + +- New `docs/kvm-provider.md`: hypervisor prerequisites (user in `libvirt` + group, a bridge to the LAN, a storage pool with enough space), the config + reference including `images:`, how the image cache works (`base-*` and + `*.sha256` volumes in the pool, when a re-download happens, how to evict + with `virsh vol-delete`), the placement table, how to log into a VM + (`ssh_authorized_keys` in user-data, `virsh console` on the host), and the + known gaps (no quota detection, orphan volumes without a domain, + synchronous first-time image fetch). +- Update the provider tables in `README.md`, `docs/architecture.md` §5, + `docs/testing.md` (integration test section). + +### Towards the generic runner (not built now, just kept open) + +Provider knows nothing about squid. Future hooks: `vendor-data` for injecting an +agent, `QEMUDomainAgentCommand` for reading exit status/output, and a future +`Placement`-independent "job" mode in the CRD. None of this changes the KVM +provider's shape. + +## Steps + +0. **Save plan + branch.** Copy this file to `docs/plans/-kvm-provider.md`, + commit alone, then `git checkout -b feat/kvm-provider` off `feat/proxy-operator` + (the working main for this project). +1. **Config.** `KVMConfig` & friends, `validate()` case, table tests. +2. **Pure builders.** `internal/provider/kvm/{domainxml.go,volume.go,seed.go,id.go,metadata.go}` + + tests: domain XML fields, volume XML, seed contents (unpack the ISO with + the same library's reader and assert the three files + label `cidata`), + providerID round-trip, metadata marshal/parse. +3. **Transport + errors.** `conn.go` (SSH dial, host-key policy, per-host cache, + reconnect), `errors.go` + classification table test. SSH dialer tested + against an in-process `x/crypto/ssh` server that serves a unix socket + forward (cheap, keeps the dial code covered). +3b. **Image cache.** `image.go` + `image_test.go`: `httptest.Server` serving a + small fake image over the `fakeAPI` (which records uploaded bytes); + cases: fresh fetch, already ready, marker missing (partial upload → redo), + checksum changed → redo, checksum mismatch → Permanent + cleanup, 404 → + Permanent, 503 → Transient, two concurrent callers → one download. +4. **Provider methods.** `kvm.go`: `New`, `newWithDialer`, `Create/Get/Delete/ListByTag` + over `libvirtAPI`; `kvm_test.go` with a hand-written `fakeAPI` (domains and + volumes in maps, per-call error injection) covering idempotent re-Create + after each partial step, state/IP mapping, delete-when-gone, ListByTag + across two hosts and with one host down. +5. **Wiring, manifests, samples, docs, CHANGELOG.** +6. **Integration test** (`//go:build integration`, `kvm_integration_test.go`): + skipped unless `KVM_TEST_HOST`, `KVM_TEST_USER`, `KVM_TEST_SSH_KEY`, + `KVM_TEST_BRIDGE`, `KVM_TEST_IMAGE_URL` are set; fetches the image (or finds + it cached), creates, waits for Running+IP, deletes, asserts per-VM volumes + gone and the base image still cached. Deviation from the testcontainers convention, + called out in `docs/testing.md`: libvirt+KVM can't run inside a container + without `/dev/kvm`, so this targets a real hypervisor. +7. **Manual validation** on one real hypervisor, results in + `docs/plans-executions/` (like `docs/gcp-vm-validation.md`), then MR via `tea`. + +## Verification + +- `make test` green (`-race`, envtest suite untouched but must still pass). +- `go test -short ./internal/provider/...` covers builders, fake-API provider, errors, config. +- Integration: `KVM_TEST_HOST=nas1.home.lan … go test -tags=integration -run TestKVM ./internal/provider/kvm/`. +- End to end: deploy with the `kvm-ssh` Secret and `kvm-home` config against a + host with no cached image, apply `config/samples/proxy_kvm.yaml`, watch the + operator log the image fetch and `virsh vol-list` show `base-debian-13.qcow2` + plus its `.sha256` marker, watch `kubectl get px -w` reach `Ready` with a LAN + IP, apply a second Proxy and confirm no second download, confirm the VM shows up in `GET /v1/proxies` and works with + `hack/` demo egress-IP check, delete the Proxy and confirm `virsh list --all` + and `virsh vol-list default` are clean.