# Plan: KVM/libvirt provider **Created:** 2026-09-04 14:04 ## Context The operator provisions egress proxies through named provider instances (today `gcp` and `kubernetes`). The user has several KVM hypervisors on the home network and wants proxy VMs created there, using stock Debian/Ubuntu cloud images so the VM stays a normal machine they can customise through their own cloud-init user-data. Longer term the operator may become a generic "run a VM or container with a set of scripts" tool; the KVM provider should not foreclose that. The provider contract (`internal/provider/provider.go`: `Create`/`Get`/`Delete`/ `ListByTag`, keyed by `NameFromUID`, cloud-init passed verbatim, GC-by-tag) fits libvirt with **no CRD or reconciler changes**. All work is a new provider package plus config/wiring/docs, following the GCP provider's shape. Decisions taken with the user (2026-09-04): - **Transport:** SSH to each hypervisor, forwarding the libvirt unix socket (`ssh.Client.Dial("unix", "/var/run/libvirt/libvirt-sock")` → `libvirt.New(conn)`). Pure Go, no cgo, no libvirt libs in the image. Verified: `x/crypto/ssh` `Client.Dial` has a `"unix"` case (`dialStreamLocal`), go-libvirt `New(net.Conn)`. - **Topology:** one provider instance can span several hosts from day one (config has a `hosts` list; providerID encodes the host), so adding hosts later is a config edit with no providerID migration. With one host configured `placement.zone` may be omitted. - **Networking:** VM NIC on a host bridge to the LAN, IP from the router's DHCP. IP discovery via `DomainInterfaceAddresses` (ARP source first, then guest agent). Provider-injected `vendor-data` installs `qemu-guest-agent` so agent discovery works on stock images without touching user-data. - **Base image:** a stock Debian/Ubuntu cloud image (qcow2, cloud-init preinstalled) that **the operator fetches itself** on first use per host: `placement.image` names an entry in the provider's `images:` map (url + optional sha256) or is an `https://` URL directly. The provider streams the download through libvirt's `StorageVolUpload` into the host's pool, so nothing is pre-placed and the operator needs no filesystem access on the host. Per-VM disk is a qcow2 overlay with the cached base as backing store. - **Seed:** NoCloud ISO (`user-data`, `meta-data`, `vendor-data`) built in pure Go with `github.com/kdomanski/iso9660` (`WriteTo(w, "cidata")`) and streamed to the pool with `StorageVolUpload(vol, io.Reader, ...)`. Nothing to install on the hypervisors. ## Design ### Config (`internal/provider/config.go`) ```yaml - name: kvm-home type: kvm kvm: sshKeyFile: /etc/proxy-operator/kvm/id_ed25519 # required knownHostsFile: /etc/proxy-operator/kvm/known_hosts # required unless insecureIgnoreHostKey insecureIgnoreHostKey: false diskSizeGb: 10 # overlay capacity; cloud-init growpart expands root fs vendorData: | # optional override; default installs qemu-guest-agent #cloud-config ... images: # optional; placement.image names a key (or is an https URL) debian-13: url: https://cloud.debian.org/images/cloud/trixie/latest/debian-13-generic-amd64.qcow2 sha256: "…" # optional; verified after download, re-fetch when it changes format: qcow2 # default qcow2 ubuntu-24.04: url: https://cloud-images.ubuntu.com/noble/current/noble-server-cloudimg-amd64.img machineTypes: # required, ≥1; placement.machineType names a key small: {vcpus: 1, memoryMiB: 1024} medium: {vcpus: 2, memoryMiB: 2048} hosts: # required, ≥1, unique names - name: nas1 # placement.zone value address: nas1.home.lan:22 # required user: virt # required; must be in the libvirt group socket: /var/run/libvirt/libvirt-sock # default pool: default # default "default" bridge: br0 # required ``` Add `KVMConfig`, `KVMHost`, `KVMMachineType`, `KVMImage` structs; a `"kvm"` case in `validate()` (cross-block checks like the others, required fields above, unique host names, image URLs must be `http(s)`, sha256 if set must be 64 hex chars). Tests in `config_test.go` extend the existing table. ### Placement mapping | Placement field | KVM meaning | |---|---| | `zone` | host name from `hosts`; empty → the single configured host, else `ErrPermanent` | | `machineType` | key in `machineTypes`; unknown → `ErrPermanent` | | `image` | key in `images:`, or an `https://` URL; anything else → `ErrPermanent`. Fetched into the host's pool on first use (see *Image cache*) | | `region` | ignored | ### providerID `hosts//domains/` with `formatProviderID`/`parseProviderID` (same self-contained style as GCP's `zones//instances/`). ### GC contract without labels Domain XML gets a `` child in a private namespace, e.g. `xmlns="https://gitea.home.hrajfrisbee.cz/kacerr/egress-proxies-operator/kvm"`, carrying `true…RFC3339`. `Get` reads it via `DomainGetMetadata(dom, DomainMetadataElement, uri)`; `ListByTag` does `ConnectListAllDomains` on every host and keeps domains where that call succeeds and `managed == "true"`. `CreatedAt` comes from the metadata (libvirt has no creation time). A host that is unreachable makes `ListByTag` return an error (GC skips the sweep that round rather than acting on a partial view). ### Connection management (`internal/provider/kvm/conn.go`) Per-host lazily dialed `*ssh.Client` + `*libvirt.Libvirt`, cached under a mutex, re-dialed when `Disconnected()` is closed or a call fails with a transport error. SSH auth: private key from `sshKeyFile`; host key check via `knownhosts.New(knownHostsFile)` unless `insecureIgnoreHostKey`. Dial timeout from `ctx`. The provider calls libvirt through a narrow interface: ```go type libvirtAPI interface { DomainLookupByName(name string) (libvirt.Domain, error) DomainDefineXML(xml string) (libvirt.Domain, error) DomainCreate(d libvirt.Domain) error DomainDestroy(d libvirt.Domain) error DomainUndefineFlags(d libvirt.Domain, f libvirt.DomainUndefineFlagsValues) error DomainGetState(d libvirt.Domain, flags uint32) (int32, int32, error) DomainGetMetadata(d libvirt.Domain, typ int32, uri libvirt.OptString, f libvirt.DomainModificationImpact) (string, error) DomainInterfaceAddresses(d libvirt.Domain, src uint32, flags uint32) ([]libvirt.DomainInterface, error) ConnectListAllDomains(need int32, f libvirt.ConnectListAllDomainsFlags) ([]libvirt.Domain, uint32, error) StoragePoolLookupByName(name string) (libvirt.StoragePool, error) StorageVolLookupByName(p libvirt.StoragePool, name string) (libvirt.StorageVol, error) StorageVolGetPath(v libvirt.StorageVol) (string, error) StorageVolCreateXML(p libvirt.StoragePool, xml string, f libvirt.StorageVolCreateFlags) (libvirt.StorageVol, error) StorageVolUpload(v libvirt.StorageVol, r io.Reader, off, n uint64, f libvirt.StorageVolUploadFlags) error StorageVolDownload(v libvirt.StorageVol, w io.Writer, off, n uint64, f libvirt.StorageVolDownloadFlags) error StorageVolDelete(v libvirt.StorageVol, f libvirt.StorageVolDeleteFlags) error StoragePoolRefresh(p libvirt.StoragePool, flags uint32) error } ``` (All signatures verified against go-libvirt `main` on 2026-09-04.) A `hostDialer func(ctx, KVMHost) (libvirtAPI, io.Closer, error)` is the test seam; `newWithDialer(cfg, dialer)` mirrors GCP's `newWithAPI`. ### Image cache (`internal/provider/kvm/image.go`) `ensureImage(ctx, host, api, img) (basePath string, err error)` makes the base image present in the host's pool, once per host per image: - Volume names: `base-.` for an `images:` key, or `base-.` for a URL given directly in `placement.image`. Marker volume `.sha256` (raw, 64 bytes) holds the digest of the verified upload. - **Ready check:** base volume exists *and* marker exists *and* (no `sha256` configured, or marker content == configured `sha256`). Read the marker with `StorageVolDownload`. Ready → return `StorageVolGetPath(base)`. - **Not ready:** delete both volumes if present (partial upload from a crash, or a changed checksum), then `GET url` with the ctx, require a `Content-Length` (`ErrPermanent` otherwise), `StorageVolCreateXML` the base with `Content-Length` and the configured format, `StorageVolUpload(base, io.TeeReader(body, sha256), 0, 0, 0)` (length 0 = until EOF, so nothing is buffered in the operator), then compare the digest with the configured one (mismatch → delete base, `ErrPermanent`), `StoragePoolRefresh` so libvirt re-reads the real qcow2 capacity, create the marker volume and upload the digest last. - HTTP 4xx → `ErrPermanent`; 5xx, network, or upload transport errors → `ErrTransient` (partial volumes are cleaned on the next attempt). - A per-`(host, image)` `sync.Mutex` map serialises concurrent Creates for the same image on the same host; the second caller finds it ready. - **Judgment call:** the fetch runs synchronously inside `Create`, so the first Proxy for a given image on a given host holds a reconcile worker for the download (LAN speed, a few hundred MB, once). This keeps `Create` idempotent and needs no background state. Async fetch with a `Provisioning`-style progress status is a possible later improvement, noted in the docs. ### Create (idempotent, fire-and-forget after the image is cached) 1. Resolve host, machine type, pool, and image source (`images:` key or URL; else `ErrPermanent`). `ensureImage` → base path. 2. Overlay volume `.qcow2`: `StorageVolCreateXML` with `diskSizeGb`, ``, `base`. `ErrStorageVolExist` → reuse. 3. Seed volume `-seed.iso`: build ISO in memory (`user-data` = req.CloudInit, `meta-data` = `instance-id: \nlocal-hostname: \n`, `vendor-data` = cfg or default), `StorageVolCreateXML` with exact capacity + ``, then `StorageVolUpload`. Exists → skip upload (content is deterministic per spec hash; spec changes recreate the whole instance anyway). 4. `DomainLookupByName(name)`; if `ErrNoDomain` → `DomainDefineXML(buildDomainXML(...))`. 5. `DomainGetState`; if not running → `DomainCreate`. Return providerID. `buildDomainXML` is a pure function over `encoding/xml` structs: `type="kvm"`, q35 machine, `host-passthrough` CPU, vcpus/memory from the profile, virtio disk (overlay) + sata cdrom (seed), `interface type="bridge"` with virtio model, `org.qemu.guest_agent.0` virtio channel, serial console, virtio rng, no graphics, and the `` block. Unit-tested field by field like `buildInsertRequest`. ### Get `parseProviderID` → host → `DomainLookupByName` (`ErrNoDomain` → `ErrNotFound`) → `DomainGetState` + `DomainGetMetadata` + `DomainInterfaceAddresses` (ARP, then agent; first IPv4 wins; agent errors while the agent isn't up yet are swallowed). Mapping: | libvirt state | IP | Instance.State | |---|---|---| | running | yes | Running | | running | no | Provisioning | | shutoff / crashed / paused / pmsuspended | – | Stopped | | no domain | – | `ErrNotFound` | ### Delete (idempotent) Destroy if running (`ErrOperationInvalid` on a stopped domain is fine) → `DomainUndefineFlags(managed-save | snapshots-metadata | nvram)` → delete `.qcow2` and `-seed.iso` (`ErrNoStorageVol` fine). Volumes are deleted by name even when the domain is already gone, so a crash between volume creation and define still gets cleaned up on the next reconcile. Known gap (documented, not solved now): volumes orphaned with no domain *and* no CR are invisible to `ListByTag`. ### Errors (`internal/provider/kvm/errors.go`) `classify(err)` on `libvirt.Error.Code`: `ErrNoDomain`, `ErrNoStorageVol` → `ErrNotFound`; `ErrNoStoragePool`, `ErrXMLError`, `ErrConfigUnsupported`, `ErrNoSupport`, `ErrAuthFailed`, `ErrInvalidArg` → `ErrPermanent`; SSH dial / host-key / transport errors and everything else → `ErrTransient`. Image download: HTTP 4xx, missing Content-Length, checksum mismatch → `ErrPermanent`; 5xx and network errors → `ErrTransient`. Quota is not detectable from libvirt; left as transient (revisit with a capacity check later). ### Wiring & manifests - `cmd/main.go`: add `"kvm": kvm.New` to the builtins map. - `config/manager/manager.yaml`: mount Secret `kvm-ssh` (optional) at `/etc/proxy-operator/kvm` (keys `id_ed25519`, `known_hosts`); document the `kubectl create secret generic kvm-ssh --from-file=...` command. - `config/samples/providers-config.yaml`: add the `kvm-home` block with the `images:` map (Debian 13 and Ubuntu 24.04 official cloud-image URLs); `config/samples/proxy_kvm.yaml`: Proxy with `placement: {zone: nas1, machineType: small, image: debian-13}` and the same squid cloud-init as the GCP sample plus `ssh_authorized_keys`. - `hack/providers-dev.yaml` stays kubernetes-only (needs no secrets). - Deps: `github.com/digitalocean/go-libvirt` (latest pseudo-version), `github.com/kdomanski/iso9660 v0.4.0`, promote `golang.org/x/crypto` to direct. ### Docs - New `docs/kvm-provider.md`: hypervisor prerequisites (user in `libvirt` group, a bridge to the LAN, a storage pool with enough space), the config reference including `images:`, how the image cache works (`base-*` and `*.sha256` volumes in the pool, when a re-download happens, how to evict with `virsh vol-delete`), the placement table, how to log into a VM (`ssh_authorized_keys` in user-data, `virsh console` on the host), and the known gaps (no quota detection, orphan volumes without a domain, synchronous first-time image fetch). - Update the provider tables in `README.md`, `docs/architecture.md` §5, `docs/testing.md` (integration test section). ### Towards the generic runner (not built now, just kept open) Provider knows nothing about squid. Future hooks: `vendor-data` for injecting an agent, `QEMUDomainAgentCommand` for reading exit status/output, and a future `Placement`-independent "job" mode in the CRD. None of this changes the KVM provider's shape. ## Steps 0. **Save plan + branch.** Copy this file to `docs/plans/-kvm-provider.md`, commit alone, then `git checkout -b feat/kvm-provider` off `feat/proxy-operator` (the working main for this project). 1. **Config.** `KVMConfig` & friends, `validate()` case, table tests. 2. **Pure builders.** `internal/provider/kvm/{domainxml.go,volume.go,seed.go,id.go,metadata.go}` + tests: domain XML fields, volume XML, seed contents (unpack the ISO with the same library's reader and assert the three files + label `cidata`), providerID round-trip, metadata marshal/parse. 3. **Transport + errors.** `conn.go` (SSH dial, host-key policy, per-host cache, reconnect), `errors.go` + classification table test. SSH dialer tested against an in-process `x/crypto/ssh` server that serves a unix socket forward (cheap, keeps the dial code covered). 3b. **Image cache.** `image.go` + `image_test.go`: `httptest.Server` serving a small fake image over the `fakeAPI` (which records uploaded bytes); cases: fresh fetch, already ready, marker missing (partial upload → redo), checksum changed → redo, checksum mismatch → Permanent + cleanup, 404 → Permanent, 503 → Transient, two concurrent callers → one download. 4. **Provider methods.** `kvm.go`: `New`, `newWithDialer`, `Create/Get/Delete/ListByTag` over `libvirtAPI`; `kvm_test.go` with a hand-written `fakeAPI` (domains and volumes in maps, per-call error injection) covering idempotent re-Create after each partial step, state/IP mapping, delete-when-gone, ListByTag across two hosts and with one host down. 5. **Wiring, manifests, samples, docs, CHANGELOG.** 6. **Integration test** (`//go:build integration`, `kvm_integration_test.go`): skipped unless `KVM_TEST_HOST`, `KVM_TEST_USER`, `KVM_TEST_SSH_KEY`, `KVM_TEST_BRIDGE`, `KVM_TEST_IMAGE_URL` are set; fetches the image (or finds it cached), creates, waits for Running+IP, deletes, asserts per-VM volumes gone and the base image still cached. Deviation from the testcontainers convention, called out in `docs/testing.md`: libvirt+KVM can't run inside a container without `/dev/kvm`, so this targets a real hypervisor. 7. **Manual validation** on one real hypervisor, results in `docs/plans-executions/` (like `docs/gcp-vm-validation.md`), then MR via `tea`. ## Verification - `make test` green (`-race`, envtest suite untouched but must still pass). - `go test -short ./internal/provider/...` covers builders, fake-API provider, errors, config. - Integration: `KVM_TEST_HOST=nas1.home.lan … go test -tags=integration -run TestKVM ./internal/provider/kvm/`. - End to end: deploy with the `kvm-ssh` Secret and `kvm-home` config against a host with no cached image, apply `config/samples/proxy_kvm.yaml`, watch the operator log the image fetch and `virsh vol-list` show `base-debian-13.qcow2` plus its `.sha256` marker, watch `kubectl get px -w` reach `Ready` with a LAN IP, apply a second Proxy and confirm no second download, confirm the VM shows up in `GET /v1/proxies` and works with `hack/` demo egress-IP check, delete the Proxy and confirm `virsh list --all` and `virsh vol-list default` are clean.