egress-proxies-operator/docs/plans/2026-09-04-1404-kvm-provider.md

Plan: KVM/libvirt provider

Created: 2026-09-04 14:04

Context

The operator provisions egress proxies through named provider instances (today gcp and kubernetes). The user has several KVM hypervisors on the home network and wants proxy VMs created there, using stock Debian/Ubuntu cloud images so the VM stays a normal machine they can customise through their own cloud-init user-data. Longer term the operator may become a generic "run a VM or container with a set of scripts" tool; the KVM provider should not foreclose that.

The provider contract (internal/provider/provider.go: Create/Get/Delete/ ListByTag, keyed by NameFromUID, cloud-init passed verbatim, GC-by-tag) fits libvirt with no CRD or reconciler changes. All work is a new provider package plus config/wiring/docs, following the GCP provider's shape.

Decisions taken with the user (2026-09-04):

• Transport: SSH to each hypervisor, forwarding the libvirt unix socket (ssh.Client.Dial("unix", "/var/run/libvirt/libvirt-sock") → libvirt.New(conn)). Pure Go, no cgo, no libvirt libs in the image. Verified: x/crypto/ssh Client.Dial has a "unix" case (dialStreamLocal), go-libvirt New(net.Conn).
• Topology: one provider instance can span several hosts from day one (config has a hosts list; providerID encodes the host), so adding hosts later is a config edit with no providerID migration. With one host configured placement.zone may be omitted.
• Networking: VM NIC on a host bridge to the LAN, IP from the router's DHCP. IP discovery via DomainInterfaceAddresses (ARP source first, then guest agent). Provider-injected vendor-data installs qemu-guest-agent so agent discovery works on stock images without touching user-data.
• Base image: a stock Debian/Ubuntu cloud image (qcow2, cloud-init preinstalled) that the operator fetches itself on first use per host: placement.image names an entry in the provider's images: map (url + optional sha256) or is an https:// URL directly. The provider streams the download through libvirt's StorageVolUpload into the host's pool, so nothing is pre-placed and the operator needs no filesystem access on the host. Per-VM disk is a qcow2 overlay with the cached base as backing store.
• Seed: NoCloud ISO (user-data, meta-data, vendor-data) built in pure Go with github.com/kdomanski/iso9660 (WriteTo(w, "cidata")) and streamed to the pool with StorageVolUpload(vol, io.Reader, ...). Nothing to install on the hypervisors.

Design

Config (internal/provider/config.go)

- name: kvm-home
  type: kvm
  kvm:
    sshKeyFile: /etc/proxy-operator/kvm/id_ed25519        # required
    knownHostsFile: /etc/proxy-operator/kvm/known_hosts   # required unless insecureIgnoreHostKey
    insecureIgnoreHostKey: false
    diskSizeGb: 10                    # overlay capacity; cloud-init growpart expands root fs
    vendorData: |                     # optional override; default installs qemu-guest-agent
      #cloud-config
      ...
    images:                           # optional; placement.image names a key (or is an https URL)
      debian-13:
        url: https://cloud.debian.org/images/cloud/trixie/latest/debian-13-generic-amd64.qcow2
        sha256: "…"                   # optional; verified after download, re-fetch when it changes
        format: qcow2                 # default qcow2
      ubuntu-24.04:
        url: https://cloud-images.ubuntu.com/noble/current/noble-server-cloudimg-amd64.img
    machineTypes:                     # required, ≥1; placement.machineType names a key
      small:  {vcpus: 1, memoryMiB: 1024}
      medium: {vcpus: 2, memoryMiB: 2048}
    hosts:                            # required, ≥1, unique names
      - name: nas1                    # placement.zone value
        address: nas1.home.lan:22     # required
        user: virt                    # required; must be in the libvirt group
        socket: /var/run/libvirt/libvirt-sock   # default
        pool: default                 # default "default"
        bridge: br0                   # required

Add KVMConfig, KVMHost, KVMMachineType, KVMImage structs; a "kvm" case in validate() (cross-block checks like the others, required fields above, unique host names, image URLs must be http(s), sha256 if set must be 64 hex chars). Tests in config_test.go extend the existing table.

Placement mapping

Placement field	KVM meaning
zone	host name from hosts; empty → the single configured host, else ErrPermanent
machineType	key in machineTypes; unknown → ErrPermanent
image	key in images:, or an https:// URL; anything else → ErrPermanent. Fetched into the host's pool on first use (see Image cache)
region	ignored

providerID

hosts/<host>/domains/<name> with formatProviderID/parseProviderID (same self-contained style as GCP's zones/<zone>/instances/<name>).

GC contract without labels

Domain XML gets a <metadata> child in a private namespace, e.g. xmlns="https://gitea.home.hrajfrisbee.cz/kacerr/egress-proxies-operator/kvm", carrying <managed>true</managed><uid>…</uid><createdAt>RFC3339</createdAt>. Get reads it via DomainGetMetadata(dom, DomainMetadataElement, uri); ListByTag does ConnectListAllDomains on every host and keeps domains where that call succeeds and managed == "true". CreatedAt comes from the metadata (libvirt has no creation time). A host that is unreachable makes ListByTag return an error (GC skips the sweep that round rather than acting on a partial view).

Connection management (internal/provider/kvm/conn.go)

Per-host lazily dialed *ssh.Client + *libvirt.Libvirt, cached under a mutex, re-dialed when Disconnected() is closed or a call fails with a transport error. SSH auth: private key from sshKeyFile; host key check via knownhosts.New(knownHostsFile) unless insecureIgnoreHostKey. Dial timeout from ctx. The provider calls libvirt through a narrow interface:

type libvirtAPI interface {
    DomainLookupByName(name string) (libvirt.Domain, error)
    DomainDefineXML(xml string) (libvirt.Domain, error)
    DomainCreate(d libvirt.Domain) error
    DomainDestroy(d libvirt.Domain) error
    DomainUndefineFlags(d libvirt.Domain, f libvirt.DomainUndefineFlagsValues) error
    DomainGetState(d libvirt.Domain, flags uint32) (int32, int32, error)
    DomainGetMetadata(d libvirt.Domain, typ int32, uri libvirt.OptString, f libvirt.DomainModificationImpact) (string, error)
    DomainInterfaceAddresses(d libvirt.Domain, src uint32, flags uint32) ([]libvirt.DomainInterface, error)
    ConnectListAllDomains(need int32, f libvirt.ConnectListAllDomainsFlags) ([]libvirt.Domain, uint32, error)
    StoragePoolLookupByName(name string) (libvirt.StoragePool, error)
    StorageVolLookupByName(p libvirt.StoragePool, name string) (libvirt.StorageVol, error)
    StorageVolGetPath(v libvirt.StorageVol) (string, error)
    StorageVolCreateXML(p libvirt.StoragePool, xml string, f libvirt.StorageVolCreateFlags) (libvirt.StorageVol, error)
    StorageVolUpload(v libvirt.StorageVol, r io.Reader, off, n uint64, f libvirt.StorageVolUploadFlags) error
    StorageVolDownload(v libvirt.StorageVol, w io.Writer, off, n uint64, f libvirt.StorageVolDownloadFlags) error
    StorageVolDelete(v libvirt.StorageVol, f libvirt.StorageVolDeleteFlags) error
    StoragePoolRefresh(p libvirt.StoragePool, flags uint32) error
}

(All signatures verified against go-libvirt main on 2026-09-04.) A hostDialer func(ctx, KVMHost) (libvirtAPI, io.Closer, error) is the test seam; newWithDialer(cfg, dialer) mirrors GCP's newWithAPI.

Image cache (internal/provider/kvm/image.go)

ensureImage(ctx, host, api, img) (basePath string, err error) makes the base image present in the host's pool, once per host per image:

• Volume names: base-<key>.<format> for an images: key, or base-<sha256(url)[:16]>.<format> for a URL given directly in placement.image. Marker volume <base>.sha256 (raw, 64 bytes) holds the digest of the verified upload.
• Ready check: base volume exists and marker exists and (no sha256 configured, or marker content == configured sha256). Read the marker with StorageVolDownload. Ready → return StorageVolGetPath(base).
• Not ready: delete both volumes if present (partial upload from a crash, or a changed checksum), then GET url with the ctx, require a Content-Length (ErrPermanent otherwise), StorageVolCreateXML the base with <capacity unit="bytes">Content-Length</capacity> and the configured format, StorageVolUpload(base, io.TeeReader(body, sha256), 0, 0, 0) (length 0 = until EOF, so nothing is buffered in the operator), then compare the digest with the configured one (mismatch → delete base, ErrPermanent), StoragePoolRefresh so libvirt re-reads the real qcow2 capacity, create the marker volume and upload the digest last.
• HTTP 4xx → ErrPermanent; 5xx, network, or upload transport errors → ErrTransient (partial volumes are cleaned on the next attempt).
• A per-(host, image) sync.Mutex map serialises concurrent Creates for the same image on the same host; the second caller finds it ready.
• Judgment call: the fetch runs synchronously inside Create, so the first Proxy for a given image on a given host holds a reconcile worker for the download (LAN speed, a few hundred MB, once). This keeps Create idempotent and needs no background state. Async fetch with a Provisioning-style progress status is a possible later improvement, noted in the docs.

Create (idempotent, fire-and-forget after the image is cached)

1. Resolve host, machine type, pool, and image source (images: key or URL; else ErrPermanent). ensureImage → base path.
2. Overlay volume <name>.qcow2: StorageVolCreateXML with <capacity unit="G">diskSizeGb</capacity>, <target><format type="qcow2"/></target>, <backingStore><path>base</path><format type="qcow2"/></backingStore>. ErrStorageVolExist → reuse.
3. Seed volume <name>-seed.iso: build ISO in memory (user-data = req.CloudInit, meta-data = instance-id: <name>\nlocal-hostname: <name>\n, vendor-data = cfg or default), StorageVolCreateXML with exact capacity + <format type="raw"/>, then StorageVolUpload. Exists → skip upload (content is deterministic per spec hash; spec changes recreate the whole instance anyway).
4. DomainLookupByName(name); if ErrNoDomain → DomainDefineXML(buildDomainXML(...)).
5. DomainGetState; if not running → DomainCreate. Return providerID.

buildDomainXML is a pure function over encoding/xml structs: type="kvm", q35 machine, host-passthrough CPU, vcpus/memory from the profile, virtio disk (overlay) + sata cdrom (seed), interface type="bridge" with virtio model, org.qemu.guest_agent.0 virtio channel, serial console, virtio rng, no graphics, and the <metadata> block. Unit-tested field by field like buildInsertRequest.

Get

parseProviderID → host → DomainLookupByName (ErrNoDomain → ErrNotFound) → DomainGetState + DomainGetMetadata + DomainInterfaceAddresses (ARP, then agent; first IPv4 wins; agent errors while the agent isn't up yet are swallowed). Mapping:

libvirt state	IP	Instance.State
running	yes	Running
running	no	Provisioning
shutoff / crashed / paused / pmsuspended	–	Stopped
no domain	–	ErrNotFound

Delete (idempotent)

Destroy if running (ErrOperationInvalid on a stopped domain is fine) → DomainUndefineFlags(managed-save | snapshots-metadata | nvram) → delete <name>.qcow2 and <name>-seed.iso (ErrNoStorageVol fine). Volumes are deleted by name even when the domain is already gone, so a crash between volume creation and define still gets cleaned up on the next reconcile. Known gap (documented, not solved now): volumes orphaned with no domain and no CR are invisible to ListByTag.

Errors (internal/provider/kvm/errors.go)

classify(err) on libvirt.Error.Code: ErrNoDomain, ErrNoStorageVol → ErrNotFound; ErrNoStoragePool, ErrXMLError, ErrConfigUnsupported, ErrNoSupport, ErrAuthFailed, ErrInvalidArg → ErrPermanent; SSH dial / host-key / transport errors and everything else → ErrTransient. Image download: HTTP 4xx, missing Content-Length, checksum mismatch → ErrPermanent; 5xx and network errors → ErrTransient. Quota is not detectable from libvirt; left as transient (revisit with a capacity check later).

Wiring & manifests

• cmd/main.go: add "kvm": kvm.New to the builtins map.
• config/manager/manager.yaml: mount Secret kvm-ssh (optional) at /etc/proxy-operator/kvm (keys id_ed25519, known_hosts); document the kubectl create secret generic kvm-ssh --from-file=... command.
• config/samples/providers-config.yaml: add the kvm-home block with the images: map (Debian 13 and Ubuntu 24.04 official cloud-image URLs); config/samples/proxy_kvm.yaml: Proxy with placement: {zone: nas1, machineType: small, image: debian-13} and the same squid cloud-init as the GCP sample plus ssh_authorized_keys.
• hack/providers-dev.yaml stays kubernetes-only (needs no secrets).
• Deps: github.com/digitalocean/go-libvirt (latest pseudo-version), github.com/kdomanski/iso9660 v0.4.0, promote golang.org/x/crypto to direct.

Docs

• New docs/kvm-provider.md: hypervisor prerequisites (user in libvirt group, a bridge to the LAN, a storage pool with enough space), the config reference including images:, how the image cache works (base-* and *.sha256 volumes in the pool, when a re-download happens, how to evict with virsh vol-delete), the placement table, how to log into a VM (ssh_authorized_keys in user-data, virsh console on the host), and the known gaps (no quota detection, orphan volumes without a domain, synchronous first-time image fetch).
• Update the provider tables in README.md, docs/architecture.md §5, docs/testing.md (integration test section).

Towards the generic runner (not built now, just kept open)

Provider knows nothing about squid. Future hooks: vendor-data for injecting an agent, QEMUDomainAgentCommand for reading exit status/output, and a future Placement-independent "job" mode in the CRD. None of this changes the KVM provider's shape.

Steps

0. Save plan + branch. Copy this file to docs/plans/<ts>-kvm-provider.md, commit alone, then git checkout -b feat/kvm-provider off feat/proxy-operator (the working main for this project).
1. Config. KVMConfig & friends, validate() case, table tests.
2. Pure builders. internal/provider/kvm/{domainxml.go,volume.go,seed.go,id.go,metadata.go}
   • tests: domain XML fields, volume XML, seed contents (unpack the ISO with the same library's reader and assert the three files + label cidata), providerID round-trip, metadata marshal/parse.
3. Transport + errors. conn.go (SSH dial, host-key policy, per-host cache, reconnect), errors.go + classification table test. SSH dialer tested against an in-process x/crypto/ssh server that serves a unix socket forward (cheap, keeps the dial code covered). 3b. Image cache. image.go + image_test.go: httptest.Server serving a small fake image over the fakeAPI (which records uploaded bytes); cases: fresh fetch, already ready, marker missing (partial upload → redo), checksum changed → redo, checksum mismatch → Permanent + cleanup, 404 → Permanent, 503 → Transient, two concurrent callers → one download.
4. Provider methods. kvm.go: New, newWithDialer, Create/Get/Delete/ListByTag over libvirtAPI; kvm_test.go with a hand-written fakeAPI (domains and volumes in maps, per-call error injection) covering idempotent re-Create after each partial step, state/IP mapping, delete-when-gone, ListByTag across two hosts and with one host down.
5. Wiring, manifests, samples, docs, CHANGELOG.
6. Integration test (//go:build integration, kvm_integration_test.go): skipped unless KVM_TEST_HOST, KVM_TEST_USER, KVM_TEST_SSH_KEY, KVM_TEST_BRIDGE, KVM_TEST_IMAGE_URL are set; fetches the image (or finds it cached), creates, waits for Running+IP, deletes, asserts per-VM volumes gone and the base image still cached. Deviation from the testcontainers convention, called out in docs/testing.md: libvirt+KVM can't run inside a container without /dev/kvm, so this targets a real hypervisor.
7. Manual validation on one real hypervisor, results in docs/plans-executions/ (like docs/gcp-vm-validation.md), then MR via tea.

Verification

• make test green (-race, envtest suite untouched but must still pass).
• go test -short ./internal/provider/... covers builders, fake-API provider, errors, config.
• Integration: KVM_TEST_HOST=nas1.home.lan … go test -tags=integration -run TestKVM ./internal/provider/kvm/.
• End to end: deploy with the kvm-ssh Secret and kvm-home config against a host with no cached image, apply config/samples/proxy_kvm.yaml, watch the operator log the image fetch and virsh vol-list show base-debian-13.qcow2 plus its .sha256 marker, watch kubectl get px -w reach Ready with a LAN IP, apply a second Proxy and confirm no second download, confirm the VM shows up in GET /v1/proxies and works with hack/ demo egress-IP check, delete the Proxy and confirm virsh list --all and virsh vol-list default are clean.