egress-proxies-operator/docs/plans/2026-09-04-1404-kvm-provider.md

# Plan: KVM/libvirt provider

**Created:** 2026-09-04 14:04

## Context

The operator provisions egress proxies through named provider instances (today
`gcp` and `kubernetes`). The user has several KVM hypervisors on the home
network and wants proxy VMs created there, using stock Debian/Ubuntu cloud
images so the VM stays a normal machine they can customise through their own
cloud-init user-data. Longer term the operator may become a generic "run a VM
or container with a set of scripts" tool; the KVM provider should not
foreclose that.

The provider contract (`internal/provider/provider.go`: `Create`/`Get`/`Delete`/
`ListByTag`, keyed by `NameFromUID`, cloud-init passed verbatim, GC-by-tag) fits
libvirt with **no CRD or reconciler changes**. All work is a new provider
package plus config/wiring/docs, following the GCP provider's shape.

Decisions taken with the user (2026-09-04):

- **Transport:** SSH to each hypervisor, forwarding the libvirt unix socket
  (`ssh.Client.Dial("unix", "/var/run/libvirt/libvirt-sock")` → `libvirt.New(conn)`).
  Pure Go, no cgo, no libvirt libs in the image. Verified: `x/crypto/ssh`
  `Client.Dial` has a `"unix"` case (`dialStreamLocal`), go-libvirt `New(net.Conn)`.
- **Topology:** one provider instance can span several hosts from day one
  (config has a `hosts` list; providerID encodes the host), so adding hosts
  later is a config edit with no providerID migration. With one host configured
  `placement.zone` may be omitted.
- **Networking:** VM NIC on a host bridge to the LAN, IP from the router's
  DHCP. IP discovery via `DomainInterfaceAddresses` (ARP source first, then
  guest agent). Provider-injected `vendor-data` installs `qemu-guest-agent`
  so agent discovery works on stock images without touching user-data.
- **Base image:** a stock Debian/Ubuntu cloud image (qcow2, cloud-init
  preinstalled) that **the operator fetches itself** on first use per host:
  `placement.image` names an entry in the provider's `images:` map (url +
  optional sha256) or is an `https://` URL directly. The provider streams the
  download through libvirt's `StorageVolUpload` into the host's pool, so
  nothing is pre-placed and the operator needs no filesystem access on the
  host. Per-VM disk is a qcow2 overlay with the cached base as backing store.
- **Seed:** NoCloud ISO (`user-data`, `meta-data`, `vendor-data`) built in
  pure Go with `github.com/kdomanski/iso9660` (`WriteTo(w, "cidata")`) and
  streamed to the pool with `StorageVolUpload(vol, io.Reader, ...)`. Nothing
  to install on the hypervisors.

## Design

### Config (`internal/provider/config.go`)

```yaml
- name: kvm-home
  type: kvm
  kvm:
    sshKeyFile: /etc/proxy-operator/kvm/id_ed25519        # required
    knownHostsFile: /etc/proxy-operator/kvm/known_hosts   # required unless insecureIgnoreHostKey
    insecureIgnoreHostKey: false
    diskSizeGb: 10                    # overlay capacity; cloud-init growpart expands root fs
    vendorData: |                     # optional override; default installs qemu-guest-agent
      #cloud-config
      ...
    images:                           # optional; placement.image names a key (or is an https URL)
      debian-13:
        url: https://cloud.debian.org/images/cloud/trixie/latest/debian-13-generic-amd64.qcow2
        sha256: "…"                   # optional; verified after download, re-fetch when it changes
        format: qcow2                 # default qcow2
      ubuntu-24.04:
        url: https://cloud-images.ubuntu.com/noble/current/noble-server-cloudimg-amd64.img
    machineTypes:                     # required, ≥1; placement.machineType names a key
      small:  {vcpus: 1, memoryMiB: 1024}
      medium: {vcpus: 2, memoryMiB: 2048}
    hosts:                            # required, ≥1, unique names
      - name: nas1                    # placement.zone value
        address: nas1.home.lan:22     # required
        user: virt                    # required; must be in the libvirt group
        socket: /var/run/libvirt/libvirt-sock   # default
        pool: default                 # default "default"
        bridge: br0                   # required
```

Add `KVMConfig`, `KVMHost`, `KVMMachineType`, `KVMImage` structs; a `"kvm"`
case in `validate()` (cross-block checks like the others, required fields
above, unique host names, image URLs must be `http(s)`, sha256 if set must be
64 hex chars). Tests in `config_test.go` extend the existing table.

### Placement mapping

| Placement field | KVM meaning |
|---|---|
| `zone` | host name from `hosts`; empty → the single configured host, else `ErrPermanent` |
| `machineType` | key in `machineTypes`; unknown → `ErrPermanent` |
| `image` | key in `images:`, or an `https://` URL; anything else → `ErrPermanent`. Fetched into the host's pool on first use (see *Image cache*) |
| `region` | ignored |

### providerID

`hosts/<host>/domains/<name>` with `formatProviderID`/`parseProviderID`
(same self-contained style as GCP's `zones/<zone>/instances/<name>`).

### GC contract without labels

Domain XML gets a `<metadata>` child in a private namespace, e.g.
`xmlns="https://gitea.home.hrajfrisbee.cz/kacerr/egress-proxies-operator/kvm"`,
carrying `<managed>true</managed><uid>…</uid><createdAt>RFC3339</createdAt>`.
`Get` reads it via `DomainGetMetadata(dom, DomainMetadataElement, uri)`;
`ListByTag` does `ConnectListAllDomains` on every host and keeps domains where
that call succeeds and `managed == "true"`. `CreatedAt` comes from the metadata
(libvirt has no creation time). A host that is unreachable makes `ListByTag`
return an error (GC skips the sweep that round rather than acting on a partial view).

### Connection management (`internal/provider/kvm/conn.go`)

Per-host lazily dialed `*ssh.Client` + `*libvirt.Libvirt`, cached under a
mutex, re-dialed when `Disconnected()` is closed or a call fails with a
transport error. SSH auth: private key from `sshKeyFile`; host key check via
`knownhosts.New(knownHostsFile)` unless `insecureIgnoreHostKey`. Dial timeout
from `ctx`. The provider calls libvirt through a narrow interface:

```go
type libvirtAPI interface {
    DomainLookupByName(name string) (libvirt.Domain, error)
    DomainDefineXML(xml string) (libvirt.Domain, error)
    DomainCreate(d libvirt.Domain) error
    DomainDestroy(d libvirt.Domain) error
    DomainUndefineFlags(d libvirt.Domain, f libvirt.DomainUndefineFlagsValues) error
    DomainGetState(d libvirt.Domain, flags uint32) (int32, int32, error)
    DomainGetMetadata(d libvirt.Domain, typ int32, uri libvirt.OptString, f libvirt.DomainModificationImpact) (string, error)
    DomainInterfaceAddresses(d libvirt.Domain, src uint32, flags uint32) ([]libvirt.DomainInterface, error)
    ConnectListAllDomains(need int32, f libvirt.ConnectListAllDomainsFlags) ([]libvirt.Domain, uint32, error)
    StoragePoolLookupByName(name string) (libvirt.StoragePool, error)
    StorageVolLookupByName(p libvirt.StoragePool, name string) (libvirt.StorageVol, error)
    StorageVolGetPath(v libvirt.StorageVol) (string, error)
    StorageVolCreateXML(p libvirt.StoragePool, xml string, f libvirt.StorageVolCreateFlags) (libvirt.StorageVol, error)
    StorageVolUpload(v libvirt.StorageVol, r io.Reader, off, n uint64, f libvirt.StorageVolUploadFlags) error
    StorageVolDownload(v libvirt.StorageVol, w io.Writer, off, n uint64, f libvirt.StorageVolDownloadFlags) error
    StorageVolDelete(v libvirt.StorageVol, f libvirt.StorageVolDeleteFlags) error
    StoragePoolRefresh(p libvirt.StoragePool, flags uint32) error
}
```

(All signatures verified against go-libvirt `main` on 2026-09-04.) A
`hostDialer func(ctx, KVMHost) (libvirtAPI, io.Closer, error)` is the test seam;
`newWithDialer(cfg, dialer)` mirrors GCP's `newWithAPI`.

### Image cache (`internal/provider/kvm/image.go`)

`ensureImage(ctx, host, api, img) (basePath string, err error)` makes the base
image present in the host's pool, once per host per image:

- Volume names: `base-<key>.<format>` for an `images:` key, or
  `base-<sha256(url)[:16]>.<format>` for a URL given directly in
  `placement.image`. Marker volume `<base>.sha256` (raw, 64 bytes) holds the
  digest of the verified upload.
- **Ready check:** base volume exists *and* marker exists *and* (no `sha256`
  configured, or marker content == configured `sha256`). Read the marker with
  `StorageVolDownload`. Ready → return `StorageVolGetPath(base)`.
- **Not ready:** delete both volumes if present (partial upload from a crash,
  or a changed checksum), then `GET url` with the ctx, require a
  `Content-Length` (`ErrPermanent` otherwise), `StorageVolCreateXML` the base
  with `<capacity unit="bytes">Content-Length</capacity>` and the configured
  format, `StorageVolUpload(base, io.TeeReader(body, sha256), 0, 0, 0)` (length
  0 = until EOF, so nothing is buffered in the operator), then compare the
  digest with the configured one (mismatch → delete base, `ErrPermanent`),
  `StoragePoolRefresh` so libvirt re-reads the real qcow2 capacity, create the
  marker volume and upload the digest last.
- HTTP 4xx → `ErrPermanent`; 5xx, network, or upload transport errors →
  `ErrTransient` (partial volumes are cleaned on the next attempt).
- A per-`(host, image)` `sync.Mutex` map serialises concurrent Creates for the
  same image on the same host; the second caller finds it ready.
- **Judgment call:** the fetch runs synchronously inside `Create`, so the first
  Proxy for a given image on a given host holds a reconcile worker for the
  download (LAN speed, a few hundred MB, once). This keeps `Create` idempotent
  and needs no background state. Async fetch with a `Provisioning`-style
  progress status is a possible later improvement, noted in the docs.

### Create (idempotent, fire-and-forget after the image is cached)

1. Resolve host, machine type, pool, and image source (`images:` key or URL;
   else `ErrPermanent`). `ensureImage` → base path.
2. Overlay volume `<name>.qcow2`: `StorageVolCreateXML` with
   `<capacity unit="G">diskSizeGb</capacity>`, `<target><format type="qcow2"/></target>`,
   `<backingStore><path>base</path><format type="qcow2"/></backingStore>`.
   `ErrStorageVolExist` → reuse.
3. Seed volume `<name>-seed.iso`: build ISO in memory (`user-data` = req.CloudInit,
   `meta-data` = `instance-id: <name>\nlocal-hostname: <name>\n`, `vendor-data` =
   cfg or default), `StorageVolCreateXML` with exact capacity + `<format type="raw"/>`,
   then `StorageVolUpload`. Exists → skip upload (content is deterministic per spec hash;
   spec changes recreate the whole instance anyway).
4. `DomainLookupByName(name)`; if `ErrNoDomain` → `DomainDefineXML(buildDomainXML(...))`.
5. `DomainGetState`; if not running → `DomainCreate`. Return providerID.

`buildDomainXML` is a pure function over `encoding/xml` structs: `type="kvm"`,
q35 machine, `host-passthrough` CPU, vcpus/memory from the profile, virtio disk
(overlay) + sata cdrom (seed), `interface type="bridge"` with virtio model,
`org.qemu.guest_agent.0` virtio channel, serial console, virtio rng, no
graphics, and the `<metadata>` block. Unit-tested field by field like
`buildInsertRequest`.

### Get

`parseProviderID` → host → `DomainLookupByName` (`ErrNoDomain` → `ErrNotFound`)
→ `DomainGetState` + `DomainGetMetadata` + `DomainInterfaceAddresses`
(ARP, then agent; first IPv4 wins; agent errors while the agent isn't up yet
are swallowed). Mapping:

| libvirt state | IP | Instance.State |
|---|---|---|
| running | yes | Running |
| running | no | Provisioning |
| shutoff / crashed / paused / pmsuspended | – | Stopped |
| no domain | – | `ErrNotFound` |

### Delete (idempotent)

Destroy if running (`ErrOperationInvalid` on a stopped domain is fine) →
`DomainUndefineFlags(managed-save | snapshots-metadata | nvram)` → delete
`<name>.qcow2` and `<name>-seed.iso` (`ErrNoStorageVol` fine). Volumes are
deleted by name even when the domain is already gone, so a crash between
volume creation and define still gets cleaned up on the next reconcile.
Known gap (documented, not solved now): volumes orphaned with no domain *and*
no CR are invisible to `ListByTag`.

### Errors (`internal/provider/kvm/errors.go`)

`classify(err)` on `libvirt.Error.Code`: `ErrNoDomain`, `ErrNoStorageVol` →
`ErrNotFound`; `ErrNoStoragePool`, `ErrXMLError`, `ErrConfigUnsupported`,
`ErrNoSupport`, `ErrAuthFailed`, `ErrInvalidArg` → `ErrPermanent`; SSH dial /
host-key / transport errors and everything else → `ErrTransient`. Image
download: HTTP 4xx, missing Content-Length, checksum mismatch → `ErrPermanent`;
5xx and network errors → `ErrTransient`. Quota is not detectable from libvirt;
left as transient (revisit with a capacity check later).

### Wiring & manifests

- `cmd/main.go`: add `"kvm": kvm.New` to the builtins map.
- `config/manager/manager.yaml`: mount Secret `kvm-ssh` (optional) at
  `/etc/proxy-operator/kvm` (keys `id_ed25519`, `known_hosts`); document the
  `kubectl create secret generic kvm-ssh --from-file=...` command.
- `config/samples/providers-config.yaml`: add the `kvm-home` block with the
  `images:` map (Debian 13 and Ubuntu 24.04 official cloud-image URLs);
  `config/samples/proxy_kvm.yaml`: Proxy with `placement: {zone: nas1,
  machineType: small, image: debian-13}` and the same squid cloud-init as the
  GCP sample plus `ssh_authorized_keys`.
- `hack/providers-dev.yaml` stays kubernetes-only (needs no secrets).
- Deps: `github.com/digitalocean/go-libvirt` (latest pseudo-version),
  `github.com/kdomanski/iso9660 v0.4.0`, promote `golang.org/x/crypto` to direct.

### Docs

- New `docs/kvm-provider.md`: hypervisor prerequisites (user in `libvirt`
  group, a bridge to the LAN, a storage pool with enough space), the config
  reference including `images:`, how the image cache works (`base-*` and
  `*.sha256` volumes in the pool, when a re-download happens, how to evict
  with `virsh vol-delete`), the placement table, how to log into a VM
  (`ssh_authorized_keys` in user-data, `virsh console` on the host), and the
  known gaps (no quota detection, orphan volumes without a domain,
  synchronous first-time image fetch).
- Update the provider tables in `README.md`, `docs/architecture.md` §5,
  `docs/testing.md` (integration test section).

### Towards the generic runner (not built now, just kept open)

Provider knows nothing about squid. Future hooks: `vendor-data` for injecting an
agent, `QEMUDomainAgentCommand` for reading exit status/output, and a future
`Placement`-independent "job" mode in the CRD. None of this changes the KVM
provider's shape.

## Steps

0. **Save plan + branch.** Copy this file to `docs/plans/<ts>-kvm-provider.md`,
   commit alone, then `git checkout -b feat/kvm-provider` off `feat/proxy-operator`
   (the working main for this project).
1. **Config.** `KVMConfig` & friends, `validate()` case, table tests.
2. **Pure builders.** `internal/provider/kvm/{domainxml.go,volume.go,seed.go,id.go,metadata.go}`
   + tests: domain XML fields, volume XML, seed contents (unpack the ISO with
   the same library's reader and assert the three files + label `cidata`),
   providerID round-trip, metadata marshal/parse.
3. **Transport + errors.** `conn.go` (SSH dial, host-key policy, per-host cache,
   reconnect), `errors.go` + classification table test. SSH dialer tested
   against an in-process `x/crypto/ssh` server that serves a unix socket
   forward (cheap, keeps the dial code covered).
3b. **Image cache.** `image.go` + `image_test.go`: `httptest.Server` serving a
   small fake image over the `fakeAPI` (which records uploaded bytes);
   cases: fresh fetch, already ready, marker missing (partial upload → redo),
   checksum changed → redo, checksum mismatch → Permanent + cleanup, 404 →
   Permanent, 503 → Transient, two concurrent callers → one download.
4. **Provider methods.** `kvm.go`: `New`, `newWithDialer`, `Create/Get/Delete/ListByTag`
   over `libvirtAPI`; `kvm_test.go` with a hand-written `fakeAPI` (domains and
   volumes in maps, per-call error injection) covering idempotent re-Create
   after each partial step, state/IP mapping, delete-when-gone, ListByTag
   across two hosts and with one host down.
5. **Wiring, manifests, samples, docs, CHANGELOG.**
6. **Integration test** (`//go:build integration`, `kvm_integration_test.go`):
   skipped unless `KVM_TEST_HOST`, `KVM_TEST_USER`, `KVM_TEST_SSH_KEY`,
   `KVM_TEST_BRIDGE`, `KVM_TEST_IMAGE_URL` are set; fetches the image (or finds
   it cached), creates, waits for Running+IP, deletes, asserts per-VM volumes
   gone and the base image still cached. Deviation from the testcontainers convention,
   called out in `docs/testing.md`: libvirt+KVM can't run inside a container
   without `/dev/kvm`, so this targets a real hypervisor.
7. **Manual validation** on one real hypervisor, results in
   `docs/plans-executions/` (like `docs/gcp-vm-validation.md`), then MR via `tea`.

## Verification

- `make test` green (`-race`, envtest suite untouched but must still pass).
- `go test -short ./internal/provider/...` covers builders, fake-API provider, errors, config.
- Integration: `KVM_TEST_HOST=nas1.home.lan … go test -tags=integration -run TestKVM ./internal/provider/kvm/`.
- End to end: deploy with the `kvm-ssh` Secret and `kvm-home` config against a
  host with no cached image, apply `config/samples/proxy_kvm.yaml`, watch the
  operator log the image fetch and `virsh vol-list` show `base-debian-13.qcow2`
  plus its `.sha256` marker, watch `kubectl get px -w` reach `Ready` with a LAN
  IP, apply a second Proxy and confirm no second download, confirm the VM shows up in `GET /v1/proxies` and works with
  `hack/` demo egress-IP check, delete the Proxy and confirm `virsh list --all`
  and `virsh vol-list default` are clean.