kacerr/egress-proxies-operator
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

Add plan: KVM/libvirt provider

Browse Source 
Design for a kvm provider type: SSH-forwarded libvirt socket, multi-host
config with placement.zone selecting the host, bridged networking with
ARP/guest-agent IP discovery, operator-fetched distro cloud images cached
in the host's storage pool, and a NoCloud seed ISO carrying the CRD's
cloud-init user-data.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
This commit is contained in:
kacerr
2026-09-10 19:13:51 +00:00
co-authored by Claude Fable 5.1
parent 81b44d426b
commit bdd1c6ce6c
1 changed files with 313 additions and 0 deletions
Download Patch File Download Diff File Expand all files Collapse all files
docs/plans/2026-09-04-1404-kvm-provider.md
+313
View File
@@ -0,0 +1,313 @@
# Plan: KVM/libvirt provider
**Created:** 2026-09-04 14:04
## Context
The operator provisions egress proxies through named provider instances (today
`gcp` and `kubernetes`). The user has several KVM hypervisors on the home
network and wants proxy VMs created there, using stock Debian/Ubuntu cloud
images so the VM stays a normal machine they can customise through their own
cloud-init user-data. Longer term the operator may become a generic "run a VM
or container with a set of scripts" tool; the KVM provider should not
foreclose that.
The provider contract (`internal/provider/provider.go`: `Create`/`Get`/`Delete`/
`ListByTag`, keyed by `NameFromUID`, cloud-init passed verbatim, GC-by-tag) fits
libvirt with **no CRD or reconciler changes**. All work is a new provider
package plus config/wiring/docs, following the GCP provider's shape.
Decisions taken with the user (2026-09-04):
- **Transport:** SSH to each hypervisor, forwarding the libvirt unix socket
(`ssh.Client.Dial("unix", "/var/run/libvirt/libvirt-sock")` → `libvirt.New(conn)`).
Pure Go, no cgo, no libvirt libs in the image. Verified: `x/crypto/ssh`
`Client.Dial` has a `"unix"` case (`dialStreamLocal`), go-libvirt `New(net.Conn)`.
- **Topology:** one provider instance can span several hosts from day one
(config has a `hosts` list; providerID encodes the host), so adding hosts
later is a config edit with no providerID migration. With one host configured
`placement.zone` may be omitted.
- **Networking:** VM NIC on a host bridge to the LAN, IP from the router's
DHCP. IP discovery via `DomainInterfaceAddresses` (ARP source first, then
guest agent). Provider-injected `vendor-data` installs `qemu-guest-agent`
so agent discovery works on stock images without touching user-data.
- **Base image:** a stock Debian/Ubuntu cloud image (qcow2, cloud-init
preinstalled) that **the operator fetches itself** on first use per host:
`placement.image` names an entry in the provider's `images:` map (url +
optional sha256) or is an `https://` URL directly. The provider streams the
download through libvirt's `StorageVolUpload` into the host's pool, so
nothing is pre-placed and the operator needs no filesystem access on the
host. Per-VM disk is a qcow2 overlay with the cached base as backing store.
- **Seed:** NoCloud ISO (`user-data`, `meta-data`, `vendor-data`) built in
pure Go with `github.com/kdomanski/iso9660` (`WriteTo(w, "cidata")`) and
streamed to the pool with `StorageVolUpload(vol, io.Reader, ...)`. Nothing
to install on the hypervisors.
## Design
### Config (`internal/provider/config.go`)
```yaml
- name: kvm-home
type: kvm
kvm:
sshKeyFile: /etc/proxy-operator/kvm/id_ed25519 # required
knownHostsFile: /etc/proxy-operator/kvm/known_hosts # required unless insecureIgnoreHostKey
insecureIgnoreHostKey: false
diskSizeGb: 10 # overlay capacity; cloud-init growpart expands root fs
vendorData: | # optional override; default installs qemu-guest-agent
#cloud-config
...
images: # optional; placement.image names a key (or is an https URL)
debian-13:
url: https://cloud.debian.org/images/cloud/trixie/latest/debian-13-generic-amd64.qcow2
sha256: "…" # optional; verified after download, re-fetch when it changes
format: qcow2 # default qcow2
ubuntu-24.04:
url: https://cloud-images.ubuntu.com/noble/current/noble-server-cloudimg-amd64.img
machineTypes: # required, ≥1; placement.machineType names a key
small: {vcpus: 1, memoryMiB: 1024}
medium: {vcpus: 2, memoryMiB: 2048}
hosts: # required, ≥1, unique names
- name: nas1 # placement.zone value
address: nas1.home.lan:22 # required
user: virt # required; must be in the libvirt group
socket: /var/run/libvirt/libvirt-sock # default
pool: default # default "default"
bridge: br0 # required
```
Add `KVMConfig`, `KVMHost`, `KVMMachineType`, `KVMImage` structs; a `"kvm"`
case in `validate()` (cross-block checks like the others, required fields
above, unique host names, image URLs must be `http(s)`, sha256 if set must be
64 hex chars). Tests in `config_test.go` extend the existing table.
### Placement mapping
| Placement field | KVM meaning |
|---|---|
| `zone` | host name from `hosts`; empty → the single configured host, else `ErrPermanent` |
| `machineType` | key in `machineTypes`; unknown → `ErrPermanent` |
| `image` | key in `images:`, or an `https://` URL; anything else → `ErrPermanent`. Fetched into the host's pool on first use (see *Image cache*) |
| `region` | ignored |
### providerID
`hosts/<host>/domains/<name>` with `formatProviderID`/`parseProviderID`
(same self-contained style as GCP's `zones/<zone>/instances/<name>`).
### GC contract without labels
Domain XML gets a `<metadata>` child in a private namespace, e.g.
`xmlns="https://gitea.home.hrajfrisbee.cz/kacerr/egress-proxies-operator/kvm"`,
carrying `<managed>true</managed><uid>…</uid><createdAt>RFC3339</createdAt>`.
`Get` reads it via `DomainGetMetadata(dom, DomainMetadataElement, uri)`;
`ListByTag` does `ConnectListAllDomains` on every host and keeps domains where
that call succeeds and `managed == "true"`. `CreatedAt` comes from the metadata
(libvirt has no creation time). A host that is unreachable makes `ListByTag`
return an error (GC skips the sweep that round rather than acting on a partial view).
### Connection management (`internal/provider/kvm/conn.go`)
Per-host lazily dialed `*ssh.Client` + `*libvirt.Libvirt`, cached under a
mutex, re-dialed when `Disconnected()` is closed or a call fails with a
transport error. SSH auth: private key from `sshKeyFile`; host key check via
`knownhosts.New(knownHostsFile)` unless `insecureIgnoreHostKey`. Dial timeout
from `ctx`. The provider calls libvirt through a narrow interface:
```go
type libvirtAPI interface {
DomainLookupByName(name string) (libvirt.Domain, error)
DomainDefineXML(xml string) (libvirt.Domain, error)
DomainCreate(d libvirt.Domain) error
DomainDestroy(d libvirt.Domain) error
DomainUndefineFlags(d libvirt.Domain, f libvirt.DomainUndefineFlagsValues) error
DomainGetState(d libvirt.Domain, flags uint32) (int32, int32, error)
DomainGetMetadata(d libvirt.Domain, typ int32, uri libvirt.OptString, f libvirt.DomainModificationImpact) (string, error)
DomainInterfaceAddresses(d libvirt.Domain, src uint32, flags uint32) ([]libvirt.DomainInterface, error)
ConnectListAllDomains(need int32, f libvirt.ConnectListAllDomainsFlags) ([]libvirt.Domain, uint32, error)
StoragePoolLookupByName(name string) (libvirt.StoragePool, error)
StorageVolLookupByName(p libvirt.StoragePool, name string) (libvirt.StorageVol, error)
StorageVolGetPath(v libvirt.StorageVol) (string, error)
StorageVolCreateXML(p libvirt.StoragePool, xml string, f libvirt.StorageVolCreateFlags) (libvirt.StorageVol, error)
StorageVolUpload(v libvirt.StorageVol, r io.Reader, off, n uint64, f libvirt.StorageVolUploadFlags) error
StorageVolDownload(v libvirt.StorageVol, w io.Writer, off, n uint64, f libvirt.StorageVolDownloadFlags) error
StorageVolDelete(v libvirt.StorageVol, f libvirt.StorageVolDeleteFlags) error
StoragePoolRefresh(p libvirt.StoragePool, flags uint32) error
}
```
(All signatures verified against go-libvirt `main` on 2026-09-04.) A
`hostDialer func(ctx, KVMHost) (libvirtAPI, io.Closer, error)` is the test seam;
`newWithDialer(cfg, dialer)` mirrors GCP's `newWithAPI`.
### Image cache (`internal/provider/kvm/image.go`)
`ensureImage(ctx, host, api, img) (basePath string, err error)` makes the base
image present in the host's pool, once per host per image:
- Volume names: `base-<key>.<format>` for an `images:` key, or
`base-<sha256(url)[:16]>.<format>` for a URL given directly in
`placement.image`. Marker volume `<base>.sha256` (raw, 64 bytes) holds the
digest of the verified upload.
- **Ready check:** base volume exists *and* marker exists *and* (no `sha256`
configured, or marker content == configured `sha256`). Read the marker with
`StorageVolDownload`. Ready → return `StorageVolGetPath(base)`.
- **Not ready:** delete both volumes if present (partial upload from a crash,
or a changed checksum), then `GET url` with the ctx, require a
`Content-Length` (`ErrPermanent` otherwise), `StorageVolCreateXML` the base
with `<capacity unit="bytes">Content-Length</capacity>` and the configured
format, `StorageVolUpload(base, io.TeeReader(body, sha256), 0, 0, 0)` (length
0 = until EOF, so nothing is buffered in the operator), then compare the
digest with the configured one (mismatch → delete base, `ErrPermanent`),
`StoragePoolRefresh` so libvirt re-reads the real qcow2 capacity, create the
marker volume and upload the digest last.
- HTTP 4xx → `ErrPermanent`; 5xx, network, or upload transport errors →
`ErrTransient` (partial volumes are cleaned on the next attempt).
- A per-`(host, image)` `sync.Mutex` map serialises concurrent Creates for the
same image on the same host; the second caller finds it ready.
- **Judgment call:** the fetch runs synchronously inside `Create`, so the first
Proxy for a given image on a given host holds a reconcile worker for the
download (LAN speed, a few hundred MB, once). This keeps `Create` idempotent
and needs no background state. Async fetch with a `Provisioning`-style
progress status is a possible later improvement, noted in the docs.
### Create (idempotent, fire-and-forget after the image is cached)
1. Resolve host, machine type, pool, and image source (`images:` key or URL;
else `ErrPermanent`). `ensureImage` → base path.
2. Overlay volume `<name>.qcow2`: `StorageVolCreateXML` with
`<capacity unit="G">diskSizeGb</capacity>`, `<target><format type="qcow2"/></target>`,
`<backingStore><path>base</path><format type="qcow2"/></backingStore>`.
`ErrStorageVolExist` → reuse.
3. Seed volume `<name>-seed.iso`: build ISO in memory (`user-data` = req.CloudInit,
`meta-data` = `instance-id: <name>\nlocal-hostname: <name>\n`, `vendor-data` =
cfg or default), `StorageVolCreateXML` with exact capacity + `<format type="raw"/>`,
then `StorageVolUpload`. Exists → skip upload (content is deterministic per spec hash;
spec changes recreate the whole instance anyway).
4. `DomainLookupByName(name)`; if `ErrNoDomain` → `DomainDefineXML(buildDomainXML(...))`.
5. `DomainGetState`; if not running → `DomainCreate`. Return providerID.
`buildDomainXML` is a pure function over `encoding/xml` structs: `type="kvm"`,
q35 machine, `host-passthrough` CPU, vcpus/memory from the profile, virtio disk
(overlay) + sata cdrom (seed), `interface type="bridge"` with virtio model,
`org.qemu.guest_agent.0` virtio channel, serial console, virtio rng, no
graphics, and the `<metadata>` block. Unit-tested field by field like
`buildInsertRequest`.
### Get
`parseProviderID` → host → `DomainLookupByName` (`ErrNoDomain` → `ErrNotFound`)
→ `DomainGetState` + `DomainGetMetadata` + `DomainInterfaceAddresses`
(ARP, then agent; first IPv4 wins; agent errors while the agent isn't up yet
are swallowed). Mapping:
| libvirt state | IP | Instance.State |
|---|---|---|
| running | yes | Running |
| running | no | Provisioning |
| shutoff / crashed / paused / pmsuspended | – | Stopped |
| no domain | – | `ErrNotFound` |
### Delete (idempotent)
Destroy if running (`ErrOperationInvalid` on a stopped domain is fine) →
`DomainUndefineFlags(managed-save | snapshots-metadata | nvram)` → delete
`<name>.qcow2` and `<name>-seed.iso` (`ErrNoStorageVol` fine). Volumes are
deleted by name even when the domain is already gone, so a crash between
volume creation and define still gets cleaned up on the next reconcile.
Known gap (documented, not solved now): volumes orphaned with no domain *and*
no CR are invisible to `ListByTag`.
### Errors (`internal/provider/kvm/errors.go`)
`classify(err)` on `libvirt.Error.Code`: `ErrNoDomain`, `ErrNoStorageVol` →
`ErrNotFound`; `ErrNoStoragePool`, `ErrXMLError`, `ErrConfigUnsupported`,
`ErrNoSupport`, `ErrAuthFailed`, `ErrInvalidArg` → `ErrPermanent`; SSH dial /
host-key / transport errors and everything else → `ErrTransient`. Image
download: HTTP 4xx, missing Content-Length, checksum mismatch → `ErrPermanent`;
5xx and network errors → `ErrTransient`. Quota is not detectable from libvirt;
left as transient (revisit with a capacity check later).
### Wiring & manifests
- `cmd/main.go`: add `"kvm": kvm.New` to the builtins map.
- `config/manager/manager.yaml`: mount Secret `kvm-ssh` (optional) at
`/etc/proxy-operator/kvm` (keys `id_ed25519`, `known_hosts`); document the
`kubectl create secret generic kvm-ssh --from-file=...` command.
- `config/samples/providers-config.yaml`: add the `kvm-home` block with the
`images:` map (Debian 13 and Ubuntu 24.04 official cloud-image URLs);
`config/samples/proxy_kvm.yaml`: Proxy with `placement: {zone: nas1,
machineType: small, image: debian-13}` and the same squid cloud-init as the
GCP sample plus `ssh_authorized_keys`.
- `hack/providers-dev.yaml` stays kubernetes-only (needs no secrets).
- Deps: `github.com/digitalocean/go-libvirt` (latest pseudo-version),
`github.com/kdomanski/iso9660 v0.4.0`, promote `golang.org/x/crypto` to direct.
### Docs
- New `docs/kvm-provider.md`: hypervisor prerequisites (user in `libvirt`
group, a bridge to the LAN, a storage pool with enough space), the config
reference including `images:`, how the image cache works (`base-*` and
`*.sha256` volumes in the pool, when a re-download happens, how to evict
with `virsh vol-delete`), the placement table, how to log into a VM
(`ssh_authorized_keys` in user-data, `virsh console` on the host), and the
known gaps (no quota detection, orphan volumes without a domain,
synchronous first-time image fetch).
- Update the provider tables in `README.md`, `docs/architecture.md` §5,
`docs/testing.md` (integration test section).
### Towards the generic runner (not built now, just kept open)
Provider knows nothing about squid. Future hooks: `vendor-data` for injecting an
agent, `QEMUDomainAgentCommand` for reading exit status/output, and a future
`Placement`-independent "job" mode in the CRD. None of this changes the KVM
provider's shape.
## Steps
0. **Save plan + branch.** Copy this file to `docs/plans/<ts>-kvm-provider.md`,
commit alone, then `git checkout -b feat/kvm-provider` off `feat/proxy-operator`
(the working main for this project).
1. **Config.** `KVMConfig` & friends, `validate()` case, table tests.
2. **Pure builders.** `internal/provider/kvm/{domainxml.go,volume.go,seed.go,id.go,metadata.go}`
+ tests: domain XML fields, volume XML, seed contents (unpack the ISO with
the same library's reader and assert the three files + label `cidata`),
providerID round-trip, metadata marshal/parse.
3. **Transport + errors.** `conn.go` (SSH dial, host-key policy, per-host cache,
reconnect), `errors.go` + classification table test. SSH dialer tested
against an in-process `x/crypto/ssh` server that serves a unix socket
forward (cheap, keeps the dial code covered).
3b. **Image cache.** `image.go` + `image_test.go`: `httptest.Server` serving a
small fake image over the `fakeAPI` (which records uploaded bytes);
cases: fresh fetch, already ready, marker missing (partial upload → redo),
checksum changed → redo, checksum mismatch → Permanent + cleanup, 404 →
Permanent, 503 → Transient, two concurrent callers → one download.
4. **Provider methods.** `kvm.go`: `New`, `newWithDialer`, `Create/Get/Delete/ListByTag`
over `libvirtAPI`; `kvm_test.go` with a hand-written `fakeAPI` (domains and
volumes in maps, per-call error injection) covering idempotent re-Create
after each partial step, state/IP mapping, delete-when-gone, ListByTag
across two hosts and with one host down.
5. **Wiring, manifests, samples, docs, CHANGELOG.**
6. **Integration test** (`//go:build integration`, `kvm_integration_test.go`):
skipped unless `KVM_TEST_HOST`, `KVM_TEST_USER`, `KVM_TEST_SSH_KEY`,
`KVM_TEST_BRIDGE`, `KVM_TEST_IMAGE_URL` are set; fetches the image (or finds
it cached), creates, waits for Running+IP, deletes, asserts per-VM volumes
gone and the base image still cached. Deviation from the testcontainers convention,
called out in `docs/testing.md`: libvirt+KVM can't run inside a container
without `/dev/kvm`, so this targets a real hypervisor.
7. **Manual validation** on one real hypervisor, results in
`docs/plans-executions/` (like `docs/gcp-vm-validation.md`), then MR via `tea`.
## Verification
- `make test` green (`-race`, envtest suite untouched but must still pass).
- `go test -short ./internal/provider/...` covers builders, fake-API provider, errors, config.
- Integration: `KVM_TEST_HOST=nas1.home.lan … go test -tags=integration -run TestKVM ./internal/provider/kvm/`.
- End to end: deploy with the `kvm-ssh` Secret and `kvm-home` config against a
host with no cached image, apply `config/samples/proxy_kvm.yaml`, watch the
operator log the image fetch and `virsh vol-list` show `base-debian-13.qcow2`
plus its `.sha256` marker, watch `kubectl get px -w` reach `Ready` with a LAN
IP, apply a second Proxy and confirm no second download, confirm the VM shows up in `GET /v1/proxies` and works with
`hack/` demo egress-IP check, delete the Proxy and confirm `virsh list --all`
and `virsh vol-list default` are clean.