From bdd1c6ce6c62a782d30710c9ed57bb13632bf072 Mon Sep 17 00:00:00 2001 From: kacerr Date: Fri, 4 Sep 2026 14:05:11 +0000 Subject: [PATCH 01/28] Add plan: KVM/libvirt provider Design for a kvm provider type: SSH-forwarded libvirt socket, multi-host config with placement.zone selecting the host, bridged networking with ARP/guest-agent IP discovery, operator-fetched distro cloud images cached in the host's storage pool, and a NoCloud seed ISO carrying the CRD's cloud-init user-data. Co-Authored-By: Claude Fable 5.1 --- docs/plans/2026-09-04-1404-kvm-provider.md | 313 +++++++++++++++++++++ 1 file changed, 313 insertions(+) create mode 100644 docs/plans/2026-09-04-1404-kvm-provider.md diff --git a/docs/plans/2026-09-04-1404-kvm-provider.md b/docs/plans/2026-09-04-1404-kvm-provider.md new file mode 100644 index 0000000..029e612 --- /dev/null +++ b/docs/plans/2026-09-04-1404-kvm-provider.md @@ -0,0 +1,313 @@ +# Plan: KVM/libvirt provider + +**Created:** 2026-09-04 14:04 + +## Context + +The operator provisions egress proxies through named provider instances (today +`gcp` and `kubernetes`). The user has several KVM hypervisors on the home +network and wants proxy VMs created there, using stock Debian/Ubuntu cloud +images so the VM stays a normal machine they can customise through their own +cloud-init user-data. Longer term the operator may become a generic "run a VM +or container with a set of scripts" tool; the KVM provider should not +foreclose that. + +The provider contract (`internal/provider/provider.go`: `Create`/`Get`/`Delete`/ +`ListByTag`, keyed by `NameFromUID`, cloud-init passed verbatim, GC-by-tag) fits +libvirt with **no CRD or reconciler changes**. All work is a new provider +package plus config/wiring/docs, following the GCP provider's shape. + +Decisions taken with the user (2026-09-04): + +- **Transport:** SSH to each hypervisor, forwarding the libvirt unix socket + (`ssh.Client.Dial("unix", "/var/run/libvirt/libvirt-sock")` → `libvirt.New(conn)`). + Pure Go, no cgo, no libvirt libs in the image. Verified: `x/crypto/ssh` + `Client.Dial` has a `"unix"` case (`dialStreamLocal`), go-libvirt `New(net.Conn)`. +- **Topology:** one provider instance can span several hosts from day one + (config has a `hosts` list; providerID encodes the host), so adding hosts + later is a config edit with no providerID migration. With one host configured + `placement.zone` may be omitted. +- **Networking:** VM NIC on a host bridge to the LAN, IP from the router's + DHCP. IP discovery via `DomainInterfaceAddresses` (ARP source first, then + guest agent). Provider-injected `vendor-data` installs `qemu-guest-agent` + so agent discovery works on stock images without touching user-data. +- **Base image:** a stock Debian/Ubuntu cloud image (qcow2, cloud-init + preinstalled) that **the operator fetches itself** on first use per host: + `placement.image` names an entry in the provider's `images:` map (url + + optional sha256) or is an `https://` URL directly. The provider streams the + download through libvirt's `StorageVolUpload` into the host's pool, so + nothing is pre-placed and the operator needs no filesystem access on the + host. Per-VM disk is a qcow2 overlay with the cached base as backing store. +- **Seed:** NoCloud ISO (`user-data`, `meta-data`, `vendor-data`) built in + pure Go with `github.com/kdomanski/iso9660` (`WriteTo(w, "cidata")`) and + streamed to the pool with `StorageVolUpload(vol, io.Reader, ...)`. Nothing + to install on the hypervisors. + +## Design + +### Config (`internal/provider/config.go`) + +```yaml +- name: kvm-home + type: kvm + kvm: + sshKeyFile: /etc/proxy-operator/kvm/id_ed25519 # required + knownHostsFile: /etc/proxy-operator/kvm/known_hosts # required unless insecureIgnoreHostKey + insecureIgnoreHostKey: false + diskSizeGb: 10 # overlay capacity; cloud-init growpart expands root fs + vendorData: | # optional override; default installs qemu-guest-agent + #cloud-config + ... + images: # optional; placement.image names a key (or is an https URL) + debian-13: + url: https://cloud.debian.org/images/cloud/trixie/latest/debian-13-generic-amd64.qcow2 + sha256: "…" # optional; verified after download, re-fetch when it changes + format: qcow2 # default qcow2 + ubuntu-24.04: + url: https://cloud-images.ubuntu.com/noble/current/noble-server-cloudimg-amd64.img + machineTypes: # required, ≥1; placement.machineType names a key + small: {vcpus: 1, memoryMiB: 1024} + medium: {vcpus: 2, memoryMiB: 2048} + hosts: # required, ≥1, unique names + - name: nas1 # placement.zone value + address: nas1.home.lan:22 # required + user: virt # required; must be in the libvirt group + socket: /var/run/libvirt/libvirt-sock # default + pool: default # default "default" + bridge: br0 # required +``` + +Add `KVMConfig`, `KVMHost`, `KVMMachineType`, `KVMImage` structs; a `"kvm"` +case in `validate()` (cross-block checks like the others, required fields +above, unique host names, image URLs must be `http(s)`, sha256 if set must be +64 hex chars). Tests in `config_test.go` extend the existing table. + +### Placement mapping + +| Placement field | KVM meaning | +|---|---| +| `zone` | host name from `hosts`; empty → the single configured host, else `ErrPermanent` | +| `machineType` | key in `machineTypes`; unknown → `ErrPermanent` | +| `image` | key in `images:`, or an `https://` URL; anything else → `ErrPermanent`. Fetched into the host's pool on first use (see *Image cache*) | +| `region` | ignored | + +### providerID + +`hosts//domains/` with `formatProviderID`/`parseProviderID` +(same self-contained style as GCP's `zones//instances/`). + +### GC contract without labels + +Domain XML gets a `` child in a private namespace, e.g. +`xmlns="https://gitea.home.hrajfrisbee.cz/kacerr/egress-proxies-operator/kvm"`, +carrying `true…RFC3339`. +`Get` reads it via `DomainGetMetadata(dom, DomainMetadataElement, uri)`; +`ListByTag` does `ConnectListAllDomains` on every host and keeps domains where +that call succeeds and `managed == "true"`. `CreatedAt` comes from the metadata +(libvirt has no creation time). A host that is unreachable makes `ListByTag` +return an error (GC skips the sweep that round rather than acting on a partial view). + +### Connection management (`internal/provider/kvm/conn.go`) + +Per-host lazily dialed `*ssh.Client` + `*libvirt.Libvirt`, cached under a +mutex, re-dialed when `Disconnected()` is closed or a call fails with a +transport error. SSH auth: private key from `sshKeyFile`; host key check via +`knownhosts.New(knownHostsFile)` unless `insecureIgnoreHostKey`. Dial timeout +from `ctx`. The provider calls libvirt through a narrow interface: + +```go +type libvirtAPI interface { + DomainLookupByName(name string) (libvirt.Domain, error) + DomainDefineXML(xml string) (libvirt.Domain, error) + DomainCreate(d libvirt.Domain) error + DomainDestroy(d libvirt.Domain) error + DomainUndefineFlags(d libvirt.Domain, f libvirt.DomainUndefineFlagsValues) error + DomainGetState(d libvirt.Domain, flags uint32) (int32, int32, error) + DomainGetMetadata(d libvirt.Domain, typ int32, uri libvirt.OptString, f libvirt.DomainModificationImpact) (string, error) + DomainInterfaceAddresses(d libvirt.Domain, src uint32, flags uint32) ([]libvirt.DomainInterface, error) + ConnectListAllDomains(need int32, f libvirt.ConnectListAllDomainsFlags) ([]libvirt.Domain, uint32, error) + StoragePoolLookupByName(name string) (libvirt.StoragePool, error) + StorageVolLookupByName(p libvirt.StoragePool, name string) (libvirt.StorageVol, error) + StorageVolGetPath(v libvirt.StorageVol) (string, error) + StorageVolCreateXML(p libvirt.StoragePool, xml string, f libvirt.StorageVolCreateFlags) (libvirt.StorageVol, error) + StorageVolUpload(v libvirt.StorageVol, r io.Reader, off, n uint64, f libvirt.StorageVolUploadFlags) error + StorageVolDownload(v libvirt.StorageVol, w io.Writer, off, n uint64, f libvirt.StorageVolDownloadFlags) error + StorageVolDelete(v libvirt.StorageVol, f libvirt.StorageVolDeleteFlags) error + StoragePoolRefresh(p libvirt.StoragePool, flags uint32) error +} +``` + +(All signatures verified against go-libvirt `main` on 2026-09-04.) A +`hostDialer func(ctx, KVMHost) (libvirtAPI, io.Closer, error)` is the test seam; +`newWithDialer(cfg, dialer)` mirrors GCP's `newWithAPI`. + +### Image cache (`internal/provider/kvm/image.go`) + +`ensureImage(ctx, host, api, img) (basePath string, err error)` makes the base +image present in the host's pool, once per host per image: + +- Volume names: `base-.` for an `images:` key, or + `base-.` for a URL given directly in + `placement.image`. Marker volume `.sha256` (raw, 64 bytes) holds the + digest of the verified upload. +- **Ready check:** base volume exists *and* marker exists *and* (no `sha256` + configured, or marker content == configured `sha256`). Read the marker with + `StorageVolDownload`. Ready → return `StorageVolGetPath(base)`. +- **Not ready:** delete both volumes if present (partial upload from a crash, + or a changed checksum), then `GET url` with the ctx, require a + `Content-Length` (`ErrPermanent` otherwise), `StorageVolCreateXML` the base + with `Content-Length` and the configured + format, `StorageVolUpload(base, io.TeeReader(body, sha256), 0, 0, 0)` (length + 0 = until EOF, so nothing is buffered in the operator), then compare the + digest with the configured one (mismatch → delete base, `ErrPermanent`), + `StoragePoolRefresh` so libvirt re-reads the real qcow2 capacity, create the + marker volume and upload the digest last. +- HTTP 4xx → `ErrPermanent`; 5xx, network, or upload transport errors → + `ErrTransient` (partial volumes are cleaned on the next attempt). +- A per-`(host, image)` `sync.Mutex` map serialises concurrent Creates for the + same image on the same host; the second caller finds it ready. +- **Judgment call:** the fetch runs synchronously inside `Create`, so the first + Proxy for a given image on a given host holds a reconcile worker for the + download (LAN speed, a few hundred MB, once). This keeps `Create` idempotent + and needs no background state. Async fetch with a `Provisioning`-style + progress status is a possible later improvement, noted in the docs. + +### Create (idempotent, fire-and-forget after the image is cached) + +1. Resolve host, machine type, pool, and image source (`images:` key or URL; + else `ErrPermanent`). `ensureImage` → base path. +2. Overlay volume `.qcow2`: `StorageVolCreateXML` with + `diskSizeGb`, ``, + `base`. + `ErrStorageVolExist` → reuse. +3. Seed volume `-seed.iso`: build ISO in memory (`user-data` = req.CloudInit, + `meta-data` = `instance-id: \nlocal-hostname: \n`, `vendor-data` = + cfg or default), `StorageVolCreateXML` with exact capacity + ``, + then `StorageVolUpload`. Exists → skip upload (content is deterministic per spec hash; + spec changes recreate the whole instance anyway). +4. `DomainLookupByName(name)`; if `ErrNoDomain` → `DomainDefineXML(buildDomainXML(...))`. +5. `DomainGetState`; if not running → `DomainCreate`. Return providerID. + +`buildDomainXML` is a pure function over `encoding/xml` structs: `type="kvm"`, +q35 machine, `host-passthrough` CPU, vcpus/memory from the profile, virtio disk +(overlay) + sata cdrom (seed), `interface type="bridge"` with virtio model, +`org.qemu.guest_agent.0` virtio channel, serial console, virtio rng, no +graphics, and the `` block. Unit-tested field by field like +`buildInsertRequest`. + +### Get + +`parseProviderID` → host → `DomainLookupByName` (`ErrNoDomain` → `ErrNotFound`) +→ `DomainGetState` + `DomainGetMetadata` + `DomainInterfaceAddresses` +(ARP, then agent; first IPv4 wins; agent errors while the agent isn't up yet +are swallowed). Mapping: + +| libvirt state | IP | Instance.State | +|---|---|---| +| running | yes | Running | +| running | no | Provisioning | +| shutoff / crashed / paused / pmsuspended | – | Stopped | +| no domain | – | `ErrNotFound` | + +### Delete (idempotent) + +Destroy if running (`ErrOperationInvalid` on a stopped domain is fine) → +`DomainUndefineFlags(managed-save | snapshots-metadata | nvram)` → delete +`.qcow2` and `-seed.iso` (`ErrNoStorageVol` fine). Volumes are +deleted by name even when the domain is already gone, so a crash between +volume creation and define still gets cleaned up on the next reconcile. +Known gap (documented, not solved now): volumes orphaned with no domain *and* +no CR are invisible to `ListByTag`. + +### Errors (`internal/provider/kvm/errors.go`) + +`classify(err)` on `libvirt.Error.Code`: `ErrNoDomain`, `ErrNoStorageVol` → +`ErrNotFound`; `ErrNoStoragePool`, `ErrXMLError`, `ErrConfigUnsupported`, +`ErrNoSupport`, `ErrAuthFailed`, `ErrInvalidArg` → `ErrPermanent`; SSH dial / +host-key / transport errors and everything else → `ErrTransient`. Image +download: HTTP 4xx, missing Content-Length, checksum mismatch → `ErrPermanent`; +5xx and network errors → `ErrTransient`. Quota is not detectable from libvirt; +left as transient (revisit with a capacity check later). + +### Wiring & manifests + +- `cmd/main.go`: add `"kvm": kvm.New` to the builtins map. +- `config/manager/manager.yaml`: mount Secret `kvm-ssh` (optional) at + `/etc/proxy-operator/kvm` (keys `id_ed25519`, `known_hosts`); document the + `kubectl create secret generic kvm-ssh --from-file=...` command. +- `config/samples/providers-config.yaml`: add the `kvm-home` block with the + `images:` map (Debian 13 and Ubuntu 24.04 official cloud-image URLs); + `config/samples/proxy_kvm.yaml`: Proxy with `placement: {zone: nas1, + machineType: small, image: debian-13}` and the same squid cloud-init as the + GCP sample plus `ssh_authorized_keys`. +- `hack/providers-dev.yaml` stays kubernetes-only (needs no secrets). +- Deps: `github.com/digitalocean/go-libvirt` (latest pseudo-version), + `github.com/kdomanski/iso9660 v0.4.0`, promote `golang.org/x/crypto` to direct. + +### Docs + +- New `docs/kvm-provider.md`: hypervisor prerequisites (user in `libvirt` + group, a bridge to the LAN, a storage pool with enough space), the config + reference including `images:`, how the image cache works (`base-*` and + `*.sha256` volumes in the pool, when a re-download happens, how to evict + with `virsh vol-delete`), the placement table, how to log into a VM + (`ssh_authorized_keys` in user-data, `virsh console` on the host), and the + known gaps (no quota detection, orphan volumes without a domain, + synchronous first-time image fetch). +- Update the provider tables in `README.md`, `docs/architecture.md` §5, + `docs/testing.md` (integration test section). + +### Towards the generic runner (not built now, just kept open) + +Provider knows nothing about squid. Future hooks: `vendor-data` for injecting an +agent, `QEMUDomainAgentCommand` for reading exit status/output, and a future +`Placement`-independent "job" mode in the CRD. None of this changes the KVM +provider's shape. + +## Steps + +0. **Save plan + branch.** Copy this file to `docs/plans/-kvm-provider.md`, + commit alone, then `git checkout -b feat/kvm-provider` off `feat/proxy-operator` + (the working main for this project). +1. **Config.** `KVMConfig` & friends, `validate()` case, table tests. +2. **Pure builders.** `internal/provider/kvm/{domainxml.go,volume.go,seed.go,id.go,metadata.go}` + + tests: domain XML fields, volume XML, seed contents (unpack the ISO with + the same library's reader and assert the three files + label `cidata`), + providerID round-trip, metadata marshal/parse. +3. **Transport + errors.** `conn.go` (SSH dial, host-key policy, per-host cache, + reconnect), `errors.go` + classification table test. SSH dialer tested + against an in-process `x/crypto/ssh` server that serves a unix socket + forward (cheap, keeps the dial code covered). +3b. **Image cache.** `image.go` + `image_test.go`: `httptest.Server` serving a + small fake image over the `fakeAPI` (which records uploaded bytes); + cases: fresh fetch, already ready, marker missing (partial upload → redo), + checksum changed → redo, checksum mismatch → Permanent + cleanup, 404 → + Permanent, 503 → Transient, two concurrent callers → one download. +4. **Provider methods.** `kvm.go`: `New`, `newWithDialer`, `Create/Get/Delete/ListByTag` + over `libvirtAPI`; `kvm_test.go` with a hand-written `fakeAPI` (domains and + volumes in maps, per-call error injection) covering idempotent re-Create + after each partial step, state/IP mapping, delete-when-gone, ListByTag + across two hosts and with one host down. +5. **Wiring, manifests, samples, docs, CHANGELOG.** +6. **Integration test** (`//go:build integration`, `kvm_integration_test.go`): + skipped unless `KVM_TEST_HOST`, `KVM_TEST_USER`, `KVM_TEST_SSH_KEY`, + `KVM_TEST_BRIDGE`, `KVM_TEST_IMAGE_URL` are set; fetches the image (or finds + it cached), creates, waits for Running+IP, deletes, asserts per-VM volumes + gone and the base image still cached. Deviation from the testcontainers convention, + called out in `docs/testing.md`: libvirt+KVM can't run inside a container + without `/dev/kvm`, so this targets a real hypervisor. +7. **Manual validation** on one real hypervisor, results in + `docs/plans-executions/` (like `docs/gcp-vm-validation.md`), then MR via `tea`. + +## Verification + +- `make test` green (`-race`, envtest suite untouched but must still pass). +- `go test -short ./internal/provider/...` covers builders, fake-API provider, errors, config. +- Integration: `KVM_TEST_HOST=nas1.home.lan … go test -tags=integration -run TestKVM ./internal/provider/kvm/`. +- End to end: deploy with the `kvm-ssh` Secret and `kvm-home` config against a + host with no cached image, apply `config/samples/proxy_kvm.yaml`, watch the + operator log the image fetch and `virsh vol-list` show `base-debian-13.qcow2` + plus its `.sha256` marker, watch `kubectl get px -w` reach `Ready` with a LAN + IP, apply a second Proxy and confirm no second download, confirm the VM shows up in `GET /v1/proxies` and works with + `hack/` demo egress-IP check, delete the Proxy and confirm `virsh list --all` + and `virsh vol-list default` are clean. -- 2.54.0 From 84dce0462eb13c1c0bce828a556fa04b9fc7bbb9 Mon Sep 17 00:00:00 2001 From: kacerr Date: Fri, 4 Sep 2026 14:25:06 +0000 Subject: [PATCH 02/28] Add KVM provider config: KVMConfig structs and validation Adds the kvm block to the providers config (SSH access, images map, machine-type profiles, hosts list) with validation and table tests. validate() now rejects mismatched type-specific blocks generically via setBlocks() instead of pairwise checks. Step 1 of docs/plans/2026-09-04-1404-kvm-provider.md. Co-Authored-By: Claude Fable 5.1 --- .claude/settings.json | 8 +- .../2026-09-04-1404-kvm-provider.md | 54 +++++ internal/provider/config.go | 49 +++-- internal/provider/config_kvm.go | 138 ++++++++++++ internal/provider/config_kvm_test.go | 208 ++++++++++++++++++ 5 files changed, 443 insertions(+), 14 deletions(-) create mode 100644 docs/plans-executions/2026-09-04-1404-kvm-provider.md create mode 100644 internal/provider/config_kvm.go create mode 100644 internal/provider/config_kvm_test.go diff --git a/.claude/settings.json b/.claude/settings.json index ec2dbda..e95a782 100644 --- a/.claude/settings.json +++ b/.claude/settings.json @@ -93,7 +93,13 @@ "Bash(git -C /Users/jan.novak/srv/go/egress-proxies-operator tag 0.01)", "Bash(git -C /Users/jan.novak/srv/go/egress-proxies-operator push origin 0.01)", "Bash(chmod +x docs/demo/run-demo.sh)", - "Bash(bash -n docs/demo/run-demo.sh)" + "Bash(bash -n docs/demo/run-demo.sh)", + "Bash(git add *)", + "Bash(git commit -q -m 'Add plan: KVM/libvirt provider *)", + "Bash(go get *)", + "Bash(go build *)", + "Bash(go test *)", + "Bash(git commit *)" ], "additionalDirectories": [ "/Users/jan.novak/srv/go/egress-proxies-operator/.claude", diff --git a/docs/plans-executions/2026-09-04-1404-kvm-provider.md b/docs/plans-executions/2026-09-04-1404-kvm-provider.md new file mode 100644 index 0000000..656015c --- /dev/null +++ b/docs/plans-executions/2026-09-04-1404-kvm-provider.md @@ -0,0 +1,54 @@ +# Execution: KVM/libvirt provider + +Plan: [2026-09-04-1404-kvm-provider.md](../plans/2026-09-04-1404-kvm-provider.md) + +- [x] Step 0 — Save plan + branch `feat/kvm-provider` +- [x] Step 1 — Config (`KVMConfig` & friends, `validate()` case, tests) +- [ ] Step 2 — Pure builders (`domainxml.go`, `volume.go`, `seed.go`, `id.go`, `metadata.go`) +- [ ] Step 3 — Transport + errors (`conn.go`, `errors.go`) +- [ ] Step 3b — Image cache (`image.go`) +- [ ] Step 4 — Provider methods (`kvm.go`) over a fake API +- [ ] Step 5 — Wiring, manifests, samples, docs, CHANGELOG +- [ ] Step 6 — Integration test against a real hypervisor +- [ ] Step 7 — Manual validation + MR + +## Step 0 — Save plan + branch + +Plan committed alone on `feat/proxy-operator`, then branched: + +```bash +git checkout -b feat/kvm-provider +``` + +## Step 1 — Config + +The new structs (`KVMConfig`, `KVMHost`, `KVMMachineType`, `KVMImage`) and +`validateKVM` went into their own file, `internal/provider/config_kvm.go`, +rather than being appended to `config.go` as the plan said: the KVM block has +far more fields and rules than the other two providers, and keeping it +separate leaves `config.go` readable. Tests likewise live in +`config_kvm_test.go` instead of extending the table in `config_test.go`; the +invalid-config table covers every required field, duplicate host names, image +URL scheme, sha256 shape, and the disk format enum. + +While adding the `kvm` case to `validate()`, the pairwise "type is X but a Y +block is set" checks (which would have needed six branches with three types) +were replaced by a `setBlocks()` helper that lists the non-nil blocks and +rejects any that don't match `type`. The error strings are unchanged, so the +existing GCP/kubernetes tests still pass. + +Dependencies were fetched up front so the API signatures in the plan could be +checked with `go doc`; they stay `// indirect` in `go.mod` until Step 2 imports +them: + +```bash +go get github.com/digitalocean/go-libvirt@latest github.com/kdomanski/iso9660@v0.4.0 +# go-libvirt resolved to v0.0.0-20260814190004-1a83157e1858 (no tagged release) +# side effect: golang.org/x/crypto 0.54.0 → 0.56.0, x/mod, x/text, x/tools bumped +``` + +Worth noting: `go doc` showed that go-libvirt's `New(net.Conn)` is deprecated +in favour of `NewWithDialer(socket.Dialer)`, where `Dialer` is a one-method +`Dial() (net.Conn, error)` interface. Step 3 should wrap the SSH client's unix +forward in that interface instead of using `New`; the plan's transport design +is otherwise unaffected. diff --git a/internal/provider/config.go b/internal/provider/config.go index 6a4d4b7..f9b81d8 100644 --- a/internal/provider/config.go +++ b/internal/provider/config.go @@ -21,6 +21,23 @@ type ProviderConfig struct { Type string `json:"type"` Kubernetes *KubernetesConfig `json:"kubernetes,omitempty"` GCP *GCPConfig `json:"gcp,omitempty"` + KVM *KVMConfig `json:"kvm,omitempty"` +} + +// setBlocks names the type-specific blocks that are non-nil, so validate can +// reject a block that doesn't match Type without a pairwise check per type. +func (p ProviderConfig) setBlocks() []string { + var out []string + if p.Kubernetes != nil { + out = append(out, "kubernetes") + } + if p.GCP != nil { + out = append(out, "gcp") + } + if p.KVM != nil { + out = append(out, "kvm") + } + return out } // KubernetesConfig configures the kubernetes-pod provider, which creates @@ -81,22 +98,28 @@ func (c *Config) validate() error { } seen[p.Name] = true + prefix := fmt.Sprintf("providers[%d] %q", i, p.Name) switch p.Type { - case "kubernetes": - if p.GCP != nil { - return fmt.Errorf("providers[%d] %q: type is kubernetes but a gcp block is set", i, p.Name) - } - case "gcp": - if p.Kubernetes != nil { - return fmt.Errorf("providers[%d] %q: type is gcp but a kubernetes block is set", i, p.Name) - } - if p.GCP == nil || p.GCP.Project == "" { - return fmt.Errorf("providers[%d] %q: gcp.project is required", i, p.Name) - } + case "kubernetes", "gcp", "kvm": case "": - return fmt.Errorf("providers[%d] %q: type is required", i, p.Name) + return fmt.Errorf("%s: type is required", prefix) default: - return fmt.Errorf("providers[%d] %q: unknown provider type %q", i, p.Name, p.Type) + return fmt.Errorf("%s: unknown provider type %q", prefix, p.Type) + } + for _, block := range p.setBlocks() { + if block != p.Type { + return fmt.Errorf("%s: type is %s but a %s block is set", prefix, p.Type, block) + } + } + switch p.Type { + case "gcp": + if p.GCP == nil || p.GCP.Project == "" { + return fmt.Errorf("%s: gcp.project is required", prefix) + } + case "kvm": + if err := validateKVM(prefix, p.KVM); err != nil { + return err + } } } return nil diff --git a/internal/provider/config_kvm.go b/internal/provider/config_kvm.go new file mode 100644 index 0000000..606d643 --- /dev/null +++ b/internal/provider/config_kvm.go @@ -0,0 +1,138 @@ +package provider + +import ( + "fmt" + "net/url" + "regexp" +) + +// KVMConfig configures a kvm provider instance: one or more libvirt/KVM +// hypervisors reached over SSH (the libvirt unix socket is forwarded through +// the SSH connection, so hosts need no libvirtd TCP/TLS listener). +// placement.zone on a Proxy selects a host by name; with a single host it may +// be omitted. +type KVMConfig struct { + // SSHKeyFile is the path to the private key used for every host. Required. + SSHKeyFile string `json:"sshKeyFile"` + // KnownHostsFile is an OpenSSH known_hosts file used to verify host keys. + // Required unless InsecureIgnoreHostKey is set. + KnownHostsFile string `json:"knownHostsFile,omitempty"` + // InsecureIgnoreHostKey skips host-key verification entirely. + InsecureIgnoreHostKey bool `json:"insecureIgnoreHostKey,omitempty"` + // DiskSizeGB is the capacity of each VM's overlay disk. Default 10. The + // guest's cloud-init growpart expands the root filesystem into it. + DiskSizeGB int64 `json:"diskSizeGb,omitempty"` + // VendorData is cloud-init vendor-data placed on every seed next to the + // Proxy's own user-data. Default: installs and starts qemu-guest-agent so + // the provider can read the VM's IP even before it shows up in ARP. + VendorData string `json:"vendorData,omitempty"` + // Images are named base images; placement.image names a key here (or is + // an http(s) URL directly). The provider downloads each image into a + // host's storage pool on first use and caches it there. + Images map[string]KVMImage `json:"images,omitempty"` + // MachineTypes are named size profiles; placement.machineType names a + // key here. Required, at least one. + MachineTypes map[string]KVMMachineType `json:"machineTypes"` + // Hosts are the hypervisors. Required, at least one, unique names. + Hosts []KVMHost `json:"hosts"` +} + +// KVMImage is one downloadable base image (a distro cloud image with +// cloud-init preinstalled, e.g. Debian's generic qcow2). +type KVMImage struct { + // URL is where to fetch the image from. Required, http or https. + URL string `json:"url"` + // SHA256 is the expected hex digest of the file. Optional; when set it is + // verified after download and a later change triggers a re-download. + SHA256 string `json:"sha256,omitempty"` + // Format is the image's disk format, "qcow2" (default) or "raw". + Format string `json:"format,omitempty"` +} + +// KVMMachineType is a VM size profile. +type KVMMachineType struct { + VCPUs int32 `json:"vcpus"` + MemoryMiB int64 `json:"memoryMiB"` +} + +// KVMHost is one hypervisor. +type KVMHost struct { + // Name is the placement.zone value that selects this host. Required. + Name string `json:"name"` + // Address is the SSH endpoint, host:port. Required. + Address string `json:"address"` + // User is the SSH user; it must be allowed to use the libvirt socket + // (typically a member of the libvirt group). Required. + User string `json:"user"` + // Socket is the libvirt unix socket path on the host. + // Default /var/run/libvirt/libvirt-sock. + Socket string `json:"socket,omitempty"` + // Pool is the libvirt storage pool for base images, overlay disks and + // seed ISOs. Default "default". + Pool string `json:"pool,omitempty"` + // Bridge is the host bridge the VM's NIC attaches to; it must lead to a + // network the cluster can reach and that hands out DHCP leases. Required. + Bridge string `json:"bridge"` +} + +var sha256Hex = regexp.MustCompile(`^[0-9a-fA-F]{64}$`) + +func validateKVM(prefix string, c *KVMConfig) error { + if c == nil { + return fmt.Errorf("%s: kvm block is required", prefix) + } + if c.SSHKeyFile == "" { + return fmt.Errorf("%s: kvm.sshKeyFile is required", prefix) + } + if c.KnownHostsFile == "" && !c.InsecureIgnoreHostKey { + return fmt.Errorf("%s: kvm.knownHostsFile is required unless kvm.insecureIgnoreHostKey is true", prefix) + } + if c.DiskSizeGB < 0 { + return fmt.Errorf("%s: kvm.diskSizeGb must not be negative", prefix) + } + for name, img := range c.Images { + p := fmt.Sprintf("%s: kvm.images[%q]", prefix, name) + u, err := url.Parse(img.URL) + if err != nil || (u.Scheme != "http" && u.Scheme != "https") || u.Host == "" { + return fmt.Errorf("%s.url must be an http(s) URL (got %q)", p, img.URL) + } + if img.SHA256 != "" && !sha256Hex.MatchString(img.SHA256) { + return fmt.Errorf("%s.sha256 must be 64 hex characters", p) + } + if img.Format != "" && img.Format != "qcow2" && img.Format != "raw" { + return fmt.Errorf("%s.format must be qcow2 or raw (got %q)", p, img.Format) + } + } + if len(c.MachineTypes) == 0 { + return fmt.Errorf("%s: kvm.machineTypes must define at least one profile", prefix) + } + for name, mt := range c.MachineTypes { + if mt.VCPUs <= 0 || mt.MemoryMiB <= 0 { + return fmt.Errorf("%s: kvm.machineTypes[%q] needs vcpus > 0 and memoryMiB > 0", prefix, name) + } + } + if len(c.Hosts) == 0 { + return fmt.Errorf("%s: kvm.hosts must list at least one host", prefix) + } + seen := make(map[string]bool, len(c.Hosts)) + for i, h := range c.Hosts { + p := fmt.Sprintf("%s: kvm.hosts[%d]", prefix, i) + if h.Name == "" { + return fmt.Errorf("%s: name is required", p) + } + if seen[h.Name] { + return fmt.Errorf("%s: duplicate host name %q", p, h.Name) + } + seen[h.Name] = true + if h.Address == "" { + return fmt.Errorf("%s %q: address is required", p, h.Name) + } + if h.User == "" { + return fmt.Errorf("%s %q: user is required", p, h.Name) + } + if h.Bridge == "" { + return fmt.Errorf("%s %q: bridge is required", p, h.Name) + } + } + return nil +} diff --git a/internal/provider/config_kvm_test.go b/internal/provider/config_kvm_test.go new file mode 100644 index 0000000..6a00afb --- /dev/null +++ b/internal/provider/config_kvm_test.go @@ -0,0 +1,208 @@ +package provider + +import ( + "strings" + "testing" +) + +const validKVMYAML = ` +providers: + - name: kvm-home + type: kvm + kvm: + sshKeyFile: /etc/proxy-operator/kvm/id_ed25519 + knownHostsFile: /etc/proxy-operator/kvm/known_hosts + diskSizeGb: 20 + images: + debian-13: + url: https://cloud.debian.org/images/cloud/trixie/latest/debian-13-generic-amd64.qcow2 + sha256: 0123456789abcdef0123456789abcdef0123456789abcdef0123456789abcdef + ubuntu: + url: http://mirror.local/noble.img + format: raw + machineTypes: + small: {vcpus: 1, memoryMiB: 1024} + medium: {vcpus: 2, memoryMiB: 2048} + hosts: + - name: nas1 + address: nas1.home.lan:22 + user: virt + bridge: br0 + - name: lab2 + address: 10.0.0.2:2222 + user: root + socket: /run/libvirt/virtqemud-sock + pool: fast + bridge: br1 +` + +func TestLoadConfig_kvmValid(t *testing.T) { + t.Parallel() + cfg, err := LoadConfig([]byte(validKVMYAML)) + if err != nil { + t.Fatalf("LoadConfig() error = %v, want nil", err) + } + k := cfg.Providers[0].KVM + if k == nil { + t.Fatal("providers[0].kvm = nil") + } + if k.SSHKeyFile != "/etc/proxy-operator/kvm/id_ed25519" || k.DiskSizeGB != 20 { + t.Errorf("kvm = %+v, want sshKeyFile and diskSizeGb parsed", k) + } + if got := k.Images["debian-13"].SHA256; !strings.HasPrefix(got, "0123") { + t.Errorf("images[debian-13].sha256 = %q", got) + } + if k.Images["ubuntu"].Format != "raw" { + t.Errorf("images[ubuntu].format = %q, want raw", k.Images["ubuntu"].Format) + } + if mt := k.MachineTypes["medium"]; mt.VCPUs != 2 || mt.MemoryMiB != 2048 { + t.Errorf("machineTypes[medium] = %+v, want {2 2048}", mt) + } + if len(k.Hosts) != 2 || k.Hosts[1].Socket != "/run/libvirt/virtqemud-sock" || k.Hosts[1].Pool != "fast" { + t.Errorf("hosts = %+v, want two hosts with lab2's socket and pool parsed", k.Hosts) + } +} + +func TestLoadConfig_kvmInsecureHostKeyNeedsNoKnownHosts(t *testing.T) { + t.Parallel() + yaml := strings.Replace(validKVMYAML, + "knownHostsFile: /etc/proxy-operator/kvm/known_hosts", + "insecureIgnoreHostKey: true", 1) + if _, err := LoadConfig([]byte(yaml)); err != nil { + t.Fatalf("LoadConfig() error = %v, want nil", err) + } +} + +func TestLoadConfig_kvmInvalid(t *testing.T) { + t.Parallel() + minimal := func(kvmBody string) string { + return "providers:\n - name: kvm-home\n type: kvm\n kvm:\n" + indent(kvmBody, 6) + } + hostsAndTypes := ` +machineTypes: + small: {vcpus: 1, memoryMiB: 1024} +hosts: + - {name: nas1, address: nas1:22, user: virt, bridge: br0} +` + tests := []struct { + name string + yaml string + wantErrSub string + }{ + { + name: "missing kvm block", + yaml: "providers:\n - name: kvm-home\n type: kvm\n", + wantErrSub: "kvm block is required", + }, + { + name: "missing sshKeyFile", + yaml: minimal("knownHostsFile: /kh\n" + hostsAndTypes), + wantErrSub: "kvm.sshKeyFile is required", + }, + { + name: "missing knownHostsFile without insecure flag", + yaml: minimal("sshKeyFile: /key\n" + hostsAndTypes), + wantErrSub: "kvm.knownHostsFile is required unless", + }, + { + name: "negative disk size", + yaml: minimal("sshKeyFile: /key\nknownHostsFile: /kh\ndiskSizeGb: -1\n" + hostsAndTypes), + wantErrSub: "diskSizeGb must not be negative", + }, + { + name: "no machine types", + yaml: minimal("sshKeyFile: /key\nknownHostsFile: /kh\nhosts:\n - {name: nas1, address: nas1:22, user: virt, bridge: br0}\n"), + wantErrSub: "machineTypes must define at least one", + }, + { + name: "machine type with zero memory", + yaml: minimal("sshKeyFile: /key\nknownHostsFile: /kh\nmachineTypes:\n small: {vcpus: 1}\nhosts:\n - {name: nas1, address: nas1:22, user: virt, bridge: br0}\n"), + wantErrSub: `machineTypes["small"] needs vcpus > 0 and memoryMiB > 0`, + }, + { + name: "no hosts", + yaml: minimal("sshKeyFile: /key\nknownHostsFile: /kh\nmachineTypes:\n small: {vcpus: 1, memoryMiB: 1024}\n"), + wantErrSub: "kvm.hosts must list at least one host", + }, + { + name: "duplicate host name", + yaml: minimal("sshKeyFile: /key\nknownHostsFile: /kh\nmachineTypes:\n small: {vcpus: 1, memoryMiB: 1024}\nhosts:\n - {name: nas1, address: a:22, user: virt, bridge: br0}\n - {name: nas1, address: b:22, user: virt, bridge: br0}\n"), + wantErrSub: `duplicate host name "nas1"`, + }, + { + name: "host missing address", + yaml: minimal("sshKeyFile: /key\nknownHostsFile: /kh\nmachineTypes:\n small: {vcpus: 1, memoryMiB: 1024}\nhosts:\n - {name: nas1, user: virt, bridge: br0}\n"), + wantErrSub: "address is required", + }, + { + name: "host missing user", + yaml: minimal("sshKeyFile: /key\nknownHostsFile: /kh\nmachineTypes:\n small: {vcpus: 1, memoryMiB: 1024}\nhosts:\n - {name: nas1, address: a:22, bridge: br0}\n"), + wantErrSub: "user is required", + }, + { + name: "host missing bridge", + yaml: minimal("sshKeyFile: /key\nknownHostsFile: /kh\nmachineTypes:\n small: {vcpus: 1, memoryMiB: 1024}\nhosts:\n - {name: nas1, address: a:22, user: virt}\n"), + wantErrSub: "bridge is required", + }, + { + name: "image with non-http url", + yaml: minimal("sshKeyFile: /key\nknownHostsFile: /kh\nimages:\n d: {url: file:///tmp/x.qcow2}\n" + hostsAndTypes), + wantErrSub: `images["d"].url must be an http(s) URL`, + }, + { + name: "image with bad sha256", + yaml: minimal("sshKeyFile: /key\nknownHostsFile: /kh\nimages:\n d: {url: https://x/y.qcow2, sha256: abc}\n" + hostsAndTypes), + wantErrSub: `images["d"].sha256 must be 64 hex characters`, + }, + { + name: "image with unknown format", + yaml: minimal("sshKeyFile: /key\nknownHostsFile: /kh\nimages:\n d: {url: https://x/y.vmdk, format: vmdk}\n" + hostsAndTypes), + wantErrSub: `images["d"].format must be qcow2 or raw`, + }, + { + name: "kvm type with gcp block", + yaml: ` +providers: + - name: p1 + type: kvm + gcp: + project: my-project`, + wantErrSub: "type is kvm but a gcp block is set", + }, + { + name: "gcp type with kvm block", + yaml: ` +providers: + - name: p1 + type: gcp + gcp: + project: my-project + kvm: + sshKeyFile: /key`, + wantErrSub: "type is gcp but a kvm block is set", + }, + } + for _, tc := range tests { + t.Run(tc.name, func(t *testing.T) { + t.Parallel() + _, err := LoadConfig([]byte(tc.yaml)) + if err == nil { + t.Fatalf("LoadConfig() error = nil, want error containing %q", tc.wantErrSub) + } + if !strings.Contains(err.Error(), tc.wantErrSub) { + t.Errorf("LoadConfig() error = %q, want substring %q", err.Error(), tc.wantErrSub) + } + }) + } +} + +func indent(s string, n int) string { + pad := strings.Repeat(" ", n) + lines := strings.Split(strings.TrimLeft(s, "\n"), "\n") + for i, l := range lines { + if l != "" { + lines[i] = pad + l + } + } + return strings.Join(lines, "\n") +} -- 2.54.0 From 162607dfe27a612fcea84323624896f457f300ce Mon Sep 17 00:00:00 2001 From: kacerr Date: Fri, 4 Sep 2026 20:26:17 +0000 Subject: [PATCH 03/28] Add KVM provider pure builders: domain XML, volumes, seed ISO, metadata Introduces internal/provider/kvm with the side-effect-free pieces of the provider: providerID format/parse, the namespaced element that carries the GC contract, q35 domain XML over pool volumes, volume XML for overlays and raw seeds, the NoCloud seed ISO built with kdomanski/iso9660, and config defaults. All unit-tested. Step 2 of docs/plans/2026-09-04-1404-kvm-provider.md. Co-Authored-By: Claude Fable 5.1 --- .../2026-09-04-1404-kvm-provider.md | 49 +++- go.mod | 1 + go.sum | 2 + internal/provider/kvm/domainxml.go | 224 ++++++++++++++++++ internal/provider/kvm/domainxml_test.go | 113 +++++++++ internal/provider/kvm/id.go | 21 ++ internal/provider/kvm/id_test.go | 41 ++++ internal/provider/kvm/kvm.go | 47 ++++ internal/provider/kvm/kvm_test.go | 44 ++++ internal/provider/kvm/metadata.go | 57 +++++ internal/provider/kvm/metadata_test.go | 79 ++++++ internal/provider/kvm/seed.go | 52 ++++ internal/provider/kvm/seed_test.go | 91 +++++++ internal/provider/kvm/volume.go | 78 ++++++ internal/provider/kvm/volume_test.go | 62 +++++ 15 files changed, 960 insertions(+), 1 deletion(-) create mode 100644 internal/provider/kvm/domainxml.go create mode 100644 internal/provider/kvm/domainxml_test.go create mode 100644 internal/provider/kvm/id.go create mode 100644 internal/provider/kvm/id_test.go create mode 100644 internal/provider/kvm/kvm.go create mode 100644 internal/provider/kvm/kvm_test.go create mode 100644 internal/provider/kvm/metadata.go create mode 100644 internal/provider/kvm/metadata_test.go create mode 100644 internal/provider/kvm/seed.go create mode 100644 internal/provider/kvm/seed_test.go create mode 100644 internal/provider/kvm/volume.go create mode 100644 internal/provider/kvm/volume_test.go diff --git a/docs/plans-executions/2026-09-04-1404-kvm-provider.md b/docs/plans-executions/2026-09-04-1404-kvm-provider.md index 656015c..b630134 100644 --- a/docs/plans-executions/2026-09-04-1404-kvm-provider.md +++ b/docs/plans-executions/2026-09-04-1404-kvm-provider.md @@ -4,7 +4,7 @@ Plan: [2026-09-04-1404-kvm-provider.md](../plans/2026-09-04-1404-kvm-provider.md - [x] Step 0 — Save plan + branch `feat/kvm-provider` - [x] Step 1 — Config (`KVMConfig` & friends, `validate()` case, tests) -- [ ] Step 2 — Pure builders (`domainxml.go`, `volume.go`, `seed.go`, `id.go`, `metadata.go`) +- [x] Step 2 — Pure builders (`domainxml.go`, `volume.go`, `seed.go`, `id.go`, `metadata.go`) - [ ] Step 3 — Transport + errors (`conn.go`, `errors.go`) - [ ] Step 3b — Image cache (`image.go`) - [ ] Step 4 — Provider methods (`kvm.go`) over a fake API @@ -52,3 +52,50 @@ in favour of `NewWithDialer(socket.Dialer)`, where `Dialer` is a one-method `Dial() (net.Conn, error)` interface. Step 3 should wrap the SSH client's unix forward in that interface instead of using `New`; the plan's transport design is otherwise unaffected. + +## Step 2 — Pure builders + +`internal/provider/kvm/` now has the five builders from the plan plus +`kvm.go` holding the package doc and `withDefaults` (disk size 10 GiB, the +default vendor-data, `/var/run/libvirt/libvirt-sock`, pool `default`, image +format `qcow2`). Everything is a pure function over `encoding/xml` structs or +strings, unit-tested by unmarshalling the output back and checking fields. + +Two design choices differ from the plan text: + +- **Domain disks reference pool volumes, not host paths.** The plan had + `buildDomainXML` take the overlay and seed paths (needing two + `StorageVolGetPath` round-trips per Create). The domain XML now uses + ``, so only the *base* + image path is ever looked up (the overlay's `` needs a real + path). Fewer RPCs and the builder's inputs are all names. +- **Capacities are always `unit="bytes"`.** libvirt's `G` means GiB but + `GB` means 10^9; spelling the byte count out removes the ambiguity from + both the overlay and the seed/base volumes. + +The metadata element is written with a default namespace +(``) because `encoding/xml` cannot emit prefixes on +marshal. libvirt may hand it back prefixed; `parseInstanceMetadata` is +tested against both spellings since `encoding/xml` matches on the namespace +URI. Whether libvirt accepts the default-namespace form on `DomainDefineXML` +is confirmed in Step 6 against a real host, not here. + +Checked before relying on `github.com/kdomanski/iso9660` for the seed: its +filename mangling lowercases and keeps `-`, so `user-data`/`meta-data`/ +`vendor-data` come out as `user-data;1` etc. in the directory record, which +the Linux isofs driver (and therefore cloud-init's NoCloud mount) presents as +`user-data`. The seed test opens the ISO with the same library's reader and +asserts label `cidata`, all three files, and byte-identical output for +identical inputs. `NewWriter()` stages files in a temp directory, so +`Cleanup()` is deferred. + +`go mod tidy` ran as part of this step; it promoted `kdomanski/iso9660` to a +direct dependency and dropped `go-libvirt` again because nothing imports it +yet — Step 3 brings it back. + +Worth noting: `withDefaults` copies the `Images` map and `Hosts` slice +instead of mutating the config in place, so the registry's shared +`ProviderConfig` stays untouched. The default vendor-data installs +`qemu-guest-agent` via `packages:` + `runcmd`, which means the first boot +needs network access to the distro mirrors; a Proxy on an isolated bridge +would still work (ARP discovery), just without the agent. diff --git a/go.mod b/go.mod index 48cfaa6..de8c5e3 100644 --- a/go.mod +++ b/go.mod @@ -5,6 +5,7 @@ go 1.26.0 require ( cloud.google.com/go/compute v1.65.0 github.com/go-logr/logr v1.4.4 + github.com/kdomanski/iso9660 v0.4.0 github.com/onsi/ginkgo/v2 v2.27.4 github.com/onsi/gomega v1.39.0 github.com/prometheus/client_golang v1.23.2 diff --git a/go.sum b/go.sum index 152847c..807f975 100644 --- a/go.sum +++ b/go.sum @@ -96,6 +96,8 @@ github.com/joshdk/go-junit v1.0.0 h1:S86cUKIdwBHWwA6xCmFlf3RTLfVXYQfvanM5Uh+K6GE github.com/joshdk/go-junit v1.0.0/go.mod h1:TiiV0PqkaNfFXjEiyjWM3XXrhVyCa1K4Zfga6W52ung= github.com/json-iterator/go v1.1.12 h1:PV8peI4a0ysnczrg+LtxykD8LfKY9ML6u2jnxaEnrnM= github.com/json-iterator/go v1.1.12/go.mod h1:e30LSqwooZae/UwlEbR2852Gd8hjQvJoHmT4TnhNGBo= +github.com/kdomanski/iso9660 v0.4.0 h1:BPKKdcINz3m0MdjIMwS0wx1nofsOjxOq8TOr45WGHFg= +github.com/kdomanski/iso9660 v0.4.0/go.mod h1:OxUSupHsO9ceI8lBLPJKWBTphLemjrCQY8LPXM7qSzU= github.com/klauspost/compress v1.18.0 h1:c/Cqfb0r+Yi+JtIEq73FWXVkRonBlf0CRNYc8Zttxdo= github.com/klauspost/compress v1.18.0/go.mod h1:2Pp+KzxcywXVXMr50+X0Q/Lsb43OQHYWRCY2AiWywWQ= github.com/kr/pretty v0.2.1/go.mod h1:ipq/a2n7PKx3OHsz4KJII5eveXtPO4qwEXGdVfWzfnI= diff --git a/internal/provider/kvm/domainxml.go b/internal/provider/kvm/domainxml.go new file mode 100644 index 0000000..32f0732 --- /dev/null +++ b/internal/provider/kvm/domainxml.go @@ -0,0 +1,224 @@ +package kvm + +import ( + "encoding/xml" + "fmt" + "time" +) + +const guestAgentChannel = "org.qemu.guest_agent.0" + +// domainParams is everything buildDomainXML needs, already resolved from the +// config and placement so the builder stays a pure function. +type domainParams struct { + Name string + UID string + CreatedAt time.Time + VCPUs int32 + MemoryMiB int64 + Pool string + Bridge string + DiskVol string + SeedVol string +} + +// buildDomainXML renders a q35 KVM guest: host-passthrough CPU, a virtio +// overlay disk and a SATA cdrom for the seed (both referenced as pool +// volumes so no host paths are needed), one virtio NIC on the host bridge, +// a serial console, the guest-agent channel, a virtio RNG, no graphics, and +// the operator's metadata element. +func buildDomainXML(p domainParams) (string, error) { + d := domainXML{ + Type: "kvm", + Name: p.Name, + Metadata: domainMetadataXML{Instance: newInstanceMetadata(p.UID, p.CreatedAt)}, + Memory: sizeXML{Unit: "MiB", Value: p.MemoryMiB}, + VCPU: p.VCPUs, + OS: osXML{ + Type: osTypeXML{Arch: "x86_64", Machine: "q35", Value: "hvm"}, + Boot: bootXML{Dev: "hd"}, + }, + Features: featuresXML{ACPI: &struct{}{}, APIC: &struct{}{}}, + CPU: cpuXML{Mode: "host-passthrough"}, + Clock: clockXML{Offset: "utc"}, + OnPoweroff: "destroy", + OnReboot: "restart", + OnCrash: "destroy", + Devices: devicesXML{ + Disks: []diskXML{ + { + Type: "volume", + Device: "disk", + Driver: driverXML{Name: "qemu", Type: formatQcow2}, + Source: diskSourceXML{Pool: p.Pool, Volume: p.DiskVol}, + Target: targetXML{Dev: "vda", Bus: "virtio"}, + }, + { + Type: "volume", + Device: "cdrom", + Driver: driverXML{Name: "qemu", Type: formatRaw}, + Source: diskSourceXML{Pool: p.Pool, Volume: p.SeedVol}, + Target: targetXML{Dev: "sda", Bus: "sata"}, + ReadOnly: &struct{}{}, + }, + }, + Interfaces: []interfaceXML{{ + Type: "bridge", + Source: ifaceSourceXML{Bridge: p.Bridge}, + Model: modelXML{Type: "virtio"}, + }}, + Serials: []serialXML{{Type: "pty", Target: portXML{Port: 0}}}, + Consoles: []consoleXML{{Type: "pty", Target: consoleTargetXML{Type: "serial", Port: 0}}}, + Channels: []channelXML{{ + Type: "unix", + Target: channelTargetXML{Type: "virtio", Name: guestAgentChannel}, + }}, + RNG: rngXML{Model: "virtio", Backend: rngBackendXML{Model: "random", Value: "/dev/urandom"}}, + }, + } + out, err := xml.Marshal(d) + if err != nil { + return "", fmt.Errorf("marshalling domain xml: %w", err) + } + return string(out), nil +} + +type domainXML struct { + XMLName xml.Name `xml:"domain"` + Type string `xml:"type,attr"` + Name string `xml:"name"` + Metadata domainMetadataXML `xml:"metadata"` + Memory sizeXML `xml:"memory"` + VCPU int32 `xml:"vcpu"` + OS osXML `xml:"os"` + Features featuresXML `xml:"features"` + CPU cpuXML `xml:"cpu"` + Clock clockXML `xml:"clock"` + OnPoweroff string `xml:"on_poweroff"` + OnReboot string `xml:"on_reboot"` + OnCrash string `xml:"on_crash"` + Devices devicesXML `xml:"devices"` +} + +type domainMetadataXML struct { + Instance instanceMetadata +} + +type sizeXML struct { + Unit string `xml:"unit,attr"` + Value int64 `xml:",chardata"` +} + +type osXML struct { + Type osTypeXML `xml:"type"` + Boot bootXML `xml:"boot"` +} + +type osTypeXML struct { + Arch string `xml:"arch,attr"` + Machine string `xml:"machine,attr"` + Value string `xml:",chardata"` +} + +type bootXML struct { + Dev string `xml:"dev,attr"` +} + +type featuresXML struct { + ACPI *struct{} `xml:"acpi"` + APIC *struct{} `xml:"apic"` +} + +type cpuXML struct { + Mode string `xml:"mode,attr"` +} + +type clockXML struct { + Offset string `xml:"offset,attr"` +} + +type devicesXML struct { + Disks []diskXML `xml:"disk"` + Interfaces []interfaceXML `xml:"interface"` + Serials []serialXML `xml:"serial"` + Consoles []consoleXML `xml:"console"` + Channels []channelXML `xml:"channel"` + RNG rngXML `xml:"rng"` +} + +type diskXML struct { + Type string `xml:"type,attr"` + Device string `xml:"device,attr"` + Driver driverXML `xml:"driver"` + Source diskSourceXML `xml:"source"` + Target targetXML `xml:"target"` + ReadOnly *struct{} `xml:"readonly"` +} + +type driverXML struct { + Name string `xml:"name,attr"` + Type string `xml:"type,attr"` +} + +type diskSourceXML struct { + Pool string `xml:"pool,attr"` + Volume string `xml:"volume,attr"` +} + +type targetXML struct { + Dev string `xml:"dev,attr"` + Bus string `xml:"bus,attr"` +} + +type interfaceXML struct { + Type string `xml:"type,attr"` + Source ifaceSourceXML `xml:"source"` + Model modelXML `xml:"model"` +} + +type ifaceSourceXML struct { + Bridge string `xml:"bridge,attr"` +} + +type modelXML struct { + Type string `xml:"type,attr"` +} + +type serialXML struct { + Type string `xml:"type,attr"` + Target portXML `xml:"target"` +} + +type portXML struct { + Port int `xml:"port,attr"` +} + +type consoleXML struct { + Type string `xml:"type,attr"` + Target consoleTargetXML `xml:"target"` +} + +type consoleTargetXML struct { + Type string `xml:"type,attr"` + Port int `xml:"port,attr"` +} + +type channelXML struct { + Type string `xml:"type,attr"` + Target channelTargetXML `xml:"target"` +} + +type channelTargetXML struct { + Type string `xml:"type,attr"` + Name string `xml:"name,attr"` +} + +type rngXML struct { + Model string `xml:"model,attr"` + Backend rngBackendXML `xml:"backend"` +} + +type rngBackendXML struct { + Model string `xml:"model,attr"` + Value string `xml:",chardata"` +} diff --git a/internal/provider/kvm/domainxml_test.go b/internal/provider/kvm/domainxml_test.go new file mode 100644 index 0000000..d52474c --- /dev/null +++ b/internal/provider/kvm/domainxml_test.go @@ -0,0 +1,113 @@ +package kvm + +import ( + "encoding/xml" + "strings" + "testing" + "time" +) + +func testDomainParams() domainParams { + return domainParams{ + Name: "proxy-abc123def456ghij", + UID: "11111111-2222-3333-4444-555555555555", + CreatedAt: time.Date(2026, 9, 4, 12, 0, 0, 0, time.UTC), + VCPUs: 2, + MemoryMiB: 2048, + Pool: "default", + Bridge: "br0", + DiskVol: "proxy-abc123def456ghij.qcow2", + SeedVol: "proxy-abc123def456ghij-seed.iso", + } +} + +func TestBuildDomainXML_fieldByField(t *testing.T) { + t.Parallel() + raw, err := buildDomainXML(testDomainParams()) + if err != nil { + t.Fatalf("buildDomainXML() error = %v", err) + } + var d domainXML + if err := xml.Unmarshal([]byte(raw), &d); err != nil { + t.Fatalf("output is not well-formed XML: %v\n%s", err, raw) + } + + if d.Type != "kvm" || d.Name != "proxy-abc123def456ghij" { + t.Errorf("type/name = %s/%s", d.Type, d.Name) + } + if d.VCPU != 2 || d.Memory.Value != 2048 || d.Memory.Unit != "MiB" { + t.Errorf("vcpu/memory = %d/%d %s, want 2/2048 MiB", d.VCPU, d.Memory.Value, d.Memory.Unit) + } + if d.OS.Type.Value != "hvm" || d.OS.Type.Arch != "x86_64" || d.OS.Type.Machine != "q35" || d.OS.Boot.Dev != "hd" { + t.Errorf("os = %+v", d.OS) + } + if d.Features.ACPI == nil || d.Features.APIC == nil { + t.Errorf("features = %+v, want acpi and apic", d.Features) + } + if d.CPU.Mode != "host-passthrough" { + t.Errorf("cpu mode = %q", d.CPU.Mode) + } + if d.OnPoweroff != "destroy" || d.OnReboot != "restart" || d.OnCrash != "destroy" { + t.Errorf("lifecycle = %s/%s/%s", d.OnPoweroff, d.OnReboot, d.OnCrash) + } + + m := d.Metadata.Instance + if !m.managed() || m.UID != "11111111-2222-3333-4444-555555555555" || m.CreatedAt != "2026-09-04T12:00:00Z" { + t.Errorf("metadata = %+v", m) + } + if !strings.Contains(raw, `xmlns="`+metadataURI+`"`) { + t.Errorf("metadata element is not namespaced; libvirt rejects that:\n%s", raw) + } + + if len(d.Devices.Disks) != 2 { + t.Fatalf("disks = %d, want 2", len(d.Devices.Disks)) + } + disk, seed := d.Devices.Disks[0], d.Devices.Disks[1] + if disk.Type != "volume" || disk.Device != "disk" || disk.Driver.Type != "qcow2" || + disk.Source.Pool != "default" || disk.Source.Volume != "proxy-abc123def456ghij.qcow2" || + disk.Target.Dev != "vda" || disk.Target.Bus != "virtio" || disk.ReadOnly != nil { + t.Errorf("overlay disk = %+v", disk) + } + if seed.Type != "volume" || seed.Device != "cdrom" || seed.Driver.Type != "raw" || + seed.Source.Pool != "default" || seed.Source.Volume != "proxy-abc123def456ghij-seed.iso" || + seed.Target.Bus != "sata" || seed.ReadOnly == nil { + t.Errorf("seed cdrom = %+v", seed) + } + + if len(d.Devices.Interfaces) != 1 { + t.Fatalf("interfaces = %d, want 1", len(d.Devices.Interfaces)) + } + if nic := d.Devices.Interfaces[0]; nic.Type != "bridge" || nic.Source.Bridge != "br0" || nic.Model.Type != "virtio" { + t.Errorf("interface = %+v", nic) + } + + if len(d.Devices.Channels) != 1 || d.Devices.Channels[0].Target.Name != guestAgentChannel || d.Devices.Channels[0].Target.Type != "virtio" { + t.Errorf("channels = %+v, want one virtio %s channel", d.Devices.Channels, guestAgentChannel) + } + if len(d.Devices.Serials) != 1 || len(d.Devices.Consoles) != 1 || d.Devices.Consoles[0].Target.Type != "serial" { + t.Errorf("serial/console = %+v/%+v", d.Devices.Serials, d.Devices.Consoles) + } + if d.Devices.RNG.Model != "virtio" || d.Devices.RNG.Backend.Value != "/dev/urandom" { + t.Errorf("rng = %+v", d.Devices.RNG) + } + if strings.Contains(raw, "/domains/", id) + } + return parts[1], parts[3], nil +} diff --git a/internal/provider/kvm/id_test.go b/internal/provider/kvm/id_test.go new file mode 100644 index 0000000..0a7c158 --- /dev/null +++ b/internal/provider/kvm/id_test.go @@ -0,0 +1,41 @@ +package kvm + +import "testing" + +func TestParseProviderID_roundTrip(t *testing.T) { + t.Parallel() + id := formatProviderID("nas1", "proxy-abc123def456ghij") + if id != "hosts/nas1/domains/proxy-abc123def456ghij" { + t.Fatalf("formatProviderID() = %q", id) + } + host, name, err := parseProviderID(id) + if err != nil { + t.Fatalf("parseProviderID(%q) error = %v", id, err) + } + if host != "nas1" || name != "proxy-abc123def456ghij" { + t.Errorf("parseProviderID(%q) = %q, %q", id, host, name) + } +} + +func TestParseProviderID_malformed(t *testing.T) { + t.Parallel() + tests := []struct { + name string + id string + }{ + {"empty", ""}, + {"gcp style", "zones/europe-west1-b/instances/proxy-x"}, + {"missing name", "hosts/nas1/domains/"}, + {"missing host", "hosts//domains/proxy-x"}, + {"too many segments", "hosts/nas1/domains/proxy-x/extra"}, + {"bare name", "proxy-x"}, + } + for _, tc := range tests { + t.Run(tc.name, func(t *testing.T) { + t.Parallel() + if _, _, err := parseProviderID(tc.id); err == nil { + t.Errorf("parseProviderID(%q) error = nil, want error", tc.id) + } + }) + } +} diff --git a/internal/provider/kvm/kvm.go b/internal/provider/kvm/kvm.go new file mode 100644 index 0000000..e5653e3 --- /dev/null +++ b/internal/provider/kvm/kvm.go @@ -0,0 +1,47 @@ +// Package kvm implements the provider contract on libvirt/KVM hypervisors. +// Each configured host is reached over SSH with the libvirt unix socket +// forwarded through the connection, so the hypervisors need no libvirtd TCP +// or TLS listener and the operator image needs no libvirt libraries. VMs +// boot a stock cloud image (fetched into the host's storage pool on first +// use) through a qcow2 overlay, with the Proxy's cloud-init delivered on a +// NoCloud seed ISO built in-process. +package kvm + +import "gitea.home.hrajfrisbee.cz/kacerr/egress-proxies-operator/internal/provider" + +const ( + defaultDiskSizeGB = 10 + defaultSocket = "/var/run/libvirt/libvirt-sock" + defaultPool = "default" + formatQcow2 = "qcow2" + formatRaw = "raw" +) + +func withDefaults(cfg provider.KVMConfig) provider.KVMConfig { + if cfg.DiskSizeGB == 0 { + cfg.DiskSizeGB = defaultDiskSizeGB + } + if cfg.VendorData == "" { + cfg.VendorData = defaultVendorData + } + images := make(map[string]provider.KVMImage, len(cfg.Images)) + for k, img := range cfg.Images { + if img.Format == "" { + img.Format = formatQcow2 + } + images[k] = img + } + cfg.Images = images + hosts := make([]provider.KVMHost, len(cfg.Hosts)) + for i, h := range cfg.Hosts { + if h.Socket == "" { + h.Socket = defaultSocket + } + if h.Pool == "" { + h.Pool = defaultPool + } + hosts[i] = h + } + cfg.Hosts = hosts + return cfg +} diff --git a/internal/provider/kvm/kvm_test.go b/internal/provider/kvm/kvm_test.go new file mode 100644 index 0000000..0f04970 --- /dev/null +++ b/internal/provider/kvm/kvm_test.go @@ -0,0 +1,44 @@ +package kvm + +import ( + "testing" + + "gitea.home.hrajfrisbee.cz/kacerr/egress-proxies-operator/internal/provider" +) + +func TestWithDefaults(t *testing.T) { + t.Parallel() + cfg := withDefaults(provider.KVMConfig{ + Images: map[string]provider.KVMImage{ + "debian": {URL: "https://x/d.qcow2"}, + "raw": {URL: "https://x/r.img", Format: "raw"}, + }, + Hosts: []provider.KVMHost{ + {Name: "a", Address: "a:22", User: "virt", Bridge: "br0"}, + {Name: "b", Address: "b:22", User: "virt", Bridge: "br1", Socket: "/run/s", Pool: "fast"}, + }, + }) + if cfg.DiskSizeGB != defaultDiskSizeGB { + t.Errorf("diskSizeGb = %d, want %d", cfg.DiskSizeGB, defaultDiskSizeGB) + } + if cfg.VendorData != defaultVendorData { + t.Errorf("vendorData = %q, want the default", cfg.VendorData) + } + if cfg.Images["debian"].Format != "qcow2" || cfg.Images["raw"].Format != "raw" { + t.Errorf("image formats = %q/%q, want qcow2 default and raw kept", cfg.Images["debian"].Format, cfg.Images["raw"].Format) + } + if a := cfg.Hosts[0]; a.Socket != defaultSocket || a.Pool != defaultPool { + t.Errorf("host a = %+v, want default socket and pool", a) + } + if b := cfg.Hosts[1]; b.Socket != "/run/s" || b.Pool != "fast" { + t.Errorf("host b = %+v, want explicit socket and pool kept", b) + } +} + +func TestWithDefaults_keepsExplicitValues(t *testing.T) { + t.Parallel() + cfg := withDefaults(provider.KVMConfig{DiskSizeGB: 42, VendorData: "#cloud-config\n"}) + if cfg.DiskSizeGB != 42 || cfg.VendorData != "#cloud-config\n" { + t.Errorf("cfg = %+v, want explicit values kept", cfg) + } +} diff --git a/internal/provider/kvm/metadata.go b/internal/provider/kvm/metadata.go new file mode 100644 index 0000000..55ab4e7 --- /dev/null +++ b/internal/provider/kvm/metadata.go @@ -0,0 +1,57 @@ +package kvm + +import ( + "encoding/xml" + "fmt" + "time" + + "gitea.home.hrajfrisbee.cz/kacerr/egress-proxies-operator/internal/provider" +) + +// metadataURI namespaces the operator's element inside the domain's +// . libvirt has no labels or tags, but it stores any namespaced XML +// there and hands it back through DomainGetMetadata, which is how the GC +// contract (managed flag + owning UID) and the creation time survive across +// operator restarts. +const metadataURI = "https://gitea.home.hrajfrisbee.cz/kacerr/egress-proxies-operator/kvm" + +type instanceMetadata struct { + XMLName xml.Name `xml:"https://gitea.home.hrajfrisbee.cz/kacerr/egress-proxies-operator/kvm instance"` + Managed string `xml:"managed"` + UID string `xml:"uid"` + CreatedAt string `xml:"createdAt"` +} + +func newInstanceMetadata(uid string, createdAt time.Time) instanceMetadata { + return instanceMetadata{ + Managed: provider.LabelManagedYes, + UID: uid, + CreatedAt: createdAt.UTC().Format(time.RFC3339), + } +} + +// parseInstanceMetadata decodes the element as DomainGetMetadata returns it. +// libvirt may re-serialise it with a namespace prefix rather than the default +// namespace it was defined with; encoding/xml matches on namespace URI, so +// both spellings decode. +func parseInstanceMetadata(s string) (instanceMetadata, error) { + var m instanceMetadata + if err := xml.Unmarshal([]byte(s), &m); err != nil { + return instanceMetadata{}, fmt.Errorf("parsing domain metadata: %w", err) + } + return m, nil +} + +func (m instanceMetadata) managed() bool { + return m.Managed == provider.LabelManagedYes +} + +// createdAt returns the zero time on a malformed timestamp: orphan GC treats +// zero as "old", so a corrupt value can never protect a domain forever. +func (m instanceMetadata) createdAt() time.Time { + t, err := time.Parse(time.RFC3339, m.CreatedAt) + if err != nil { + return time.Time{} + } + return t +} diff --git a/internal/provider/kvm/metadata_test.go b/internal/provider/kvm/metadata_test.go new file mode 100644 index 0000000..f3ab94f --- /dev/null +++ b/internal/provider/kvm/metadata_test.go @@ -0,0 +1,79 @@ +package kvm + +import ( + "encoding/xml" + "strings" + "testing" + "time" +) + +func TestInstanceMetadata_roundTrip(t *testing.T) { + t.Parallel() + created := time.Date(2026, 9, 4, 12, 30, 0, 0, time.FixedZone("CEST", 2*3600)) + raw, err := xml.Marshal(newInstanceMetadata("uid-1", created)) + if err != nil { + t.Fatalf("Marshal() error = %v", err) + } + if !strings.Contains(string(raw), `xmlns="`+metadataURI+`"`) { + t.Errorf("marshalled metadata lacks the namespace declaration: %s", raw) + } + m, err := parseInstanceMetadata(string(raw)) + if err != nil { + t.Fatalf("parseInstanceMetadata() error = %v", err) + } + if !m.managed() || m.UID != "uid-1" { + t.Errorf("parsed = %+v, want managed with uid-1", m) + } + if got := m.createdAt(); !got.Equal(created) || got.Location() != time.UTC { + t.Errorf("createdAt() = %v, want %v in UTC", got, created) + } +} + +func TestParseInstanceMetadata_libvirtPrefixedForm(t *testing.T) { + t.Parallel() + // libvirt re-serialises the stored element with whatever prefix it + // chooses; only the namespace URI is contractual. + in := `` + + `trueuid-2` + + `2026-09-04T10:00:00Z` + m, err := parseInstanceMetadata(in) + if err != nil { + t.Fatalf("parseInstanceMetadata() error = %v", err) + } + if !m.managed() || m.UID != "uid-2" || m.createdAt().Hour() != 10 { + t.Errorf("parsed = %+v", m) + } +} + +func TestParseInstanceMetadata_notManaged(t *testing.T) { + t.Parallel() + tests := []struct { + name string + in string + wantErr bool + }{ + {"managed false", `falseu`, false}, + {"foreign element", `true`, true}, + {"garbage", `not xml`, true}, + } + for _, tc := range tests { + t.Run(tc.name, func(t *testing.T) { + t.Parallel() + m, err := parseInstanceMetadata(tc.in) + if (err != nil) != tc.wantErr { + t.Fatalf("error = %v, wantErr %v", err, tc.wantErr) + } + if m.managed() { + t.Errorf("managed() = true for %q", tc.in) + } + }) + } +} + +func TestInstanceMetadata_createdAtMalformedIsZero(t *testing.T) { + t.Parallel() + m := instanceMetadata{CreatedAt: "yesterday"} + if !m.createdAt().IsZero() { + t.Errorf("createdAt() = %v, want zero time", m.createdAt()) + } +} diff --git a/internal/provider/kvm/seed.go b/internal/provider/kvm/seed.go new file mode 100644 index 0000000..49e23cd --- /dev/null +++ b/internal/provider/kvm/seed.go @@ -0,0 +1,52 @@ +package kvm + +import ( + "bytes" + "fmt" + "strings" + + "github.com/kdomanski/iso9660" +) + +// seedLabel is the volume label cloud-init's NoCloud datasource looks for. +const seedLabel = "cidata" + +// defaultVendorData is the cloud-init vendor-data shipped on every seed +// unless the config overrides it. cloud-init merges it under the Proxy's own +// user-data, so stock images grow a guest agent (which lets the provider +// read the VM's IP before ARP sees traffic) without the user's cloud-init +// having to know about it. +const defaultVendorData = `#cloud-config +packages: + - qemu-guest-agent +runcmd: + - [systemctl, enable, --now, qemu-guest-agent] +` + +// buildSeedISO renders a NoCloud seed: user-data verbatim, vendor-data as +// given, and a meta-data whose instance-id is the deterministic VM name, so a +// re-created seed for the same name is byte-identical and cloud-init never +// mistakes a reboot for a new instance. +func buildSeedISO(name, userData, vendorData string) ([]byte, error) { + w, err := iso9660.NewWriter() + if err != nil { + return nil, fmt.Errorf("creating iso writer: %w", err) + } + defer w.Cleanup() //nolint:errcheck // temp-dir cleanup; nothing to do on failure + + files := []struct{ name, body string }{ + {"user-data", userData}, + {"meta-data", fmt.Sprintf("instance-id: %s\nlocal-hostname: %s\n", name, name)}, + {"vendor-data", vendorData}, + } + for _, f := range files { + if err := w.AddFile(strings.NewReader(f.body), f.name); err != nil { + return nil, fmt.Errorf("adding %s to seed iso: %w", f.name, err) + } + } + var buf bytes.Buffer + if err := w.WriteTo(&buf, seedLabel); err != nil { + return nil, fmt.Errorf("writing seed iso: %w", err) + } + return buf.Bytes(), nil +} diff --git a/internal/provider/kvm/seed_test.go b/internal/provider/kvm/seed_test.go new file mode 100644 index 0000000..4bd9c08 --- /dev/null +++ b/internal/provider/kvm/seed_test.go @@ -0,0 +1,91 @@ +package kvm + +import ( + "bytes" + "io" + "strings" + "testing" + + "github.com/kdomanski/iso9660" +) + +func readSeed(t *testing.T, iso []byte) (label string, files map[string]string) { + t.Helper() + img, err := iso9660.OpenImage(bytes.NewReader(iso)) + if err != nil { + t.Fatalf("OpenImage() error = %v", err) + } + label, err = img.Label() + if err != nil { + t.Fatalf("Label() error = %v", err) + } + root, err := img.RootDir() + if err != nil { + t.Fatalf("RootDir() error = %v", err) + } + children, err := root.GetChildren() + if err != nil { + t.Fatalf("GetChildren() error = %v", err) + } + files = make(map[string]string, len(children)) + for _, c := range children { + body, err := io.ReadAll(c.Reader()) + if err != nil { + t.Fatalf("reading %s: %v", c.Name(), err) + } + files[c.Name()] = string(body) + } + return label, files +} + +func TestBuildSeedISO_contents(t *testing.T) { + t.Parallel() + userData := "#cloud-config\npackages: [squid]\n" + iso, err := buildSeedISO("proxy-abc123def456ghij", userData, defaultVendorData) + if err != nil { + t.Fatalf("buildSeedISO() error = %v", err) + } + label, files := readSeed(t, iso) + if !strings.EqualFold(label, seedLabel) { + t.Errorf("label = %q, want %q", label, seedLabel) + } + if len(files) != 3 { + t.Errorf("files = %v, want exactly user-data, meta-data, vendor-data", files) + } + if files["user-data"] != userData { + t.Errorf("user-data = %q, want verbatim cloud-init", files["user-data"]) + } + if files["vendor-data"] != defaultVendorData { + t.Errorf("vendor-data = %q", files["vendor-data"]) + } + if got, want := files["meta-data"], "instance-id: proxy-abc123def456ghij\nlocal-hostname: proxy-abc123def456ghij\n"; got != want { + t.Errorf("meta-data = %q, want %q", got, want) + } +} + +func TestBuildSeedISO_deterministic(t *testing.T) { + t.Parallel() + a, err := buildSeedISO("proxy-x", "#cloud-config\n", "") + if err != nil { + t.Fatalf("first buildSeedISO() error = %v", err) + } + b, err := buildSeedISO("proxy-x", "#cloud-config\n", "") + if err != nil { + t.Fatalf("second buildSeedISO() error = %v", err) + } + if !bytes.Equal(a, b) { + t.Error("two builds for the same inputs differ; a re-created seed must be byte-identical") + } +} + +func TestBuildSeedISO_emptyUserData(t *testing.T) { + t.Parallel() + iso, err := buildSeedISO("proxy-x", "", defaultVendorData) + if err != nil { + t.Fatalf("buildSeedISO() error = %v", err) + } + _, files := readSeed(t, iso) + if body, ok := files["user-data"]; !ok || body != "" { + t.Errorf("user-data = %q (present=%v), want an empty file to still exist", body, ok) + } +} diff --git a/internal/provider/kvm/volume.go b/internal/provider/kvm/volume.go new file mode 100644 index 0000000..da0814d --- /dev/null +++ b/internal/provider/kvm/volume.go @@ -0,0 +1,78 @@ +package kvm + +import ( + "encoding/xml" + "fmt" +) + +const bytesPerGiB = 1 << 30 + +func overlayVolumeName(name string) string { return name + ".qcow2" } + +func seedVolumeName(name string) string { return name + "-seed.iso" } + +// volumeSpec describes a storage volume to create. A non-empty BackingPath +// makes it a copy-on-write overlay of that file. +type volumeSpec struct { + Name string + SizeBytes int64 + Format string + BackingPath string + BackingFormat string +} + +func overlayVolume(name string, sizeGB int64, basePath, baseFormat string) volumeSpec { + return volumeSpec{ + Name: overlayVolumeName(name), + SizeBytes: sizeGB * bytesPerGiB, + Format: formatQcow2, + BackingPath: basePath, + BackingFormat: baseFormat, + } +} + +func rawVolume(name string, sizeBytes int64) volumeSpec { + return volumeSpec{Name: name, SizeBytes: sizeBytes, Format: formatRaw} +} + +// buildVolumeXML renders the volume for StorageVolCreateXML. Capacity is +// always given in bytes so no libvirt unit-suffix ambiguity (G vs GiB) can +// creep in. +func buildVolumeXML(v volumeSpec) (string, error) { + x := volumeXML{ + Type: "file", + Name: v.Name, + Capacity: sizeXML{Unit: "bytes", Value: v.SizeBytes}, + Target: volumeTargetXML{Format: formatXML{Type: v.Format}}, + } + if v.BackingPath != "" { + x.BackingStore = &backingStoreXML{Path: v.BackingPath, Format: formatXML{Type: v.BackingFormat}} + } + out, err := xml.Marshal(x) + if err != nil { + return "", fmt.Errorf("marshalling volume xml: %w", err) + } + return string(out), nil +} + +type volumeXML struct { + XMLName xml.Name `xml:"volume"` + Type string `xml:"type,attr"` + Name string `xml:"name"` + Capacity sizeXML `xml:"capacity"` + Target volumeTargetXML `xml:"target"` + BackingStore *backingStoreXML `xml:"backingStore"` +} + +type volumeTargetXML struct { + Format formatXML `xml:"format"` +} + +type formatXML struct { + Type string `xml:"type,attr"` +} + +type backingStoreXML struct { + Path string `xml:"path"` + Format formatXML `xml:"format"` +} diff --git a/internal/provider/kvm/volume_test.go b/internal/provider/kvm/volume_test.go new file mode 100644 index 0000000..c9b68a8 --- /dev/null +++ b/internal/provider/kvm/volume_test.go @@ -0,0 +1,62 @@ +package kvm + +import ( + "encoding/xml" + "strings" + "testing" +) + +func TestVolumeNames(t *testing.T) { + t.Parallel() + if got := overlayVolumeName("proxy-x"); got != "proxy-x.qcow2" { + t.Errorf("overlayVolumeName = %q", got) + } + if got := seedVolumeName("proxy-x"); got != "proxy-x-seed.iso" { + t.Errorf("seedVolumeName = %q", got) + } +} + +func TestBuildVolumeXML_overlay(t *testing.T) { + t.Parallel() + raw, err := buildVolumeXML(overlayVolume("proxy-x", 10, "/var/lib/libvirt/images/base-debian-13.qcow2", "qcow2")) + if err != nil { + t.Fatalf("buildVolumeXML() error = %v", err) + } + var v volumeXML + if err := xml.Unmarshal([]byte(raw), &v); err != nil { + t.Fatalf("output is not well-formed XML: %v", err) + } + if v.Type != "file" || v.Name != "proxy-x.qcow2" { + t.Errorf("type/name = %s/%s", v.Type, v.Name) + } + if v.Capacity.Unit != "bytes" || v.Capacity.Value != 10*bytesPerGiB { + t.Errorf("capacity = %d %s, want %d bytes", v.Capacity.Value, v.Capacity.Unit, 10*bytesPerGiB) + } + if v.Target.Format.Type != "qcow2" { + t.Errorf("format = %q", v.Target.Format.Type) + } + if v.BackingStore == nil { + t.Fatalf("backingStore missing:\n%s", raw) + } + if v.BackingStore.Path != "/var/lib/libvirt/images/base-debian-13.qcow2" || v.BackingStore.Format.Type != "qcow2" { + t.Errorf("backingStore = %+v", v.BackingStore) + } +} + +func TestBuildVolumeXML_rawHasNoBackingStore(t *testing.T) { + t.Parallel() + raw, err := buildVolumeXML(rawVolume("proxy-x-seed.iso", 372736)) + if err != nil { + t.Fatalf("buildVolumeXML() error = %v", err) + } + if strings.Contains(raw, "backingStore") { + t.Errorf("raw volume must not declare a backing store:\n%s", raw) + } + var v volumeXML + if err := xml.Unmarshal([]byte(raw), &v); err != nil { + t.Fatalf("output is not well-formed XML: %v", err) + } + if v.Name != "proxy-x-seed.iso" || v.Capacity.Value != 372736 || v.Target.Format.Type != "raw" { + t.Errorf("volume = %+v", v) + } +} -- 2.54.0 From 6322184d50541358af2db34e64a6ea1bff8d2e30 Mon Sep 17 00:00:00 2001 From: kacerr Date: Sat, 5 Sep 2026 03:27:57 +0000 Subject: [PATCH 04/28] Add KVM provider transport: SSH-forwarded libvirt connections and error classification Per-host connection pool over SSH with the libvirt unix socket forwarded through the session, lazy dial and redial after disconnect, context cancellation for go-libvirt's context-less RPCs, handshake deadline and SSH keepalives. Error classification onto the provider taxonomy. Tested with a fake dialer and an in-process SSH server that forwards unix sockets. Step 3 of docs/plans/2026-09-04-1404-kvm-provider.md. Co-Authored-By: Claude Fable 5.1 --- .../2026-09-04-1404-kvm-provider.md | 54 +- go.mod | 3 +- go.sum | 2 + internal/provider/kvm/conn.go | 301 +++++++++ internal/provider/kvm/conn_test.go | 569 ++++++++++++++++++ internal/provider/kvm/errors.go | 65 ++ internal/provider/kvm/errors_test.go | 67 +++ 7 files changed, 1059 insertions(+), 2 deletions(-) create mode 100644 internal/provider/kvm/conn.go create mode 100644 internal/provider/kvm/conn_test.go create mode 100644 internal/provider/kvm/errors.go create mode 100644 internal/provider/kvm/errors_test.go diff --git a/docs/plans-executions/2026-09-04-1404-kvm-provider.md b/docs/plans-executions/2026-09-04-1404-kvm-provider.md index b630134..533ebf8 100644 --- a/docs/plans-executions/2026-09-04-1404-kvm-provider.md +++ b/docs/plans-executions/2026-09-04-1404-kvm-provider.md @@ -5,7 +5,7 @@ Plan: [2026-09-04-1404-kvm-provider.md](../plans/2026-09-04-1404-kvm-provider.md - [x] Step 0 — Save plan + branch `feat/kvm-provider` - [x] Step 1 — Config (`KVMConfig` & friends, `validate()` case, tests) - [x] Step 2 — Pure builders (`domainxml.go`, `volume.go`, `seed.go`, `id.go`, `metadata.go`) -- [ ] Step 3 — Transport + errors (`conn.go`, `errors.go`) +- [x] Step 3 — Transport + errors (`conn.go`, `errors.go`) - [ ] Step 3b — Image cache (`image.go`) - [ ] Step 4 — Provider methods (`kvm.go`) over a fake API - [ ] Step 5 — Wiring, manifests, samples, docs, CHANGELOG @@ -99,3 +99,55 @@ instead of mutating the config in place, so the registry's shared `qemu-guest-agent` via `packages:` + `runcmd`, which means the first boot needs network access to the distro mirrors; a Proxy on an isolated bridge would still work (ARP discovery), just without the agent. + +## Step 3 — Transport + errors + +`conn.go` holds the `libvirtAPI` seam (17 methods, signatures as in the +plan), a per-host connection pool, and the SSH dialer; `errors.go` holds +`classify`, `isLibvirtError`, `hasCode` and `logAPIError`. Both are +unit-tested; the whole package still runs in about a second under `-race`. + +The one thing the plan did not anticipate: **go-libvirt RPCs take no +context and never time out**. A hung hypervisor would pin a reconcile worker +forever. So the pool exposes `call(ctx, host, fn)` rather than handing out +raw clients: `fn` runs in a goroutine, and if `ctx` ends first the pool +closes the SSH connection out from under it (which makes the blocked RPC +return) and reports `ctx.Err()`. The same helper (`awaitOrCancel`) bounds +`lv.Connect()` during dial. Two more liveness measures for the same reason: +the SSH handshake gets a deadline (ctx's, or 30 s), and each connection runs +an OpenSSH `keepalive@openssh.com` loop every 15 s that closes the client +when a request errors or goes unanswered for one interval. + +Connection-drop policy in `call`: any error that is *not* a `libvirt.Error` +(so io.EOF, "connection closed", SSH failures) drops the cached connection +and the next call redials; a daemon-reported `libvirt.Error` keeps it. Dials +are serialised per host (concurrent Creates on one host share a dial) but +hosts don't block each other. + +As Step 1 noted, the dialer uses `libvirt.NewWithDialer` with a +`socket.Dialer` that wraps `ssh.Client.Dial("unix", socket)` rather than the +deprecated `New(net.Conn)`. + +Tests: the pool is exercised through a fake dialer (reuse, redial after +`Disconnected`, drop on transport error, keep on libvirt error, cancel +releases a blocked call, concurrent callers share one dial, hosts +independent). The SSH layer is tested against an in-process +`x/crypto/ssh` server that serves `direct-streamlocal@openssh.com` channels +by dialing the requested unix path (an echo socket standing in for +`libvirt-sock`), with keys generated per test and a real `known_hosts` file +written via `knownhosts.Line`. Covered: forward round-trip and the socket +path the server saw, unknown host key rejected with `knownhosts.KeyError`, +`insecureIgnoreHostKey`, wrong client key, cancelled ctx, missing key or +known_hosts file, and keepalive both closing a stalled client and leaving a +healthy one alone. The libvirt handshake (`Connect()`) itself is *not* +covered here — faking libvirt's XDR RPC just for that is not worth it; the +Step 6 integration test covers it against a real host. + +`go mod tidy` promoted `go-libvirt` and `golang.org/x/crypto` to direct +dependencies. + +Worth noting: `classify` maps `ErrOperationInvalid` (e.g. destroying a +domain that is already off) to Transient; Delete in Step 4 must check that +code itself via `hasCode` and treat it as success rather than relying on +the classifier. Nothing maps to `ErrQuotaExceeded` — libvirt has no quota +concept; an out-of-space pool surfaces as a Transient `ErrInternalError`. diff --git a/go.mod b/go.mod index de8c5e3..8e6cf30 100644 --- a/go.mod +++ b/go.mod @@ -4,6 +4,7 @@ go 1.26.0 require ( cloud.google.com/go/compute v1.65.0 + github.com/digitalocean/go-libvirt v0.0.0-20260814190004-1a83157e1858 github.com/go-logr/logr v1.4.4 github.com/kdomanski/iso9660 v0.4.0 github.com/onsi/ginkgo/v2 v2.27.4 @@ -16,6 +17,7 @@ require ( go.opentelemetry.io/otel/exporters/stdout/stdouttrace v1.45.0 go.opentelemetry.io/otel/sdk v1.45.0 go.opentelemetry.io/otel/trace v1.45.0 + golang.org/x/crypto v0.54.0 google.golang.org/api v0.292.0 google.golang.org/protobuf v1.36.12-0.20260120151049-f2248ac996af k8s.io/api v0.36.0 @@ -81,7 +83,6 @@ require ( go.uber.org/zap v1.27.1 // indirect go.yaml.in/yaml/v2 v2.4.3 // indirect go.yaml.in/yaml/v3 v3.0.4 // indirect - golang.org/x/crypto v0.54.0 // indirect golang.org/x/exp v0.0.0-20251219203646-944ab1f22d93 // indirect golang.org/x/mod v0.37.0 // indirect golang.org/x/net v0.57.0 // indirect diff --git a/go.sum b/go.sum index 807f975..b0c2886 100644 --- a/go.sum +++ b/go.sum @@ -28,6 +28,8 @@ github.com/davecgh/go-spew v1.1.0/go.mod h1:J7Y8YcW2NihsgmVo/mv3lAwl/skON4iLHjSs github.com/davecgh/go-spew v1.1.1/go.mod h1:J7Y8YcW2NihsgmVo/mv3lAwl/skON4iLHjSsI+c5H38= github.com/davecgh/go-spew v1.1.2-0.20180830191138-d8f796af33cc h1:U9qPSI2PIWSS1VwoXQT9A3Wy9MM3WgvqSxFWenqJduM= github.com/davecgh/go-spew v1.1.2-0.20180830191138-d8f796af33cc/go.mod h1:J7Y8YcW2NihsgmVo/mv3lAwl/skON4iLHjSsI+c5H38= +github.com/digitalocean/go-libvirt v0.0.0-20260814190004-1a83157e1858 h1:8xCFt73OddCD5WXxoYidJBWp3bJMC+CpE7Zwf5tcwk8= +github.com/digitalocean/go-libvirt v0.0.0-20260814190004-1a83157e1858/go.mod h1:qb0Ofa71d3oXARQf633h2tNaeBxLsVxuDp+jcsVO2+4= github.com/emicklei/go-restful/v3 v3.13.0 h1:C4Bl2xDndpU6nJ4bc1jXd+uTmYPVUwkD6bFY/oTyCes= github.com/emicklei/go-restful/v3 v3.13.0/go.mod h1:6n3XBCmQQb25CM2LCACGz8ukIrRry+4bhvbpWn3mrbc= github.com/evanphx/json-patch v0.5.2 h1:xVCHIVMUu1wtM/VkR9jVZ45N3FhZfYMMYGorLCR8P3k= diff --git a/internal/provider/kvm/conn.go b/internal/provider/kvm/conn.go new file mode 100644 index 0000000..b0034b0 --- /dev/null +++ b/internal/provider/kvm/conn.go @@ -0,0 +1,301 @@ +package kvm + +import ( + "context" + "fmt" + "io" + "net" + "os" + "sync" + "time" + + libvirt "github.com/digitalocean/go-libvirt" + "golang.org/x/crypto/ssh" + "golang.org/x/crypto/ssh/knownhosts" + + "gitea.home.hrajfrisbee.cz/kacerr/egress-proxies-operator/internal/provider" +) + +const ( + defaultHandshakeTimeout = 30 * time.Second + keepaliveInterval = 15 * time.Second +) + +// libvirtAPI is the slice of *libvirt.Libvirt the provider uses, and the +// test seam: the fake in kvm_test.go implements exactly this. +type libvirtAPI interface { + DomainLookupByName(name string) (libvirt.Domain, error) + DomainDefineXML(xml string) (libvirt.Domain, error) + DomainCreate(d libvirt.Domain) error + DomainDestroy(d libvirt.Domain) error + DomainUndefineFlags(d libvirt.Domain, flags libvirt.DomainUndefineFlagsValues) error + DomainGetState(d libvirt.Domain, flags uint32) (state int32, reason int32, err error) + DomainGetMetadata(d libvirt.Domain, typ int32, uri libvirt.OptString, flags libvirt.DomainModificationImpact) (string, error) + DomainInterfaceAddresses(d libvirt.Domain, source uint32, flags uint32) ([]libvirt.DomainInterface, error) + ConnectListAllDomains(needResults int32, flags libvirt.ConnectListAllDomainsFlags) ([]libvirt.Domain, uint32, error) + StoragePoolLookupByName(name string) (libvirt.StoragePool, error) + StoragePoolRefresh(pool libvirt.StoragePool, flags uint32) error + StorageVolLookupByName(pool libvirt.StoragePool, name string) (libvirt.StorageVol, error) + StorageVolGetPath(vol libvirt.StorageVol) (string, error) + StorageVolCreateXML(pool libvirt.StoragePool, xml string, flags libvirt.StorageVolCreateFlags) (libvirt.StorageVol, error) + StorageVolUpload(vol libvirt.StorageVol, r io.Reader, offset, length uint64, flags libvirt.StorageVolUploadFlags) error + StorageVolDownload(vol libvirt.StorageVol, w io.Writer, offset, length uint64, flags libvirt.StorageVolDownloadFlags) error + StorageVolDelete(vol libvirt.StorageVol, flags libvirt.StorageVolDeleteFlags) error +} + +// hostClient is one live connection to a hypervisor. +type hostClient struct { + api libvirtAPI + // done is closed once the connection is known to be lost. A nil channel + // never fires, which suits fakes. + done <-chan struct{} + close func() +} + +func (c *hostClient) alive() bool { + select { + case <-c.done: + return false + default: + return true + } +} + +// hostDialer opens a connection to one host. The SSH implementation is the +// real one; tests inject their own. +type hostDialer func(ctx context.Context, host provider.KVMHost) (*hostClient, error) + +// hostPool caches one connection per host, dialing lazily and redialing +// after the connection drops. Dials for different hosts don't block each +// other; concurrent callers for the same host share a single dial. +type hostPool struct { + dial hostDialer + + mu sync.Mutex + entries map[string]*hostEntry +} + +type hostEntry struct { + mu sync.Mutex + client *hostClient +} + +func newHostPool(dial hostDialer) *hostPool { + return &hostPool{dial: dial, entries: make(map[string]*hostEntry)} +} + +func (p *hostPool) entry(name string) *hostEntry { + p.mu.Lock() + defer p.mu.Unlock() + e, ok := p.entries[name] + if !ok { + e = &hostEntry{} + p.entries[name] = e + } + return e +} + +func (p *hostPool) get(ctx context.Context, host provider.KVMHost) (*hostClient, error) { + e := p.entry(host.Name) + e.mu.Lock() + defer e.mu.Unlock() + if e.client != nil { + if e.client.alive() { + return e.client, nil + } + e.client.close() + e.client = nil + } + c, err := p.dial(ctx, host) + if err != nil { + return nil, fmt.Errorf("connecting to host %s (%s): %w", host.Name, host.Address, err) + } + e.client = c + return c, nil +} + +// drop closes and forgets c if it is still the cached client for host, so a +// connection that was already replaced by a redial is left alone. +func (p *hostPool) drop(host string, c *hostClient) { + e := p.entry(host) + e.mu.Lock() + defer e.mu.Unlock() + if e.client == c { + e.client = nil + } + c.close() +} + +// call runs fn against host's connection. go-libvirt RPCs take no context +// and block until the daemon answers, so when ctx ends first the connection +// is closed — which unblocks fn with an error — and ctx.Err() is returned. +// A transport error (anything that is not a libvirt.Error) also drops the +// cached connection so the next call redials instead of reusing a dead +// link. +func (p *hostPool) call(ctx context.Context, host provider.KVMHost, fn func(api libvirtAPI) error) error { + c, err := p.get(ctx, host) + if err != nil { + return err + } + err = awaitOrCancel(ctx, func() error { return fn(c.api) }, func() { p.drop(host.Name, c) }) + if err != nil && !isLibvirtError(err) { + p.drop(host.Name, c) + } + return err +} + +// awaitOrCancel runs fn in the background and returns its result, unless ctx +// ends first: then abort is invoked (it must make fn return eventually, e.g. +// by closing the connection fn is blocked on) and ctx.Err() is returned. The +// abandoned fn's result is discarded. +func awaitOrCancel(ctx context.Context, fn func() error, abort func()) error { + errc := make(chan error, 1) + go func() { errc <- fn() }() + select { + case err := <-errc: + return err + case <-ctx.Done(): + abort() + return ctx.Err() + } +} + +// sshAuth holds the pieces of the SSH client config shared by every host: +// the private key and the host-key policy, both loaded once at startup so a +// missing or unreadable file fails the operator fast rather than on the +// first Create. +type sshAuth struct { + signer ssh.Signer + hostKey ssh.HostKeyCallback +} + +func newSSHAuth(cfg provider.KVMConfig) (*sshAuth, error) { + key, err := os.ReadFile(cfg.SSHKeyFile) + if err != nil { + return nil, fmt.Errorf("reading kvm.sshKeyFile: %w", err) + } + signer, err := ssh.ParsePrivateKey(key) + if err != nil { + return nil, fmt.Errorf("parsing kvm.sshKeyFile: %w", err) + } + hostKey := ssh.InsecureIgnoreHostKey() //nolint:gosec // opt-in via config, documented as insecure + if !cfg.InsecureIgnoreHostKey { + hostKey, err = knownhosts.New(cfg.KnownHostsFile) + if err != nil { + return nil, fmt.Errorf("loading kvm.knownHostsFile: %w", err) + } + } + return &sshAuth{signer: signer, hostKey: hostKey}, nil +} + +// dialer returns the production hostDialer: SSH to the host, forward its +// libvirt unix socket through the connection, and open a libvirt session +// over that. +func (a *sshAuth) dialer() hostDialer { + return func(ctx context.Context, host provider.KVMHost) (*hostClient, error) { + client, err := a.dialSSH(ctx, host) + if err != nil { + return nil, err + } + hc, err := connectLibvirt(ctx, client, host.Socket) + if err != nil { + client.Close() + return nil, err + } + return hc, nil + } +} + +// dialSSH establishes the SSH connection. The handshake is bounded by ctx's +// deadline, or defaultHandshakeTimeout when ctx has none, since a +// half-open hypervisor must not pin a reconcile worker indefinitely. +func (a *sshAuth) dialSSH(ctx context.Context, host provider.KVMHost) (*ssh.Client, error) { + deadline, ok := ctx.Deadline() + if !ok { + deadline = time.Now().Add(defaultHandshakeTimeout) + } + var d net.Dialer + nc, err := d.DialContext(ctx, "tcp", host.Address) + if err != nil { + return nil, fmt.Errorf("ssh dial: %w", err) + } + _ = nc.SetDeadline(deadline) + conn, chans, reqs, err := ssh.NewClientConn(nc, host.Address, &ssh.ClientConfig{ + User: host.User, + Auth: []ssh.AuthMethod{ssh.PublicKeys(a.signer)}, + HostKeyCallback: a.hostKey, + }) + if err != nil { + nc.Close() + return nil, fmt.Errorf("ssh handshake: %w", err) + } + _ = nc.SetDeadline(time.Time{}) + return ssh.NewClient(conn, chans, reqs), nil +} + +// unixForward adapts an SSH client to go-libvirt's socket.Dialer: each Dial +// opens a direct-streamlocal channel to the libvirt socket on the host. +type unixForward struct { + client *ssh.Client + path string +} + +func (u unixForward) Dial() (net.Conn, error) { + return u.client.Dial("unix", u.path) +} + +// connectLibvirt opens the libvirt session over the SSH client and starts a +// keepalive loop so a dead link is noticed within roughly two intervals +// rather than at the TCP stack's leisure. Closing the returned client tears +// down the SSH connection, which in turn ends the libvirt session. +func connectLibvirt(ctx context.Context, client *ssh.Client, socket string) (*hostClient, error) { + lv := libvirt.NewWithDialer(unixForward{client: client, path: socket}) + if err := awaitOrCancel(ctx, lv.Connect, func() { client.Close() }); err != nil { + return nil, fmt.Errorf("libvirt connect via %s: %w", socket, err) + } + stop := make(chan struct{}) + var once sync.Once + go keepalive(client, keepaliveInterval, stop) + return &hostClient{ + api: lv, + done: lv.Disconnected(), + close: func() { + once.Do(func() { + close(stop) + client.Close() + }) + }, + }, nil +} + +// keepalive sends OpenSSH keepalive requests until stop closes; a failed or +// unanswered request closes the client, which surfaces as Disconnected on +// the libvirt side and makes the pool redial. +func keepalive(client *ssh.Client, interval time.Duration, stop <-chan struct{}) { + t := time.NewTicker(interval) + defer t.Stop() + for { + select { + case <-stop: + return + case <-t.C: + } + replied := make(chan error, 1) + go func() { + _, _, err := client.SendRequest("keepalive@openssh.com", true, nil) + replied <- err + }() + select { + case <-stop: + return + case err := <-replied: + if err != nil { + client.Close() + return + } + case <-time.After(interval): + client.Close() + return + } + } +} diff --git a/internal/provider/kvm/conn_test.go b/internal/provider/kvm/conn_test.go new file mode 100644 index 0000000..732413b --- /dev/null +++ b/internal/provider/kvm/conn_test.go @@ -0,0 +1,569 @@ +package kvm + +import ( + "bytes" + "context" + "crypto/ed25519" + "crypto/rand" + "encoding/pem" + "errors" + "io" + "net" + "os" + "path/filepath" + "strings" + "sync" + "sync/atomic" + "testing" + "time" + + libvirt "github.com/digitalocean/go-libvirt" + "golang.org/x/crypto/ssh" + "golang.org/x/crypto/ssh/knownhosts" + + "gitea.home.hrajfrisbee.cz/kacerr/egress-proxies-operator/internal/provider" +) + +// --- hostPool over a fake dialer --- + +type fakeClient struct { + done chan struct{} + closed atomic.Bool +} + +func (c *fakeClient) close() { + if c.closed.CompareAndSwap(false, true) { + close(c.done) + } +} + +type fakeDialer struct { + mu sync.Mutex + dials int + err error + delay time.Duration + clients []*fakeClient +} + +func (d *fakeDialer) dial(ctx context.Context, _ provider.KVMHost) (*hostClient, error) { + if d.delay > 0 { + select { + case <-time.After(d.delay): + case <-ctx.Done(): + return nil, ctx.Err() + } + } + d.mu.Lock() + defer d.mu.Unlock() + d.dials++ + if d.err != nil { + return nil, d.err + } + fc := &fakeClient{done: make(chan struct{})} + d.clients = append(d.clients, fc) + return &hostClient{api: nil, done: fc.done, close: fc.close}, nil +} + +func (d *fakeDialer) dialCount() int { + d.mu.Lock() + defer d.mu.Unlock() + return d.dials +} + +var testHost = provider.KVMHost{Name: "nas1", Address: "nas1:22", User: "virt", Socket: defaultSocket, Pool: "default", Bridge: "br0"} + +func noop(libvirtAPI) error { return nil } + +func TestHostPool_call_reusesConnection(t *testing.T) { + t.Parallel() + d := &fakeDialer{} + p := newHostPool(d.dial) + for range 3 { + if err := p.call(context.Background(), testHost, noop); err != nil { + t.Fatalf("call() error = %v", err) + } + } + if d.dialCount() != 1 { + t.Errorf("dials = %d, want 1", d.dialCount()) + } +} + +func TestHostPool_call_redialsAfterDisconnect(t *testing.T) { + t.Parallel() + d := &fakeDialer{} + p := newHostPool(d.dial) + if err := p.call(context.Background(), testHost, noop); err != nil { + t.Fatalf("call() error = %v", err) + } + d.clients[0].close() + if err := p.call(context.Background(), testHost, noop); err != nil { + t.Fatalf("second call() error = %v", err) + } + if d.dialCount() != 2 { + t.Errorf("dials = %d, want 2 after the first connection dropped", d.dialCount()) + } +} + +func TestHostPool_call_transportErrorDropsConnection(t *testing.T) { + t.Parallel() + d := &fakeDialer{} + p := newHostPool(d.dial) + err := p.call(context.Background(), testHost, func(libvirtAPI) error { return io.ErrUnexpectedEOF }) + if !errors.Is(err, io.ErrUnexpectedEOF) { + t.Fatalf("call() error = %v, want the transport error passed through", err) + } + if !d.clients[0].closed.Load() { + t.Error("connection not closed after a transport error") + } + if err := p.call(context.Background(), testHost, noop); err != nil { + t.Fatalf("call() after transport error = %v", err) + } + if d.dialCount() != 2 { + t.Errorf("dials = %d, want 2", d.dialCount()) + } +} + +func TestHostPool_call_libvirtErrorKeepsConnection(t *testing.T) { + t.Parallel() + d := &fakeDialer{} + p := newHostPool(d.dial) + err := p.call(context.Background(), testHost, func(libvirtAPI) error { return lvErr(libvirt.ErrNoDomain) }) + if !hasCode(err, libvirt.ErrNoDomain) { + t.Fatalf("call() error = %v, want the libvirt error passed through", err) + } + if err := p.call(context.Background(), testHost, noop); err != nil { + t.Fatalf("call() error = %v", err) + } + if d.dialCount() != 1 { + t.Errorf("dials = %d, want 1: a daemon-reported error must not cost the connection", d.dialCount()) + } + if d.clients[0].closed.Load() { + t.Error("connection closed after a libvirt error") + } +} + +func TestHostPool_call_dialErrorPropagates(t *testing.T) { + t.Parallel() + d := &fakeDialer{err: errors.New("ssh: handshake failed")} + p := newHostPool(d.dial) + err := p.call(context.Background(), testHost, noop) + if err == nil || !strings.Contains(err.Error(), "nas1") || !strings.Contains(err.Error(), "handshake failed") { + t.Errorf("call() error = %v, want the dial error naming the host", err) + } +} + +func TestHostPool_call_cancelAbandonsBlockedCall(t *testing.T) { + t.Parallel() + d := &fakeDialer{} + p := newHostPool(d.dial) + ctx, cancel := context.WithCancel(context.Background()) + released := make(chan struct{}) + go func() { + time.Sleep(20 * time.Millisecond) + cancel() + }() + err := p.call(ctx, testHost, func(libvirtAPI) error { + // Simulates an RPC stuck on a hung hypervisor: it only returns once + // the pool closes the connection out from under it. + <-d.clients[0].done + close(released) + return io.EOF + }) + if !errors.Is(err, context.Canceled) { + t.Fatalf("call() error = %v, want context.Canceled", err) + } + select { + case <-released: + case <-time.After(time.Second): + t.Fatal("blocked call was not released by closing the connection") + } + if err := p.call(context.Background(), testHost, noop); err != nil { + t.Fatalf("call() after cancel = %v", err) + } + if d.dialCount() != 2 { + t.Errorf("dials = %d, want 2", d.dialCount()) + } +} + +func TestHostPool_call_concurrentCallersShareOneDial(t *testing.T) { + t.Parallel() + d := &fakeDialer{delay: 30 * time.Millisecond} + p := newHostPool(d.dial) + var wg sync.WaitGroup + for range 8 { + wg.Go(func() { + if err := p.call(context.Background(), testHost, noop); err != nil { + t.Errorf("call() error = %v", err) + } + }) + } + wg.Wait() + if d.dialCount() != 1 { + t.Errorf("dials = %d, want 1", d.dialCount()) + } +} + +func TestHostPool_call_hostsAreIndependent(t *testing.T) { + t.Parallel() + d := &fakeDialer{} + p := newHostPool(d.dial) + other := testHost + other.Name = "lab2" + for _, h := range []provider.KVMHost{testHost, other, testHost, other} { + if err := p.call(context.Background(), h, noop); err != nil { + t.Fatalf("call(%s) error = %v", h.Name, err) + } + } + if d.dialCount() != 2 { + t.Errorf("dials = %d, want one per host", d.dialCount()) + } +} + +// --- SSH dial against an in-process server --- + +func newKeyPair(t *testing.T) (ssh.Signer, []byte) { + t.Helper() + _, priv, err := ed25519.GenerateKey(rand.Reader) + if err != nil { + t.Fatal(err) + } + signer, err := ssh.NewSignerFromKey(priv) + if err != nil { + t.Fatal(err) + } + block, err := ssh.MarshalPrivateKey(priv, "") + if err != nil { + t.Fatal(err) + } + return signer, pem.EncodeToMemory(block) +} + +// testSSHServer accepts one public key and serves +// direct-streamlocal@openssh.com channels by dialing the requested unix +// path, which is exactly what OpenSSH does for ssh.Client.Dial("unix"). +type testSSHServer struct { + addr string + hostKey ssh.PublicKey + + mu sync.Mutex + paths []string +} + +func startSSHServer(t *testing.T, clientKey ssh.PublicKey, answerGlobalRequests bool) *testSSHServer { + t.Helper() + hostSigner, _ := newKeyPair(t) + cfg := &ssh.ServerConfig{ + PublicKeyCallback: func(_ ssh.ConnMetadata, key ssh.PublicKey) (*ssh.Permissions, error) { + if bytes.Equal(key.Marshal(), clientKey.Marshal()) { + return nil, nil + } + return nil, errors.New("unknown public key") + }, + } + cfg.AddHostKey(hostSigner) + ln, err := net.Listen("tcp", "127.0.0.1:0") + if err != nil { + t.Fatal(err) + } + t.Cleanup(func() { ln.Close() }) + s := &testSSHServer{addr: ln.Addr().String(), hostKey: hostSigner.PublicKey()} + go func() { + for { + nc, err := ln.Accept() + if err != nil { + return + } + go s.serve(nc, cfg, answerGlobalRequests) + } + }() + return s +} + +func (s *testSSHServer) serve(nc net.Conn, cfg *ssh.ServerConfig, answerGlobalRequests bool) { + _, chans, reqs, err := ssh.NewServerConn(nc, cfg) + if err != nil { + return + } + if answerGlobalRequests { + go ssh.DiscardRequests(reqs) + } else { + go func() { + for range reqs { //nolint:revive // deliberately never replies, to stall keepalives + } + }() + } + for newCh := range chans { + if newCh.ChannelType() != "direct-streamlocal@openssh.com" { + _ = newCh.Reject(ssh.UnknownChannelType, "test server only forwards unix sockets") + continue + } + var payload struct { + Path string + Reserved0 string + Reserved1 uint32 + } + if err := ssh.Unmarshal(newCh.ExtraData(), &payload); err != nil { + _ = newCh.Reject(ssh.ConnectionFailed, err.Error()) + continue + } + s.mu.Lock() + s.paths = append(s.paths, payload.Path) + s.mu.Unlock() + target, err := net.Dial("unix", payload.Path) + if err != nil { + _ = newCh.Reject(ssh.ConnectionFailed, err.Error()) + continue + } + ch, chReqs, err := newCh.Accept() + if err != nil { + target.Close() + continue + } + go ssh.DiscardRequests(chReqs) + go func() { _, _ = io.Copy(ch, target); ch.Close() }() + go func() { _, _ = io.Copy(target, ch); target.Close() }() + } +} + +func (s *testSSHServer) requestedPaths() []string { + s.mu.Lock() + defer s.mu.Unlock() + return append([]string(nil), s.paths...) +} + +// startEchoSocket serves a unix socket that echoes everything back, standing +// in for libvirt-sock. +func startEchoSocket(t *testing.T) string { + t.Helper() + path := filepath.Join(t.TempDir(), "libvirt-sock") + ln, err := net.Listen("unix", path) + if err != nil { + t.Fatal(err) + } + t.Cleanup(func() { ln.Close() }) + go func() { + for { + c, err := ln.Accept() + if err != nil { + return + } + go func() { _, _ = io.Copy(c, c); c.Close() }() + } + }() + return path +} + +type sshFixture struct { + server *testSSHServer + cfg provider.KVMConfig + host provider.KVMHost + keyFile string +} + +func newSSHFixture(t *testing.T, answerGlobalRequests bool) *sshFixture { + t.Helper() + clientSigner, clientPEM := newKeyPair(t) + server := startSSHServer(t, clientSigner.PublicKey(), answerGlobalRequests) + dir := t.TempDir() + keyFile := filepath.Join(dir, "id_ed25519") + if err := os.WriteFile(keyFile, clientPEM, 0o600); err != nil { + t.Fatal(err) + } + khFile := filepath.Join(dir, "known_hosts") + line := knownhosts.Line([]string{server.addr}, server.hostKey) + if err := os.WriteFile(khFile, []byte(line+"\n"), 0o600); err != nil { + t.Fatal(err) + } + return &sshFixture{ + server: server, + keyFile: keyFile, + cfg: provider.KVMConfig{SSHKeyFile: keyFile, KnownHostsFile: khFile}, + host: provider.KVMHost{Name: "nas1", Address: server.addr, User: "virt", Socket: startEchoSocket(t)}, + } +} + +func TestSSHAuth_dialSSH_forwardsUnixSocket(t *testing.T) { + t.Parallel() + f := newSSHFixture(t, true) + auth, err := newSSHAuth(f.cfg) + if err != nil { + t.Fatalf("newSSHAuth() error = %v", err) + } + ctx, cancel := context.WithTimeout(context.Background(), 5*time.Second) + defer cancel() + client, err := auth.dialSSH(ctx, f.host) + if err != nil { + t.Fatalf("dialSSH() error = %v", err) + } + defer client.Close() + + conn, err := unixForward{client: client, path: f.host.Socket}.Dial() + if err != nil { + t.Fatalf("unixForward.Dial() error = %v", err) + } + defer conn.Close() + if _, err := conn.Write([]byte("ping")); err != nil { + t.Fatal(err) + } + buf := make([]byte, 4) + if _, err := io.ReadFull(conn, buf); err != nil || string(buf) != "ping" { + t.Fatalf("echo through the forward = %q, %v; want ping", buf, err) + } + if paths := f.server.requestedPaths(); len(paths) != 1 || paths[0] != f.host.Socket { + t.Errorf("server saw forward requests %v, want [%s]", paths, f.host.Socket) + } +} + +func TestSSHAuth_dialSSH_rejectsUnknownHostKey(t *testing.T) { + t.Parallel() + f := newSSHFixture(t, true) + impostor, _ := newKeyPair(t) + line := knownhosts.Line([]string{f.server.addr}, impostor.PublicKey()) + if err := os.WriteFile(f.cfg.KnownHostsFile, []byte(line+"\n"), 0o600); err != nil { + t.Fatal(err) + } + auth, err := newSSHAuth(f.cfg) + if err != nil { + t.Fatalf("newSSHAuth() error = %v", err) + } + client, err := auth.dialSSH(context.Background(), f.host) + if err == nil { + client.Close() + t.Fatal("dialSSH() succeeded against a host whose key is not in known_hosts") + } + var keyErr *knownhosts.KeyError + if !errors.As(err, &keyErr) { + t.Errorf("dialSSH() error = %v, want a knownhosts.KeyError", err) + } +} + +func TestSSHAuth_dialSSH_insecureIgnoreHostKey(t *testing.T) { + t.Parallel() + f := newSSHFixture(t, true) + f.cfg.KnownHostsFile = "" + f.cfg.InsecureIgnoreHostKey = true + auth, err := newSSHAuth(f.cfg) + if err != nil { + t.Fatalf("newSSHAuth() error = %v", err) + } + client, err := auth.dialSSH(context.Background(), f.host) + if err != nil { + t.Fatalf("dialSSH() error = %v", err) + } + client.Close() +} + +func TestSSHAuth_dialSSH_rejectsWrongClientKey(t *testing.T) { + t.Parallel() + f := newSSHFixture(t, true) + _, otherPEM := newKeyPair(t) + if err := os.WriteFile(f.keyFile, otherPEM, 0o600); err != nil { + t.Fatal(err) + } + auth, err := newSSHAuth(f.cfg) + if err != nil { + t.Fatalf("newSSHAuth() error = %v", err) + } + client, err := auth.dialSSH(context.Background(), f.host) + if err == nil { + client.Close() + t.Fatal("dialSSH() succeeded with a key the server does not know") + } + if !strings.Contains(err.Error(), "ssh handshake") { + t.Errorf("dialSSH() error = %v, want an auth failure from the handshake", err) + } +} + +func TestSSHAuth_dialSSH_cancelledContext(t *testing.T) { + t.Parallel() + f := newSSHFixture(t, true) + auth, err := newSSHAuth(f.cfg) + if err != nil { + t.Fatalf("newSSHAuth() error = %v", err) + } + ctx, cancel := context.WithCancel(context.Background()) + cancel() + if client, err := auth.dialSSH(ctx, f.host); err == nil { + client.Close() + t.Fatal("dialSSH() succeeded with an already-cancelled context") + } +} + +func TestNewSSHAuth_missingFiles(t *testing.T) { + t.Parallel() + _, keyPEM := newKeyPair(t) + keyFile := filepath.Join(t.TempDir(), "key") + if err := os.WriteFile(keyFile, keyPEM, 0o600); err != nil { + t.Fatal(err) + } + tests := []struct { + name string + cfg provider.KVMConfig + wantSub string + }{ + {"key file missing", provider.KVMConfig{SSHKeyFile: "/nonexistent/key", InsecureIgnoreHostKey: true}, "sshKeyFile"}, + {"key file garbage", provider.KVMConfig{SSHKeyFile: keyFile + ".bad", InsecureIgnoreHostKey: true}, "sshKeyFile"}, + {"known_hosts missing", provider.KVMConfig{SSHKeyFile: keyFile, KnownHostsFile: "/nonexistent/kh"}, "knownHostsFile"}, + } + if err := os.WriteFile(keyFile+".bad", []byte("not a key"), 0o600); err != nil { + t.Fatal(err) + } + for _, tc := range tests { + t.Run(tc.name, func(t *testing.T) { + t.Parallel() + _, err := newSSHAuth(tc.cfg) + if err == nil || !strings.Contains(err.Error(), tc.wantSub) { + t.Errorf("newSSHAuth() error = %v, want one mentioning %s", err, tc.wantSub) + } + }) + } +} + +func TestKeepalive_closesClientWhenServerStalls(t *testing.T) { + t.Parallel() + f := newSSHFixture(t, false) + auth, err := newSSHAuth(f.cfg) + if err != nil { + t.Fatalf("newSSHAuth() error = %v", err) + } + client, err := auth.dialSSH(context.Background(), f.host) + if err != nil { + t.Fatalf("dialSSH() error = %v", err) + } + defer client.Close() + stop := make(chan struct{}) + defer close(stop) + go keepalive(client, 20*time.Millisecond, stop) + + closed := make(chan struct{}) + go func() { _ = client.Wait(); close(closed) }() + select { + case <-closed: + case <-time.After(2 * time.Second): + t.Fatal("client still open after keepalives went unanswered") + } +} + +func TestKeepalive_keepsHealthyClientOpen(t *testing.T) { + t.Parallel() + f := newSSHFixture(t, true) + auth, err := newSSHAuth(f.cfg) + if err != nil { + t.Fatalf("newSSHAuth() error = %v", err) + } + client, err := auth.dialSSH(context.Background(), f.host) + if err != nil { + t.Fatalf("dialSSH() error = %v", err) + } + defer client.Close() + stop := make(chan struct{}) + defer close(stop) + go keepalive(client, 10*time.Millisecond, stop) + time.Sleep(60 * time.Millisecond) + fwd := unixForward{client: client, path: f.host.Socket} + conn, err := fwd.Dial() + if err != nil { + t.Fatalf("forward after several keepalives failed: %v", err) + } + conn.Close() +} diff --git a/internal/provider/kvm/errors.go b/internal/provider/kvm/errors.go new file mode 100644 index 0000000..cacf2f0 --- /dev/null +++ b/internal/provider/kvm/errors.go @@ -0,0 +1,65 @@ +package kvm + +import ( + "errors" + + libvirt "github.com/digitalocean/go-libvirt" + "github.com/go-logr/logr" + + "gitea.home.hrajfrisbee.cz/kacerr/egress-proxies-operator/internal/provider" +) + +// classify maps a libvirt error onto the provider taxonomy. Missing domain +// or volume → NotFound; anything the host will keep rejecting (bad XML, +// unsupported config, denied access, missing pool) → Permanent; every other +// libvirt code, plus SSH/transport failures and cancellation, → Transient, +// because retrying is safer than latching Failed on an error nobody taught +// this function to recognise. libvirt exposes no quota signal, so nothing +// maps to QuotaExceeded. +func classify(err error) error { + var lerr libvirt.Error + if !errors.As(err, &lerr) { + return provider.ErrTransient + } + switch libvirt.ErrorNumber(lerr.Code) { + case libvirt.ErrNoDomain, libvirt.ErrNoStorageVol: + return provider.ErrNotFound + case libvirt.ErrNoStoragePool, + libvirt.ErrXMLError, libvirt.ErrXMLDetail, libvirt.ErrXMLInvalidSchema, + libvirt.ErrConfigUnsupported, libvirt.ErrNoSupport, + libvirt.ErrOperationUnsupported, libvirt.ErrArgumentUnsupported, + libvirt.ErrInvalidArg, + libvirt.ErrAuthFailed, libvirt.ErrAccessDenied, libvirt.ErrOperationDenied: + return provider.ErrPermanent + default: + return provider.ErrTransient + } +} + +// isLibvirtError reports whether err came back from the libvirt daemon (as +// opposed to the SSH or RPC transport), which is what decides whether the +// cached connection is still trustworthy. +func isLibvirtError(err error) bool { + var lerr libvirt.Error + return errors.As(err, &lerr) +} + +func hasCode(err error, code libvirt.ErrorNumber) bool { + var lerr libvirt.Error + return errors.As(err, &lerr) && libvirt.ErrorNumber(lerr.Code) == code +} + +// logAPIError records the raw libvirt code and message at V(1); classify +// collapses them onto the coarser taxonomy, so this is the only place the +// original detail survives. +func logAPIError(log logr.Logger, op string, err error) { + if !log.V(1).Enabled() { + return + } + kv := []any{"op", op, "error", err.Error()} + var lerr libvirt.Error + if errors.As(err, &lerr) { + kv = append(kv, "libvirtCode", lerr.Code, "libvirtMessage", lerr.Message) + } + log.V(1).Info("libvirt call failed", kv...) +} diff --git a/internal/provider/kvm/errors_test.go b/internal/provider/kvm/errors_test.go new file mode 100644 index 0000000..9369c49 --- /dev/null +++ b/internal/provider/kvm/errors_test.go @@ -0,0 +1,67 @@ +package kvm + +import ( + "context" + "errors" + "fmt" + "io" + "testing" + + libvirt "github.com/digitalocean/go-libvirt" + + "gitea.home.hrajfrisbee.cz/kacerr/egress-proxies-operator/internal/provider" +) + +func lvErr(code libvirt.ErrorNumber) error { + return libvirt.Error{Code: uint32(code), Message: code.String()} +} + +func TestClassify(t *testing.T) { + t.Parallel() + tests := []struct { + name string + err error + want error + }{ + {"no domain", lvErr(libvirt.ErrNoDomain), provider.ErrNotFound}, + {"no storage vol", lvErr(libvirt.ErrNoStorageVol), provider.ErrNotFound}, + {"no storage pool", lvErr(libvirt.ErrNoStoragePool), provider.ErrPermanent}, + {"xml error", lvErr(libvirt.ErrXMLError), provider.ErrPermanent}, + {"xml detail", lvErr(libvirt.ErrXMLDetail), provider.ErrPermanent}, + {"config unsupported", lvErr(libvirt.ErrConfigUnsupported), provider.ErrPermanent}, + {"no support", lvErr(libvirt.ErrNoSupport), provider.ErrPermanent}, + {"invalid arg", lvErr(libvirt.ErrInvalidArg), provider.ErrPermanent}, + {"auth failed", lvErr(libvirt.ErrAuthFailed), provider.ErrPermanent}, + {"access denied", lvErr(libvirt.ErrAccessDenied), provider.ErrPermanent}, + {"operation invalid", lvErr(libvirt.ErrOperationInvalid), provider.ErrTransient}, + {"operation failed", lvErr(libvirt.ErrOperationFailed), provider.ErrTransient}, + {"internal error", lvErr(libvirt.ErrInternalError), provider.ErrTransient}, + {"storage vol exists", lvErr(libvirt.ErrStorageVolExist), provider.ErrTransient}, + {"wrapped libvirt error", fmt.Errorf("defining: %w", lvErr(libvirt.ErrNoDomain)), provider.ErrNotFound}, + {"transport eof", io.EOF, provider.ErrTransient}, + {"context cancelled", context.Canceled, provider.ErrTransient}, + {"plain error", errors.New("ssh: handshake failed"), provider.ErrTransient}, + } + for _, tc := range tests { + t.Run(tc.name, func(t *testing.T) { + t.Parallel() + if got := classify(tc.err); !errors.Is(got, tc.want) { + t.Errorf("classify(%v) = %v, want %v", tc.err, got, tc.want) + } + }) + } +} + +func TestHasCode(t *testing.T) { + t.Parallel() + err := fmt.Errorf("lookup: %w", lvErr(libvirt.ErrNoDomain)) + if !hasCode(err, libvirt.ErrNoDomain) { + t.Error("hasCode(ErrNoDomain) = false through a wrap") + } + if hasCode(err, libvirt.ErrNoStorageVol) { + t.Error("hasCode(ErrNoStorageVol) = true for an ErrNoDomain error") + } + if hasCode(io.EOF, libvirt.ErrNoDomain) || isLibvirtError(io.EOF) { + t.Error("io.EOF classified as a libvirt error") + } +} -- 2.54.0 From 6dba33adf223d097d5d9149399c51330c4e5587c Mon Sep 17 00:00:00 2001 From: kacerr Date: Sat, 5 Sep 2026 03:32:39 +0000 Subject: [PATCH 05/28] Add KVM provider image cache: fetch base images into host pools on first use resolveImage maps placement.image to a configured key or URL; imageCache streams the download through StorageVolUpload with a sha256 tee, writes a marker volume when the upload is verified, and re-fetches when the marker is missing or the configured checksum changed. Errors carry taxonomy sentinels that classify now honours. Adds the in-memory fake libvirt API used from here on. Step 3b of docs/plans/2026-09-04-1404-kvm-provider.md. Co-Authored-By: Claude Fable 5.1 --- .../2026-09-04-1404-kvm-provider.md | 47 +- internal/provider/kvm/errors.go | 21 +- internal/provider/kvm/errors_test.go | 2 + internal/provider/kvm/fake_test.go | 408 ++++++++++++++++++ internal/provider/kvm/image.go | 232 ++++++++++ internal/provider/kvm/image_test.go | 385 +++++++++++++++++ 6 files changed, 1087 insertions(+), 8 deletions(-) create mode 100644 internal/provider/kvm/fake_test.go create mode 100644 internal/provider/kvm/image.go create mode 100644 internal/provider/kvm/image_test.go diff --git a/docs/plans-executions/2026-09-04-1404-kvm-provider.md b/docs/plans-executions/2026-09-04-1404-kvm-provider.md index 533ebf8..beebc58 100644 --- a/docs/plans-executions/2026-09-04-1404-kvm-provider.md +++ b/docs/plans-executions/2026-09-04-1404-kvm-provider.md @@ -6,7 +6,7 @@ Plan: [2026-09-04-1404-kvm-provider.md](../plans/2026-09-04-1404-kvm-provider.md - [x] Step 1 — Config (`KVMConfig` & friends, `validate()` case, tests) - [x] Step 2 — Pure builders (`domainxml.go`, `volume.go`, `seed.go`, `id.go`, `metadata.go`) - [x] Step 3 — Transport + errors (`conn.go`, `errors.go`) -- [ ] Step 3b — Image cache (`image.go`) +- [x] Step 3b — Image cache (`image.go`) - [ ] Step 4 — Provider methods (`kvm.go`) over a fake API - [ ] Step 5 — Wiring, manifests, samples, docs, CHANGELOG - [ ] Step 6 — Integration test against a real hypervisor @@ -151,3 +151,48 @@ domain that is already off) to Transient; Delete in Step 4 must check that code itself via `hasCode` and treat it as success rather than relying on the classifier. Nothing maps to `ErrQuotaExceeded` — libvirt has no quota concept; an out-of-space pool surfaces as a Transient `ErrInternalError`. + +## Step 3b — Image cache + +`image.go` follows the plan closely: `resolveImage` maps `placement.image` +to a configured key or a direct http(s) URL (else Permanent); +`imageCache.ensure` checks base volume + marker (+ marker content when a +checksum is configured), tears down and re-fetches when anything is off, +streams the HTTP body through `StorageVolUpload` with a `sha256` tee, +refreshes the pool, then writes the marker last. A per-`(host, volume)` +mutex serialises concurrent callers; the pool itself is the only cache +state. + +Errors from this file are pre-tagged with the taxonomy sentinel via +`fmt.Errorf("%w: …", provider.ErrPermanent, …)` and `classify` now honours +such tags before looking at libvirt codes. That keeps one `wrapErr` path in +the provider instead of special-casing image errors. + +`fake_test.go` landed with this step rather than Step 4: an in-memory +`libvirtAPI` with pools, volumes (content, format, capacity, backing path), +domains (state, metadata, per-source interface addresses), per-method error +injection (`fail` always / `failOnce`) and an `uploadHook` to break a stream +mid-way. It models the libvirt error codes the provider branches on +(`ErrStorageVolExist`, `ErrNoStorageVol`, `ErrNoDomain`, +`ErrOperationInvalid`, `ErrNoDomainMetadata`, `ErrAgentUnresponsive`). + +Tests cover every case in the plan plus: no checksum configured (marker is +still written with the computed digest; second call needs no marker read), +403/500 alongside 404/503, missing pool → Permanent, cancelled ctx → +Transient with nothing created, and different hosts fetching independently. + +One test-harness lesson worth keeping: `httptest`'s server drops +`Content-Length` and switches to chunked encoding for bodies larger than +its 2 KiB write buffer, so the fake mirror sets the header explicitly. +Real distro mirrors send it; a server that doesn't is reported as +Permanent with a message naming the header. + +Worth noting: the base volume is created with `capacity = Content-Length` +and the *image's* format (`qcow2`), and the raw bytes are uploaded over it. +That relies on libvirt's upload writing the file verbatim (which is how +`virsh vol-upload` populates volumes); the following `StoragePoolRefresh` +makes libvirt re-read the real virtual size from the qcow2 header. Step 6 +confirms this on a real host. Also, the whole download runs inside one +`pool.call`, so cancellation aborts the HTTP body (ctx-bound) and the +upload stream fails; the half-written base volume is deleted on the spot +and, failing that, on the next attempt via the missing marker. diff --git a/internal/provider/kvm/errors.go b/internal/provider/kvm/errors.go index cacf2f0..51bb20e 100644 --- a/internal/provider/kvm/errors.go +++ b/internal/provider/kvm/errors.go @@ -9,14 +9,21 @@ import ( "gitea.home.hrajfrisbee.cz/kacerr/egress-proxies-operator/internal/provider" ) -// classify maps a libvirt error onto the provider taxonomy. Missing domain -// or volume → NotFound; anything the host will keep rejecting (bad XML, -// unsupported config, denied access, missing pool) → Permanent; every other -// libvirt code, plus SSH/transport failures and cancellation, → Transient, -// because retrying is safer than latching Failed on an error nobody taught -// this function to recognise. libvirt exposes no quota signal, so nothing -// maps to QuotaExceeded. +// classify maps an error onto the provider taxonomy. Errors the package +// already tagged with a sentinel (image download failures, placement +// errors) keep their class. For libvirt errors: missing domain or volume → +// NotFound; anything the host will keep rejecting (bad XML, unsupported +// config, denied access, missing pool) → Permanent; every other libvirt +// code, plus SSH/transport failures and cancellation, → Transient, because +// retrying is safer than latching Failed on an error nobody taught this +// function to recognise. libvirt exposes no quota signal, so nothing maps +// to QuotaExceeded. func classify(err error) error { + for _, class := range []error{provider.ErrNotFound, provider.ErrQuotaExceeded, provider.ErrPermanent, provider.ErrTransient} { + if errors.Is(err, class) { + return class + } + } var lerr libvirt.Error if !errors.As(err, &lerr) { return provider.ErrTransient diff --git a/internal/provider/kvm/errors_test.go b/internal/provider/kvm/errors_test.go index 9369c49..1f00ad7 100644 --- a/internal/provider/kvm/errors_test.go +++ b/internal/provider/kvm/errors_test.go @@ -38,6 +38,8 @@ func TestClassify(t *testing.T) { {"internal error", lvErr(libvirt.ErrInternalError), provider.ErrTransient}, {"storage vol exists", lvErr(libvirt.ErrStorageVolExist), provider.ErrTransient}, {"wrapped libvirt error", fmt.Errorf("defining: %w", lvErr(libvirt.ErrNoDomain)), provider.ErrNotFound}, + {"pre-tagged permanent", fmt.Errorf("%w: image returned HTTP 404", provider.ErrPermanent), provider.ErrPermanent}, + {"pre-tagged transient wrapping libvirt not-found", fmt.Errorf("%w: %w", provider.ErrTransient, lvErr(libvirt.ErrNoDomain)), provider.ErrTransient}, {"transport eof", io.EOF, provider.ErrTransient}, {"context cancelled", context.Canceled, provider.ErrTransient}, {"plain error", errors.New("ssh: handshake failed"), provider.ErrTransient}, diff --git a/internal/provider/kvm/fake_test.go b/internal/provider/kvm/fake_test.go new file mode 100644 index 0000000..02c52e4 --- /dev/null +++ b/internal/provider/kvm/fake_test.go @@ -0,0 +1,408 @@ +package kvm + +import ( + "bytes" + "encoding/xml" + "fmt" + "io" + "slices" + "sync" + + libvirt "github.com/digitalocean/go-libvirt" +) + +// fakeAPI is an in-memory libvirtAPI: pools of volumes and a set of +// domains, with per-method error injection. It models just enough libvirt +// semantics for the provider's decisions to be observable — ErrStorageVolExist +// on duplicate create, ErrNoDomain/ErrNoStorageVol on missing lookups, +// ErrOperationInvalid on destroying a stopped domain, and so on. +type fakeAPI struct { + mu sync.Mutex + poolPath string + pools map[string]map[string]*fakeVol + domains map[string]*fakeDomain + calls []string + fail map[string]error + failOnce map[string]error + // uploadHook, when set, sees every StorageVolUpload before the fake + // reads the stream; it can return an error to simulate a broken upload. + uploadHook func(vol string, r io.Reader) error +} + +type fakeVol struct { + name string + format string + capacity int64 + backing string + data []byte +} + +type fakeDomain struct { + name string + xml string + metadata string + running bool + // addrs is what DomainInterfaceAddresses returns per source. + addrs map[uint32][]libvirt.DomainInterface +} + +func newFakeAPI(pools ...string) *fakeAPI { + f := &fakeAPI{ + poolPath: "/var/lib/libvirt/images", + pools: map[string]map[string]*fakeVol{}, + domains: map[string]*fakeDomain{}, + fail: map[string]error{}, + failOnce: map[string]error{}, + } + for _, p := range pools { + f.pools[p] = map[string]*fakeVol{} + } + return f +} + +// enter records the call and returns any injected error for method. +func (f *fakeAPI) enter(method string) error { + f.calls = append(f.calls, method) + if err, ok := f.failOnce[method]; ok { + delete(f.failOnce, method) + return err + } + return f.fail[method] +} + +func (f *fakeAPI) callCount(method string) int { + f.mu.Lock() + defer f.mu.Unlock() + n := 0 + for _, c := range f.calls { + if c == method { + n++ + } + } + return n +} + +func (f *fakeAPI) hasCall(method string) bool { return f.callCount(method) > 0 } + +func (f *fakeAPI) vol(pool, name string) *fakeVol { + f.mu.Lock() + defer f.mu.Unlock() + return f.pools[pool][name] +} + +func (f *fakeAPI) putVol(pool, name, format string, data []byte) { + f.mu.Lock() + defer f.mu.Unlock() + f.pools[pool][name] = &fakeVol{name: name, format: format, capacity: int64(len(data)), data: data} +} + +func (f *fakeAPI) volNames(pool string) []string { + f.mu.Lock() + defer f.mu.Unlock() + var out []string + for n := range f.pools[pool] { + out = append(out, n) + } + slices.Sort(out) + return out +} + +func (f *fakeAPI) domain(name string) *fakeDomain { + f.mu.Lock() + defer f.mu.Unlock() + return f.domains[name] +} + +func (f *fakeAPI) putDomain(d *fakeDomain) { + f.mu.Lock() + defer f.mu.Unlock() + f.domains[d.name] = d +} + +func lvError(code libvirt.ErrorNumber, format string, args ...any) error { + return libvirt.Error{Code: uint32(code), Message: fmt.Sprintf(format, args...)} +} + +// --- domains --- + +func (f *fakeAPI) DomainLookupByName(name string) (libvirt.Domain, error) { + f.mu.Lock() + defer f.mu.Unlock() + if err := f.enter("DomainLookupByName"); err != nil { + return libvirt.Domain{}, err + } + if _, ok := f.domains[name]; !ok { + return libvirt.Domain{}, lvError(libvirt.ErrNoDomain, "Domain not found: no domain with matching name '%s'", name) + } + return libvirt.Domain{Name: name}, nil +} + +func (f *fakeAPI) DomainDefineXML(raw string) (libvirt.Domain, error) { + f.mu.Lock() + defer f.mu.Unlock() + if err := f.enter("DomainDefineXML"); err != nil { + return libvirt.Domain{}, err + } + var d domainXML + if err := xml.Unmarshal([]byte(raw), &d); err != nil { + return libvirt.Domain{}, lvError(libvirt.ErrXMLError, "%v", err) + } + meta, err := xml.Marshal(d.Metadata.Instance) + if err != nil { + return libvirt.Domain{}, lvError(libvirt.ErrXMLError, "%v", err) + } + existing := f.domains[d.Name] + nd := &fakeDomain{name: d.Name, xml: raw, metadata: string(meta)} + if existing != nil { + nd.running = existing.running + nd.addrs = existing.addrs + } + f.domains[d.Name] = nd + return libvirt.Domain{Name: d.Name}, nil +} + +func (f *fakeAPI) DomainCreate(d libvirt.Domain) error { + f.mu.Lock() + defer f.mu.Unlock() + if err := f.enter("DomainCreate"); err != nil { + return err + } + dom, ok := f.domains[d.Name] + if !ok { + return lvError(libvirt.ErrNoDomain, "no domain %s", d.Name) + } + if dom.running { + return lvError(libvirt.ErrOperationInvalid, "domain is already running") + } + dom.running = true + return nil +} + +func (f *fakeAPI) DomainDestroy(d libvirt.Domain) error { + f.mu.Lock() + defer f.mu.Unlock() + if err := f.enter("DomainDestroy"); err != nil { + return err + } + dom, ok := f.domains[d.Name] + if !ok { + return lvError(libvirt.ErrNoDomain, "no domain %s", d.Name) + } + if !dom.running { + return lvError(libvirt.ErrOperationInvalid, "domain is not running") + } + dom.running = false + return nil +} + +func (f *fakeAPI) DomainUndefineFlags(d libvirt.Domain, _ libvirt.DomainUndefineFlagsValues) error { + f.mu.Lock() + defer f.mu.Unlock() + if err := f.enter("DomainUndefineFlags"); err != nil { + return err + } + if _, ok := f.domains[d.Name]; !ok { + return lvError(libvirt.ErrNoDomain, "no domain %s", d.Name) + } + delete(f.domains, d.Name) + return nil +} + +func (f *fakeAPI) DomainGetState(d libvirt.Domain, _ uint32) (int32, int32, error) { + f.mu.Lock() + defer f.mu.Unlock() + if err := f.enter("DomainGetState"); err != nil { + return 0, 0, err + } + dom, ok := f.domains[d.Name] + if !ok { + return 0, 0, lvError(libvirt.ErrNoDomain, "no domain %s", d.Name) + } + if dom.running { + return int32(libvirt.DomainRunning), 0, nil + } + return int32(libvirt.DomainShutoff), 0, nil +} + +func (f *fakeAPI) DomainGetMetadata(d libvirt.Domain, typ int32, uri libvirt.OptString, _ libvirt.DomainModificationImpact) (string, error) { + f.mu.Lock() + defer f.mu.Unlock() + if err := f.enter("DomainGetMetadata"); err != nil { + return "", err + } + dom, ok := f.domains[d.Name] + if !ok { + return "", lvError(libvirt.ErrNoDomain, "no domain %s", d.Name) + } + if typ != int32(libvirt.DomainMetadataElement) || len(uri) != 1 || uri[0] != metadataURI || dom.metadata == "" { + return "", lvError(libvirt.ErrNoDomainMetadata, "Requested metadata element is not present") + } + return dom.metadata, nil +} + +func (f *fakeAPI) DomainInterfaceAddresses(d libvirt.Domain, source uint32, _ uint32) ([]libvirt.DomainInterface, error) { + f.mu.Lock() + defer f.mu.Unlock() + if err := f.enter("DomainInterfaceAddresses"); err != nil { + return nil, err + } + dom, ok := f.domains[d.Name] + if !ok { + return nil, lvError(libvirt.ErrNoDomain, "no domain %s", d.Name) + } + if !dom.running { + return nil, lvError(libvirt.ErrOperationInvalid, "domain is not running") + } + if source == uint32(libvirt.DomainInterfaceAddressesSrcAgent) && dom.addrs[source] == nil { + return nil, lvError(libvirt.ErrAgentUnresponsive, "Guest agent is not responding") + } + return dom.addrs[source], nil +} + +func (f *fakeAPI) ConnectListAllDomains(_ int32, _ libvirt.ConnectListAllDomainsFlags) ([]libvirt.Domain, uint32, error) { + f.mu.Lock() + defer f.mu.Unlock() + if err := f.enter("ConnectListAllDomains"); err != nil { + return nil, 0, err + } + names := make([]string, 0, len(f.domains)) + for n := range f.domains { + names = append(names, n) + } + slices.Sort(names) + out := make([]libvirt.Domain, 0, len(names)) + for _, n := range names { + out = append(out, libvirt.Domain{Name: n}) + } + return out, uint32(len(out)), nil +} + +// --- storage --- + +func (f *fakeAPI) StoragePoolLookupByName(name string) (libvirt.StoragePool, error) { + f.mu.Lock() + defer f.mu.Unlock() + if err := f.enter("StoragePoolLookupByName"); err != nil { + return libvirt.StoragePool{}, err + } + if _, ok := f.pools[name]; !ok { + return libvirt.StoragePool{}, lvError(libvirt.ErrNoStoragePool, "Storage pool not found: no storage pool with matching name '%s'", name) + } + return libvirt.StoragePool{Name: name}, nil +} + +func (f *fakeAPI) StoragePoolRefresh(pool libvirt.StoragePool, _ uint32) error { + f.mu.Lock() + defer f.mu.Unlock() + if err := f.enter("StoragePoolRefresh"); err != nil { + return err + } + if _, ok := f.pools[pool.Name]; !ok { + return lvError(libvirt.ErrNoStoragePool, "no pool %s", pool.Name) + } + return nil +} + +func (f *fakeAPI) StorageVolLookupByName(pool libvirt.StoragePool, name string) (libvirt.StorageVol, error) { + f.mu.Lock() + defer f.mu.Unlock() + if err := f.enter("StorageVolLookupByName"); err != nil { + return libvirt.StorageVol{}, err + } + if _, ok := f.pools[pool.Name][name]; !ok { + return libvirt.StorageVol{}, lvError(libvirt.ErrNoStorageVol, "Storage volume not found: no storage vol with matching name '%s'", name) + } + return libvirt.StorageVol{Pool: pool.Name, Name: name, Key: f.poolPath + "/" + name}, nil +} + +func (f *fakeAPI) StorageVolGetPath(vol libvirt.StorageVol) (string, error) { + f.mu.Lock() + defer f.mu.Unlock() + if err := f.enter("StorageVolGetPath"); err != nil { + return "", err + } + if _, ok := f.pools[vol.Pool][vol.Name]; !ok { + return "", lvError(libvirt.ErrNoStorageVol, "no vol %s", vol.Name) + } + return f.poolPath + "/" + vol.Name, nil +} + +func (f *fakeAPI) StorageVolCreateXML(pool libvirt.StoragePool, raw string, _ libvirt.StorageVolCreateFlags) (libvirt.StorageVol, error) { + f.mu.Lock() + defer f.mu.Unlock() + if err := f.enter("StorageVolCreateXML"); err != nil { + return libvirt.StorageVol{}, err + } + vols, ok := f.pools[pool.Name] + if !ok { + return libvirt.StorageVol{}, lvError(libvirt.ErrNoStoragePool, "no pool %s", pool.Name) + } + var v volumeXML + if err := xml.Unmarshal([]byte(raw), &v); err != nil { + return libvirt.StorageVol{}, lvError(libvirt.ErrXMLError, "%v", err) + } + if _, exists := vols[v.Name]; exists { + return libvirt.StorageVol{}, lvError(libvirt.ErrStorageVolExist, "storage volume '%s' exists already", v.Name) + } + fv := &fakeVol{name: v.Name, format: v.Target.Format.Type, capacity: v.Capacity.Value} + if v.BackingStore != nil { + fv.backing = v.BackingStore.Path + } + vols[v.Name] = fv + return libvirt.StorageVol{Pool: pool.Name, Name: v.Name, Key: f.poolPath + "/" + v.Name}, nil +} + +func (f *fakeAPI) StorageVolUpload(vol libvirt.StorageVol, r io.Reader, _, _ uint64, _ libvirt.StorageVolUploadFlags) error { + f.mu.Lock() + if err := f.enter("StorageVolUpload"); err != nil { + f.mu.Unlock() + return err + } + fv, ok := f.pools[vol.Pool][vol.Name] + hook := f.uploadHook + f.mu.Unlock() + if !ok { + return lvError(libvirt.ErrNoStorageVol, "no vol %s", vol.Name) + } + if hook != nil { + if err := hook(vol.Name, r); err != nil { + return err + } + } + // Read outside the lock: the stream may be a slow HTTP body. + data, err := io.ReadAll(r) + if err != nil { + return fmt.Errorf("reading upload stream: %w", err) + } + f.mu.Lock() + defer f.mu.Unlock() + fv.data = data + return nil +} + +func (f *fakeAPI) StorageVolDownload(vol libvirt.StorageVol, w io.Writer, _, _ uint64, _ libvirt.StorageVolDownloadFlags) error { + f.mu.Lock() + defer f.mu.Unlock() + if err := f.enter("StorageVolDownload"); err != nil { + return err + } + fv, ok := f.pools[vol.Pool][vol.Name] + if !ok { + return lvError(libvirt.ErrNoStorageVol, "no vol %s", vol.Name) + } + _, err := io.Copy(w, bytes.NewReader(fv.data)) + return err +} + +func (f *fakeAPI) StorageVolDelete(vol libvirt.StorageVol, _ libvirt.StorageVolDeleteFlags) error { + f.mu.Lock() + defer f.mu.Unlock() + if err := f.enter("StorageVolDelete"); err != nil { + return err + } + if _, ok := f.pools[vol.Pool][vol.Name]; !ok { + return lvError(libvirt.ErrNoStorageVol, "no vol %s", vol.Name) + } + delete(f.pools[vol.Pool], vol.Name) + return nil +} diff --git a/internal/provider/kvm/image.go b/internal/provider/kvm/image.go new file mode 100644 index 0000000..ffdc0e1 --- /dev/null +++ b/internal/provider/kvm/image.go @@ -0,0 +1,232 @@ +package kvm + +import ( + "bytes" + "context" + "crypto/sha256" + "encoding/hex" + "fmt" + "io" + "net/http" + "net/url" + "strings" + "sync" + "time" + + libvirt "github.com/digitalocean/go-libvirt" + "github.com/go-logr/logr" + + "gitea.home.hrajfrisbee.cz/kacerr/egress-proxies-operator/internal/provider" +) + +// imageSource is a resolved placement.image: either an entry from the +// provider's images map (Key set) or a URL given directly. +type imageSource struct { + Key string + URL string + SHA256 string + Format string +} + +// resolveImage maps placement.image to a source. Anything that is neither a +// configured key nor an http(s) URL is a spec error the reconciler should +// surface, not retry. +func resolveImage(cfg provider.KVMConfig, image string) (imageSource, error) { + if img, ok := cfg.Images[image]; ok { + return imageSource{Key: image, URL: img.URL, SHA256: img.SHA256, Format: img.Format}, nil + } + u, err := url.Parse(image) + if err == nil && (u.Scheme == "http" || u.Scheme == "https") && u.Host != "" { + return imageSource{URL: image, Format: formatQcow2}, nil + } + return imageSource{}, fmt.Errorf("%w: placement.image %q is neither a key in kvm.images nor an http(s) URL", + provider.ErrPermanent, image) +} + +// volumeName is the cached base image's name in the pool: readable for +// configured keys, a URL hash for ad-hoc URLs. +func (s imageSource) volumeName() string { + if s.Key != "" { + return "base-" + s.Key + "." + s.Format + } + sum := sha256.Sum256([]byte(s.URL)) + return "base-" + hex.EncodeToString(sum[:8]) + "." + s.Format +} + +// markerName is the sidecar volume holding the sha256 of the verified +// upload. Its presence is what says "the base volume is complete". +func (s imageSource) markerName() string { + return s.volumeName() + ".sha256" +} + +// imageCache fetches base images into a host's pool on first use. It keeps +// no state about what is cached — the pool is the source of truth, checked +// on every call — only a per-(host, image) lock so concurrent Creates for +// the same image on the same host result in one download. +type imageCache struct { + http *http.Client + + mu sync.Mutex + locks map[string]*sync.Mutex +} + +func newImageCache(client *http.Client) *imageCache { + if client == nil { + client = http.DefaultClient + } + return &imageCache{http: client, locks: make(map[string]*sync.Mutex)} +} + +func (c *imageCache) lockFor(key string) *sync.Mutex { + c.mu.Lock() + defer c.mu.Unlock() + m, ok := c.locks[key] + if !ok { + m = &sync.Mutex{} + c.locks[key] = m + } + return m +} + +// ensure makes src present in pool and returns the base volume's path on +// the host. The image is downloaded synchronously through the libvirt +// stream when the pool does not already hold a verified copy. +func (c *imageCache) ensure(ctx context.Context, log logr.Logger, api libvirtAPI, hostName string, pool libvirt.StoragePool, src imageSource) (string, error) { + m := c.lockFor(hostName + "/" + src.volumeName()) + m.Lock() + defer m.Unlock() + + base, ready, err := c.lookup(api, pool, src) + if err != nil { + return "", err + } + if ready { + log.V(1).Info("base image already cached", "host", hostName, "volume", src.volumeName()) + return api.StorageVolGetPath(base) + } + if err := deleteIfExists(api, pool, src.markerName()); err != nil { + return "", err + } + if err := deleteIfExists(api, pool, src.volumeName()); err != nil { + return "", err + } + base, err = c.fetch(ctx, log, api, hostName, pool, src) + if err != nil { + return "", err + } + return api.StorageVolGetPath(base) +} + +// lookup reports whether the pool holds a complete, verified base volume: +// the volume and its marker both exist and, when a checksum is configured, +// the marker records that checksum. +func (c *imageCache) lookup(api libvirtAPI, pool libvirt.StoragePool, src imageSource) (libvirt.StorageVol, bool, error) { + base, err := api.StorageVolLookupByName(pool, src.volumeName()) + if hasCode(err, libvirt.ErrNoStorageVol) { + return libvirt.StorageVol{}, false, nil + } + if err != nil { + return libvirt.StorageVol{}, false, fmt.Errorf("looking up base volume: %w", err) + } + marker, err := api.StorageVolLookupByName(pool, src.markerName()) + if hasCode(err, libvirt.ErrNoStorageVol) { + return base, false, nil + } + if err != nil { + return libvirt.StorageVol{}, false, fmt.Errorf("looking up image marker: %w", err) + } + if src.SHA256 == "" { + return base, true, nil + } + var buf bytes.Buffer + if err := api.StorageVolDownload(marker, &buf, 0, 0, 0); err != nil { + return libvirt.StorageVol{}, false, fmt.Errorf("reading image marker: %w", err) + } + return base, strings.EqualFold(strings.TrimSpace(buf.String()), src.SHA256), nil +} + +// fetch streams the image from its URL straight into a new base volume, +// hashing on the way, then writes the marker. Nothing is buffered in the +// operator beyond the HTTP body's read buffer. +func (c *imageCache) fetch(ctx context.Context, log logr.Logger, api libvirtAPI, hostName string, pool libvirt.StoragePool, src imageSource) (libvirt.StorageVol, error) { + req, err := http.NewRequestWithContext(ctx, http.MethodGet, src.URL, nil) + if err != nil { + return libvirt.StorageVol{}, fmt.Errorf("%w: building image request: %v", provider.ErrPermanent, err) + } + resp, err := c.http.Do(req) + if err != nil { + return libvirt.StorageVol{}, fmt.Errorf("%w: fetching image %s: %v", provider.ErrTransient, src.URL, err) + } + defer resp.Body.Close() + switch { + case resp.StatusCode >= 200 && resp.StatusCode < 300: + case resp.StatusCode >= 400 && resp.StatusCode < 500: + return libvirt.StorageVol{}, fmt.Errorf("%w: image %s returned HTTP %d", provider.ErrPermanent, src.URL, resp.StatusCode) + default: + return libvirt.StorageVol{}, fmt.Errorf("%w: image %s returned HTTP %d", provider.ErrTransient, src.URL, resp.StatusCode) + } + if resp.ContentLength < 0 { + return libvirt.StorageVol{}, fmt.Errorf("%w: image %s has no Content-Length; the base volume needs a known size", provider.ErrPermanent, src.URL) + } + + log.Info("fetching base image into host pool", + "host", hostName, "pool", pool.Name, "volume", src.volumeName(), + "url", src.URL, "bytes", resp.ContentLength) + start := time.Now() + + baseXML, err := buildVolumeXML(volumeSpec{Name: src.volumeName(), SizeBytes: resp.ContentLength, Format: src.Format}) + if err != nil { + return libvirt.StorageVol{}, err + } + base, err := api.StorageVolCreateXML(pool, baseXML, 0) + if err != nil { + return libvirt.StorageVol{}, fmt.Errorf("creating base volume: %w", err) + } + hasher := sha256.New() + if err := api.StorageVolUpload(base, io.TeeReader(resp.Body, hasher), 0, 0, 0); err != nil { + _ = api.StorageVolDelete(base, 0) + return libvirt.StorageVol{}, fmt.Errorf("uploading image to base volume: %w", err) + } + digest := hex.EncodeToString(hasher.Sum(nil)) + if src.SHA256 != "" && !strings.EqualFold(digest, src.SHA256) { + _ = api.StorageVolDelete(base, 0) + return libvirt.StorageVol{}, fmt.Errorf("%w: image %s sha256 is %s, config expects %s", + provider.ErrPermanent, src.URL, digest, src.SHA256) + } + // libvirt sized the volume from Content-Length; a refresh makes it + // re-read the real virtual size from the uploaded qcow2 header. + if err := api.StoragePoolRefresh(pool, 0); err != nil { + return libvirt.StorageVol{}, fmt.Errorf("refreshing pool after upload: %w", err) + } + + markerXML, err := buildVolumeXML(rawVolume(src.markerName(), int64(len(digest)))) + if err != nil { + return libvirt.StorageVol{}, err + } + marker, err := api.StorageVolCreateXML(pool, markerXML, 0) + if err != nil { + return libvirt.StorageVol{}, fmt.Errorf("creating image marker: %w", err) + } + if err := api.StorageVolUpload(marker, strings.NewReader(digest), 0, 0, 0); err != nil { + _ = api.StorageVolDelete(marker, 0) + return libvirt.StorageVol{}, fmt.Errorf("writing image marker: %w", err) + } + log.Info("base image cached", + "host", hostName, "volume", src.volumeName(), "sha256", digest, + "bytes", resp.ContentLength, "duration", time.Since(start).Round(time.Second)) + return base, nil +} + +func deleteIfExists(api libvirtAPI, pool libvirt.StoragePool, name string) error { + vol, err := api.StorageVolLookupByName(pool, name) + if hasCode(err, libvirt.ErrNoStorageVol) { + return nil + } + if err != nil { + return fmt.Errorf("looking up volume %s: %w", name, err) + } + if err := api.StorageVolDelete(vol, 0); err != nil && !hasCode(err, libvirt.ErrNoStorageVol) { + return fmt.Errorf("deleting volume %s: %w", name, err) + } + return nil +} diff --git a/internal/provider/kvm/image_test.go b/internal/provider/kvm/image_test.go new file mode 100644 index 0000000..2493c44 --- /dev/null +++ b/internal/provider/kvm/image_test.go @@ -0,0 +1,385 @@ +package kvm + +import ( + "bytes" + "context" + "crypto/sha256" + "encoding/hex" + "errors" + "io" + "net/http" + "net/http/httptest" + "strconv" + "strings" + "sync" + "sync/atomic" + "testing" + "time" + + "github.com/go-logr/logr" + + "gitea.home.hrajfrisbee.cz/kacerr/egress-proxies-operator/internal/provider" +) + +var testImage = bytes.Repeat([]byte("QFI\xfb fake qcow2 payload "), 1024) + +func testImageSHA() string { + sum := sha256.Sum256(testImage) + return hex.EncodeToString(sum[:]) +} + +// imageServer serves testImage and counts hits. Status overrides the +// response, noLength forces chunked encoding (no Content-Length), delay +// slows each response so concurrent callers overlap. +type imageServer struct { + *httptest.Server + hits atomic.Int32 + status int + noLength bool + delay time.Duration + body []byte +} + +func newImageServer(t *testing.T) *imageServer { + t.Helper() + s := &imageServer{status: http.StatusOK, body: testImage} + s.Server = httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, _ *http.Request) { + s.hits.Add(1) + time.Sleep(s.delay) + if s.status != http.StatusOK { + w.WriteHeader(s.status) + return + } + // Bodies larger than the server's write buffer would otherwise go + // out chunked without a Content-Length, unlike real image mirrors. + if s.noLength { + w.(http.Flusher).Flush() + } else { + w.Header().Set("Content-Length", strconv.Itoa(len(s.body))) + } + _, _ = w.Write(s.body) + })) + t.Cleanup(s.Close) + return s +} + +func testSource(srv *imageServer, sha string) imageSource { + return imageSource{Key: "debian-13", URL: srv.URL + "/debian.qcow2", SHA256: sha, Format: formatQcow2} +} + +func ensureTest(t *testing.T, api *fakeAPI, src imageSource) (string, error) { + t.Helper() + c := newImageCache(nil) + pool, err := api.StoragePoolLookupByName("default") + if err != nil { + t.Fatal(err) + } + return c.ensure(context.Background(), logr.Discard(), api, "nas1", pool, src) +} + +func TestResolveImage(t *testing.T) { + t.Parallel() + cfg := withDefaults(provider.KVMConfig{Images: map[string]provider.KVMImage{ + "debian-13": {URL: "https://cloud.debian.org/d.qcow2", SHA256: "abc"}, + }}) + tests := []struct { + name string + image string + want imageSource + wantErr bool + }{ + {"configured key", "debian-13", imageSource{Key: "debian-13", URL: "https://cloud.debian.org/d.qcow2", SHA256: "abc", Format: "qcow2"}, false}, + {"direct https url", "https://x.example/img.qcow2", imageSource{URL: "https://x.example/img.qcow2", Format: "qcow2"}, false}, + {"direct http url", "http://mirror/img.qcow2", imageSource{URL: "http://mirror/img.qcow2", Format: "qcow2"}, false}, + {"unknown key", "ubuntu", imageSource{}, true}, + {"file url", "file:///tmp/x.qcow2", imageSource{}, true}, + {"empty", "", imageSource{}, true}, + } + for _, tc := range tests { + t.Run(tc.name, func(t *testing.T) { + t.Parallel() + got, err := resolveImage(cfg, tc.image) + if tc.wantErr { + if !errors.Is(err, provider.ErrPermanent) { + t.Fatalf("resolveImage(%q) error = %v, want ErrPermanent", tc.image, err) + } + return + } + if err != nil { + t.Fatalf("resolveImage(%q) error = %v", tc.image, err) + } + if got != tc.want { + t.Errorf("resolveImage(%q) = %+v, want %+v", tc.image, got, tc.want) + } + }) + } +} + +func TestImageSource_volumeNames(t *testing.T) { + t.Parallel() + keyed := imageSource{Key: "debian-13", URL: "https://a/b.qcow2", Format: "qcow2"} + if keyed.volumeName() != "base-debian-13.qcow2" || keyed.markerName() != "base-debian-13.qcow2.sha256" { + t.Errorf("keyed names = %s / %s", keyed.volumeName(), keyed.markerName()) + } + direct := imageSource{URL: "https://a/b.qcow2", Format: "qcow2"} + name := direct.volumeName() + if !strings.HasPrefix(name, "base-") || !strings.HasSuffix(name, ".qcow2") || len(name) != len("base-")+16+len(".qcow2") { + t.Errorf("direct-url name = %q, want base-<16 hex>.qcow2", name) + } + other := imageSource{URL: "https://a/c.qcow2", Format: "qcow2"} + if other.volumeName() == name { + t.Error("different URLs produced the same volume name") + } +} + +func TestImageCache_ensure_freshFetch(t *testing.T) { + t.Parallel() + srv := newImageServer(t) + api := newFakeAPI("default") + path, err := ensureTest(t, api, testSource(srv, testImageSHA())) + if err != nil { + t.Fatalf("ensure() error = %v", err) + } + if path != "/var/lib/libvirt/images/base-debian-13.qcow2" { + t.Errorf("path = %q", path) + } + base := api.vol("default", "base-debian-13.qcow2") + if base == nil || !bytes.Equal(base.data, testImage) { + t.Fatalf("base volume missing or content differs") + } + if base.format != "qcow2" || base.capacity != int64(len(testImage)) { + t.Errorf("base volume format/capacity = %s/%d, want qcow2/%d", base.format, base.capacity, len(testImage)) + } + marker := api.vol("default", "base-debian-13.qcow2.sha256") + if marker == nil || string(marker.data) != testImageSHA() { + t.Fatalf("marker missing or wrong: %+v", marker) + } + if !api.hasCall("StoragePoolRefresh") { + t.Error("pool was not refreshed after the upload") + } + if srv.hits.Load() != 1 { + t.Errorf("http hits = %d, want 1", srv.hits.Load()) + } +} + +func TestImageCache_ensure_noChecksumConfigured(t *testing.T) { + t.Parallel() + srv := newImageServer(t) + api := newFakeAPI("default") + if _, err := ensureTest(t, api, testSource(srv, "")); err != nil { + t.Fatalf("ensure() error = %v", err) + } + if marker := api.vol("default", "base-debian-13.qcow2.sha256"); marker == nil || string(marker.data) != testImageSHA() { + t.Errorf("marker should still record the computed digest, got %+v", marker) + } + // Second call: ready without a checksum to compare, so no marker read. + if _, err := ensureTest(t, api, testSource(srv, "")); err != nil { + t.Fatalf("second ensure() error = %v", err) + } + if srv.hits.Load() != 1 { + t.Errorf("http hits = %d, want 1", srv.hits.Load()) + } +} + +func TestImageCache_ensure_alreadyCached(t *testing.T) { + t.Parallel() + srv := newImageServer(t) + api := newFakeAPI("default") + api.putVol("default", "base-debian-13.qcow2", "qcow2", testImage) + api.putVol("default", "base-debian-13.qcow2.sha256", "raw", []byte(testImageSHA())) + path, err := ensureTest(t, api, testSource(srv, testImageSHA())) + if err != nil { + t.Fatalf("ensure() error = %v", err) + } + if path != "/var/lib/libvirt/images/base-debian-13.qcow2" { + t.Errorf("path = %q", path) + } + if srv.hits.Load() != 0 { + t.Errorf("http hits = %d, want 0 for a cached image", srv.hits.Load()) + } + if api.hasCall("StorageVolCreateXML") || api.hasCall("StorageVolDelete") { + t.Errorf("cached image was touched: calls = %v", api.calls) + } +} + +func TestImageCache_ensure_markerMissingRedownloads(t *testing.T) { + t.Parallel() + srv := newImageServer(t) + api := newFakeAPI("default") + api.putVol("default", "base-debian-13.qcow2", "qcow2", []byte("half an image")) + if _, err := ensureTest(t, api, testSource(srv, testImageSHA())); err != nil { + t.Fatalf("ensure() error = %v", err) + } + if srv.hits.Load() != 1 { + t.Errorf("http hits = %d, want 1: a base without its marker is a partial upload", srv.hits.Load()) + } + if base := api.vol("default", "base-debian-13.qcow2"); base == nil || !bytes.Equal(base.data, testImage) { + t.Error("partial base volume was not replaced") + } +} + +func TestImageCache_ensure_checksumChangedRedownloads(t *testing.T) { + t.Parallel() + srv := newImageServer(t) + api := newFakeAPI("default") + api.putVol("default", "base-debian-13.qcow2", "qcow2", []byte("an older release")) + api.putVol("default", "base-debian-13.qcow2.sha256", "raw", []byte(strings.Repeat("0", 64))) + if _, err := ensureTest(t, api, testSource(srv, testImageSHA())); err != nil { + t.Fatalf("ensure() error = %v", err) + } + if srv.hits.Load() != 1 { + t.Errorf("http hits = %d, want 1 after the configured checksum changed", srv.hits.Load()) + } + if marker := api.vol("default", "base-debian-13.qcow2.sha256"); marker == nil || string(marker.data) != testImageSHA() { + t.Errorf("marker not updated: %+v", marker) + } +} + +func TestImageCache_ensure_checksumMismatchIsPermanent(t *testing.T) { + t.Parallel() + srv := newImageServer(t) + api := newFakeAPI("default") + _, err := ensureTest(t, api, testSource(srv, strings.Repeat("a", 64))) + if !errors.Is(err, provider.ErrPermanent) { + t.Fatalf("ensure() error = %v, want ErrPermanent", err) + } + if names := api.volNames("default"); len(names) != 0 { + t.Errorf("volumes left behind after a checksum mismatch: %v", names) + } +} + +func TestImageCache_ensure_httpErrors(t *testing.T) { + t.Parallel() + tests := []struct { + name string + status int + want error + }{ + {"404 is permanent", http.StatusNotFound, provider.ErrPermanent}, + {"403 is permanent", http.StatusForbidden, provider.ErrPermanent}, + {"503 is transient", http.StatusServiceUnavailable, provider.ErrTransient}, + {"500 is transient", http.StatusInternalServerError, provider.ErrTransient}, + } + for _, tc := range tests { + t.Run(tc.name, func(t *testing.T) { + t.Parallel() + srv := newImageServer(t) + srv.status = tc.status + api := newFakeAPI("default") + _, err := ensureTest(t, api, testSource(srv, "")) + if !errors.Is(err, tc.want) { + t.Fatalf("ensure() error = %v, want %v", err, tc.want) + } + if names := api.volNames("default"); len(names) != 0 { + t.Errorf("volumes created despite HTTP %d: %v", tc.status, names) + } + }) + } +} + +func TestImageCache_ensure_noContentLengthIsPermanent(t *testing.T) { + t.Parallel() + srv := newImageServer(t) + srv.noLength = true + api := newFakeAPI("default") + _, err := ensureTest(t, api, testSource(srv, "")) + if !errors.Is(err, provider.ErrPermanent) || !strings.Contains(err.Error(), "Content-Length") { + t.Fatalf("ensure() error = %v, want ErrPermanent about Content-Length", err) + } +} + +func TestImageCache_ensure_uploadFailureCleansUp(t *testing.T) { + t.Parallel() + srv := newImageServer(t) + api := newFakeAPI("default") + api.uploadHook = func(_ string, r io.Reader) error { + _, _ = io.CopyN(io.Discard, r, 100) + return io.ErrUnexpectedEOF + } + _, err := ensureTest(t, api, testSource(srv, testImageSHA())) + if !errors.Is(err, provider.ErrTransient) && classify(err) != provider.ErrTransient { + t.Fatalf("ensure() error = %v, want a transient failure", err) + } + if names := api.volNames("default"); len(names) != 0 { + t.Errorf("volumes left behind after a failed upload: %v", names) + } + // Next attempt starts clean and succeeds. + api.uploadHook = nil + if _, err := ensureTest(t, api, testSource(srv, testImageSHA())); err != nil { + t.Fatalf("retry ensure() error = %v", err) + } +} + +func TestImageCache_ensure_missingPoolIsPermanent(t *testing.T) { + t.Parallel() + srv := newImageServer(t) + api := newFakeAPI("default") + c := newImageCache(nil) + pool, _ := api.StoragePoolLookupByName("default") + delete(api.pools, "default") + _, err := c.ensure(context.Background(), logr.Discard(), api, "nas1", pool, testSource(srv, "")) + if classify(err) != provider.ErrPermanent { + t.Fatalf("ensure() error = %v, want one classifying as Permanent", err) + } +} + +func TestImageCache_ensure_concurrentCallersDownloadOnce(t *testing.T) { + t.Parallel() + srv := newImageServer(t) + srv.delay = 50 * time.Millisecond + api := newFakeAPI("default") + c := newImageCache(nil) + pool, _ := api.StoragePoolLookupByName("default") + src := testSource(srv, testImageSHA()) + var wg sync.WaitGroup + paths := make([]string, 4) + errs := make([]error, 4) + for i := range 4 { + wg.Go(func() { + paths[i], errs[i] = c.ensure(context.Background(), logr.Discard(), api, "nas1", pool, src) + }) + } + wg.Wait() + for i := range 4 { + if errs[i] != nil || paths[i] != "/var/lib/libvirt/images/base-debian-13.qcow2" { + t.Errorf("caller %d: path=%q err=%v", i, paths[i], errs[i]) + } + } + if srv.hits.Load() != 1 { + t.Errorf("http hits = %d, want 1 for four concurrent callers", srv.hits.Load()) + } +} + +func TestImageCache_ensure_differentHostsFetchIndependently(t *testing.T) { + t.Parallel() + srv := newImageServer(t) + c := newImageCache(nil) + src := testSource(srv, "") + for _, host := range []string{"nas1", "lab2"} { + api := newFakeAPI("default") + pool, _ := api.StoragePoolLookupByName("default") + if _, err := c.ensure(context.Background(), logr.Discard(), api, host, pool, src); err != nil { + t.Fatalf("ensure(%s) error = %v", host, err) + } + } + if srv.hits.Load() != 2 { + t.Errorf("http hits = %d, want one per host", srv.hits.Load()) + } +} + +func TestImageCache_ensure_cancelledContextIsTransient(t *testing.T) { + t.Parallel() + srv := newImageServer(t) + api := newFakeAPI("default") + c := newImageCache(nil) + pool, _ := api.StoragePoolLookupByName("default") + ctx, cancel := context.WithCancel(context.Background()) + cancel() + _, err := c.ensure(ctx, logr.Discard(), api, "nas1", pool, testSource(srv, "")) + if classify(err) != provider.ErrTransient { + t.Fatalf("ensure() error = %v, want transient", err) + } + if names := api.volNames("default"); len(names) != 0 { + t.Errorf("volumes created despite cancelled context: %v", names) + } +} -- 2.54.0 From 97cb854693e629d6ee510e0099cd80a533846f5c Mon Sep 17 00:00:00 2001 From: kacerr Date: Tue, 8 Sep 2026 10:48:59 +0000 Subject: [PATCH 06/28] Add KVM provider methods: Create/Get/Delete/ListByTag over the libvirt seam Create looks the domain up first and only starts it when it exists; otherwise it caches the image, reuses or creates the overlay, rebuilds the seed ISO unconditionally (a cancelled upload can leave a truncated seed that no later attempt would repair), defines and starts the domain. Get maps libvirt state plus the first routable IPv4 from ARP or the guest agent onto the provider states. Delete destroys, undefines and removes the per-VM volumes by name, tolerating every already-gone case. ListByTag keeps managed domains across all hosts, skipping domains it cannot read and failing the sweep only for an unreachable host. Also corrects the Step 2 seed determinism test: the iso9660 writer stamps the build time into the volume descriptor, so builds differ by a few bytes; the test now compares label, files and size. Co-Authored-By: Claude Fable 5.1 --- .../2026-09-04-1404-kvm-provider.md | 89 +- internal/provider/kvm/kvm.go | 515 ++++++++++- internal/provider/kvm/kvm_test.go | 836 ++++++++++++++++++ internal/provider/kvm/seed.go | 7 +- internal/provider/kvm/seed_test.go | 15 +- 5 files changed, 1454 insertions(+), 8 deletions(-) diff --git a/docs/plans-executions/2026-09-04-1404-kvm-provider.md b/docs/plans-executions/2026-09-04-1404-kvm-provider.md index beebc58..02da6f9 100644 --- a/docs/plans-executions/2026-09-04-1404-kvm-provider.md +++ b/docs/plans-executions/2026-09-04-1404-kvm-provider.md @@ -7,7 +7,7 @@ Plan: [2026-09-04-1404-kvm-provider.md](../plans/2026-09-04-1404-kvm-provider.md - [x] Step 2 — Pure builders (`domainxml.go`, `volume.go`, `seed.go`, `id.go`, `metadata.go`) - [x] Step 3 — Transport + errors (`conn.go`, `errors.go`) - [x] Step 3b — Image cache (`image.go`) -- [ ] Step 4 — Provider methods (`kvm.go`) over a fake API +- [x] Step 4 — Provider methods (`kvm.go`) over a fake API - [ ] Step 5 — Wiring, manifests, samples, docs, CHANGELOG - [ ] Step 6 — Integration test against a real hypervisor - [ ] Step 7 — Manual validation + MR @@ -196,3 +196,90 @@ confirms this on a real host. Also, the whole download runs inside one `pool.call`, so cancellation aborts the HTTP body (ctx-bound) and the upload stream fails; the half-written base volume is deleted on the spot and, failing that, on the next attempt via the missing marker. + +## Step 4 — Provider methods + +`kvm.go` now holds `Provider`, `New` (loads the SSH key and known_hosts +up front via `newSSHAuth`, so a bad path fails at startup), the +`newWithDialer` seam, and `Create`/`Get`/`Delete`/`ListByTag`. Each +method resolves its host from config, then runs the whole libvirt +conversation inside one `hostPool.call`, so a cancelled reconcile drops +the connection and the blocked RPC returns. Errors pass through `classify` +in one `wrapErr`, as the Step 3b note intended. + +Behaviour that goes beyond the plan text, or that a future reader should +know before touching this file: + +- **Create looks up the domain first.** If it exists, only the running + check runs and the volumes are never touched. If not, nothing can be + using the volumes yet: the overlay is reused when it exists (its + creation is atomic; `ErrStorageVolExist` → reuse) but the **seed is + always deleted and rebuilt**. The plan said "exists → skip upload"; the + reviewer pointed out that a reconcile cancelled mid-upload leaves a + truncated seed that the compensating delete cannot remove, because + `hostPool.call` has already closed the connection the delete would go + over. Skipping such a seed would boot the VM without cloud-init and no + retry would ever repair it. Rebuilding costs one ~40 KiB upload per + attempt before the domain exists, nothing afterwards. The table test + breaks Create after each step and re-runs it, and a separate test + starts from a leftover truncated seed. +- **Get swallows address-source errors.** `DomainInterfaceAddresses` from + ARP returns nothing until the guest has sent traffic and the agent + source errors with `ErrAgentUnresponsive` until `qemu-guest-agent` is + up, so both are logged at V(2) and mean "no IP yet" → `Provisioning`. + Loopback, link-local and unspecified addresses are skipped when + choosing the first IPv4 (the agent reports `lo` too). +- **State mapping:** `running`/`blocked` are active; `nostate` → + Provisioning (the reconciler answers Stopped with delete-and-recreate, + too drastic for a state libvirt could not determine); everything else + (paused, pmsuspended, shutdown, shutoff, crashed) → Stopped. +- **Delete treats a missing pool as "nothing to clean"** rather than + Permanent: failing there would pin the Proxy's finalizer on a config + edit that renamed the pool. Domain removal is destroy (`ErrOperationInvalid` + on a stopped domain is fine) → undefine with managed-save, snapshot + metadata and nvram flags → overlay and seed deleted by name. +- **ListByTag** reads each domain's metadata once (passed into the same + `inspect` that Get uses, so no second `DomainGetMetadata` round-trip) + and skips domains without the operator's element. A domain that cannot + be read at all — vanished mid-sweep, corrupt metadata, state call + failing — is *skipped*, not fatal: GC never deletes what it does not + see, so leaving one out is safe, whereas one bad domain blocking GC for + every other one (the first version's behaviour, caught in review) is + not. An unreachable host still fails the whole sweep. + `ConnectListAllDomains` is called with flags 0 (active and inactive). +- A host named in a providerID but no longer in `kvm.hosts` is Permanent + for Get and Delete: the operator cannot reach it, and retrying forever + would hide the config problem. + +Tests run the provider over a `hostDialer` that maps host names to +`fakeAPI` instances (an unmapped host is "unreachable"). The image is +pre-seeded in the pool for most tests; one test runs the full first-use +download through `httptest` and confirms a second Proxy on the same host +does not download again. + +Found while running the package: `TestBuildSeedISO_deterministic` from +Step 2 was wrong. `iso9660.ImageWriter` stamps `time.Now()` into the +volume descriptor, so two builds in different seconds differ by six +timestamp bytes; the test had only passed because both builds landed in +the same second. Byte identity is nothing the provider depends on (an +existing seed is never re-uploaded), so the test now compares label, +files and size, and the comment on `buildSeedISO` no longer claims +identical output. + +The `go-operator-reviewer` agent ran on the first version of this step +and found the two bugs above (truncated seed, wedged GC sweep) plus the +double metadata read and the `nostate` mapping; all were fixed before +committing. One finding it raised is deliberately **not** fixed here +because it lives in Step 3b's `image.go` and changes the volume naming +the plan chose: re-fetching a base image (checksum edited in config, or +upstream content changed) deletes `base-.` while existing +overlays still back onto it, breaking those VMs' disks. Content-addressed +base names would avoid it. Flagged for the user; until decided, Step 5's +docs must warn that an image's `sha256`/`url` must not change while +Proxies using it exist. + +Worth noting: `go test -cover` reports 88.2 % for the package; the +uncovered lines are the real SSH/libvirt dial path and log-only +branches. `go vet` and `-race` (three runs) are clean. Wiring into +`cmd/main.go` is Step 5, so `New` is not yet reachable from a running +operator. diff --git a/internal/provider/kvm/kvm.go b/internal/provider/kvm/kvm.go index e5653e3..e510d08 100644 --- a/internal/provider/kvm/kvm.go +++ b/internal/provider/kvm/kvm.go @@ -7,7 +7,20 @@ // NoCloud seed ISO built in-process. package kvm -import "gitea.home.hrajfrisbee.cz/kacerr/egress-proxies-operator/internal/provider" +import ( + "bytes" + "context" + "fmt" + "net" + "net/http" + "time" + + libvirt "github.com/digitalocean/go-libvirt" + "github.com/go-logr/logr" + logf "sigs.k8s.io/controller-runtime/pkg/log" + + "gitea.home.hrajfrisbee.cz/kacerr/egress-proxies-operator/internal/provider" +) const ( defaultDiskSizeGB = 10 @@ -15,6 +28,12 @@ const ( defaultPool = "default" formatQcow2 = "qcow2" formatRaw = "raw" + + // undefineFlags removes everything libvirt may keep alongside a domain + // definition, so an undefine never fails on leftover state. + undefineFlags = libvirt.DomainUndefineManagedSave | + libvirt.DomainUndefineSnapshotsMetadata | + libvirt.DomainUndefineNvram ) func withDefaults(cfg provider.KVMConfig) provider.KVMConfig { @@ -45,3 +64,497 @@ func withDefaults(cfg provider.KVMConfig) provider.KVMConfig { cfg.Hosts = hosts return cfg } + +// Provider implements provider.Provider on one or more libvirt hosts. +type Provider struct { + name string + cfg provider.KVMConfig + hosts *hostPool + images *imageCache + now func() time.Time +} + +// New builds a Provider that reaches its hosts over SSH. The key and +// known_hosts files are read here so a bad path fails at startup rather +// than on the first Create. Satisfies registry.Constructor. +func New(_ context.Context, pc provider.ProviderConfig) (provider.Provider, error) { + cfg := provider.KVMConfig{} + if pc.KVM != nil { + cfg = *pc.KVM + } + auth, err := newSSHAuth(cfg) + if err != nil { + return nil, fmt.Errorf("kvm provider %q: %w", pc.Name, err) + } + return newWithDialer(pc, auth.dialer(), nil), nil +} + +// newWithDialer is the test seam: the dialer decides what libvirtAPI each +// host resolves to, and the HTTP client is what image downloads go through. +func newWithDialer(pc provider.ProviderConfig, dial hostDialer, httpClient *http.Client) *Provider { + cfg := provider.KVMConfig{} + if pc.KVM != nil { + cfg = *pc.KVM + } + return &Provider{ + name: pc.Name, + cfg: withDefaults(cfg), + hosts: newHostPool(dial), + images: newImageCache(httpClient), + now: time.Now, + } +} + +func (p *Provider) logger(ctx context.Context) logr.Logger { + return logf.FromContext(ctx).WithName("kvm").WithValues("provider", p.name) +} + +func (p *Provider) wrapErr(op, id string, err error) error { + return provider.Wrap(classify(err), op, p.name, id, err) +} + +// host resolves placement.zone (or a providerID's host segment) to a +// configured host. An empty zone is allowed only when exactly one host is +// configured; anything else is a spec error. +func (p *Provider) host(zone string) (provider.KVMHost, error) { + if zone == "" { + switch len(p.cfg.Hosts) { + case 0: + return provider.KVMHost{}, fmt.Errorf("%w: kvm.hosts is empty", provider.ErrPermanent) + case 1: + return p.cfg.Hosts[0], nil + default: + return provider.KVMHost{}, fmt.Errorf("%w: placement.zone is required when kvm.hosts lists more than one host", + provider.ErrPermanent) + } + } + for _, h := range p.cfg.Hosts { + if h.Name == zone { + return h, nil + } + } + return provider.KVMHost{}, fmt.Errorf("%w: host %q is not in kvm.hosts", provider.ErrPermanent, zone) +} + +// Create makes the VM exist and running on the placed host, and is +// idempotent: a defined domain is only started if needed, its volumes +// untouched. Before the domain exists nothing can be using the volumes, +// so an existing overlay is reused (its creation is atomic) but the seed +// is rebuilt from scratch — a previous attempt may have been cut off mid +// upload, and a truncated seed would boot a VM without cloud-init. The +// first Create for an image on a host also downloads that image into the +// host's pool, synchronously. +func (p *Provider) Create(ctx context.Context, req provider.CreateRequest) (string, error) { + pl := req.Placement + host, err := p.host(pl.Zone) + if err != nil { + return "", p.wrapErr("create", "", err) + } + mt, ok := p.cfg.MachineTypes[pl.MachineType] + if !ok { + return "", p.wrapErr("create", "", fmt.Errorf("%w: placement.machineType %q is not in kvm.machineTypes", + provider.ErrPermanent, pl.MachineType)) + } + src, err := resolveImage(p.cfg, pl.Image) + if err != nil { + return "", p.wrapErr("create", "", err) + } + id := formatProviderID(host.Name, req.Name) + log := p.logger(ctx).WithValues("host", host.Name, "name", req.Name) + log.V(2).Info("KVM create resolved", + "pool", host.Pool, "bridge", host.Bridge, "machineType", pl.MachineType, + "vcpus", mt.VCPUs, "memoryMiB", mt.MemoryMiB, "diskSizeGb", p.cfg.DiskSizeGB, + "image", src.volumeName(), "cloudInitBytes", len(req.CloudInit)) + + err = p.hosts.call(ctx, host, func(api libvirtAPI) error { + dom, err := api.DomainLookupByName(req.Name) + if err == nil { + log.V(1).Info("domain already defined") + return p.ensureRunning(log, api, dom) + } + if !hasCode(err, libvirt.ErrNoDomain) { + return fmt.Errorf("looking up domain: %w", err) + } + pool, err := api.StoragePoolLookupByName(host.Pool) + if err != nil { + return fmt.Errorf("looking up pool %s: %w", host.Pool, err) + } + basePath, err := p.images.ensure(ctx, log, api, host.Name, pool, src) + if err != nil { + return err + } + if err := p.ensureOverlay(log, api, pool, req.Name, basePath, src.Format); err != nil { + return err + } + if err := p.writeSeed(log, api, pool, req.Name, req.CloudInit); err != nil { + return err + } + dom, err = p.defineDomain(log, api, host, req, mt) + if err != nil { + return err + } + return p.ensureRunning(log, api, dom) + }) + if err != nil { + logAPIError(log, "create", err) + return "", p.wrapErr("create", id, err) + } + log.V(1).Info("KVM domain ensured running", "id", id) + return id, nil +} + +func (p *Provider) ensureOverlay(log logr.Logger, api libvirtAPI, pool libvirt.StoragePool, name, basePath, baseFormat string) error { + xml, err := buildVolumeXML(overlayVolume(name, p.cfg.DiskSizeGB, basePath, baseFormat)) + if err != nil { + return err + } + _, err = api.StorageVolCreateXML(pool, xml, 0) + switch { + case err == nil: + log.V(1).Info("overlay volume created", "volume", overlayVolumeName(name), "base", basePath) + case hasCode(err, libvirt.ErrStorageVolExist): + log.V(1).Info("overlay volume already exists", "volume", overlayVolumeName(name)) + default: + return fmt.Errorf("creating overlay volume: %w", err) + } + return nil +} + +// writeSeed replaces the seed volume with a freshly built NoCloud ISO. +// Only called while no domain references it. +func (p *Provider) writeSeed(log logr.Logger, api libvirtAPI, pool libvirt.StoragePool, name, userData string) error { + seedName := seedVolumeName(name) + if err := deleteIfExists(api, pool, seedName); err != nil { + return err + } + iso, err := buildSeedISO(name, userData, p.cfg.VendorData) + if err != nil { + return fmt.Errorf("%w: %v", provider.ErrPermanent, err) + } + xml, err := buildVolumeXML(rawVolume(seedName, int64(len(iso)))) + if err != nil { + return err + } + vol, err := api.StorageVolCreateXML(pool, xml, 0) + if err != nil { + return fmt.Errorf("creating seed volume: %w", err) + } + if err := api.StorageVolUpload(vol, bytes.NewReader(iso), 0, 0, 0); err != nil { + return fmt.Errorf("uploading seed volume: %w", err) + } + log.V(1).Info("seed volume uploaded", "volume", seedName, "bytes", len(iso)) + return nil +} + +func (p *Provider) defineDomain(log logr.Logger, api libvirtAPI, host provider.KVMHost, req provider.CreateRequest, mt provider.KVMMachineType) (libvirt.Domain, error) { + xml, err := buildDomainXML(domainParams{ + Name: req.Name, + UID: req.UID, + CreatedAt: p.now(), + VCPUs: mt.VCPUs, + MemoryMiB: mt.MemoryMiB, + Pool: host.Pool, + Bridge: host.Bridge, + DiskVol: overlayVolumeName(req.Name), + SeedVol: seedVolumeName(req.Name), + }) + if err != nil { + return libvirt.Domain{}, err + } + dom, err := api.DomainDefineXML(xml) + if err != nil { + return libvirt.Domain{}, fmt.Errorf("defining domain: %w", err) + } + log.V(1).Info("domain defined", "uid", req.UID) + return dom, nil +} + +func (p *Provider) ensureRunning(log logr.Logger, api libvirtAPI, dom libvirt.Domain) error { + state, _, err := api.DomainGetState(dom, 0) + if err != nil { + return fmt.Errorf("reading domain state: %w", err) + } + if isActive(libvirt.DomainState(state)) { + return nil + } + err = api.DomainCreate(dom) + switch { + case err == nil: + log.V(1).Info("domain started") + case hasCode(err, libvirt.ErrOperationInvalid): + log.V(1).Info("domain already running, start treated as success") + default: + return fmt.Errorf("starting domain: %w", err) + } + return nil +} + +// Get reads the domain's state, metadata and addresses. The providerID +// carries the host, so this looks at the right hypervisor even after a +// placement.zone edit. +func (p *Provider) Get(ctx context.Context, providerID string) (*provider.Instance, error) { + hostName, name, err := parseProviderID(providerID) + if err != nil { + return nil, provider.Wrap(provider.ErrPermanent, "get", p.name, providerID, err) + } + host, err := p.host(hostName) + if err != nil { + return nil, p.wrapErr("get", providerID, err) + } + log := p.logger(ctx).WithValues("host", host.Name, "name", name) + var inst *provider.Instance + err = p.hosts.call(ctx, host, func(api libvirtAPI) error { + dom, err := api.DomainLookupByName(name) + if err != nil { + return fmt.Errorf("looking up domain: %w", err) + } + meta, err := p.metadata(api, dom) + if err != nil { + return err + } + inst, err = p.inspect(log, api, host.Name, dom, meta) + return err + }) + if err != nil { + logAPIError(log, "get", err) + return nil, p.wrapErr("get", providerID, err) + } + log.V(1).Info("KVM domain fetched", "state", inst.State, "ip", inst.IP, "uid", inst.UID) + return inst, nil +} + +// inspect assembles an Instance from a domain that is known to exist and +// whose metadata has already been read (the zero value for a domain +// without the operator's element, which reports an empty UID as the +// contract requires). +func (p *Provider) inspect(log logr.Logger, api libvirtAPI, hostName string, dom libvirt.Domain, meta instanceMetadata) (*provider.Instance, error) { + state, _, err := api.DomainGetState(dom, 0) + if err != nil { + return nil, fmt.Errorf("reading domain state: %w", err) + } + var ip string + if isActive(libvirt.DomainState(state)) { + ip = p.address(log, api, dom) + } + return &provider.Instance{ + ID: formatProviderID(hostName, dom.Name), + IP: ip, + State: mapState(libvirt.DomainState(state), ip), + UID: meta.UID, + CreatedAt: meta.createdAt(), + }, nil +} + +// metadata returns the operator's element, or the zero value when the +// domain has none (it was not created by this operator). +func (p *Provider) metadata(api libvirtAPI, dom libvirt.Domain) (instanceMetadata, error) { + raw, err := api.DomainGetMetadata(dom, int32(libvirt.DomainMetadataElement), libvirt.OptString{metadataURI}, libvirt.DomainAffectCurrent) + if hasCode(err, libvirt.ErrNoDomainMetadata) { + return instanceMetadata{}, nil + } + if err != nil { + return instanceMetadata{}, fmt.Errorf("reading domain metadata: %w", err) + } + meta, err := parseInstanceMetadata(raw) + if err != nil { + return instanceMetadata{}, fmt.Errorf("%w: %v", provider.ErrPermanent, err) + } + return meta, nil +} + +// address returns the domain's first routable IPv4, asking the host's ARP +// table first and the guest agent second. Both sources fail routinely while +// the guest is still booting (no traffic seen yet, agent not installed +// yet), so their errors mean "no address yet", not "Get failed". +func (p *Provider) address(log logr.Logger, api libvirtAPI, dom libvirt.Domain) string { + for _, source := range []libvirt.DomainInterfaceAddressesSource{ + libvirt.DomainInterfaceAddressesSrcArp, + libvirt.DomainInterfaceAddressesSrcAgent, + } { + ifaces, err := api.DomainInterfaceAddresses(dom, uint32(source), 0) + if err != nil { + log.V(2).Info("no addresses from source", "source", source.String(), "error", err.Error()) + continue + } + if ip := firstIPv4(ifaces); ip != "" { + return ip + } + } + return "" +} + +func firstIPv4(ifaces []libvirt.DomainInterface) string { + for _, iface := range ifaces { + for _, a := range iface.Addrs { + if a.Type != int32(libvirt.IPAddrTypeIpv4) { + continue + } + ip := net.ParseIP(a.Addr) + if ip == nil || ip.IsLoopback() || ip.IsLinkLocalUnicast() || ip.IsUnspecified() { + continue + } + return a.Addr + } + } + return "" +} + +// isActive reports whether the guest is executing (or blocked on the +// hypervisor, which is still "up" from the outside). +func isActive(s libvirt.DomainState) bool { + return s == libvirt.DomainRunning || s == libvirt.DomainBlocked +} + +// mapState collapses libvirt's states onto the provider's. An active +// domain without an address is still Provisioning: an empty IP must never +// be published as Running. "No state" is also Provisioning — the +// reconciler answers Stopped with delete-and-recreate, too drastic for a +// state libvirt could not determine; the next poll decides. Everything +// else — paused, suspended, shut off, crashed, shutting down — is Stopped. +func mapState(s libvirt.DomainState, ip string) provider.InstanceState { + if s == libvirt.DomainNostate { + return provider.StateProvisioning + } + if !isActive(s) { + return provider.StateStopped + } + if ip == "" { + return provider.StateProvisioning + } + return provider.StateRunning +} + +// Delete tears the VM down and is idempotent at every step: a stopped +// domain is not destroyed again, a missing domain is not undefined, and +// the per-VM volumes are removed by name even when the domain is already +// gone, so a crash between volume creation and define is cleaned up on the +// next reconcile. +func (p *Provider) Delete(ctx context.Context, providerID string) error { + hostName, name, err := parseProviderID(providerID) + if err != nil { + return provider.Wrap(provider.ErrPermanent, "delete", p.name, providerID, err) + } + host, err := p.host(hostName) + if err != nil { + return p.wrapErr("delete", providerID, err) + } + log := p.logger(ctx).WithValues("host", host.Name, "name", name) + err = p.hosts.call(ctx, host, func(api libvirtAPI) error { + if err := p.removeDomain(log, api, name); err != nil { + return err + } + return p.removeVolumes(log, api, host.Pool, name) + }) + if err != nil { + logAPIError(log, "delete", err) + return p.wrapErr("delete", providerID, err) + } + log.V(1).Info("KVM domain deleted") + return nil +} + +func (p *Provider) removeDomain(log logr.Logger, api libvirtAPI, name string) error { + dom, err := api.DomainLookupByName(name) + if hasCode(err, libvirt.ErrNoDomain) { + log.V(1).Info("domain already gone") + return nil + } + if err != nil { + return fmt.Errorf("looking up domain: %w", err) + } + err = api.DomainDestroy(dom) + switch { + case err == nil: + log.V(1).Info("domain destroyed") + case hasCode(err, libvirt.ErrOperationInvalid): + log.V(1).Info("domain not running, destroy skipped") + case hasCode(err, libvirt.ErrNoDomain): + return nil + default: + return fmt.Errorf("destroying domain: %w", err) + } + if err := api.DomainUndefineFlags(dom, undefineFlags); err != nil && !hasCode(err, libvirt.ErrNoDomain) { + return fmt.Errorf("undefining domain: %w", err) + } + log.V(1).Info("domain undefined") + return nil +} + +// removeVolumes deletes the overlay and seed. A missing pool means there +// is nothing left to clean, not a failure: refusing to finish a Delete +// over a pool that no longer exists would pin the Proxy's finalizer. +func (p *Provider) removeVolumes(log logr.Logger, api libvirtAPI, poolName, name string) error { + pool, err := api.StoragePoolLookupByName(poolName) + if hasCode(err, libvirt.ErrNoStoragePool) { + log.Info("pool not found, no volumes to delete", "pool", poolName) + return nil + } + if err != nil { + return fmt.Errorf("looking up pool %s: %w", poolName, err) + } + for _, vol := range []string{overlayVolumeName(name), seedVolumeName(name)} { + if err := deleteIfExists(api, pool, vol); err != nil { + return err + } + } + log.V(1).Info("per-VM volumes deleted") + return nil +} + +// ListByTag inspects every domain on every host and keeps those carrying +// the operator's managed metadata. One unreachable host fails the whole +// sweep, so orphan GC never acts on a partial view of the fleet; a single +// domain that cannot be read is skipped instead, which is safe in the +// same direction — a domain missing from the list is never collected. +func (p *Provider) ListByTag(ctx context.Context) ([]provider.Instance, error) { + log := p.logger(ctx) + var out []provider.Instance + for _, host := range p.cfg.Hosts { + hlog := log.WithValues("host", host.Name) + err := p.hosts.call(ctx, host, func(api libvirtAPI) error { + doms, _, err := api.ConnectListAllDomains(1, 0) + if err != nil { + return fmt.Errorf("listing domains: %w", err) + } + for _, dom := range doms { + if inst := p.managedInstance(hlog, api, host.Name, dom); inst != nil { + out = append(out, *inst) + } + } + return nil + }) + if err != nil { + logAPIError(hlog, "list", err) + return nil, p.wrapErr("list", "", err) + } + } + log.V(1).Info("KVM domains listed", "hosts", len(p.cfg.Hosts), "managed", len(out)) + for _, inst := range out { + log.V(2).Info("KVM listed instance", "id", inst.ID, "state", inst.State, "uid", inst.UID, "createdAt", inst.CreatedAt) + } + return out, nil +} + +// managedInstance returns nil for domains the operator does not own, for +// domains that vanished between the listing and the inspection, and for +// domains whose metadata or state cannot be read (corrupt element, mid +// teardown): one bad domain must not block GC for every other one. A +// transport failure would also land here, but it fails the next RPC in +// the same sweep too, so the host still counts as unreachable. +func (p *Provider) managedInstance(log logr.Logger, api libvirtAPI, hostName string, dom libvirt.Domain) *provider.Instance { + log = log.WithValues("name", dom.Name) + meta, err := p.metadata(api, dom) + if err != nil { + log.V(1).Info("skipping domain with unreadable metadata", "error", err.Error()) + return nil + } + if !meta.managed() { + return nil + } + inst, err := p.inspect(log, api, hostName, dom, meta) + if err != nil { + log.V(1).Info("skipping domain that could not be inspected", "error", err.Error()) + return nil + } + return inst +} diff --git a/internal/provider/kvm/kvm_test.go b/internal/provider/kvm/kvm_test.go index 0f04970..c627c34 100644 --- a/internal/provider/kvm/kvm_test.go +++ b/internal/provider/kvm/kvm_test.go @@ -1,7 +1,16 @@ package kvm import ( + "context" + "encoding/xml" + "errors" + "fmt" + "io" + "strings" "testing" + "time" + + libvirt "github.com/digitalocean/go-libvirt" "gitea.home.hrajfrisbee.cz/kacerr/egress-proxies-operator/internal/provider" ) @@ -42,3 +51,830 @@ func TestWithDefaults_keepsExplicitValues(t *testing.T) { t.Errorf("cfg = %+v, want explicit values kept", cfg) } } + +// --- provider over fake hosts --- + +const ( + testName = "proxy-abc123def456ghij" + testUID = "11111111-2222-3333-4444-555555555555" + testImageURL = "https://images.example/debian-13.qcow2" + testCloudInit = "#cloud-config\npackages: [squid]\n" +) + +var ( + testNow = time.Date(2026, 9, 8, 10, 0, 0, 0, time.UTC) + errHostOff = errors.New("ssh dial: connection refused") +) + +// hostFakes routes each configured host name to its own fakeAPI; a host +// with no entry is unreachable. +type hostFakes map[string]*fakeAPI + +func (h hostFakes) dial(_ context.Context, host provider.KVMHost) (*hostClient, error) { + api, ok := h[host.Name] + if !ok { + return nil, errHostOff + } + return &hostClient{api: api, close: func() {}}, nil +} + +func testConfig(hosts ...string) provider.ProviderConfig { + cfg := &provider.KVMConfig{ + SSHKeyFile: "/k", + KnownHostsFile: "/kh", + Images: map[string]provider.KVMImage{"debian-13": {URL: testImageURL}}, + MachineTypes: map[string]provider.KVMMachineType{"small": {VCPUs: 2, MemoryMiB: 2048}}, + } + for _, h := range hosts { + cfg.Hosts = append(cfg.Hosts, provider.KVMHost{Name: h, Address: h + ":22", User: "virt", Bridge: "br0"}) + } + return provider.ProviderConfig{Name: "kvm-home", Type: "kvm", KVM: cfg} +} + +func newTestProvider(fakes hostFakes, hosts ...string) *Provider { + p := newWithDialer(testConfig(hosts...), fakes.dial, nil) + p.now = func() time.Time { return testNow } + return p +} + +// cacheImage pre-populates the pool with the configured image and its +// marker so Create never touches the network. +func cacheImage(api *fakeAPI) { + src := imageSource{Key: "debian-13", URL: testImageURL, Format: formatQcow2} + api.putVol("default", src.volumeName(), formatQcow2, []byte("qcow2")) + api.putVol("default", src.markerName(), formatRaw, []byte(testImageSHA())) +} + +func readyHost() *fakeAPI { + api := newFakeAPI("default") + cacheImage(api) + return api +} + +func testRequest() provider.CreateRequest { + return provider.CreateRequest{ + Name: testName, + UID: testUID, + Namespace: "default", + ProxyName: "home-proxy-1", + Placement: provider.Placement{Zone: "nas1", MachineType: "small", Image: "debian-13"}, + CloudInit: testCloudInit, + Port: 3128, + } +} + +func metadataXML(t *testing.T, uid string, created time.Time) string { + t.Helper() + out, err := xml.Marshal(newInstanceMetadata(uid, created)) + if err != nil { + t.Fatal(err) + } + return string(out) +} + +func ipv4(addr string) libvirt.DomainIPAddr { + return libvirt.DomainIPAddr{Type: int32(libvirt.IPAddrTypeIpv4), Addr: addr, Prefix: 24} +} + +func ipv6(addr string) libvirt.DomainIPAddr { + return libvirt.DomainIPAddr{Type: int32(libvirt.IPAddrTypeIpv6), Addr: addr, Prefix: 64} +} + +// managedDomain seeds a domain as Create would have left it. +func managedDomain(t *testing.T, api *fakeAPI, name, uid string, running bool, addrs map[uint32][]libvirt.DomainInterface) { + t.Helper() + api.putDomain(&fakeDomain{name: name, metadata: metadataXML(t, uid, testNow), running: running, addrs: addrs}) + api.putVol("default", overlayVolumeName(name), formatQcow2, nil) + api.putVol("default", seedVolumeName(name), formatRaw, []byte("iso")) +} + +func wantClass(t *testing.T, err error, class error) { + t.Helper() + if err == nil { + t.Fatalf("got nil error, want %v", class) + } + if !errors.Is(err, class) { + t.Fatalf("error class = %v, want %v (err: %v)", provider.Class(err), class, err) + } +} + +func assertCreated(t *testing.T, api *fakeAPI) { + t.Helper() + dom := api.domain(testName) + if dom == nil { + t.Fatal("domain not defined") + } + if !dom.running { + t.Error("domain not running") + } + overlay := api.vol("default", overlayVolumeName(testName)) + if overlay == nil { + t.Fatal("overlay volume missing") + } + if overlay.backing != "/var/lib/libvirt/images/base-debian-13.qcow2" || overlay.format != formatQcow2 { + t.Errorf("overlay = %+v, want qcow2 backed by the cached base", overlay) + } + if overlay.capacity != defaultDiskSizeGB*bytesPerGiB { + t.Errorf("overlay capacity = %d, want %d", overlay.capacity, defaultDiskSizeGB*bytesPerGiB) + } + seed := api.vol("default", seedVolumeName(testName)) + if seed == nil { + t.Fatal("seed volume missing") + } + if seed.format != formatRaw || len(seed.data) == 0 || int64(len(seed.data)) != seed.capacity { + t.Errorf("seed = format %s, %d bytes, capacity %d; want raw with capacity == uploaded size", seed.format, len(seed.data), seed.capacity) + } +} + +func TestCreate_freshHost(t *testing.T) { + t.Parallel() + api := readyHost() + p := newTestProvider(hostFakes{"nas1": api}, "nas1") + + id, err := p.Create(context.Background(), testRequest()) + if err != nil { + t.Fatalf("Create: %v", err) + } + if want := "hosts/nas1/domains/" + testName; id != want { + t.Errorf("providerID = %s, want %s", id, want) + } + assertCreated(t, api) + + var d domainXML + if err := xml.Unmarshal([]byte(api.domain(testName).xml), &d); err != nil { + t.Fatal(err) + } + if d.VCPU != 2 || d.Memory.Value != 2048 || d.Memory.Unit != "MiB" { + t.Errorf("domain size = %d vcpu / %d %s, want 2 / 2048 MiB", d.VCPU, d.Memory.Value, d.Memory.Unit) + } + if d.Devices.Interfaces[0].Source.Bridge != "br0" { + t.Errorf("bridge = %q, want br0", d.Devices.Interfaces[0].Source.Bridge) + } + if m := d.Metadata.Instance; m.UID != testUID || !m.managed() || !m.createdAt().Equal(testNow) { + t.Errorf("metadata = %+v, want managed, uid %s, createdAt %s", m, testUID, testNow) + } + if api.hasCall("StorageVolDownload") || api.callCount("StorageVolUpload") != 1 { + t.Errorf("calls = %v, want no image download and exactly one upload (the seed)", api.calls) + } +} + +func TestCreate_fetchesImageOnFirstUse(t *testing.T) { + t.Parallel() + srv := newImageServer(t) + api := newFakeAPI("default") + cfg := testConfig("nas1") + cfg.KVM.Images["debian-13"] = provider.KVMImage{URL: srv.URL + "/debian.qcow2", SHA256: testImageSHA()} + p := newWithDialer(cfg, hostFakes{"nas1": api}.dial, nil) + + if _, err := p.Create(context.Background(), testRequest()); err != nil { + t.Fatalf("Create: %v", err) + } + if srv.hits.Load() != 1 { + t.Errorf("image server hits = %d, want 1", srv.hits.Load()) + } + base := api.vol("default", "base-debian-13.qcow2") + if base == nil || string(base.data) != string(testImage) { + t.Fatal("base image not cached in the pool") + } + if api.domain(testName) == nil || !api.domain(testName).running { + t.Error("domain not running after image fetch") + } + + // A second Proxy with the same image on the same host reuses the cache. + req := testRequest() + req.Name, req.UID = "proxy-second", "uid-2" + if _, err := p.Create(context.Background(), req); err != nil { + t.Fatalf("second Create: %v", err) + } + if srv.hits.Load() != 1 { + t.Errorf("image server hits after second Create = %d, want still 1", srv.hits.Load()) + } +} + +func TestCreate_isIdempotentWhenComplete(t *testing.T) { + t.Parallel() + api := readyHost() + p := newTestProvider(hostFakes{"nas1": api}, "nas1") + if _, err := p.Create(context.Background(), testRequest()); err != nil { + t.Fatal(err) + } + before := len(api.calls) + + id, err := p.Create(context.Background(), testRequest()) + if err != nil { + t.Fatalf("second Create: %v", err) + } + if id != "hosts/nas1/domains/"+testName { + t.Errorf("providerID = %s", id) + } + for _, m := range []string{"DomainDefineXML", "DomainCreate", "StorageVolUpload"} { + if n := api.callCount(m); n != 1 { + t.Errorf("%s called %d times across two Creates, want 1", m, n) + } + } + if len(api.calls) == before { + t.Error("second Create made no calls at all; expected lookups") + } +} + +// Each case breaks Create after a different step, then re-runs it and +// expects the second attempt to pick up where the first stopped without +// duplicating what already exists. +func TestCreate_resumesAfterPartialFailure(t *testing.T) { + t.Parallel() + cases := []struct { + name string + sabotage func(api *fakeAPI) + class error + check func(t *testing.T, api *fakeAPI) + }{ + { + name: "seed upload fails after overlay exists", + sabotage: func(api *fakeAPI) { + api.uploadHook = func(vol string, _ io.Reader) error { + api.uploadHook = nil + return io.ErrUnexpectedEOF + } + }, + class: provider.ErrTransient, + check: func(t *testing.T, api *fakeAPI) { + if n := api.callCount("StorageVolUpload"); n != 2 { + t.Errorf("seed uploads = %d, want 2 (failed, then redone)", n) + } + if n := api.callCount("StorageVolCreateXML"); n != 4 { + t.Errorf("volume creates = %d, want 4 (overlay, seed; overlay rejected as existing, seed again)", n) + } + if names := api.volNames("default"); len(names) != 4 { + t.Errorf("pool holds %v, want base, marker, one overlay, one seed", names) + } + }, + }, + { + name: "define fails after volumes exist", + sabotage: func(api *fakeAPI) { api.failOnce["DomainDefineXML"] = lvError(libvirt.ErrInternalError, "boom") }, + class: provider.ErrTransient, + check: func(t *testing.T, api *fakeAPI) { + if n := api.callCount("StorageVolUpload"); n != 2 { + t.Errorf("seed uploads = %d, want 2 (seed rebuilt while no domain exists)", n) + } + if n := api.callCount("DomainDefineXML"); n != 2 { + t.Errorf("defines = %d, want 2", n) + } + }, + }, + { + name: "start fails after define", + sabotage: func(api *fakeAPI) { api.failOnce["DomainCreate"] = lvError(libvirt.ErrInternalError, "boom") }, + class: provider.ErrTransient, + check: func(t *testing.T, api *fakeAPI) { + if n := api.callCount("DomainDefineXML"); n != 1 { + t.Errorf("defines = %d, want 1 (existing domain reused)", n) + } + if n := api.callCount("DomainCreate"); n != 2 { + t.Errorf("starts = %d, want 2", n) + } + }, + }, + { + name: "transport drops mid-create", + sabotage: func(api *fakeAPI) { api.failOnce["DomainLookupByName"] = io.EOF }, + class: provider.ErrTransient, + check: func(t *testing.T, api *fakeAPI) { + if n := api.callCount("DomainDefineXML"); n != 1 { + t.Errorf("defines = %d, want 1", n) + } + }, + }, + } + for _, tc := range cases { + t.Run(tc.name, func(t *testing.T) { + t.Parallel() + api := readyHost() + tc.sabotage(api) + p := newTestProvider(hostFakes{"nas1": api}, "nas1") + + _, err := p.Create(context.Background(), testRequest()) + wantClass(t, err, tc.class) + if _, err := p.Create(context.Background(), testRequest()); err != nil { + t.Fatalf("retry Create: %v", err) + } + assertCreated(t, api) + tc.check(t, api) + }) + } +} + +// A reconcile cancelled mid-upload leaves a seed volume whose content is +// short; the compensating delete cannot run because the pool has already +// closed the connection. The next Create must not trust that volume. +func TestCreate_replacesLeftoverSeed(t *testing.T) { + t.Parallel() + api := readyHost() + api.putVol("default", seedVolumeName(testName), formatRaw, []byte("trunc")) + p := newTestProvider(hostFakes{"nas1": api}, "nas1") + + if _, err := p.Create(context.Background(), testRequest()); err != nil { + t.Fatal(err) + } + assertCreated(t, api) + if seed := api.vol("default", seedVolumeName(testName)); string(seed.data) == "trunc" { + t.Error("leftover seed was kept") + } + if n := api.callCount("StorageVolUpload"); n != 1 { + t.Errorf("uploads = %d, want 1 (the rebuilt seed)", n) + } +} + +func TestCreate_startsStoppedDomain(t *testing.T) { + t.Parallel() + api := readyHost() + managedDomain(t, api, testName, testUID, false, nil) + p := newTestProvider(hostFakes{"nas1": api}, "nas1") + + if _, err := p.Create(context.Background(), testRequest()); err != nil { + t.Fatal(err) + } + if !api.domain(testName).running { + t.Error("domain still stopped") + } + if api.hasCall("DomainDefineXML") || api.hasCall("StorageVolUpload") || api.hasCall("StorageVolDelete") { + t.Errorf("calls = %v, want no define, upload or volume delete for an existing domain", api.calls) + } +} + +func TestCreate_singleHostNeedsNoZone(t *testing.T) { + t.Parallel() + api := readyHost() + p := newTestProvider(hostFakes{"nas1": api}, "nas1") + req := testRequest() + req.Placement.Zone = "" + + id, err := p.Create(context.Background(), req) + if err != nil { + t.Fatal(err) + } + if id != "hosts/nas1/domains/"+testName { + t.Errorf("providerID = %s, want the single host filled in", id) + } +} + +func TestCreate_placementErrors(t *testing.T) { + t.Parallel() + cases := []struct { + name string + hosts []string + edit func(*provider.CreateRequest) + want string + }{ + {"unknown host", []string{"nas1"}, func(r *provider.CreateRequest) { r.Placement.Zone = "nas9" }, "not in kvm.hosts"}, + {"empty zone with two hosts", []string{"nas1", "nas2"}, func(r *provider.CreateRequest) { r.Placement.Zone = "" }, "placement.zone is required"}, + {"empty zone with no hosts", nil, func(r *provider.CreateRequest) { r.Placement.Zone = "" }, "kvm.hosts is empty"}, + {"unknown machine type", []string{"nas1"}, func(r *provider.CreateRequest) { r.Placement.MachineType = "huge" }, "not in kvm.machineTypes"}, + {"empty machine type", []string{"nas1"}, func(r *provider.CreateRequest) { r.Placement.MachineType = "" }, "not in kvm.machineTypes"}, + {"unknown image", []string{"nas1"}, func(r *provider.CreateRequest) { r.Placement.Image = "windows" }, "neither a key"}, + } + for _, tc := range cases { + t.Run(tc.name, func(t *testing.T) { + t.Parallel() + fakes := hostFakes{} + for _, h := range tc.hosts { + fakes[h] = readyHost() + } + p := newTestProvider(fakes, tc.hosts...) + req := testRequest() + tc.edit(&req) + + _, err := p.Create(context.Background(), req) + wantClass(t, err, provider.ErrPermanent) + if !strings.Contains(err.Error(), tc.want) { + t.Errorf("error = %q, want it to mention %q", err, tc.want) + } + for _, api := range fakes { + if len(api.calls) != 0 { + t.Errorf("host was called: %v", api.calls) + } + } + }) + } +} + +func TestCreate_hostFailures(t *testing.T) { + t.Parallel() + cases := []struct { + name string + setup func() (hostFakes, string) + class error + }{ + { + name: "host unreachable", + setup: func() (hostFakes, string) { return hostFakes{}, "nas1" }, + class: provider.ErrTransient, + }, + { + name: "pool missing", + setup: func() (hostFakes, string) { + api := newFakeAPI("other") + return hostFakes{"nas1": api}, "nas1" + }, + class: provider.ErrPermanent, + }, + { + name: "domain xml rejected", + setup: func() (hostFakes, string) { + api := readyHost() + api.fail["DomainDefineXML"] = lvError(libvirt.ErrConfigUnsupported, "no q35 here") + return hostFakes{"nas1": api}, "nas1" + }, + class: provider.ErrPermanent, + }, + } + for _, tc := range cases { + t.Run(tc.name, func(t *testing.T) { + t.Parallel() + fakes, host := tc.setup() + p := newTestProvider(fakes, host) + _, err := p.Create(context.Background(), testRequest()) + wantClass(t, err, tc.class) + }) + } +} + +func TestCreate_cancelledContext(t *testing.T) { + t.Parallel() + api := readyHost() + p := newTestProvider(hostFakes{"nas1": api}, "nas1") + ctx, cancel := context.WithCancel(context.Background()) + cancel() + + _, err := p.Create(ctx, testRequest()) + wantClass(t, err, provider.ErrTransient) + if !errors.Is(err, context.Canceled) { + t.Errorf("error = %v, want it to unwrap to context.Canceled", err) + } +} + +func TestGet(t *testing.T) { + t.Parallel() + arp := map[uint32][]libvirt.DomainInterface{ + uint32(libvirt.DomainInterfaceAddressesSrcArp): {{Name: "vnet0", Addrs: []libvirt.DomainIPAddr{ipv4("192.168.1.50")}}}, + } + agentOnly := map[uint32][]libvirt.DomainInterface{ + uint32(libvirt.DomainInterfaceAddressesSrcAgent): { + {Name: "lo", Addrs: []libvirt.DomainIPAddr{ipv4("127.0.0.1"), ipv6("::1")}}, + {Name: "enp1s0", Addrs: []libvirt.DomainIPAddr{ipv6("fe80::1"), ipv4("192.168.1.51")}}, + }, + } + linkLocalOnly := map[uint32][]libvirt.DomainInterface{ + uint32(libvirt.DomainInterfaceAddressesSrcAgent): { + {Name: "enp1s0", Addrs: []libvirt.DomainIPAddr{ipv4("169.254.10.10")}}, + }, + } + cases := []struct { + name string + running bool + addrs map[uint32][]libvirt.DomainInterface + state provider.InstanceState + ip string + }{ + {"running with arp address", true, arp, provider.StateRunning, "192.168.1.50"}, + {"running, agent only, skips loopback and v6", true, agentOnly, provider.StateRunning, "192.168.1.51"}, + {"running, no address yet", true, nil, provider.StateProvisioning, ""}, + {"running, only link-local", true, linkLocalOnly, provider.StateProvisioning, ""}, + {"shut off", false, arp, provider.StateStopped, ""}, + } + for _, tc := range cases { + t.Run(tc.name, func(t *testing.T) { + t.Parallel() + api := readyHost() + managedDomain(t, api, testName, testUID, tc.running, tc.addrs) + p := newTestProvider(hostFakes{"nas1": api}, "nas1") + + inst, err := p.Get(context.Background(), "hosts/nas1/domains/"+testName) + if err != nil { + t.Fatalf("Get: %v", err) + } + want := provider.Instance{ID: "hosts/nas1/domains/" + testName, IP: tc.ip, State: tc.state, UID: testUID, CreatedAt: testNow} + if *inst != want { + t.Errorf("instance = %+v, want %+v", *inst, want) + } + }) + } +} + +func TestGet_unmanagedDomainHasNoUID(t *testing.T) { + t.Parallel() + api := readyHost() + api.putDomain(&fakeDomain{name: "hand-made", running: true}) + p := newTestProvider(hostFakes{"nas1": api}, "nas1") + + inst, err := p.Get(context.Background(), "hosts/nas1/domains/hand-made") + if err != nil { + t.Fatalf("Get: %v", err) + } + if inst.UID != "" || !inst.CreatedAt.IsZero() { + t.Errorf("instance = %+v, want empty UID and zero CreatedAt", inst) + } +} + +func TestGet_errors(t *testing.T) { + t.Parallel() + cases := []struct { + name string + id string + setup func(t *testing.T, api *fakeAPI) + class error + }{ + {"missing domain", "hosts/nas1/domains/" + testName, func(*testing.T, *fakeAPI) {}, provider.ErrNotFound}, + {"malformed id", "zones/x/instances/y", func(*testing.T, *fakeAPI) {}, provider.ErrPermanent}, + {"host not configured", "hosts/nas9/domains/" + testName, func(*testing.T, *fakeAPI) {}, provider.ErrPermanent}, + { + "state read fails", "hosts/nas1/domains/" + testName, + func(t *testing.T, api *fakeAPI) { + managedDomain(t, api, testName, testUID, true, nil) + api.fail["DomainGetState"] = lvError(libvirt.ErrInternalError, "boom") + }, + provider.ErrTransient, + }, + { + "corrupt metadata", "hosts/nas1/domains/" + testName, + func(_ *testing.T, api *fakeAPI) { + api.putDomain(&fakeDomain{name: testName, metadata: "false`}) + p := newTestProvider(hostFakes{"nas1": nas1, "nas2": nas2}, "nas1", "nas2") + + got, err := p.ListByTag(context.Background()) + if err != nil { + t.Fatalf("ListByTag: %v", err) + } + want := []provider.Instance{ + {ID: "hosts/nas1/domains/proxy-one", IP: "192.168.1.10", State: provider.StateRunning, UID: "uid-1", CreatedAt: testNow}, + {ID: "hosts/nas2/domains/proxy-two", State: provider.StateStopped, UID: "uid-2", CreatedAt: testNow}, + } + if len(got) != len(want) { + t.Fatalf("got %d instances %+v, want %d", len(got), got, len(want)) + } + for i := range want { + if got[i] != want[i] { + t.Errorf("instance[%d] = %+v, want %+v", i, got[i], want[i]) + } + } +} + +func TestListByTag_oneHostDownFailsSweep(t *testing.T) { + t.Parallel() + nas1 := readyHost() + managedDomain(t, nas1, "proxy-one", "uid-1", true, nil) + p := newTestProvider(hostFakes{"nas1": nas1}, "nas1", "nas2") + + got, err := p.ListByTag(context.Background()) + wantClass(t, err, provider.ErrTransient) + if got != nil { + t.Errorf("got %+v alongside the error, want nil (no partial view)", got) + } +} + +// A domain that cannot be read is left out of the sweep rather than +// failing it: GC never deletes what it does not see, so skipping is the +// safe direction, while one bad domain blocking GC for all others is not. +func TestListByTag_skipsUnreadableDomain(t *testing.T) { + t.Parallel() + cases := []struct { + name string + setup func(t *testing.T, api *fakeAPI) + }{ + { + "deleted mid-sweep", + func(t *testing.T, api *fakeAPI) { + managedDomain(t, api, "proxy-one", "uid-1", true, nil) + api.failOnce["DomainGetMetadata"] = lvError(libvirt.ErrNoDomain, "gone") + }, + }, + { + "corrupt metadata", + func(_ *testing.T, api *fakeAPI) { + api.putDomain(&fakeDomain{name: "proxy-one", metadata: " Date: Tue, 8 Sep 2026 10:57:05 +0000 Subject: [PATCH 07/28] Add KVM provider integration test against a real hypervisor Behind -tags=integration and self-skipping unless KVM_TEST_* names a host: one sequential lifecycle (image fetch, Create, idempotent re-Create, Running with IP, ListByTag, Delete, pool state afterwards), driven through the provider directly so it needs no Step 5 wiring. Adds `make test-integration-kvm` and documents the run and the deviation from the testcontainers convention in docs/testing.md. Co-Authored-By: Claude Fable 5.1 --- Makefile | 4 + .../2026-09-04-1404-kvm-provider.md | 57 +++- docs/testing.md | 49 +++ internal/provider/kvm/kvm_integration_test.go | 291 ++++++++++++++++++ 4 files changed, 400 insertions(+), 1 deletion(-) create mode 100644 internal/provider/kvm/kvm_integration_test.go diff --git a/Makefile b/Makefile index d348f5d..b621dac 100644 --- a/Makefile +++ b/Makefile @@ -66,6 +66,10 @@ vet: ## Run go vet against code. test: manifests generate fmt vet setup-envtest ## Run tests. KUBEBUILDER_ASSETS="$(shell "$(ENVTEST)" use $(ENVTEST_K8S_VERSION) --bin-dir "$(LOCALBIN)" -p path)" go test -race $$(go list ./... | grep -v /e2e) -coverprofile cover.out +.PHONY: test-integration-kvm +test-integration-kvm: ## Run the KVM provider test against a real hypervisor. Needs KVM_TEST_HOST/USER/SSH_KEY/BRIDGE/IMAGE_URL; see docs/testing.md. + go test -race -tags=integration -count=1 -v -timeout 40m -run TestKVM ./internal/provider/kvm/ + # TODO(user): To use a different vendor for e2e tests, modify the setup under 'tests/e2e'. # The default setup assumes Kind is pre-installed and builds/loads the Manager Docker image locally. # kubectl kuberc is disabled by default for test isolation; enable with: diff --git a/docs/plans-executions/2026-09-04-1404-kvm-provider.md b/docs/plans-executions/2026-09-04-1404-kvm-provider.md index 02da6f9..215080a 100644 --- a/docs/plans-executions/2026-09-04-1404-kvm-provider.md +++ b/docs/plans-executions/2026-09-04-1404-kvm-provider.md @@ -9,7 +9,7 @@ Plan: [2026-09-04-1404-kvm-provider.md](../plans/2026-09-04-1404-kvm-provider.md - [x] Step 3b — Image cache (`image.go`) - [x] Step 4 — Provider methods (`kvm.go`) over a fake API - [ ] Step 5 — Wiring, manifests, samples, docs, CHANGELOG -- [ ] Step 6 — Integration test against a real hypervisor +- [x] Step 6 — Integration test against a real hypervisor (written and compiled; first real run is part of Step 7) - [ ] Step 7 — Manual validation + MR ## Step 0 — Save plan + branch @@ -283,3 +283,58 @@ uncovered lines are the real SSH/libvirt dial path and log-only branches. `go vet` and `-race` (three runs) are clean. Wiring into `cmd/main.go` is Step 5, so `New` is not yet reachable from a running operator. + +## Step 6 — Integration test + +Done before Step 5, at the user's request, after a discussion of what the +unit tests can and cannot prove: they run over an in-memory fake of the +libvirt calls, so every assumption about libvirt's real behaviour (domain +XML accepted, raw qcow2 upload + pool refresh usable as a backing image, +guest boots and reports an address) needed a test against a real host. +The wiring in Step 5 is not a prerequisite — the test drives the provider +directly. + +`internal/provider/kvm/kvm_integration_test.go` is behind +`//go:build integration` and skips itself under `-short` or when any of +`KVM_TEST_HOST`, `KVM_TEST_USER`, `KVM_TEST_SSH_KEY`, `KVM_TEST_BRIDGE`, +`KVM_TEST_IMAGE_URL` is unset. Optional knobs beyond the plan's list: +`KVM_TEST_KNOWN_HOSTS` (without it the run disables host-key checking +and logs that), `KVM_TEST_IMAGE_SHA256`, `KVM_TEST_POOL`, +`KVM_TEST_SOCKET`, `KVM_TEST_TIMEOUT` (default 20 m, must cover a cold +image download) and `KVM_TEST_KEEP` to leave a failed run's VM behind for +`virsh console`. + +One sequential test, `TestKVM_lifecycle`, not subtests: each stage +depends on hypervisor state the previous one left. Create (timed, so the +log shows the image fetch), re-Create returns the same ID, poll Get to +Running with an IPv4 (Stopped fails fast with a hint to use the console), +UID and CreatedAt survive the metadata round-trip, ListByTag includes it, +Delete, Get → NotFound, second Delete is a no-op, ListByTag excludes it, +and the pool is inspected through the provider's own connection pool +(`p.hosts.call`, reachable because the test is in package `kvm`): overlay +and seed gone, base image and marker kept. A TCP connect to the guest's +port 22 is attempted but only logged, since the test machine may not +route to the bridge network. Cleanup deletes the VM on failure with its +own two-minute budget, because the test context may already be spent. + +Makefile target, so the invocation is not something to remember: + +```bash +make test-integration-kvm +# = go test -race -tags=integration -count=1 -v -timeout 40m -run TestKVM ./internal/provider/kvm/ +``` + +`docs/testing.md` gained the kvm package rows in the suite table, a +"live libvirt" bullet under the deliberately-not-covered list, and a +section on running this test, with the deviation from the testcontainers +convention called out as the plan asked. + +Worth noting: **the test has not been executed yet.** This session had +no hypervisor credentials; it was verified with +`go vet -tags=integration ./...` and by running it without the +environment (self-skip). Its first real run is the opening act of +Step 7's manual validation, and it is the point at which the Step 2 and +Step 3b "confirm on a real host" notes get answered. The test's config +is built directly rather than through the provider package's validation +(unexported), so a typo in an env var surfaces from `New` or from +libvirt, not from a validation message. diff --git a/docs/testing.md b/docs/testing.md index 03fee39..5ee1633 100644 --- a/docs/testing.md +++ b/docs/testing.md @@ -54,6 +54,10 @@ manual runs too. | | `metrics_test.go` | `WithMetrics` decorator: classified result labels, error passthrough | fake recorder + static provider | | `internal/provider/kubernetes` | `kubernetes_test.go`, `pod_test.go` | Create/Get/Delete/ListByTag over real Pod objects; pure `buildPod`/`squidConf` incl. the `max_filedescriptors` OOM-regression assertion | `client/fake` (real `corev1.Pod`s, no kubelet) | | `internal/provider/gcp` | `insert_test.go`, `errors_test.go`, `gcp_test.go` | field-by-field `buildInsertRequest`; HTTP-code classification table; zone-qualified IDs, 409-is-success, 9-row state mapping, ListByTag filter + `ReturnPartialSuccess` | pure builder needs **no fake**; the rest uses the flattened `instancesAPI` seam | +| `internal/provider/kvm` | `domainxml_test.go`, `volume_test.go`, `seed_test.go`, `id_test.go`, `metadata_test.go` | pure builders: domain/volume XML unmarshalled back and checked field by field, seed ISO reopened with the same library, providerID round-trip, metadata in both namespace spellings | pure | +| | `errors_test.go`, `conn_test.go` | libvirt-code classification table; connection pool (reuse, redial, drop on transport error, cancel releases a blocked RPC, one dial per host under concurrency); SSH dial, host-key policy, keepalive | fake dialer; a **real in-process `x/crypto/ssh` server** forwarding a unix socket | +| | `image_test.go`, `kvm_test.go` | image cache (fresh fetch, ready, marker missing, checksum change/mismatch, 4xx/5xx, concurrent callers); Create resume after every partial step, leftover truncated seed, state/IP mapping, every Delete already-gone case, ListByTag across hosts / one down / unreadable domain | `httptest` image server + in-memory `fakeAPI` behind the `libvirtAPI` seam | +| | `kvm_integration_test.go` (`-tags=integration`) | one real VM: image fetch, Create, idempotent re-Create, Running with IP, ListByTag, Delete, pool state afterwards | a **real hypervisor** — see below | | `internal/health` | `probe_test.go` | probes through a **real CONNECT-capable proxy stub** to a real TLS server: tunnel success, refused CONNECT, wrong status, dead proxy, plain-http forward | `httptest` + hijacked bidirectional tunnel | | | `engine_test.go` | thresholds, first-verdict, latency suppression matrix, dropped-event retry, stale-probe UID guard, tick scheduling/seeding/pruning, `Start` end-to-end | fake `client.Reader` + injected `probeFn` | | `internal/lease` | `store_test.go` | capacity, `MaxLeases=0`, all three selection tie-breaks, target-scoped vs global cooldowns, expiry via fake clock, report-on-retained-lease, idempotent release, **40 concurrent acquires never exceeding capacity** | injectable clock; the concurrency case is why `-race` matters | @@ -83,6 +87,51 @@ envtest suite the scaffold generated. bug (below). - **Live GCP**: no test talks to Google. The seam boundary (`buildInsertRequest` + classification) is tested exhaustively instead. +- **Live libvirt, by default**: the KVM unit tests run over an in-memory + fake of the 17 libvirt calls the provider uses. Whether libvirt accepts + the domain XML, whether a raw qcow2 upload plus pool refresh yields a + usable backing image, and whether the guest actually boots and reports + an address are provable only on a real host — that is the opt-in + integration test below. + +## The KVM integration test (real hypervisor) + +`internal/provider/kvm/kvm_integration_test.go` compiles only under +`-tags=integration` and skips itself unless the environment names a +host. It deviates from the "testcontainers for real infra" convention on +purpose: libvirt needs `/dev/kvm` and a bridge to a DHCP network, and a +container has neither. So it targets whichever hypervisor you point it +at and leaves that host's pool with one more cached base image than it +had. + +```sh +export KVM_TEST_HOST=nas1.home.lan:22 # SSH endpoint +export KVM_TEST_USER=virt # must be in the libvirt group +export KVM_TEST_SSH_KEY=$HOME/.ssh/id_ed25519 # private key for that user +export KVM_TEST_BRIDGE=br0 # bridge to a network with DHCP +export KVM_TEST_IMAGE_URL=https://cloud.debian.org/images/cloud/trixie/latest/debian-13-generic-amd64.qcow2 +make test-integration-kvm +``` + +Optional: `KVM_TEST_KNOWN_HOSTS` (without it host keys are **not** +verified — the log says so), `KVM_TEST_IMAGE_SHA256`, `KVM_TEST_POOL` +(default `default`), `KVM_TEST_SOCKET`, `KVM_TEST_TIMEOUT` (default +`20m`, must cover the first-time image download), and `KVM_TEST_KEEP` +to leave the VM behind for `virsh console` when something goes wrong. + +What one run asserts, in order: `New` dials the host; `Create` returns a +`hosts//domains/` ID (downloading the image on a cold host — +the log prints how long it took); a second `Create` returns the same ID; +`Get` reaches `Running` with an IPv4 within the budget, and the UID and +`CreatedAt` survive the metadata round-trip through libvirt; `ListByTag` +lists it; `Delete` removes it, `Get` says `ErrNotFound`, a second +`Delete` is a no-op, and the pool holds the base image and its marker +but no per-VM overlay or seed. It also tries a TCP connect to port 22 on +the guest, but only logs the outcome — the machine running the test may +not route to the bridge network. + +The test is not part of `make test` and not run in CI. Keep it compiling +with `go vet -tags=integration ./...` when the provider's internals move. ## The scaffolded `make test-e2e` suite diff --git a/internal/provider/kvm/kvm_integration_test.go b/internal/provider/kvm/kvm_integration_test.go new file mode 100644 index 0000000..1e1faf7 --- /dev/null +++ b/internal/provider/kvm/kvm_integration_test.go @@ -0,0 +1,291 @@ +//go:build integration + +package kvm + +// Integration test against a real libvirt/KVM hypervisor. It deviates from +// the repo's testcontainers convention on purpose: libvirt needs /dev/kvm +// and a bridge, neither of which a container provides, so this runs +// against whatever host the KVM_TEST_* variables name. It creates one VM +// from a stock cloud image, waits for it to report an address, deletes it, +// and checks the pool afterwards. The base image stays cached on the host. +// +// Required: +// +// KVM_TEST_HOST SSH endpoint, host:port +// KVM_TEST_USER SSH user (member of the libvirt group) +// KVM_TEST_SSH_KEY path to the private key +// KVM_TEST_BRIDGE host bridge for the VM's NIC +// KVM_TEST_IMAGE_URL cloud image to fetch (http/https) +// +// Optional: +// +// KVM_TEST_KNOWN_HOSTS known_hosts file; without it host keys are NOT verified +// KVM_TEST_IMAGE_SHA256 expected digest of the image +// KVM_TEST_POOL storage pool (default "default") +// KVM_TEST_SOCKET libvirt socket path on the host +// KVM_TEST_TIMEOUT overall budget incl. the image download (default 20m) +// KVM_TEST_KEEP set to anything to leave the VM behind for inspection + +import ( + "context" + "crypto/rand" + "encoding/hex" + "errors" + "fmt" + "net" + "os" + "testing" + "time" + + libvirt "github.com/digitalocean/go-libvirt" + "k8s.io/apimachinery/pkg/types" + + "gitea.home.hrajfrisbee.cz/kacerr/egress-proxies-operator/internal/provider" +) + +const ( + integrationHostName = "kvm-test" + integrationImageKey = "test-image" + integrationPoll = 5 * time.Second +) + +type integrationEnv struct { + cfg provider.ProviderConfig + timeout time.Duration + keep bool +} + +func loadIntegrationEnv(t *testing.T) integrationEnv { + t.Helper() + if testing.Short() { + t.Skip("integration test skipped under -short") + } + required := map[string]string{} + for _, key := range []string{"KVM_TEST_HOST", "KVM_TEST_USER", "KVM_TEST_SSH_KEY", "KVM_TEST_BRIDGE", "KVM_TEST_IMAGE_URL"} { + v := os.Getenv(key) + if v == "" { + t.Skipf("%s not set; skipping KVM integration test", key) + } + required[key] = v + } + cfg := &provider.KVMConfig{ + SSHKeyFile: required["KVM_TEST_SSH_KEY"], + KnownHostsFile: os.Getenv("KVM_TEST_KNOWN_HOSTS"), + Images: map[string]provider.KVMImage{ + integrationImageKey: {URL: required["KVM_TEST_IMAGE_URL"], SHA256: os.Getenv("KVM_TEST_IMAGE_SHA256")}, + }, + MachineTypes: map[string]provider.KVMMachineType{"small": {VCPUs: 1, MemoryMiB: 1024}}, + Hosts: []provider.KVMHost{{ + Name: integrationHostName, + Address: required["KVM_TEST_HOST"], + User: required["KVM_TEST_USER"], + Socket: os.Getenv("KVM_TEST_SOCKET"), + Pool: os.Getenv("KVM_TEST_POOL"), + Bridge: required["KVM_TEST_BRIDGE"], + }}, + } + if cfg.KnownHostsFile == "" { + cfg.InsecureIgnoreHostKey = true + t.Log("KVM_TEST_KNOWN_HOSTS not set: host key verification disabled for this run") + } + pc := provider.ProviderConfig{Name: "kvm-integration", Type: "kvm", KVM: cfg} + timeout := 20 * time.Minute + if v := os.Getenv("KVM_TEST_TIMEOUT"); v != "" { + d, err := time.ParseDuration(v) + if err != nil { + t.Fatalf("KVM_TEST_TIMEOUT: %v", err) + } + timeout = d + } + return integrationEnv{cfg: pc, timeout: timeout, keep: os.Getenv("KVM_TEST_KEEP") != ""} +} + +func randomUID(t *testing.T) string { + t.Helper() + var b [16]byte + if _, err := rand.Read(b[:]); err != nil { + t.Fatal(err) + } + h := hex.EncodeToString(b[:]) + return fmt.Sprintf("%s-%s-%s-%s-%s", h[0:8], h[8:12], h[12:16], h[16:20], h[20:32]) +} + +// TestKVM_lifecycle is one sequential story, not parallel subtests: each +// stage depends on the state the previous one left on the hypervisor. +func TestKVM_lifecycle(t *testing.T) { + env := loadIntegrationEnv(t) + ctx, cancel := context.WithTimeout(context.Background(), env.timeout) + defer cancel() + + prov, err := New(ctx, env.cfg) + if err != nil { + t.Fatalf("New: %v", err) + } + p := prov.(*Provider) + host := p.cfg.Hosts[0] + + uid := randomUID(t) + req := provider.CreateRequest{ + Name: provider.NameFromUID(types.UID(uid)), + UID: uid, + Namespace: "integration", + ProxyName: "kvm-integration", + Placement: provider.Placement{Zone: integrationHostName, MachineType: "small", Image: integrationImageKey}, + CloudInit: "#cloud-config\n", + Port: 3128, + } + t.Logf("host %s (%s), pool %q, bridge %s, domain %s", host.Name, host.Address, host.Pool, host.Bridge, req.Name) + + // Cleanup runs even when a stage fails, with its own budget: the test + // ctx may already be spent by then. + t.Cleanup(func() { + if env.keep { + t.Logf("KVM_TEST_KEEP set: leaving domain %s and its volumes on %s", req.Name, host.Address) + return + } + cctx, ccancel := context.WithTimeout(context.Background(), 2*time.Minute) + defer ccancel() + if err := p.Delete(cctx, formatProviderID(host.Name, req.Name)); err != nil { + t.Errorf("cleanup Delete: %v", err) + } + }) + + // --- create (includes the image fetch on a cold host) --- + start := time.Now() + id, err := p.Create(ctx, req) + if err != nil { + t.Fatalf("Create: %v", err) + } + t.Logf("Create returned %s after %s", id, time.Since(start).Round(time.Second)) + if want := formatProviderID(host.Name, req.Name); id != want { + t.Fatalf("providerID = %s, want %s", id, want) + } + + // --- create again: must find what it made, not duplicate it --- + id2, err := p.Create(ctx, req) + if err != nil { + t.Fatalf("second Create: %v", err) + } + if id2 != id { + t.Fatalf("second Create returned %s, want %s", id2, id) + } + + // --- wait for the guest to come up with an address --- + inst := waitForRunning(ctx, t, p, id) + t.Logf("domain Running with IP %s (state reported after %s)", inst.IP, time.Since(start).Round(time.Second)) + if inst.UID != uid { + t.Errorf("UID read back = %q, want %q (metadata round-trip through libvirt)", inst.UID, uid) + } + if age := time.Since(inst.CreatedAt); inst.CreatedAt.IsZero() || age < 0 || age > time.Hour { + t.Errorf("CreatedAt = %s (age %s), want a recent timestamp", inst.CreatedAt, age) + } + if conn, err := net.DialTimeout("tcp", net.JoinHostPort(inst.IP, "22"), 5*time.Second); err != nil { + t.Logf("note: tcp %s:22 not reachable from this machine (%v); fine if the bridge network is not routed here", inst.IP, err) + } else { + conn.Close() + t.Logf("tcp %s:22 reachable from this machine", inst.IP) + } + + // --- the GC view must include it --- + listed, err := p.ListByTag(ctx) + if err != nil { + t.Fatalf("ListByTag: %v", err) + } + if !containsID(listed, id) { + t.Errorf("ListByTag = %+v, want it to include %s", listed, id) + } + + // --- delete, then verify nothing of the VM remains but the base image does --- + if err := p.Delete(ctx, id); err != nil { + t.Fatalf("Delete: %v", err) + } + if _, err := p.Get(ctx, id); !errors.Is(err, provider.ErrNotFound) { + t.Errorf("Get after Delete = %v, want ErrNotFound", err) + } + if err := p.Delete(ctx, id); err != nil { + t.Errorf("second Delete: %v, want nil (idempotent)", err) + } + listed, err = p.ListByTag(ctx) + if err != nil { + t.Fatalf("ListByTag after Delete: %v", err) + } + if containsID(listed, id) { + t.Errorf("ListByTag still includes %s after Delete", id) + } + + src, err := resolveImage(p.cfg, integrationImageKey) + if err != nil { + t.Fatal(err) + } + err = p.hosts.call(ctx, host, func(api libvirtAPI) error { + pool, err := api.StoragePoolLookupByName(host.Pool) + if err != nil { + return err + } + for _, gone := range []string{overlayVolumeName(req.Name), seedVolumeName(req.Name)} { + if _, err := api.StorageVolLookupByName(pool, gone); !hasCode(err, libvirt.ErrNoStorageVol) { + t.Errorf("volume %s after Delete: lookup err = %v, want ErrNoStorageVol", gone, err) + } + } + for _, kept := range []string{src.volumeName(), src.markerName()} { + if _, err := api.StorageVolLookupByName(pool, kept); err != nil { + t.Errorf("cached image volume %s: %v, want it kept after Delete", kept, err) + } + } + return nil + }) + if err != nil { + t.Fatalf("inspecting pool after Delete: %v", err) + } + // KVM_TEST_KEEP is for inspecting a failed run; a passed run has + // nothing left to keep, so let cleanup's Delete confirm that. + env.keep = false +} + +// waitForRunning polls Get until the instance is Running with an IP or ctx +// ends. Transient errors are retried; anything else fails the test. +func waitForRunning(ctx context.Context, t *testing.T, p *Provider, id string) *provider.Instance { + t.Helper() + var last provider.InstanceState + ticker := time.NewTicker(integrationPoll) + defer ticker.Stop() + for { + inst, err := p.Get(ctx, id) + switch { + case err == nil: + if inst.State != last { + t.Logf("state %s ip %q", inst.State, inst.IP) + last = inst.State + } + if inst.State == provider.StateRunning { + return inst + } + if inst.State == provider.StateStopped { + t.Fatalf("domain reported %s while waiting for Running; check `virsh console %s` on the host", inst.State, idName(id)) + } + case errors.Is(err, provider.ErrTransient): + t.Logf("Get: transient error, retrying: %v", err) + default: + t.Fatalf("Get: %v", err) + } + select { + case <-ctx.Done(): + t.Fatalf("timed out waiting for Running (last state %s): %v", last, ctx.Err()) + case <-ticker.C: + } + } +} + +func idName(id string) string { + _, name, _ := parseProviderID(id) + return name +} + +func containsID(list []provider.Instance, id string) bool { + for _, inst := range list { + if inst.ID == id { + return true + } + } + return false +} -- 2.54.0 From 791988b5a0392eb6f6d7853f0572256dc2abd690 Mon Sep 17 00:00:00 2001 From: kacerr Date: Tue, 8 Sep 2026 11:03:48 +0000 Subject: [PATCH 08/28] Wire the KVM provider: main.go, kvm-ssh Secret mount, samples, docs, CHANGELOG Registers "kvm" in the provider builtins, mounts an optional kvm-ssh Secret at /etc/proxy-operator/kvm, adds a kvm-home providers-config sample and a proxy_kvm.yaml Proxy sample, documents the provider in docs/kvm-provider.md (prerequisites, config reference, image cache and its do-not-replace-in-use rule, known gaps), and updates the README and architecture provider sections and decision log. Co-Authored-By: Claude Fable 5.1 --- CHANGELOG.md | 16 ++ README.md | 30 ++- cmd/main.go | 2 + config/manager/manager.yaml | 13 ++ config/manager/providers_config.yaml | 5 +- config/samples/kustomization.yaml | 1 + config/samples/providers-config.yaml | 33 +++ config/samples/proxy_kvm.yaml | 43 ++++ docs/architecture.md | 44 ++++ docs/kvm-provider.md | 214 ++++++++++++++++++ .../2026-09-04-1404-kvm-provider.md | 53 ++++- 11 files changed, 448 insertions(+), 6 deletions(-) create mode 100644 config/samples/proxy_kvm.yaml create mode 100644 docs/kvm-provider.md diff --git a/CHANGELOG.md b/CHANGELOG.md index 4fe773f..e82db70 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -1,5 +1,21 @@ # Changelog +## 2026-09-08 11:00 UTC — KVM/libvirt provider + +- New `kvm` provider type: Managed proxies as VMs on your own libvirt/KVM hosts. Each + host is reached over SSH with the libvirt unix socket forwarded through the + connection (pure Go, nothing to expose on the hypervisors beyond sshd); one provider + instance can span several hosts, selected by `placement.zone`. +- Base images (`images:` map or a direct URL in `placement.image`) are downloaded by the + operator into each host's storage pool on first use, verified by sha256 when configured, + and cached there. Per-VM disks are qcow2 overlays; cloud-init arrives on a NoCloud seed + ISO built in-process. GC identity lives in the domain's `` element. +- Wiring: `"kvm": kvm.New` in `cmd/main.go`; optional `kvm-ssh` Secret mounted at + `/etc/proxy-operator/kvm`; samples `config/samples/proxy_kvm.yaml` and a `kvm-home` + block in `providers-config.yaml`; `docs/kvm-provider.md`; opt-in integration test + (`make test-integration-kvm`) against a real hypervisor. +- Not yet validated on a real host — that is the next step on the branch. + ## 2026-08-10 09:34 CEST — kind e2e verified; fix Squid OOM in containers; quickstart goes in-cluster - Full end-to-end pass on a throwaway kind cluster: Squid pod Ready with a real CONNECT diff --git a/README.md b/README.md index a3a80dd..8dc4df5 100644 --- a/README.md +++ b/README.md @@ -15,9 +15,10 @@ itself**, and hands out to crawler clients via an HTTP list/lease API. cattle**: any meaningful spec change (placement, cloud-init, port) deletes and recreates the VM — never in-place mutation. - **Providers** behind one minimal interface: `kubernetes` (a real Squid - pod in this cluster — local dev/CI) and `gcp` (Compute Engine VMs with - ephemeral external IPs — the real egress fleet). Config is a YAML file - (`--providers-config`) with named instances (`gcp-eu`, `gcp-us`, ...). + pod in this cluster — local dev/CI), `gcp` (Compute Engine VMs with + ephemeral external IPs — the real egress fleet) and `kvm` (VMs on your + own libvirt hypervisors, reached over SSH). Config is a YAML file + (`--providers-config`) with named instances (`gcp-eu`, `kvm-home`, ...). - **Health engine** probes every proxy by fetching a URL *through* it (a real CONNECT tunnel — a proxy that accepts TCP but can't egress goes Unhealthy), with threshold logic and transition-only status writes. @@ -131,6 +132,29 @@ make deploy IMG=/egress-proxies-operator:dev which also installs Squid via cloud-init). A missing field fails the Proxy with a message naming it. +## KVM setup + +Managed proxies as VMs on libvirt/KVM hosts you own. Full guide: +[docs/kvm-provider.md](docs/kvm-provider.md). + +1. Each host needs `libvirtd`, an SSH user in the `libvirt` group, a + bridge to a network with DHCP, and a storage pool. +2. Create the `kvm-ssh` Secret (private key + `known_hosts`); the manager + mounts it at `/etc/proxy-operator/kvm`: + + ```sh + kubectl -n egress-proxies-operator-system create secret generic kvm-ssh \ + --from-file=id_ed25519= --from-file=known_hosts= + ``` + +3. Add a `kvm` entry to the providers config with `images`, + `machineTypes` and `hosts` (see + [config/samples/providers-config.yaml](config/samples/providers-config.yaml)). + Base images are downloaded into each host's pool on first use. +4. A Proxy names a host (`placement.zone`, optional with one host), a + size (`placement.machineType`) and an image (`placement.image`) — see + [config/samples/proxy_kvm.yaml](config/samples/proxy_kvm.yaml). + Cloud-init from a Secret: the Secret **must** carry the label `crawl.example.com/cloud-init: "true"` — the operator's cache only holds labelled Secrets, so an unlabelled one is invisible (the Proxy reports diff --git a/cmd/main.go b/cmd/main.go index 2b85b59..18d76cf 100644 --- a/cmd/main.go +++ b/cmd/main.go @@ -58,6 +58,7 @@ import ( "gitea.home.hrajfrisbee.cz/kacerr/egress-proxies-operator/internal/provider" "gitea.home.hrajfrisbee.cz/kacerr/egress-proxies-operator/internal/provider/gcp" "gitea.home.hrajfrisbee.cz/kacerr/egress-proxies-operator/internal/provider/kubernetes" + "gitea.home.hrajfrisbee.cz/kacerr/egress-proxies-operator/internal/provider/kvm" "gitea.home.hrajfrisbee.cz/kacerr/egress-proxies-operator/internal/provider/registry" "gitea.home.hrajfrisbee.cz/kacerr/egress-proxies-operator/internal/tracing" "gitea.home.hrajfrisbee.cz/kacerr/egress-proxies-operator/internal/version" @@ -187,6 +188,7 @@ func main() { "gcp": func(ctx context.Context, pc provider.ProviderConfig) (provider.Provider, error) { return gcp.NewWithWireOptions(ctx, pc, gcp.WireLogOptions{FullPayloads: gcpWireFullPayloads}) }, + "kvm": kvm.New, }) if err != nil { setupLog.Error(err, "Failed to build providers") diff --git a/config/manager/manager.yaml b/config/manager/manager.yaml index 5676abb..e115f10 100644 --- a/config/manager/manager.yaml +++ b/config/manager/manager.yaml @@ -145,9 +145,22 @@ spec: - name: providers-config mountPath: /etc/proxy-operator readOnly: true + # SSH material for kvm providers (sshKeyFile / knownHostsFile in the + # providers config point here). Optional: the mount is empty when the + # Secret does not exist, and only kvm entries need it. Create it with: + # kubectl -n egress-proxies-operator-system create secret generic kvm-ssh \ + # --from-file=id_ed25519= \ + # --from-file=known_hosts= + - name: kvm-ssh + mountPath: /etc/proxy-operator/kvm + readOnly: true volumes: - name: providers-config configMap: name: providers-config + - name: kvm-ssh + secret: + secretName: kvm-ssh + optional: true serviceAccountName: controller-manager terminationGracePeriodSeconds: 10 diff --git a/config/manager/providers_config.yaml b/config/manager/providers_config.yaml index 6c383c0..916ac14 100644 --- a/config/manager/providers_config.yaml +++ b/config/manager/providers_config.yaml @@ -9,8 +9,9 @@ metadata: data: # Mounted at /etc/proxy-operator/providers.yaml (--providers-config). # The default ships only the kubernetes-pod provider so the operator runs - # out of the box; add gcp entries (type: gcp, gcp.project: ...) for real - # egress fleets — see config/samples/providers-config.yaml. + # out of the box; add gcp entries (type: gcp, gcp.project: ...) or kvm + # entries (type: kvm, hosts reached over SSH) for real egress fleets — + # see config/samples/providers-config.yaml. providers.yaml: | providers: - name: kubernetes diff --git a/config/samples/kustomization.yaml b/config/samples/kustomization.yaml index d1c025f..4545440 100644 --- a/config/samples/kustomization.yaml +++ b/config/samples/kustomization.yaml @@ -4,5 +4,6 @@ resources: - proxy_kubernetes.yaml - proxy_gcp.yaml +- proxy_kvm.yaml - proxy_external.yaml # +kubebuilder:scaffold:manifestskustomizesamples diff --git a/config/samples/providers-config.yaml b/config/samples/providers-config.yaml index 34220d8..af9e575 100644 --- a/config/samples/providers-config.yaml +++ b/config/samples/providers-config.yaml @@ -19,3 +19,36 @@ providers: # diskSizeGb: 10 # auth: Application Default Credentials (workload identity # in-cluster, gcloud ADC locally). No key-file plumbing. + - name: kvm-home + type: kvm + kvm: + # SSH to each host with the libvirt unix socket forwarded through the + # connection; nothing to open on the hypervisors beyond sshd. Both + # files come from the optional kvm-ssh Secret mounted by + # config/manager/manager.yaml — see docs/kvm-provider.md. + sshKeyFile: /etc/proxy-operator/kvm/id_ed25519 + knownHostsFile: /etc/proxy-operator/kvm/known_hosts + # insecureIgnoreHostKey: true # skip host-key checks (not recommended) + # diskSizeGb: 10 # per-VM overlay; cloud-init grows the root fs into it + # vendorData: | # default installs qemu-guest-agent for IP discovery + # #cloud-config + # ... + images: + # placement.image names a key here (or is an https:// URL directly). + # Fetched into each host's pool on first use and cached there as + # base-.qcow2; do not change an entry while Proxies use it. + debian-13: + url: https://cloud.debian.org/images/cloud/trixie/latest/debian-13-generic-amd64.qcow2 + # sha256: "<64 hex>" # optional; verified after download + ubuntu-24.04: + url: https://cloud-images.ubuntu.com/noble/current/noble-server-cloudimg-amd64.img + machineTypes: # placement.machineType names a key here + small: {vcpus: 1, memoryMiB: 1024} + medium: {vcpus: 2, memoryMiB: 2048} + hosts: # placement.zone names a host; optional with one host + - name: nas1 + address: nas1.home.lan:22 + user: virt # must be in the libvirt group + bridge: br0 # to a LAN with DHCP; the VM's IP comes from there + # socket: /var/run/libvirt/libvirt-sock + # pool: default diff --git a/config/samples/proxy_kvm.yaml b/config/samples/proxy_kvm.yaml new file mode 100644 index 0000000..e84e4d6 --- /dev/null +++ b/config/samples/proxy_kvm.yaml @@ -0,0 +1,43 @@ +# A Managed proxy backed by a libvirt/KVM hypervisor on the home network: +# the operator boots a stock Debian cloud image as a VM on the named host, +# with a NIC on the host's bridge (IP from the LAN's DHCP), and installs +# Squid via cloud-init. Requires a providers-config entry named "kvm-home" +# (see providers-config.yaml) and the kvm-ssh Secret (docs/kvm-provider.md). +# +# placement.zone is the host name from kvm.hosts (optional when only one +# host is configured); machineType and image are keys in the provider's +# machineTypes and images maps. The first Proxy for an image on a host +# waits for the image download; later ones reuse the cached copy. +apiVersion: crawl.example.com/v1alpha1 +kind: Proxy +metadata: + name: proxy-kvm-sample +spec: + mode: Managed + provider: kvm-home + placement: + zone: nas1 + machineType: small + image: debian-13 + port: 3128 + cloudInit: + inline: | + #cloud-config + packages: + - squid + write_files: + - path: /etc/squid/conf.d/proxy-operator.conf + content: | + http_port 3128 + http_access allow all + via off + forwarded_for off + runcmd: + - systemctl restart squid + # Optional: log in with `ssh debian@` (the Debian image's default + # user) to look around; `virsh console ` on the host works too. + # ssh_authorized_keys: + # - ssh-ed25519 AAAA... you@laptop + attributes: + geo: home + purpose: crawl diff --git a/docs/architecture.md b/docs/architecture.md index 5c1053b..328c069 100644 --- a/docs/architecture.md +++ b/docs/architecture.md @@ -9,6 +9,7 @@ | Provider contract | `internal/provider` | library | — | `Provider` interface, error taxonomy, deterministic naming, config, metrics decorator | | kubernetes provider | `internal/provider/kubernetes` | library | — | real Squid pods in this cluster (local dev/CI) | | gcp provider | `internal/provider/gcp` | library | — | Compute Engine VMs, four API calls, fire-and-forget ops | +| kvm provider | `internal/provider/kvm` | library | — | VMs on libvirt hosts over SSH-forwarded sockets; image cache, overlay disks, NoCloud seeds | | Health engine | `internal/health` | Runnable | yes | through-the-proxy probes, thresholds, transition events | | Lease store | `internal/lease` | Runnable (expiry sweep) | no | in-memory leases + cooldowns, single mutex | | Discovery API | `internal/discovery` | Runnable | no | HTTP list/lease/release/report on `:8090` | @@ -147,6 +148,25 @@ prov.ListByTag ─► AggregatedList(label filter, ─┘ prov ReturnPartialSuccess: true) providerID = zones//instances/ — zone-qualified, so Get/Delete stay correct even mid-replacement after a zone edit + +kvm provider (internal/provider/kvm/) — libvirt RPC over an SSH-forwarded unix socket, per host +prov.Create ──► DomainLookupByName: exists → start if needed, done ─┐ every step tolerates + else: ensure base image in pool (download on first use) │ the previous attempt + overlay volume (exists = reuse) │ having got that far; + seed ISO (always rebuilt — a cut-off upload must │ the seed is the one + never be trusted) │ thing redone + DomainDefineXML + DomainCreate ─┘ +prov.Get ──► DomainGetState + DomainGetMetadata + DomainInterfaceAddresses + (ARP, then guest agent; first routable IPv4) + running without IP = Provisioning; nostate = Provisioning +prov.Delete ──► destroy (not running = fine) → undefine → delete overlay + seed by name +prov.ListByTag ─► ConnectListAllDomains on every host, keep domains whose + carries managed=true; one host down fails the sweep, + one unreadable domain is skipped +providerID = hosts//domains/; GC identity lives in the domain's + element (libvirt has no labels). All RPCs run under a context- +bounded call: go-libvirt never times out on its own, so a cancelled reconcile +closes the SSH connection to unblock it. ``` The reconciler never watches provider-side resources (Pods or GCP VMs). @@ -415,6 +435,30 @@ they read as choices, not accidents. Chronological by build step. - **Unknown GCP instance statuses map to `Stopped`:** the reconciler's answer to Stopped is delete-and-recreate, the always-safe move for cattle when the API grows a new state. +- **KVM hosts are reached over SSH with the libvirt socket forwarded,** + not libvirtd's TCP/TLS listener: nothing to configure or expose on the + hypervisors beyond sshd, pure Go on the operator side (no cgo, no + libvirt libraries in the image), and the hosts list can grow by a + config edit because the providerID carries the host name. +- **KVM base images are fetched by the operator into the host's pool** + through libvirt's upload stream, rather than expected to be pre-placed: + the operator then needs no filesystem access on the host and a new + host is usable with zero manual staging. The cost is a synchronous + first-time download inside Create. +- **The KVM seed ISO is rebuilt on every Create attempt** before the + domain exists, instead of "exists → skip": a reconcile cancelled mid + upload leaves a truncated seed, and the compensating delete cannot run + because the connection was already closed to unblock the RPC. Skipping + it would boot a VM with no cloud-init and no retry would ever repair it. +- **KVM `nostate` maps to `Provisioning`, not `Stopped`:** unlike GCP's + unknown statuses, `nostate` means libvirt could not determine the + state at all; answering that with delete-and-recreate is too drastic, + so the next poll decides. +- **A KVM domain GC cannot read is skipped, a host it cannot reach fails + the sweep.** GC never deletes what it does not see, so leaving one + domain out is safe; one corrupt domain blocking GC for every other one + is not. A whole host missing from the view would hide many, so that + still aborts. - **Kubernetes 403s classify as `ErrPermanent`** even though quota exhaustion also surfaces as 403 (indistinguishable from RBAC denial in `apierrors`): not hammering an API server that may never allow the diff --git a/docs/kvm-provider.md b/docs/kvm-provider.md new file mode 100644 index 0000000..c79899e --- /dev/null +++ b/docs/kvm-provider.md @@ -0,0 +1,214 @@ +# The KVM provider + +Runs Managed proxies as VMs on your own libvirt/KVM hypervisors. The +operator reaches each host over SSH, forwards the libvirt unix socket +through that connection, and talks libvirt's own RPC over it — so the +hosts need nothing beyond `sshd` and `libvirtd`, and the operator image +carries no libvirt libraries. VMs boot a stock distro cloud image that the +operator downloads into the host's storage pool on first use, with the +Proxy's cloud-init delivered on a NoCloud seed ISO built in-process. + +## Hypervisor prerequisites + +Per host, once: + +1. **libvirt + KVM** (`libvirtd` running, `/dev/kvm` present). +2. **An SSH user in the `libvirt` group**, so it may open + `/var/run/libvirt/libvirt-sock`. The operator authenticates with a + private key; nothing else on the host is touched. +3. **A bridge to a network with DHCP** (e.g. `br0` bridged onto the LAN). + Each VM gets one virtio NIC on it and takes its IP from that network's + DHCP; the cluster must be able to reach those IPs for health probes + and for crawlers to use the proxy. +4. **A storage pool** (default `default`, usually `/var/lib/libvirt/images`) + with room for one base image per distro (a few hundred MB) plus the + per-VM overlays (`diskSizeGb` each, thin-provisioned). +5. Optionally, **outbound internet from the VMs**: the default + vendor-data installs `qemu-guest-agent` from the distro mirrors on + first boot. Without it the VM still works — the operator falls back to + the host's ARP table to find the IP, which fills in once the guest + sends traffic. + +Check from the operator's point of view before wiring it in: + +```sh +ssh virt@nas1.home.lan virsh -c qemu:///system list --all # user can drive libvirt +ssh virt@nas1.home.lan virsh -c qemu:///system pool-info default +ssh virt@nas1.home.lan ip link show br0 +``` + +## Cluster-side setup + +1. Create the SSH Secret. `known_hosts` must contain every host's key + (`ssh-keyscan -H nas1.home.lan >> known_hosts`); without it, set + `insecureIgnoreHostKey: true` and accept that a MITM could hand you a + fake hypervisor. + + ```sh + kubectl -n egress-proxies-operator-system create secret generic kvm-ssh \ + --from-file=id_ed25519=$HOME/.ssh/proxy-operator_ed25519 \ + --from-file=known_hosts=./known_hosts + ``` + + `config/manager/manager.yaml` mounts it (optional) at + `/etc/proxy-operator/kvm`; the deployment starts fine without it as + long as no `kvm` provider is configured. + +2. Add a provider entry to the providers config + (`config/manager/providers_config.yaml` in-cluster; full sample in + `config/samples/providers-config.yaml`): + + ```yaml + - name: kvm-home + type: kvm + kvm: + sshKeyFile: /etc/proxy-operator/kvm/id_ed25519 + knownHostsFile: /etc/proxy-operator/kvm/known_hosts + images: + debian-13: + url: https://cloud.debian.org/images/cloud/trixie/latest/debian-13-generic-amd64.qcow2 + machineTypes: + small: {vcpus: 1, memoryMiB: 1024} + hosts: + - name: nas1 + address: nas1.home.lan:22 + user: virt + bridge: br0 + ``` + + The operator reads the key and known_hosts at startup and refuses to + start if either is missing — a bad path fails fast, not on the first + Proxy. + +3. Apply a Proxy (`config/samples/proxy_kvm.yaml`) and watch it: + + ```sh + kubectl get px -w + # NAME MODE PROVIDER PHASE IP HEALTHY + # proxy-kvm-sample Managed kvm-home Provisioning + # proxy-kvm-sample Managed kvm-home Running 192.168.1.73 + # proxy-kvm-sample Managed kvm-home Ready 192.168.1.73 True + ``` + + The first Proxy for a given image on a given host sits in + `Provisioning` for the length of the image download; the operator log + shows `fetching base image into host pool` and `base image cached` + with the size and duration. + +## Config reference + +| Field | Required | Default | Meaning | +|---|---|---|---| +| `sshKeyFile` | yes | | private key used for every host | +| `knownHostsFile` | unless `insecureIgnoreHostKey` | | OpenSSH `known_hosts` to verify host keys against | +| `insecureIgnoreHostKey` | | `false` | skip host-key verification entirely | +| `diskSizeGb` | | `10` | capacity of each VM's overlay disk; cloud-init's growpart expands the root fs into it | +| `vendorData` | | installs `qemu-guest-agent` | cloud-init vendor-data placed on every seed next to the Proxy's user-data | +| `images..url` | yes per entry | | http(s) URL of a cloud image (qcow2 or raw) | +| `images..sha256` | | | expected digest; verified after download, a change triggers a re-download | +| `images..format` | | `qcow2` | `qcow2` or `raw` | +| `machineTypes.` | ≥ 1 | | `{vcpus, memoryMiB}` size profile | +| `hosts[].name` | yes | | the `placement.zone` value that selects this host; unique | +| `hosts[].address` | yes | | SSH endpoint, `host:port` | +| `hosts[].user` | yes | | SSH user, member of the `libvirt` group | +| `hosts[].socket` | | `/var/run/libvirt/libvirt-sock` | libvirt socket path on the host | +| `hosts[].pool` | | `default` | storage pool for base images, overlays and seeds | +| `hosts[].bridge` | yes | | host bridge for the VM's NIC | + +## Placement + +| Proxy field | Meaning for kvm | +|---|---| +| `placement.zone` | host name from `hosts`. May be omitted when exactly one host is configured; otherwise required. Unknown → `Failed` | +| `placement.machineType` | key in `machineTypes`. Unknown → `Failed` | +| `placement.image` | key in `images`, or an `http(s)://` URL used directly. Anything else → `Failed` | +| `placement.region` | ignored | + +`Failed` here means the reconciler stops retrying and the message names +the offending value; fix the spec (which replaces the VM) or the config. + +## What lands on the host + +For a Proxy whose deterministic name is `proxy-abc…`, on the chosen +host's pool: + +| Object | Name | Lifetime | +|---|---|---| +| base image | `base-.qcow2` (or `base-.qcow2` for a URL) | cached; survives Proxy deletion | +| image marker | `base-.qcow2.sha256` | with the base image | +| overlay disk | `proxy-abc….qcow2`, qcow2 backed by the base | with the Proxy | +| seed ISO | `proxy-abc…-seed.iso` | with the Proxy | +| domain | `proxy-abc…` | with the Proxy | + +The domain is a q35 machine with host-passthrough CPU, the overlay on +virtio, the seed on a SATA cdrom, one virtio NIC on the bridge, a serial +console, the guest-agent channel, a virtio RNG and no graphics. Its +`` carries the operator's element (managed flag, owning UID, +creation time) — that is how orphan GC recognises its own VMs, since +libvirt has no labels. + +## The image cache + +The base image is downloaded **by the operator, through libvirt**, into +the host's pool the first time a Proxy asks for it on that host — nothing +is pre-placed on the hypervisor, and the operator needs no filesystem +access there. The bytes stream from the URL through the operator into +the pool with a running sha256; nothing is buffered. The marker volume is +written last and holds the digest, so a crash mid-download leaves a base +volume without a marker, which the next attempt deletes and redoes. + +A re-download happens when: the marker is missing, or `sha256` is +configured and differs from the marker's content. Otherwise the cached +copy is used for as long as it exists, even if the URL's content changed +upstream (a `latest` URL is pinned at the moment of first fetch, per +host). + +To evict manually: + +```sh +ssh virt@nas1 virsh -c qemu:///system vol-delete --pool default base-debian-13.qcow2.sha256 +ssh virt@nas1 virsh -c qemu:///system vol-delete --pool default base-debian-13.qcow2 +``` + +⚠️ **Do not change an image entry's `url` or `sha256`, and do not evict +its base volume, while Proxies using that image exist on the host.** The +overlays back onto the base file; deleting or replacing it corrupts every +VM that uses it. Delete those Proxies first, or add the new image under a +new key and move Proxies over by changing their `placement.image` (which +replaces the VM). + +## Logging into a VM + +- Put your key in the Proxy's cloud-init (`ssh_authorized_keys`) and + `ssh debian@` (Debian) / `ssh ubuntu@` (Ubuntu). +- Or on the host: `virsh console proxy-abc…` attaches to the serial + console; cloud-init's output scrolls by there on first boot. +- `virsh domifaddr proxy-abc… --source arp` / `--source agent` shows what + the operator sees when it looks for the IP. + +## Known gaps + +- **No quota or capacity detection.** libvirt reports an out-of-space + pool or an over-committed host as an ordinary error; the reconciler + treats it as transient and retries with backoff rather than the slower + quota backoff. +- **Orphan volumes without a domain are invisible to GC.** A crash + between creating the volumes and defining the domain leaves an overlay + and seed that `ListByTag` cannot see (it lists domains). They are + cleaned when the same Proxy reconciles again (Delete removes volumes by + name even with no domain), but a Proxy CR that is gone for good leaves + them for `virsh vol-list` to find. +- **The first image fetch is synchronous.** It holds a reconcile worker + for the download (LAN speed, a few hundred MB, once per image per + host). Other Proxies keep reconciling on the remaining workers. +- **Replacing an image in use breaks running VMs** — see the warning + under *The image cache*. +- **IP discovery needs either guest traffic or the agent.** A VM on a + bridge with no DHCP never gets an address and stays `Provisioning` + forever; check `virsh console`. + +## Testing + +Unit tests run over an in-memory fake of the libvirt API. The optional +integration test against a real host is described in +[testing.md](testing.md#the-kvm-integration-test-real-hypervisor). diff --git a/docs/plans-executions/2026-09-04-1404-kvm-provider.md b/docs/plans-executions/2026-09-04-1404-kvm-provider.md index 215080a..a256fa4 100644 --- a/docs/plans-executions/2026-09-04-1404-kvm-provider.md +++ b/docs/plans-executions/2026-09-04-1404-kvm-provider.md @@ -8,7 +8,7 @@ Plan: [2026-09-04-1404-kvm-provider.md](../plans/2026-09-04-1404-kvm-provider.md - [x] Step 3 — Transport + errors (`conn.go`, `errors.go`) - [x] Step 3b — Image cache (`image.go`) - [x] Step 4 — Provider methods (`kvm.go`) over a fake API -- [ ] Step 5 — Wiring, manifests, samples, docs, CHANGELOG +- [x] Step 5 — Wiring, manifests, samples, docs, CHANGELOG - [x] Step 6 — Integration test against a real hypervisor (written and compiled; first real run is part of Step 7) - [ ] Step 7 — Manual validation + MR @@ -338,3 +338,54 @@ Step 3b "confirm on a real host" notes get answered. The test's config is built directly rather than through the provider package's validation (unexported), so a typo in an env var surfaces from `New` or from libvirt, not from a validation message. + +## Step 5 — Wiring, manifests, samples, docs, CHANGELOG + +Done after Step 6 (see that entry for why the order flipped). Everything +the plan listed, plus a few things it did not: + +- `cmd/main.go`: `"kvm": kvm.New` in the builtins map. No RBAC change — + the provider talks to hypervisors, not the API server. +- `config/manager/manager.yaml`: `kvm-ssh` Secret mounted read-only at + `/etc/proxy-operator/kvm` with `optional: true`, so deployments without + a kvm provider are unaffected. It nests under the providers-config + mount at `/etc/proxy-operator`; kustomize renders it (`kubectl + kustomize config/default` checked). The container runs as uid 65532 + and Secret files default to 0644, so the key is readable without an + `fsGroup`; the `kubectl create secret` recipe is in a comment next to + the mount, in `docs/kvm-provider.md` and in the README. +- `config/samples/providers-config.yaml`: `kvm-home` block with Debian 13 + and Ubuntu 24.04 official image URLs, two machine types, one host, and + every optional field shown commented out. Parsed through + `provider.LoadConfigFile` in a throwaway test to make sure the sample + validates. `config/samples/proxy_kvm.yaml` uses the GCP sample's squid + cloud-init and shows `ssh_authorized_keys` commented out rather than + set — a sample with a placeholder key would either fail cloud-init or + invite pasting a real key into a committed file. Added to the samples + kustomization; `config/manager/providers_config.yaml`'s comment now + mentions kvm. +- `docs/kvm-provider.md`: prerequisites with a three-line preflight + (`virsh list`, `pool-info`, `ip link` over ssh), cluster setup, the full + config reference table, placement table, a table of what lands on the + host and its lifetime, how the image cache works and how to evict, the + **warning not to change or evict an image while Proxies use it** (the + reviewer's finding from Step 4, carried as a documented rule since the + content-addressed-naming alternative was not chosen), how to log in, + and the known gaps. +- README: providers bullet and a "KVM setup" section mirroring the GCP + one. `docs/architecture.md`: components row, a §5 block for the kvm + call flow in the same style as the other two, and five decision-log + entries (SSH transport, operator-fetched images, seed always rebuilt, + `nostate` → Provisioning, skip-vs-fail in the GC sweep). +- `CHANGELOG.md` entry at the top, stating explicitly that the provider + is not yet validated on a real host. + +Verification: `go build ./...`, `go vet ./...`, `go test -short` on +`cmd` and `internal/provider/...`, and both kustomizations rendered. +`hack/providers-dev.yaml` untouched, as planned. + +Worth noting: the CRD has no rule requiring all three placement fields +(that is GCP's own check), so the single-host "zone may be omitted" +behaviour works without an API change. And with this step the branch is +feature-complete but **unproven**: Step 7 runs the integration test and +the end-to-end checklist against a real hypervisor before the MR. -- 2.54.0 From cd51886139de4eb0140c6a7fb312f2ff031f0551 Mon Sep 17 00:00:00 2001 From: kacerr Date: Thu, 10 Sep 2026 07:37:51 +0000 Subject: [PATCH 09/28] Reference KVM domain disks by host path so AppArmor-confined QEMU can open them The first run against a real hypervisor (beelink, Ubuntu, libvirt 11.6) failed at DomainCreate with "Could not open base image: Permission denied" even with correct DAC ownership. libvirt's virt-aa-helper only grants QEMU access to disk sources that carry a path; the form chosen in Step 2 gives it pool and volume names instead, so the overlay and its backing image get no AppArmor rule on any stock Debian/Ubuntu host. buildDomainXML now takes DiskPath/SeedPath and emits type="file" disks. ensureOverlay and writeSeed return the volume handles and Create resolves both paths with StorageVolGetPath before defining the domain. The domain XML test rejects any volume/pool reference so this cannot drift back; the Create test asserts the sources equal the paths libvirt reported. Execution log: Step 2 note marked reverted, Step 7 entry opened with the two failed runs and the diagnosis. Co-Authored-By: Claude Fable 5.1 --- .../2026-09-04-1404-kvm-provider.md | 56 +++++++++++++++++++ internal/provider/kvm/domainxml.go | 32 ++++++----- internal/provider/kvm/domainxml_test.go | 16 +++--- internal/provider/kvm/kvm.go | 54 +++++++++++------- internal/provider/kvm/kvm_test.go | 7 +++ 5 files changed, 125 insertions(+), 40 deletions(-) diff --git a/docs/plans-executions/2026-09-04-1404-kvm-provider.md b/docs/plans-executions/2026-09-04-1404-kvm-provider.md index a256fa4..e9dc2bb 100644 --- a/docs/plans-executions/2026-09-04-1404-kvm-provider.md +++ b/docs/plans-executions/2026-09-04-1404-kvm-provider.md @@ -69,6 +69,9 @@ Two design choices differ from the plan text: ``, so only the *base* image path is ever looked up (the overlay's `` needs a real path). Fewer RPCs and the builder's inputs are all names. + **Reverted in Step 7**: volume-referenced disks cannot start on an + AppArmor host, see there. The domain XML is back to `type="file"` with + the two `StorageVolGetPath` calls the plan had. - **Capacities are always `unit="bytes"`.** libvirt's `G` means GiB but `GB` means 10^9; spelling the byte count out removes the ambiguity from both the overlay and the seed/base volumes. @@ -389,3 +392,56 @@ Worth noting: the CRD has no rule requiring all three placement fields behaviour works without an API change. And with this step the branch is feature-complete but **unproven**: Step 7 runs the integration test and the end-to-end checklist against a real hypervisor before the MR. + +## Step 7 — Manual validation + MR (in progress) + +Host: `beelink` (Ubuntu, libvirt 11.6.0, QEMU 10.1.0, pool `default` at +`/srv/vms/images` on local ext4, bridge `br0`), reached over Tailscale as +`beelink.tail8c69c.ts.net`, SSH user `novakj`. Not the `nas1.home.lan` +the plan and samples name; that host does not resolve from the dev box. + +```bash +export KVM_TEST_HOST=beelink.tail8c69c.ts.net:22 +export KVM_TEST_USER=novakj +export KVM_TEST_SSH_KEY=$HOME/.ssh/proxy-operator_ed25519 +export KVM_TEST_KNOWN_HOSTS=$HOME/.ssh/kvm_known_hosts +export KVM_TEST_BRIDGE=br0 +export KVM_TEST_IMAGE_URL=https://cloud.debian.org/images/cloud/trixie/latest/debian-13-generic-amd64.qcow2 +make test-integration-kvm +``` + +### Run 1 and 2: `Could not open '/srv/vms/images/base-test-image.qcow2': Permission denied` + +Everything up to `DomainCreate` worked on the first try against a real +host, which answers the open notes from Steps 2 and 3b: the SSH-forwarded +libvirt handshake, the raw qcow2 upload plus pool refresh as a backing +image, the default-namespace metadata element, and `DomainDefineXML` for +the generated XML were all accepted by libvirt 11.6. QEMU then exited at +start with `Permission denied` on the base image. + +Ownership was ruled out the slow way: the file was `root:root 0600` (the +dir-pool default; libvirt's DAC driver chowns the chain at start and +restores it after a failed start, so that observation meant nothing), +`qemu.conf` was all defaults (`libvirt-qemu`, `dynamic_ownership=1`), and +after a manual `chown libvirt-qemu:kvm` the second run failed identically. + +The actual cause is the Step 2 choice of ``. +libvirt's AppArmor helper builds each domain's profile from the domain XML +without consulting the storage driver, and its `add_file_path` only emits a +rule when the source has a `path`; a volume-referenced disk has pool and +volume names instead, so neither the overlay nor its backing image gets a +rule and QEMU is denied on the first one it opens. Every stock Debian and +Ubuntu hypervisor confines QEMU this way. The unit tests could not have +caught it: the fake accepts either form. + +Fix, in this step: `buildDomainXML` takes `DiskPath`/`SeedPath` and emits +`type="file"` disks; `ensureOverlay` and `writeSeed` return the volume +handles (the exists-already branch now does a `StorageVolLookupByName`), +and Create resolves both paths with `StorageVolGetPath` before defining +the domain. Two extra cheap RPCs per Create. `TestBuildDomainXML_fieldByField` +now fails on any `type="volume"` or `pool=` attribute so the design cannot +drift back, and `TestCreate_freshHost` asserts the disk sources equal the +paths the fake reports. + +Worth noting: the first two runs left the base image and its marker in +the pool as designed, so later runs skip the 435 MB download. diff --git a/internal/provider/kvm/domainxml.go b/internal/provider/kvm/domainxml.go index 32f0732..bfdcfe4 100644 --- a/internal/provider/kvm/domainxml.go +++ b/internal/provider/kvm/domainxml.go @@ -9,24 +9,29 @@ import ( const guestAgentChannel = "org.qemu.guest_agent.0" // domainParams is everything buildDomainXML needs, already resolved from the -// config and placement so the builder stays a pure function. +// config and placement so the builder stays a pure function. DiskPath and +// SeedPath are the volumes' host paths as reported by libvirt. type domainParams struct { Name string UID string CreatedAt time.Time VCPUs int32 MemoryMiB int64 - Pool string Bridge string - DiskVol string - SeedVol string + DiskPath string + SeedPath string } // buildDomainXML renders a q35 KVM guest: host-passthrough CPU, a virtio -// overlay disk and a SATA cdrom for the seed (both referenced as pool -// volumes so no host paths are needed), one virtio NIC on the host bridge, -// a serial console, the guest-agent channel, a virtio RNG, no graphics, and -// the operator's metadata element. +// overlay disk and a SATA cdrom for the seed, one virtio NIC on the host +// bridge, a serial console, the guest-agent channel, a virtio RNG, no +// graphics, and the operator's metadata element. +// +// Disks are type="file" with host paths, not type="volume" with pool/volume +// names: libvirt's AppArmor helper (virt-aa-helper) only grants QEMU access +// to sources that carry a path, so a volume-referenced disk fails to start +// with "Permission denied" on any AppArmor-confined host (stock Debian and +// Ubuntu). The paths cost one StorageVolGetPath round-trip each per Create. func buildDomainXML(p domainParams) (string, error) { d := domainXML{ Type: "kvm", @@ -47,17 +52,17 @@ func buildDomainXML(p domainParams) (string, error) { Devices: devicesXML{ Disks: []diskXML{ { - Type: "volume", + Type: "file", Device: "disk", Driver: driverXML{Name: "qemu", Type: formatQcow2}, - Source: diskSourceXML{Pool: p.Pool, Volume: p.DiskVol}, + Source: diskSourceXML{File: p.DiskPath}, Target: targetXML{Dev: "vda", Bus: "virtio"}, }, { - Type: "volume", + Type: "file", Device: "cdrom", Driver: driverXML{Name: "qemu", Type: formatRaw}, - Source: diskSourceXML{Pool: p.Pool, Volume: p.SeedVol}, + Source: diskSourceXML{File: p.SeedPath}, Target: targetXML{Dev: "sda", Bus: "sata"}, ReadOnly: &struct{}{}, }, @@ -161,8 +166,7 @@ type driverXML struct { } type diskSourceXML struct { - Pool string `xml:"pool,attr"` - Volume string `xml:"volume,attr"` + File string `xml:"file,attr"` } type targetXML struct { diff --git a/internal/provider/kvm/domainxml_test.go b/internal/provider/kvm/domainxml_test.go index d52474c..b401670 100644 --- a/internal/provider/kvm/domainxml_test.go +++ b/internal/provider/kvm/domainxml_test.go @@ -14,10 +14,9 @@ func testDomainParams() domainParams { CreatedAt: time.Date(2026, 9, 4, 12, 0, 0, 0, time.UTC), VCPUs: 2, MemoryMiB: 2048, - Pool: "default", Bridge: "br0", - DiskVol: "proxy-abc123def456ghij.qcow2", - SeedVol: "proxy-abc123def456ghij-seed.iso", + DiskPath: "/var/lib/libvirt/images/proxy-abc123def456ghij.qcow2", + SeedPath: "/var/lib/libvirt/images/proxy-abc123def456ghij-seed.iso", } } @@ -63,16 +62,19 @@ func TestBuildDomainXML_fieldByField(t *testing.T) { t.Fatalf("disks = %d, want 2", len(d.Devices.Disks)) } disk, seed := d.Devices.Disks[0], d.Devices.Disks[1] - if disk.Type != "volume" || disk.Device != "disk" || disk.Driver.Type != "qcow2" || - disk.Source.Pool != "default" || disk.Source.Volume != "proxy-abc123def456ghij.qcow2" || + if disk.Type != "file" || disk.Device != "disk" || disk.Driver.Type != "qcow2" || + disk.Source.File != "/var/lib/libvirt/images/proxy-abc123def456ghij.qcow2" || disk.Target.Dev != "vda" || disk.Target.Bus != "virtio" || disk.ReadOnly != nil { t.Errorf("overlay disk = %+v", disk) } - if seed.Type != "volume" || seed.Device != "cdrom" || seed.Driver.Type != "raw" || - seed.Source.Pool != "default" || seed.Source.Volume != "proxy-abc123def456ghij-seed.iso" || + if seed.Type != "file" || seed.Device != "cdrom" || seed.Driver.Type != "raw" || + seed.Source.File != "/var/lib/libvirt/images/proxy-abc123def456ghij-seed.iso" || seed.Target.Bus != "sata" || seed.ReadOnly == nil { t.Errorf("seed cdrom = %+v", seed) } + if strings.Contains(raw, `type="volume"`) || strings.Contains(raw, " pool=") { + t.Errorf("disks reference pool volumes; virt-aa-helper cannot grant QEMU access to those:\n%s", raw) + } if len(d.Devices.Interfaces) != 1 { t.Fatalf("interfaces = %d, want 1", len(d.Devices.Interfaces)) diff --git a/internal/provider/kvm/kvm.go b/internal/provider/kvm/kvm.go index e510d08..6d94dc4 100644 --- a/internal/provider/kvm/kvm.go +++ b/internal/provider/kvm/kvm.go @@ -183,13 +183,23 @@ func (p *Provider) Create(ctx context.Context, req provider.CreateRequest) (stri if err != nil { return err } - if err := p.ensureOverlay(log, api, pool, req.Name, basePath, src.Format); err != nil { + overlay, err := p.ensureOverlay(log, api, pool, req.Name, basePath, src.Format) + if err != nil { return err } - if err := p.writeSeed(log, api, pool, req.Name, req.CloudInit); err != nil { + seed, err := p.writeSeed(log, api, pool, req.Name, req.CloudInit) + if err != nil { return err } - dom, err = p.defineDomain(log, api, host, req, mt) + diskPath, err := api.StorageVolGetPath(overlay) + if err != nil { + return fmt.Errorf("resolving overlay path: %w", err) + } + seedPath, err := api.StorageVolGetPath(seed) + if err != nil { + return fmt.Errorf("resolving seed path: %w", err) + } + dom, err = p.defineDomain(log, api, host, req, mt, diskPath, seedPath) if err != nil { return err } @@ -203,60 +213,66 @@ func (p *Provider) Create(ctx context.Context, req provider.CreateRequest) (stri return id, nil } -func (p *Provider) ensureOverlay(log logr.Logger, api libvirtAPI, pool libvirt.StoragePool, name, basePath, baseFormat string) error { +// ensureOverlay creates the per-VM overlay volume, or returns the existing +// one: creation is atomic, so an overlay that exists is complete. +func (p *Provider) ensureOverlay(log logr.Logger, api libvirtAPI, pool libvirt.StoragePool, name, basePath, baseFormat string) (libvirt.StorageVol, error) { xml, err := buildVolumeXML(overlayVolume(name, p.cfg.DiskSizeGB, basePath, baseFormat)) if err != nil { - return err + return libvirt.StorageVol{}, err } - _, err = api.StorageVolCreateXML(pool, xml, 0) + vol, err := api.StorageVolCreateXML(pool, xml, 0) switch { case err == nil: log.V(1).Info("overlay volume created", "volume", overlayVolumeName(name), "base", basePath) + return vol, nil case hasCode(err, libvirt.ErrStorageVolExist): log.V(1).Info("overlay volume already exists", "volume", overlayVolumeName(name)) + vol, err := api.StorageVolLookupByName(pool, overlayVolumeName(name)) + if err != nil { + return libvirt.StorageVol{}, fmt.Errorf("looking up existing overlay volume: %w", err) + } + return vol, nil default: - return fmt.Errorf("creating overlay volume: %w", err) + return libvirt.StorageVol{}, fmt.Errorf("creating overlay volume: %w", err) } - return nil } // writeSeed replaces the seed volume with a freshly built NoCloud ISO. // Only called while no domain references it. -func (p *Provider) writeSeed(log logr.Logger, api libvirtAPI, pool libvirt.StoragePool, name, userData string) error { +func (p *Provider) writeSeed(log logr.Logger, api libvirtAPI, pool libvirt.StoragePool, name, userData string) (libvirt.StorageVol, error) { seedName := seedVolumeName(name) if err := deleteIfExists(api, pool, seedName); err != nil { - return err + return libvirt.StorageVol{}, err } iso, err := buildSeedISO(name, userData, p.cfg.VendorData) if err != nil { - return fmt.Errorf("%w: %v", provider.ErrPermanent, err) + return libvirt.StorageVol{}, fmt.Errorf("%w: %v", provider.ErrPermanent, err) } xml, err := buildVolumeXML(rawVolume(seedName, int64(len(iso)))) if err != nil { - return err + return libvirt.StorageVol{}, err } vol, err := api.StorageVolCreateXML(pool, xml, 0) if err != nil { - return fmt.Errorf("creating seed volume: %w", err) + return libvirt.StorageVol{}, fmt.Errorf("creating seed volume: %w", err) } if err := api.StorageVolUpload(vol, bytes.NewReader(iso), 0, 0, 0); err != nil { - return fmt.Errorf("uploading seed volume: %w", err) + return libvirt.StorageVol{}, fmt.Errorf("uploading seed volume: %w", err) } log.V(1).Info("seed volume uploaded", "volume", seedName, "bytes", len(iso)) - return nil + return vol, nil } -func (p *Provider) defineDomain(log logr.Logger, api libvirtAPI, host provider.KVMHost, req provider.CreateRequest, mt provider.KVMMachineType) (libvirt.Domain, error) { +func (p *Provider) defineDomain(log logr.Logger, api libvirtAPI, host provider.KVMHost, req provider.CreateRequest, mt provider.KVMMachineType, diskPath, seedPath string) (libvirt.Domain, error) { xml, err := buildDomainXML(domainParams{ Name: req.Name, UID: req.UID, CreatedAt: p.now(), VCPUs: mt.VCPUs, MemoryMiB: mt.MemoryMiB, - Pool: host.Pool, Bridge: host.Bridge, - DiskVol: overlayVolumeName(req.Name), - SeedVol: seedVolumeName(req.Name), + DiskPath: diskPath, + SeedPath: seedPath, }) if err != nil { return libvirt.Domain{}, err diff --git a/internal/provider/kvm/kvm_test.go b/internal/provider/kvm/kvm_test.go index c627c34..087c371 100644 --- a/internal/provider/kvm/kvm_test.go +++ b/internal/provider/kvm/kvm_test.go @@ -210,6 +210,13 @@ func TestCreate_freshHost(t *testing.T) { if d.Devices.Interfaces[0].Source.Bridge != "br0" { t.Errorf("bridge = %q, want br0", d.Devices.Interfaces[0].Source.Bridge) } + wantDisk, wantSeed := api.poolPath+"/"+overlayVolumeName(testName), api.poolPath+"/"+seedVolumeName(testName) + if got := d.Devices.Disks[0].Source.File; got != wantDisk { + t.Errorf("overlay disk source = %q, want the path libvirt reported, %q", got, wantDisk) + } + if got := d.Devices.Disks[1].Source.File; got != wantSeed { + t.Errorf("seed cdrom source = %q, want the path libvirt reported, %q", got, wantSeed) + } if m := d.Metadata.Instance; m.UID != testUID || !m.managed() || !m.createdAt().Equal(testNow) { t.Errorf("metadata = %+v, want managed, uid %s, createdAt %s", m, testUID, testNow) } -- 2.54.0 From ffb328661d27d2f3ef1a2aa263cefaf13dc70919 Mon Sep 17 00:00:00 2001 From: kacerr Date: Thu, 10 Sep 2026 07:40:23 +0000 Subject: [PATCH 10/28] Parse KVM domain metadata by local name: libvirt returns it without a namespace Run 3 on the real host got past Create and failed in Get: libvirt 11.6 selects the element by our URI in DomainGetMetadata but hands it back with no namespace declaration, and the parser required the URI on the element. The lookup already guarantees the namespace, so match on the local name only. The fake now strips the declaration the way libvirt does, and a new test pins the bare form. Execution log updated. Co-Authored-By: Claude Fable 5.1 --- .../2026-09-04-1404-kvm-provider.md | 19 ++++++++++++++++++- internal/provider/kvm/fake_test.go | 6 +++++- internal/provider/kvm/metadata.go | 18 ++++++++++++------ internal/provider/kvm/metadata_test.go | 15 +++++++++++++++ 4 files changed, 50 insertions(+), 8 deletions(-) diff --git a/docs/plans-executions/2026-09-04-1404-kvm-provider.md b/docs/plans-executions/2026-09-04-1404-kvm-provider.md index e9dc2bb..18f61b3 100644 --- a/docs/plans-executions/2026-09-04-1404-kvm-provider.md +++ b/docs/plans-executions/2026-09-04-1404-kvm-provider.md @@ -81,7 +81,9 @@ The metadata element is written with a default namespace marshal. libvirt may hand it back prefixed; `parseInstanceMetadata` is tested against both spellings since `encoding/xml` matches on the namespace URI. Whether libvirt accepts the default-namespace form on `DomainDefineXML` -is confirmed in Step 6 against a real host, not here. +is confirmed in Step 6 against a real host, not here. (Step 7: it does, +but `DomainGetMetadata` returns the element with no namespace at all; the +parser now matches on local name only.) Checked before relying on `github.com/kdomanski/iso9660` for the seed: its filename mangling lowercases and keeps `-`, so `user-data`/`meta-data`/ @@ -445,3 +447,18 @@ paths the fake reports. Worth noting: the first two runs left the base image and its marker in the pool as designed, so later runs skip the 435 MB download. + +### Run 3: `parsing domain metadata: expected element in name space … but have no name space` + +With file-backed disks, Create returned in 1 s (image cached) and the +domain started. Get then failed to parse the metadata: libvirt 11.6 finds +the element by our URI but returns it as a bare `` with no +namespace declaration at all. Step 2 had tested the default-namespace and +prefixed spellings and left "does libvirt accept the default-namespace +form" for the real host; it does, it just does not give it back that way. + +Fix: `parseInstanceMetadata` matches on the local name only, since the +lookup already selected the element by URI. The fake now strips the +declaration when storing metadata, so the provider's Get and ListByTag +tests parse the shape libvirt really returns, and a new +`TestParseInstanceMetadata_libvirtBareForm` pins it. diff --git a/internal/provider/kvm/fake_test.go b/internal/provider/kvm/fake_test.go index 02c52e4..2e0928d 100644 --- a/internal/provider/kvm/fake_test.go +++ b/internal/provider/kvm/fake_test.go @@ -6,6 +6,7 @@ import ( "fmt" "io" "slices" + "strings" "sync" libvirt "github.com/digitalocean/go-libvirt" @@ -151,8 +152,11 @@ func (f *fakeAPI) DomainDefineXML(raw string) (libvirt.Domain, error) { if err != nil { return libvirt.Domain{}, lvError(libvirt.ErrXMLError, "%v", err) } + // libvirt hands the element back without its namespace declaration + // (seen on 11.6); mimic that so Get/ListByTag parse the real shape. + stored := strings.Replace(string(meta), ` xmlns="`+metadataURI+`"`, "", 1) existing := f.domains[d.Name] - nd := &fakeDomain{name: d.Name, xml: raw, metadata: string(meta)} + nd := &fakeDomain{name: d.Name, xml: raw, metadata: stored} if existing != nil { nd.running = existing.running nd.addrs = existing.addrs diff --git a/internal/provider/kvm/metadata.go b/internal/provider/kvm/metadata.go index 55ab4e7..7eeb110 100644 --- a/internal/provider/kvm/metadata.go +++ b/internal/provider/kvm/metadata.go @@ -31,15 +31,21 @@ func newInstanceMetadata(uid string, createdAt time.Time) instanceMetadata { } // parseInstanceMetadata decodes the element as DomainGetMetadata returns it. -// libvirt may re-serialise it with a namespace prefix rather than the default -// namespace it was defined with; encoding/xml matches on namespace URI, so -// both spellings decode. +// libvirt selects the element by namespace URI but serialises it without the +// declaration (observed on libvirt 11.6: a bare ), and other +// versions may keep a default or prefixed form. The URI is already checked +// by the lookup, so only the local name is matched here. func parseInstanceMetadata(s string) (instanceMetadata, error) { - var m instanceMetadata - if err := xml.Unmarshal([]byte(s), &m); err != nil { + var raw struct { + XMLName xml.Name `xml:"instance"` + Managed string `xml:"managed"` + UID string `xml:"uid"` + CreatedAt string `xml:"createdAt"` + } + if err := xml.Unmarshal([]byte(s), &raw); err != nil { return instanceMetadata{}, fmt.Errorf("parsing domain metadata: %w", err) } - return m, nil + return instanceMetadata{Managed: raw.Managed, UID: raw.UID, CreatedAt: raw.CreatedAt}, nil } func (m instanceMetadata) managed() bool { diff --git a/internal/provider/kvm/metadata_test.go b/internal/provider/kvm/metadata_test.go index f3ab94f..d7727f2 100644 --- a/internal/provider/kvm/metadata_test.go +++ b/internal/provider/kvm/metadata_test.go @@ -45,6 +45,21 @@ func TestParseInstanceMetadata_libvirtPrefixedForm(t *testing.T) { } } +func TestParseInstanceMetadata_libvirtBareForm(t *testing.T) { + t.Parallel() + // What libvirt 11.6 actually returns from DomainGetMetadata: the element + // selected by our URI, serialised without any namespace declaration. + in := `trueuid-3` + + `2026-09-10T07:30:00Z` + m, err := parseInstanceMetadata(in) + if err != nil { + t.Fatalf("parseInstanceMetadata() error = %v", err) + } + if !m.managed() || m.UID != "uid-3" || m.createdAt().Minute() != 30 { + t.Errorf("parsed = %+v", m) + } +} + func TestParseInstanceMetadata_notManaged(t *testing.T) { t.Parallel() tests := []struct { -- 2.54.0 From aa8b36e1d0f793cca16bde3fe6c419657a52f617 Mon Sep 17 00:00:00 2001 From: kacerr Date: Thu, 10 Sep 2026 08:43:45 +0000 Subject: [PATCH 11/28] Add plan: trace every libvirt call in the KVM provider Co-Authored-By: Claude Fable 5.1 --- .../2026-09-10-0805-kvm-libvirt-tracing.md | 186 ++++++++++++++++++ 1 file changed, 186 insertions(+) create mode 100644 docs/plans/2026-09-10-0805-kvm-libvirt-tracing.md diff --git a/docs/plans/2026-09-10-0805-kvm-libvirt-tracing.md b/docs/plans/2026-09-10-0805-kvm-libvirt-tracing.md new file mode 100644 index 0000000..2c48cf7 --- /dev/null +++ b/docs/plans/2026-09-10-0805-kvm-libvirt-tracing.md @@ -0,0 +1,186 @@ +# Plan: Trace every libvirt call in the KVM provider +**Created:** 2026-09-10 08:05 + +## Context + +Step 7 of the KVM provider plan (live validation on `beelink`) has gone through +three fixes found only on the real host, and the current failure (guest hangs at +GRUB's "Booting" line) is being diagnosed blind: the provider logs at V(1)/V(2) +but nothing shows them. The integration test never installs a logger, so +controller-runtime prints "log.SetLogger(...) was never called" and swallows +everything; and even with a logger, the provider only logs per *operation* +(Create, Get…), not per libvirt RPC, so the sequence and timing of what was +actually sent to the hypervisor is invisible. + +Goal: one log line per libvirt call at debug level (`--zap-log-level=debug`, logr +V(1) in this repo's scheme: V(1) = outcome per operation, V(2) = curated detail, +V(5) = raw wire, per the GCP verbose-logging plans), with method, host, domain or +volume, duration and the libvirt error code on failure; and those lines visible in +`make test-integration-kvm` output. + +Assumption: this lands on the current `feat/kvm-provider` branch, since it exists +to finish that branch's validation, and the plan file plus execution log follow +CLAUDE.md conventions (plan committed alone first, execution summary alongside +each step). Say so if you want a separate branch instead. + +## Design + +- **Decorator, not call-site logging.** New `tracingAPI` implements the 17-method + `libvirtAPI` seam (`internal/provider/kvm/conn.go:26-44`), wrapping the real + client. Every method logs one V(1) line after the inner call returns. Applied in + `hostPool.call`, so every provider method, `image.go`'s cache and the + integration test's own `p.hosts.call` are traced with no per-site changes. +- **Logger passed explicitly.** `call(ctx, log, host, fn)` and `get(ctx, log, host)` + gain a `logr.Logger` parameter, matching how this package already threads `log` + into every helper. The five call sites (`kvm.go:169, 322, 458, 530`, + `kvm_integration_test.go:220`) already hold the right logger with + provider/host/name values. `call` does `log = log.WithName("libvirt")`, so trace + lines are named `kvm.libvirt`. +- **Wrap only when `log.V(1).Enabled()`.** Zero cost at info level; the + controller-runtime unfulfilled root sink reports disabled, so unit tests and + `conn_test.go`'s `hostClient{api: nil}` cases are unaffected. +- **One line per RPC at every level.** At V(2) the same line gains extra keys + (`xml`, `metadata`, `addresses`, `names`) instead of a second line. Upload and + download log offset/length/duration only, never bytes (the seed ISO carries + cloud-init). Domain and volume XML contain no secrets, so V(2) is fine for them. +- **Connection events traced too** in the pool: dial with duration, dial failure, + dead cached client redial, and both drop reasons (context cancelled, transport + error). `dialSSH`/`connectLibvirt`/`keepalive` stay untouched: changing + `hostDialer`'s signature would ripple into every test dialer, and the pool + already observes every event. +- **Rendering.** Handles by name only (`domain`, `pool`, `volume`), never UUID/Key. + `libvirt.DomainState` and `DomainInterfaceAddressesSource` have stringers, pass + them typed. Bitmask flags don't stringify, log as integers only when non-zero. + `libvirt.OptString` → first element or `""` under `uri`. +- **awaitOrCancel.** `done` runs inside fn's goroutine; on the normal path it logs + before `call` returns. On cancel, the abandoned RPC logs one late line when the + closed connection makes it fail: desirable in production, and in the integration + test only reachable in an already-failing timeout run (testr panics on a log after + the test ends; note that in the header, don't engineer around it). + +## Steps + +### Step 0 — Save plan +Copy this file to `docs/plans/2026-09-10-0805-kvm-libvirt-tracing.md`, commit alone. + +### Step 1 — `errors.go`: shared error fields +Factor `libvirtErrorFields(err error) []any` out of `logAPIError` (`errors.go:62-72`): +returns `libvirtCode` (uint32), `libvirtCodeName` (`libvirt.ErrorNumber(code)`, +e.g. `ErrNoDomain`), `libvirtMessage`; nil for non-libvirt errors; works through +`%w` wrapping via `errors.As`. `logAPIError` keeps its message and gains +`libvirtCodeName`. Test `TestLibvirtErrorFields` (table: libvirt.Error, wrapped, +plain error) in `errors_test.go`. + +### Step 2 — new `trace.go` +```go +type tracingAPI struct { inner libvirtAPI; log logr.Logger } +type tracedCall struct { t *tracingAPI; method string; start time.Time; kv []any } +func (t *tracingAPI) begin(method string, kv ...any) *tracedCall +func (c *tracedCall) detail(kv ...any) *tracedCall // appended only if V(2) enabled +func (c *tracedCall) done(err error, kv ...any) // one V(1) line +``` +`done` emits `log.V(1).Info("libvirt call", ...)` with keys in order: `method`, +begin args, result kv, `duration` (rounded to µs), then on error `error` plus +`libvirtErrorFields`. Per-method keys: + +| method | V(1) args | V(1) results | V(2) extra | +|---|---|---|---| +| DomainLookupByName | domain | | | +| DomainDefineXML | xmlBytes | domain | xml | +| DomainCreate, DomainDestroy | domain | | | +| DomainUndefineFlags | domain, flags | | | +| DomainGetState | domain | state, reason | | +| DomainGetMetadata | domain, type, uri | metadataBytes | metadata | +| DomainInterfaceAddresses | domain, source | interfaces (count) | addresses | +| ConnectListAllDomains | | count | names | +| StoragePoolLookupByName, StoragePoolRefresh | pool | | | +| StorageVolLookupByName | pool, volume | | | +| StorageVolGetPath | pool, volume | path | | +| StorageVolCreateXML | pool, xmlBytes | volume | xml | +| StorageVolUpload, StorageVolDownload | pool, volume, offset, length | | (never content) | +| StorageVolDelete | pool, volume | | | + +Return values pass through untouched. + +### Step 3 — `conn.go`: pool logging and wrapping +- `get(ctx, log, host)`: dead cached client → `"libvirt connection lost, redialing"`; + time the dial; `"libvirt host connected"` with `address`, `duration`; failure → + `"libvirt host dial failed"` with `address`, `duration`, `error`. +- `call(ctx, log, host, fn)`: `log = log.WithName("libvirt")`; wrap when + `log.V(1).Enabled()`; abort closure logs `"libvirt connection dropped"` + `reason="context cancelled"` before `drop`; transport-error branch logs the same + message with `reason="transport error"`. Update `call`'s doc comment. + +### Step 4 — call sites +`kvm.go` Create/Get/Delete pass `log`, ListByTag passes `hlog`; +`kvm_integration_test.go:220` passes `p.logger(ctx).WithValues("host", host.Name)`; +`conn_test.go` (12 `p.call(` sites) gets `var discard = logr.Discard()`. + +### Step 5 — integration test output +- `integrationLogger(t)`: `KVM_TEST_LOG_LEVEL` (default `1`, `strconv.Atoi`, + `t.Fatalf` on garbage) → `testr.NewWithOptions(t, testr.Options{LogTimestamp: + true, Verbosity: n})` (`github.com/go-logr/logr/testr`, present in logr v1.4.3, + no go.mod change); `logf.SetLogger(log)` once so nothing falls back to the + unfulfilled root. +- `TestKVM_lifecycle`: `ctx = logr.NewContext(ctx, log)` after `WithTimeout`; the + cleanup ctx gets the same logger. +- Header comment: document `KVM_TEST_LOG_LEVEL` (1 = one line per RPC, 2 adds + XML/metadata/addresses, 0 = only `t.Log`) and the late-line-after-timeout caveat. + +### Step 6 — unit tests +- `kvm_test.go`: `captureContext(verbosity)` like `gcp_test.go:277-286` but with a + mutex around the slice; `TestLogging_verbosityTiers` (mirror `gcp_test.go:322-399`): + V0 → no lines; V1 → `"msg"="libvirt call"`, `"method"="DomainDefineXML"`, + `"method"="StorageVolUpload"`, `"msg"="libvirt host connected"`, provider/host/name + values, `"libvirtCodeName"="ErrNoDomain"` from the first lookup, no `"xml"=`; + V2 → `"xml"="`, lines like + +``` +kvm.libvirt: "level"=1 "msg"="libvirt call" "provider"="kvm-integration" "host"="kvm-test" "name"="proxy-…" "method"="DomainLookupByName" "domain"="proxy-…" "duration"="2.1ms" "error"="…" "libvirtCode"=42 "libvirtCodeName"="ErrNoDomain" "libvirtMessage"="…" +``` + +plus one `libvirt host connected` line with `duration`, a `StorageVolUpload` line +whose duration is the seed upload, and no "log.SetLogger(...) was never called" +text. `KVM_TEST_LOG_LEVEL=2` shows `"xml"=" Date: Thu, 10 Sep 2026 08:57:17 +0000 Subject: [PATCH 12/28] Trace every libvirt call in the KVM provider at V(1), visible in the integration test A tracingAPI decorator wraps the 17-method libvirt seam and logs one "libvirt call" line per RPC after it returns: method, the identifiers it was given, curated results, duration, and on failure the error with its libvirt code, name and message. At V(2) the same line carries the domain and volume XML, metadata and addresses; upload bodies are never logged. The pool applies it in call() when V(1) is enabled and logs connect, dial-failure and drop events, taking the caller's logger explicitly like every other helper in the package. The integration test installs a testr logger at KVM_TEST_LOG_LEVEL (default 1) so the trace interleaves with t.Log output, which also ends the "log.SetLogger(...) was never called" warning. Docs gained a Logging section and the test env var; the KVM provider execution log records run 4 (guest hang at GRUB) that prompted this. Co-Authored-By: Claude Fable 5.1 --- docs/kvm-provider.md | 25 +- .../2026-09-04-1404-kvm-provider.md | 15 + .../2026-09-10-0805-kvm-libvirt-tracing.md | 57 ++++ docs/testing.md | 13 +- internal/provider/kvm/conn.go | 28 +- internal/provider/kvm/conn_test.go | 111 ++++++- internal/provider/kvm/errors.go | 21 +- internal/provider/kvm/errors_test.go | 24 ++ internal/provider/kvm/kvm.go | 8 +- internal/provider/kvm/kvm_integration_test.go | 47 ++- internal/provider/kvm/kvm_test.go | 93 ++++++ internal/provider/kvm/trace.go | 201 ++++++++++++ internal/provider/kvm/trace_test.go | 300 ++++++++++++++++++ 13 files changed, 904 insertions(+), 39 deletions(-) create mode 100644 docs/plans-executions/2026-09-10-0805-kvm-libvirt-tracing.md create mode 100644 internal/provider/kvm/trace.go create mode 100644 internal/provider/kvm/trace_test.go diff --git a/docs/kvm-provider.md b/docs/kvm-provider.md index c79899e..b7aed5f 100644 --- a/docs/kvm-provider.md +++ b/docs/kvm-provider.md @@ -186,6 +186,27 @@ replaces the VM). - `virsh domifaddr proxy-abc… --source arp` / `--source agent` shows what the operator sees when it looks for the IP. +## Logging + +The provider logs through controller-runtime's logger, named `kvm`, with +`provider`, `host` and `name` (the domain) on every line. + +- `--zap-log-level=debug` (logr V(1)): one line per provider operation + (`domain defined`, `KVM domain ensured running`, ...) **and one + `libvirt call` line per libvirt RPC**, named `kvm.libvirt`, with + `method`, the identifiers it was given (`domain`, `pool`, `volume`), + curated results (`state`, `path`, `interfaces`), `duration`, and on + failure `error` plus `libvirtCode`, `libvirtCodeName` and + `libvirtMessage`. Connection events (`libvirt host connected`, dial + failures, `libvirt connection dropped` with a reason) log at the same + level. A Create is about ten RPCs; a Get is four. +- `--zap-log-level=2`: the same lines also carry the domain and volume + `xml`, the `metadata` element, the interface `addresses` and the + domain `names` from a listing. +- Cloud-init is never logged at any level: the seed ISO upload reports + only `length` and `duration`. There is no raw wire tier like the GCP + provider's V(5); go-libvirt exposes no hook for it. + ## Known gaps - **No quota or capacity detection.** libvirt reports an out-of-space @@ -211,4 +232,6 @@ replaces the VM). Unit tests run over an in-memory fake of the libvirt API. The optional integration test against a real host is described in -[testing.md](testing.md#the-kvm-integration-test-real-hypervisor). +[testing.md](testing.md#the-kvm-integration-test-real-hypervisor); its +`-v` output includes the `libvirt call` trace above, at the verbosity +`KVM_TEST_LOG_LEVEL` selects (default 1). diff --git a/docs/plans-executions/2026-09-04-1404-kvm-provider.md b/docs/plans-executions/2026-09-04-1404-kvm-provider.md index 18f61b3..ff9d7b1 100644 --- a/docs/plans-executions/2026-09-04-1404-kvm-provider.md +++ b/docs/plans-executions/2026-09-04-1404-kvm-provider.md @@ -462,3 +462,18 @@ lookup already selected the element by URI. The fake now strips the declaration when storing metadata, so the provider's Get and ListByTag tests parse the shape libvirt really returns, and a new `TestParseInstanceMetadata_libvirtBareForm` pins it. + +### Run 4: guest hangs at GRUB's "Booting" line + +Create and start passed; Get polled `Provisioning` with no IP while the +VM sat at 100 % CPU after GRUB printed "Booting `Debian GNU/Linux`" and +before the kernel wrote anything to the serial console. The domain XML +matches a working VM on the same host (q35, BIOS, host-passthrough, +virtio disk, SATA cdrom, br0), so the difference is the disk content: +the cached image was downloaded without a checksum and never verified. +Diagnosis was handed to the user (sha512 against Debian's published +sums, `qemu-img check`, and a `virt-install --import` of the same image +outside the operator). Because the provider's own logs were invisible in +the test output, this run also produced a side plan: +[2026-09-10-0805-kvm-libvirt-tracing](2026-09-10-0805-kvm-libvirt-tracing.md), +which traces every libvirt RPC at V(1) and routes it into `-v` output. diff --git a/docs/plans-executions/2026-09-10-0805-kvm-libvirt-tracing.md b/docs/plans-executions/2026-09-10-0805-kvm-libvirt-tracing.md new file mode 100644 index 0000000..3a4ac3c --- /dev/null +++ b/docs/plans-executions/2026-09-10-0805-kvm-libvirt-tracing.md @@ -0,0 +1,57 @@ +# Execution: Trace every libvirt call in the KVM provider + +Plan: [2026-09-10-0805-kvm-libvirt-tracing.md](../plans/2026-09-10-0805-kvm-libvirt-tracing.md) + +- [x] Step 0 — Save plan +- [x] Step 1 — `errors.go`: shared error fields +- [x] Step 2 — `trace.go`: the `tracingAPI` decorator +- [x] Step 3 — `conn.go`: pool logging and wrapping +- [x] Step 4 — Call sites +- [x] Step 5 — Integration test output +- [x] Step 6 — Unit tests +- [ ] Step 7 — Docs, execution log, CHANGELOG (docs and this log done; CHANGELOG waits for the live run) + +## Steps 1–6 — Tracer, pool logging, tests, integration wiring + +Done as one commit on `feat/kvm-provider`; the plan's assumption that +this belongs on the KVM branch rather than its own feature branch was +accepted at plan approval. + +The code follows the plan closely. Things that came out differently +or are worth knowing: + +- **Logger names join with `/` in funcr, `.` in zap.** The trace lines + are `kvm/libvirt` in the unit-test capture and in `testr` output, and + `kvm.libvirt` under the operator's zap logger. The docs use the zap + spelling; the unit test asserts the funcr one. First run of the tier + test failed on exactly this. +- **Cancel is not reported as a transport error.** The original `call` + dropped the connection twice on cancellation (once from the abort + closure, once from the "not a libvirt error" branch, harmlessly, since + drop is idempotent). With a log line on each drop that would have + printed both `context cancelled` and `transport error` for one event, + so the transport branch now skips `ctx.Err()`. Covered by + `TestHostPool_call_logsDropReason`. +- **`ConnectListAllDomains` in the fake includes every defined domain**, + so the trace test's fixture (one running managed domain plus one + stopped domain for `DomainCreate` to start) lists two; the V(2) + assertion checks for the `names` key rather than an exact list. +- **`libvirtErrorFields` also feeds `logAPIError`**, which therefore + gains `libvirtCodeName` (e.g. `ErrNoDomain`) next to the numeric code. +- The seam-coverage guard (`reflect` count of `libvirtAPI` methods vs + the test table) is the only thing standing between a future 18th RPC + and a silently untraced call; keep it. + +Verification run before committing: + +```bash +go build ./... && go vet ./... && go vet -tags=integration ./internal/provider/kvm/ +go test -short ./... +go test -race -count=1 -cover ./internal/provider/kvm/ # 90.2 %, up from 88.2 +go test -tags=integration -count=1 -run TestKVM ./internal/provider/kvm/ # self-skips without env +``` + +Worth noting: the live run (`make test-integration-kvm` against beelink) +had not happened when this was written; it is what Step 7 of the KVM +provider plan is waiting on, and its output is the acceptance test for +this feature. Expected shape is in the plan's Verification section. diff --git a/docs/testing.md b/docs/testing.md index 5ee1633..206a63c 100644 --- a/docs/testing.md +++ b/docs/testing.md @@ -56,7 +56,8 @@ manual runs too. | `internal/provider/gcp` | `insert_test.go`, `errors_test.go`, `gcp_test.go` | field-by-field `buildInsertRequest`; HTTP-code classification table; zone-qualified IDs, 409-is-success, 9-row state mapping, ListByTag filter + `ReturnPartialSuccess` | pure builder needs **no fake**; the rest uses the flattened `instancesAPI` seam | | `internal/provider/kvm` | `domainxml_test.go`, `volume_test.go`, `seed_test.go`, `id_test.go`, `metadata_test.go` | pure builders: domain/volume XML unmarshalled back and checked field by field, seed ISO reopened with the same library, providerID round-trip, metadata in both namespace spellings | pure | | | `errors_test.go`, `conn_test.go` | libvirt-code classification table; connection pool (reuse, redial, drop on transport error, cancel releases a blocked RPC, one dial per host under concurrency); SSH dial, host-key policy, keepalive | fake dialer; a **real in-process `x/crypto/ssh` server** forwarding a unix socket | -| | `image_test.go`, `kvm_test.go` | image cache (fresh fetch, ready, marker missing, checksum change/mismatch, 4xx/5xx, concurrent callers); Create resume after every partial step, leftover truncated seed, state/IP mapping, every Delete already-gone case, ListByTag across hosts / one down / unreadable domain | `httptest` image server + in-memory `fakeAPI` behind the `libvirtAPI` seam | +| | `image_test.go`, `kvm_test.go` | image cache (fresh fetch, ready, marker missing, checksum change/mismatch, 4xx/5xx, concurrent callers); Create resume after every partial step, leftover truncated seed, state/IP mapping, every Delete already-gone case, ListByTag across hosts / one down / unreadable domain; logging tiers (V0 silent, V1 one line per RPC, V2 XML) with cloud-init never logged | `httptest` image server + in-memory `fakeAPI` behind the `libvirtAPI` seam | +| | `trace_test.go`, `conn_test.go`, `errors_test.go` | every `libvirtAPI` method traced exactly once (guarded by reflection on the interface), results and errors pass through, libvirt error fields, detail keys only at V2, upload content never logged; pool connect/drop lines, no wrapping below V1 | `fakeAPI` + `funcr` capture | | | `kvm_integration_test.go` (`-tags=integration`) | one real VM: image fetch, Create, idempotent re-Create, Running with IP, ListByTag, Delete, pool state afterwards | a **real hypervisor** — see below | | `internal/health` | `probe_test.go` | probes through a **real CONNECT-capable proxy stub** to a real TLS server: tunnel success, refused CONNECT, wrong status, dead proxy, plain-http forward | `httptest` + hijacked bidirectional tunnel | | | `engine_test.go` | thresholds, first-verdict, latency suppression matrix, dropped-event retry, stale-probe UID guard, tick scheduling/seeding/pruning, `Start` end-to-end | fake `client.Reader` + injected `probeFn` | @@ -116,8 +117,14 @@ make test-integration-kvm Optional: `KVM_TEST_KNOWN_HOSTS` (without it host keys are **not** verified — the log says so), `KVM_TEST_IMAGE_SHA256`, `KVM_TEST_POOL` (default `default`), `KVM_TEST_SOCKET`, `KVM_TEST_TIMEOUT` (default -`20m`, must cover the first-time image download), and `KVM_TEST_KEEP` -to leave the VM behind for `virsh console` when something goes wrong. +`20m`, must cover the first-time image download), `KVM_TEST_KEEP` +to leave the VM behind for `virsh console` when something goes wrong, +and `KVM_TEST_LOG_LEVEL` (default `1`) for how much of the provider's +logging the `-v` output shows: at `1` every libvirt RPC is one +`libvirt call` line with method, identifiers, duration and error code, +interleaved with the test's own `t.Log` lines; `2` adds the domain and +volume XML, metadata and addresses; `0` leaves only `t.Log`. See the +Logging section of [kvm-provider.md](kvm-provider.md#logging). What one run asserts, in order: `New` dials the host; `Create` returns a `hosts//domains/` ID (downloading the image on a cold host — diff --git a/internal/provider/kvm/conn.go b/internal/provider/kvm/conn.go index b0034b0..4531532 100644 --- a/internal/provider/kvm/conn.go +++ b/internal/provider/kvm/conn.go @@ -2,6 +2,7 @@ package kvm import ( "context" + "errors" "fmt" "io" "net" @@ -10,6 +11,7 @@ import ( "time" libvirt "github.com/digitalocean/go-libvirt" + "github.com/go-logr/logr" "golang.org/x/crypto/ssh" "golang.org/x/crypto/ssh/knownhosts" @@ -95,7 +97,7 @@ func (p *hostPool) entry(name string) *hostEntry { return e } -func (p *hostPool) get(ctx context.Context, host provider.KVMHost) (*hostClient, error) { +func (p *hostPool) get(ctx context.Context, log logr.Logger, host provider.KVMHost) (*hostClient, error) { e := p.entry(host.Name) e.mu.Lock() defer e.mu.Unlock() @@ -103,13 +105,17 @@ func (p *hostPool) get(ctx context.Context, host provider.KVMHost) (*hostClient, if e.client.alive() { return e.client, nil } + log.V(1).Info("libvirt connection lost, redialing", "address", host.Address) e.client.close() e.client = nil } + start := time.Now() c, err := p.dial(ctx, host) if err != nil { + log.V(1).Info("libvirt host dial failed", "address", host.Address, "duration", time.Since(start).Round(time.Millisecond), "error", err.Error()) return nil, fmt.Errorf("connecting to host %s (%s): %w", host.Name, host.Address, err) } + log.V(1).Info("libvirt host connected", "address", host.Address, "duration", time.Since(start).Round(time.Millisecond)) e.client = c return c, nil } @@ -131,14 +137,24 @@ func (p *hostPool) drop(host string, c *hostClient) { // is closed — which unblocks fn with an error — and ctx.Err() is returned. // A transport error (anything that is not a libvirt.Error) also drops the // cached connection so the next call redials instead of reusing a dead -// link. -func (p *hostPool) call(ctx context.Context, host provider.KVMHost, fn func(api libvirtAPI) error) error { - c, err := p.get(ctx, host) +// link. When log has V(1) enabled, every RPC fn makes is traced through it +// (see tracingAPI), as are connection events. +func (p *hostPool) call(ctx context.Context, log logr.Logger, host provider.KVMHost, fn func(api libvirtAPI) error) error { + log = log.WithName("libvirt") + c, err := p.get(ctx, log, host) if err != nil { return err } - err = awaitOrCancel(ctx, func() error { return fn(c.api) }, func() { p.drop(host.Name, c) }) - if err != nil && !isLibvirtError(err) { + api := c.api + if log.V(1).Enabled() { + api = &tracingAPI{inner: c.api, log: log} + } + err = awaitOrCancel(ctx, func() error { return fn(api) }, func() { + log.V(1).Info("libvirt connection dropped", "reason", "context cancelled", "error", ctx.Err().Error()) + p.drop(host.Name, c) + }) + if err != nil && !isLibvirtError(err) && !errors.Is(err, ctx.Err()) { + log.V(1).Info("libvirt connection dropped", "reason", "transport error", "error", err.Error()) p.drop(host.Name, c) } return err diff --git a/internal/provider/kvm/conn_test.go b/internal/provider/kvm/conn_test.go index 732413b..d6105f1 100644 --- a/internal/provider/kvm/conn_test.go +++ b/internal/provider/kvm/conn_test.go @@ -18,6 +18,7 @@ import ( "time" libvirt "github.com/digitalocean/go-libvirt" + "github.com/go-logr/logr" "golang.org/x/crypto/ssh" "golang.org/x/crypto/ssh/knownhosts" @@ -43,6 +44,8 @@ type fakeDialer struct { err error delay time.Duration clients []*fakeClient + // api is what the dialled client exposes; nil for pool-only tests. + api libvirtAPI } func (d *fakeDialer) dial(ctx context.Context, _ provider.KVMHost) (*hostClient, error) { @@ -61,7 +64,7 @@ func (d *fakeDialer) dial(ctx context.Context, _ provider.KVMHost) (*hostClient, } fc := &fakeClient{done: make(chan struct{})} d.clients = append(d.clients, fc) - return &hostClient{api: nil, done: fc.done, close: fc.close}, nil + return &hostClient{api: d.api, done: fc.done, close: fc.close}, nil } func (d *fakeDialer) dialCount() int { @@ -72,6 +75,8 @@ func (d *fakeDialer) dialCount() int { var testHost = provider.KVMHost{Name: "nas1", Address: "nas1:22", User: "virt", Socket: defaultSocket, Pool: "default", Bridge: "br0"} +var discard = logr.Discard() + func noop(libvirtAPI) error { return nil } func TestHostPool_call_reusesConnection(t *testing.T) { @@ -79,7 +84,7 @@ func TestHostPool_call_reusesConnection(t *testing.T) { d := &fakeDialer{} p := newHostPool(d.dial) for range 3 { - if err := p.call(context.Background(), testHost, noop); err != nil { + if err := p.call(context.Background(), discard, testHost, noop); err != nil { t.Fatalf("call() error = %v", err) } } @@ -92,11 +97,11 @@ func TestHostPool_call_redialsAfterDisconnect(t *testing.T) { t.Parallel() d := &fakeDialer{} p := newHostPool(d.dial) - if err := p.call(context.Background(), testHost, noop); err != nil { + if err := p.call(context.Background(), discard, testHost, noop); err != nil { t.Fatalf("call() error = %v", err) } d.clients[0].close() - if err := p.call(context.Background(), testHost, noop); err != nil { + if err := p.call(context.Background(), discard, testHost, noop); err != nil { t.Fatalf("second call() error = %v", err) } if d.dialCount() != 2 { @@ -108,14 +113,14 @@ func TestHostPool_call_transportErrorDropsConnection(t *testing.T) { t.Parallel() d := &fakeDialer{} p := newHostPool(d.dial) - err := p.call(context.Background(), testHost, func(libvirtAPI) error { return io.ErrUnexpectedEOF }) + err := p.call(context.Background(), discard, testHost, func(libvirtAPI) error { return io.ErrUnexpectedEOF }) if !errors.Is(err, io.ErrUnexpectedEOF) { t.Fatalf("call() error = %v, want the transport error passed through", err) } if !d.clients[0].closed.Load() { t.Error("connection not closed after a transport error") } - if err := p.call(context.Background(), testHost, noop); err != nil { + if err := p.call(context.Background(), discard, testHost, noop); err != nil { t.Fatalf("call() after transport error = %v", err) } if d.dialCount() != 2 { @@ -127,11 +132,11 @@ func TestHostPool_call_libvirtErrorKeepsConnection(t *testing.T) { t.Parallel() d := &fakeDialer{} p := newHostPool(d.dial) - err := p.call(context.Background(), testHost, func(libvirtAPI) error { return lvErr(libvirt.ErrNoDomain) }) + err := p.call(context.Background(), discard, testHost, func(libvirtAPI) error { return lvErr(libvirt.ErrNoDomain) }) if !hasCode(err, libvirt.ErrNoDomain) { t.Fatalf("call() error = %v, want the libvirt error passed through", err) } - if err := p.call(context.Background(), testHost, noop); err != nil { + if err := p.call(context.Background(), discard, testHost, noop); err != nil { t.Fatalf("call() error = %v", err) } if d.dialCount() != 1 { @@ -146,7 +151,7 @@ func TestHostPool_call_dialErrorPropagates(t *testing.T) { t.Parallel() d := &fakeDialer{err: errors.New("ssh: handshake failed")} p := newHostPool(d.dial) - err := p.call(context.Background(), testHost, noop) + err := p.call(context.Background(), discard, testHost, noop) if err == nil || !strings.Contains(err.Error(), "nas1") || !strings.Contains(err.Error(), "handshake failed") { t.Errorf("call() error = %v, want the dial error naming the host", err) } @@ -162,7 +167,7 @@ func TestHostPool_call_cancelAbandonsBlockedCall(t *testing.T) { time.Sleep(20 * time.Millisecond) cancel() }() - err := p.call(ctx, testHost, func(libvirtAPI) error { + err := p.call(ctx, discard, testHost, func(libvirtAPI) error { // Simulates an RPC stuck on a hung hypervisor: it only returns once // the pool closes the connection out from under it. <-d.clients[0].done @@ -177,7 +182,7 @@ func TestHostPool_call_cancelAbandonsBlockedCall(t *testing.T) { case <-time.After(time.Second): t.Fatal("blocked call was not released by closing the connection") } - if err := p.call(context.Background(), testHost, noop); err != nil { + if err := p.call(context.Background(), discard, testHost, noop); err != nil { t.Fatalf("call() after cancel = %v", err) } if d.dialCount() != 2 { @@ -192,7 +197,7 @@ func TestHostPool_call_concurrentCallersShareOneDial(t *testing.T) { var wg sync.WaitGroup for range 8 { wg.Go(func() { - if err := p.call(context.Background(), testHost, noop); err != nil { + if err := p.call(context.Background(), discard, testHost, noop); err != nil { t.Errorf("call() error = %v", err) } }) @@ -210,7 +215,7 @@ func TestHostPool_call_hostsAreIndependent(t *testing.T) { other := testHost other.Name = "lab2" for _, h := range []provider.KVMHost{testHost, other, testHost, other} { - if err := p.call(context.Background(), h, noop); err != nil { + if err := p.call(context.Background(), discard, h, noop); err != nil { t.Fatalf("call(%s) error = %v", h.Name, err) } } @@ -567,3 +572,83 @@ func TestKeepalive_keepsHealthyClientOpen(t *testing.T) { } conn.Close() } + +func TestHostPool_call_tracesAtV1(t *testing.T) { + t.Parallel() + d := &fakeDialer{api: readyHost()} + p := newHostPool(d.dial) + log, snapshot := captureLogger(1) + err := p.call(context.Background(), log, testHost, func(api libvirtAPI) error { + _, err := api.StoragePoolLookupByName("default") + return err + }) + if err != nil { + t.Fatalf("call() error = %v", err) + } + joined := strings.Join(snapshot(), "\n") + for _, want := range []string{`"msg"="libvirt host connected"`, `"address"="nas1:22"`, `"method"="StoragePoolLookupByName"`, `"pool"="default"`, "libvirt "} { + if !strings.Contains(joined, want) { + t.Errorf("output missing %s:\n%s", want, joined) + } + } +} + +func TestHostPool_call_noTracingBelowV1(t *testing.T) { + t.Parallel() + inner := readyHost() + d := &fakeDialer{api: inner} + p := newHostPool(d.dial) + log, snapshot := captureLogger(0) + err := p.call(context.Background(), log, testHost, func(api libvirtAPI) error { + if api != libvirtAPI(inner) { + t.Errorf("fn received %T, want the bare fake when V(1) is off", api) + } + return nil + }) + if err != nil { + t.Fatalf("call() error = %v", err) + } + if lines := snapshot(); len(lines) != 0 { + t.Errorf("verbosity 0 logged %d lines:\n%s", len(lines), strings.Join(lines, "\n")) + } +} + +func TestHostPool_call_logsDropReason(t *testing.T) { + t.Parallel() + t.Run("transport error", func(t *testing.T) { + t.Parallel() + d := &fakeDialer{} + p := newHostPool(d.dial) + log, snapshot := captureLogger(1) + _ = p.call(context.Background(), log, testHost, func(libvirtAPI) error { return io.ErrUnexpectedEOF }) + joined := strings.Join(snapshot(), "\n") + if !strings.Contains(joined, `"msg"="libvirt connection dropped"`) || !strings.Contains(joined, `"reason"="transport error"`) { + t.Errorf("output missing the drop line:\n%s", joined) + } + }) + t.Run("context cancelled", func(t *testing.T) { + t.Parallel() + d := &fakeDialer{} + p := newHostPool(d.dial) + log, snapshot := captureLogger(1) + ctx, cancel := context.WithCancel(context.Background()) + cancel() + released := make(chan struct{}) + err := p.call(ctx, log, testHost, func(libvirtAPI) error { + <-d.clients[0].done + close(released) + return io.EOF + }) + if !errors.Is(err, context.Canceled) { + t.Fatalf("call() error = %v, want context.Canceled", err) + } + <-released + joined := strings.Join(snapshot(), "\n") + if !strings.Contains(joined, `"reason"="context cancelled"`) { + t.Errorf("output missing the cancel drop line:\n%s", joined) + } + if strings.Contains(joined, `"reason"="transport error"`) { + t.Errorf("cancel was also reported as a transport error:\n%s", joined) + } + }) +} diff --git a/internal/provider/kvm/errors.go b/internal/provider/kvm/errors.go index 51bb20e..6896c01 100644 --- a/internal/provider/kvm/errors.go +++ b/internal/provider/kvm/errors.go @@ -63,10 +63,21 @@ func logAPIError(log logr.Logger, op string, err error) { if !log.V(1).Enabled() { return } - kv := []any{"op", op, "error", err.Error()} - var lerr libvirt.Error - if errors.As(err, &lerr) { - kv = append(kv, "libvirtCode", lerr.Code, "libvirtMessage", lerr.Message) - } + kv := append([]any{"op", op, "error", err.Error()}, libvirtErrorFields(err)...) log.V(1).Info("libvirt call failed", kv...) } + +// libvirtErrorFields renders a daemon-reported error as log key/values: +// the numeric code, its constant name (ErrNoDomain, ...) and the daemon's +// message. Nil for transport and other non-libvirt errors. +func libvirtErrorFields(err error) []any { + var lerr libvirt.Error + if !errors.As(err, &lerr) { + return nil + } + return []any{ + "libvirtCode", lerr.Code, + "libvirtCodeName", libvirt.ErrorNumber(lerr.Code).String(), + "libvirtMessage", lerr.Message, + } +} diff --git a/internal/provider/kvm/errors_test.go b/internal/provider/kvm/errors_test.go index 1f00ad7..6ce0065 100644 --- a/internal/provider/kvm/errors_test.go +++ b/internal/provider/kvm/errors_test.go @@ -5,6 +5,7 @@ import ( "errors" "fmt" "io" + "reflect" "testing" libvirt "github.com/digitalocean/go-libvirt" @@ -67,3 +68,26 @@ func TestHasCode(t *testing.T) { t.Error("io.EOF classified as a libvirt error") } } + +func TestLibvirtErrorFields(t *testing.T) { + t.Parallel() + tests := []struct { + name string + err error + want []any + }{ + {"libvirt error", lvErr(libvirt.ErrNoDomain), []any{"libvirtCode", uint32(42), "libvirtCodeName", "ErrNoDomain", "libvirtMessage", "ErrNoDomain"}}, + {"wrapped libvirt error", fmt.Errorf("looking up: %w", lvErr(libvirt.ErrNoDomain)), []any{"libvirtCode", uint32(42), "libvirtCodeName", "ErrNoDomain", "libvirtMessage", "ErrNoDomain"}}, + {"transport error", io.EOF, nil}, + {"cancelled", context.Canceled, nil}, + } + for _, tc := range tests { + t.Run(tc.name, func(t *testing.T) { + t.Parallel() + got := libvirtErrorFields(tc.err) + if !reflect.DeepEqual(got, tc.want) { + t.Errorf("libvirtErrorFields() = %#v, want %#v", got, tc.want) + } + }) + } +} diff --git a/internal/provider/kvm/kvm.go b/internal/provider/kvm/kvm.go index 6d94dc4..ed4613e 100644 --- a/internal/provider/kvm/kvm.go +++ b/internal/provider/kvm/kvm.go @@ -166,7 +166,7 @@ func (p *Provider) Create(ctx context.Context, req provider.CreateRequest) (stri "vcpus", mt.VCPUs, "memoryMiB", mt.MemoryMiB, "diskSizeGb", p.cfg.DiskSizeGB, "image", src.volumeName(), "cloudInitBytes", len(req.CloudInit)) - err = p.hosts.call(ctx, host, func(api libvirtAPI) error { + err = p.hosts.call(ctx, log, host, func(api libvirtAPI) error { dom, err := api.DomainLookupByName(req.Name) if err == nil { log.V(1).Info("domain already defined") @@ -319,7 +319,7 @@ func (p *Provider) Get(ctx context.Context, providerID string) (*provider.Instan } log := p.logger(ctx).WithValues("host", host.Name, "name", name) var inst *provider.Instance - err = p.hosts.call(ctx, host, func(api libvirtAPI) error { + err = p.hosts.call(ctx, log, host, func(api libvirtAPI) error { dom, err := api.DomainLookupByName(name) if err != nil { return fmt.Errorf("looking up domain: %w", err) @@ -455,7 +455,7 @@ func (p *Provider) Delete(ctx context.Context, providerID string) error { return p.wrapErr("delete", providerID, err) } log := p.logger(ctx).WithValues("host", host.Name, "name", name) - err = p.hosts.call(ctx, host, func(api libvirtAPI) error { + err = p.hosts.call(ctx, log, host, func(api libvirtAPI) error { if err := p.removeDomain(log, api, name); err != nil { return err } @@ -527,7 +527,7 @@ func (p *Provider) ListByTag(ctx context.Context) ([]provider.Instance, error) { var out []provider.Instance for _, host := range p.cfg.Hosts { hlog := log.WithValues("host", host.Name) - err := p.hosts.call(ctx, host, func(api libvirtAPI) error { + err := p.hosts.call(ctx, hlog, host, func(api libvirtAPI) error { doms, _, err := api.ConnectListAllDomains(1, 0) if err != nil { return fmt.Errorf("listing domains: %w", err) diff --git a/internal/provider/kvm/kvm_integration_test.go b/internal/provider/kvm/kvm_integration_test.go index 1e1faf7..32bf5a7 100644 --- a/internal/provider/kvm/kvm_integration_test.go +++ b/internal/provider/kvm/kvm_integration_test.go @@ -25,6 +25,14 @@ package kvm // KVM_TEST_SOCKET libvirt socket path on the host // KVM_TEST_TIMEOUT overall budget incl. the image download (default 20m) // KVM_TEST_KEEP set to anything to leave the VM behind for inspection +// KVM_TEST_LOG_LEVEL provider log verbosity shown in -v output (default 1: +// one line per libvirt call; 2 adds XML, metadata and +// addresses; 0 leaves only the t.Log lines) +// +// Provider logs go through a testr logger, so they interleave with t.Log +// output. A run that hits its timeout may log one more line from the RPC +// the pool abandoned, after the test has ended; that is a symptom of the +// timeout, not a bug in the tracer. import ( "context" @@ -34,11 +42,15 @@ import ( "fmt" "net" "os" + "strconv" "testing" "time" libvirt "github.com/digitalocean/go-libvirt" + "github.com/go-logr/logr" + "github.com/go-logr/logr/testr" "k8s.io/apimachinery/pkg/types" + logf "sigs.k8s.io/controller-runtime/pkg/log" "gitea.home.hrajfrisbee.cz/kacerr/egress-proxies-operator/internal/provider" ) @@ -50,9 +62,10 @@ const ( ) type integrationEnv struct { - cfg provider.ProviderConfig - timeout time.Duration - keep bool + cfg provider.ProviderConfig + timeout time.Duration + keep bool + logLevel int } func loadIntegrationEnv(t *testing.T) integrationEnv { @@ -97,7 +110,26 @@ func loadIntegrationEnv(t *testing.T) integrationEnv { } timeout = d } - return integrationEnv{cfg: pc, timeout: timeout, keep: os.Getenv("KVM_TEST_KEEP") != ""} + logLevel := 1 + if v := os.Getenv("KVM_TEST_LOG_LEVEL"); v != "" { + n, err := strconv.Atoi(v) + if err != nil { + t.Fatalf("KVM_TEST_LOG_LEVEL: %v", err) + } + logLevel = n + } + return integrationEnv{cfg: pc, timeout: timeout, keep: os.Getenv("KVM_TEST_KEEP") != "", logLevel: logLevel} +} + +// integrationLogger routes provider logs into the test output at the +// requested verbosity, and installs it as controller-runtime's root logger +// so nothing falls back to the unfulfilled delegating sink (which would +// print its "SetLogger was never called" warning and swallow the lines). +func integrationLogger(t *testing.T, verbosity int) logr.Logger { + t.Helper() + log := testr.NewWithOptions(t, testr.Options{LogTimestamp: true, Verbosity: verbosity}) + logf.SetLogger(log) + return log } func randomUID(t *testing.T) string { @@ -114,7 +146,8 @@ func randomUID(t *testing.T) string { // stage depends on the state the previous one left on the hypervisor. func TestKVM_lifecycle(t *testing.T) { env := loadIntegrationEnv(t) - ctx, cancel := context.WithTimeout(context.Background(), env.timeout) + log := integrationLogger(t, env.logLevel) + ctx, cancel := context.WithTimeout(logr.NewContext(context.Background(), log), env.timeout) defer cancel() prov, err := New(ctx, env.cfg) @@ -143,7 +176,7 @@ func TestKVM_lifecycle(t *testing.T) { t.Logf("KVM_TEST_KEEP set: leaving domain %s and its volumes on %s", req.Name, host.Address) return } - cctx, ccancel := context.WithTimeout(context.Background(), 2*time.Minute) + cctx, ccancel := context.WithTimeout(logr.NewContext(context.Background(), log), 2*time.Minute) defer ccancel() if err := p.Delete(cctx, formatProviderID(host.Name, req.Name)); err != nil { t.Errorf("cleanup Delete: %v", err) @@ -217,7 +250,7 @@ func TestKVM_lifecycle(t *testing.T) { if err != nil { t.Fatal(err) } - err = p.hosts.call(ctx, host, func(api libvirtAPI) error { + err = p.hosts.call(ctx, p.logger(ctx).WithValues("host", host.Name), host, func(api libvirtAPI) error { pool, err := api.StoragePoolLookupByName(host.Pool) if err != nil { return err diff --git a/internal/provider/kvm/kvm_test.go b/internal/provider/kvm/kvm_test.go index 087c371..60d50da 100644 --- a/internal/provider/kvm/kvm_test.go +++ b/internal/provider/kvm/kvm_test.go @@ -885,3 +885,96 @@ func TestNew_badKeyPathFailsFast(t *testing.T) { t.Fatal("New succeeded with a missing key file") } } + +// runAllOps drives one full lifecycle over a ready fake host so logging +// tests see every provider method and every libvirt call. +func runAllOps(t *testing.T, ctx context.Context, req provider.CreateRequest) { + t.Helper() + p := newTestProvider(hostFakes{"nas1": readyHost()}, "nas1") + id, err := p.Create(ctx, req) + if err != nil { + t.Fatalf("Create: %v", err) + } + if _, err := p.Get(ctx, id); err != nil { + t.Fatalf("Get: %v", err) + } + if _, err := p.ListByTag(ctx); err != nil { + t.Fatalf("ListByTag: %v", err) + } + if err := p.Delete(ctx, id); err != nil { + t.Fatalf("Delete: %v", err) + } +} + +func TestLogging_verbosityTiers(t *testing.T) { + t.Parallel() + tests := []struct { + name string + verbosity int + wantLines []string + absentLines []string + }{ + { + name: "v0 stays silent", + verbosity: 0, + absentLines: []string{"libvirt call", "KVM domain ensured running"}, + }, + { + name: "v1 logs one line per libvirt call plus the operation outcomes", + verbosity: 1, + wantLines: []string{ + `kvm/libvirt "level"=1 "msg"="libvirt host connected"`, + `"msg"="libvirt call"`, + `"method"="DomainLookupByName"`, + `"libvirtCodeName"="ErrNoDomain"`, + `"method"="StorageVolUpload"`, + `"method"="DomainDefineXML"`, + `"method"="DomainCreate"`, + `"method"="DomainInterfaceAddresses"`, + `"method"="ConnectListAllDomains"`, + `"method"="DomainUndefineFlags"`, + `"method"="StorageVolDelete"`, + `"provider"="kvm-home"`, + `"host"="nas1"`, + `"name"="` + testName + `"`, + `"msg"="KVM domain ensured running"`, + }, + absentLines: []string{`"xml"=`, `"metadata"=`}, + }, + { + name: "v2 adds xml, metadata and addresses on the same lines", + verbosity: 2, + wantLines: []string{`"xml"=") and a snapshot function for the +// lines it recorded. The sink is mutex-guarded: a cancelled call logs from +// the abandoned goroutine. +func captureLogger(verbosity int) (logr.Logger, func() []string) { + var mu sync.Mutex + var lines []string + log := funcr.New(func(prefix, args string) { + mu.Lock() + defer mu.Unlock() + lines = append(lines, prefix+" "+args) + }, funcr.Options{Verbosity: verbosity}) + return log, func() []string { + mu.Lock() + defer mu.Unlock() + return append([]string(nil), lines...) + } +} + +func captureContext(verbosity int) (context.Context, func() []string) { + log, snapshot := captureLogger(verbosity) + return logr.NewContext(context.Background(), log), snapshot +} + +const stoppedName = "proxy-stopped" + +// tracedHost is a fake with one managed running domain and its volumes, so +// every method of the seam has something to act on. +func tracedHost(t *testing.T) *fakeAPI { + t.Helper() + api := readyHost() + managedDomain(t, api, testName, testUID, true, map[uint32][]libvirt.DomainInterface{ + uint32(libvirt.DomainInterfaceAddressesSrcArp): {{Name: "vnet0", Addrs: []libvirt.DomainIPAddr{ipv4("192.168.1.50")}}}, + }) + api.putDomain(&fakeDomain{name: stoppedName}) + return api +} + +func TestTracingAPI_logsEveryMethod(t *testing.T) { + t.Parallel() + pool := libvirt.StoragePool{Name: "default"} + dom := libvirt.Domain{Name: testName} + overlay := libvirt.StorageVol{Pool: "default", Name: overlayVolumeName(testName)} + cases := []struct { + method string + call func(api libvirtAPI) error + }{ + {"DomainLookupByName", func(api libvirtAPI) error { _, err := api.DomainLookupByName(testName); return err }}, + {"DomainDefineXML", func(api libvirtAPI) error { + xml, err := buildDomainXML(testDomainParams()) + if err != nil { + return err + } + _, err = api.DomainDefineXML(xml) + return err + }}, + {"DomainCreate", func(api libvirtAPI) error { return api.DomainCreate(libvirt.Domain{Name: stoppedName}) }}, + {"DomainDestroy", func(api libvirtAPI) error { return api.DomainDestroy(dom) }}, + {"DomainUndefineFlags", func(api libvirtAPI) error { return api.DomainUndefineFlags(dom, undefineFlags) }}, + {"DomainGetState", func(api libvirtAPI) error { _, _, err := api.DomainGetState(dom, 0); return err }}, + {"DomainGetMetadata", func(api libvirtAPI) error { + _, err := api.DomainGetMetadata(dom, int32(libvirt.DomainMetadataElement), libvirt.OptString{metadataURI}, 0) + return err + }}, + {"DomainInterfaceAddresses", func(api libvirtAPI) error { + _, err := api.DomainInterfaceAddresses(dom, uint32(libvirt.DomainInterfaceAddressesSrcArp), 0) + return err + }}, + {"ConnectListAllDomains", func(api libvirtAPI) error { _, _, err := api.ConnectListAllDomains(1, 0); return err }}, + {"StoragePoolLookupByName", func(api libvirtAPI) error { _, err := api.StoragePoolLookupByName("default"); return err }}, + {"StoragePoolRefresh", func(api libvirtAPI) error { return api.StoragePoolRefresh(pool, 0) }}, + {"StorageVolLookupByName", func(api libvirtAPI) error { _, err := api.StorageVolLookupByName(pool, overlay.Name); return err }}, + {"StorageVolGetPath", func(api libvirtAPI) error { _, err := api.StorageVolGetPath(overlay); return err }}, + {"StorageVolCreateXML", func(api libvirtAPI) error { + xml, err := buildVolumeXML(rawVolume("new.iso", 3)) + if err != nil { + return err + } + _, err = api.StorageVolCreateXML(pool, xml, 0) + return err + }}, + {"StorageVolUpload", func(api libvirtAPI) error { + return api.StorageVolUpload(overlay, strings.NewReader("abc"), 0, 3, 0) + }}, + {"StorageVolDownload", func(api libvirtAPI) error { + return api.StorageVolDownload(overlay, io.Discard, 0, 0, 0) + }}, + {"StorageVolDelete", func(api libvirtAPI) error { return api.StorageVolDelete(overlay, 0) }}, + } + if n := reflect.TypeOf((*libvirtAPI)(nil)).Elem().NumMethod(); n != len(cases) { + t.Fatalf("libvirtAPI has %d methods, this test covers %d; trace the new one", n, len(cases)) + } + for _, tc := range cases { + t.Run(tc.method, func(t *testing.T) { + t.Parallel() + log, snapshot := captureLogger(1) + api := &tracingAPI{inner: tracedHost(t), log: log} + if err := tc.call(api); err != nil { + t.Fatalf("%s: %v", tc.method, err) + } + lines := snapshot() + if len(lines) != 1 { + t.Fatalf("logged %d lines, want exactly 1:\n%s", len(lines), strings.Join(lines, "\n")) + } + for _, want := range []string{`"msg"="libvirt call"`, `"method"="` + tc.method + `"`, `"duration"=`} { + if !strings.Contains(lines[0], want) { + t.Errorf("line missing %s:\n%s", want, lines[0]) + } + } + if strings.Contains(lines[0], `"error"=`) { + t.Errorf("successful call logged an error:\n%s", lines[0]) + } + }) + } +} + +func TestTracingAPI_passesResultsThrough(t *testing.T) { + t.Parallel() + inner := tracedHost(t) + log, _ := captureLogger(1) + traced := &tracingAPI{inner: inner, log: log} + dom := libvirt.Domain{Name: testName} + + wantState, wantReason, err := inner.DomainGetState(dom, 0) + if err != nil { + t.Fatal(err) + } + state, reason, err := traced.DomainGetState(dom, 0) + if err != nil || state != wantState || reason != wantReason { + t.Errorf("DomainGetState = %d/%d/%v, want %d/%d/nil", state, reason, err, wantState, wantReason) + } + + overlay := libvirt.StorageVol{Pool: "default", Name: overlayVolumeName(testName)} + wantPath, _ := inner.StorageVolGetPath(overlay) + if path, err := traced.StorageVolGetPath(overlay); err != nil || path != wantPath { + t.Errorf("StorageVolGetPath = %q/%v, want %q", path, err, wantPath) + } + + wantDoms, _, _ := inner.ConnectListAllDomains(1, 0) + doms, _, err := traced.ConnectListAllDomains(1, 0) + if err != nil || !reflect.DeepEqual(doms, wantDoms) { + t.Errorf("ConnectListAllDomains = %v/%v, want %v", doms, err, wantDoms) + } + + inner.fail["DomainCreate"] = lvError(libvirt.ErrOperationInvalid, "already running") + if err := traced.DomainCreate(dom); !hasCode(err, libvirt.ErrOperationInvalid) { + t.Errorf("DomainCreate error = %v, want the injected libvirt error unchanged", err) + } +} + +func TestTracingAPI_errorFields(t *testing.T) { + t.Parallel() + tests := []struct { + name string + inject error + want []string + notWant []string + }{ + { + name: "libvirt error carries code, name and message", + inject: lvError(libvirt.ErrNoDomain, "Domain not found: no domain with matching name"), + want: []string{`"error"=`, `"libvirtCode"=42`, `"libvirtCodeName"="ErrNoDomain"`, `"libvirtMessage"="Domain not found`}, + }, + { + name: "transport error has no libvirt fields", + inject: io.EOF, + want: []string{`"error"="EOF"`}, + notWant: []string{`"libvirtCode"`}, + }, + } + for _, tc := range tests { + t.Run(tc.name, func(t *testing.T) { + t.Parallel() + inner := tracedHost(t) + inner.fail["DomainLookupByName"] = tc.inject + log, snapshot := captureLogger(1) + api := &tracingAPI{inner: inner, log: log} + if _, err := api.DomainLookupByName(testName); !errors.Is(err, tc.inject) { + t.Fatalf("error = %v, want the injected one", err) + } + joined := strings.Join(snapshot(), "\n") + for _, want := range tc.want { + if !strings.Contains(joined, want) { + t.Errorf("output missing %s:\n%s", want, joined) + } + } + for _, absent := range tc.notWant { + if strings.Contains(joined, absent) { + t.Errorf("output unexpectedly contains %s:\n%s", absent, joined) + } + } + }) + } +} + +func TestTracingAPI_detailOnlyAtV2(t *testing.T) { + t.Parallel() + dom := libvirt.Domain{Name: testName} + calls := []struct { + method string + call func(api libvirtAPI) error + always string + detail string + }{ + {"DomainDefineXML", func(api libvirtAPI) error { + xml, err := buildDomainXML(testDomainParams()) + if err != nil { + return err + } + _, err = api.DomainDefineXML(xml) + return err + }, `"xmlBytes"=`, `"xml"="= 2) { + t.Errorf("v%d output contains %s = %v, want %v:\n%s", verbosity, tc.detail, has, verbosity >= 2, joined) + } + }) + } + } +} + +func TestTracingAPI_uploadLogsSizeNotContent(t *testing.T) { + t.Parallel() + const sentinel = "SENTINEL-seed-content-must-never-be-logged" + log, snapshot := captureLogger(2) + api := &tracingAPI{inner: tracedHost(t), log: log} + vol := libvirt.StorageVol{Pool: "default", Name: seedVolumeName(testName)} + if err := api.StorageVolUpload(vol, bytes.NewReader([]byte(sentinel)), 0, uint64(len(sentinel)), 0); err != nil { + t.Fatal(err) + } + joined := strings.Join(snapshot(), "\n") + if !strings.Contains(joined, `"length"=42`) { + t.Errorf("upload line missing the length:\n%s", joined) + } + if strings.Contains(joined, sentinel) { + t.Errorf("upload content leaked into logs:\n%s", joined) + } +} + +func TestTracingAPI_silentBelowV1(t *testing.T) { + t.Parallel() + log, snapshot := captureLogger(0) + api := &tracingAPI{inner: tracedHost(t), log: log} + if _, err := api.DomainLookupByName(testName); err != nil { + t.Fatal(err) + } + if lines := snapshot(); len(lines) != 0 { + t.Errorf("verbosity 0 logged %d lines:\n%s", len(lines), strings.Join(lines, "\n")) + } +} -- 2.54.0 From 5a6c8895317dec7eb696e77b34cc90988a9e051e Mon Sep 17 00:00:00 2001 From: kacerr Date: Thu, 10 Sep 2026 09:05:15 +0000 Subject: [PATCH 13/28] Make the libvirt trace compact: libvirt API names, one error field, terse test output The first live run showed the trace working but noisy: testr printed ts/level/msg and the per-run constants on every line, the error text appeared twice, host appeared twice on image-cache lines, and the Go method name was less useful than the libvirt function it maps to. The tracer now logs rpc=virDomainLookupByName (go-libvirt's names are the public API names without the vir prefix), error plus libvirtError (code name) and libvirtCode, and image.go stops re-adding host. The integration test replaces testr with a compact logr sink: time, logger name, the rpc as headline, then fields, with provider/host/name omitted since the first line of the run states them. Docs and execution log updated. Co-Authored-By: Claude Fable 5.1 --- docs/kvm-provider.md | 9 +- .../2026-09-10-0805-kvm-libvirt-tracing.md | 43 ++++++++++ docs/testing.md | 12 +-- internal/provider/kvm/conn_test.go | 2 +- internal/provider/kvm/errors.go | 11 +-- internal/provider/kvm/errors_test.go | 4 +- internal/provider/kvm/image.go | 6 +- internal/provider/kvm/kvm_integration_test.go | 84 +++++++++++++++++-- internal/provider/kvm/kvm_test.go | 18 ++-- internal/provider/kvm/trace.go | 16 ++-- internal/provider/kvm/trace_test.go | 6 +- 11 files changed, 165 insertions(+), 46 deletions(-) diff --git a/docs/kvm-provider.md b/docs/kvm-provider.md index b7aed5f..d861acb 100644 --- a/docs/kvm-provider.md +++ b/docs/kvm-provider.md @@ -194,10 +194,11 @@ The provider logs through controller-runtime's logger, named `kvm`, with - `--zap-log-level=debug` (logr V(1)): one line per provider operation (`domain defined`, `KVM domain ensured running`, ...) **and one `libvirt call` line per libvirt RPC**, named `kvm.libvirt`, with - `method`, the identifiers it was given (`domain`, `pool`, `volume`), - curated results (`state`, `path`, `interfaces`), `duration`, and on - failure `error` plus `libvirtCode`, `libvirtCodeName` and - `libvirtMessage`. Connection events (`libvirt host connected`, dial + `rpc` (the libvirt API function, e.g. `virDomainLookupByName`), the + identifiers it was given (`domain`, `pool`, `volume`), curated results + (`state`, `path`, `interfaces`), `duration`, and on failure `error` + plus `libvirtError` (the code name, e.g. `ErrNoDomain`) and + `libvirtCode`. Connection events (`libvirt host connected`, dial failures, `libvirt connection dropped` with a reason) log at the same level. A Create is about ten RPCs; a Get is four. - `--zap-log-level=2`: the same lines also carry the domain and volume diff --git a/docs/plans-executions/2026-09-10-0805-kvm-libvirt-tracing.md b/docs/plans-executions/2026-09-10-0805-kvm-libvirt-tracing.md index 3a4ac3c..f6f747c 100644 --- a/docs/plans-executions/2026-09-10-0805-kvm-libvirt-tracing.md +++ b/docs/plans-executions/2026-09-10-0805-kvm-libvirt-tracing.md @@ -55,3 +55,46 @@ Worth noting: the live run (`make test-integration-kvm` against beelink) had not happened when this was written; it is what Step 7 of the KVM provider plan is waiting on, and its output is the acceptance test for this feature. Expected shape is in the plan's Verification section. + +## Revision after the first live run — less noise, libvirt names + +The first live run on beelink showed the trace working but hard to read: +`testr` printed `"ts"`, `"level"`, `"msg"` and the three per-run +constants on every line, the error text appeared twice (`error` and +`libvirtMessage`), `host` appeared twice on image-cache lines (the +logger already carried it), and `method=DomainLookupByName` was +go-libvirt's name rather than something to look up in libvirt's docs. +The user asked for "less verbose but more informative". Changes: + +- The tracer's headline field is now `rpc=virDomainLookupByName`: the + public libvirt API function, which go-libvirt's method names are + exactly minus the `vir` prefix (they come from the same + `remote_protocol.x`). Not gRPC, for the record: libvirt's own + XDR-based RPC over the forwarded unix socket. +- Errors are one `error` field plus `libvirtError=ErrNoDomain` and + `libvirtCode=42`; `libvirtMessage` is gone from both the tracer and + `logAPIError`, since it is the error text. +- `image.go` no longer adds `host` to lines whose logger has it. +- The integration test replaces `testr` with a small `compactSink` + (`logr.LogSink` plus `GetCallStackHelper`, so `t.Log` still + attributes lines to `trace.go`): `HH:MM:SS.mmm libvirt + virDomainCreate domain=… duration=… error=… libvirtError=…`. The + `rpc` value becomes the headline, `provider`/`host`/`name` are + dropped (the run's first `t.Log` line states them), and values with + spaces are quoted. Production zap output is unchanged apart from the + field renames; structured logs are meant to repeat context. + +Previewed by driving the sink through the fake provider with an +injected `DomainCreate` failure (a throwaway integration-tagged test, +not committed): + +``` +trace.go:55: 09:04:33.165 libvirt virDomainLookupByName domain=proxy-abc123def456ghij duration=4µs error="Domain not found: …" libvirtError=ErrNoDomain libvirtCode=42 +trace.go:55: 09:04:33.165 libvirt virStorageVolUpload pool=default volume=proxy-abc123def456ghij-seed.iso offset=0 length=0 duration=71µs +kvm.go:262: 09:04:33.165 kvm seed volume uploaded volume=proxy-abc123def456ghij-seed.iso bytes=45056 +trace.go:55: 09:04:33.165 libvirt virDomainCreate domain=proxy-abc123def456ghij duration=0s error="internal error: process exited…" libvirtError=ErrResourceBusy libvirtCode=87 +``` + +Worth noting: `length=0` on `virStorageVolUpload` is what the provider +passes (libvirt reads to EOF); the following `seed volume uploaded` +line has the byte count. diff --git a/docs/testing.md b/docs/testing.md index 206a63c..c4d79d1 100644 --- a/docs/testing.md +++ b/docs/testing.md @@ -120,11 +120,13 @@ verified — the log says so), `KVM_TEST_IMAGE_SHA256`, `KVM_TEST_POOL` `20m`, must cover the first-time image download), `KVM_TEST_KEEP` to leave the VM behind for `virsh console` when something goes wrong, and `KVM_TEST_LOG_LEVEL` (default `1`) for how much of the provider's -logging the `-v` output shows: at `1` every libvirt RPC is one -`libvirt call` line with method, identifiers, duration and error code, -interleaved with the test's own `t.Log` lines; `2` adds the domain and -volume XML, metadata and addresses; `0` leaves only `t.Log`. See the -Logging section of [kvm-provider.md](kvm-provider.md#logging). +logging the `-v` output shows: at `1` every libvirt RPC is one compact +line (`HH:MM:SS.mmm libvirt virDomainLookupByName domain=... duration=... +error=... libvirtError=ErrNoDomain`), interleaved with the test's own +`t.Log` lines; `2` adds the domain and volume XML, metadata and +addresses; `0` leaves only `t.Log`. The per-run constants (provider, +host, domain name) are left off each line. See the Logging section of +[kvm-provider.md](kvm-provider.md#logging). What one run asserts, in order: `New` dials the host; `Create` returns a `hosts//domains/` ID (downloading the image on a cold host — diff --git a/internal/provider/kvm/conn_test.go b/internal/provider/kvm/conn_test.go index d6105f1..452d66a 100644 --- a/internal/provider/kvm/conn_test.go +++ b/internal/provider/kvm/conn_test.go @@ -586,7 +586,7 @@ func TestHostPool_call_tracesAtV1(t *testing.T) { t.Fatalf("call() error = %v", err) } joined := strings.Join(snapshot(), "\n") - for _, want := range []string{`"msg"="libvirt host connected"`, `"address"="nas1:22"`, `"method"="StoragePoolLookupByName"`, `"pool"="default"`, "libvirt "} { + for _, want := range []string{`"msg"="libvirt host connected"`, `"address"="nas1:22"`, `"rpc"="virStoragePoolLookupByName"`, `"pool"="default"`, "libvirt "} { if !strings.Contains(joined, want) { t.Errorf("output missing %s:\n%s", want, joined) } diff --git a/internal/provider/kvm/errors.go b/internal/provider/kvm/errors.go index 6896c01..158449f 100644 --- a/internal/provider/kvm/errors.go +++ b/internal/provider/kvm/errors.go @@ -68,16 +68,13 @@ func logAPIError(log logr.Logger, op string, err error) { } // libvirtErrorFields renders a daemon-reported error as log key/values: -// the numeric code, its constant name (ErrNoDomain, ...) and the daemon's -// message. Nil for transport and other non-libvirt errors. +// its constant name (ErrNoDomain, ...) and numeric code. The daemon's +// message is already the error text. Nil for transport and other +// non-libvirt errors. func libvirtErrorFields(err error) []any { var lerr libvirt.Error if !errors.As(err, &lerr) { return nil } - return []any{ - "libvirtCode", lerr.Code, - "libvirtCodeName", libvirt.ErrorNumber(lerr.Code).String(), - "libvirtMessage", lerr.Message, - } + return []any{"libvirtError", libvirt.ErrorNumber(lerr.Code).String(), "libvirtCode", lerr.Code} } diff --git a/internal/provider/kvm/errors_test.go b/internal/provider/kvm/errors_test.go index 6ce0065..b8ccd65 100644 --- a/internal/provider/kvm/errors_test.go +++ b/internal/provider/kvm/errors_test.go @@ -76,8 +76,8 @@ func TestLibvirtErrorFields(t *testing.T) { err error want []any }{ - {"libvirt error", lvErr(libvirt.ErrNoDomain), []any{"libvirtCode", uint32(42), "libvirtCodeName", "ErrNoDomain", "libvirtMessage", "ErrNoDomain"}}, - {"wrapped libvirt error", fmt.Errorf("looking up: %w", lvErr(libvirt.ErrNoDomain)), []any{"libvirtCode", uint32(42), "libvirtCodeName", "ErrNoDomain", "libvirtMessage", "ErrNoDomain"}}, + {"libvirt error", lvErr(libvirt.ErrNoDomain), []any{"libvirtError", "ErrNoDomain", "libvirtCode", uint32(42)}}, + {"wrapped libvirt error", fmt.Errorf("looking up: %w", lvErr(libvirt.ErrNoDomain)), []any{"libvirtError", "ErrNoDomain", "libvirtCode", uint32(42)}}, {"transport error", io.EOF, nil}, {"cancelled", context.Canceled, nil}, } diff --git a/internal/provider/kvm/image.go b/internal/provider/kvm/image.go index ffdc0e1..a13645c 100644 --- a/internal/provider/kvm/image.go +++ b/internal/provider/kvm/image.go @@ -101,7 +101,7 @@ func (c *imageCache) ensure(ctx context.Context, log logr.Logger, api libvirtAPI return "", err } if ready { - log.V(1).Info("base image already cached", "host", hostName, "volume", src.volumeName()) + log.V(1).Info("base image already cached", "volume", src.volumeName()) return api.StorageVolGetPath(base) } if err := deleteIfExists(api, pool, src.markerName()); err != nil { @@ -170,7 +170,7 @@ func (c *imageCache) fetch(ctx context.Context, log logr.Logger, api libvirtAPI, } log.Info("fetching base image into host pool", - "host", hostName, "pool", pool.Name, "volume", src.volumeName(), + "pool", pool.Name, "volume", src.volumeName(), "url", src.URL, "bytes", resp.ContentLength) start := time.Now() @@ -212,7 +212,7 @@ func (c *imageCache) fetch(ctx context.Context, log logr.Logger, api libvirtAPI, return libvirt.StorageVol{}, fmt.Errorf("writing image marker: %w", err) } log.Info("base image cached", - "host", hostName, "volume", src.volumeName(), "sha256", digest, + "volume", src.volumeName(), "sha256", digest, "bytes", resp.ContentLength, "duration", time.Since(start).Round(time.Second)) return base, nil } diff --git a/internal/provider/kvm/kvm_integration_test.go b/internal/provider/kvm/kvm_integration_test.go index 32bf5a7..6e9e157 100644 --- a/internal/provider/kvm/kvm_integration_test.go +++ b/internal/provider/kvm/kvm_integration_test.go @@ -29,10 +29,13 @@ package kvm // one line per libvirt call; 2 adds XML, metadata and // addresses; 0 leaves only the t.Log lines) // -// Provider logs go through a testr logger, so they interleave with t.Log -// output. A run that hits its timeout may log one more line from the RPC -// the pool abandoned, after the test has ended; that is a symptom of the -// timeout, not a bug in the tracer. +// Provider logs are printed through t.Log in a compact form: time, logger +// name, then the libvirt API function (or the message) and its fields. +// The provider/host/name values that are the same on every line of a run +// are dropped; the first t.Log line of the run states them. A run that +// hits its timeout may log one more line from the RPC the pool abandoned, +// after the test has ended; that is a symptom of the timeout, not a bug +// in the tracer. import ( "context" @@ -43,12 +46,12 @@ import ( "net" "os" "strconv" + "strings" "testing" "time" libvirt "github.com/digitalocean/go-libvirt" "github.com/go-logr/logr" - "github.com/go-logr/logr/testr" "k8s.io/apimachinery/pkg/types" logf "sigs.k8s.io/controller-runtime/pkg/log" @@ -127,11 +130,80 @@ func loadIntegrationEnv(t *testing.T) integrationEnv { // print its "SetLogger was never called" warning and swallow the lines). func integrationLogger(t *testing.T, verbosity int) logr.Logger { t.Helper() - log := testr.NewWithOptions(t, testr.Options{LogTimestamp: true, Verbosity: verbosity}) + log := logr.New(&compactSink{t: t, verbosity: verbosity}) logf.SetLogger(log) return log } +// compactSink is a logr sink for test output: one line per record as +// "HH:MM:SS.mmm name headline k=v ...", where the headline is the rpc +// field when present (the libvirt API function) and the message +// otherwise. Keys that are constant for the whole run are omitted. +type compactSink struct { + t *testing.T + verbosity int + name string + values []any +} + +var compactOmit = map[string]bool{"provider": true, "host": true, "name": true} + +func (s *compactSink) Init(logr.RuntimeInfo) {} + +func (s *compactSink) Enabled(level int) bool { return level <= s.verbosity } + +// GetCallStackHelper lets logr mark its frames as helpers, so t.Log +// attributes each line to the provider code that logged it. +func (s *compactSink) GetCallStackHelper() func() { return s.t.Helper } + +func (s *compactSink) Info(_ int, msg string, kv ...any) { + s.t.Helper() + s.t.Log(s.format(msg, kv)) +} + +func (s *compactSink) Error(err error, msg string, kv ...any) { + s.t.Helper() + s.t.Log(s.format(msg, append(kv, "error", err))) +} + +func (s *compactSink) WithValues(kv ...any) logr.LogSink { + c := *s + c.values = append(append([]any(nil), s.values...), kv...) + return &c +} + +func (s *compactSink) WithName(name string) logr.LogSink { + c := *s + if c.name != "" { + name = c.name + "/" + name + } + c.name = name + return &c +} + +func (s *compactSink) format(msg string, kv []any) string { + all := append(append([]any(nil), s.values...), kv...) + headline := msg + var b strings.Builder + for i := 0; i+1 < len(all); i += 2 { + key := fmt.Sprint(all[i]) + if key == "rpc" { + headline = fmt.Sprint(all[i+1]) + continue + } + if compactOmit[key] { + continue + } + val := fmt.Sprint(all[i+1]) + if strings.ContainsAny(val, " \t\"=") { + val = strconv.Quote(val) + } + b.WriteString(" " + key + "=" + val) + } + name := s.name[strings.LastIndex(s.name, "/")+1:] + return fmt.Sprintf("%s %s %s%s", time.Now().Format("15:04:05.000"), name, headline, b.String()) +} + func randomUID(t *testing.T) string { t.Helper() var b [16]byte diff --git a/internal/provider/kvm/kvm_test.go b/internal/provider/kvm/kvm_test.go index 60d50da..c1f3603 100644 --- a/internal/provider/kvm/kvm_test.go +++ b/internal/provider/kvm/kvm_test.go @@ -925,15 +925,15 @@ func TestLogging_verbosityTiers(t *testing.T) { wantLines: []string{ `kvm/libvirt "level"=1 "msg"="libvirt host connected"`, `"msg"="libvirt call"`, - `"method"="DomainLookupByName"`, - `"libvirtCodeName"="ErrNoDomain"`, - `"method"="StorageVolUpload"`, - `"method"="DomainDefineXML"`, - `"method"="DomainCreate"`, - `"method"="DomainInterfaceAddresses"`, - `"method"="ConnectListAllDomains"`, - `"method"="DomainUndefineFlags"`, - `"method"="StorageVolDelete"`, + `"rpc"="virDomainLookupByName"`, + `"libvirtError"="ErrNoDomain"`, + `"rpc"="virStorageVolUpload"`, + `"rpc"="virDomainDefineXML"`, + `"rpc"="virDomainCreate"`, + `"rpc"="virDomainInterfaceAddresses"`, + `"rpc"="virConnectListAllDomains"`, + `"rpc"="virDomainUndefineFlags"`, + `"rpc"="virStorageVolDelete"`, `"provider"="kvm-home"`, `"host"="nas1"`, `"name"="` + testName + `"`, diff --git a/internal/provider/kvm/trace.go b/internal/provider/kvm/trace.go index 152f65b..c3d8e21 100644 --- a/internal/provider/kvm/trace.go +++ b/internal/provider/kvm/trace.go @@ -10,10 +10,11 @@ import ( ) // tracingAPI wraps a libvirtAPI and logs one V(1) line per call after it -// returns: method, the identifiers it was given, curated results, duration, -// and the libvirt error fields on failure. At V(2) the same line also -// carries XML bodies, metadata and address lists. Upload and download -// bodies are never logged: the seed ISO carries cloud-init. +// returns: the libvirt API function (rpc=virDomainLookupByName, ...), the +// identifiers it was given, curated results, duration, and on failure the +// error with libvirt's code name. At V(2) the same line also carries XML +// bodies, metadata and address lists. Upload and download bodies are never +// logged: the seed ISO carries cloud-init. type tracingAPI struct { inner libvirtAPI log logr.Logger @@ -26,8 +27,11 @@ type tracedCall struct { kv []any } +// begin names the call by its libvirt API function: go-libvirt's method +// names are the remote procedure names, which are the public vir* API +// names without the prefix. func (t *tracingAPI) begin(method string, kv ...any) *tracedCall { - return &tracedCall{t: t, method: method, start: time.Now(), kv: kv} + return &tracedCall{t: t, method: "vir" + method, start: time.Now(), kv: kv} } // detail adds key/values that are only worth the bytes at V(2). @@ -40,7 +44,7 @@ func (c *tracedCall) detail(kv ...any) *tracedCall { func (c *tracedCall) done(err error, kv ...any) { fields := make([]any, 0, 2+len(c.kv)+len(kv)+10) - fields = append(fields, "method", c.method) + fields = append(fields, "rpc", c.method) fields = append(fields, c.kv...) fields = append(fields, kv...) fields = append(fields, "duration", time.Since(c.start).Round(time.Microsecond)) diff --git a/internal/provider/kvm/trace_test.go b/internal/provider/kvm/trace_test.go index 98e60a8..0a11a30 100644 --- a/internal/provider/kvm/trace_test.go +++ b/internal/provider/kvm/trace_test.go @@ -119,7 +119,7 @@ func TestTracingAPI_logsEveryMethod(t *testing.T) { if len(lines) != 1 { t.Fatalf("logged %d lines, want exactly 1:\n%s", len(lines), strings.Join(lines, "\n")) } - for _, want := range []string{`"msg"="libvirt call"`, `"method"="` + tc.method + `"`, `"duration"=`} { + for _, want := range []string{`"msg"="libvirt call"`, `"rpc"="vir` + tc.method + `"`, `"duration"=`} { if !strings.Contains(lines[0], want) { t.Errorf("line missing %s:\n%s", want, lines[0]) } @@ -176,13 +176,13 @@ func TestTracingAPI_errorFields(t *testing.T) { { name: "libvirt error carries code, name and message", inject: lvError(libvirt.ErrNoDomain, "Domain not found: no domain with matching name"), - want: []string{`"error"=`, `"libvirtCode"=42`, `"libvirtCodeName"="ErrNoDomain"`, `"libvirtMessage"="Domain not found`}, + want: []string{`"error"="Domain not found`, `"libvirtError"="ErrNoDomain"`, `"libvirtCode"=42`}, }, { name: "transport error has no libvirt fields", inject: io.EOF, want: []string{`"error"="EOF"`}, - notWant: []string{`"libvirtCode"`}, + notWant: []string{`"libvirtError"`, `"libvirtCode"`}, }, } for _, tc := range tests { -- 2.54.0 From 9a393b1df8f908830becc6c69ee32d3588f9875b Mon Sep 17 00:00:00 2001 From: kacerr Date: Thu, 10 Sep 2026 09:42:33 +0000 Subject: [PATCH 14/28] Boot KVM guests via UEFI by default; per-image firmware setting Run 5 on the real host bisected the guest hang: the Debian 13 image hangs after GRUB under SeaBIOS with or without our seed and with either CPU model, and boots under OVMF. The image's legacy boot chain is the problem, not the provider. images..firmware is "uefi" (default) or "bios". The domain XML emits with secure boot off, so libvirt selects the OVMF binary and manages the NVRAM file, which the existing nvram undefine flag removes. Direct-URL images default to UEFI too. OVMF joins the hypervisor prerequisites; sample, config table, known gaps, decision log and the execution log record the finding. Co-Authored-By: Claude Fable 5.1 --- config/samples/providers-config.yaml | 2 + docs/architecture.md | 7 +++ docs/kvm-provider.md | 13 ++++- .../2026-09-04-1404-kvm-provider.md | 37 +++++++++++++++ internal/provider/config_kvm.go | 8 ++++ internal/provider/config_kvm_test.go | 5 ++ internal/provider/kvm/domainxml.go | 47 ++++++++++++++----- internal/provider/kvm/domainxml_test.go | 22 +++++++++ internal/provider/kvm/image.go | 13 ++--- internal/provider/kvm/image_test.go | 6 +-- internal/provider/kvm/kvm.go | 12 +++-- internal/provider/kvm/kvm_test.go | 5 +- 12 files changed, 151 insertions(+), 26 deletions(-) diff --git a/config/samples/providers-config.yaml b/config/samples/providers-config.yaml index af9e575..871b9d0 100644 --- a/config/samples/providers-config.yaml +++ b/config/samples/providers-config.yaml @@ -40,6 +40,8 @@ providers: debian-13: url: https://cloud.debian.org/images/cloud/trixie/latest/debian-13-generic-amd64.qcow2 # sha256: "<64 hex>" # optional; verified after download + # firmware: uefi # default; "bios" only for images with no EFI partition + # # (Debian 13 does not boot via BIOS, see docs/kvm-provider.md) ubuntu-24.04: url: https://cloud-images.ubuntu.com/noble/current/noble-server-cloudimg-amd64.img machineTypes: # placement.machineType names a key here diff --git a/docs/architecture.md b/docs/architecture.md index 328c069..969017a 100644 --- a/docs/architecture.md +++ b/docs/architecture.md @@ -454,6 +454,13 @@ they read as choices, not accidents. Chronological by build step. unknown statuses, `nostate` means libvirt could not determine the state at all; answering that with delete-and-recreate is too drastic, so the next poll decides. +- **KVM guests boot UEFI by default, secure boot off.** Found on the + first real host: Debian 13's generic image hangs after GRUB under + SeaBIOS regardless of CPU model or seed, and boots under OVMF. Every + current distro cloud image supports UEFI, so it is the default and + `firmware: bios` is the per-image escape; libvirt's firmware + autoselection (``) keeps the OVMF path out of the + operator's config. Secure boot is off so unsigned images boot too. - **A KVM domain GC cannot read is skipped, a host it cannot reach fails the sweep.** GC never deletes what it does not see, so leaving one domain out is safe; one corrupt domain blocking GC for every other one diff --git a/docs/kvm-provider.md b/docs/kvm-provider.md index d861acb..4c45b08 100644 --- a/docs/kvm-provider.md +++ b/docs/kvm-provider.md @@ -23,7 +23,11 @@ Per host, once: 4. **A storage pool** (default `default`, usually `/var/lib/libvirt/images`) with room for one base image per distro (a few hundred MB) plus the per-VM overlays (`diskSizeGb` each, thin-provisioned). -5. Optionally, **outbound internet from the VMs**: the default +5. **UEFI firmware for guests**: the `ovmf` package on Debian/Ubuntu, + `edk2-ovmf` on Fedora/RHEL. VMs boot UEFI by default (libvirt picks + the OVMF binary itself); a domain start fails with "no firmware + found" without it. Only `firmware: bios` images skip this. +6. Optionally, **outbound internet from the VMs**: the default vendor-data installs `qemu-guest-agent` from the distro mirrors on first boot. Without it the VM still works — the operator falls back to the host's ARP table to find the IP, which fills in once the guest @@ -107,6 +111,7 @@ ssh virt@nas1.home.lan ip link show br0 | `images..url` | yes per entry | | http(s) URL of a cloud image (qcow2 or raw) | | `images..sha256` | | | expected digest; verified after download, a change triggers a re-download | | `images..format` | | `qcow2` | `qcow2` or `raw` | +| `images..firmware` | | `uefi` | `uefi` (OVMF, secure boot off) or `bios` (SeaBIOS) for images without an EFI system partition; Debian 13's BIOS path does not boot, see *Known gaps* | | `machineTypes.` | ≥ 1 | | `{vcpus, memoryMiB}` size profile | | `hosts[].name` | yes | | the `placement.zone` value that selects this host; unique | | `hosts[].address` | yes | | SSH endpoint, `host:port` | @@ -228,6 +233,12 @@ The provider logs through controller-runtime's logger, named `kvm`, with - **IP discovery needs either guest traffic or the agent.** A VM on a bridge with no DHCP never gets an address and stays `Provisioning` forever; check `virsh console`. +- **Debian 13 cloud images do not boot via BIOS on the tested host** + (Ryzen 8845HS, libvirt 11.6, QEMU 10.1): GRUB prints "Booting" and + the kernel never reaches the console, with or without a seed, with + host-passthrough and host-model CPUs alike. The same image boots + under UEFI, which is why `firmware` defaults to `uefi`. Images that + need `firmware: bios` are the exception, not the rule. ## Testing diff --git a/docs/plans-executions/2026-09-04-1404-kvm-provider.md b/docs/plans-executions/2026-09-04-1404-kvm-provider.md index ff9d7b1..00991aa 100644 --- a/docs/plans-executions/2026-09-04-1404-kvm-provider.md +++ b/docs/plans-executions/2026-09-04-1404-kvm-provider.md @@ -477,3 +477,40 @@ outside the operator). Because the provider's own logs were invisible in the test output, this run also produced a side plan: [2026-09-10-0805-kvm-libvirt-tracing](2026-09-10-0805-kvm-libvirt-tracing.md), which traces every libvirt RPC at V(1) and routes it into `-v` output. + +### Run 5: the guest hang is Debian 13's BIOS boot path; UEFI fixes it + +The cached image's sha512 matched Debian's published sum, and +`qemu-img info` showed a sound 3 GiB qcow2, which also confirms that +the raw upload over a libvirt-created volume plus pool refresh yields a +byte-identical file (the Step 3b note is answered). Bisected by hand on +beelink with `virt-install --import` on a fresh overlay of the same +image: + +| variant | seed | CPU | firmware | result | +|---|---|---|---|---| +| operator's domain | ours | host-passthrough | SeaBIOS | hangs after GRUB "Booting", 100 % CPU | +| bare | none | host-passthrough | SeaBIOS | same hang | +| bare | none | host-model | SeaBIOS | same hang | +| bare | none | host-passthrough | **OVMF (`--boot uefi`)** | **boots to login** | + +So neither the seed ISO, the domain XML nor the CPU model was the +problem: the image's legacy GRUB → kernel handoff does not work on this +host, while its EFI stub path does. Not diagnosed further; it is +Debian's chain, not ours, and every current cloud image boots UEFI. + +Change: `images..firmware` (`uefi` default, `bios` escape), +threaded through `withDefaults` → `imageSource` → `domainParams`, and +`buildDomainXML` emits `` so libvirt selects the +OVMF binary and creates the NVRAM file; the `nvram` undefine flag Delete +already passes removes it (seen on the host: `virsh undefine` without +`--nvram` refuses such a domain). Direct-URL images default to UEFI +too. Docs: OVMF is a new hypervisor prerequisite, config table row, +sample comment, known-gaps entry, decision-log entry. + +Worth noting: the first image download went through with no checksum +configured, and the cache reuses a base image by key, so switching +`KVM_TEST_IMAGE_URL` requires deleting `base-test-image.qcow2` and its +marker by hand first. That is the reviewer's Step 4 finding showing up +in practice; still documented rather than fixed. diff --git a/internal/provider/config_kvm.go b/internal/provider/config_kvm.go index 606d643..61521c7 100644 --- a/internal/provider/config_kvm.go +++ b/internal/provider/config_kvm.go @@ -47,6 +47,11 @@ type KVMImage struct { SHA256 string `json:"sha256,omitempty"` // Format is the image's disk format, "qcow2" (default) or "raw". Format string `json:"format,omitempty"` + // Firmware is how VMs from this image boot: "uefi" (default; OVMF with + // secure boot off) or "bios" (SeaBIOS) for images with no EFI system + // partition. Current Debian, Ubuntu, Fedora and Rocky cloud images all + // boot UEFI, and Debian 13's BIOS path was found not to boot at all. + Firmware string `json:"firmware,omitempty"` } // KVMMachineType is a VM size profile. @@ -102,6 +107,9 @@ func validateKVM(prefix string, c *KVMConfig) error { if img.Format != "" && img.Format != "qcow2" && img.Format != "raw" { return fmt.Errorf("%s.format must be qcow2 or raw (got %q)", p, img.Format) } + if img.Firmware != "" && img.Firmware != "uefi" && img.Firmware != "bios" { + return fmt.Errorf("%s.firmware must be uefi or bios (got %q)", p, img.Firmware) + } } if len(c.MachineTypes) == 0 { return fmt.Errorf("%s: kvm.machineTypes must define at least one profile", prefix) diff --git a/internal/provider/config_kvm_test.go b/internal/provider/config_kvm_test.go index 6a00afb..7438e38 100644 --- a/internal/provider/config_kvm_test.go +++ b/internal/provider/config_kvm_test.go @@ -154,6 +154,11 @@ hosts: yaml: minimal("sshKeyFile: /key\nknownHostsFile: /kh\nimages:\n d: {url: https://x/y.qcow2, sha256: abc}\n" + hostsAndTypes), wantErrSub: `images["d"].sha256 must be 64 hex characters`, }, + { + name: "image with unknown firmware", + yaml: minimal("sshKeyFile: /key\nknownHostsFile: /kh\nimages:\n d: {url: https://x/y.qcow2, firmware: coreboot}\n" + hostsAndTypes), + wantErrSub: `images["d"].firmware must be uefi or bios`, + }, { name: "image with unknown format", yaml: minimal("sshKeyFile: /key\nknownHostsFile: /kh\nimages:\n d: {url: https://x/y.vmdk, format: vmdk}\n" + hostsAndTypes), diff --git a/internal/provider/kvm/domainxml.go b/internal/provider/kvm/domainxml.go index bfdcfe4..4951fac 100644 --- a/internal/provider/kvm/domainxml.go +++ b/internal/provider/kvm/domainxml.go @@ -10,13 +10,15 @@ const guestAgentChannel = "org.qemu.guest_agent.0" // domainParams is everything buildDomainXML needs, already resolved from the // config and placement so the builder stays a pure function. DiskPath and -// SeedPath are the volumes' host paths as reported by libvirt. +// SeedPath are the volumes' host paths as reported by libvirt; Firmware is +// firmwareUEFI or firmwareBIOS. type domainParams struct { Name string UID string CreatedAt time.Time VCPUs int32 MemoryMiB int64 + Firmware string Bridge string DiskPath string SeedPath string @@ -32,17 +34,27 @@ type domainParams struct { // to sources that carry a path, so a volume-referenced disk fails to start // with "Permission denied" on any AppArmor-confined host (stock Debian and // Ubuntu). The paths cost one StorageVolGetPath round-trip each per Create. +// +// UEFI is expressed as with secure boot off, and libvirt +// picks the OVMF binary and creates the NVRAM file itself (removed again by +// the nvram undefine flag). Debian 13's BIOS boot path was found to hang +// after GRUB on a real host, while the same image boots under UEFI. func buildDomainXML(p domainParams) (string, error) { + os := osXML{ + Type: osTypeXML{Arch: "x86_64", Machine: "q35", Value: "hvm"}, + Boot: bootXML{Dev: "hd"}, + } + if p.Firmware == firmwareUEFI { + os.Firmware = "efi" + os.FirmwareFeatures = &firmwareXML{Features: []firmwareFeatureXML{{Enabled: "no", Name: "secure-boot"}}} + } d := domainXML{ - Type: "kvm", - Name: p.Name, - Metadata: domainMetadataXML{Instance: newInstanceMetadata(p.UID, p.CreatedAt)}, - Memory: sizeXML{Unit: "MiB", Value: p.MemoryMiB}, - VCPU: p.VCPUs, - OS: osXML{ - Type: osTypeXML{Arch: "x86_64", Machine: "q35", Value: "hvm"}, - Boot: bootXML{Dev: "hd"}, - }, + Type: "kvm", + Name: p.Name, + Metadata: domainMetadataXML{Instance: newInstanceMetadata(p.UID, p.CreatedAt)}, + Memory: sizeXML{Unit: "MiB", Value: p.MemoryMiB}, + VCPU: p.VCPUs, + OS: os, Features: featuresXML{ACPI: &struct{}{}, APIC: &struct{}{}}, CPU: cpuXML{Mode: "host-passthrough"}, Clock: clockXML{Offset: "utc"}, @@ -115,8 +127,19 @@ type sizeXML struct { } type osXML struct { - Type osTypeXML `xml:"type"` - Boot bootXML `xml:"boot"` + Firmware string `xml:"firmware,attr,omitempty"` + Type osTypeXML `xml:"type"` + FirmwareFeatures *firmwareXML `xml:"firmware"` + Boot bootXML `xml:"boot"` +} + +type firmwareXML struct { + Features []firmwareFeatureXML `xml:"feature"` +} + +type firmwareFeatureXML struct { + Enabled string `xml:"enabled,attr"` + Name string `xml:"name,attr"` } type osTypeXML struct { diff --git a/internal/provider/kvm/domainxml_test.go b/internal/provider/kvm/domainxml_test.go index b401670..8ba3803 100644 --- a/internal/provider/kvm/domainxml_test.go +++ b/internal/provider/kvm/domainxml_test.go @@ -14,6 +14,7 @@ func testDomainParams() domainParams { CreatedAt: time.Date(2026, 9, 4, 12, 0, 0, 0, time.UTC), VCPUs: 2, MemoryMiB: 2048, + Firmware: firmwareUEFI, Bridge: "br0", DiskPath: "/var/lib/libvirt/images/proxy-abc123def456ghij.qcow2", SeedPath: "/var/lib/libvirt/images/proxy-abc123def456ghij-seed.iso", @@ -40,6 +41,10 @@ func TestBuildDomainXML_fieldByField(t *testing.T) { if d.OS.Type.Value != "hvm" || d.OS.Type.Arch != "x86_64" || d.OS.Type.Machine != "q35" || d.OS.Boot.Dev != "hd" { t.Errorf("os = %+v", d.OS) } + if d.OS.Firmware != "efi" || d.OS.FirmwareFeatures == nil || len(d.OS.FirmwareFeatures.Features) != 1 || + d.OS.FirmwareFeatures.Features[0] != (firmwareFeatureXML{Enabled: "no", Name: "secure-boot"}) { + t.Errorf("os firmware = %q %+v, want efi with secure-boot disabled", d.OS.Firmware, d.OS.FirmwareFeatures) + } if d.Features.ACPI == nil || d.Features.APIC == nil { t.Errorf("features = %+v, want acpi and apic", d.Features) } @@ -113,3 +118,20 @@ func TestBuildDomainXML_escapesUserValues(t *testing.T) { t.Errorf("bridge = %q, want the raw value back", d.Devices.Interfaces[0].Source.Bridge) } } + +func TestBuildDomainXML_biosFirmware(t *testing.T) { + t.Parallel() + p := testDomainParams() + p.Firmware = firmwareBIOS + raw, err := buildDomainXML(p) + if err != nil { + t.Fatalf("buildDomainXML() error = %v", err) + } + var d domainXML + if err := xml.Unmarshal([]byte(raw), &d); err != nil { + t.Fatalf("output is not well-formed XML: %v", err) + } + if d.OS.Firmware != "" || d.OS.FirmwareFeatures != nil || strings.Contains(raw, "firmware") { + t.Errorf("bios domain still carries firmware settings:\n%s", raw) + } +} diff --git a/internal/provider/kvm/image.go b/internal/provider/kvm/image.go index a13645c..5fcbda6 100644 --- a/internal/provider/kvm/image.go +++ b/internal/provider/kvm/image.go @@ -22,10 +22,11 @@ import ( // imageSource is a resolved placement.image: either an entry from the // provider's images map (Key set) or a URL given directly. type imageSource struct { - Key string - URL string - SHA256 string - Format string + Key string + URL string + SHA256 string + Format string + Firmware string } // resolveImage maps placement.image to a source. Anything that is neither a @@ -33,11 +34,11 @@ type imageSource struct { // surface, not retry. func resolveImage(cfg provider.KVMConfig, image string) (imageSource, error) { if img, ok := cfg.Images[image]; ok { - return imageSource{Key: image, URL: img.URL, SHA256: img.SHA256, Format: img.Format}, nil + return imageSource{Key: image, URL: img.URL, SHA256: img.SHA256, Format: img.Format, Firmware: img.Firmware}, nil } u, err := url.Parse(image) if err == nil && (u.Scheme == "http" || u.Scheme == "https") && u.Host != "" { - return imageSource{URL: image, Format: formatQcow2}, nil + return imageSource{URL: image, Format: formatQcow2, Firmware: firmwareUEFI}, nil } return imageSource{}, fmt.Errorf("%w: placement.image %q is neither a key in kvm.images nor an http(s) URL", provider.ErrPermanent, image) diff --git a/internal/provider/kvm/image_test.go b/internal/provider/kvm/image_test.go index 2493c44..93ffdc6 100644 --- a/internal/provider/kvm/image_test.go +++ b/internal/provider/kvm/image_test.go @@ -88,9 +88,9 @@ func TestResolveImage(t *testing.T) { want imageSource wantErr bool }{ - {"configured key", "debian-13", imageSource{Key: "debian-13", URL: "https://cloud.debian.org/d.qcow2", SHA256: "abc", Format: "qcow2"}, false}, - {"direct https url", "https://x.example/img.qcow2", imageSource{URL: "https://x.example/img.qcow2", Format: "qcow2"}, false}, - {"direct http url", "http://mirror/img.qcow2", imageSource{URL: "http://mirror/img.qcow2", Format: "qcow2"}, false}, + {"configured key", "debian-13", imageSource{Key: "debian-13", URL: "https://cloud.debian.org/d.qcow2", SHA256: "abc", Format: "qcow2", Firmware: "uefi"}, false}, + {"direct https url", "https://x.example/img.qcow2", imageSource{URL: "https://x.example/img.qcow2", Format: "qcow2", Firmware: "uefi"}, false}, + {"direct http url", "http://mirror/img.qcow2", imageSource{URL: "http://mirror/img.qcow2", Format: "qcow2", Firmware: "uefi"}, false}, {"unknown key", "ubuntu", imageSource{}, true}, {"file url", "file:///tmp/x.qcow2", imageSource{}, true}, {"empty", "", imageSource{}, true}, diff --git a/internal/provider/kvm/kvm.go b/internal/provider/kvm/kvm.go index ed4613e..a42e5eb 100644 --- a/internal/provider/kvm/kvm.go +++ b/internal/provider/kvm/kvm.go @@ -28,6 +28,8 @@ const ( defaultPool = "default" formatQcow2 = "qcow2" formatRaw = "raw" + firmwareUEFI = "uefi" + firmwareBIOS = "bios" // undefineFlags removes everything libvirt may keep alongside a domain // definition, so an undefine never fails on leftover state. @@ -48,6 +50,9 @@ func withDefaults(cfg provider.KVMConfig) provider.KVMConfig { if img.Format == "" { img.Format = formatQcow2 } + if img.Firmware == "" { + img.Firmware = firmwareUEFI + } images[k] = img } cfg.Images = images @@ -199,7 +204,7 @@ func (p *Provider) Create(ctx context.Context, req provider.CreateRequest) (stri if err != nil { return fmt.Errorf("resolving seed path: %w", err) } - dom, err = p.defineDomain(log, api, host, req, mt, diskPath, seedPath) + dom, err = p.defineDomain(log, api, host, req, mt, src.Firmware, diskPath, seedPath) if err != nil { return err } @@ -263,13 +268,14 @@ func (p *Provider) writeSeed(log logr.Logger, api libvirtAPI, pool libvirt.Stora return vol, nil } -func (p *Provider) defineDomain(log logr.Logger, api libvirtAPI, host provider.KVMHost, req provider.CreateRequest, mt provider.KVMMachineType, diskPath, seedPath string) (libvirt.Domain, error) { +func (p *Provider) defineDomain(log logr.Logger, api libvirtAPI, host provider.KVMHost, req provider.CreateRequest, mt provider.KVMMachineType, firmware, diskPath, seedPath string) (libvirt.Domain, error) { xml, err := buildDomainXML(domainParams{ Name: req.Name, UID: req.UID, CreatedAt: p.now(), VCPUs: mt.VCPUs, MemoryMiB: mt.MemoryMiB, + Firmware: firmware, Bridge: host.Bridge, DiskPath: diskPath, SeedPath: seedPath, @@ -281,7 +287,7 @@ func (p *Provider) defineDomain(log logr.Logger, api libvirtAPI, host provider.K if err != nil { return libvirt.Domain{}, fmt.Errorf("defining domain: %w", err) } - log.V(1).Info("domain defined", "uid", req.UID) + log.V(1).Info("domain defined", "uid", req.UID, "firmware", firmware) return dom, nil } diff --git a/internal/provider/kvm/kvm_test.go b/internal/provider/kvm/kvm_test.go index c1f3603..17c7190 100644 --- a/internal/provider/kvm/kvm_test.go +++ b/internal/provider/kvm/kvm_test.go @@ -20,7 +20,7 @@ func TestWithDefaults(t *testing.T) { cfg := withDefaults(provider.KVMConfig{ Images: map[string]provider.KVMImage{ "debian": {URL: "https://x/d.qcow2"}, - "raw": {URL: "https://x/r.img", Format: "raw"}, + "raw": {URL: "https://x/r.img", Format: "raw", Firmware: "bios"}, }, Hosts: []provider.KVMHost{ {Name: "a", Address: "a:22", User: "virt", Bridge: "br0"}, @@ -36,6 +36,9 @@ func TestWithDefaults(t *testing.T) { if cfg.Images["debian"].Format != "qcow2" || cfg.Images["raw"].Format != "raw" { t.Errorf("image formats = %q/%q, want qcow2 default and raw kept", cfg.Images["debian"].Format, cfg.Images["raw"].Format) } + if cfg.Images["debian"].Firmware != firmwareUEFI || cfg.Images["raw"].Firmware != firmwareBIOS { + t.Errorf("image firmware = %q/%q, want uefi default and bios kept", cfg.Images["debian"].Firmware, cfg.Images["raw"].Firmware) + } if a := cfg.Hosts[0]; a.Socket != defaultSocket || a.Pool != defaultPool { t.Errorf("host a = %+v, want default socket and pool", a) } -- 2.54.0 From c4f59bd1535acb9a5824d8389bc5a263386b4ace Mon Sep 17 00:00:00 2001 From: kacerr Date: Thu, 10 Sep 2026 09:58:01 +0000 Subject: [PATCH 15/28] Name the upstream cause of the Debian 13 BIOS hang: Debian bug #1111240 The known-gaps entry, decision log, sample comment and builder comment blamed the host; the cause is trixie's GRUB config ("terminal_output gfxterm serial" with a serial console and no video device), reported and reproduced upstream and still open. UEFI remains the default for the reason recorded; bookworm images are noted as unaffected. Co-Authored-By: Claude Fable 5.1 --- config/samples/providers-config.yaml | 2 +- docs/architecture.md | 8 ++++++-- docs/kvm-provider.md | 16 ++++++++++------ .../2026-09-04-1404-kvm-provider.md | 18 ++++++++++++++++++ internal/provider/kvm/domainxml.go | 6 ++++-- 5 files changed, 39 insertions(+), 11 deletions(-) diff --git a/config/samples/providers-config.yaml b/config/samples/providers-config.yaml index 871b9d0..f80cca4 100644 --- a/config/samples/providers-config.yaml +++ b/config/samples/providers-config.yaml @@ -41,7 +41,7 @@ providers: url: https://cloud.debian.org/images/cloud/trixie/latest/debian-13-generic-amd64.qcow2 # sha256: "<64 hex>" # optional; verified after download # firmware: uefi # default; "bios" only for images with no EFI partition - # # (Debian 13 does not boot via BIOS, see docs/kvm-provider.md) + # # (Debian 13 hangs in GRUB under BIOS headless: Debian bug #1111240) ubuntu-24.04: url: https://cloud-images.ubuntu.com/noble/current/noble-server-cloudimg-amd64.img machineTypes: # placement.machineType names a key here diff --git a/docs/architecture.md b/docs/architecture.md index 969017a..ee1ff9d 100644 --- a/docs/architecture.md +++ b/docs/architecture.md @@ -455,8 +455,12 @@ they read as choices, not accidents. Chronological by build step. state at all; answering that with delete-and-recreate is too drastic, so the next poll decides. - **KVM guests boot UEFI by default, secure boot off.** Found on the - first real host: Debian 13's generic image hangs after GRUB under - SeaBIOS regardless of CPU model or seed, and boots under OVMF. Every + first real host: Debian 13's generic image hangs in GRUB under SeaBIOS + on a headless VM (Debian bug #1111240: `terminal_output gfxterm + serial` with a serial console and no video device) regardless of CPU + model or seed, and boots under OVMF with the serial console working. + The alternatives were a virtio console (loses the kernel's serial + output) or a video device the VMs have no use for. Every current distro cloud image supports UEFI, so it is the default and `firmware: bios` is the per-image escape; libvirt's firmware autoselection (``) keeps the OVMF path out of the diff --git a/docs/kvm-provider.md b/docs/kvm-provider.md index 4c45b08..718d71f 100644 --- a/docs/kvm-provider.md +++ b/docs/kvm-provider.md @@ -111,7 +111,7 @@ ssh virt@nas1.home.lan ip link show br0 | `images..url` | yes per entry | | http(s) URL of a cloud image (qcow2 or raw) | | `images..sha256` | | | expected digest; verified after download, a change triggers a re-download | | `images..format` | | `qcow2` | `qcow2` or `raw` | -| `images..firmware` | | `uefi` | `uefi` (OVMF, secure boot off) or `bios` (SeaBIOS) for images without an EFI system partition; Debian 13's BIOS path does not boot, see *Known gaps* | +| `images..firmware` | | `uefi` | `uefi` (OVMF, secure boot off) or `bios` (SeaBIOS) for images without an EFI system partition; Debian 13 hangs in GRUB under BIOS on a headless VM, see *Known gaps* | | `machineTypes.` | ≥ 1 | | `{vcpus, memoryMiB}` size profile | | `hosts[].name` | yes | | the `placement.zone` value that selects this host; unique | | `hosts[].address` | yes | | SSH endpoint, `host:port` | @@ -233,11 +233,15 @@ The provider logs through controller-runtime's logger, named `kvm`, with - **IP discovery needs either guest traffic or the agent.** A VM on a bridge with no DHCP never gets an address and stays `Provisioning` forever; check `virsh console`. -- **Debian 13 cloud images do not boot via BIOS on the tested host** - (Ryzen 8845HS, libvirt 11.6, QEMU 10.1): GRUB prints "Booting" and - the kernel never reaches the console, with or without a seed, with - host-passthrough and host-model CPUs alike. The same image boots - under UEFI, which is why `firmware` defaults to `uefi`. Images that +- **Debian 13 cloud images hang in GRUB under BIOS on a headless VM** + ([Debian bug #1111240](https://bugs.debian.org/1111240), open, in + `grub2`): trixie's image ships `terminal_output gfxterm serial` in its + GRUB config, and with a serial console but no video device (exactly + the domain this provider defines) GRUB prints "Booting" and never + hands over to the kernel. Not host-specific; reproduced by the Debian + cloud team. The same image boots under UEFI with the serial console + intact, which is why `firmware` defaults to `uefi`. Debian 12 + (bookworm) images are unaffected and boot either way. Images that need `firmware: bios` are the exception, not the rule. ## Testing diff --git a/docs/plans-executions/2026-09-04-1404-kvm-provider.md b/docs/plans-executions/2026-09-04-1404-kvm-provider.md index 00991aa..27b016b 100644 --- a/docs/plans-executions/2026-09-04-1404-kvm-provider.md +++ b/docs/plans-executions/2026-09-04-1404-kvm-provider.md @@ -514,3 +514,21 @@ configured, and the cache reuses a base image by key, so switching `KVM_TEST_IMAGE_URL` requires deleting `base-test-image.qcow2` and its marker by hand first. That is the reviewer's Step 4 finding showing up in practice; still documented rather than fixed. + +Follow-up, same day: the cause is known upstream. [Debian bug +#1111240](https://bugs.debian.org/1111240) (filed 2025-08-16 against +the cloud image, reassigned to `grub2`, still open as of 2026-09-10; +trixie ships grub2 2.12-9+deb13u2) reports exactly this with a domain +XML almost identical to ours: BIOS, virtio disk, SATA cdrom seed, bridge +NIC, ``, no ``/`