Author SHA1 Message Date
kacerrandClaude Fable 5.1 a114201ad3 Add KVM provider methods: Create/Get/Delete/ListByTag over the libvirt seam
Create looks the domain up first and only starts it when it exists;
otherwise it caches the image, reuses or creates the overlay, rebuilds
the seed ISO unconditionally (a cancelled upload can leave a truncated
seed that no later attempt would repair), defines and starts the domain.
Get maps libvirt state plus the first routable IPv4 from ARP or the
guest agent onto the provider states. Delete destroys, undefines and
removes the per-VM volumes by name, tolerating every already-gone case.
ListByTag keeps managed domains across all hosts, skipping domains it
cannot read and failing the sweep only for an unreachable host.

Also corrects the Step 2 seed determinism test: the iso9660 writer
stamps the build time into the volume descriptor, so builds differ by a
few bytes; the test now compares label, files and size.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
2026-09-08 10:48:59 +00:00
kacerrandClaude Fable 5.1 7d177b4817 Add KVM provider image cache: fetch base images into host pools on first use
resolveImage maps placement.image to a configured key or URL; imageCache
streams the download through StorageVolUpload with a sha256 tee, writes
a marker volume when the upload is verified, and re-fetches when the
marker is missing or the configured checksum changed. Errors carry
taxonomy sentinels that classify now honours. Adds the in-memory fake
libvirt API used from here on.

Step 3b of docs/plans/2026-09-04-1404-kvm-provider.md.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
2026-09-05 03:32:39 +00:00
kacerrandClaude Fable 5.1 d171c0426e Add KVM provider transport: SSH-forwarded libvirt connections and error classification
Per-host connection pool over SSH with the libvirt unix socket forwarded
through the session, lazy dial and redial after disconnect, context
cancellation for go-libvirt's context-less RPCs, handshake deadline and
SSH keepalives. Error classification onto the provider taxonomy. Tested
with a fake dialer and an in-process SSH server that forwards unix sockets.

Step 3 of docs/plans/2026-09-04-1404-kvm-provider.md.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
2026-09-05 03:27:57 +00:00
kacerrandClaude Fable 5.1 7aea14fbb3 Add KVM provider pure builders: domain XML, volumes, seed ISO, metadata
Introduces internal/provider/kvm with the side-effect-free pieces of the
provider: providerID format/parse, the namespaced <metadata> element that
carries the GC contract, q35 domain XML over pool volumes, volume XML for
overlays and raw seeds, the NoCloud seed ISO built with kdomanski/iso9660,
and config defaults. All unit-tested.

Step 2 of docs/plans/2026-09-04-1404-kvm-provider.md.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
2026-09-04 20:26:17 +00:00
kacerrandClaude Fable 5.1 bd633c0288 Add KVM provider config: KVMConfig structs and validation
Adds the kvm block to the providers config (SSH access, images map,
machine-type profiles, hosts list) with validation and table tests.
validate() now rejects mismatched type-specific blocks generically via
setBlocks() instead of pairwise checks.

Step 1 of docs/plans/2026-09-04-1404-kvm-provider.md.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
2026-09-04 14:25:06 +00:00
kacerrandClaude Fable 5.1 1b5ddb8349 Add plan: KVM/libvirt provider
Design for a kvm provider type: SSH-forwarded libvirt socket, multi-host
config with placement.zone selecting the host, bridged networking with
ARP/guest-agent IP discovery, operator-fetched distro cloud images cached
in the host's storage pool, and a NoCloud seed ISO carrying the CRD's
cloud-init user-data.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
2026-09-04 14:05:11 +00:00
kacerr 20ffba604b Merge pull request 'feat/demo-scripts' (#3) from feat/demo-scripts into feat/proxy-operator
Reviewed-on: #3
2026-08-12 00:11:20 +02:00
27 changed files with 5309 additions and 26 deletions
+7 -1
View File
@@ -83,7 +83,13 @@
"Bash(kubectl -n egress-proxies-operator-system get pods -o wide)",
"Bash(kubectl -n egress-proxies-operator-system get deploy egress-proxies-operator-controller-manager -o jsonpath='{.spec.template.spec.containers[0].args}')",
"Bash(kubectl -n egress-proxies-operator-system logs deploy/egress-proxies-operator-controller-manager)",
"Bash(python3 -c \"import json; d=json.load\\(open\\('docs/deploy/sa_key.json'\\)\\); print\\(d.get\\('type'\\), d.get\\('client_email'\\)\\)\")"
"Bash(python3 -c \"import json; d=json.load\\(open\\('docs/deploy/sa_key.json'\\)\\); print\\(d.get\\('type'\\), d.get\\('client_email'\\)\\)\")",
"Bash(git add *)",
"Bash(git commit -q -m 'Add plan: KVM/libvirt provider *)",
"Bash(go get *)",
"Bash(go build *)",
"Bash(go test *)",
"Bash(git commit *)"
],
"additionalDirectories": [
"/Users/jan.novak/srv/go/egress-proxies-operator/.claude",
@@ -0,0 +1,285 @@
# Execution: KVM/libvirt provider
Plan: [2026-09-04-1404-kvm-provider.md](../plans/2026-09-04-1404-kvm-provider.md)
- [x] Step 0 — Save plan + branch `feat/kvm-provider`
- [x] Step 1 — Config (`KVMConfig` & friends, `validate()` case, tests)
- [x] Step 2 — Pure builders (`domainxml.go`, `volume.go`, `seed.go`, `id.go`, `metadata.go`)
- [x] Step 3 — Transport + errors (`conn.go`, `errors.go`)
- [x] Step 3b — Image cache (`image.go`)
- [x] Step 4 — Provider methods (`kvm.go`) over a fake API
- [ ] Step 5 — Wiring, manifests, samples, docs, CHANGELOG
- [ ] Step 6 — Integration test against a real hypervisor
- [ ] Step 7 — Manual validation + MR
## Step 0 — Save plan + branch
Plan committed alone on `feat/proxy-operator`, then branched:
```bash
git checkout -b feat/kvm-provider
```
## Step 1 — Config
The new structs (`KVMConfig`, `KVMHost`, `KVMMachineType`, `KVMImage`) and
`validateKVM` went into their own file, `internal/provider/config_kvm.go`,
rather than being appended to `config.go` as the plan said: the KVM block has
far more fields and rules than the other two providers, and keeping it
separate leaves `config.go` readable. Tests likewise live in
`config_kvm_test.go` instead of extending the table in `config_test.go`; the
invalid-config table covers every required field, duplicate host names, image
URL scheme, sha256 shape, and the disk format enum.
While adding the `kvm` case to `validate()`, the pairwise "type is X but a Y
block is set" checks (which would have needed six branches with three types)
were replaced by a `setBlocks()` helper that lists the non-nil blocks and
rejects any that don't match `type`. The error strings are unchanged, so the
existing GCP/kubernetes tests still pass.
Dependencies were fetched up front so the API signatures in the plan could be
checked with `go doc`; they stay `// indirect` in `go.mod` until Step 2 imports
them:
```bash
go get github.com/digitalocean/go-libvirt@latest github.com/kdomanski/iso9660@v0.4.0
# go-libvirt resolved to v0.0.0-20260814190004-1a83157e1858 (no tagged release)
# side effect: golang.org/x/crypto 0.54.0 → 0.56.0, x/mod, x/text, x/tools bumped
```
Worth noting: `go doc` showed that go-libvirt's `New(net.Conn)` is deprecated
in favour of `NewWithDialer(socket.Dialer)`, where `Dialer` is a one-method
`Dial() (net.Conn, error)` interface. Step 3 should wrap the SSH client's unix
forward in that interface instead of using `New`; the plan's transport design
is otherwise unaffected.
## Step 2 — Pure builders
`internal/provider/kvm/` now has the five builders from the plan plus
`kvm.go` holding the package doc and `withDefaults` (disk size 10 GiB, the
default vendor-data, `/var/run/libvirt/libvirt-sock`, pool `default`, image
format `qcow2`). Everything is a pure function over `encoding/xml` structs or
strings, unit-tested by unmarshalling the output back and checking fields.
Two design choices differ from the plan text:
- **Domain disks reference pool volumes, not host paths.** The plan had
`buildDomainXML` take the overlay and seed paths (needing two
`StorageVolGetPath` round-trips per Create). The domain XML now uses
`<disk type="volume"><source pool=... volume=.../>`, so only the *base*
image path is ever looked up (the overlay's `<backingStore>` needs a real
path). Fewer RPCs and the builder's inputs are all names.
- **Capacities are always `unit="bytes"`.** libvirt's `G` means GiB but
`GB` means 10^9; spelling the byte count out removes the ambiguity from
both the overlay and the seed/base volumes.
The metadata element is written with a default namespace
(`<instance xmlns="…/kvm">`) because `encoding/xml` cannot emit prefixes on
marshal. libvirt may hand it back prefixed; `parseInstanceMetadata` is
tested against both spellings since `encoding/xml` matches on the namespace
URI. Whether libvirt accepts the default-namespace form on `DomainDefineXML`
is confirmed in Step 6 against a real host, not here.
Checked before relying on `github.com/kdomanski/iso9660` for the seed: its
filename mangling lowercases and keeps `-`, so `user-data`/`meta-data`/
`vendor-data` come out as `user-data;1` etc. in the directory record, which
the Linux isofs driver (and therefore cloud-init's NoCloud mount) presents as
`user-data`. The seed test opens the ISO with the same library's reader and
asserts label `cidata`, all three files, and byte-identical output for
identical inputs. `NewWriter()` stages files in a temp directory, so
`Cleanup()` is deferred.
`go mod tidy` ran as part of this step; it promoted `kdomanski/iso9660` to a
direct dependency and dropped `go-libvirt` again because nothing imports it
yet — Step 3 brings it back.
Worth noting: `withDefaults` copies the `Images` map and `Hosts` slice
instead of mutating the config in place, so the registry's shared
`ProviderConfig` stays untouched. The default vendor-data installs
`qemu-guest-agent` via `packages:` + `runcmd`, which means the first boot
needs network access to the distro mirrors; a Proxy on an isolated bridge
would still work (ARP discovery), just without the agent.
## Step 3 — Transport + errors
`conn.go` holds the `libvirtAPI` seam (17 methods, signatures as in the
plan), a per-host connection pool, and the SSH dialer; `errors.go` holds
`classify`, `isLibvirtError`, `hasCode` and `logAPIError`. Both are
unit-tested; the whole package still runs in about a second under `-race`.
The one thing the plan did not anticipate: **go-libvirt RPCs take no
context and never time out**. A hung hypervisor would pin a reconcile worker
forever. So the pool exposes `call(ctx, host, fn)` rather than handing out
raw clients: `fn` runs in a goroutine, and if `ctx` ends first the pool
closes the SSH connection out from under it (which makes the blocked RPC
return) and reports `ctx.Err()`. The same helper (`awaitOrCancel`) bounds
`lv.Connect()` during dial. Two more liveness measures for the same reason:
the SSH handshake gets a deadline (ctx's, or 30 s), and each connection runs
an OpenSSH `keepalive@openssh.com` loop every 15 s that closes the client
when a request errors or goes unanswered for one interval.
Connection-drop policy in `call`: any error that is *not* a `libvirt.Error`
(so io.EOF, "connection closed", SSH failures) drops the cached connection
and the next call redials; a daemon-reported `libvirt.Error` keeps it. Dials
are serialised per host (concurrent Creates on one host share a dial) but
hosts don't block each other.
As Step 1 noted, the dialer uses `libvirt.NewWithDialer` with a
`socket.Dialer` that wraps `ssh.Client.Dial("unix", socket)` rather than the
deprecated `New(net.Conn)`.
Tests: the pool is exercised through a fake dialer (reuse, redial after
`Disconnected`, drop on transport error, keep on libvirt error, cancel
releases a blocked call, concurrent callers share one dial, hosts
independent). The SSH layer is tested against an in-process
`x/crypto/ssh` server that serves `direct-streamlocal@openssh.com` channels
by dialing the requested unix path (an echo socket standing in for
`libvirt-sock`), with keys generated per test and a real `known_hosts` file
written via `knownhosts.Line`. Covered: forward round-trip and the socket
path the server saw, unknown host key rejected with `knownhosts.KeyError`,
`insecureIgnoreHostKey`, wrong client key, cancelled ctx, missing key or
known_hosts file, and keepalive both closing a stalled client and leaving a
healthy one alone. The libvirt handshake (`Connect()`) itself is *not*
covered here — faking libvirt's XDR RPC just for that is not worth it; the
Step 6 integration test covers it against a real host.
`go mod tidy` promoted `go-libvirt` and `golang.org/x/crypto` to direct
dependencies.
Worth noting: `classify` maps `ErrOperationInvalid` (e.g. destroying a
domain that is already off) to Transient; Delete in Step 4 must check that
code itself via `hasCode` and treat it as success rather than relying on
the classifier. Nothing maps to `ErrQuotaExceeded` — libvirt has no quota
concept; an out-of-space pool surfaces as a Transient `ErrInternalError`.
## Step 3b — Image cache
`image.go` follows the plan closely: `resolveImage` maps `placement.image`
to a configured key or a direct http(s) URL (else Permanent);
`imageCache.ensure` checks base volume + marker (+ marker content when a
checksum is configured), tears down and re-fetches when anything is off,
streams the HTTP body through `StorageVolUpload` with a `sha256` tee,
refreshes the pool, then writes the marker last. A per-`(host, volume)`
mutex serialises concurrent callers; the pool itself is the only cache
state.
Errors from this file are pre-tagged with the taxonomy sentinel via
`fmt.Errorf("%w: …", provider.ErrPermanent, …)` and `classify` now honours
such tags before looking at libvirt codes. That keeps one `wrapErr` path in
the provider instead of special-casing image errors.
`fake_test.go` landed with this step rather than Step 4: an in-memory
`libvirtAPI` with pools, volumes (content, format, capacity, backing path),
domains (state, metadata, per-source interface addresses), per-method error
injection (`fail` always / `failOnce`) and an `uploadHook` to break a stream
mid-way. It models the libvirt error codes the provider branches on
(`ErrStorageVolExist`, `ErrNoStorageVol`, `ErrNoDomain`,
`ErrOperationInvalid`, `ErrNoDomainMetadata`, `ErrAgentUnresponsive`).
Tests cover every case in the plan plus: no checksum configured (marker is
still written with the computed digest; second call needs no marker read),
403/500 alongside 404/503, missing pool → Permanent, cancelled ctx →
Transient with nothing created, and different hosts fetching independently.
One test-harness lesson worth keeping: `httptest`'s server drops
`Content-Length` and switches to chunked encoding for bodies larger than
its 2 KiB write buffer, so the fake mirror sets the header explicitly.
Real distro mirrors send it; a server that doesn't is reported as
Permanent with a message naming the header.
Worth noting: the base volume is created with `capacity = Content-Length`
and the *image's* format (`qcow2`), and the raw bytes are uploaded over it.
That relies on libvirt's upload writing the file verbatim (which is how
`virsh vol-upload` populates volumes); the following `StoragePoolRefresh`
makes libvirt re-read the real virtual size from the qcow2 header. Step 6
confirms this on a real host. Also, the whole download runs inside one
`pool.call`, so cancellation aborts the HTTP body (ctx-bound) and the
upload stream fails; the half-written base volume is deleted on the spot
and, failing that, on the next attempt via the missing marker.
## Step 4 — Provider methods
`kvm.go` now holds `Provider`, `New` (loads the SSH key and known_hosts
up front via `newSSHAuth`, so a bad path fails at startup), the
`newWithDialer` seam, and `Create`/`Get`/`Delete`/`ListByTag`. Each
method resolves its host from config, then runs the whole libvirt
conversation inside one `hostPool.call`, so a cancelled reconcile drops
the connection and the blocked RPC returns. Errors pass through `classify`
in one `wrapErr`, as the Step 3b note intended.
Behaviour that goes beyond the plan text, or that a future reader should
know before touching this file:
- **Create looks up the domain first.** If it exists, only the running
check runs and the volumes are never touched. If not, nothing can be
using the volumes yet: the overlay is reused when it exists (its
creation is atomic; `ErrStorageVolExist` → reuse) but the **seed is
always deleted and rebuilt**. The plan said "exists → skip upload"; the
reviewer pointed out that a reconcile cancelled mid-upload leaves a
truncated seed that the compensating delete cannot remove, because
`hostPool.call` has already closed the connection the delete would go
over. Skipping such a seed would boot the VM without cloud-init and no
retry would ever repair it. Rebuilding costs one ~40 KiB upload per
attempt before the domain exists, nothing afterwards. The table test
breaks Create after each step and re-runs it, and a separate test
starts from a leftover truncated seed.
- **Get swallows address-source errors.** `DomainInterfaceAddresses` from
ARP returns nothing until the guest has sent traffic and the agent
source errors with `ErrAgentUnresponsive` until `qemu-guest-agent` is
up, so both are logged at V(2) and mean "no IP yet" → `Provisioning`.
Loopback, link-local and unspecified addresses are skipped when
choosing the first IPv4 (the agent reports `lo` too).
- **State mapping:** `running`/`blocked` are active; `nostate` →
Provisioning (the reconciler answers Stopped with delete-and-recreate,
too drastic for a state libvirt could not determine); everything else
(paused, pmsuspended, shutdown, shutoff, crashed) → Stopped.
- **Delete treats a missing pool as "nothing to clean"** rather than
Permanent: failing there would pin the Proxy's finalizer on a config
edit that renamed the pool. Domain removal is destroy (`ErrOperationInvalid`
on a stopped domain is fine) → undefine with managed-save, snapshot
metadata and nvram flags → overlay and seed deleted by name.
- **ListByTag** reads each domain's metadata once (passed into the same
`inspect` that Get uses, so no second `DomainGetMetadata` round-trip)
and skips domains without the operator's element. A domain that cannot
be read at all — vanished mid-sweep, corrupt metadata, state call
failing — is *skipped*, not fatal: GC never deletes what it does not
see, so leaving one out is safe, whereas one bad domain blocking GC for
every other one (the first version's behaviour, caught in review) is
not. An unreachable host still fails the whole sweep.
`ConnectListAllDomains` is called with flags 0 (active and inactive).
- A host named in a providerID but no longer in `kvm.hosts` is Permanent
for Get and Delete: the operator cannot reach it, and retrying forever
would hide the config problem.
Tests run the provider over a `hostDialer` that maps host names to
`fakeAPI` instances (an unmapped host is "unreachable"). The image is
pre-seeded in the pool for most tests; one test runs the full first-use
download through `httptest` and confirms a second Proxy on the same host
does not download again.
Found while running the package: `TestBuildSeedISO_deterministic` from
Step 2 was wrong. `iso9660.ImageWriter` stamps `time.Now()` into the
volume descriptor, so two builds in different seconds differ by six
timestamp bytes; the test had only passed because both builds landed in
the same second. Byte identity is nothing the provider depends on (an
existing seed is never re-uploaded), so the test now compares label,
files and size, and the comment on `buildSeedISO` no longer claims
identical output.
The `go-operator-reviewer` agent ran on the first version of this step
and found the two bugs above (truncated seed, wedged GC sweep) plus the
double metadata read and the `nostate` mapping; all were fixed before
committing. One finding it raised is deliberately **not** fixed here
because it lives in Step 3b's `image.go` and changes the volume naming
the plan chose: re-fetching a base image (checksum edited in config, or
upstream content changed) deletes `base-<key>.<fmt>` while existing
overlays still back onto it, breaking those VMs' disks. Content-addressed
base names would avoid it. Flagged for the user; until decided, Step 5's
docs must warn that an image's `sha256`/`url` must not change while
Proxies using it exist.
Worth noting: `go test -cover` reports 88.2 % for the package; the
uncovered lines are the real SSH/libvirt dial path and log-only
branches. `go vet` and `-race` (three runs) are clean. Wiring into
`cmd/main.go` is Step 5, so `New` is not yet reachable from a running
operator.
+313
View File
@@ -0,0 +1,313 @@
# Plan: KVM/libvirt provider
**Created:** 2026-09-04 14:04
## Context
The operator provisions egress proxies through named provider instances (today
`gcp` and `kubernetes`). The user has several KVM hypervisors on the home
network and wants proxy VMs created there, using stock Debian/Ubuntu cloud
images so the VM stays a normal machine they can customise through their own
cloud-init user-data. Longer term the operator may become a generic "run a VM
or container with a set of scripts" tool; the KVM provider should not
foreclose that.
The provider contract (`internal/provider/provider.go`: `Create`/`Get`/`Delete`/
`ListByTag`, keyed by `NameFromUID`, cloud-init passed verbatim, GC-by-tag) fits
libvirt with **no CRD or reconciler changes**. All work is a new provider
package plus config/wiring/docs, following the GCP provider's shape.
Decisions taken with the user (2026-09-04):
- **Transport:** SSH to each hypervisor, forwarding the libvirt unix socket
(`ssh.Client.Dial("unix", "/var/run/libvirt/libvirt-sock")` → `libvirt.New(conn)`).
Pure Go, no cgo, no libvirt libs in the image. Verified: `x/crypto/ssh`
`Client.Dial` has a `"unix"` case (`dialStreamLocal`), go-libvirt `New(net.Conn)`.
- **Topology:** one provider instance can span several hosts from day one
(config has a `hosts` list; providerID encodes the host), so adding hosts
later is a config edit with no providerID migration. With one host configured
`placement.zone` may be omitted.
- **Networking:** VM NIC on a host bridge to the LAN, IP from the router's
DHCP. IP discovery via `DomainInterfaceAddresses` (ARP source first, then
guest agent). Provider-injected `vendor-data` installs `qemu-guest-agent`
so agent discovery works on stock images without touching user-data.
- **Base image:** a stock Debian/Ubuntu cloud image (qcow2, cloud-init
preinstalled) that **the operator fetches itself** on first use per host:
`placement.image` names an entry in the provider's `images:` map (url +
optional sha256) or is an `https://` URL directly. The provider streams the
download through libvirt's `StorageVolUpload` into the host's pool, so
nothing is pre-placed and the operator needs no filesystem access on the
host. Per-VM disk is a qcow2 overlay with the cached base as backing store.
- **Seed:** NoCloud ISO (`user-data`, `meta-data`, `vendor-data`) built in
pure Go with `github.com/kdomanski/iso9660` (`WriteTo(w, "cidata")`) and
streamed to the pool with `StorageVolUpload(vol, io.Reader, ...)`. Nothing
to install on the hypervisors.
## Design
### Config (`internal/provider/config.go`)
```yaml
- name: kvm-home
type: kvm
kvm:
sshKeyFile: /etc/proxy-operator/kvm/id_ed25519 # required
knownHostsFile: /etc/proxy-operator/kvm/known_hosts # required unless insecureIgnoreHostKey
insecureIgnoreHostKey: false
diskSizeGb: 10 # overlay capacity; cloud-init growpart expands root fs
vendorData: | # optional override; default installs qemu-guest-agent
#cloud-config
...
images: # optional; placement.image names a key (or is an https URL)
debian-13:
url: https://cloud.debian.org/images/cloud/trixie/latest/debian-13-generic-amd64.qcow2
sha256: "…" # optional; verified after download, re-fetch when it changes
format: qcow2 # default qcow2
ubuntu-24.04:
url: https://cloud-images.ubuntu.com/noble/current/noble-server-cloudimg-amd64.img
machineTypes: # required, ≥1; placement.machineType names a key
small: {vcpus: 1, memoryMiB: 1024}
medium: {vcpus: 2, memoryMiB: 2048}
hosts: # required, ≥1, unique names
- name: nas1 # placement.zone value
address: nas1.home.lan:22 # required
user: virt # required; must be in the libvirt group
socket: /var/run/libvirt/libvirt-sock # default
pool: default # default "default"
bridge: br0 # required
```
Add `KVMConfig`, `KVMHost`, `KVMMachineType`, `KVMImage` structs; a `"kvm"`
case in `validate()` (cross-block checks like the others, required fields
above, unique host names, image URLs must be `http(s)`, sha256 if set must be
64 hex chars). Tests in `config_test.go` extend the existing table.
### Placement mapping
| Placement field | KVM meaning |
|---|---|
| `zone` | host name from `hosts`; empty → the single configured host, else `ErrPermanent` |
| `machineType` | key in `machineTypes`; unknown → `ErrPermanent` |
| `image` | key in `images:`, or an `https://` URL; anything else → `ErrPermanent`. Fetched into the host's pool on first use (see *Image cache*) |
| `region` | ignored |
### providerID
`hosts/<host>/domains/<name>` with `formatProviderID`/`parseProviderID`
(same self-contained style as GCP's `zones/<zone>/instances/<name>`).
### GC contract without labels
Domain XML gets a `<metadata>` child in a private namespace, e.g.
`xmlns="https://gitea.home.hrajfrisbee.cz/kacerr/egress-proxies-operator/kvm"`,
carrying `<managed>true</managed><uid>…</uid><createdAt>RFC3339</createdAt>`.
`Get` reads it via `DomainGetMetadata(dom, DomainMetadataElement, uri)`;
`ListByTag` does `ConnectListAllDomains` on every host and keeps domains where
that call succeeds and `managed == "true"`. `CreatedAt` comes from the metadata
(libvirt has no creation time). A host that is unreachable makes `ListByTag`
return an error (GC skips the sweep that round rather than acting on a partial view).
### Connection management (`internal/provider/kvm/conn.go`)
Per-host lazily dialed `*ssh.Client` + `*libvirt.Libvirt`, cached under a
mutex, re-dialed when `Disconnected()` is closed or a call fails with a
transport error. SSH auth: private key from `sshKeyFile`; host key check via
`knownhosts.New(knownHostsFile)` unless `insecureIgnoreHostKey`. Dial timeout
from `ctx`. The provider calls libvirt through a narrow interface:
```go
type libvirtAPI interface {
DomainLookupByName(name string) (libvirt.Domain, error)
DomainDefineXML(xml string) (libvirt.Domain, error)
DomainCreate(d libvirt.Domain) error
DomainDestroy(d libvirt.Domain) error
DomainUndefineFlags(d libvirt.Domain, f libvirt.DomainUndefineFlagsValues) error
DomainGetState(d libvirt.Domain, flags uint32) (int32, int32, error)
DomainGetMetadata(d libvirt.Domain, typ int32, uri libvirt.OptString, f libvirt.DomainModificationImpact) (string, error)
DomainInterfaceAddresses(d libvirt.Domain, src uint32, flags uint32) ([]libvirt.DomainInterface, error)
ConnectListAllDomains(need int32, f libvirt.ConnectListAllDomainsFlags) ([]libvirt.Domain, uint32, error)
StoragePoolLookupByName(name string) (libvirt.StoragePool, error)
StorageVolLookupByName(p libvirt.StoragePool, name string) (libvirt.StorageVol, error)
StorageVolGetPath(v libvirt.StorageVol) (string, error)
StorageVolCreateXML(p libvirt.StoragePool, xml string, f libvirt.StorageVolCreateFlags) (libvirt.StorageVol, error)
StorageVolUpload(v libvirt.StorageVol, r io.Reader, off, n uint64, f libvirt.StorageVolUploadFlags) error
StorageVolDownload(v libvirt.StorageVol, w io.Writer, off, n uint64, f libvirt.StorageVolDownloadFlags) error
StorageVolDelete(v libvirt.StorageVol, f libvirt.StorageVolDeleteFlags) error
StoragePoolRefresh(p libvirt.StoragePool, flags uint32) error
}
```
(All signatures verified against go-libvirt `main` on 2026-09-04.) A
`hostDialer func(ctx, KVMHost) (libvirtAPI, io.Closer, error)` is the test seam;
`newWithDialer(cfg, dialer)` mirrors GCP's `newWithAPI`.
### Image cache (`internal/provider/kvm/image.go`)
`ensureImage(ctx, host, api, img) (basePath string, err error)` makes the base
image present in the host's pool, once per host per image:
- Volume names: `base-<key>.<format>` for an `images:` key, or
`base-<sha256(url)[:16]>.<format>` for a URL given directly in
`placement.image`. Marker volume `<base>.sha256` (raw, 64 bytes) holds the
digest of the verified upload.
- **Ready check:** base volume exists *and* marker exists *and* (no `sha256`
configured, or marker content == configured `sha256`). Read the marker with
`StorageVolDownload`. Ready → return `StorageVolGetPath(base)`.
- **Not ready:** delete both volumes if present (partial upload from a crash,
or a changed checksum), then `GET url` with the ctx, require a
`Content-Length` (`ErrPermanent` otherwise), `StorageVolCreateXML` the base
with `<capacity unit="bytes">Content-Length</capacity>` and the configured
format, `StorageVolUpload(base, io.TeeReader(body, sha256), 0, 0, 0)` (length
0 = until EOF, so nothing is buffered in the operator), then compare the
digest with the configured one (mismatch → delete base, `ErrPermanent`),
`StoragePoolRefresh` so libvirt re-reads the real qcow2 capacity, create the
marker volume and upload the digest last.
- HTTP 4xx → `ErrPermanent`; 5xx, network, or upload transport errors →
`ErrTransient` (partial volumes are cleaned on the next attempt).
- A per-`(host, image)` `sync.Mutex` map serialises concurrent Creates for the
same image on the same host; the second caller finds it ready.
- **Judgment call:** the fetch runs synchronously inside `Create`, so the first
Proxy for a given image on a given host holds a reconcile worker for the
download (LAN speed, a few hundred MB, once). This keeps `Create` idempotent
and needs no background state. Async fetch with a `Provisioning`-style
progress status is a possible later improvement, noted in the docs.
### Create (idempotent, fire-and-forget after the image is cached)
1. Resolve host, machine type, pool, and image source (`images:` key or URL;
else `ErrPermanent`). `ensureImage` → base path.
2. Overlay volume `<name>.qcow2`: `StorageVolCreateXML` with
`<capacity unit="G">diskSizeGb</capacity>`, `<target><format type="qcow2"/></target>`,
`<backingStore><path>base</path><format type="qcow2"/></backingStore>`.
`ErrStorageVolExist` → reuse.
3. Seed volume `<name>-seed.iso`: build ISO in memory (`user-data` = req.CloudInit,
`meta-data` = `instance-id: <name>\nlocal-hostname: <name>\n`, `vendor-data` =
cfg or default), `StorageVolCreateXML` with exact capacity + `<format type="raw"/>`,
then `StorageVolUpload`. Exists → skip upload (content is deterministic per spec hash;
spec changes recreate the whole instance anyway).
4. `DomainLookupByName(name)`; if `ErrNoDomain` → `DomainDefineXML(buildDomainXML(...))`.
5. `DomainGetState`; if not running → `DomainCreate`. Return providerID.
`buildDomainXML` is a pure function over `encoding/xml` structs: `type="kvm"`,
q35 machine, `host-passthrough` CPU, vcpus/memory from the profile, virtio disk
(overlay) + sata cdrom (seed), `interface type="bridge"` with virtio model,
`org.qemu.guest_agent.0` virtio channel, serial console, virtio rng, no
graphics, and the `<metadata>` block. Unit-tested field by field like
`buildInsertRequest`.
### Get
`parseProviderID` → host → `DomainLookupByName` (`ErrNoDomain` → `ErrNotFound`)
→ `DomainGetState` + `DomainGetMetadata` + `DomainInterfaceAddresses`
(ARP, then agent; first IPv4 wins; agent errors while the agent isn't up yet
are swallowed). Mapping:
| libvirt state | IP | Instance.State |
|---|---|---|
| running | yes | Running |
| running | no | Provisioning |
| shutoff / crashed / paused / pmsuspended | – | Stopped |
| no domain | – | `ErrNotFound` |
### Delete (idempotent)
Destroy if running (`ErrOperationInvalid` on a stopped domain is fine) →
`DomainUndefineFlags(managed-save | snapshots-metadata | nvram)` → delete
`<name>.qcow2` and `<name>-seed.iso` (`ErrNoStorageVol` fine). Volumes are
deleted by name even when the domain is already gone, so a crash between
volume creation and define still gets cleaned up on the next reconcile.
Known gap (documented, not solved now): volumes orphaned with no domain *and*
no CR are invisible to `ListByTag`.
### Errors (`internal/provider/kvm/errors.go`)
`classify(err)` on `libvirt.Error.Code`: `ErrNoDomain`, `ErrNoStorageVol` →
`ErrNotFound`; `ErrNoStoragePool`, `ErrXMLError`, `ErrConfigUnsupported`,
`ErrNoSupport`, `ErrAuthFailed`, `ErrInvalidArg` → `ErrPermanent`; SSH dial /
host-key / transport errors and everything else → `ErrTransient`. Image
download: HTTP 4xx, missing Content-Length, checksum mismatch → `ErrPermanent`;
5xx and network errors → `ErrTransient`. Quota is not detectable from libvirt;
left as transient (revisit with a capacity check later).
### Wiring & manifests
- `cmd/main.go`: add `"kvm": kvm.New` to the builtins map.
- `config/manager/manager.yaml`: mount Secret `kvm-ssh` (optional) at
`/etc/proxy-operator/kvm` (keys `id_ed25519`, `known_hosts`); document the
`kubectl create secret generic kvm-ssh --from-file=...` command.
- `config/samples/providers-config.yaml`: add the `kvm-home` block with the
`images:` map (Debian 13 and Ubuntu 24.04 official cloud-image URLs);
`config/samples/proxy_kvm.yaml`: Proxy with `placement: {zone: nas1,
machineType: small, image: debian-13}` and the same squid cloud-init as the
GCP sample plus `ssh_authorized_keys`.
- `hack/providers-dev.yaml` stays kubernetes-only (needs no secrets).
- Deps: `github.com/digitalocean/go-libvirt` (latest pseudo-version),
`github.com/kdomanski/iso9660 v0.4.0`, promote `golang.org/x/crypto` to direct.
### Docs
- New `docs/kvm-provider.md`: hypervisor prerequisites (user in `libvirt`
group, a bridge to the LAN, a storage pool with enough space), the config
reference including `images:`, how the image cache works (`base-*` and
`*.sha256` volumes in the pool, when a re-download happens, how to evict
with `virsh vol-delete`), the placement table, how to log into a VM
(`ssh_authorized_keys` in user-data, `virsh console` on the host), and the
known gaps (no quota detection, orphan volumes without a domain,
synchronous first-time image fetch).
- Update the provider tables in `README.md`, `docs/architecture.md` §5,
`docs/testing.md` (integration test section).
### Towards the generic runner (not built now, just kept open)
Provider knows nothing about squid. Future hooks: `vendor-data` for injecting an
agent, `QEMUDomainAgentCommand` for reading exit status/output, and a future
`Placement`-independent "job" mode in the CRD. None of this changes the KVM
provider's shape.
## Steps
0. **Save plan + branch.** Copy this file to `docs/plans/<ts>-kvm-provider.md`,
commit alone, then `git checkout -b feat/kvm-provider` off `feat/proxy-operator`
(the working main for this project).
1. **Config.** `KVMConfig` & friends, `validate()` case, table tests.
2. **Pure builders.** `internal/provider/kvm/{domainxml.go,volume.go,seed.go,id.go,metadata.go}`
+ tests: domain XML fields, volume XML, seed contents (unpack the ISO with
the same library's reader and assert the three files + label `cidata`),
providerID round-trip, metadata marshal/parse.
3. **Transport + errors.** `conn.go` (SSH dial, host-key policy, per-host cache,
reconnect), `errors.go` + classification table test. SSH dialer tested
against an in-process `x/crypto/ssh` server that serves a unix socket
forward (cheap, keeps the dial code covered).
3b. **Image cache.** `image.go` + `image_test.go`: `httptest.Server` serving a
small fake image over the `fakeAPI` (which records uploaded bytes);
cases: fresh fetch, already ready, marker missing (partial upload → redo),
checksum changed → redo, checksum mismatch → Permanent + cleanup, 404 →
Permanent, 503 → Transient, two concurrent callers → one download.
4. **Provider methods.** `kvm.go`: `New`, `newWithDialer`, `Create/Get/Delete/ListByTag`
over `libvirtAPI`; `kvm_test.go` with a hand-written `fakeAPI` (domains and
volumes in maps, per-call error injection) covering idempotent re-Create
after each partial step, state/IP mapping, delete-when-gone, ListByTag
across two hosts and with one host down.
5. **Wiring, manifests, samples, docs, CHANGELOG.**
6. **Integration test** (`//go:build integration`, `kvm_integration_test.go`):
skipped unless `KVM_TEST_HOST`, `KVM_TEST_USER`, `KVM_TEST_SSH_KEY`,
`KVM_TEST_BRIDGE`, `KVM_TEST_IMAGE_URL` are set; fetches the image (or finds
it cached), creates, waits for Running+IP, deletes, asserts per-VM volumes
gone and the base image still cached. Deviation from the testcontainers convention,
called out in `docs/testing.md`: libvirt+KVM can't run inside a container
without `/dev/kvm`, so this targets a real hypervisor.
7. **Manual validation** on one real hypervisor, results in
`docs/plans-executions/` (like `docs/gcp-vm-validation.md`), then MR via `tea`.
## Verification
- `make test` green (`-race`, envtest suite untouched but must still pass).
- `go test -short ./internal/provider/...` covers builders, fake-API provider, errors, config.
- Integration: `KVM_TEST_HOST=nas1.home.lan … go test -tags=integration -run TestKVM ./internal/provider/kvm/`.
- End to end: deploy with the `kvm-ssh` Secret and `kvm-home` config against a
host with no cached image, apply `config/samples/proxy_kvm.yaml`, watch the
operator log the image fetch and `virsh vol-list` show `base-debian-13.qcow2`
plus its `.sha256` marker, watch `kubectl get px -w` reach `Ready` with a LAN
IP, apply a second Proxy and confirm no second download, confirm the VM shows up in `GET /v1/proxies` and works with
`hack/` demo egress-IP check, delete the Proxy and confirm `virsh list --all`
and `virsh vol-list default` are clean.
+6 -4
View File
@@ -4,10 +4,13 @@ go 1.26.0
require (
cloud.google.com/go/compute v1.65.0
github.com/digitalocean/go-libvirt v0.0.0-20260814190004-1a83157e1858
github.com/go-logr/logr v1.4.3
github.com/kdomanski/iso9660 v0.4.0
github.com/onsi/ginkgo/v2 v2.27.4
github.com/onsi/gomega v1.39.0
github.com/prometheus/client_golang v1.23.2
golang.org/x/crypto v0.56.0
google.golang.org/api v0.292.0
google.golang.org/protobuf v1.36.12-0.20260120151049-f2248ac996af
k8s.io/api v0.36.0
@@ -78,17 +81,16 @@ require (
go.uber.org/zap v1.27.1 // indirect
go.yaml.in/yaml/v2 v2.4.3 // indirect
go.yaml.in/yaml/v3 v3.0.4 // indirect
golang.org/x/crypto v0.54.0 // indirect
golang.org/x/exp v0.0.0-20251219203646-944ab1f22d93 // indirect
golang.org/x/mod v0.37.0 // indirect
golang.org/x/mod v0.38.0 // indirect
golang.org/x/net v0.57.0 // indirect
golang.org/x/oauth2 v0.36.0 // indirect
golang.org/x/sync v0.22.0 // indirect
golang.org/x/sys v0.47.0 // indirect
golang.org/x/term v0.45.0 // indirect
golang.org/x/text v0.40.0 // indirect
golang.org/x/text v0.41.0 // indirect
golang.org/x/time v0.15.0 // indirect
golang.org/x/tools v0.47.0 // indirect
golang.org/x/tools v0.48.0 // indirect
gomodules.xyz/jsonpatch/v2 v2.4.0 // indirect
google.golang.org/genproto v0.0.0-20260319201613-d00831a3d3e7 // indirect
google.golang.org/genproto/googleapis/api v0.0.0-20260630182238-925bb5da69e7 // indirect
+12 -8
View File
@@ -28,6 +28,8 @@ github.com/davecgh/go-spew v1.1.0/go.mod h1:J7Y8YcW2NihsgmVo/mv3lAwl/skON4iLHjSs
github.com/davecgh/go-spew v1.1.1/go.mod h1:J7Y8YcW2NihsgmVo/mv3lAwl/skON4iLHjSsI+c5H38=
github.com/davecgh/go-spew v1.1.2-0.20180830191138-d8f796af33cc h1:U9qPSI2PIWSS1VwoXQT9A3Wy9MM3WgvqSxFWenqJduM=
github.com/davecgh/go-spew v1.1.2-0.20180830191138-d8f796af33cc/go.mod h1:J7Y8YcW2NihsgmVo/mv3lAwl/skON4iLHjSsI+c5H38=
github.com/digitalocean/go-libvirt v0.0.0-20260814190004-1a83157e1858 h1:8xCFt73OddCD5WXxoYidJBWp3bJMC+CpE7Zwf5tcwk8=
github.com/digitalocean/go-libvirt v0.0.0-20260814190004-1a83157e1858/go.mod h1:qb0Ofa71d3oXARQf633h2tNaeBxLsVxuDp+jcsVO2+4=
github.com/emicklei/go-restful/v3 v3.13.0 h1:C4Bl2xDndpU6nJ4bc1jXd+uTmYPVUwkD6bFY/oTyCes=
github.com/emicklei/go-restful/v3 v3.13.0/go.mod h1:6n3XBCmQQb25CM2LCACGz8ukIrRry+4bhvbpWn3mrbc=
github.com/evanphx/json-patch v0.5.2 h1:xVCHIVMUu1wtM/VkR9jVZ45N3FhZfYMMYGorLCR8P3k=
@@ -96,6 +98,8 @@ github.com/joshdk/go-junit v1.0.0 h1:S86cUKIdwBHWwA6xCmFlf3RTLfVXYQfvanM5Uh+K6GE
github.com/joshdk/go-junit v1.0.0/go.mod h1:TiiV0PqkaNfFXjEiyjWM3XXrhVyCa1K4Zfga6W52ung=
github.com/json-iterator/go v1.1.12 h1:PV8peI4a0ysnczrg+LtxykD8LfKY9ML6u2jnxaEnrnM=
github.com/json-iterator/go v1.1.12/go.mod h1:e30LSqwooZae/UwlEbR2852Gd8hjQvJoHmT4TnhNGBo=
github.com/kdomanski/iso9660 v0.4.0 h1:BPKKdcINz3m0MdjIMwS0wx1nofsOjxOq8TOr45WGHFg=
github.com/kdomanski/iso9660 v0.4.0/go.mod h1:OxUSupHsO9ceI8lBLPJKWBTphLemjrCQY8LPXM7qSzU=
github.com/klauspost/compress v1.18.0 h1:c/Cqfb0r+Yi+JtIEq73FWXVkRonBlf0CRNYc8Zttxdo=
github.com/klauspost/compress v1.18.0/go.mod h1:2Pp+KzxcywXVXMr50+X0Q/Lsb43OQHYWRCY2AiWywWQ=
github.com/kr/pretty v0.2.1/go.mod h1:ipq/a2n7PKx3OHsz4KJII5eveXtPO4qwEXGdVfWzfnI=
@@ -198,12 +202,12 @@ go.yaml.in/yaml/v2 v2.4.3 h1:6gvOSjQoTB3vt1l+CU+tSyi/HOjfOjRLJ4YwYZGwRO0=
go.yaml.in/yaml/v2 v2.4.3/go.mod h1:zSxWcmIDjOzPXpjlTTbAsKokqkDNAVtZO0WOMiT90s8=
go.yaml.in/yaml/v3 v3.0.4 h1:tfq32ie2Jv2UxXFdLJdh3jXuOzWiL1fo0bu/FbuKpbc=
go.yaml.in/yaml/v3 v3.0.4/go.mod h1:DhzuOOF2ATzADvBadXxruRBLzYTpT36CKvDb3+aBEFg=
golang.org/x/crypto v0.54.0 h1:YLIA59K4fiNzHzjnZt2tUJQjQtUWfWbeHBqKtk3eScw=
golang.org/x/crypto v0.54.0/go.mod h1:KWL8ny2AZdGR2cWmzeHrp2azQPGogOv+HeQaVEXC2dk=
golang.org/x/crypto v0.56.0 h1:GUh5Ii4J5jtcseSMiRqr1jXCNHoxjeV9Fmekc2oLy6Y=
golang.org/x/crypto v0.56.0/go.mod h1:OMW5y6CY9l38uPLmxU6l6pwcXp1obtLo3e6gT7gQR2I=
golang.org/x/exp v0.0.0-20251219203646-944ab1f22d93 h1:fQsdNF2N+/YewlRZiricy4P1iimyPKZ/xwniHj8Q2a0=
golang.org/x/exp v0.0.0-20251219203646-944ab1f22d93/go.mod h1:EPRbTFwzwjXj9NpYyyrvenVh9Y+GFeEvMNh7Xuz7xgU=
golang.org/x/mod v0.37.0 h1:vF1DjpVEshcIqoEaauuHebaLk1O1forxjxBaVn884JQ=
golang.org/x/mod v0.37.0/go.mod h1:m8S8VeM9r4dzDwjrKO0a1sZP3YjeMamRRlD+fmR2Q/0=
golang.org/x/mod v0.38.0 h1:MECBjubtXD7yj4HrhIUcywNaGeNVUdfVnxmPajOk4yk=
golang.org/x/mod v0.38.0/go.mod h1:V6Xz0pq8TQ3dGqVQ1FVHuelZpAL0uNhSkk9ogYP3c40=
golang.org/x/net v0.57.0 h1:K5+3DljvIuDG9/Jv9rvyMywYNFCQ9RSUY6OOTTkT+tE=
golang.org/x/net v0.57.0/go.mod h1:KpXc8iv+r3XplLAG/f7Jsf9RPszJzdR0f58q9vGOuEU=
golang.org/x/oauth2 v0.36.0 h1:peZ/1z27fi9hUOFCAZaHyrpWG5lwe0RJEEEeH0ThlIs=
@@ -214,12 +218,12 @@ golang.org/x/sys v0.47.0 h1:o7XGOvZQCADBQQ4Y7VNq2dRWQR7JmOUW8Kxx4ZsNgWs=
golang.org/x/sys v0.47.0/go.mod h1:4GL1E5IUh+htKOUEOaiffhrAeqysfVGipDYzABqnCmw=
golang.org/x/term v0.45.0 h1:NwWyBmoJCbfTHpxrWoZ9C6/VxOf7ic219I8xZZFdrf0=
golang.org/x/term v0.45.0/go.mod h1:9aqxs0blBcrm/n0L9QW0aRVD+ktan8ssZromtqJC43w=
golang.org/x/text v0.40.0 h1:Ub2Z6/xjgF1WrYQz2nuITOEegKFtiIy+rieRJ5lHZKs=
golang.org/x/text v0.40.0/go.mod h1:hpnzDAfGV753zIKo+wk3u1bVKCGPbrnF7+7LBF/UHVY=
golang.org/x/text v0.41.0 h1:vz/seA0lnX87Othu2f/0L24RcgrXD9/YFTSuGjj3rH8=
golang.org/x/text v0.41.0/go.mod h1:jvf1O8ajNzZqhSrQBPbutR/EB83Cc0CFrezNQIwbb5M=
golang.org/x/time v0.15.0 h1:bbrp8t3bGUeFOx08pvsMYRTCVSMk89u4tKbNOZbp88U=
golang.org/x/time v0.15.0/go.mod h1:Y4YMaQmXwGQZoFaVFk4YpCt4FLQMYKZe9oeV/f4MSno=
golang.org/x/tools v0.47.0 h1:7Kn5x/d1svx/PzryTsqeoZN4TZwqeH5pGWjefhLi/1Q=
golang.org/x/tools v0.47.0/go.mod h1:dFHnyTvFWY212G+h7ZY4Vsp/K3U4/7W9TyVaAul8uCA=
golang.org/x/tools v0.48.0 h1:3+hClM1aLL5mjMKm5ovokw9epgRXPuu2tILgismM6RE=
golang.org/x/tools v0.48.0/go.mod h1:08xX0orndb/F7jJxGDicx061tyd5pcMto75YMAXr6lk=
gomodules.xyz/jsonpatch/v2 v2.4.0 h1:Ci3iUJyx9UeRx7CeFN8ARgGbkESwJK+KB9lLcWxY/Zw=
gomodules.xyz/jsonpatch/v2 v2.4.0/go.mod h1:AH3dM2RI6uoBZxn3LVrfvJ3E0/9dG4cSrbuBJT4moAY=
gonum.org/v1/gonum v0.17.0 h1:VbpOemQlsSMrYmn7T2OUvQ4dqxQXU+ouZFQsZOx50z4=
+36 -13
View File
@@ -21,6 +21,23 @@ type ProviderConfig struct {
Type string `json:"type"`
Kubernetes *KubernetesConfig `json:"kubernetes,omitempty"`
GCP *GCPConfig `json:"gcp,omitempty"`
KVM *KVMConfig `json:"kvm,omitempty"`
}
// setBlocks names the type-specific blocks that are non-nil, so validate can
// reject a block that doesn't match Type without a pairwise check per type.
func (p ProviderConfig) setBlocks() []string {
var out []string
if p.Kubernetes != nil {
out = append(out, "kubernetes")
}
if p.GCP != nil {
out = append(out, "gcp")
}
if p.KVM != nil {
out = append(out, "kvm")
}
return out
}
// KubernetesConfig configures the kubernetes-pod provider, which creates
@@ -81,22 +98,28 @@ func (c *Config) validate() error {
}
seen[p.Name] = true
prefix := fmt.Sprintf("providers[%d] %q", i, p.Name)
switch p.Type {
case "kubernetes":
if p.GCP != nil {
return fmt.Errorf("providers[%d] %q: type is kubernetes but a gcp block is set", i, p.Name)
}
case "gcp":
if p.Kubernetes != nil {
return fmt.Errorf("providers[%d] %q: type is gcp but a kubernetes block is set", i, p.Name)
}
if p.GCP == nil || p.GCP.Project == "" {
return fmt.Errorf("providers[%d] %q: gcp.project is required", i, p.Name)
}
case "kubernetes", "gcp", "kvm":
case "":
return fmt.Errorf("providers[%d] %q: type is required", i, p.Name)
return fmt.Errorf("%s: type is required", prefix)
default:
return fmt.Errorf("providers[%d] %q: unknown provider type %q", i, p.Name, p.Type)
return fmt.Errorf("%s: unknown provider type %q", prefix, p.Type)
}
for _, block := range p.setBlocks() {
if block != p.Type {
return fmt.Errorf("%s: type is %s but a %s block is set", prefix, p.Type, block)
}
}
switch p.Type {
case "gcp":
if p.GCP == nil || p.GCP.Project == "" {
return fmt.Errorf("%s: gcp.project is required", prefix)
}
case "kvm":
if err := validateKVM(prefix, p.KVM); err != nil {
return err
}
}
}
return nil
+138
View File
@@ -0,0 +1,138 @@
package provider
import (
"fmt"
"net/url"
"regexp"
)
// KVMConfig configures a kvm provider instance: one or more libvirt/KVM
// hypervisors reached over SSH (the libvirt unix socket is forwarded through
// the SSH connection, so hosts need no libvirtd TCP/TLS listener).
// placement.zone on a Proxy selects a host by name; with a single host it may
// be omitted.
type KVMConfig struct {
// SSHKeyFile is the path to the private key used for every host. Required.
SSHKeyFile string `json:"sshKeyFile"`
// KnownHostsFile is an OpenSSH known_hosts file used to verify host keys.
// Required unless InsecureIgnoreHostKey is set.
KnownHostsFile string `json:"knownHostsFile,omitempty"`
// InsecureIgnoreHostKey skips host-key verification entirely.
InsecureIgnoreHostKey bool `json:"insecureIgnoreHostKey,omitempty"`
// DiskSizeGB is the capacity of each VM's overlay disk. Default 10. The
// guest's cloud-init growpart expands the root filesystem into it.
DiskSizeGB int64 `json:"diskSizeGb,omitempty"`
// VendorData is cloud-init vendor-data placed on every seed next to the
// Proxy's own user-data. Default: installs and starts qemu-guest-agent so
// the provider can read the VM's IP even before it shows up in ARP.
VendorData string `json:"vendorData,omitempty"`
// Images are named base images; placement.image names a key here (or is
// an http(s) URL directly). The provider downloads each image into a
// host's storage pool on first use and caches it there.
Images map[string]KVMImage `json:"images,omitempty"`
// MachineTypes are named size profiles; placement.machineType names a
// key here. Required, at least one.
MachineTypes map[string]KVMMachineType `json:"machineTypes"`
// Hosts are the hypervisors. Required, at least one, unique names.
Hosts []KVMHost `json:"hosts"`
}
// KVMImage is one downloadable base image (a distro cloud image with
// cloud-init preinstalled, e.g. Debian's generic qcow2).
type KVMImage struct {
// URL is where to fetch the image from. Required, http or https.
URL string `json:"url"`
// SHA256 is the expected hex digest of the file. Optional; when set it is
// verified after download and a later change triggers a re-download.
SHA256 string `json:"sha256,omitempty"`
// Format is the image's disk format, "qcow2" (default) or "raw".
Format string `json:"format,omitempty"`
}
// KVMMachineType is a VM size profile.
type KVMMachineType struct {
VCPUs int32 `json:"vcpus"`
MemoryMiB int64 `json:"memoryMiB"`
}
// KVMHost is one hypervisor.
type KVMHost struct {
// Name is the placement.zone value that selects this host. Required.
Name string `json:"name"`
// Address is the SSH endpoint, host:port. Required.
Address string `json:"address"`
// User is the SSH user; it must be allowed to use the libvirt socket
// (typically a member of the libvirt group). Required.
User string `json:"user"`
// Socket is the libvirt unix socket path on the host.
// Default /var/run/libvirt/libvirt-sock.
Socket string `json:"socket,omitempty"`
// Pool is the libvirt storage pool for base images, overlay disks and
// seed ISOs. Default "default".
Pool string `json:"pool,omitempty"`
// Bridge is the host bridge the VM's NIC attaches to; it must lead to a
// network the cluster can reach and that hands out DHCP leases. Required.
Bridge string `json:"bridge"`
}
var sha256Hex = regexp.MustCompile(`^[0-9a-fA-F]{64}$`)
func validateKVM(prefix string, c *KVMConfig) error {
if c == nil {
return fmt.Errorf("%s: kvm block is required", prefix)
}
if c.SSHKeyFile == "" {
return fmt.Errorf("%s: kvm.sshKeyFile is required", prefix)
}
if c.KnownHostsFile == "" && !c.InsecureIgnoreHostKey {
return fmt.Errorf("%s: kvm.knownHostsFile is required unless kvm.insecureIgnoreHostKey is true", prefix)
}
if c.DiskSizeGB < 0 {
return fmt.Errorf("%s: kvm.diskSizeGb must not be negative", prefix)
}
for name, img := range c.Images {
p := fmt.Sprintf("%s: kvm.images[%q]", prefix, name)
u, err := url.Parse(img.URL)
if err != nil || (u.Scheme != "http" && u.Scheme != "https") || u.Host == "" {
return fmt.Errorf("%s.url must be an http(s) URL (got %q)", p, img.URL)
}
if img.SHA256 != "" && !sha256Hex.MatchString(img.SHA256) {
return fmt.Errorf("%s.sha256 must be 64 hex characters", p)
}
if img.Format != "" && img.Format != "qcow2" && img.Format != "raw" {
return fmt.Errorf("%s.format must be qcow2 or raw (got %q)", p, img.Format)
}
}
if len(c.MachineTypes) == 0 {
return fmt.Errorf("%s: kvm.machineTypes must define at least one profile", prefix)
}
for name, mt := range c.MachineTypes {
if mt.VCPUs <= 0 || mt.MemoryMiB <= 0 {
return fmt.Errorf("%s: kvm.machineTypes[%q] needs vcpus > 0 and memoryMiB > 0", prefix, name)
}
}
if len(c.Hosts) == 0 {
return fmt.Errorf("%s: kvm.hosts must list at least one host", prefix)
}
seen := make(map[string]bool, len(c.Hosts))
for i, h := range c.Hosts {
p := fmt.Sprintf("%s: kvm.hosts[%d]", prefix, i)
if h.Name == "" {
return fmt.Errorf("%s: name is required", p)
}
if seen[h.Name] {
return fmt.Errorf("%s: duplicate host name %q", p, h.Name)
}
seen[h.Name] = true
if h.Address == "" {
return fmt.Errorf("%s %q: address is required", p, h.Name)
}
if h.User == "" {
return fmt.Errorf("%s %q: user is required", p, h.Name)
}
if h.Bridge == "" {
return fmt.Errorf("%s %q: bridge is required", p, h.Name)
}
}
return nil
}
+208
View File
@@ -0,0 +1,208 @@
package provider
import (
"strings"
"testing"
)
const validKVMYAML = `
providers:
- name: kvm-home
type: kvm
kvm:
sshKeyFile: /etc/proxy-operator/kvm/id_ed25519
knownHostsFile: /etc/proxy-operator/kvm/known_hosts
diskSizeGb: 20
images:
debian-13:
url: https://cloud.debian.org/images/cloud/trixie/latest/debian-13-generic-amd64.qcow2
sha256: 0123456789abcdef0123456789abcdef0123456789abcdef0123456789abcdef
ubuntu:
url: http://mirror.local/noble.img
format: raw
machineTypes:
small: {vcpus: 1, memoryMiB: 1024}
medium: {vcpus: 2, memoryMiB: 2048}
hosts:
- name: nas1
address: nas1.home.lan:22
user: virt
bridge: br0
- name: lab2
address: 10.0.0.2:2222
user: root
socket: /run/libvirt/virtqemud-sock
pool: fast
bridge: br1
`
func TestLoadConfig_kvmValid(t *testing.T) {
t.Parallel()
cfg, err := LoadConfig([]byte(validKVMYAML))
if err != nil {
t.Fatalf("LoadConfig() error = %v, want nil", err)
}
k := cfg.Providers[0].KVM
if k == nil {
t.Fatal("providers[0].kvm = nil")
}
if k.SSHKeyFile != "/etc/proxy-operator/kvm/id_ed25519" || k.DiskSizeGB != 20 {
t.Errorf("kvm = %+v, want sshKeyFile and diskSizeGb parsed", k)
}
if got := k.Images["debian-13"].SHA256; !strings.HasPrefix(got, "0123") {
t.Errorf("images[debian-13].sha256 = %q", got)
}
if k.Images["ubuntu"].Format != "raw" {
t.Errorf("images[ubuntu].format = %q, want raw", k.Images["ubuntu"].Format)
}
if mt := k.MachineTypes["medium"]; mt.VCPUs != 2 || mt.MemoryMiB != 2048 {
t.Errorf("machineTypes[medium] = %+v, want {2 2048}", mt)
}
if len(k.Hosts) != 2 || k.Hosts[1].Socket != "/run/libvirt/virtqemud-sock" || k.Hosts[1].Pool != "fast" {
t.Errorf("hosts = %+v, want two hosts with lab2's socket and pool parsed", k.Hosts)
}
}
func TestLoadConfig_kvmInsecureHostKeyNeedsNoKnownHosts(t *testing.T) {
t.Parallel()
yaml := strings.Replace(validKVMYAML,
"knownHostsFile: /etc/proxy-operator/kvm/known_hosts",
"insecureIgnoreHostKey: true", 1)
if _, err := LoadConfig([]byte(yaml)); err != nil {
t.Fatalf("LoadConfig() error = %v, want nil", err)
}
}
func TestLoadConfig_kvmInvalid(t *testing.T) {
t.Parallel()
minimal := func(kvmBody string) string {
return "providers:\n - name: kvm-home\n type: kvm\n kvm:\n" + indent(kvmBody, 6)
}
hostsAndTypes := `
machineTypes:
small: {vcpus: 1, memoryMiB: 1024}
hosts:
- {name: nas1, address: nas1:22, user: virt, bridge: br0}
`
tests := []struct {
name string
yaml string
wantErrSub string
}{
{
name: "missing kvm block",
yaml: "providers:\n - name: kvm-home\n type: kvm\n",
wantErrSub: "kvm block is required",
},
{
name: "missing sshKeyFile",
yaml: minimal("knownHostsFile: /kh\n" + hostsAndTypes),
wantErrSub: "kvm.sshKeyFile is required",
},
{
name: "missing knownHostsFile without insecure flag",
yaml: minimal("sshKeyFile: /key\n" + hostsAndTypes),
wantErrSub: "kvm.knownHostsFile is required unless",
},
{
name: "negative disk size",
yaml: minimal("sshKeyFile: /key\nknownHostsFile: /kh\ndiskSizeGb: -1\n" + hostsAndTypes),
wantErrSub: "diskSizeGb must not be negative",
},
{
name: "no machine types",
yaml: minimal("sshKeyFile: /key\nknownHostsFile: /kh\nhosts:\n - {name: nas1, address: nas1:22, user: virt, bridge: br0}\n"),
wantErrSub: "machineTypes must define at least one",
},
{
name: "machine type with zero memory",
yaml: minimal("sshKeyFile: /key\nknownHostsFile: /kh\nmachineTypes:\n small: {vcpus: 1}\nhosts:\n - {name: nas1, address: nas1:22, user: virt, bridge: br0}\n"),
wantErrSub: `machineTypes["small"] needs vcpus > 0 and memoryMiB > 0`,
},
{
name: "no hosts",
yaml: minimal("sshKeyFile: /key\nknownHostsFile: /kh\nmachineTypes:\n small: {vcpus: 1, memoryMiB: 1024}\n"),
wantErrSub: "kvm.hosts must list at least one host",
},
{
name: "duplicate host name",
yaml: minimal("sshKeyFile: /key\nknownHostsFile: /kh\nmachineTypes:\n small: {vcpus: 1, memoryMiB: 1024}\nhosts:\n - {name: nas1, address: a:22, user: virt, bridge: br0}\n - {name: nas1, address: b:22, user: virt, bridge: br0}\n"),
wantErrSub: `duplicate host name "nas1"`,
},
{
name: "host missing address",
yaml: minimal("sshKeyFile: /key\nknownHostsFile: /kh\nmachineTypes:\n small: {vcpus: 1, memoryMiB: 1024}\nhosts:\n - {name: nas1, user: virt, bridge: br0}\n"),
wantErrSub: "address is required",
},
{
name: "host missing user",
yaml: minimal("sshKeyFile: /key\nknownHostsFile: /kh\nmachineTypes:\n small: {vcpus: 1, memoryMiB: 1024}\nhosts:\n - {name: nas1, address: a:22, bridge: br0}\n"),
wantErrSub: "user is required",
},
{
name: "host missing bridge",
yaml: minimal("sshKeyFile: /key\nknownHostsFile: /kh\nmachineTypes:\n small: {vcpus: 1, memoryMiB: 1024}\nhosts:\n - {name: nas1, address: a:22, user: virt}\n"),
wantErrSub: "bridge is required",
},
{
name: "image with non-http url",
yaml: minimal("sshKeyFile: /key\nknownHostsFile: /kh\nimages:\n d: {url: file:///tmp/x.qcow2}\n" + hostsAndTypes),
wantErrSub: `images["d"].url must be an http(s) URL`,
},
{
name: "image with bad sha256",
yaml: minimal("sshKeyFile: /key\nknownHostsFile: /kh\nimages:\n d: {url: https://x/y.qcow2, sha256: abc}\n" + hostsAndTypes),
wantErrSub: `images["d"].sha256 must be 64 hex characters`,
},
{
name: "image with unknown format",
yaml: minimal("sshKeyFile: /key\nknownHostsFile: /kh\nimages:\n d: {url: https://x/y.vmdk, format: vmdk}\n" + hostsAndTypes),
wantErrSub: `images["d"].format must be qcow2 or raw`,
},
{
name: "kvm type with gcp block",
yaml: `
providers:
- name: p1
type: kvm
gcp:
project: my-project`,
wantErrSub: "type is kvm but a gcp block is set",
},
{
name: "gcp type with kvm block",
yaml: `
providers:
- name: p1
type: gcp
gcp:
project: my-project
kvm:
sshKeyFile: /key`,
wantErrSub: "type is gcp but a kvm block is set",
},
}
for _, tc := range tests {
t.Run(tc.name, func(t *testing.T) {
t.Parallel()
_, err := LoadConfig([]byte(tc.yaml))
if err == nil {
t.Fatalf("LoadConfig() error = nil, want error containing %q", tc.wantErrSub)
}
if !strings.Contains(err.Error(), tc.wantErrSub) {
t.Errorf("LoadConfig() error = %q, want substring %q", err.Error(), tc.wantErrSub)
}
})
}
}
func indent(s string, n int) string {
pad := strings.Repeat(" ", n)
lines := strings.Split(strings.TrimLeft(s, "\n"), "\n")
for i, l := range lines {
if l != "" {
lines[i] = pad + l
}
}
return strings.Join(lines, "\n")
}
+301
View File
@@ -0,0 +1,301 @@
package kvm
import (
"context"
"fmt"
"io"
"net"
"os"
"sync"
"time"
libvirt "github.com/digitalocean/go-libvirt"
"golang.org/x/crypto/ssh"
"golang.org/x/crypto/ssh/knownhosts"
"gitea.home.hrajfrisbee.cz/kacerr/egress-proxies-operator/internal/provider"
)
const (
defaultHandshakeTimeout = 30 * time.Second
keepaliveInterval = 15 * time.Second
)
// libvirtAPI is the slice of *libvirt.Libvirt the provider uses, and the
// test seam: the fake in kvm_test.go implements exactly this.
type libvirtAPI interface {
DomainLookupByName(name string) (libvirt.Domain, error)
DomainDefineXML(xml string) (libvirt.Domain, error)
DomainCreate(d libvirt.Domain) error
DomainDestroy(d libvirt.Domain) error
DomainUndefineFlags(d libvirt.Domain, flags libvirt.DomainUndefineFlagsValues) error
DomainGetState(d libvirt.Domain, flags uint32) (state int32, reason int32, err error)
DomainGetMetadata(d libvirt.Domain, typ int32, uri libvirt.OptString, flags libvirt.DomainModificationImpact) (string, error)
DomainInterfaceAddresses(d libvirt.Domain, source uint32, flags uint32) ([]libvirt.DomainInterface, error)
ConnectListAllDomains(needResults int32, flags libvirt.ConnectListAllDomainsFlags) ([]libvirt.Domain, uint32, error)
StoragePoolLookupByName(name string) (libvirt.StoragePool, error)
StoragePoolRefresh(pool libvirt.StoragePool, flags uint32) error
StorageVolLookupByName(pool libvirt.StoragePool, name string) (libvirt.StorageVol, error)
StorageVolGetPath(vol libvirt.StorageVol) (string, error)
StorageVolCreateXML(pool libvirt.StoragePool, xml string, flags libvirt.StorageVolCreateFlags) (libvirt.StorageVol, error)
StorageVolUpload(vol libvirt.StorageVol, r io.Reader, offset, length uint64, flags libvirt.StorageVolUploadFlags) error
StorageVolDownload(vol libvirt.StorageVol, w io.Writer, offset, length uint64, flags libvirt.StorageVolDownloadFlags) error
StorageVolDelete(vol libvirt.StorageVol, flags libvirt.StorageVolDeleteFlags) error
}
// hostClient is one live connection to a hypervisor.
type hostClient struct {
api libvirtAPI
// done is closed once the connection is known to be lost. A nil channel
// never fires, which suits fakes.
done <-chan struct{}
close func()
}
func (c *hostClient) alive() bool {
select {
case <-c.done:
return false
default:
return true
}
}
// hostDialer opens a connection to one host. The SSH implementation is the
// real one; tests inject their own.
type hostDialer func(ctx context.Context, host provider.KVMHost) (*hostClient, error)
// hostPool caches one connection per host, dialing lazily and redialing
// after the connection drops. Dials for different hosts don't block each
// other; concurrent callers for the same host share a single dial.
type hostPool struct {
dial hostDialer
mu sync.Mutex
entries map[string]*hostEntry
}
type hostEntry struct {
mu sync.Mutex
client *hostClient
}
func newHostPool(dial hostDialer) *hostPool {
return &hostPool{dial: dial, entries: make(map[string]*hostEntry)}
}
func (p *hostPool) entry(name string) *hostEntry {
p.mu.Lock()
defer p.mu.Unlock()
e, ok := p.entries[name]
if !ok {
e = &hostEntry{}
p.entries[name] = e
}
return e
}
func (p *hostPool) get(ctx context.Context, host provider.KVMHost) (*hostClient, error) {
e := p.entry(host.Name)
e.mu.Lock()
defer e.mu.Unlock()
if e.client != nil {
if e.client.alive() {
return e.client, nil
}
e.client.close()
e.client = nil
}
c, err := p.dial(ctx, host)
if err != nil {
return nil, fmt.Errorf("connecting to host %s (%s): %w", host.Name, host.Address, err)
}
e.client = c
return c, nil
}
// drop closes and forgets c if it is still the cached client for host, so a
// connection that was already replaced by a redial is left alone.
func (p *hostPool) drop(host string, c *hostClient) {
e := p.entry(host)
e.mu.Lock()
defer e.mu.Unlock()
if e.client == c {
e.client = nil
}
c.close()
}
// call runs fn against host's connection. go-libvirt RPCs take no context
// and block until the daemon answers, so when ctx ends first the connection
// is closed — which unblocks fn with an error — and ctx.Err() is returned.
// A transport error (anything that is not a libvirt.Error) also drops the
// cached connection so the next call redials instead of reusing a dead
// link.
func (p *hostPool) call(ctx context.Context, host provider.KVMHost, fn func(api libvirtAPI) error) error {
c, err := p.get(ctx, host)
if err != nil {
return err
}
err = awaitOrCancel(ctx, func() error { return fn(c.api) }, func() { p.drop(host.Name, c) })
if err != nil && !isLibvirtError(err) {
p.drop(host.Name, c)
}
return err
}
// awaitOrCancel runs fn in the background and returns its result, unless ctx
// ends first: then abort is invoked (it must make fn return eventually, e.g.
// by closing the connection fn is blocked on) and ctx.Err() is returned. The
// abandoned fn's result is discarded.
func awaitOrCancel(ctx context.Context, fn func() error, abort func()) error {
errc := make(chan error, 1)
go func() { errc <- fn() }()
select {
case err := <-errc:
return err
case <-ctx.Done():
abort()
return ctx.Err()
}
}
// sshAuth holds the pieces of the SSH client config shared by every host:
// the private key and the host-key policy, both loaded once at startup so a
// missing or unreadable file fails the operator fast rather than on the
// first Create.
type sshAuth struct {
signer ssh.Signer
hostKey ssh.HostKeyCallback
}
func newSSHAuth(cfg provider.KVMConfig) (*sshAuth, error) {
key, err := os.ReadFile(cfg.SSHKeyFile)
if err != nil {
return nil, fmt.Errorf("reading kvm.sshKeyFile: %w", err)
}
signer, err := ssh.ParsePrivateKey(key)
if err != nil {
return nil, fmt.Errorf("parsing kvm.sshKeyFile: %w", err)
}
hostKey := ssh.InsecureIgnoreHostKey() //nolint:gosec // opt-in via config, documented as insecure
if !cfg.InsecureIgnoreHostKey {
hostKey, err = knownhosts.New(cfg.KnownHostsFile)
if err != nil {
return nil, fmt.Errorf("loading kvm.knownHostsFile: %w", err)
}
}
return &sshAuth{signer: signer, hostKey: hostKey}, nil
}
// dialer returns the production hostDialer: SSH to the host, forward its
// libvirt unix socket through the connection, and open a libvirt session
// over that.
func (a *sshAuth) dialer() hostDialer {
return func(ctx context.Context, host provider.KVMHost) (*hostClient, error) {
client, err := a.dialSSH(ctx, host)
if err != nil {
return nil, err
}
hc, err := connectLibvirt(ctx, client, host.Socket)
if err != nil {
client.Close()
return nil, err
}
return hc, nil
}
}
// dialSSH establishes the SSH connection. The handshake is bounded by ctx's
// deadline, or defaultHandshakeTimeout when ctx has none, since a
// half-open hypervisor must not pin a reconcile worker indefinitely.
func (a *sshAuth) dialSSH(ctx context.Context, host provider.KVMHost) (*ssh.Client, error) {
deadline, ok := ctx.Deadline()
if !ok {
deadline = time.Now().Add(defaultHandshakeTimeout)
}
var d net.Dialer
nc, err := d.DialContext(ctx, "tcp", host.Address)
if err != nil {
return nil, fmt.Errorf("ssh dial: %w", err)
}
_ = nc.SetDeadline(deadline)
conn, chans, reqs, err := ssh.NewClientConn(nc, host.Address, &ssh.ClientConfig{
User: host.User,
Auth: []ssh.AuthMethod{ssh.PublicKeys(a.signer)},
HostKeyCallback: a.hostKey,
})
if err != nil {
nc.Close()
return nil, fmt.Errorf("ssh handshake: %w", err)
}
_ = nc.SetDeadline(time.Time{})
return ssh.NewClient(conn, chans, reqs), nil
}
// unixForward adapts an SSH client to go-libvirt's socket.Dialer: each Dial
// opens a direct-streamlocal channel to the libvirt socket on the host.
type unixForward struct {
client *ssh.Client
path string
}
func (u unixForward) Dial() (net.Conn, error) {
return u.client.Dial("unix", u.path)
}
// connectLibvirt opens the libvirt session over the SSH client and starts a
// keepalive loop so a dead link is noticed within roughly two intervals
// rather than at the TCP stack's leisure. Closing the returned client tears
// down the SSH connection, which in turn ends the libvirt session.
func connectLibvirt(ctx context.Context, client *ssh.Client, socket string) (*hostClient, error) {
lv := libvirt.NewWithDialer(unixForward{client: client, path: socket})
if err := awaitOrCancel(ctx, lv.Connect, func() { client.Close() }); err != nil {
return nil, fmt.Errorf("libvirt connect via %s: %w", socket, err)
}
stop := make(chan struct{})
var once sync.Once
go keepalive(client, keepaliveInterval, stop)
return &hostClient{
api: lv,
done: lv.Disconnected(),
close: func() {
once.Do(func() {
close(stop)
client.Close()
})
},
}, nil
}
// keepalive sends OpenSSH keepalive requests until stop closes; a failed or
// unanswered request closes the client, which surfaces as Disconnected on
// the libvirt side and makes the pool redial.
func keepalive(client *ssh.Client, interval time.Duration, stop <-chan struct{}) {
t := time.NewTicker(interval)
defer t.Stop()
for {
select {
case <-stop:
return
case <-t.C:
}
replied := make(chan error, 1)
go func() {
_, _, err := client.SendRequest("keepalive@openssh.com", true, nil)
replied <- err
}()
select {
case <-stop:
return
case err := <-replied:
if err != nil {
client.Close()
return
}
case <-time.After(interval):
client.Close()
return
}
}
}
+569
View File
@@ -0,0 +1,569 @@
package kvm
import (
"bytes"
"context"
"crypto/ed25519"
"crypto/rand"
"encoding/pem"
"errors"
"io"
"net"
"os"
"path/filepath"
"strings"
"sync"
"sync/atomic"
"testing"
"time"
libvirt "github.com/digitalocean/go-libvirt"
"golang.org/x/crypto/ssh"
"golang.org/x/crypto/ssh/knownhosts"
"gitea.home.hrajfrisbee.cz/kacerr/egress-proxies-operator/internal/provider"
)
// --- hostPool over a fake dialer ---
type fakeClient struct {
done chan struct{}
closed atomic.Bool
}
func (c *fakeClient) close() {
if c.closed.CompareAndSwap(false, true) {
close(c.done)
}
}
type fakeDialer struct {
mu sync.Mutex
dials int
err error
delay time.Duration
clients []*fakeClient
}
func (d *fakeDialer) dial(ctx context.Context, _ provider.KVMHost) (*hostClient, error) {
if d.delay > 0 {
select {
case <-time.After(d.delay):
case <-ctx.Done():
return nil, ctx.Err()
}
}
d.mu.Lock()
defer d.mu.Unlock()
d.dials++
if d.err != nil {
return nil, d.err
}
fc := &fakeClient{done: make(chan struct{})}
d.clients = append(d.clients, fc)
return &hostClient{api: nil, done: fc.done, close: fc.close}, nil
}
func (d *fakeDialer) dialCount() int {
d.mu.Lock()
defer d.mu.Unlock()
return d.dials
}
var testHost = provider.KVMHost{Name: "nas1", Address: "nas1:22", User: "virt", Socket: defaultSocket, Pool: "default", Bridge: "br0"}
func noop(libvirtAPI) error { return nil }
func TestHostPool_call_reusesConnection(t *testing.T) {
t.Parallel()
d := &fakeDialer{}
p := newHostPool(d.dial)
for range 3 {
if err := p.call(context.Background(), testHost, noop); err != nil {
t.Fatalf("call() error = %v", err)
}
}
if d.dialCount() != 1 {
t.Errorf("dials = %d, want 1", d.dialCount())
}
}
func TestHostPool_call_redialsAfterDisconnect(t *testing.T) {
t.Parallel()
d := &fakeDialer{}
p := newHostPool(d.dial)
if err := p.call(context.Background(), testHost, noop); err != nil {
t.Fatalf("call() error = %v", err)
}
d.clients[0].close()
if err := p.call(context.Background(), testHost, noop); err != nil {
t.Fatalf("second call() error = %v", err)
}
if d.dialCount() != 2 {
t.Errorf("dials = %d, want 2 after the first connection dropped", d.dialCount())
}
}
func TestHostPool_call_transportErrorDropsConnection(t *testing.T) {
t.Parallel()
d := &fakeDialer{}
p := newHostPool(d.dial)
err := p.call(context.Background(), testHost, func(libvirtAPI) error { return io.ErrUnexpectedEOF })
if !errors.Is(err, io.ErrUnexpectedEOF) {
t.Fatalf("call() error = %v, want the transport error passed through", err)
}
if !d.clients[0].closed.Load() {
t.Error("connection not closed after a transport error")
}
if err := p.call(context.Background(), testHost, noop); err != nil {
t.Fatalf("call() after transport error = %v", err)
}
if d.dialCount() != 2 {
t.Errorf("dials = %d, want 2", d.dialCount())
}
}
func TestHostPool_call_libvirtErrorKeepsConnection(t *testing.T) {
t.Parallel()
d := &fakeDialer{}
p := newHostPool(d.dial)
err := p.call(context.Background(), testHost, func(libvirtAPI) error { return lvErr(libvirt.ErrNoDomain) })
if !hasCode(err, libvirt.ErrNoDomain) {
t.Fatalf("call() error = %v, want the libvirt error passed through", err)
}
if err := p.call(context.Background(), testHost, noop); err != nil {
t.Fatalf("call() error = %v", err)
}
if d.dialCount() != 1 {
t.Errorf("dials = %d, want 1: a daemon-reported error must not cost the connection", d.dialCount())
}
if d.clients[0].closed.Load() {
t.Error("connection closed after a libvirt error")
}
}
func TestHostPool_call_dialErrorPropagates(t *testing.T) {
t.Parallel()
d := &fakeDialer{err: errors.New("ssh: handshake failed")}
p := newHostPool(d.dial)
err := p.call(context.Background(), testHost, noop)
if err == nil || !strings.Contains(err.Error(), "nas1") || !strings.Contains(err.Error(), "handshake failed") {
t.Errorf("call() error = %v, want the dial error naming the host", err)
}
}
func TestHostPool_call_cancelAbandonsBlockedCall(t *testing.T) {
t.Parallel()
d := &fakeDialer{}
p := newHostPool(d.dial)
ctx, cancel := context.WithCancel(context.Background())
released := make(chan struct{})
go func() {
time.Sleep(20 * time.Millisecond)
cancel()
}()
err := p.call(ctx, testHost, func(libvirtAPI) error {
// Simulates an RPC stuck on a hung hypervisor: it only returns once
// the pool closes the connection out from under it.
<-d.clients[0].done
close(released)
return io.EOF
})
if !errors.Is(err, context.Canceled) {
t.Fatalf("call() error = %v, want context.Canceled", err)
}
select {
case <-released:
case <-time.After(time.Second):
t.Fatal("blocked call was not released by closing the connection")
}
if err := p.call(context.Background(), testHost, noop); err != nil {
t.Fatalf("call() after cancel = %v", err)
}
if d.dialCount() != 2 {
t.Errorf("dials = %d, want 2", d.dialCount())
}
}
func TestHostPool_call_concurrentCallersShareOneDial(t *testing.T) {
t.Parallel()
d := &fakeDialer{delay: 30 * time.Millisecond}
p := newHostPool(d.dial)
var wg sync.WaitGroup
for range 8 {
wg.Go(func() {
if err := p.call(context.Background(), testHost, noop); err != nil {
t.Errorf("call() error = %v", err)
}
})
}
wg.Wait()
if d.dialCount() != 1 {
t.Errorf("dials = %d, want 1", d.dialCount())
}
}
func TestHostPool_call_hostsAreIndependent(t *testing.T) {
t.Parallel()
d := &fakeDialer{}
p := newHostPool(d.dial)
other := testHost
other.Name = "lab2"
for _, h := range []provider.KVMHost{testHost, other, testHost, other} {
if err := p.call(context.Background(), h, noop); err != nil {
t.Fatalf("call(%s) error = %v", h.Name, err)
}
}
if d.dialCount() != 2 {
t.Errorf("dials = %d, want one per host", d.dialCount())
}
}
// --- SSH dial against an in-process server ---
func newKeyPair(t *testing.T) (ssh.Signer, []byte) {
t.Helper()
_, priv, err := ed25519.GenerateKey(rand.Reader)
if err != nil {
t.Fatal(err)
}
signer, err := ssh.NewSignerFromKey(priv)
if err != nil {
t.Fatal(err)
}
block, err := ssh.MarshalPrivateKey(priv, "")
if err != nil {
t.Fatal(err)
}
return signer, pem.EncodeToMemory(block)
}
// testSSHServer accepts one public key and serves
// direct-streamlocal@openssh.com channels by dialing the requested unix
// path, which is exactly what OpenSSH does for ssh.Client.Dial("unix").
type testSSHServer struct {
addr string
hostKey ssh.PublicKey
mu sync.Mutex
paths []string
}
func startSSHServer(t *testing.T, clientKey ssh.PublicKey, answerGlobalRequests bool) *testSSHServer {
t.Helper()
hostSigner, _ := newKeyPair(t)
cfg := &ssh.ServerConfig{
PublicKeyCallback: func(_ ssh.ConnMetadata, key ssh.PublicKey) (*ssh.Permissions, error) {
if bytes.Equal(key.Marshal(), clientKey.Marshal()) {
return nil, nil
}
return nil, errors.New("unknown public key")
},
}
cfg.AddHostKey(hostSigner)
ln, err := net.Listen("tcp", "127.0.0.1:0")
if err != nil {
t.Fatal(err)
}
t.Cleanup(func() { ln.Close() })
s := &testSSHServer{addr: ln.Addr().String(), hostKey: hostSigner.PublicKey()}
go func() {
for {
nc, err := ln.Accept()
if err != nil {
return
}
go s.serve(nc, cfg, answerGlobalRequests)
}
}()
return s
}
func (s *testSSHServer) serve(nc net.Conn, cfg *ssh.ServerConfig, answerGlobalRequests bool) {
_, chans, reqs, err := ssh.NewServerConn(nc, cfg)
if err != nil {
return
}
if answerGlobalRequests {
go ssh.DiscardRequests(reqs)
} else {
go func() {
for range reqs { //nolint:revive // deliberately never replies, to stall keepalives
}
}()
}
for newCh := range chans {
if newCh.ChannelType() != "direct-streamlocal@openssh.com" {
_ = newCh.Reject(ssh.UnknownChannelType, "test server only forwards unix sockets")
continue
}
var payload struct {
Path string
Reserved0 string
Reserved1 uint32
}
if err := ssh.Unmarshal(newCh.ExtraData(), &payload); err != nil {
_ = newCh.Reject(ssh.ConnectionFailed, err.Error())
continue
}
s.mu.Lock()
s.paths = append(s.paths, payload.Path)
s.mu.Unlock()
target, err := net.Dial("unix", payload.Path)
if err != nil {
_ = newCh.Reject(ssh.ConnectionFailed, err.Error())
continue
}
ch, chReqs, err := newCh.Accept()
if err != nil {
target.Close()
continue
}
go ssh.DiscardRequests(chReqs)
go func() { _, _ = io.Copy(ch, target); ch.Close() }()
go func() { _, _ = io.Copy(target, ch); target.Close() }()
}
}
func (s *testSSHServer) requestedPaths() []string {
s.mu.Lock()
defer s.mu.Unlock()
return append([]string(nil), s.paths...)
}
// startEchoSocket serves a unix socket that echoes everything back, standing
// in for libvirt-sock.
func startEchoSocket(t *testing.T) string {
t.Helper()
path := filepath.Join(t.TempDir(), "libvirt-sock")
ln, err := net.Listen("unix", path)
if err != nil {
t.Fatal(err)
}
t.Cleanup(func() { ln.Close() })
go func() {
for {
c, err := ln.Accept()
if err != nil {
return
}
go func() { _, _ = io.Copy(c, c); c.Close() }()
}
}()
return path
}
type sshFixture struct {
server *testSSHServer
cfg provider.KVMConfig
host provider.KVMHost
keyFile string
}
func newSSHFixture(t *testing.T, answerGlobalRequests bool) *sshFixture {
t.Helper()
clientSigner, clientPEM := newKeyPair(t)
server := startSSHServer(t, clientSigner.PublicKey(), answerGlobalRequests)
dir := t.TempDir()
keyFile := filepath.Join(dir, "id_ed25519")
if err := os.WriteFile(keyFile, clientPEM, 0o600); err != nil {
t.Fatal(err)
}
khFile := filepath.Join(dir, "known_hosts")
line := knownhosts.Line([]string{server.addr}, server.hostKey)
if err := os.WriteFile(khFile, []byte(line+"\n"), 0o600); err != nil {
t.Fatal(err)
}
return &sshFixture{
server: server,
keyFile: keyFile,
cfg: provider.KVMConfig{SSHKeyFile: keyFile, KnownHostsFile: khFile},
host: provider.KVMHost{Name: "nas1", Address: server.addr, User: "virt", Socket: startEchoSocket(t)},
}
}
func TestSSHAuth_dialSSH_forwardsUnixSocket(t *testing.T) {
t.Parallel()
f := newSSHFixture(t, true)
auth, err := newSSHAuth(f.cfg)
if err != nil {
t.Fatalf("newSSHAuth() error = %v", err)
}
ctx, cancel := context.WithTimeout(context.Background(), 5*time.Second)
defer cancel()
client, err := auth.dialSSH(ctx, f.host)
if err != nil {
t.Fatalf("dialSSH() error = %v", err)
}
defer client.Close()
conn, err := unixForward{client: client, path: f.host.Socket}.Dial()
if err != nil {
t.Fatalf("unixForward.Dial() error = %v", err)
}
defer conn.Close()
if _, err := conn.Write([]byte("ping")); err != nil {
t.Fatal(err)
}
buf := make([]byte, 4)
if _, err := io.ReadFull(conn, buf); err != nil || string(buf) != "ping" {
t.Fatalf("echo through the forward = %q, %v; want ping", buf, err)
}
if paths := f.server.requestedPaths(); len(paths) != 1 || paths[0] != f.host.Socket {
t.Errorf("server saw forward requests %v, want [%s]", paths, f.host.Socket)
}
}
func TestSSHAuth_dialSSH_rejectsUnknownHostKey(t *testing.T) {
t.Parallel()
f := newSSHFixture(t, true)
impostor, _ := newKeyPair(t)
line := knownhosts.Line([]string{f.server.addr}, impostor.PublicKey())
if err := os.WriteFile(f.cfg.KnownHostsFile, []byte(line+"\n"), 0o600); err != nil {
t.Fatal(err)
}
auth, err := newSSHAuth(f.cfg)
if err != nil {
t.Fatalf("newSSHAuth() error = %v", err)
}
client, err := auth.dialSSH(context.Background(), f.host)
if err == nil {
client.Close()
t.Fatal("dialSSH() succeeded against a host whose key is not in known_hosts")
}
var keyErr *knownhosts.KeyError
if !errors.As(err, &keyErr) {
t.Errorf("dialSSH() error = %v, want a knownhosts.KeyError", err)
}
}
func TestSSHAuth_dialSSH_insecureIgnoreHostKey(t *testing.T) {
t.Parallel()
f := newSSHFixture(t, true)
f.cfg.KnownHostsFile = ""
f.cfg.InsecureIgnoreHostKey = true
auth, err := newSSHAuth(f.cfg)
if err != nil {
t.Fatalf("newSSHAuth() error = %v", err)
}
client, err := auth.dialSSH(context.Background(), f.host)
if err != nil {
t.Fatalf("dialSSH() error = %v", err)
}
client.Close()
}
func TestSSHAuth_dialSSH_rejectsWrongClientKey(t *testing.T) {
t.Parallel()
f := newSSHFixture(t, true)
_, otherPEM := newKeyPair(t)
if err := os.WriteFile(f.keyFile, otherPEM, 0o600); err != nil {
t.Fatal(err)
}
auth, err := newSSHAuth(f.cfg)
if err != nil {
t.Fatalf("newSSHAuth() error = %v", err)
}
client, err := auth.dialSSH(context.Background(), f.host)
if err == nil {
client.Close()
t.Fatal("dialSSH() succeeded with a key the server does not know")
}
if !strings.Contains(err.Error(), "ssh handshake") {
t.Errorf("dialSSH() error = %v, want an auth failure from the handshake", err)
}
}
func TestSSHAuth_dialSSH_cancelledContext(t *testing.T) {
t.Parallel()
f := newSSHFixture(t, true)
auth, err := newSSHAuth(f.cfg)
if err != nil {
t.Fatalf("newSSHAuth() error = %v", err)
}
ctx, cancel := context.WithCancel(context.Background())
cancel()
if client, err := auth.dialSSH(ctx, f.host); err == nil {
client.Close()
t.Fatal("dialSSH() succeeded with an already-cancelled context")
}
}
func TestNewSSHAuth_missingFiles(t *testing.T) {
t.Parallel()
_, keyPEM := newKeyPair(t)
keyFile := filepath.Join(t.TempDir(), "key")
if err := os.WriteFile(keyFile, keyPEM, 0o600); err != nil {
t.Fatal(err)
}
tests := []struct {
name string
cfg provider.KVMConfig
wantSub string
}{
{"key file missing", provider.KVMConfig{SSHKeyFile: "/nonexistent/key", InsecureIgnoreHostKey: true}, "sshKeyFile"},
{"key file garbage", provider.KVMConfig{SSHKeyFile: keyFile + ".bad", InsecureIgnoreHostKey: true}, "sshKeyFile"},
{"known_hosts missing", provider.KVMConfig{SSHKeyFile: keyFile, KnownHostsFile: "/nonexistent/kh"}, "knownHostsFile"},
}
if err := os.WriteFile(keyFile+".bad", []byte("not a key"), 0o600); err != nil {
t.Fatal(err)
}
for _, tc := range tests {
t.Run(tc.name, func(t *testing.T) {
t.Parallel()
_, err := newSSHAuth(tc.cfg)
if err == nil || !strings.Contains(err.Error(), tc.wantSub) {
t.Errorf("newSSHAuth() error = %v, want one mentioning %s", err, tc.wantSub)
}
})
}
}
func TestKeepalive_closesClientWhenServerStalls(t *testing.T) {
t.Parallel()
f := newSSHFixture(t, false)
auth, err := newSSHAuth(f.cfg)
if err != nil {
t.Fatalf("newSSHAuth() error = %v", err)
}
client, err := auth.dialSSH(context.Background(), f.host)
if err != nil {
t.Fatalf("dialSSH() error = %v", err)
}
defer client.Close()
stop := make(chan struct{})
defer close(stop)
go keepalive(client, 20*time.Millisecond, stop)
closed := make(chan struct{})
go func() { _ = client.Wait(); close(closed) }()
select {
case <-closed:
case <-time.After(2 * time.Second):
t.Fatal("client still open after keepalives went unanswered")
}
}
func TestKeepalive_keepsHealthyClientOpen(t *testing.T) {
t.Parallel()
f := newSSHFixture(t, true)
auth, err := newSSHAuth(f.cfg)
if err != nil {
t.Fatalf("newSSHAuth() error = %v", err)
}
client, err := auth.dialSSH(context.Background(), f.host)
if err != nil {
t.Fatalf("dialSSH() error = %v", err)
}
defer client.Close()
stop := make(chan struct{})
defer close(stop)
go keepalive(client, 10*time.Millisecond, stop)
time.Sleep(60 * time.Millisecond)
fwd := unixForward{client: client, path: f.host.Socket}
conn, err := fwd.Dial()
if err != nil {
t.Fatalf("forward after several keepalives failed: %v", err)
}
conn.Close()
}
+224
View File
@@ -0,0 +1,224 @@
package kvm
import (
"encoding/xml"
"fmt"
"time"
)
const guestAgentChannel = "org.qemu.guest_agent.0"
// domainParams is everything buildDomainXML needs, already resolved from the
// config and placement so the builder stays a pure function.
type domainParams struct {
Name string
UID string
CreatedAt time.Time
VCPUs int32
MemoryMiB int64
Pool string
Bridge string
DiskVol string
SeedVol string
}
// buildDomainXML renders a q35 KVM guest: host-passthrough CPU, a virtio
// overlay disk and a SATA cdrom for the seed (both referenced as pool
// volumes so no host paths are needed), one virtio NIC on the host bridge,
// a serial console, the guest-agent channel, a virtio RNG, no graphics, and
// the operator's metadata element.
func buildDomainXML(p domainParams) (string, error) {
d := domainXML{
Type: "kvm",
Name: p.Name,
Metadata: domainMetadataXML{Instance: newInstanceMetadata(p.UID, p.CreatedAt)},
Memory: sizeXML{Unit: "MiB", Value: p.MemoryMiB},
VCPU: p.VCPUs,
OS: osXML{
Type: osTypeXML{Arch: "x86_64", Machine: "q35", Value: "hvm"},
Boot: bootXML{Dev: "hd"},
},
Features: featuresXML{ACPI: &struct{}{}, APIC: &struct{}{}},
CPU: cpuXML{Mode: "host-passthrough"},
Clock: clockXML{Offset: "utc"},
OnPoweroff: "destroy",
OnReboot: "restart",
OnCrash: "destroy",
Devices: devicesXML{
Disks: []diskXML{
{
Type: "volume",
Device: "disk",
Driver: driverXML{Name: "qemu", Type: formatQcow2},
Source: diskSourceXML{Pool: p.Pool, Volume: p.DiskVol},
Target: targetXML{Dev: "vda", Bus: "virtio"},
},
{
Type: "volume",
Device: "cdrom",
Driver: driverXML{Name: "qemu", Type: formatRaw},
Source: diskSourceXML{Pool: p.Pool, Volume: p.SeedVol},
Target: targetXML{Dev: "sda", Bus: "sata"},
ReadOnly: &struct{}{},
},
},
Interfaces: []interfaceXML{{
Type: "bridge",
Source: ifaceSourceXML{Bridge: p.Bridge},
Model: modelXML{Type: "virtio"},
}},
Serials: []serialXML{{Type: "pty", Target: portXML{Port: 0}}},
Consoles: []consoleXML{{Type: "pty", Target: consoleTargetXML{Type: "serial", Port: 0}}},
Channels: []channelXML{{
Type: "unix",
Target: channelTargetXML{Type: "virtio", Name: guestAgentChannel},
}},
RNG: rngXML{Model: "virtio", Backend: rngBackendXML{Model: "random", Value: "/dev/urandom"}},
},
}
out, err := xml.Marshal(d)
if err != nil {
return "", fmt.Errorf("marshalling domain xml: %w", err)
}
return string(out), nil
}
type domainXML struct {
XMLName xml.Name `xml:"domain"`
Type string `xml:"type,attr"`
Name string `xml:"name"`
Metadata domainMetadataXML `xml:"metadata"`
Memory sizeXML `xml:"memory"`
VCPU int32 `xml:"vcpu"`
OS osXML `xml:"os"`
Features featuresXML `xml:"features"`
CPU cpuXML `xml:"cpu"`
Clock clockXML `xml:"clock"`
OnPoweroff string `xml:"on_poweroff"`
OnReboot string `xml:"on_reboot"`
OnCrash string `xml:"on_crash"`
Devices devicesXML `xml:"devices"`
}
type domainMetadataXML struct {
Instance instanceMetadata
}
type sizeXML struct {
Unit string `xml:"unit,attr"`
Value int64 `xml:",chardata"`
}
type osXML struct {
Type osTypeXML `xml:"type"`
Boot bootXML `xml:"boot"`
}
type osTypeXML struct {
Arch string `xml:"arch,attr"`
Machine string `xml:"machine,attr"`
Value string `xml:",chardata"`
}
type bootXML struct {
Dev string `xml:"dev,attr"`
}
type featuresXML struct {
ACPI *struct{} `xml:"acpi"`
APIC *struct{} `xml:"apic"`
}
type cpuXML struct {
Mode string `xml:"mode,attr"`
}
type clockXML struct {
Offset string `xml:"offset,attr"`
}
type devicesXML struct {
Disks []diskXML `xml:"disk"`
Interfaces []interfaceXML `xml:"interface"`
Serials []serialXML `xml:"serial"`
Consoles []consoleXML `xml:"console"`
Channels []channelXML `xml:"channel"`
RNG rngXML `xml:"rng"`
}
type diskXML struct {
Type string `xml:"type,attr"`
Device string `xml:"device,attr"`
Driver driverXML `xml:"driver"`
Source diskSourceXML `xml:"source"`
Target targetXML `xml:"target"`
ReadOnly *struct{} `xml:"readonly"`
}
type driverXML struct {
Name string `xml:"name,attr"`
Type string `xml:"type,attr"`
}
type diskSourceXML struct {
Pool string `xml:"pool,attr"`
Volume string `xml:"volume,attr"`
}
type targetXML struct {
Dev string `xml:"dev,attr"`
Bus string `xml:"bus,attr"`
}
type interfaceXML struct {
Type string `xml:"type,attr"`
Source ifaceSourceXML `xml:"source"`
Model modelXML `xml:"model"`
}
type ifaceSourceXML struct {
Bridge string `xml:"bridge,attr"`
}
type modelXML struct {
Type string `xml:"type,attr"`
}
type serialXML struct {
Type string `xml:"type,attr"`
Target portXML `xml:"target"`
}
type portXML struct {
Port int `xml:"port,attr"`
}
type consoleXML struct {
Type string `xml:"type,attr"`
Target consoleTargetXML `xml:"target"`
}
type consoleTargetXML struct {
Type string `xml:"type,attr"`
Port int `xml:"port,attr"`
}
type channelXML struct {
Type string `xml:"type,attr"`
Target channelTargetXML `xml:"target"`
}
type channelTargetXML struct {
Type string `xml:"type,attr"`
Name string `xml:"name,attr"`
}
type rngXML struct {
Model string `xml:"model,attr"`
Backend rngBackendXML `xml:"backend"`
}
type rngBackendXML struct {
Model string `xml:"model,attr"`
Value string `xml:",chardata"`
}
+113
View File
@@ -0,0 +1,113 @@
package kvm
import (
"encoding/xml"
"strings"
"testing"
"time"
)
func testDomainParams() domainParams {
return domainParams{
Name: "proxy-abc123def456ghij",
UID: "11111111-2222-3333-4444-555555555555",
CreatedAt: time.Date(2026, 9, 4, 12, 0, 0, 0, time.UTC),
VCPUs: 2,
MemoryMiB: 2048,
Pool: "default",
Bridge: "br0",
DiskVol: "proxy-abc123def456ghij.qcow2",
SeedVol: "proxy-abc123def456ghij-seed.iso",
}
}
func TestBuildDomainXML_fieldByField(t *testing.T) {
t.Parallel()
raw, err := buildDomainXML(testDomainParams())
if err != nil {
t.Fatalf("buildDomainXML() error = %v", err)
}
var d domainXML
if err := xml.Unmarshal([]byte(raw), &d); err != nil {
t.Fatalf("output is not well-formed XML: %v\n%s", err, raw)
}
if d.Type != "kvm" || d.Name != "proxy-abc123def456ghij" {
t.Errorf("type/name = %s/%s", d.Type, d.Name)
}
if d.VCPU != 2 || d.Memory.Value != 2048 || d.Memory.Unit != "MiB" {
t.Errorf("vcpu/memory = %d/%d %s, want 2/2048 MiB", d.VCPU, d.Memory.Value, d.Memory.Unit)
}
if d.OS.Type.Value != "hvm" || d.OS.Type.Arch != "x86_64" || d.OS.Type.Machine != "q35" || d.OS.Boot.Dev != "hd" {
t.Errorf("os = %+v", d.OS)
}
if d.Features.ACPI == nil || d.Features.APIC == nil {
t.Errorf("features = %+v, want acpi and apic", d.Features)
}
if d.CPU.Mode != "host-passthrough" {
t.Errorf("cpu mode = %q", d.CPU.Mode)
}
if d.OnPoweroff != "destroy" || d.OnReboot != "restart" || d.OnCrash != "destroy" {
t.Errorf("lifecycle = %s/%s/%s", d.OnPoweroff, d.OnReboot, d.OnCrash)
}
m := d.Metadata.Instance
if !m.managed() || m.UID != "11111111-2222-3333-4444-555555555555" || m.CreatedAt != "2026-09-04T12:00:00Z" {
t.Errorf("metadata = %+v", m)
}
if !strings.Contains(raw, `xmlns="`+metadataURI+`"`) {
t.Errorf("metadata element is not namespaced; libvirt rejects that:\n%s", raw)
}
if len(d.Devices.Disks) != 2 {
t.Fatalf("disks = %d, want 2", len(d.Devices.Disks))
}
disk, seed := d.Devices.Disks[0], d.Devices.Disks[1]
if disk.Type != "volume" || disk.Device != "disk" || disk.Driver.Type != "qcow2" ||
disk.Source.Pool != "default" || disk.Source.Volume != "proxy-abc123def456ghij.qcow2" ||
disk.Target.Dev != "vda" || disk.Target.Bus != "virtio" || disk.ReadOnly != nil {
t.Errorf("overlay disk = %+v", disk)
}
if seed.Type != "volume" || seed.Device != "cdrom" || seed.Driver.Type != "raw" ||
seed.Source.Pool != "default" || seed.Source.Volume != "proxy-abc123def456ghij-seed.iso" ||
seed.Target.Bus != "sata" || seed.ReadOnly == nil {
t.Errorf("seed cdrom = %+v", seed)
}
if len(d.Devices.Interfaces) != 1 {
t.Fatalf("interfaces = %d, want 1", len(d.Devices.Interfaces))
}
if nic := d.Devices.Interfaces[0]; nic.Type != "bridge" || nic.Source.Bridge != "br0" || nic.Model.Type != "virtio" {
t.Errorf("interface = %+v", nic)
}
if len(d.Devices.Channels) != 1 || d.Devices.Channels[0].Target.Name != guestAgentChannel || d.Devices.Channels[0].Target.Type != "virtio" {
t.Errorf("channels = %+v, want one virtio %s channel", d.Devices.Channels, guestAgentChannel)
}
if len(d.Devices.Serials) != 1 || len(d.Devices.Consoles) != 1 || d.Devices.Consoles[0].Target.Type != "serial" {
t.Errorf("serial/console = %+v/%+v", d.Devices.Serials, d.Devices.Consoles)
}
if d.Devices.RNG.Model != "virtio" || d.Devices.RNG.Backend.Value != "/dev/urandom" {
t.Errorf("rng = %+v", d.Devices.RNG)
}
if strings.Contains(raw, "<graphics") {
t.Errorf("domain has a graphics device; headless VMs must not:\n%s", raw)
}
}
func TestBuildDomainXML_escapesUserValues(t *testing.T) {
t.Parallel()
p := testDomainParams()
p.Bridge = `br0" onload="x`
raw, err := buildDomainXML(p)
if err != nil {
t.Fatalf("buildDomainXML() error = %v", err)
}
var d domainXML
if err := xml.Unmarshal([]byte(raw), &d); err != nil {
t.Fatalf("output is not well-formed XML: %v", err)
}
if d.Devices.Interfaces[0].Source.Bridge != p.Bridge {
t.Errorf("bridge = %q, want the raw value back", d.Devices.Interfaces[0].Source.Bridge)
}
}
+72
View File
@@ -0,0 +1,72 @@
package kvm
import (
"errors"
libvirt "github.com/digitalocean/go-libvirt"
"github.com/go-logr/logr"
"gitea.home.hrajfrisbee.cz/kacerr/egress-proxies-operator/internal/provider"
)
// classify maps an error onto the provider taxonomy. Errors the package
// already tagged with a sentinel (image download failures, placement
// errors) keep their class. For libvirt errors: missing domain or volume →
// NotFound; anything the host will keep rejecting (bad XML, unsupported
// config, denied access, missing pool) → Permanent; every other libvirt
// code, plus SSH/transport failures and cancellation, → Transient, because
// retrying is safer than latching Failed on an error nobody taught this
// function to recognise. libvirt exposes no quota signal, so nothing maps
// to QuotaExceeded.
func classify(err error) error {
for _, class := range []error{provider.ErrNotFound, provider.ErrQuotaExceeded, provider.ErrPermanent, provider.ErrTransient} {
if errors.Is(err, class) {
return class
}
}
var lerr libvirt.Error
if !errors.As(err, &lerr) {
return provider.ErrTransient
}
switch libvirt.ErrorNumber(lerr.Code) {
case libvirt.ErrNoDomain, libvirt.ErrNoStorageVol:
return provider.ErrNotFound
case libvirt.ErrNoStoragePool,
libvirt.ErrXMLError, libvirt.ErrXMLDetail, libvirt.ErrXMLInvalidSchema,
libvirt.ErrConfigUnsupported, libvirt.ErrNoSupport,
libvirt.ErrOperationUnsupported, libvirt.ErrArgumentUnsupported,
libvirt.ErrInvalidArg,
libvirt.ErrAuthFailed, libvirt.ErrAccessDenied, libvirt.ErrOperationDenied:
return provider.ErrPermanent
default:
return provider.ErrTransient
}
}
// isLibvirtError reports whether err came back from the libvirt daemon (as
// opposed to the SSH or RPC transport), which is what decides whether the
// cached connection is still trustworthy.
func isLibvirtError(err error) bool {
var lerr libvirt.Error
return errors.As(err, &lerr)
}
func hasCode(err error, code libvirt.ErrorNumber) bool {
var lerr libvirt.Error
return errors.As(err, &lerr) && libvirt.ErrorNumber(lerr.Code) == code
}
// logAPIError records the raw libvirt code and message at V(1); classify
// collapses them onto the coarser taxonomy, so this is the only place the
// original detail survives.
func logAPIError(log logr.Logger, op string, err error) {
if !log.V(1).Enabled() {
return
}
kv := []any{"op", op, "error", err.Error()}
var lerr libvirt.Error
if errors.As(err, &lerr) {
kv = append(kv, "libvirtCode", lerr.Code, "libvirtMessage", lerr.Message)
}
log.V(1).Info("libvirt call failed", kv...)
}
+69
View File
@@ -0,0 +1,69 @@
package kvm
import (
"context"
"errors"
"fmt"
"io"
"testing"
libvirt "github.com/digitalocean/go-libvirt"
"gitea.home.hrajfrisbee.cz/kacerr/egress-proxies-operator/internal/provider"
)
func lvErr(code libvirt.ErrorNumber) error {
return libvirt.Error{Code: uint32(code), Message: code.String()}
}
func TestClassify(t *testing.T) {
t.Parallel()
tests := []struct {
name string
err error
want error
}{
{"no domain", lvErr(libvirt.ErrNoDomain), provider.ErrNotFound},
{"no storage vol", lvErr(libvirt.ErrNoStorageVol), provider.ErrNotFound},
{"no storage pool", lvErr(libvirt.ErrNoStoragePool), provider.ErrPermanent},
{"xml error", lvErr(libvirt.ErrXMLError), provider.ErrPermanent},
{"xml detail", lvErr(libvirt.ErrXMLDetail), provider.ErrPermanent},
{"config unsupported", lvErr(libvirt.ErrConfigUnsupported), provider.ErrPermanent},
{"no support", lvErr(libvirt.ErrNoSupport), provider.ErrPermanent},
{"invalid arg", lvErr(libvirt.ErrInvalidArg), provider.ErrPermanent},
{"auth failed", lvErr(libvirt.ErrAuthFailed), provider.ErrPermanent},
{"access denied", lvErr(libvirt.ErrAccessDenied), provider.ErrPermanent},
{"operation invalid", lvErr(libvirt.ErrOperationInvalid), provider.ErrTransient},
{"operation failed", lvErr(libvirt.ErrOperationFailed), provider.ErrTransient},
{"internal error", lvErr(libvirt.ErrInternalError), provider.ErrTransient},
{"storage vol exists", lvErr(libvirt.ErrStorageVolExist), provider.ErrTransient},
{"wrapped libvirt error", fmt.Errorf("defining: %w", lvErr(libvirt.ErrNoDomain)), provider.ErrNotFound},
{"pre-tagged permanent", fmt.Errorf("%w: image returned HTTP 404", provider.ErrPermanent), provider.ErrPermanent},
{"pre-tagged transient wrapping libvirt not-found", fmt.Errorf("%w: %w", provider.ErrTransient, lvErr(libvirt.ErrNoDomain)), provider.ErrTransient},
{"transport eof", io.EOF, provider.ErrTransient},
{"context cancelled", context.Canceled, provider.ErrTransient},
{"plain error", errors.New("ssh: handshake failed"), provider.ErrTransient},
}
for _, tc := range tests {
t.Run(tc.name, func(t *testing.T) {
t.Parallel()
if got := classify(tc.err); !errors.Is(got, tc.want) {
t.Errorf("classify(%v) = %v, want %v", tc.err, got, tc.want)
}
})
}
}
func TestHasCode(t *testing.T) {
t.Parallel()
err := fmt.Errorf("lookup: %w", lvErr(libvirt.ErrNoDomain))
if !hasCode(err, libvirt.ErrNoDomain) {
t.Error("hasCode(ErrNoDomain) = false through a wrap")
}
if hasCode(err, libvirt.ErrNoStorageVol) {
t.Error("hasCode(ErrNoStorageVol) = true for an ErrNoDomain error")
}
if hasCode(io.EOF, libvirt.ErrNoDomain) || isLibvirtError(io.EOF) {
t.Error("io.EOF classified as a libvirt error")
}
}
+408
View File
@@ -0,0 +1,408 @@
package kvm
import (
"bytes"
"encoding/xml"
"fmt"
"io"
"slices"
"sync"
libvirt "github.com/digitalocean/go-libvirt"
)
// fakeAPI is an in-memory libvirtAPI: pools of volumes and a set of
// domains, with per-method error injection. It models just enough libvirt
// semantics for the provider's decisions to be observable — ErrStorageVolExist
// on duplicate create, ErrNoDomain/ErrNoStorageVol on missing lookups,
// ErrOperationInvalid on destroying a stopped domain, and so on.
type fakeAPI struct {
mu sync.Mutex
poolPath string
pools map[string]map[string]*fakeVol
domains map[string]*fakeDomain
calls []string
fail map[string]error
failOnce map[string]error
// uploadHook, when set, sees every StorageVolUpload before the fake
// reads the stream; it can return an error to simulate a broken upload.
uploadHook func(vol string, r io.Reader) error
}
type fakeVol struct {
name string
format string
capacity int64
backing string
data []byte
}
type fakeDomain struct {
name string
xml string
metadata string
running bool
// addrs is what DomainInterfaceAddresses returns per source.
addrs map[uint32][]libvirt.DomainInterface
}
func newFakeAPI(pools ...string) *fakeAPI {
f := &fakeAPI{
poolPath: "/var/lib/libvirt/images",
pools: map[string]map[string]*fakeVol{},
domains: map[string]*fakeDomain{},
fail: map[string]error{},
failOnce: map[string]error{},
}
for _, p := range pools {
f.pools[p] = map[string]*fakeVol{}
}
return f
}
// enter records the call and returns any injected error for method.
func (f *fakeAPI) enter(method string) error {
f.calls = append(f.calls, method)
if err, ok := f.failOnce[method]; ok {
delete(f.failOnce, method)
return err
}
return f.fail[method]
}
func (f *fakeAPI) callCount(method string) int {
f.mu.Lock()
defer f.mu.Unlock()
n := 0
for _, c := range f.calls {
if c == method {
n++
}
}
return n
}
func (f *fakeAPI) hasCall(method string) bool { return f.callCount(method) > 0 }
func (f *fakeAPI) vol(pool, name string) *fakeVol {
f.mu.Lock()
defer f.mu.Unlock()
return f.pools[pool][name]
}
func (f *fakeAPI) putVol(pool, name, format string, data []byte) {
f.mu.Lock()
defer f.mu.Unlock()
f.pools[pool][name] = &fakeVol{name: name, format: format, capacity: int64(len(data)), data: data}
}
func (f *fakeAPI) volNames(pool string) []string {
f.mu.Lock()
defer f.mu.Unlock()
var out []string
for n := range f.pools[pool] {
out = append(out, n)
}
slices.Sort(out)
return out
}
func (f *fakeAPI) domain(name string) *fakeDomain {
f.mu.Lock()
defer f.mu.Unlock()
return f.domains[name]
}
func (f *fakeAPI) putDomain(d *fakeDomain) {
f.mu.Lock()
defer f.mu.Unlock()
f.domains[d.name] = d
}
func lvError(code libvirt.ErrorNumber, format string, args ...any) error {
return libvirt.Error{Code: uint32(code), Message: fmt.Sprintf(format, args...)}
}
// --- domains ---
func (f *fakeAPI) DomainLookupByName(name string) (libvirt.Domain, error) {
f.mu.Lock()
defer f.mu.Unlock()
if err := f.enter("DomainLookupByName"); err != nil {
return libvirt.Domain{}, err
}
if _, ok := f.domains[name]; !ok {
return libvirt.Domain{}, lvError(libvirt.ErrNoDomain, "Domain not found: no domain with matching name '%s'", name)
}
return libvirt.Domain{Name: name}, nil
}
func (f *fakeAPI) DomainDefineXML(raw string) (libvirt.Domain, error) {
f.mu.Lock()
defer f.mu.Unlock()
if err := f.enter("DomainDefineXML"); err != nil {
return libvirt.Domain{}, err
}
var d domainXML
if err := xml.Unmarshal([]byte(raw), &d); err != nil {
return libvirt.Domain{}, lvError(libvirt.ErrXMLError, "%v", err)
}
meta, err := xml.Marshal(d.Metadata.Instance)
if err != nil {
return libvirt.Domain{}, lvError(libvirt.ErrXMLError, "%v", err)
}
existing := f.domains[d.Name]
nd := &fakeDomain{name: d.Name, xml: raw, metadata: string(meta)}
if existing != nil {
nd.running = existing.running
nd.addrs = existing.addrs
}
f.domains[d.Name] = nd
return libvirt.Domain{Name: d.Name}, nil
}
func (f *fakeAPI) DomainCreate(d libvirt.Domain) error {
f.mu.Lock()
defer f.mu.Unlock()
if err := f.enter("DomainCreate"); err != nil {
return err
}
dom, ok := f.domains[d.Name]
if !ok {
return lvError(libvirt.ErrNoDomain, "no domain %s", d.Name)
}
if dom.running {
return lvError(libvirt.ErrOperationInvalid, "domain is already running")
}
dom.running = true
return nil
}
func (f *fakeAPI) DomainDestroy(d libvirt.Domain) error {
f.mu.Lock()
defer f.mu.Unlock()
if err := f.enter("DomainDestroy"); err != nil {
return err
}
dom, ok := f.domains[d.Name]
if !ok {
return lvError(libvirt.ErrNoDomain, "no domain %s", d.Name)
}
if !dom.running {
return lvError(libvirt.ErrOperationInvalid, "domain is not running")
}
dom.running = false
return nil
}
func (f *fakeAPI) DomainUndefineFlags(d libvirt.Domain, _ libvirt.DomainUndefineFlagsValues) error {
f.mu.Lock()
defer f.mu.Unlock()
if err := f.enter("DomainUndefineFlags"); err != nil {
return err
}
if _, ok := f.domains[d.Name]; !ok {
return lvError(libvirt.ErrNoDomain, "no domain %s", d.Name)
}
delete(f.domains, d.Name)
return nil
}
func (f *fakeAPI) DomainGetState(d libvirt.Domain, _ uint32) (int32, int32, error) {
f.mu.Lock()
defer f.mu.Unlock()
if err := f.enter("DomainGetState"); err != nil {
return 0, 0, err
}
dom, ok := f.domains[d.Name]
if !ok {
return 0, 0, lvError(libvirt.ErrNoDomain, "no domain %s", d.Name)
}
if dom.running {
return int32(libvirt.DomainRunning), 0, nil
}
return int32(libvirt.DomainShutoff), 0, nil
}
func (f *fakeAPI) DomainGetMetadata(d libvirt.Domain, typ int32, uri libvirt.OptString, _ libvirt.DomainModificationImpact) (string, error) {
f.mu.Lock()
defer f.mu.Unlock()
if err := f.enter("DomainGetMetadata"); err != nil {
return "", err
}
dom, ok := f.domains[d.Name]
if !ok {
return "", lvError(libvirt.ErrNoDomain, "no domain %s", d.Name)
}
if typ != int32(libvirt.DomainMetadataElement) || len(uri) != 1 || uri[0] != metadataURI || dom.metadata == "" {
return "", lvError(libvirt.ErrNoDomainMetadata, "Requested metadata element is not present")
}
return dom.metadata, nil
}
func (f *fakeAPI) DomainInterfaceAddresses(d libvirt.Domain, source uint32, _ uint32) ([]libvirt.DomainInterface, error) {
f.mu.Lock()
defer f.mu.Unlock()
if err := f.enter("DomainInterfaceAddresses"); err != nil {
return nil, err
}
dom, ok := f.domains[d.Name]
if !ok {
return nil, lvError(libvirt.ErrNoDomain, "no domain %s", d.Name)
}
if !dom.running {
return nil, lvError(libvirt.ErrOperationInvalid, "domain is not running")
}
if source == uint32(libvirt.DomainInterfaceAddressesSrcAgent) && dom.addrs[source] == nil {
return nil, lvError(libvirt.ErrAgentUnresponsive, "Guest agent is not responding")
}
return dom.addrs[source], nil
}
func (f *fakeAPI) ConnectListAllDomains(_ int32, _ libvirt.ConnectListAllDomainsFlags) ([]libvirt.Domain, uint32, error) {
f.mu.Lock()
defer f.mu.Unlock()
if err := f.enter("ConnectListAllDomains"); err != nil {
return nil, 0, err
}
names := make([]string, 0, len(f.domains))
for n := range f.domains {
names = append(names, n)
}
slices.Sort(names)
out := make([]libvirt.Domain, 0, len(names))
for _, n := range names {
out = append(out, libvirt.Domain{Name: n})
}
return out, uint32(len(out)), nil
}
// --- storage ---
func (f *fakeAPI) StoragePoolLookupByName(name string) (libvirt.StoragePool, error) {
f.mu.Lock()
defer f.mu.Unlock()
if err := f.enter("StoragePoolLookupByName"); err != nil {
return libvirt.StoragePool{}, err
}
if _, ok := f.pools[name]; !ok {
return libvirt.StoragePool{}, lvError(libvirt.ErrNoStoragePool, "Storage pool not found: no storage pool with matching name '%s'", name)
}
return libvirt.StoragePool{Name: name}, nil
}
func (f *fakeAPI) StoragePoolRefresh(pool libvirt.StoragePool, _ uint32) error {
f.mu.Lock()
defer f.mu.Unlock()
if err := f.enter("StoragePoolRefresh"); err != nil {
return err
}
if _, ok := f.pools[pool.Name]; !ok {
return lvError(libvirt.ErrNoStoragePool, "no pool %s", pool.Name)
}
return nil
}
func (f *fakeAPI) StorageVolLookupByName(pool libvirt.StoragePool, name string) (libvirt.StorageVol, error) {
f.mu.Lock()
defer f.mu.Unlock()
if err := f.enter("StorageVolLookupByName"); err != nil {
return libvirt.StorageVol{}, err
}
if _, ok := f.pools[pool.Name][name]; !ok {
return libvirt.StorageVol{}, lvError(libvirt.ErrNoStorageVol, "Storage volume not found: no storage vol with matching name '%s'", name)
}
return libvirt.StorageVol{Pool: pool.Name, Name: name, Key: f.poolPath + "/" + name}, nil
}
func (f *fakeAPI) StorageVolGetPath(vol libvirt.StorageVol) (string, error) {
f.mu.Lock()
defer f.mu.Unlock()
if err := f.enter("StorageVolGetPath"); err != nil {
return "", err
}
if _, ok := f.pools[vol.Pool][vol.Name]; !ok {
return "", lvError(libvirt.ErrNoStorageVol, "no vol %s", vol.Name)
}
return f.poolPath + "/" + vol.Name, nil
}
func (f *fakeAPI) StorageVolCreateXML(pool libvirt.StoragePool, raw string, _ libvirt.StorageVolCreateFlags) (libvirt.StorageVol, error) {
f.mu.Lock()
defer f.mu.Unlock()
if err := f.enter("StorageVolCreateXML"); err != nil {
return libvirt.StorageVol{}, err
}
vols, ok := f.pools[pool.Name]
if !ok {
return libvirt.StorageVol{}, lvError(libvirt.ErrNoStoragePool, "no pool %s", pool.Name)
}
var v volumeXML
if err := xml.Unmarshal([]byte(raw), &v); err != nil {
return libvirt.StorageVol{}, lvError(libvirt.ErrXMLError, "%v", err)
}
if _, exists := vols[v.Name]; exists {
return libvirt.StorageVol{}, lvError(libvirt.ErrStorageVolExist, "storage volume '%s' exists already", v.Name)
}
fv := &fakeVol{name: v.Name, format: v.Target.Format.Type, capacity: v.Capacity.Value}
if v.BackingStore != nil {
fv.backing = v.BackingStore.Path
}
vols[v.Name] = fv
return libvirt.StorageVol{Pool: pool.Name, Name: v.Name, Key: f.poolPath + "/" + v.Name}, nil
}
func (f *fakeAPI) StorageVolUpload(vol libvirt.StorageVol, r io.Reader, _, _ uint64, _ libvirt.StorageVolUploadFlags) error {
f.mu.Lock()
if err := f.enter("StorageVolUpload"); err != nil {
f.mu.Unlock()
return err
}
fv, ok := f.pools[vol.Pool][vol.Name]
hook := f.uploadHook
f.mu.Unlock()
if !ok {
return lvError(libvirt.ErrNoStorageVol, "no vol %s", vol.Name)
}
if hook != nil {
if err := hook(vol.Name, r); err != nil {
return err
}
}
// Read outside the lock: the stream may be a slow HTTP body.
data, err := io.ReadAll(r)
if err != nil {
return fmt.Errorf("reading upload stream: %w", err)
}
f.mu.Lock()
defer f.mu.Unlock()
fv.data = data
return nil
}
func (f *fakeAPI) StorageVolDownload(vol libvirt.StorageVol, w io.Writer, _, _ uint64, _ libvirt.StorageVolDownloadFlags) error {
f.mu.Lock()
defer f.mu.Unlock()
if err := f.enter("StorageVolDownload"); err != nil {
return err
}
fv, ok := f.pools[vol.Pool][vol.Name]
if !ok {
return lvError(libvirt.ErrNoStorageVol, "no vol %s", vol.Name)
}
_, err := io.Copy(w, bytes.NewReader(fv.data))
return err
}
func (f *fakeAPI) StorageVolDelete(vol libvirt.StorageVol, _ libvirt.StorageVolDeleteFlags) error {
f.mu.Lock()
defer f.mu.Unlock()
if err := f.enter("StorageVolDelete"); err != nil {
return err
}
if _, ok := f.pools[vol.Pool][vol.Name]; !ok {
return lvError(libvirt.ErrNoStorageVol, "no vol %s", vol.Name)
}
delete(f.pools[vol.Pool], vol.Name)
return nil
}
+21
View File
@@ -0,0 +1,21 @@
package kvm
import (
"fmt"
"strings"
)
// The providerID carries the host, so Get/Delete stay correct after a
// placement.zone edit mid-replacement — re-reading the spec then would look
// at the wrong hypervisor.
func formatProviderID(host, name string) string {
return fmt.Sprintf("hosts/%s/domains/%s", host, name)
}
func parseProviderID(id string) (host, name string, err error) {
parts := strings.Split(id, "/")
if len(parts) != 4 || parts[0] != "hosts" || parts[2] != "domains" || parts[1] == "" || parts[3] == "" {
return "", "", fmt.Errorf("malformed kvm providerID %q, want hosts/<host>/domains/<name>", id)
}
return parts[1], parts[3], nil
}
+41
View File
@@ -0,0 +1,41 @@
package kvm
import "testing"
func TestParseProviderID_roundTrip(t *testing.T) {
t.Parallel()
id := formatProviderID("nas1", "proxy-abc123def456ghij")
if id != "hosts/nas1/domains/proxy-abc123def456ghij" {
t.Fatalf("formatProviderID() = %q", id)
}
host, name, err := parseProviderID(id)
if err != nil {
t.Fatalf("parseProviderID(%q) error = %v", id, err)
}
if host != "nas1" || name != "proxy-abc123def456ghij" {
t.Errorf("parseProviderID(%q) = %q, %q", id, host, name)
}
}
func TestParseProviderID_malformed(t *testing.T) {
t.Parallel()
tests := []struct {
name string
id string
}{
{"empty", ""},
{"gcp style", "zones/europe-west1-b/instances/proxy-x"},
{"missing name", "hosts/nas1/domains/"},
{"missing host", "hosts//domains/proxy-x"},
{"too many segments", "hosts/nas1/domains/proxy-x/extra"},
{"bare name", "proxy-x"},
}
for _, tc := range tests {
t.Run(tc.name, func(t *testing.T) {
t.Parallel()
if _, _, err := parseProviderID(tc.id); err == nil {
t.Errorf("parseProviderID(%q) error = nil, want error", tc.id)
}
})
}
}
+232
View File
@@ -0,0 +1,232 @@
package kvm
import (
"bytes"
"context"
"crypto/sha256"
"encoding/hex"
"fmt"
"io"
"net/http"
"net/url"
"strings"
"sync"
"time"
libvirt "github.com/digitalocean/go-libvirt"
"github.com/go-logr/logr"
"gitea.home.hrajfrisbee.cz/kacerr/egress-proxies-operator/internal/provider"
)
// imageSource is a resolved placement.image: either an entry from the
// provider's images map (Key set) or a URL given directly.
type imageSource struct {
Key string
URL string
SHA256 string
Format string
}
// resolveImage maps placement.image to a source. Anything that is neither a
// configured key nor an http(s) URL is a spec error the reconciler should
// surface, not retry.
func resolveImage(cfg provider.KVMConfig, image string) (imageSource, error) {
if img, ok := cfg.Images[image]; ok {
return imageSource{Key: image, URL: img.URL, SHA256: img.SHA256, Format: img.Format}, nil
}
u, err := url.Parse(image)
if err == nil && (u.Scheme == "http" || u.Scheme == "https") && u.Host != "" {
return imageSource{URL: image, Format: formatQcow2}, nil
}
return imageSource{}, fmt.Errorf("%w: placement.image %q is neither a key in kvm.images nor an http(s) URL",
provider.ErrPermanent, image)
}
// volumeName is the cached base image's name in the pool: readable for
// configured keys, a URL hash for ad-hoc URLs.
func (s imageSource) volumeName() string {
if s.Key != "" {
return "base-" + s.Key + "." + s.Format
}
sum := sha256.Sum256([]byte(s.URL))
return "base-" + hex.EncodeToString(sum[:8]) + "." + s.Format
}
// markerName is the sidecar volume holding the sha256 of the verified
// upload. Its presence is what says "the base volume is complete".
func (s imageSource) markerName() string {
return s.volumeName() + ".sha256"
}
// imageCache fetches base images into a host's pool on first use. It keeps
// no state about what is cached — the pool is the source of truth, checked
// on every call — only a per-(host, image) lock so concurrent Creates for
// the same image on the same host result in one download.
type imageCache struct {
http *http.Client
mu sync.Mutex
locks map[string]*sync.Mutex
}
func newImageCache(client *http.Client) *imageCache {
if client == nil {
client = http.DefaultClient
}
return &imageCache{http: client, locks: make(map[string]*sync.Mutex)}
}
func (c *imageCache) lockFor(key string) *sync.Mutex {
c.mu.Lock()
defer c.mu.Unlock()
m, ok := c.locks[key]
if !ok {
m = &sync.Mutex{}
c.locks[key] = m
}
return m
}
// ensure makes src present in pool and returns the base volume's path on
// the host. The image is downloaded synchronously through the libvirt
// stream when the pool does not already hold a verified copy.
func (c *imageCache) ensure(ctx context.Context, log logr.Logger, api libvirtAPI, hostName string, pool libvirt.StoragePool, src imageSource) (string, error) {
m := c.lockFor(hostName + "/" + src.volumeName())
m.Lock()
defer m.Unlock()
base, ready, err := c.lookup(api, pool, src)
if err != nil {
return "", err
}
if ready {
log.V(1).Info("base image already cached", "host", hostName, "volume", src.volumeName())
return api.StorageVolGetPath(base)
}
if err := deleteIfExists(api, pool, src.markerName()); err != nil {
return "", err
}
if err := deleteIfExists(api, pool, src.volumeName()); err != nil {
return "", err
}
base, err = c.fetch(ctx, log, api, hostName, pool, src)
if err != nil {
return "", err
}
return api.StorageVolGetPath(base)
}
// lookup reports whether the pool holds a complete, verified base volume:
// the volume and its marker both exist and, when a checksum is configured,
// the marker records that checksum.
func (c *imageCache) lookup(api libvirtAPI, pool libvirt.StoragePool, src imageSource) (libvirt.StorageVol, bool, error) {
base, err := api.StorageVolLookupByName(pool, src.volumeName())
if hasCode(err, libvirt.ErrNoStorageVol) {
return libvirt.StorageVol{}, false, nil
}
if err != nil {
return libvirt.StorageVol{}, false, fmt.Errorf("looking up base volume: %w", err)
}
marker, err := api.StorageVolLookupByName(pool, src.markerName())
if hasCode(err, libvirt.ErrNoStorageVol) {
return base, false, nil
}
if err != nil {
return libvirt.StorageVol{}, false, fmt.Errorf("looking up image marker: %w", err)
}
if src.SHA256 == "" {
return base, true, nil
}
var buf bytes.Buffer
if err := api.StorageVolDownload(marker, &buf, 0, 0, 0); err != nil {
return libvirt.StorageVol{}, false, fmt.Errorf("reading image marker: %w", err)
}
return base, strings.EqualFold(strings.TrimSpace(buf.String()), src.SHA256), nil
}
// fetch streams the image from its URL straight into a new base volume,
// hashing on the way, then writes the marker. Nothing is buffered in the
// operator beyond the HTTP body's read buffer.
func (c *imageCache) fetch(ctx context.Context, log logr.Logger, api libvirtAPI, hostName string, pool libvirt.StoragePool, src imageSource) (libvirt.StorageVol, error) {
req, err := http.NewRequestWithContext(ctx, http.MethodGet, src.URL, nil)
if err != nil {
return libvirt.StorageVol{}, fmt.Errorf("%w: building image request: %v", provider.ErrPermanent, err)
}
resp, err := c.http.Do(req)
if err != nil {
return libvirt.StorageVol{}, fmt.Errorf("%w: fetching image %s: %v", provider.ErrTransient, src.URL, err)
}
defer resp.Body.Close()
switch {
case resp.StatusCode >= 200 && resp.StatusCode < 300:
case resp.StatusCode >= 400 && resp.StatusCode < 500:
return libvirt.StorageVol{}, fmt.Errorf("%w: image %s returned HTTP %d", provider.ErrPermanent, src.URL, resp.StatusCode)
default:
return libvirt.StorageVol{}, fmt.Errorf("%w: image %s returned HTTP %d", provider.ErrTransient, src.URL, resp.StatusCode)
}
if resp.ContentLength < 0 {
return libvirt.StorageVol{}, fmt.Errorf("%w: image %s has no Content-Length; the base volume needs a known size", provider.ErrPermanent, src.URL)
}
log.Info("fetching base image into host pool",
"host", hostName, "pool", pool.Name, "volume", src.volumeName(),
"url", src.URL, "bytes", resp.ContentLength)
start := time.Now()
baseXML, err := buildVolumeXML(volumeSpec{Name: src.volumeName(), SizeBytes: resp.ContentLength, Format: src.Format})
if err != nil {
return libvirt.StorageVol{}, err
}
base, err := api.StorageVolCreateXML(pool, baseXML, 0)
if err != nil {
return libvirt.StorageVol{}, fmt.Errorf("creating base volume: %w", err)
}
hasher := sha256.New()
if err := api.StorageVolUpload(base, io.TeeReader(resp.Body, hasher), 0, 0, 0); err != nil {
_ = api.StorageVolDelete(base, 0)
return libvirt.StorageVol{}, fmt.Errorf("uploading image to base volume: %w", err)
}
digest := hex.EncodeToString(hasher.Sum(nil))
if src.SHA256 != "" && !strings.EqualFold(digest, src.SHA256) {
_ = api.StorageVolDelete(base, 0)
return libvirt.StorageVol{}, fmt.Errorf("%w: image %s sha256 is %s, config expects %s",
provider.ErrPermanent, src.URL, digest, src.SHA256)
}
// libvirt sized the volume from Content-Length; a refresh makes it
// re-read the real virtual size from the uploaded qcow2 header.
if err := api.StoragePoolRefresh(pool, 0); err != nil {
return libvirt.StorageVol{}, fmt.Errorf("refreshing pool after upload: %w", err)
}
markerXML, err := buildVolumeXML(rawVolume(src.markerName(), int64(len(digest))))
if err != nil {
return libvirt.StorageVol{}, err
}
marker, err := api.StorageVolCreateXML(pool, markerXML, 0)
if err != nil {
return libvirt.StorageVol{}, fmt.Errorf("creating image marker: %w", err)
}
if err := api.StorageVolUpload(marker, strings.NewReader(digest), 0, 0, 0); err != nil {
_ = api.StorageVolDelete(marker, 0)
return libvirt.StorageVol{}, fmt.Errorf("writing image marker: %w", err)
}
log.Info("base image cached",
"host", hostName, "volume", src.volumeName(), "sha256", digest,
"bytes", resp.ContentLength, "duration", time.Since(start).Round(time.Second))
return base, nil
}
func deleteIfExists(api libvirtAPI, pool libvirt.StoragePool, name string) error {
vol, err := api.StorageVolLookupByName(pool, name)
if hasCode(err, libvirt.ErrNoStorageVol) {
return nil
}
if err != nil {
return fmt.Errorf("looking up volume %s: %w", name, err)
}
if err := api.StorageVolDelete(vol, 0); err != nil && !hasCode(err, libvirt.ErrNoStorageVol) {
return fmt.Errorf("deleting volume %s: %w", name, err)
}
return nil
}
+385
View File
@@ -0,0 +1,385 @@
package kvm
import (
"bytes"
"context"
"crypto/sha256"
"encoding/hex"
"errors"
"io"
"net/http"
"net/http/httptest"
"strconv"
"strings"
"sync"
"sync/atomic"
"testing"
"time"
"github.com/go-logr/logr"
"gitea.home.hrajfrisbee.cz/kacerr/egress-proxies-operator/internal/provider"
)
var testImage = bytes.Repeat([]byte("QFI\xfb fake qcow2 payload "), 1024)
func testImageSHA() string {
sum := sha256.Sum256(testImage)
return hex.EncodeToString(sum[:])
}
// imageServer serves testImage and counts hits. Status overrides the
// response, noLength forces chunked encoding (no Content-Length), delay
// slows each response so concurrent callers overlap.
type imageServer struct {
*httptest.Server
hits atomic.Int32
status int
noLength bool
delay time.Duration
body []byte
}
func newImageServer(t *testing.T) *imageServer {
t.Helper()
s := &imageServer{status: http.StatusOK, body: testImage}
s.Server = httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, _ *http.Request) {
s.hits.Add(1)
time.Sleep(s.delay)
if s.status != http.StatusOK {
w.WriteHeader(s.status)
return
}
// Bodies larger than the server's write buffer would otherwise go
// out chunked without a Content-Length, unlike real image mirrors.
if s.noLength {
w.(http.Flusher).Flush()
} else {
w.Header().Set("Content-Length", strconv.Itoa(len(s.body)))
}
_, _ = w.Write(s.body)
}))
t.Cleanup(s.Close)
return s
}
func testSource(srv *imageServer, sha string) imageSource {
return imageSource{Key: "debian-13", URL: srv.URL + "/debian.qcow2", SHA256: sha, Format: formatQcow2}
}
func ensureTest(t *testing.T, api *fakeAPI, src imageSource) (string, error) {
t.Helper()
c := newImageCache(nil)
pool, err := api.StoragePoolLookupByName("default")
if err != nil {
t.Fatal(err)
}
return c.ensure(context.Background(), logr.Discard(), api, "nas1", pool, src)
}
func TestResolveImage(t *testing.T) {
t.Parallel()
cfg := withDefaults(provider.KVMConfig{Images: map[string]provider.KVMImage{
"debian-13": {URL: "https://cloud.debian.org/d.qcow2", SHA256: "abc"},
}})
tests := []struct {
name string
image string
want imageSource
wantErr bool
}{
{"configured key", "debian-13", imageSource{Key: "debian-13", URL: "https://cloud.debian.org/d.qcow2", SHA256: "abc", Format: "qcow2"}, false},
{"direct https url", "https://x.example/img.qcow2", imageSource{URL: "https://x.example/img.qcow2", Format: "qcow2"}, false},
{"direct http url", "http://mirror/img.qcow2", imageSource{URL: "http://mirror/img.qcow2", Format: "qcow2"}, false},
{"unknown key", "ubuntu", imageSource{}, true},
{"file url", "file:///tmp/x.qcow2", imageSource{}, true},
{"empty", "", imageSource{}, true},
}
for _, tc := range tests {
t.Run(tc.name, func(t *testing.T) {
t.Parallel()
got, err := resolveImage(cfg, tc.image)
if tc.wantErr {
if !errors.Is(err, provider.ErrPermanent) {
t.Fatalf("resolveImage(%q) error = %v, want ErrPermanent", tc.image, err)
}
return
}
if err != nil {
t.Fatalf("resolveImage(%q) error = %v", tc.image, err)
}
if got != tc.want {
t.Errorf("resolveImage(%q) = %+v, want %+v", tc.image, got, tc.want)
}
})
}
}
func TestImageSource_volumeNames(t *testing.T) {
t.Parallel()
keyed := imageSource{Key: "debian-13", URL: "https://a/b.qcow2", Format: "qcow2"}
if keyed.volumeName() != "base-debian-13.qcow2" || keyed.markerName() != "base-debian-13.qcow2.sha256" {
t.Errorf("keyed names = %s / %s", keyed.volumeName(), keyed.markerName())
}
direct := imageSource{URL: "https://a/b.qcow2", Format: "qcow2"}
name := direct.volumeName()
if !strings.HasPrefix(name, "base-") || !strings.HasSuffix(name, ".qcow2") || len(name) != len("base-")+16+len(".qcow2") {
t.Errorf("direct-url name = %q, want base-<16 hex>.qcow2", name)
}
other := imageSource{URL: "https://a/c.qcow2", Format: "qcow2"}
if other.volumeName() == name {
t.Error("different URLs produced the same volume name")
}
}
func TestImageCache_ensure_freshFetch(t *testing.T) {
t.Parallel()
srv := newImageServer(t)
api := newFakeAPI("default")
path, err := ensureTest(t, api, testSource(srv, testImageSHA()))
if err != nil {
t.Fatalf("ensure() error = %v", err)
}
if path != "/var/lib/libvirt/images/base-debian-13.qcow2" {
t.Errorf("path = %q", path)
}
base := api.vol("default", "base-debian-13.qcow2")
if base == nil || !bytes.Equal(base.data, testImage) {
t.Fatalf("base volume missing or content differs")
}
if base.format != "qcow2" || base.capacity != int64(len(testImage)) {
t.Errorf("base volume format/capacity = %s/%d, want qcow2/%d", base.format, base.capacity, len(testImage))
}
marker := api.vol("default", "base-debian-13.qcow2.sha256")
if marker == nil || string(marker.data) != testImageSHA() {
t.Fatalf("marker missing or wrong: %+v", marker)
}
if !api.hasCall("StoragePoolRefresh") {
t.Error("pool was not refreshed after the upload")
}
if srv.hits.Load() != 1 {
t.Errorf("http hits = %d, want 1", srv.hits.Load())
}
}
func TestImageCache_ensure_noChecksumConfigured(t *testing.T) {
t.Parallel()
srv := newImageServer(t)
api := newFakeAPI("default")
if _, err := ensureTest(t, api, testSource(srv, "")); err != nil {
t.Fatalf("ensure() error = %v", err)
}
if marker := api.vol("default", "base-debian-13.qcow2.sha256"); marker == nil || string(marker.data) != testImageSHA() {
t.Errorf("marker should still record the computed digest, got %+v", marker)
}
// Second call: ready without a checksum to compare, so no marker read.
if _, err := ensureTest(t, api, testSource(srv, "")); err != nil {
t.Fatalf("second ensure() error = %v", err)
}
if srv.hits.Load() != 1 {
t.Errorf("http hits = %d, want 1", srv.hits.Load())
}
}
func TestImageCache_ensure_alreadyCached(t *testing.T) {
t.Parallel()
srv := newImageServer(t)
api := newFakeAPI("default")
api.putVol("default", "base-debian-13.qcow2", "qcow2", testImage)
api.putVol("default", "base-debian-13.qcow2.sha256", "raw", []byte(testImageSHA()))
path, err := ensureTest(t, api, testSource(srv, testImageSHA()))
if err != nil {
t.Fatalf("ensure() error = %v", err)
}
if path != "/var/lib/libvirt/images/base-debian-13.qcow2" {
t.Errorf("path = %q", path)
}
if srv.hits.Load() != 0 {
t.Errorf("http hits = %d, want 0 for a cached image", srv.hits.Load())
}
if api.hasCall("StorageVolCreateXML") || api.hasCall("StorageVolDelete") {
t.Errorf("cached image was touched: calls = %v", api.calls)
}
}
func TestImageCache_ensure_markerMissingRedownloads(t *testing.T) {
t.Parallel()
srv := newImageServer(t)
api := newFakeAPI("default")
api.putVol("default", "base-debian-13.qcow2", "qcow2", []byte("half an image"))
if _, err := ensureTest(t, api, testSource(srv, testImageSHA())); err != nil {
t.Fatalf("ensure() error = %v", err)
}
if srv.hits.Load() != 1 {
t.Errorf("http hits = %d, want 1: a base without its marker is a partial upload", srv.hits.Load())
}
if base := api.vol("default", "base-debian-13.qcow2"); base == nil || !bytes.Equal(base.data, testImage) {
t.Error("partial base volume was not replaced")
}
}
func TestImageCache_ensure_checksumChangedRedownloads(t *testing.T) {
t.Parallel()
srv := newImageServer(t)
api := newFakeAPI("default")
api.putVol("default", "base-debian-13.qcow2", "qcow2", []byte("an older release"))
api.putVol("default", "base-debian-13.qcow2.sha256", "raw", []byte(strings.Repeat("0", 64)))
if _, err := ensureTest(t, api, testSource(srv, testImageSHA())); err != nil {
t.Fatalf("ensure() error = %v", err)
}
if srv.hits.Load() != 1 {
t.Errorf("http hits = %d, want 1 after the configured checksum changed", srv.hits.Load())
}
if marker := api.vol("default", "base-debian-13.qcow2.sha256"); marker == nil || string(marker.data) != testImageSHA() {
t.Errorf("marker not updated: %+v", marker)
}
}
func TestImageCache_ensure_checksumMismatchIsPermanent(t *testing.T) {
t.Parallel()
srv := newImageServer(t)
api := newFakeAPI("default")
_, err := ensureTest(t, api, testSource(srv, strings.Repeat("a", 64)))
if !errors.Is(err, provider.ErrPermanent) {
t.Fatalf("ensure() error = %v, want ErrPermanent", err)
}
if names := api.volNames("default"); len(names) != 0 {
t.Errorf("volumes left behind after a checksum mismatch: %v", names)
}
}
func TestImageCache_ensure_httpErrors(t *testing.T) {
t.Parallel()
tests := []struct {
name string
status int
want error
}{
{"404 is permanent", http.StatusNotFound, provider.ErrPermanent},
{"403 is permanent", http.StatusForbidden, provider.ErrPermanent},
{"503 is transient", http.StatusServiceUnavailable, provider.ErrTransient},
{"500 is transient", http.StatusInternalServerError, provider.ErrTransient},
}
for _, tc := range tests {
t.Run(tc.name, func(t *testing.T) {
t.Parallel()
srv := newImageServer(t)
srv.status = tc.status
api := newFakeAPI("default")
_, err := ensureTest(t, api, testSource(srv, ""))
if !errors.Is(err, tc.want) {
t.Fatalf("ensure() error = %v, want %v", err, tc.want)
}
if names := api.volNames("default"); len(names) != 0 {
t.Errorf("volumes created despite HTTP %d: %v", tc.status, names)
}
})
}
}
func TestImageCache_ensure_noContentLengthIsPermanent(t *testing.T) {
t.Parallel()
srv := newImageServer(t)
srv.noLength = true
api := newFakeAPI("default")
_, err := ensureTest(t, api, testSource(srv, ""))
if !errors.Is(err, provider.ErrPermanent) || !strings.Contains(err.Error(), "Content-Length") {
t.Fatalf("ensure() error = %v, want ErrPermanent about Content-Length", err)
}
}
func TestImageCache_ensure_uploadFailureCleansUp(t *testing.T) {
t.Parallel()
srv := newImageServer(t)
api := newFakeAPI("default")
api.uploadHook = func(_ string, r io.Reader) error {
_, _ = io.CopyN(io.Discard, r, 100)
return io.ErrUnexpectedEOF
}
_, err := ensureTest(t, api, testSource(srv, testImageSHA()))
if !errors.Is(err, provider.ErrTransient) && classify(err) != provider.ErrTransient {
t.Fatalf("ensure() error = %v, want a transient failure", err)
}
if names := api.volNames("default"); len(names) != 0 {
t.Errorf("volumes left behind after a failed upload: %v", names)
}
// Next attempt starts clean and succeeds.
api.uploadHook = nil
if _, err := ensureTest(t, api, testSource(srv, testImageSHA())); err != nil {
t.Fatalf("retry ensure() error = %v", err)
}
}
func TestImageCache_ensure_missingPoolIsPermanent(t *testing.T) {
t.Parallel()
srv := newImageServer(t)
api := newFakeAPI("default")
c := newImageCache(nil)
pool, _ := api.StoragePoolLookupByName("default")
delete(api.pools, "default")
_, err := c.ensure(context.Background(), logr.Discard(), api, "nas1", pool, testSource(srv, ""))
if classify(err) != provider.ErrPermanent {
t.Fatalf("ensure() error = %v, want one classifying as Permanent", err)
}
}
func TestImageCache_ensure_concurrentCallersDownloadOnce(t *testing.T) {
t.Parallel()
srv := newImageServer(t)
srv.delay = 50 * time.Millisecond
api := newFakeAPI("default")
c := newImageCache(nil)
pool, _ := api.StoragePoolLookupByName("default")
src := testSource(srv, testImageSHA())
var wg sync.WaitGroup
paths := make([]string, 4)
errs := make([]error, 4)
for i := range 4 {
wg.Go(func() {
paths[i], errs[i] = c.ensure(context.Background(), logr.Discard(), api, "nas1", pool, src)
})
}
wg.Wait()
for i := range 4 {
if errs[i] != nil || paths[i] != "/var/lib/libvirt/images/base-debian-13.qcow2" {
t.Errorf("caller %d: path=%q err=%v", i, paths[i], errs[i])
}
}
if srv.hits.Load() != 1 {
t.Errorf("http hits = %d, want 1 for four concurrent callers", srv.hits.Load())
}
}
func TestImageCache_ensure_differentHostsFetchIndependently(t *testing.T) {
t.Parallel()
srv := newImageServer(t)
c := newImageCache(nil)
src := testSource(srv, "")
for _, host := range []string{"nas1", "lab2"} {
api := newFakeAPI("default")
pool, _ := api.StoragePoolLookupByName("default")
if _, err := c.ensure(context.Background(), logr.Discard(), api, host, pool, src); err != nil {
t.Fatalf("ensure(%s) error = %v", host, err)
}
}
if srv.hits.Load() != 2 {
t.Errorf("http hits = %d, want one per host", srv.hits.Load())
}
}
func TestImageCache_ensure_cancelledContextIsTransient(t *testing.T) {
t.Parallel()
srv := newImageServer(t)
api := newFakeAPI("default")
c := newImageCache(nil)
pool, _ := api.StoragePoolLookupByName("default")
ctx, cancel := context.WithCancel(context.Background())
cancel()
_, err := c.ensure(ctx, logr.Discard(), api, "nas1", pool, testSource(srv, ""))
if classify(err) != provider.ErrTransient {
t.Fatalf("ensure() error = %v, want transient", err)
}
if names := api.volNames("default"); len(names) != 0 {
t.Errorf("volumes created despite cancelled context: %v", names)
}
}
+560
View File
@@ -0,0 +1,560 @@
// Package kvm implements the provider contract on libvirt/KVM hypervisors.
// Each configured host is reached over SSH with the libvirt unix socket
// forwarded through the connection, so the hypervisors need no libvirtd TCP
// or TLS listener and the operator image needs no libvirt libraries. VMs
// boot a stock cloud image (fetched into the host's storage pool on first
// use) through a qcow2 overlay, with the Proxy's cloud-init delivered on a
// NoCloud seed ISO built in-process.
package kvm
import (
"bytes"
"context"
"fmt"
"net"
"net/http"
"time"
libvirt "github.com/digitalocean/go-libvirt"
"github.com/go-logr/logr"
logf "sigs.k8s.io/controller-runtime/pkg/log"
"gitea.home.hrajfrisbee.cz/kacerr/egress-proxies-operator/internal/provider"
)
const (
defaultDiskSizeGB = 10
defaultSocket = "/var/run/libvirt/libvirt-sock"
defaultPool = "default"
formatQcow2 = "qcow2"
formatRaw = "raw"
// undefineFlags removes everything libvirt may keep alongside a domain
// definition, so an undefine never fails on leftover state.
undefineFlags = libvirt.DomainUndefineManagedSave |
libvirt.DomainUndefineSnapshotsMetadata |
libvirt.DomainUndefineNvram
)
func withDefaults(cfg provider.KVMConfig) provider.KVMConfig {
if cfg.DiskSizeGB == 0 {
cfg.DiskSizeGB = defaultDiskSizeGB
}
if cfg.VendorData == "" {
cfg.VendorData = defaultVendorData
}
images := make(map[string]provider.KVMImage, len(cfg.Images))
for k, img := range cfg.Images {
if img.Format == "" {
img.Format = formatQcow2
}
images[k] = img
}
cfg.Images = images
hosts := make([]provider.KVMHost, len(cfg.Hosts))
for i, h := range cfg.Hosts {
if h.Socket == "" {
h.Socket = defaultSocket
}
if h.Pool == "" {
h.Pool = defaultPool
}
hosts[i] = h
}
cfg.Hosts = hosts
return cfg
}
// Provider implements provider.Provider on one or more libvirt hosts.
type Provider struct {
name string
cfg provider.KVMConfig
hosts *hostPool
images *imageCache
now func() time.Time
}
// New builds a Provider that reaches its hosts over SSH. The key and
// known_hosts files are read here so a bad path fails at startup rather
// than on the first Create. Satisfies registry.Constructor.
func New(_ context.Context, pc provider.ProviderConfig) (provider.Provider, error) {
cfg := provider.KVMConfig{}
if pc.KVM != nil {
cfg = *pc.KVM
}
auth, err := newSSHAuth(cfg)
if err != nil {
return nil, fmt.Errorf("kvm provider %q: %w", pc.Name, err)
}
return newWithDialer(pc, auth.dialer(), nil), nil
}
// newWithDialer is the test seam: the dialer decides what libvirtAPI each
// host resolves to, and the HTTP client is what image downloads go through.
func newWithDialer(pc provider.ProviderConfig, dial hostDialer, httpClient *http.Client) *Provider {
cfg := provider.KVMConfig{}
if pc.KVM != nil {
cfg = *pc.KVM
}
return &Provider{
name: pc.Name,
cfg: withDefaults(cfg),
hosts: newHostPool(dial),
images: newImageCache(httpClient),
now: time.Now,
}
}
func (p *Provider) logger(ctx context.Context) logr.Logger {
return logf.FromContext(ctx).WithName("kvm").WithValues("provider", p.name)
}
func (p *Provider) wrapErr(op, id string, err error) error {
return provider.Wrap(classify(err), op, p.name, id, err)
}
// host resolves placement.zone (or a providerID's host segment) to a
// configured host. An empty zone is allowed only when exactly one host is
// configured; anything else is a spec error.
func (p *Provider) host(zone string) (provider.KVMHost, error) {
if zone == "" {
switch len(p.cfg.Hosts) {
case 0:
return provider.KVMHost{}, fmt.Errorf("%w: kvm.hosts is empty", provider.ErrPermanent)
case 1:
return p.cfg.Hosts[0], nil
default:
return provider.KVMHost{}, fmt.Errorf("%w: placement.zone is required when kvm.hosts lists more than one host",
provider.ErrPermanent)
}
}
for _, h := range p.cfg.Hosts {
if h.Name == zone {
return h, nil
}
}
return provider.KVMHost{}, fmt.Errorf("%w: host %q is not in kvm.hosts", provider.ErrPermanent, zone)
}
// Create makes the VM exist and running on the placed host, and is
// idempotent: a defined domain is only started if needed, its volumes
// untouched. Before the domain exists nothing can be using the volumes,
// so an existing overlay is reused (its creation is atomic) but the seed
// is rebuilt from scratch — a previous attempt may have been cut off mid
// upload, and a truncated seed would boot a VM without cloud-init. The
// first Create for an image on a host also downloads that image into the
// host's pool, synchronously.
func (p *Provider) Create(ctx context.Context, req provider.CreateRequest) (string, error) {
pl := req.Placement
host, err := p.host(pl.Zone)
if err != nil {
return "", p.wrapErr("create", "", err)
}
mt, ok := p.cfg.MachineTypes[pl.MachineType]
if !ok {
return "", p.wrapErr("create", "", fmt.Errorf("%w: placement.machineType %q is not in kvm.machineTypes",
provider.ErrPermanent, pl.MachineType))
}
src, err := resolveImage(p.cfg, pl.Image)
if err != nil {
return "", p.wrapErr("create", "", err)
}
id := formatProviderID(host.Name, req.Name)
log := p.logger(ctx).WithValues("host", host.Name, "name", req.Name)
log.V(2).Info("KVM create resolved",
"pool", host.Pool, "bridge", host.Bridge, "machineType", pl.MachineType,
"vcpus", mt.VCPUs, "memoryMiB", mt.MemoryMiB, "diskSizeGb", p.cfg.DiskSizeGB,
"image", src.volumeName(), "cloudInitBytes", len(req.CloudInit))
err = p.hosts.call(ctx, host, func(api libvirtAPI) error {
dom, err := api.DomainLookupByName(req.Name)
if err == nil {
log.V(1).Info("domain already defined")
return p.ensureRunning(log, api, dom)
}
if !hasCode(err, libvirt.ErrNoDomain) {
return fmt.Errorf("looking up domain: %w", err)
}
pool, err := api.StoragePoolLookupByName(host.Pool)
if err != nil {
return fmt.Errorf("looking up pool %s: %w", host.Pool, err)
}
basePath, err := p.images.ensure(ctx, log, api, host.Name, pool, src)
if err != nil {
return err
}
if err := p.ensureOverlay(log, api, pool, req.Name, basePath, src.Format); err != nil {
return err
}
if err := p.writeSeed(log, api, pool, req.Name, req.CloudInit); err != nil {
return err
}
dom, err = p.defineDomain(log, api, host, req, mt)
if err != nil {
return err
}
return p.ensureRunning(log, api, dom)
})
if err != nil {
logAPIError(log, "create", err)
return "", p.wrapErr("create", id, err)
}
log.V(1).Info("KVM domain ensured running", "id", id)
return id, nil
}
func (p *Provider) ensureOverlay(log logr.Logger, api libvirtAPI, pool libvirt.StoragePool, name, basePath, baseFormat string) error {
xml, err := buildVolumeXML(overlayVolume(name, p.cfg.DiskSizeGB, basePath, baseFormat))
if err != nil {
return err
}
_, err = api.StorageVolCreateXML(pool, xml, 0)
switch {
case err == nil:
log.V(1).Info("overlay volume created", "volume", overlayVolumeName(name), "base", basePath)
case hasCode(err, libvirt.ErrStorageVolExist):
log.V(1).Info("overlay volume already exists", "volume", overlayVolumeName(name))
default:
return fmt.Errorf("creating overlay volume: %w", err)
}
return nil
}
// writeSeed replaces the seed volume with a freshly built NoCloud ISO.
// Only called while no domain references it.
func (p *Provider) writeSeed(log logr.Logger, api libvirtAPI, pool libvirt.StoragePool, name, userData string) error {
seedName := seedVolumeName(name)
if err := deleteIfExists(api, pool, seedName); err != nil {
return err
}
iso, err := buildSeedISO(name, userData, p.cfg.VendorData)
if err != nil {
return fmt.Errorf("%w: %v", provider.ErrPermanent, err)
}
xml, err := buildVolumeXML(rawVolume(seedName, int64(len(iso))))
if err != nil {
return err
}
vol, err := api.StorageVolCreateXML(pool, xml, 0)
if err != nil {
return fmt.Errorf("creating seed volume: %w", err)
}
if err := api.StorageVolUpload(vol, bytes.NewReader(iso), 0, 0, 0); err != nil {
return fmt.Errorf("uploading seed volume: %w", err)
}
log.V(1).Info("seed volume uploaded", "volume", seedName, "bytes", len(iso))
return nil
}
func (p *Provider) defineDomain(log logr.Logger, api libvirtAPI, host provider.KVMHost, req provider.CreateRequest, mt provider.KVMMachineType) (libvirt.Domain, error) {
xml, err := buildDomainXML(domainParams{
Name: req.Name,
UID: req.UID,
CreatedAt: p.now(),
VCPUs: mt.VCPUs,
MemoryMiB: mt.MemoryMiB,
Pool: host.Pool,
Bridge: host.Bridge,
DiskVol: overlayVolumeName(req.Name),
SeedVol: seedVolumeName(req.Name),
})
if err != nil {
return libvirt.Domain{}, err
}
dom, err := api.DomainDefineXML(xml)
if err != nil {
return libvirt.Domain{}, fmt.Errorf("defining domain: %w", err)
}
log.V(1).Info("domain defined", "uid", req.UID)
return dom, nil
}
func (p *Provider) ensureRunning(log logr.Logger, api libvirtAPI, dom libvirt.Domain) error {
state, _, err := api.DomainGetState(dom, 0)
if err != nil {
return fmt.Errorf("reading domain state: %w", err)
}
if isActive(libvirt.DomainState(state)) {
return nil
}
err = api.DomainCreate(dom)
switch {
case err == nil:
log.V(1).Info("domain started")
case hasCode(err, libvirt.ErrOperationInvalid):
log.V(1).Info("domain already running, start treated as success")
default:
return fmt.Errorf("starting domain: %w", err)
}
return nil
}
// Get reads the domain's state, metadata and addresses. The providerID
// carries the host, so this looks at the right hypervisor even after a
// placement.zone edit.
func (p *Provider) Get(ctx context.Context, providerID string) (*provider.Instance, error) {
hostName, name, err := parseProviderID(providerID)
if err != nil {
return nil, provider.Wrap(provider.ErrPermanent, "get", p.name, providerID, err)
}
host, err := p.host(hostName)
if err != nil {
return nil, p.wrapErr("get", providerID, err)
}
log := p.logger(ctx).WithValues("host", host.Name, "name", name)
var inst *provider.Instance
err = p.hosts.call(ctx, host, func(api libvirtAPI) error {
dom, err := api.DomainLookupByName(name)
if err != nil {
return fmt.Errorf("looking up domain: %w", err)
}
meta, err := p.metadata(api, dom)
if err != nil {
return err
}
inst, err = p.inspect(log, api, host.Name, dom, meta)
return err
})
if err != nil {
logAPIError(log, "get", err)
return nil, p.wrapErr("get", providerID, err)
}
log.V(1).Info("KVM domain fetched", "state", inst.State, "ip", inst.IP, "uid", inst.UID)
return inst, nil
}
// inspect assembles an Instance from a domain that is known to exist and
// whose metadata has already been read (the zero value for a domain
// without the operator's element, which reports an empty UID as the
// contract requires).
func (p *Provider) inspect(log logr.Logger, api libvirtAPI, hostName string, dom libvirt.Domain, meta instanceMetadata) (*provider.Instance, error) {
state, _, err := api.DomainGetState(dom, 0)
if err != nil {
return nil, fmt.Errorf("reading domain state: %w", err)
}
var ip string
if isActive(libvirt.DomainState(state)) {
ip = p.address(log, api, dom)
}
return &provider.Instance{
ID: formatProviderID(hostName, dom.Name),
IP: ip,
State: mapState(libvirt.DomainState(state), ip),
UID: meta.UID,
CreatedAt: meta.createdAt(),
}, nil
}
// metadata returns the operator's element, or the zero value when the
// domain has none (it was not created by this operator).
func (p *Provider) metadata(api libvirtAPI, dom libvirt.Domain) (instanceMetadata, error) {
raw, err := api.DomainGetMetadata(dom, int32(libvirt.DomainMetadataElement), libvirt.OptString{metadataURI}, libvirt.DomainAffectCurrent)
if hasCode(err, libvirt.ErrNoDomainMetadata) {
return instanceMetadata{}, nil
}
if err != nil {
return instanceMetadata{}, fmt.Errorf("reading domain metadata: %w", err)
}
meta, err := parseInstanceMetadata(raw)
if err != nil {
return instanceMetadata{}, fmt.Errorf("%w: %v", provider.ErrPermanent, err)
}
return meta, nil
}
// address returns the domain's first routable IPv4, asking the host's ARP
// table first and the guest agent second. Both sources fail routinely while
// the guest is still booting (no traffic seen yet, agent not installed
// yet), so their errors mean "no address yet", not "Get failed".
func (p *Provider) address(log logr.Logger, api libvirtAPI, dom libvirt.Domain) string {
for _, source := range []libvirt.DomainInterfaceAddressesSource{
libvirt.DomainInterfaceAddressesSrcArp,
libvirt.DomainInterfaceAddressesSrcAgent,
} {
ifaces, err := api.DomainInterfaceAddresses(dom, uint32(source), 0)
if err != nil {
log.V(2).Info("no addresses from source", "source", source.String(), "error", err.Error())
continue
}
if ip := firstIPv4(ifaces); ip != "" {
return ip
}
}
return ""
}
func firstIPv4(ifaces []libvirt.DomainInterface) string {
for _, iface := range ifaces {
for _, a := range iface.Addrs {
if a.Type != int32(libvirt.IPAddrTypeIpv4) {
continue
}
ip := net.ParseIP(a.Addr)
if ip == nil || ip.IsLoopback() || ip.IsLinkLocalUnicast() || ip.IsUnspecified() {
continue
}
return a.Addr
}
}
return ""
}
// isActive reports whether the guest is executing (or blocked on the
// hypervisor, which is still "up" from the outside).
func isActive(s libvirt.DomainState) bool {
return s == libvirt.DomainRunning || s == libvirt.DomainBlocked
}
// mapState collapses libvirt's states onto the provider's. An active
// domain without an address is still Provisioning: an empty IP must never
// be published as Running. "No state" is also Provisioning — the
// reconciler answers Stopped with delete-and-recreate, too drastic for a
// state libvirt could not determine; the next poll decides. Everything
// else — paused, suspended, shut off, crashed, shutting down — is Stopped.
func mapState(s libvirt.DomainState, ip string) provider.InstanceState {
if s == libvirt.DomainNostate {
return provider.StateProvisioning
}
if !isActive(s) {
return provider.StateStopped
}
if ip == "" {
return provider.StateProvisioning
}
return provider.StateRunning
}
// Delete tears the VM down and is idempotent at every step: a stopped
// domain is not destroyed again, a missing domain is not undefined, and
// the per-VM volumes are removed by name even when the domain is already
// gone, so a crash between volume creation and define is cleaned up on the
// next reconcile.
func (p *Provider) Delete(ctx context.Context, providerID string) error {
hostName, name, err := parseProviderID(providerID)
if err != nil {
return provider.Wrap(provider.ErrPermanent, "delete", p.name, providerID, err)
}
host, err := p.host(hostName)
if err != nil {
return p.wrapErr("delete", providerID, err)
}
log := p.logger(ctx).WithValues("host", host.Name, "name", name)
err = p.hosts.call(ctx, host, func(api libvirtAPI) error {
if err := p.removeDomain(log, api, name); err != nil {
return err
}
return p.removeVolumes(log, api, host.Pool, name)
})
if err != nil {
logAPIError(log, "delete", err)
return p.wrapErr("delete", providerID, err)
}
log.V(1).Info("KVM domain deleted")
return nil
}
func (p *Provider) removeDomain(log logr.Logger, api libvirtAPI, name string) error {
dom, err := api.DomainLookupByName(name)
if hasCode(err, libvirt.ErrNoDomain) {
log.V(1).Info("domain already gone")
return nil
}
if err != nil {
return fmt.Errorf("looking up domain: %w", err)
}
err = api.DomainDestroy(dom)
switch {
case err == nil:
log.V(1).Info("domain destroyed")
case hasCode(err, libvirt.ErrOperationInvalid):
log.V(1).Info("domain not running, destroy skipped")
case hasCode(err, libvirt.ErrNoDomain):
return nil
default:
return fmt.Errorf("destroying domain: %w", err)
}
if err := api.DomainUndefineFlags(dom, undefineFlags); err != nil && !hasCode(err, libvirt.ErrNoDomain) {
return fmt.Errorf("undefining domain: %w", err)
}
log.V(1).Info("domain undefined")
return nil
}
// removeVolumes deletes the overlay and seed. A missing pool means there
// is nothing left to clean, not a failure: refusing to finish a Delete
// over a pool that no longer exists would pin the Proxy's finalizer.
func (p *Provider) removeVolumes(log logr.Logger, api libvirtAPI, poolName, name string) error {
pool, err := api.StoragePoolLookupByName(poolName)
if hasCode(err, libvirt.ErrNoStoragePool) {
log.Info("pool not found, no volumes to delete", "pool", poolName)
return nil
}
if err != nil {
return fmt.Errorf("looking up pool %s: %w", poolName, err)
}
for _, vol := range []string{overlayVolumeName(name), seedVolumeName(name)} {
if err := deleteIfExists(api, pool, vol); err != nil {
return err
}
}
log.V(1).Info("per-VM volumes deleted")
return nil
}
// ListByTag inspects every domain on every host and keeps those carrying
// the operator's managed metadata. One unreachable host fails the whole
// sweep, so orphan GC never acts on a partial view of the fleet; a single
// domain that cannot be read is skipped instead, which is safe in the
// same direction — a domain missing from the list is never collected.
func (p *Provider) ListByTag(ctx context.Context) ([]provider.Instance, error) {
log := p.logger(ctx)
var out []provider.Instance
for _, host := range p.cfg.Hosts {
hlog := log.WithValues("host", host.Name)
err := p.hosts.call(ctx, host, func(api libvirtAPI) error {
doms, _, err := api.ConnectListAllDomains(1, 0)
if err != nil {
return fmt.Errorf("listing domains: %w", err)
}
for _, dom := range doms {
if inst := p.managedInstance(hlog, api, host.Name, dom); inst != nil {
out = append(out, *inst)
}
}
return nil
})
if err != nil {
logAPIError(hlog, "list", err)
return nil, p.wrapErr("list", "", err)
}
}
log.V(1).Info("KVM domains listed", "hosts", len(p.cfg.Hosts), "managed", len(out))
for _, inst := range out {
log.V(2).Info("KVM listed instance", "id", inst.ID, "state", inst.State, "uid", inst.UID, "createdAt", inst.CreatedAt)
}
return out, nil
}
// managedInstance returns nil for domains the operator does not own, for
// domains that vanished between the listing and the inspection, and for
// domains whose metadata or state cannot be read (corrupt element, mid
// teardown): one bad domain must not block GC for every other one. A
// transport failure would also land here, but it fails the next RPC in
// the same sweep too, so the host still counts as unreachable.
func (p *Provider) managedInstance(log logr.Logger, api libvirtAPI, hostName string, dom libvirt.Domain) *provider.Instance {
log = log.WithValues("name", dom.Name)
meta, err := p.metadata(api, dom)
if err != nil {
log.V(1).Info("skipping domain with unreadable metadata", "error", err.Error())
return nil
}
if !meta.managed() {
return nil
}
inst, err := p.inspect(log, api, hostName, dom, meta)
if err != nil {
log.V(1).Info("skipping domain that could not be inspected", "error", err.Error())
return nil
}
return inst
}
+880
View File
@@ -0,0 +1,880 @@
package kvm
import (
"context"
"encoding/xml"
"errors"
"fmt"
"io"
"strings"
"testing"
"time"
libvirt "github.com/digitalocean/go-libvirt"
"gitea.home.hrajfrisbee.cz/kacerr/egress-proxies-operator/internal/provider"
)
func TestWithDefaults(t *testing.T) {
t.Parallel()
cfg := withDefaults(provider.KVMConfig{
Images: map[string]provider.KVMImage{
"debian": {URL: "https://x/d.qcow2"},
"raw": {URL: "https://x/r.img", Format: "raw"},
},
Hosts: []provider.KVMHost{
{Name: "a", Address: "a:22", User: "virt", Bridge: "br0"},
{Name: "b", Address: "b:22", User: "virt", Bridge: "br1", Socket: "/run/s", Pool: "fast"},
},
})
if cfg.DiskSizeGB != defaultDiskSizeGB {
t.Errorf("diskSizeGb = %d, want %d", cfg.DiskSizeGB, defaultDiskSizeGB)
}
if cfg.VendorData != defaultVendorData {
t.Errorf("vendorData = %q, want the default", cfg.VendorData)
}
if cfg.Images["debian"].Format != "qcow2" || cfg.Images["raw"].Format != "raw" {
t.Errorf("image formats = %q/%q, want qcow2 default and raw kept", cfg.Images["debian"].Format, cfg.Images["raw"].Format)
}
if a := cfg.Hosts[0]; a.Socket != defaultSocket || a.Pool != defaultPool {
t.Errorf("host a = %+v, want default socket and pool", a)
}
if b := cfg.Hosts[1]; b.Socket != "/run/s" || b.Pool != "fast" {
t.Errorf("host b = %+v, want explicit socket and pool kept", b)
}
}
func TestWithDefaults_keepsExplicitValues(t *testing.T) {
t.Parallel()
cfg := withDefaults(provider.KVMConfig{DiskSizeGB: 42, VendorData: "#cloud-config\n"})
if cfg.DiskSizeGB != 42 || cfg.VendorData != "#cloud-config\n" {
t.Errorf("cfg = %+v, want explicit values kept", cfg)
}
}
// --- provider over fake hosts ---
const (
testName = "proxy-abc123def456ghij"
testUID = "11111111-2222-3333-4444-555555555555"
testImageURL = "https://images.example/debian-13.qcow2"
testCloudInit = "#cloud-config\npackages: [squid]\n"
)
var (
testNow = time.Date(2026, 9, 8, 10, 0, 0, 0, time.UTC)
errHostOff = errors.New("ssh dial: connection refused")
)
// hostFakes routes each configured host name to its own fakeAPI; a host
// with no entry is unreachable.
type hostFakes map[string]*fakeAPI
func (h hostFakes) dial(_ context.Context, host provider.KVMHost) (*hostClient, error) {
api, ok := h[host.Name]
if !ok {
return nil, errHostOff
}
return &hostClient{api: api, close: func() {}}, nil
}
func testConfig(hosts ...string) provider.ProviderConfig {
cfg := &provider.KVMConfig{
SSHKeyFile: "/k",
KnownHostsFile: "/kh",
Images: map[string]provider.KVMImage{"debian-13": {URL: testImageURL}},
MachineTypes: map[string]provider.KVMMachineType{"small": {VCPUs: 2, MemoryMiB: 2048}},
}
for _, h := range hosts {
cfg.Hosts = append(cfg.Hosts, provider.KVMHost{Name: h, Address: h + ":22", User: "virt", Bridge: "br0"})
}
return provider.ProviderConfig{Name: "kvm-home", Type: "kvm", KVM: cfg}
}
func newTestProvider(fakes hostFakes, hosts ...string) *Provider {
p := newWithDialer(testConfig(hosts...), fakes.dial, nil)
p.now = func() time.Time { return testNow }
return p
}
// cacheImage pre-populates the pool with the configured image and its
// marker so Create never touches the network.
func cacheImage(api *fakeAPI) {
src := imageSource{Key: "debian-13", URL: testImageURL, Format: formatQcow2}
api.putVol("default", src.volumeName(), formatQcow2, []byte("qcow2"))
api.putVol("default", src.markerName(), formatRaw, []byte(testImageSHA()))
}
func readyHost() *fakeAPI {
api := newFakeAPI("default")
cacheImage(api)
return api
}
func testRequest() provider.CreateRequest {
return provider.CreateRequest{
Name: testName,
UID: testUID,
Namespace: "default",
ProxyName: "home-proxy-1",
Placement: provider.Placement{Zone: "nas1", MachineType: "small", Image: "debian-13"},
CloudInit: testCloudInit,
Port: 3128,
}
}
func metadataXML(t *testing.T, uid string, created time.Time) string {
t.Helper()
out, err := xml.Marshal(newInstanceMetadata(uid, created))
if err != nil {
t.Fatal(err)
}
return string(out)
}
func ipv4(addr string) libvirt.DomainIPAddr {
return libvirt.DomainIPAddr{Type: int32(libvirt.IPAddrTypeIpv4), Addr: addr, Prefix: 24}
}
func ipv6(addr string) libvirt.DomainIPAddr {
return libvirt.DomainIPAddr{Type: int32(libvirt.IPAddrTypeIpv6), Addr: addr, Prefix: 64}
}
// managedDomain seeds a domain as Create would have left it.
func managedDomain(t *testing.T, api *fakeAPI, name, uid string, running bool, addrs map[uint32][]libvirt.DomainInterface) {
t.Helper()
api.putDomain(&fakeDomain{name: name, metadata: metadataXML(t, uid, testNow), running: running, addrs: addrs})
api.putVol("default", overlayVolumeName(name), formatQcow2, nil)
api.putVol("default", seedVolumeName(name), formatRaw, []byte("iso"))
}
func wantClass(t *testing.T, err error, class error) {
t.Helper()
if err == nil {
t.Fatalf("got nil error, want %v", class)
}
if !errors.Is(err, class) {
t.Fatalf("error class = %v, want %v (err: %v)", provider.Class(err), class, err)
}
}
func assertCreated(t *testing.T, api *fakeAPI) {
t.Helper()
dom := api.domain(testName)
if dom == nil {
t.Fatal("domain not defined")
}
if !dom.running {
t.Error("domain not running")
}
overlay := api.vol("default", overlayVolumeName(testName))
if overlay == nil {
t.Fatal("overlay volume missing")
}
if overlay.backing != "/var/lib/libvirt/images/base-debian-13.qcow2" || overlay.format != formatQcow2 {
t.Errorf("overlay = %+v, want qcow2 backed by the cached base", overlay)
}
if overlay.capacity != defaultDiskSizeGB*bytesPerGiB {
t.Errorf("overlay capacity = %d, want %d", overlay.capacity, defaultDiskSizeGB*bytesPerGiB)
}
seed := api.vol("default", seedVolumeName(testName))
if seed == nil {
t.Fatal("seed volume missing")
}
if seed.format != formatRaw || len(seed.data) == 0 || int64(len(seed.data)) != seed.capacity {
t.Errorf("seed = format %s, %d bytes, capacity %d; want raw with capacity == uploaded size", seed.format, len(seed.data), seed.capacity)
}
}
func TestCreate_freshHost(t *testing.T) {
t.Parallel()
api := readyHost()
p := newTestProvider(hostFakes{"nas1": api}, "nas1")
id, err := p.Create(context.Background(), testRequest())
if err != nil {
t.Fatalf("Create: %v", err)
}
if want := "hosts/nas1/domains/" + testName; id != want {
t.Errorf("providerID = %s, want %s", id, want)
}
assertCreated(t, api)
var d domainXML
if err := xml.Unmarshal([]byte(api.domain(testName).xml), &d); err != nil {
t.Fatal(err)
}
if d.VCPU != 2 || d.Memory.Value != 2048 || d.Memory.Unit != "MiB" {
t.Errorf("domain size = %d vcpu / %d %s, want 2 / 2048 MiB", d.VCPU, d.Memory.Value, d.Memory.Unit)
}
if d.Devices.Interfaces[0].Source.Bridge != "br0" {
t.Errorf("bridge = %q, want br0", d.Devices.Interfaces[0].Source.Bridge)
}
if m := d.Metadata.Instance; m.UID != testUID || !m.managed() || !m.createdAt().Equal(testNow) {
t.Errorf("metadata = %+v, want managed, uid %s, createdAt %s", m, testUID, testNow)
}
if api.hasCall("StorageVolDownload") || api.callCount("StorageVolUpload") != 1 {
t.Errorf("calls = %v, want no image download and exactly one upload (the seed)", api.calls)
}
}
func TestCreate_fetchesImageOnFirstUse(t *testing.T) {
t.Parallel()
srv := newImageServer(t)
api := newFakeAPI("default")
cfg := testConfig("nas1")
cfg.KVM.Images["debian-13"] = provider.KVMImage{URL: srv.URL + "/debian.qcow2", SHA256: testImageSHA()}
p := newWithDialer(cfg, hostFakes{"nas1": api}.dial, nil)
if _, err := p.Create(context.Background(), testRequest()); err != nil {
t.Fatalf("Create: %v", err)
}
if srv.hits.Load() != 1 {
t.Errorf("image server hits = %d, want 1", srv.hits.Load())
}
base := api.vol("default", "base-debian-13.qcow2")
if base == nil || string(base.data) != string(testImage) {
t.Fatal("base image not cached in the pool")
}
if api.domain(testName) == nil || !api.domain(testName).running {
t.Error("domain not running after image fetch")
}
// A second Proxy with the same image on the same host reuses the cache.
req := testRequest()
req.Name, req.UID = "proxy-second", "uid-2"
if _, err := p.Create(context.Background(), req); err != nil {
t.Fatalf("second Create: %v", err)
}
if srv.hits.Load() != 1 {
t.Errorf("image server hits after second Create = %d, want still 1", srv.hits.Load())
}
}
func TestCreate_isIdempotentWhenComplete(t *testing.T) {
t.Parallel()
api := readyHost()
p := newTestProvider(hostFakes{"nas1": api}, "nas1")
if _, err := p.Create(context.Background(), testRequest()); err != nil {
t.Fatal(err)
}
before := len(api.calls)
id, err := p.Create(context.Background(), testRequest())
if err != nil {
t.Fatalf("second Create: %v", err)
}
if id != "hosts/nas1/domains/"+testName {
t.Errorf("providerID = %s", id)
}
for _, m := range []string{"DomainDefineXML", "DomainCreate", "StorageVolUpload"} {
if n := api.callCount(m); n != 1 {
t.Errorf("%s called %d times across two Creates, want 1", m, n)
}
}
if len(api.calls) == before {
t.Error("second Create made no calls at all; expected lookups")
}
}
// Each case breaks Create after a different step, then re-runs it and
// expects the second attempt to pick up where the first stopped without
// duplicating what already exists.
func TestCreate_resumesAfterPartialFailure(t *testing.T) {
t.Parallel()
cases := []struct {
name string
sabotage func(api *fakeAPI)
class error
check func(t *testing.T, api *fakeAPI)
}{
{
name: "seed upload fails after overlay exists",
sabotage: func(api *fakeAPI) {
api.uploadHook = func(vol string, _ io.Reader) error {
api.uploadHook = nil
return io.ErrUnexpectedEOF
}
},
class: provider.ErrTransient,
check: func(t *testing.T, api *fakeAPI) {
if n := api.callCount("StorageVolUpload"); n != 2 {
t.Errorf("seed uploads = %d, want 2 (failed, then redone)", n)
}
if n := api.callCount("StorageVolCreateXML"); n != 4 {
t.Errorf("volume creates = %d, want 4 (overlay, seed; overlay rejected as existing, seed again)", n)
}
if names := api.volNames("default"); len(names) != 4 {
t.Errorf("pool holds %v, want base, marker, one overlay, one seed", names)
}
},
},
{
name: "define fails after volumes exist",
sabotage: func(api *fakeAPI) { api.failOnce["DomainDefineXML"] = lvError(libvirt.ErrInternalError, "boom") },
class: provider.ErrTransient,
check: func(t *testing.T, api *fakeAPI) {
if n := api.callCount("StorageVolUpload"); n != 2 {
t.Errorf("seed uploads = %d, want 2 (seed rebuilt while no domain exists)", n)
}
if n := api.callCount("DomainDefineXML"); n != 2 {
t.Errorf("defines = %d, want 2", n)
}
},
},
{
name: "start fails after define",
sabotage: func(api *fakeAPI) { api.failOnce["DomainCreate"] = lvError(libvirt.ErrInternalError, "boom") },
class: provider.ErrTransient,
check: func(t *testing.T, api *fakeAPI) {
if n := api.callCount("DomainDefineXML"); n != 1 {
t.Errorf("defines = %d, want 1 (existing domain reused)", n)
}
if n := api.callCount("DomainCreate"); n != 2 {
t.Errorf("starts = %d, want 2", n)
}
},
},
{
name: "transport drops mid-create",
sabotage: func(api *fakeAPI) { api.failOnce["DomainLookupByName"] = io.EOF },
class: provider.ErrTransient,
check: func(t *testing.T, api *fakeAPI) {
if n := api.callCount("DomainDefineXML"); n != 1 {
t.Errorf("defines = %d, want 1", n)
}
},
},
}
for _, tc := range cases {
t.Run(tc.name, func(t *testing.T) {
t.Parallel()
api := readyHost()
tc.sabotage(api)
p := newTestProvider(hostFakes{"nas1": api}, "nas1")
_, err := p.Create(context.Background(), testRequest())
wantClass(t, err, tc.class)
if _, err := p.Create(context.Background(), testRequest()); err != nil {
t.Fatalf("retry Create: %v", err)
}
assertCreated(t, api)
tc.check(t, api)
})
}
}
// A reconcile cancelled mid-upload leaves a seed volume whose content is
// short; the compensating delete cannot run because the pool has already
// closed the connection. The next Create must not trust that volume.
func TestCreate_replacesLeftoverSeed(t *testing.T) {
t.Parallel()
api := readyHost()
api.putVol("default", seedVolumeName(testName), formatRaw, []byte("trunc"))
p := newTestProvider(hostFakes{"nas1": api}, "nas1")
if _, err := p.Create(context.Background(), testRequest()); err != nil {
t.Fatal(err)
}
assertCreated(t, api)
if seed := api.vol("default", seedVolumeName(testName)); string(seed.data) == "trunc" {
t.Error("leftover seed was kept")
}
if n := api.callCount("StorageVolUpload"); n != 1 {
t.Errorf("uploads = %d, want 1 (the rebuilt seed)", n)
}
}
func TestCreate_startsStoppedDomain(t *testing.T) {
t.Parallel()
api := readyHost()
managedDomain(t, api, testName, testUID, false, nil)
p := newTestProvider(hostFakes{"nas1": api}, "nas1")
if _, err := p.Create(context.Background(), testRequest()); err != nil {
t.Fatal(err)
}
if !api.domain(testName).running {
t.Error("domain still stopped")
}
if api.hasCall("DomainDefineXML") || api.hasCall("StorageVolUpload") || api.hasCall("StorageVolDelete") {
t.Errorf("calls = %v, want no define, upload or volume delete for an existing domain", api.calls)
}
}
func TestCreate_singleHostNeedsNoZone(t *testing.T) {
t.Parallel()
api := readyHost()
p := newTestProvider(hostFakes{"nas1": api}, "nas1")
req := testRequest()
req.Placement.Zone = ""
id, err := p.Create(context.Background(), req)
if err != nil {
t.Fatal(err)
}
if id != "hosts/nas1/domains/"+testName {
t.Errorf("providerID = %s, want the single host filled in", id)
}
}
func TestCreate_placementErrors(t *testing.T) {
t.Parallel()
cases := []struct {
name string
hosts []string
edit func(*provider.CreateRequest)
want string
}{
{"unknown host", []string{"nas1"}, func(r *provider.CreateRequest) { r.Placement.Zone = "nas9" }, "not in kvm.hosts"},
{"empty zone with two hosts", []string{"nas1", "nas2"}, func(r *provider.CreateRequest) { r.Placement.Zone = "" }, "placement.zone is required"},
{"empty zone with no hosts", nil, func(r *provider.CreateRequest) { r.Placement.Zone = "" }, "kvm.hosts is empty"},
{"unknown machine type", []string{"nas1"}, func(r *provider.CreateRequest) { r.Placement.MachineType = "huge" }, "not in kvm.machineTypes"},
{"empty machine type", []string{"nas1"}, func(r *provider.CreateRequest) { r.Placement.MachineType = "" }, "not in kvm.machineTypes"},
{"unknown image", []string{"nas1"}, func(r *provider.CreateRequest) { r.Placement.Image = "windows" }, "neither a key"},
}
for _, tc := range cases {
t.Run(tc.name, func(t *testing.T) {
t.Parallel()
fakes := hostFakes{}
for _, h := range tc.hosts {
fakes[h] = readyHost()
}
p := newTestProvider(fakes, tc.hosts...)
req := testRequest()
tc.edit(&req)
_, err := p.Create(context.Background(), req)
wantClass(t, err, provider.ErrPermanent)
if !strings.Contains(err.Error(), tc.want) {
t.Errorf("error = %q, want it to mention %q", err, tc.want)
}
for _, api := range fakes {
if len(api.calls) != 0 {
t.Errorf("host was called: %v", api.calls)
}
}
})
}
}
func TestCreate_hostFailures(t *testing.T) {
t.Parallel()
cases := []struct {
name string
setup func() (hostFakes, string)
class error
}{
{
name: "host unreachable",
setup: func() (hostFakes, string) { return hostFakes{}, "nas1" },
class: provider.ErrTransient,
},
{
name: "pool missing",
setup: func() (hostFakes, string) {
api := newFakeAPI("other")
return hostFakes{"nas1": api}, "nas1"
},
class: provider.ErrPermanent,
},
{
name: "domain xml rejected",
setup: func() (hostFakes, string) {
api := readyHost()
api.fail["DomainDefineXML"] = lvError(libvirt.ErrConfigUnsupported, "no q35 here")
return hostFakes{"nas1": api}, "nas1"
},
class: provider.ErrPermanent,
},
}
for _, tc := range cases {
t.Run(tc.name, func(t *testing.T) {
t.Parallel()
fakes, host := tc.setup()
p := newTestProvider(fakes, host)
_, err := p.Create(context.Background(), testRequest())
wantClass(t, err, tc.class)
})
}
}
func TestCreate_cancelledContext(t *testing.T) {
t.Parallel()
api := readyHost()
p := newTestProvider(hostFakes{"nas1": api}, "nas1")
ctx, cancel := context.WithCancel(context.Background())
cancel()
_, err := p.Create(ctx, testRequest())
wantClass(t, err, provider.ErrTransient)
if !errors.Is(err, context.Canceled) {
t.Errorf("error = %v, want it to unwrap to context.Canceled", err)
}
}
func TestGet(t *testing.T) {
t.Parallel()
arp := map[uint32][]libvirt.DomainInterface{
uint32(libvirt.DomainInterfaceAddressesSrcArp): {{Name: "vnet0", Addrs: []libvirt.DomainIPAddr{ipv4("192.168.1.50")}}},
}
agentOnly := map[uint32][]libvirt.DomainInterface{
uint32(libvirt.DomainInterfaceAddressesSrcAgent): {
{Name: "lo", Addrs: []libvirt.DomainIPAddr{ipv4("127.0.0.1"), ipv6("::1")}},
{Name: "enp1s0", Addrs: []libvirt.DomainIPAddr{ipv6("fe80::1"), ipv4("192.168.1.51")}},
},
}
linkLocalOnly := map[uint32][]libvirt.DomainInterface{
uint32(libvirt.DomainInterfaceAddressesSrcAgent): {
{Name: "enp1s0", Addrs: []libvirt.DomainIPAddr{ipv4("169.254.10.10")}},
},
}
cases := []struct {
name string
running bool
addrs map[uint32][]libvirt.DomainInterface
state provider.InstanceState
ip string
}{
{"running with arp address", true, arp, provider.StateRunning, "192.168.1.50"},
{"running, agent only, skips loopback and v6", true, agentOnly, provider.StateRunning, "192.168.1.51"},
{"running, no address yet", true, nil, provider.StateProvisioning, ""},
{"running, only link-local", true, linkLocalOnly, provider.StateProvisioning, ""},
{"shut off", false, arp, provider.StateStopped, ""},
}
for _, tc := range cases {
t.Run(tc.name, func(t *testing.T) {
t.Parallel()
api := readyHost()
managedDomain(t, api, testName, testUID, tc.running, tc.addrs)
p := newTestProvider(hostFakes{"nas1": api}, "nas1")
inst, err := p.Get(context.Background(), "hosts/nas1/domains/"+testName)
if err != nil {
t.Fatalf("Get: %v", err)
}
want := provider.Instance{ID: "hosts/nas1/domains/" + testName, IP: tc.ip, State: tc.state, UID: testUID, CreatedAt: testNow}
if *inst != want {
t.Errorf("instance = %+v, want %+v", *inst, want)
}
})
}
}
func TestGet_unmanagedDomainHasNoUID(t *testing.T) {
t.Parallel()
api := readyHost()
api.putDomain(&fakeDomain{name: "hand-made", running: true})
p := newTestProvider(hostFakes{"nas1": api}, "nas1")
inst, err := p.Get(context.Background(), "hosts/nas1/domains/hand-made")
if err != nil {
t.Fatalf("Get: %v", err)
}
if inst.UID != "" || !inst.CreatedAt.IsZero() {
t.Errorf("instance = %+v, want empty UID and zero CreatedAt", inst)
}
}
func TestGet_errors(t *testing.T) {
t.Parallel()
cases := []struct {
name string
id string
setup func(t *testing.T, api *fakeAPI)
class error
}{
{"missing domain", "hosts/nas1/domains/" + testName, func(*testing.T, *fakeAPI) {}, provider.ErrNotFound},
{"malformed id", "zones/x/instances/y", func(*testing.T, *fakeAPI) {}, provider.ErrPermanent},
{"host not configured", "hosts/nas9/domains/" + testName, func(*testing.T, *fakeAPI) {}, provider.ErrPermanent},
{
"state read fails", "hosts/nas1/domains/" + testName,
func(t *testing.T, api *fakeAPI) {
managedDomain(t, api, testName, testUID, true, nil)
api.fail["DomainGetState"] = lvError(libvirt.ErrInternalError, "boom")
},
provider.ErrTransient,
},
{
"corrupt metadata", "hosts/nas1/domains/" + testName,
func(_ *testing.T, api *fakeAPI) {
api.putDomain(&fakeDomain{name: testName, metadata: "<instance", running: true})
},
provider.ErrPermanent,
},
}
for _, tc := range cases {
t.Run(tc.name, func(t *testing.T) {
t.Parallel()
api := readyHost()
tc.setup(t, api)
p := newTestProvider(hostFakes{"nas1": api}, "nas1")
_, err := p.Get(context.Background(), tc.id)
wantClass(t, err, tc.class)
})
}
}
func TestGet_hostUnreachable(t *testing.T) {
t.Parallel()
p := newTestProvider(hostFakes{}, "nas1")
_, err := p.Get(context.Background(), "hosts/nas1/domains/"+testName)
wantClass(t, err, provider.ErrTransient)
}
func assertGone(t *testing.T, api *fakeAPI) {
t.Helper()
if api.domain(testName) != nil {
t.Error("domain still defined")
}
if names := api.volNames("default"); len(names) != 2 {
t.Errorf("pool holds %v, want only the cached base image and its marker", names)
}
}
func TestDelete(t *testing.T) {
t.Parallel()
cases := []struct {
name string
setup func(t *testing.T, api *fakeAPI)
check func(t *testing.T, api *fakeAPI)
}{
{
name: "running domain",
setup: func(t *testing.T, api *fakeAPI) { managedDomain(t, api, testName, testUID, true, nil) },
check: func(t *testing.T, api *fakeAPI) {
if !api.hasCall("DomainDestroy") || !api.hasCall("DomainUndefineFlags") {
t.Errorf("calls = %v, want destroy and undefine", api.calls)
}
},
},
{
name: "stopped domain",
setup: func(t *testing.T, api *fakeAPI) { managedDomain(t, api, testName, testUID, false, nil) },
check: func(t *testing.T, api *fakeAPI) {
if !api.hasCall("DomainUndefineFlags") {
t.Errorf("calls = %v, want undefine despite destroy being invalid", api.calls)
}
},
},
{
name: "domain gone, volumes left",
setup: func(_ *testing.T, api *fakeAPI) {
api.putVol("default", overlayVolumeName(testName), formatQcow2, nil)
api.putVol("default", seedVolumeName(testName), formatRaw, nil)
},
check: func(t *testing.T, api *fakeAPI) {
if api.hasCall("DomainDestroy") || api.hasCall("DomainUndefineFlags") {
t.Errorf("calls = %v, want no domain operations", api.calls)
}
},
},
{
name: "nothing exists",
setup: func(*testing.T, *fakeAPI) {},
check: func(t *testing.T, api *fakeAPI) {
if api.hasCall("StorageVolDelete") {
t.Errorf("calls = %v, want no deletes", api.calls)
}
},
},
}
for _, tc := range cases {
t.Run(tc.name, func(t *testing.T) {
t.Parallel()
api := readyHost()
tc.setup(t, api)
p := newTestProvider(hostFakes{"nas1": api}, "nas1")
if err := p.Delete(context.Background(), "hosts/nas1/domains/"+testName); err != nil {
t.Fatalf("Delete: %v", err)
}
assertGone(t, api)
tc.check(t, api)
if err := p.Delete(context.Background(), "hosts/nas1/domains/"+testName); err != nil {
t.Fatalf("second Delete: %v", err)
}
})
}
}
func TestDelete_missingPoolIsNotAnError(t *testing.T) {
t.Parallel()
api := newFakeAPI("other")
api.putDomain(&fakeDomain{name: testName, metadata: metadataXML(t, testUID, testNow), running: true})
p := newTestProvider(hostFakes{"nas1": api}, "nas1")
if err := p.Delete(context.Background(), "hosts/nas1/domains/"+testName); err != nil {
t.Fatalf("Delete: %v", err)
}
if api.domain(testName) != nil {
t.Error("domain still defined")
}
}
func TestDelete_errors(t *testing.T) {
t.Parallel()
cases := []struct {
name string
id string
fakes func(t *testing.T) hostFakes
class error
}{
{"malformed id", "nope", func(*testing.T) hostFakes { return hostFakes{"nas1": readyHost()} }, provider.ErrPermanent},
{"host not configured", "hosts/nas9/domains/x", func(*testing.T) hostFakes { return hostFakes{"nas1": readyHost()} }, provider.ErrPermanent},
{"host unreachable", "hosts/nas1/domains/x", func(*testing.T) hostFakes { return hostFakes{} }, provider.ErrTransient},
{
"undefine fails", "hosts/nas1/domains/" + testName,
func(t *testing.T) hostFakes {
api := readyHost()
managedDomain(t, api, testName, testUID, true, nil)
api.fail["DomainUndefineFlags"] = lvError(libvirt.ErrInternalError, "boom")
return hostFakes{"nas1": api}
},
provider.ErrTransient,
},
}
for _, tc := range cases {
t.Run(tc.name, func(t *testing.T) {
t.Parallel()
p := newTestProvider(tc.fakes(t), "nas1")
wantClass(t, p.Delete(context.Background(), tc.id), tc.class)
})
}
}
func TestListByTag_acrossHosts(t *testing.T) {
t.Parallel()
nas1, nas2 := readyHost(), readyHost()
managedDomain(t, nas1, "proxy-one", "uid-1", true, map[uint32][]libvirt.DomainInterface{
uint32(libvirt.DomainInterfaceAddressesSrcArp): {{Addrs: []libvirt.DomainIPAddr{ipv4("192.168.1.10")}}},
})
nas1.putDomain(&fakeDomain{name: "hand-made", running: true})
managedDomain(t, nas2, "proxy-two", "uid-2", false, nil)
nas2.putDomain(&fakeDomain{name: "foreign", metadata: `<instance xmlns="` + metadataURI + `"><managed>false</managed></instance>`})
p := newTestProvider(hostFakes{"nas1": nas1, "nas2": nas2}, "nas1", "nas2")
got, err := p.ListByTag(context.Background())
if err != nil {
t.Fatalf("ListByTag: %v", err)
}
want := []provider.Instance{
{ID: "hosts/nas1/domains/proxy-one", IP: "192.168.1.10", State: provider.StateRunning, UID: "uid-1", CreatedAt: testNow},
{ID: "hosts/nas2/domains/proxy-two", State: provider.StateStopped, UID: "uid-2", CreatedAt: testNow},
}
if len(got) != len(want) {
t.Fatalf("got %d instances %+v, want %d", len(got), got, len(want))
}
for i := range want {
if got[i] != want[i] {
t.Errorf("instance[%d] = %+v, want %+v", i, got[i], want[i])
}
}
}
func TestListByTag_oneHostDownFailsSweep(t *testing.T) {
t.Parallel()
nas1 := readyHost()
managedDomain(t, nas1, "proxy-one", "uid-1", true, nil)
p := newTestProvider(hostFakes{"nas1": nas1}, "nas1", "nas2")
got, err := p.ListByTag(context.Background())
wantClass(t, err, provider.ErrTransient)
if got != nil {
t.Errorf("got %+v alongside the error, want nil (no partial view)", got)
}
}
// A domain that cannot be read is left out of the sweep rather than
// failing it: GC never deletes what it does not see, so skipping is the
// safe direction, while one bad domain blocking GC for all others is not.
func TestListByTag_skipsUnreadableDomain(t *testing.T) {
t.Parallel()
cases := []struct {
name string
setup func(t *testing.T, api *fakeAPI)
}{
{
"deleted mid-sweep",
func(t *testing.T, api *fakeAPI) {
managedDomain(t, api, "proxy-one", "uid-1", true, nil)
api.failOnce["DomainGetMetadata"] = lvError(libvirt.ErrNoDomain, "gone")
},
},
{
"corrupt metadata",
func(_ *testing.T, api *fakeAPI) {
api.putDomain(&fakeDomain{name: "proxy-one", metadata: "<instance", running: true})
},
},
{
"state unreadable",
func(t *testing.T, api *fakeAPI) {
managedDomain(t, api, "proxy-one", "uid-1", true, nil)
api.failOnce["DomainGetState"] = lvError(libvirt.ErrInternalError, "boom")
},
},
}
for _, tc := range cases {
t.Run(tc.name, func(t *testing.T) {
t.Parallel()
nas1 := readyHost()
tc.setup(t, nas1)
managedDomain(t, nas1, "proxy-two", "uid-2", false, nil)
p := newTestProvider(hostFakes{"nas1": nas1}, "nas1")
got, err := p.ListByTag(context.Background())
if err != nil {
t.Fatalf("ListByTag: %v", err)
}
if len(got) != 1 || got[0].UID != "uid-2" {
t.Errorf("got %+v, want only proxy-two", got)
}
})
}
}
func TestListByTag_noHosts(t *testing.T) {
t.Parallel()
p := newTestProvider(hostFakes{})
got, err := p.ListByTag(context.Background())
if err != nil || len(got) != 0 {
t.Errorf("got %v, %v; want empty, nil", got, err)
}
}
func TestMapState(t *testing.T) {
t.Parallel()
cases := []struct {
state libvirt.DomainState
ip string
want provider.InstanceState
}{
{libvirt.DomainRunning, "10.0.0.1", provider.StateRunning},
{libvirt.DomainRunning, "", provider.StateProvisioning},
{libvirt.DomainBlocked, "10.0.0.1", provider.StateRunning},
{libvirt.DomainPaused, "10.0.0.1", provider.StateStopped},
{libvirt.DomainShutdown, "", provider.StateStopped},
{libvirt.DomainShutoff, "", provider.StateStopped},
{libvirt.DomainCrashed, "", provider.StateStopped},
{libvirt.DomainPmsuspended, "", provider.StateStopped},
{libvirt.DomainNostate, "", provider.StateProvisioning},
}
for _, tc := range cases {
t.Run(fmt.Sprintf("%s/%q", tc.state, tc.ip), func(t *testing.T) {
t.Parallel()
if got := mapState(tc.state, tc.ip); got != tc.want {
t.Errorf("mapState(%s, %q) = %s, want %s", tc.state, tc.ip, got, tc.want)
}
})
}
}
func TestNew_badKeyPathFailsFast(t *testing.T) {
t.Parallel()
cfg := testConfig("nas1")
cfg.KVM.SSHKeyFile = t.TempDir() + "/missing"
if _, err := New(context.Background(), cfg); err == nil {
t.Fatal("New succeeded with a missing key file")
}
}
+57
View File
@@ -0,0 +1,57 @@
package kvm
import (
"encoding/xml"
"fmt"
"time"
"gitea.home.hrajfrisbee.cz/kacerr/egress-proxies-operator/internal/provider"
)
// metadataURI namespaces the operator's element inside the domain's
// <metadata>. libvirt has no labels or tags, but it stores any namespaced XML
// there and hands it back through DomainGetMetadata, which is how the GC
// contract (managed flag + owning UID) and the creation time survive across
// operator restarts.
const metadataURI = "https://gitea.home.hrajfrisbee.cz/kacerr/egress-proxies-operator/kvm"
type instanceMetadata struct {
XMLName xml.Name `xml:"https://gitea.home.hrajfrisbee.cz/kacerr/egress-proxies-operator/kvm instance"`
Managed string `xml:"managed"`
UID string `xml:"uid"`
CreatedAt string `xml:"createdAt"`
}
func newInstanceMetadata(uid string, createdAt time.Time) instanceMetadata {
return instanceMetadata{
Managed: provider.LabelManagedYes,
UID: uid,
CreatedAt: createdAt.UTC().Format(time.RFC3339),
}
}
// parseInstanceMetadata decodes the element as DomainGetMetadata returns it.
// libvirt may re-serialise it with a namespace prefix rather than the default
// namespace it was defined with; encoding/xml matches on namespace URI, so
// both spellings decode.
func parseInstanceMetadata(s string) (instanceMetadata, error) {
var m instanceMetadata
if err := xml.Unmarshal([]byte(s), &m); err != nil {
return instanceMetadata{}, fmt.Errorf("parsing domain metadata: %w", err)
}
return m, nil
}
func (m instanceMetadata) managed() bool {
return m.Managed == provider.LabelManagedYes
}
// createdAt returns the zero time on a malformed timestamp: orphan GC treats
// zero as "old", so a corrupt value can never protect a domain forever.
func (m instanceMetadata) createdAt() time.Time {
t, err := time.Parse(time.RFC3339, m.CreatedAt)
if err != nil {
return time.Time{}
}
return t
}
+79
View File
@@ -0,0 +1,79 @@
package kvm
import (
"encoding/xml"
"strings"
"testing"
"time"
)
func TestInstanceMetadata_roundTrip(t *testing.T) {
t.Parallel()
created := time.Date(2026, 9, 4, 12, 30, 0, 0, time.FixedZone("CEST", 2*3600))
raw, err := xml.Marshal(newInstanceMetadata("uid-1", created))
if err != nil {
t.Fatalf("Marshal() error = %v", err)
}
if !strings.Contains(string(raw), `xmlns="`+metadataURI+`"`) {
t.Errorf("marshalled metadata lacks the namespace declaration: %s", raw)
}
m, err := parseInstanceMetadata(string(raw))
if err != nil {
t.Fatalf("parseInstanceMetadata() error = %v", err)
}
if !m.managed() || m.UID != "uid-1" {
t.Errorf("parsed = %+v, want managed with uid-1", m)
}
if got := m.createdAt(); !got.Equal(created) || got.Location() != time.UTC {
t.Errorf("createdAt() = %v, want %v in UTC", got, created)
}
}
func TestParseInstanceMetadata_libvirtPrefixedForm(t *testing.T) {
t.Parallel()
// libvirt re-serialises the stored element with whatever prefix it
// chooses; only the namespace URI is contractual.
in := `<op:instance xmlns:op="` + metadataURI + `">` +
`<op:managed>true</op:managed><op:uid>uid-2</op:uid>` +
`<op:createdAt>2026-09-04T10:00:00Z</op:createdAt></op:instance>`
m, err := parseInstanceMetadata(in)
if err != nil {
t.Fatalf("parseInstanceMetadata() error = %v", err)
}
if !m.managed() || m.UID != "uid-2" || m.createdAt().Hour() != 10 {
t.Errorf("parsed = %+v", m)
}
}
func TestParseInstanceMetadata_notManaged(t *testing.T) {
t.Parallel()
tests := []struct {
name string
in string
wantErr bool
}{
{"managed false", `<instance xmlns="` + metadataURI + `"><managed>false</managed><uid>u</uid></instance>`, false},
{"foreign element", `<foo xmlns="urn:other"><managed>true</managed></foo>`, true},
{"garbage", `not xml`, true},
}
for _, tc := range tests {
t.Run(tc.name, func(t *testing.T) {
t.Parallel()
m, err := parseInstanceMetadata(tc.in)
if (err != nil) != tc.wantErr {
t.Fatalf("error = %v, wantErr %v", err, tc.wantErr)
}
if m.managed() {
t.Errorf("managed() = true for %q", tc.in)
}
})
}
}
func TestInstanceMetadata_createdAtMalformedIsZero(t *testing.T) {
t.Parallel()
m := instanceMetadata{CreatedAt: "yesterday"}
if !m.createdAt().IsZero() {
t.Errorf("createdAt() = %v, want zero time", m.createdAt())
}
}
+53
View File
@@ -0,0 +1,53 @@
package kvm
import (
"bytes"
"fmt"
"strings"
"github.com/kdomanski/iso9660"
)
// seedLabel is the volume label cloud-init's NoCloud datasource looks for.
const seedLabel = "cidata"
// defaultVendorData is the cloud-init vendor-data shipped on every seed
// unless the config overrides it. cloud-init merges it under the Proxy's own
// user-data, so stock images grow a guest agent (which lets the provider
// read the VM's IP before ARP sees traffic) without the user's cloud-init
// having to know about it.
const defaultVendorData = `#cloud-config
packages:
- qemu-guest-agent
runcmd:
- [systemctl, enable, --now, qemu-guest-agent]
`
// buildSeedISO renders a NoCloud seed: user-data verbatim, vendor-data as
// given, and a meta-data whose instance-id is the deterministic VM name, so
// cloud-init never mistakes a reboot for a new instance. The ISO's volume
// descriptor carries the build time, so two builds differ in a few
// timestamp bytes; the files and label are identical.
func buildSeedISO(name, userData, vendorData string) ([]byte, error) {
w, err := iso9660.NewWriter()
if err != nil {
return nil, fmt.Errorf("creating iso writer: %w", err)
}
defer w.Cleanup() //nolint:errcheck // temp-dir cleanup; nothing to do on failure
files := []struct{ name, body string }{
{"user-data", userData},
{"meta-data", fmt.Sprintf("instance-id: %s\nlocal-hostname: %s\n", name, name)},
{"vendor-data", vendorData},
}
for _, f := range files {
if err := w.AddFile(strings.NewReader(f.body), f.name); err != nil {
return nil, fmt.Errorf("adding %s to seed iso: %w", f.name, err)
}
}
var buf bytes.Buffer
if err := w.WriteTo(&buf, seedLabel); err != nil {
return nil, fmt.Errorf("writing seed iso: %w", err)
}
return buf.Bytes(), nil
}
+100
View File
@@ -0,0 +1,100 @@
package kvm
import (
"bytes"
"io"
"maps"
"strings"
"testing"
"github.com/kdomanski/iso9660"
)
func readSeed(t *testing.T, iso []byte) (label string, files map[string]string) {
t.Helper()
img, err := iso9660.OpenImage(bytes.NewReader(iso))
if err != nil {
t.Fatalf("OpenImage() error = %v", err)
}
label, err = img.Label()
if err != nil {
t.Fatalf("Label() error = %v", err)
}
root, err := img.RootDir()
if err != nil {
t.Fatalf("RootDir() error = %v", err)
}
children, err := root.GetChildren()
if err != nil {
t.Fatalf("GetChildren() error = %v", err)
}
files = make(map[string]string, len(children))
for _, c := range children {
body, err := io.ReadAll(c.Reader())
if err != nil {
t.Fatalf("reading %s: %v", c.Name(), err)
}
files[c.Name()] = string(body)
}
return label, files
}
func TestBuildSeedISO_contents(t *testing.T) {
t.Parallel()
userData := "#cloud-config\npackages: [squid]\n"
iso, err := buildSeedISO("proxy-abc123def456ghij", userData, defaultVendorData)
if err != nil {
t.Fatalf("buildSeedISO() error = %v", err)
}
label, files := readSeed(t, iso)
if !strings.EqualFold(label, seedLabel) {
t.Errorf("label = %q, want %q", label, seedLabel)
}
if len(files) != 3 {
t.Errorf("files = %v, want exactly user-data, meta-data, vendor-data", files)
}
if files["user-data"] != userData {
t.Errorf("user-data = %q, want verbatim cloud-init", files["user-data"])
}
if files["vendor-data"] != defaultVendorData {
t.Errorf("vendor-data = %q", files["vendor-data"])
}
if got, want := files["meta-data"], "instance-id: proxy-abc123def456ghij\nlocal-hostname: proxy-abc123def456ghij\n"; got != want {
t.Errorf("meta-data = %q, want %q", got, want)
}
}
// The writer stamps the build time into the volume descriptor, so two
// builds are not byte-identical; what must not change between them is
// what cloud-init reads: the label and the three files.
func TestBuildSeedISO_deterministicContent(t *testing.T) {
t.Parallel()
a, err := buildSeedISO("proxy-x", "#cloud-config\n", "")
if err != nil {
t.Fatalf("first buildSeedISO() error = %v", err)
}
b, err := buildSeedISO("proxy-x", "#cloud-config\n", "")
if err != nil {
t.Fatalf("second buildSeedISO() error = %v", err)
}
labelA, filesA := readSeed(t, a)
labelB, filesB := readSeed(t, b)
if labelA != labelB || !maps.Equal(filesA, filesB) {
t.Errorf("two builds for the same inputs differ: %q %v vs %q %v", labelA, filesA, labelB, filesB)
}
if len(a) != len(b) {
t.Errorf("sizes differ: %d vs %d; the seed volume is created with the ISO's exact size", len(a), len(b))
}
}
func TestBuildSeedISO_emptyUserData(t *testing.T) {
t.Parallel()
iso, err := buildSeedISO("proxy-x", "", defaultVendorData)
if err != nil {
t.Fatalf("buildSeedISO() error = %v", err)
}
_, files := readSeed(t, iso)
if body, ok := files["user-data"]; !ok || body != "" {
t.Errorf("user-data = %q (present=%v), want an empty file to still exist", body, ok)
}
}
+78
View File
@@ -0,0 +1,78 @@
package kvm
import (
"encoding/xml"
"fmt"
)
const bytesPerGiB = 1 << 30
func overlayVolumeName(name string) string { return name + ".qcow2" }
func seedVolumeName(name string) string { return name + "-seed.iso" }
// volumeSpec describes a storage volume to create. A non-empty BackingPath
// makes it a copy-on-write overlay of that file.
type volumeSpec struct {
Name string
SizeBytes int64
Format string
BackingPath string
BackingFormat string
}
func overlayVolume(name string, sizeGB int64, basePath, baseFormat string) volumeSpec {
return volumeSpec{
Name: overlayVolumeName(name),
SizeBytes: sizeGB * bytesPerGiB,
Format: formatQcow2,
BackingPath: basePath,
BackingFormat: baseFormat,
}
}
func rawVolume(name string, sizeBytes int64) volumeSpec {
return volumeSpec{Name: name, SizeBytes: sizeBytes, Format: formatRaw}
}
// buildVolumeXML renders the volume for StorageVolCreateXML. Capacity is
// always given in bytes so no libvirt unit-suffix ambiguity (G vs GiB) can
// creep in.
func buildVolumeXML(v volumeSpec) (string, error) {
x := volumeXML{
Type: "file",
Name: v.Name,
Capacity: sizeXML{Unit: "bytes", Value: v.SizeBytes},
Target: volumeTargetXML{Format: formatXML{Type: v.Format}},
}
if v.BackingPath != "" {
x.BackingStore = &backingStoreXML{Path: v.BackingPath, Format: formatXML{Type: v.BackingFormat}}
}
out, err := xml.Marshal(x)
if err != nil {
return "", fmt.Errorf("marshalling volume xml: %w", err)
}
return string(out), nil
}
type volumeXML struct {
XMLName xml.Name `xml:"volume"`
Type string `xml:"type,attr"`
Name string `xml:"name"`
Capacity sizeXML `xml:"capacity"`
Target volumeTargetXML `xml:"target"`
BackingStore *backingStoreXML `xml:"backingStore"`
}
type volumeTargetXML struct {
Format formatXML `xml:"format"`
}
type formatXML struct {
Type string `xml:"type,attr"`
}
type backingStoreXML struct {
Path string `xml:"path"`
Format formatXML `xml:"format"`
}
+62
View File
@@ -0,0 +1,62 @@
package kvm
import (
"encoding/xml"
"strings"
"testing"
)
func TestVolumeNames(t *testing.T) {
t.Parallel()
if got := overlayVolumeName("proxy-x"); got != "proxy-x.qcow2" {
t.Errorf("overlayVolumeName = %q", got)
}
if got := seedVolumeName("proxy-x"); got != "proxy-x-seed.iso" {
t.Errorf("seedVolumeName = %q", got)
}
}
func TestBuildVolumeXML_overlay(t *testing.T) {
t.Parallel()
raw, err := buildVolumeXML(overlayVolume("proxy-x", 10, "/var/lib/libvirt/images/base-debian-13.qcow2", "qcow2"))
if err != nil {
t.Fatalf("buildVolumeXML() error = %v", err)
}
var v volumeXML
if err := xml.Unmarshal([]byte(raw), &v); err != nil {
t.Fatalf("output is not well-formed XML: %v", err)
}
if v.Type != "file" || v.Name != "proxy-x.qcow2" {
t.Errorf("type/name = %s/%s", v.Type, v.Name)
}
if v.Capacity.Unit != "bytes" || v.Capacity.Value != 10*bytesPerGiB {
t.Errorf("capacity = %d %s, want %d bytes", v.Capacity.Value, v.Capacity.Unit, 10*bytesPerGiB)
}
if v.Target.Format.Type != "qcow2" {
t.Errorf("format = %q", v.Target.Format.Type)
}
if v.BackingStore == nil {
t.Fatalf("backingStore missing:\n%s", raw)
}
if v.BackingStore.Path != "/var/lib/libvirt/images/base-debian-13.qcow2" || v.BackingStore.Format.Type != "qcow2" {
t.Errorf("backingStore = %+v", v.BackingStore)
}
}
func TestBuildVolumeXML_rawHasNoBackingStore(t *testing.T) {
t.Parallel()
raw, err := buildVolumeXML(rawVolume("proxy-x-seed.iso", 372736))
if err != nil {
t.Fatalf("buildVolumeXML() error = %v", err)
}
if strings.Contains(raw, "backingStore") {
t.Errorf("raw volume must not declare a backing store:\n%s", raw)
}
var v volumeXML
if err := xml.Unmarshal([]byte(raw), &v); err != nil {
t.Fatalf("output is not well-formed XML: %v", err)
}
if v.Name != "proxy-x-seed.iso" || v.Capacity.Value != 372736 || v.Target.Format.Type != "raw" {
t.Errorf("volume = %+v", v)
}
}