Add KVM provider integration test against a real hypervisor
Behind -tags=integration and self-skipping unless KVM_TEST_* names a host: one sequential lifecycle (image fetch, Create, idempotent re-Create, Running with IP, ListByTag, Delete, pool state afterwards), driven through the provider directly so it needs no Step 5 wiring. Adds `make test-integration-kvm` and documents the run and the deviation from the testcontainers convention in docs/testing.md. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
This commit is contained in:
Claude Fable 5.1
@@ -9,7 +9,7 @@ Plan: [2026-09-04-1404-kvm-provider.md](../plans/2026-09-04-1404-kvm-provider.md
|
||||
- [x] Step 3b — Image cache (`image.go`)
|
||||
- [x] Step 4 — Provider methods (`kvm.go`) over a fake API
|
||||
- [ ] Step 5 — Wiring, manifests, samples, docs, CHANGELOG
|
||||
- [ ] Step 6 — Integration test against a real hypervisor
|
||||
- [x] Step 6 — Integration test against a real hypervisor (written and compiled; first real run is part of Step 7)
|
||||
- [ ] Step 7 — Manual validation + MR
|
||||
|
||||
## Step 0 — Save plan + branch
|
||||
@@ -283,3 +283,58 @@ uncovered lines are the real SSH/libvirt dial path and log-only
|
||||
branches. `go vet` and `-race` (three runs) are clean. Wiring into
|
||||
`cmd/main.go` is Step 5, so `New` is not yet reachable from a running
|
||||
operator.
|
||||
|
||||
## Step 6 — Integration test
|
||||
|
||||
Done before Step 5, at the user's request, after a discussion of what the
|
||||
unit tests can and cannot prove: they run over an in-memory fake of the
|
||||
libvirt calls, so every assumption about libvirt's real behaviour (domain
|
||||
XML accepted, raw qcow2 upload + pool refresh usable as a backing image,
|
||||
guest boots and reports an address) needed a test against a real host.
|
||||
The wiring in Step 5 is not a prerequisite — the test drives the provider
|
||||
directly.
|
||||
|
||||
`internal/provider/kvm/kvm_integration_test.go` is behind
|
||||
`//go:build integration` and skips itself under `-short` or when any of
|
||||
`KVM_TEST_HOST`, `KVM_TEST_USER`, `KVM_TEST_SSH_KEY`, `KVM_TEST_BRIDGE`,
|
||||
`KVM_TEST_IMAGE_URL` is unset. Optional knobs beyond the plan's list:
|
||||
`KVM_TEST_KNOWN_HOSTS` (without it the run disables host-key checking
|
||||
and logs that), `KVM_TEST_IMAGE_SHA256`, `KVM_TEST_POOL`,
|
||||
`KVM_TEST_SOCKET`, `KVM_TEST_TIMEOUT` (default 20 m, must cover a cold
|
||||
image download) and `KVM_TEST_KEEP` to leave a failed run's VM behind for
|
||||
`virsh console`.
|
||||
|
||||
One sequential test, `TestKVM_lifecycle`, not subtests: each stage
|
||||
depends on hypervisor state the previous one left. Create (timed, so the
|
||||
log shows the image fetch), re-Create returns the same ID, poll Get to
|
||||
Running with an IPv4 (Stopped fails fast with a hint to use the console),
|
||||
UID and CreatedAt survive the metadata round-trip, ListByTag includes it,
|
||||
Delete, Get → NotFound, second Delete is a no-op, ListByTag excludes it,
|
||||
and the pool is inspected through the provider's own connection pool
|
||||
(`p.hosts.call`, reachable because the test is in package `kvm`): overlay
|
||||
and seed gone, base image and marker kept. A TCP connect to the guest's
|
||||
port 22 is attempted but only logged, since the test machine may not
|
||||
route to the bridge network. Cleanup deletes the VM on failure with its
|
||||
own two-minute budget, because the test context may already be spent.
|
||||
|
||||
Makefile target, so the invocation is not something to remember:
|
||||
|
||||
```bash
|
||||
make test-integration-kvm
|
||||
# = go test -race -tags=integration -count=1 -v -timeout 40m -run TestKVM ./internal/provider/kvm/
|
||||
```
|
||||
|
||||
`docs/testing.md` gained the kvm package rows in the suite table, a
|
||||
"live libvirt" bullet under the deliberately-not-covered list, and a
|
||||
section on running this test, with the deviation from the testcontainers
|
||||
convention called out as the plan asked.
|
||||
|
||||
Worth noting: **the test has not been executed yet.** This session had
|
||||
no hypervisor credentials; it was verified with
|
||||
`go vet -tags=integration ./...` and by running it without the
|
||||
environment (self-skip). Its first real run is the opening act of
|
||||
Step 7's manual validation, and it is the point at which the Step 2 and
|
||||
Step 3b "confirm on a real host" notes get answered. The test's config
|
||||
is built directly rather than through the provider package's validation
|
||||
(unexported), so a typo in an env var surfaces from `New` or from
|
||||
libvirt, not from a validation message.
|
||||
|
||||
@@ -54,6 +54,10 @@ manual runs too.
|
||||
| | `metrics_test.go` | `WithMetrics` decorator: classified result labels, error passthrough | fake recorder + static provider |
|
||||
| `internal/provider/kubernetes` | `kubernetes_test.go`, `pod_test.go` | Create/Get/Delete/ListByTag over real Pod objects; pure `buildPod`/`squidConf` incl. the `max_filedescriptors` OOM-regression assertion | `client/fake` (real `corev1.Pod`s, no kubelet) |
|
||||
| `internal/provider/gcp` | `insert_test.go`, `errors_test.go`, `gcp_test.go` | field-by-field `buildInsertRequest`; HTTP-code classification table; zone-qualified IDs, 409-is-success, 9-row state mapping, ListByTag filter + `ReturnPartialSuccess` | pure builder needs **no fake**; the rest uses the flattened `instancesAPI` seam |
|
||||
| `internal/provider/kvm` | `domainxml_test.go`, `volume_test.go`, `seed_test.go`, `id_test.go`, `metadata_test.go` | pure builders: domain/volume XML unmarshalled back and checked field by field, seed ISO reopened with the same library, providerID round-trip, metadata in both namespace spellings | pure |
|
||||
| | `errors_test.go`, `conn_test.go` | libvirt-code classification table; connection pool (reuse, redial, drop on transport error, cancel releases a blocked RPC, one dial per host under concurrency); SSH dial, host-key policy, keepalive | fake dialer; a **real in-process `x/crypto/ssh` server** forwarding a unix socket |
|
||||
| | `image_test.go`, `kvm_test.go` | image cache (fresh fetch, ready, marker missing, checksum change/mismatch, 4xx/5xx, concurrent callers); Create resume after every partial step, leftover truncated seed, state/IP mapping, every Delete already-gone case, ListByTag across hosts / one down / unreadable domain | `httptest` image server + in-memory `fakeAPI` behind the `libvirtAPI` seam |
|
||||
| | `kvm_integration_test.go` (`-tags=integration`) | one real VM: image fetch, Create, idempotent re-Create, Running with IP, ListByTag, Delete, pool state afterwards | a **real hypervisor** — see below |
|
||||
| `internal/health` | `probe_test.go` | probes through a **real CONNECT-capable proxy stub** to a real TLS server: tunnel success, refused CONNECT, wrong status, dead proxy, plain-http forward | `httptest` + hijacked bidirectional tunnel |
|
||||
| | `engine_test.go` | thresholds, first-verdict, latency suppression matrix, dropped-event retry, stale-probe UID guard, tick scheduling/seeding/pruning, `Start` end-to-end | fake `client.Reader` + injected `probeFn` |
|
||||
| `internal/lease` | `store_test.go` | capacity, `MaxLeases=0`, all three selection tie-breaks, target-scoped vs global cooldowns, expiry via fake clock, report-on-retained-lease, idempotent release, **40 concurrent acquires never exceeding capacity** | injectable clock; the concurrency case is why `-race` matters |
|
||||
@@ -83,6 +87,51 @@ envtest suite the scaffold generated.
|
||||
bug (below).
|
||||
- **Live GCP**: no test talks to Google. The seam boundary
|
||||
(`buildInsertRequest` + classification) is tested exhaustively instead.
|
||||
- **Live libvirt, by default**: the KVM unit tests run over an in-memory
|
||||
fake of the 17 libvirt calls the provider uses. Whether libvirt accepts
|
||||
the domain XML, whether a raw qcow2 upload plus pool refresh yields a
|
||||
usable backing image, and whether the guest actually boots and reports
|
||||
an address are provable only on a real host — that is the opt-in
|
||||
integration test below.
|
||||
|
||||
## The KVM integration test (real hypervisor)
|
||||
|
||||
`internal/provider/kvm/kvm_integration_test.go` compiles only under
|
||||
`-tags=integration` and skips itself unless the environment names a
|
||||
host. It deviates from the "testcontainers for real infra" convention on
|
||||
purpose: libvirt needs `/dev/kvm` and a bridge to a DHCP network, and a
|
||||
container has neither. So it targets whichever hypervisor you point it
|
||||
at and leaves that host's pool with one more cached base image than it
|
||||
had.
|
||||
|
||||
```sh
|
||||
export KVM_TEST_HOST=nas1.home.lan:22 # SSH endpoint
|
||||
export KVM_TEST_USER=virt # must be in the libvirt group
|
||||
export KVM_TEST_SSH_KEY=$HOME/.ssh/id_ed25519 # private key for that user
|
||||
export KVM_TEST_BRIDGE=br0 # bridge to a network with DHCP
|
||||
export KVM_TEST_IMAGE_URL=https://cloud.debian.org/images/cloud/trixie/latest/debian-13-generic-amd64.qcow2
|
||||
make test-integration-kvm
|
||||
```
|
||||
|
||||
Optional: `KVM_TEST_KNOWN_HOSTS` (without it host keys are **not**
|
||||
verified — the log says so), `KVM_TEST_IMAGE_SHA256`, `KVM_TEST_POOL`
|
||||
(default `default`), `KVM_TEST_SOCKET`, `KVM_TEST_TIMEOUT` (default
|
||||
`20m`, must cover the first-time image download), and `KVM_TEST_KEEP`
|
||||
to leave the VM behind for `virsh console` when something goes wrong.
|
||||
|
||||
What one run asserts, in order: `New` dials the host; `Create` returns a
|
||||
`hosts/<host>/domains/<name>` ID (downloading the image on a cold host —
|
||||
the log prints how long it took); a second `Create` returns the same ID;
|
||||
`Get` reaches `Running` with an IPv4 within the budget, and the UID and
|
||||
`CreatedAt` survive the metadata round-trip through libvirt; `ListByTag`
|
||||
lists it; `Delete` removes it, `Get` says `ErrNotFound`, a second
|
||||
`Delete` is a no-op, and the pool holds the base image and its marker
|
||||
but no per-VM overlay or seed. It also tries a TCP connect to port 22 on
|
||||
the guest, but only logs the outcome — the machine running the test may
|
||||
not route to the bridge network.
|
||||
|
||||
The test is not part of `make test` and not run in CI. Keep it compiling
|
||||
with `go vet -tags=integration ./...` when the provider's internals move.
|
||||
|
||||
## The scaffolded `make test-e2e` suite
|
||||
|
||||
|
||||
Reference in New Issue
Block a user