kacerr/egress-proxies-operator
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

kvm2: integration test, sample Proxy, providers-config entries and Makefile target

Browse Source 
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
This commit is contained in:
kacerr
2026-09-16 19:40:50 +00:00
co-authored by Claude Fable 5.1
parent ce62fe57f6
commit fb6f59090b
9 changed files with 844 additions and 10 deletions
Download Patch File Download Diff File Expand all files Collapse all files
docs/testing.md
+100 -6
View File
@@ -62,6 +62,11 @@ manual runs too.
| | `trace_test.go`, `conn_test.go`, `errors_test.go` | every `libvirtAPI` method traced exactly once (guarded by reflection on the interface), results and errors pass through, libvirt error fields, detail keys only at V2, upload content never logged; pool connect/drop lines, no wrapping below V1 | `fakeAPI` + `funcr` capture |
| | `check_test.go`, `conn_test.go` | health check: details, per-host verdicts in config order, hosts checked concurrently, a timeout that keeps the connection and refuses a second check while the first still waits, no waiting behind a reconcile's dial, the reason table; `probe` abandons without closing, but still drops on a transport error; waiting for a host's slot ends with the caller's deadline; an inactive or missing pool and daemon errors keep the connection; every dial stage tagged through the real SSH server, with classification unchanged | `fakeAPI`, fake dialers, in-process SSH server |
| | `kvm_integration_test.go` (`-tags=integration`) | one real VM: image fetch, Create, idempotent re-Create, Running with IP, ListByTag, Delete, pool state afterwards; `TestKVMCheck_realHost` runs the health check against the host and creates nothing | a **real hypervisor** — see below |
| `internal/provider/kvm2` | `xml_test.go`, `volume_test.go`, `seed_test.go`, `id_test.go`, `metadata_test.go`, `state_test.go` | pure builders: domain XML (file disks with resolved paths, UEFI with secure boot off, BIOS escape, the metadata element, the guest-agent channel), volume XML with and without a backing store, seed ISO reopened (files, label, user-data byte for byte, temp dir removed) and the vendor-data script vs. multipart rules, providerID round-trip and malformed forms, metadata in both namespace spellings, state mapping and the IPv4 filter | pure |
| | `errors_test.go`, `backend_test.go`, `hosts_test.go`, `ssh_test.go` | classification table (libvirt codes, SSH auth text, `knownhosts.KeyError`, missing files); the per-call budget adapter (deadline kills the connection and is Transient, cancel unblocks, killed by another caller, close idempotent) against a mocked libvirt socket; the host table (one dial per host under concurrency, redial after drop, `dialFresh` never cached, waiting ends with ctx); the real dialer end to end: socket forwarding, unknown and mismatched host keys, insecure mode, missing vs. unparsable key and known_hosts, wrong key, handshake timeout, cancel, every stage tagged, key material never in errors or logs | fake dialer; a **real in-process `x/crypto/ssh` server** forwarding a unix socket |
| | `image_test.go`, `create_test.go`, `get_test.go`, `delete_test.go`, `list_test.go`, `kvm2_test.go`, `provider_test.go` | image cache (fetch with the marker written last, cached, marker missing or without a digest, sha256 change or mismatch, 4xx/5xx, no Content-Length, short read, concurrent callers fetch once, cancel mid-stream, streamed without buffering, upload on a dedicated connection) and the pool-space quota precheck; Create resume after every partial step, returns only when running, adoption guard on UID, seed always rebuilt, cloud-init sentinel never logged; Get state and address mapping incl. agent-before-ARP; every Delete already-gone case, NVRAM flag, per-name lock; ListByTag across hosts, one down fails the whole call, unreadable domain skipped; `New` never dials and warns on missing files; every error wrapped with the configured name; logging tiers V0/V1/V2 | `httptest` image server + in-memory `fakeBackend` behind the `backend` seam |
| | `observe_test.go`, `check_test.go` | every seam method traced and logged exactly once (guarded by reflection), child spans under the parent, detail only at V2, upload bodies never logged, error fields; health check: one entry per host in config order, hosts checked concurrently, expired ctx returns promptly, never touches the pooled connection, the reason table, detail keys | `fakeBackend` + `funcr` capture + span recorder |
| | `kvm2_integration_test.go` (`-tags=integration`) | one real VM: `New` without a dial, `Check` Ready with details, Create, idempotent re-Create with an unchanged domain count, Running with IP and the metadata round-trip, ListByTag, Delete, NotFound, pool state afterwards, `kvm2.*` spans under `provider.create`; `TestKVM2Check_realHost` runs the health check against the host and creates nothing | a **real hypervisor** — see below |
| `internal/health` | `probe_test.go` | probes through a **real CONNECT-capable proxy stub** to a real TLS server: tunnel success, refused CONNECT, wrong status, dead proxy, plain-http forward | `httptest` + hijacked bidirectional tunnel |
| | `engine_test.go` | thresholds, first-verdict, latency suppression matrix, dropped-event retry, stale-probe UID guard, tick scheduling/seeding/pruning, `Start` end-to-end | fake `client.Reader` + injected `probeFn` |
| `internal/lease` | `store_test.go` | capacity, `MaxLeases=0`, all three selection tie-breaks, target-scoped vs global cooldowns, expiry via fake clock, report-on-retained-lease, idempotent release, **40 concurrent acquires never exceeding capacity** | injectable clock; the concurrency case is why `-race` matters |
@@ -93,12 +98,12 @@ envtest suite the scaffold generated.
bug (below).
- **Live GCP**: no test talks to Google. The seam boundary
(`buildInsertRequest` + classification) is tested exhaustively instead.
- **Live libvirt, by default**: the KVM unit tests run over an in-memory
fake of the 20 libvirt calls the provider uses. Whether libvirt accepts
the domain XML, whether a raw qcow2 upload plus pool refresh yields a
usable backing image, and whether the guest actually boots and reports
an address are provable only on a real host — that is the opt-in
integration test below.
- **Live libvirt, by default**: the KVM and kvm2 unit tests run over an
in-memory fake of the 20 libvirt calls each provider uses. Whether
libvirt accepts the domain XML, whether a raw qcow2 upload plus pool
refresh yields a usable backing image, and whether the guest actually
boots and reports an address are provable only on a real host — that is
the opt-in integration test below (one per provider).
## The KVM integration test (real hypervisor)
@@ -162,6 +167,95 @@ go test -tags=integration -count=1 -v -run TestKVMCheck ./internal/provider/kvm/
The test is not part of `make test` and not run in CI. Keep it compiling
with `go vet -tags=integration ./...` when the provider's internals move.
## The kvm2 integration test (real hypervisor)
`internal/provider/kvm2/kvm2_integration_test.go` compiles only under
`-tags=integration` and skips itself unless the environment names a
host. Like the kvm one it deviates from the "testcontainers for real
infra" convention on purpose: libvirt needs `/dev/kvm` and a bridge to a
DHCP network, and a container has neither. It targets whichever
hypervisor you point it at and leaves that host's pool with one more
cached base image (`base-test-image.qcow2` and its `.sha256` marker) than
it had. The kvm and kvm2 tests can share a host: kvm2 marks its domains
in its own metadata namespace and never lists or adopts the other's.
```sh
export KVM2_TEST_HOST=beelink:22 # SSH endpoint
export KVM2_TEST_USER=novakj # must be in the libvirt group
export KVM2_TEST_SSH_KEY=$HOME/.ssh/id_ed25519 # private key for that user
export KVM2_TEST_KNOWN_HOSTS=$HOME/.ssh/kvm_known_hosts # the hypervisors' host keys
export KVM2_TEST_BRIDGE=br0 # bridge to a network with DHCP
export KVM2_TEST_IMAGE_URL=https://cloud.debian.org/images/cloud/trixie/latest/debian-13-generic-amd64.qcow2
make test-integration-kvm2
```
`KVM2_TEST_KNOWN_HOSTS` is technically optional, but leaving it out turns
host-key verification **off** for the run (the log says so); keep a
`kvm_known_hosts` holding just the hypervisors rather than pointing the
test at your everyday `known_hosts`, so a re-keyed hypervisor is a
one-line fix in a file nothing else uses.
Also optional: `KVM2_TEST_IMAGE_SHA256` (verified after download and
compared with the cached marker), `KVM2_TEST_POOL` (default `default`),
`KVM2_TEST_SOCKET` (default `/var/run/libvirt/libvirt-sock`),
`KVM2_TEST_TIMEOUT` (default `20m`, the budget for the whole lifecycle;
it must cover the first-time image download), `KVM2_TEST_KEEP` to leave
the VM behind for `virsh console` when something goes wrong (the log then
prints the `virsh undefine --nvram` and `virsh vol-delete` lines that
clean it up), and `KVM2_TEST_LOG_LEVEL` (default `1`) for how much of the
provider's logging the `-v` output shows: at `1` every libvirt RPC is one
compact `libvirt call` line (`HH:MM:SS.mmm kvm2 "level"=1 "msg"="libvirt
call" "rpc"="DomainLookupByName" "domain"=... "duration"=...`, plus the
per-operation outcome lines and connection events), interleaved with the
test's own `t.Log` lines; `2` adds the domain and volume XML, metadata
and addresses; `0` leaves only `t.Log`. The per-run constants (provider,
host, domain name) are stated once in the first line and left off every
provider line. See the Logging section of
[kvm2-provider.md](kvm2-provider.md).
What one run asserts, in order, each step logged with its duration:
1. `New` returns within 100 ms and without dialling (no `libvirt host
connected` line before the first call).
2. `Check` reports the one host `Ready` with reason `Reachable` and the
`hostname`, `libvirtVersion`, `poolState` (`running`) and
`poolAvailableGiB` details, which are logged.
3. Every later call goes through `provider.WithTracing` with an in-memory
span recorder that the package's own spans also land in.
4. `Create` returns `hosts/kvm2-test/domains/<name>` (downloading the
image on a cold pool — the log prints how long it took).
5. A second `Create` returns the same ID and the host's domain count,
read over a dedicated connection, is unchanged.
6. `Get`, polled every 5 s, reaches `Running` with an IPv4 within the
budget; its `UID` is the one Create wrote and `CreatedAt` is non-zero
and within a few minutes of now. The time from Create's return to the
first address is logged.
7. `ListByTag` contains the ID with that UID.
8. A TCP connect to port 22 on the guest, logged only — the machine
running the test may not route to the bridge network.
9. `Delete` returns nil, `Get` is `ErrNotFound`, a second `Delete` is a
no-op.
10. The pool, inspected over a dedicated connection, holds the base image
and its marker but no `<name>.qcow2` or `<name>-seed.iso`.
11. Among the recorded spans, a `provider.create` span has at least one
`kvm2.DomainDefineXML` and one `kvm2.dial` child, and no span
attribute carries the cloud-init text.
With `KVM2_TEST_KEEP` steps 9 and 10 are skipped; otherwise a `t.Cleanup`
attempts the Delete even when an earlier assertion failed.
`TestKVM2Check_realHost` runs the provider health check
([kvm2-provider.md](kvm2-provider.md)) against the same host. It must
find the host reachable and the pool running, and it creates nothing. It
needs the same variables; to run only this test:
```sh
go test -tags=integration -count=1 -v -run TestKVM2Check ./internal/provider/kvm2/
```
The test is not part of `make test` and not run in CI. Keep it compiling
with `go vet -tags=integration ./...` when the provider's internals move.
## The scaffolded `make test-e2e` suite
`test/e2e/` is the kubebuilder-generated smoke suite (build image → kind