egress-proxies-operator/docs/testing.md

Testing

What tests exist, how to run them, and where the boundaries of the automated suites are. The philosophy throughout: test against the most real double available — a real envtest API server over a fake client, a real CONNECT-capable proxy stub over a mocked HTTP client, the real lease store under the discovery handlers — and leave the gaps that only real infrastructure can close to the kind verification run, documented at the bottom.

Running

make test              # THE canonical run: codegen + fmt + vet + envtest setup,
                       # then `go test -race` on every package, with coverage
go test -short ./...   # skip the envtest suite (runs in <1s per package)
go tool cover -html=cover.out   # browse coverage from the last make test

Targeted runs:

go test -race ./internal/lease/                  # one package
go test -race -run TestAcquire ./internal/lease/ # one test (or a prefix)
go test -race -count=2 ./internal/health/        # flake-shaking: run twice

Gotcha — plain go test ./... fails the controller package with an error like /usr/local/kubebuilder/bin/etcd: no such file. That is not a code problem: the envtest suite needs KUBEBUILDER_ASSETS pointing at the API-server/etcd binaries, which only make test sets up (via setup-envtest). Either use make test, or export it once:

KUBEBUILDER_ASSETS="$PWD/bin/k8s/<version>-<os>-<arch>" go test -race ./...

Everything runs with -race — make test enforces it; keep the flag on manual runs too.

The suites, package by package

Package	Files	What is covered	Test double / technique
api/v1alpha1	helpers_test.go	EffectivePort/EffectiveHost/HealthCheckOrDefault/MaxLeasesOrDefault	pure table-driven; CEL rules are not unit-testable — see envtest below
internal/controller	reconcile_test.go	every row of the reconciler's action table: create, poll, publish IP, replace, adopt, NotFound recovery, delete/finalizer, quota/permanent/transient errors, cloud-init resolution	controller-runtime/pkg/client/fake + an in-test stubProvider
	status_test.go	computePhase truth table (10 rows)	pure
	spechash_test.go	hash stability, nil/empty normalization, sensitivity to every replacement-triggering field	pure
	health_test.go	Healthy-condition representation: Ready/Unhealthy phases, no-verdict, stale-verdict cleanup on replacement, nil snapshotter	fake client + fakeSnapshotter
	proxy_controller_test.go (envtest, ginkgo)	full lifecycles against a real API server: provision→Running, spec-change replacement, finalizer deletion, External tracking, Ready-through-health, quota-vs-permanent, adoption; all CEL rules (six invalid creates, mode/provider immutability incl. removal) and the default={} materialization	envtest apiserver + stub provider; the only place CEL and structural defaulting actually execute
internal/provider	name_test.go	deterministic naming: idempotency, ^proxy-[a-z2-7]{16}$, 10k-UID distinctness	pure
	errors_test.go	taxonomy: Class() mapping, errors.Is and errors.As through the multi-unwrap	pure
	config_test.go	providers-config parsing + fail-fast validation	pure
	metrics_test.go	WithMetrics decorator: classified result labels, error passthrough	fake recorder + static provider
internal/provider/kubernetes	kubernetes_test.go, pod_test.go	Create/Get/Delete/ListByTag over real Pod objects; pure buildPod/squidConf incl. the max_filedescriptors OOM-regression assertion	client/fake (real corev1.Pods, no kubelet)
	check_test.go	health check: a single cluster-wide list with Limit=1 and the managed-label selector; the reason for every API error class	client/fake + interceptor
internal/provider/gcp	insert_test.go, errors_test.go, gcp_test.go	field-by-field buildInsertRequest; HTTP-code classification table; zone-qualified IDs, 409-is-success, 9-row state mapping, ListByTag filter + ReturnPartialSuccess	pure builder needs no fake; the rest uses the flattened instancesAPI seam
	check_test.go	health check: request shape and the reason per HTTP code or network error; the real SDK client makes exactly one request with maxResults=1, even when a next page exists	fake seam + httptest
internal/provider/kvm	domainxml_test.go, volume_test.go, seed_test.go, id_test.go, metadata_test.go	pure builders: domain/volume XML unmarshalled back and checked field by field, seed ISO reopened with the same library, providerID round-trip, metadata in both namespace spellings	pure
	errors_test.go, conn_test.go	libvirt-code classification table; connection pool (reuse, redial, drop on transport error, cancel releases a blocked RPC, one dial per host under concurrency); SSH dial, host-key policy, keepalive	fake dialer; a real in-process x/crypto/ssh server forwarding a unix socket
	image_test.go, kvm_test.go	image cache (fresh fetch, ready, marker missing, checksum change/mismatch, 4xx/5xx, concurrent callers); Create resume after every partial step, leftover truncated seed, state/IP mapping, every Delete already-gone case, ListByTag across hosts / one down / unreadable domain; logging tiers (V0 silent, V1 one line per RPC, V2 XML) with cloud-init never logged	httptest image server + in-memory fakeAPI behind the libvirtAPI seam
	trace_test.go, conn_test.go, errors_test.go	every libvirtAPI method traced exactly once (guarded by reflection on the interface), results and errors pass through, libvirt error fields, detail keys only at V2, upload content never logged; pool connect/drop lines, no wrapping below V1	fakeAPI + funcr capture
	check_test.go, conn_test.go	health check: details, per-host verdicts in config order, hosts checked concurrently, a timeout that keeps the connection and refuses a second check while the first still waits, no waiting behind a reconcile's dial, the reason table; probe abandons without closing, but still drops on a transport error; waiting for a host's slot ends with the caller's deadline; an inactive or missing pool and daemon errors keep the connection; every dial stage tagged through the real SSH server, with classification unchanged	fakeAPI, fake dialers, in-process SSH server
	kvm_integration_test.go (-tags=integration)	one real VM: image fetch, Create, idempotent re-Create, Running with IP, ListByTag, Delete, pool state afterwards; TestKVMCheck_realHost runs the health check against the host and creates nothing	a real hypervisor — see below
internal/provider/kvm2	xml_test.go, volume_test.go, seed_test.go, id_test.go, metadata_test.go, state_test.go	pure builders: domain XML (file disks with resolved paths, UEFI with secure boot off, BIOS escape, the metadata element, the guest-agent channel), volume XML with and without a backing store, seed ISO reopened (files, label, user-data byte for byte, temp dir removed) and the vendor-data script vs. multipart rules, providerID round-trip and malformed forms, metadata in both namespace spellings, state mapping and the IPv4 filter	pure
	errors_test.go, backend_test.go, hosts_test.go, ssh_test.go	classification table (libvirt codes, SSH auth text, knownhosts.KeyError, missing files); the per-call budget adapter (deadline kills the connection and is Transient, cancel unblocks, killed by another caller, close idempotent) against a mocked libvirt socket; the host table (one dial per host under concurrency, redial after drop, dialFresh never cached, waiting ends with ctx); the real dialer end to end: socket forwarding, unknown and mismatched host keys, insecure mode, missing vs. unparsable key and known_hosts, wrong key, handshake timeout, cancel, every stage tagged, key material never in errors or logs	fake dialer; a real in-process x/crypto/ssh server forwarding a unix socket
	image_test.go, create_test.go, get_test.go, delete_test.go, list_test.go, kvm2_test.go, provider_test.go	image cache (fetch with the marker written last, cached, marker missing or without a digest, sha256 change or mismatch, 4xx/5xx, no Content-Length, short read, concurrent callers fetch once, cancel mid-stream, streamed without buffering, upload on a dedicated connection) and the pool-space quota precheck; Create resume after every partial step, returns only when running, adoption guard on UID, seed always rebuilt, cloud-init sentinel never logged; Get state and address mapping incl. agent-before-ARP; every Delete already-gone case, NVRAM flag, per-name lock; ListByTag across hosts, one down fails the whole call, unreadable domain skipped; New never dials and warns on missing files; every error wrapped with the configured name; logging tiers V0/V1/V2	httptest image server + in-memory fakeBackend behind the backend seam
	observe_test.go, check_test.go	every seam method traced and logged exactly once (guarded by reflection), child spans under the parent, detail only at V2, upload bodies never logged, error fields; health check: one entry per host in config order, hosts checked concurrently, expired ctx returns promptly, never touches the pooled connection, the reason table, detail keys	fakeBackend + funcr capture + span recorder
	kvm2_integration_test.go (-tags=integration)	one real VM: New without a dial, Check Ready with details, Create, idempotent re-Create with an unchanged domain count, Running with IP and the metadata round-trip, ListByTag, Delete, NotFound, pool state afterwards, kvm2.* spans under provider.create; TestKVM2Check_realHost runs the health check against the host and creates nothing	a real hypervisor — see below
internal/health	probe_test.go	probes through a real CONNECT-capable proxy stub to a real TLS server: tunnel success, refused CONNECT, wrong status, dead proxy, plain-http forward	httptest + hijacked bidirectional tunnel
	engine_test.go	thresholds, first-verdict, latency suppression matrix, dropped-event retry, stale-probe UID guard, tick scheduling/seeding/pruning, Start end-to-end	fake client.Reader + injected probeFn
internal/lease	store_test.go	capacity, MaxLeases=0, all three selection tie-breaks, target-scoped vs global cooldowns, expiry via fake clock, report-on-retained-lease, idempotent release, 40 concurrent acquires never exceeding capacity	injectable clock; the concurrency case is why -race matters
internal/discovery	server_test.go	auth matrix (incl. /healthz exemption), list filtering, grant shape, full 409 arithmetic, invalid TTL/body/result, idempotent release, report→cooldown→409 round trip, Start/shutdown	httptest over the real handler chain, fake cache reader, real lease.Store
internal/gc	gc_test.go	the true-orphan matrix (live/deleting/young/unlabelled all kept), broken-provider isolation, list-failure skips sweep, namespace guard, sweep loop	fake reader + canned-list provider
internal/providerhealth	status_test.go, monitor_test.go	buildStatus (the Ready condition, per-target transition times, truncation); objects created with the label and patched, the round timeout, no-checker Unknown, invalid names, transition-only logging, cleanup of labelled unconfigured objects, the checks-off path, ValidateSchedule	fake client with a cluster-scoped RESTMapper + funcr capture
	envtest_test.go	the generated ProviderHealth schema against a real API server: cluster scope, status subresource, required fields, list-map keys, the 1024-character message limit	envtest (skipped under -short)
internal/metrics	metrics_test.go	scrape-time collectors (GatherAndCompare) incl. the provider-target gauges, per-proxy series cleanup via ForgetProxy, label counts, fresh-registry-per-test property	prometheus/client_golang/testutil

Conventions (from CLAUDE.md): stdlib testing, table-driven by default, t.Parallel() where safe, tests next to source, ginkgo only in the envtest suite the scaffold generated.

What the automated suites deliberately do NOT cover

• New() constructors that dial real infrastructure: the kubernetes provider's New (connects to whatever your kubeconfig points at), the GCP provider's New and its realInstances SDK adapter (ADC + real Google endpoints). Both are thin; both are exercised by the kind run / real deployments. Their 0% coverage is by design — do not "fix" it.
• CEL and structural defaulting under the fake client: the fake client runs neither. Every unit test that relies on defaults calls the *OrDefault helpers; every CEL rule is asserted in the envtest suite instead.
• A container actually starting and serving: envtest has no kubelet, so a Pod created there sits Pending forever. Whether Squid really comes up and tunnels CONNECT is provable only on a real cluster — that is the kind verification's job, and it is exactly what caught the Squid OOM bug (below).
• Live GCP: no test talks to Google. The seam boundary (buildInsertRequest + classification) is tested exhaustively instead.
• Live libvirt, by default: the KVM and kvm2 unit tests run over an in-memory fake of the 20 libvirt calls each provider uses. Whether libvirt accepts the domain XML, whether a raw qcow2 upload plus pool refresh yields a usable backing image, and whether the guest actually boots and reports an address are provable only on a real host — that is the opt-in integration test below (one per provider).

The KVM integration test (real hypervisor)

internal/provider/kvm/kvm_integration_test.go compiles only under -tags=integration and skips itself unless the environment names a host. It deviates from the "testcontainers for real infra" convention on purpose: libvirt needs /dev/kvm and a bridge to a DHCP network, and a container has neither. So it targets whichever hypervisor you point it at and leaves that host's pool with one more cached base image than it had.

export KVM_TEST_HOST=nas1.home.lan:22          # SSH endpoint
export KVM_TEST_USER=virt                      # must be in the libvirt group
export KVM_TEST_SSH_KEY=$HOME/.ssh/id_ed25519  # private key for that user
export KVM_TEST_KNOWN_HOSTS=$HOME/.ssh/kvm_known_hosts  # the hypervisors' host keys
export KVM_TEST_BRIDGE=br0                     # bridge to a network with DHCP
export KVM_TEST_IMAGE_URL=https://cloud.debian.org/images/cloud/trixie/latest/debian-13-generic-amd64.qcow2
make test-integration-kvm

KVM_TEST_KNOWN_HOSTS is technically optional, but leaving it out turns host-key verification off for the run (the log says so); keep a kvm_known_hosts holding just the hypervisors rather than pointing the test at your everyday known_hosts, so a re-keyed hypervisor is a one-line fix in a file nothing else uses.

Also optional: KVM_TEST_IMAGE_SHA256, KVM_TEST_POOL (default default), KVM_TEST_SOCKET, KVM_TEST_TIMEOUT (default 20m, must cover the first-time image download), KVM_TEST_KEEP to leave the VM behind for virsh console when something goes wrong, and KVM_TEST_LOG_LEVEL (default 1) for how much of the provider's logging the -v output shows: at 1 every libvirt RPC is one compact line (HH:MM:SS.mmm libvirt virDomainLookupByName domain=... duration=... error=... libvirtError=ErrNoDomain), interleaved with the test's own t.Log lines; 2 adds the domain and volume XML, metadata and addresses; 0 leaves only t.Log. The per-run constants (provider, host, domain name) are left off each line. See the Logging section of kvm-provider.md.

What one run asserts, in order: New dials the host; Create returns a hosts/<host>/domains/<name> ID (downloading the image on a cold host — the log prints how long it took); a second Create returns the same ID; Get reaches Running with an IPv4 within the budget, and the UID and CreatedAt survive the metadata round-trip through libvirt; ListByTag lists it; Delete removes it, Get says ErrNotFound, a second Delete is a no-op, and the pool holds the base image and its marker but no per-VM overlay or seed. It also tries a TCP connect to port 22 on the guest, but only logs the outcome — the machine running the test may not route to the bridge network.

TestKVMCheck_realHost runs the provider health check (kvm-provider.md) against the same host. It must find the host reachable and the pool running, and it creates nothing. It needs the same variables; to run only this test:

go test -tags=integration -count=1 -v -run TestKVMCheck ./internal/provider/kvm/

The test is not part of make test and not run in CI. Keep it compiling with go vet -tags=integration ./... when the provider's internals move.

The kvm2 integration test (real hypervisor)

internal/provider/kvm2/kvm2_integration_test.go compiles only under -tags=integration and skips itself unless the environment names a host. Like the kvm one it deviates from the "testcontainers for real infra" convention on purpose: libvirt needs /dev/kvm and a bridge to a DHCP network, and a container has neither. It targets whichever hypervisor you point it at and leaves that host's pool with one more cached base image (base-test-image.qcow2 and its .sha256 marker) than it had. The kvm and kvm2 tests can share a host: kvm2 marks its domains in its own metadata namespace and never lists or adopts the other's.

export KVM2_TEST_HOST=beelink:22               # SSH endpoint
export KVM2_TEST_USER=novakj                   # must be in the libvirt group
export KVM2_TEST_SSH_KEY=$HOME/.ssh/id_ed25519 # private key for that user
export KVM2_TEST_KNOWN_HOSTS=$HOME/.ssh/kvm_known_hosts  # the hypervisors' host keys
export KVM2_TEST_BRIDGE=br0                    # bridge to a network with DHCP
export KVM2_TEST_IMAGE_URL=https://cloud.debian.org/images/cloud/trixie/latest/debian-13-generic-amd64.qcow2
make test-integration-kvm2

KVM2_TEST_KNOWN_HOSTS is technically optional, but leaving it out turns host-key verification off for the run (the log says so); keep a kvm_known_hosts holding just the hypervisors rather than pointing the test at your everyday known_hosts, so a re-keyed hypervisor is a one-line fix in a file nothing else uses.

Also optional: KVM2_TEST_IMAGE_SHA256 (verified after download and compared with the cached marker), KVM2_TEST_POOL (default default), KVM2_TEST_SOCKET (default /var/run/libvirt/libvirt-sock), KVM2_TEST_TIMEOUT (default 20m, the budget for the whole lifecycle; it must cover the first-time image download), KVM2_TEST_KEEP to leave the VM behind for virsh console when something goes wrong (the log then prints the virsh undefine --nvram and virsh vol-delete lines that clean it up), and KVM2_TEST_LOG_LEVEL (default 1) for how much of the provider's logging the -v output shows: at 1 every libvirt RPC is one compact libvirt call line (HH:MM:SS.mmm kvm2 "level"=1 "msg"="libvirt call" "rpc"="DomainLookupByName" "domain"=... "duration"=..., plus the per-operation outcome lines and connection events), interleaved with the test's own t.Log lines; 2 adds the domain and volume XML, metadata and addresses; 0 leaves only t.Log. The per-run constants (provider, host, domain name) are stated once in the first line and left off every provider line. See the Logging section of kvm2-provider.md.

What one run asserts, in order, each step logged with its duration:

1. New returns within 100 ms and without dialling (no libvirt host connected line before the first call).
2. Check reports the one host Ready with reason Reachable and the hostname, libvirtVersion, poolState (running) and poolAvailableGiB details, which are logged.
3. Every later call goes through provider.WithTracing with an in-memory span recorder that the package's own spans also land in.
4. Create returns hosts/kvm2-test/domains/<name> (downloading the image on a cold pool — the log prints how long it took).
5. A second Create returns the same ID and the host's domain count, read over a dedicated connection, is unchanged.
6. Get, polled every 5 s, reaches Running with an IPv4 within the budget; its UID is the one Create wrote and CreatedAt is non-zero and within a few minutes of now. The time from Create's return to the first address is logged.
7. ListByTag contains the ID with that UID.
8. A TCP connect to port 22 on the guest, logged only — the machine running the test may not route to the bridge network.
9. Delete returns nil, Get is ErrNotFound, a second Delete is a no-op.
10. The pool, inspected over a dedicated connection, holds the base image and its marker but no <name>.qcow2 or <name>-seed.iso.
11. Among the recorded spans, a provider.create span has at least one kvm2.DomainDefineXML and one kvm2.dial child, and no span attribute carries the cloud-init text.

With KVM2_TEST_KEEP steps 9 and 10 are skipped; otherwise a t.Cleanup attempts the Delete even when an earlier assertion failed.

TestKVM2Check_realHost runs the provider health check (kvm2-provider.md) against the same host. It must find the host reachable and the pool running, and it creates nothing. It needs the same variables; to run only this test:

go test -tags=integration -count=1 -v -run TestKVM2Check ./internal/provider/kvm2/

The test is not part of make test and not run in CI. Keep it compiling with go vet -tags=integration ./... when the provider's internals move.

The scaffolded make test-e2e suite

test/e2e/ is the kubebuilder-generated smoke suite (build image → kind cluster → deploy → assert the manager pod runs, publishes a Ready ProviderHealth for the kubernetes provider, and serves metrics that include that provider's proxy_operator_provider_target_up series). It compiles only under -tags=e2e, manages its own kind cluster (make test-e2e / make cleanup-test-e2e), and has been kept compiling (go vet -tags=e2e ./... is part of the routine) but is not part of make test and was not used for the release verification — the manual kind run below covers strictly more.

The OTel tracing spec (Tempo-gated)

test/e2e/tracing_test.go proves the full tracing pipeline against a real Tempo: deploy in kind with tracing enabled, create two kubernetes-provider proxies, delete them, and assert in Tempo that the traces exist with the documented span topology (Reconcile Proxy → reconcile.managed / provider.create / status.patch, and reconcile.delete / provider.delete on the way out).

It skips unless both env vars are set (so the rest of the suite runs anywhere). Homelab invocation:

TEMPO_URL=http://192.168.0.30:3200 \
OTLP_ENDPOINT=http://192.168.0.30:4318 \
make test-e2e

Worth knowing:

• Every span of a run carries the resource attribute test.run.id=e2e-<nanos> (injected via OTEL_RESOURCE_ATTRIBUTES, no code involved); the run ID is printed in the test log, and {resource.test.run.id="<id>"} in Grafana shows exactly that run.
• The spec preflights the OTLP endpoint from inside the cluster with a curl pod and fails fast with a clear message if it's unreachable — export failures are otherwise only visible at -zap-log-level=1.
• Proxy CRs are created in default, not the operator namespace: the squid pods carry no securityContext and the operator namespace enforces restricted PSS.

The kind verification run (the real end-to-end)

The README quickstart is the e2e test, run by hand before the MR: deploy the operator in-cluster on a throwaway kind cluster, watch a real Squid pod reach Ready via a real CONNECT probe, then exercise the whole lease lifecycle through the discovery API (list → lease → report rate_limited → same-target lease answered 409 inCooldown:1 → release 204 twice) and delete the CR to watch the finalizer remove the pod.

Worth knowing about that run (full detail in plans-executions/2026-08-07-1747-proxy-operator.md, "Verification" section): it was not a clean pass-through. It caught two real bugs the automated suites structurally could not —

1. make run-dev on a laptop can never produce a Ready kubernetes-provider proxy: health probes originate on the host, which cannot route to kind's pod network. The quickstart was rewritten to deploy in-cluster.
2. Squid was OOM-killed at startup: it sizes file-descriptor tables from RLIMIT_NOFILE, which containerd under kind sets effectively unlimited. Fixed with max_filedescriptors 1024 in the generated config, plus a regression assertion in pod_test.go.

That is the pattern to keep: when the kind run finds something, the fix lands with a unit-level regression test, so the manual run stays a discovery tool rather than a recurring gate.