egress-proxies-operator/docs/testing.md

Testing

What tests exist, how to run them, and where the boundaries of the automated suites are. The philosophy throughout: test against the most real double available — a real envtest API server over a fake client, a real CONNECT-capable proxy stub over a mocked HTTP client, the real lease store under the discovery handlers — and leave the gaps that only real infrastructure can close to the kind verification run, documented at the bottom.

Running

make test              # THE canonical run: codegen + fmt + vet + envtest setup,
                       # then `go test -race` on every package, with coverage
go test -short ./...   # skip the envtest suite (runs in <1s per package)
go tool cover -html=cover.out   # browse coverage from the last make test

Targeted runs:

go test -race ./internal/lease/                  # one package
go test -race -run TestAcquire ./internal/lease/ # one test (or a prefix)
go test -race -count=2 ./internal/health/        # flake-shaking: run twice

Gotcha — plain go test ./... fails the controller package with an error like /usr/local/kubebuilder/bin/etcd: no such file. That is not a code problem: the envtest suite needs KUBEBUILDER_ASSETS pointing at the API-server/etcd binaries, which only make test sets up (via setup-envtest). Either use make test, or export it once:

KUBEBUILDER_ASSETS="$PWD/bin/k8s/<version>-<os>-<arch>" go test -race ./...

Everything runs with -race — make test enforces it; keep the flag on manual runs too.

The suites, package by package

Package	Files	What is covered	Test double / technique
api/v1alpha1	helpers_test.go	EffectivePort/EffectiveHost/HealthCheckOrDefault/MaxLeasesOrDefault	pure table-driven; CEL rules are not unit-testable — see envtest below
internal/controller	reconcile_test.go	every row of the reconciler's action table: create, poll, publish IP, replace, adopt, NotFound recovery, delete/finalizer, quota/permanent/transient errors, cloud-init resolution	controller-runtime/pkg/client/fake + an in-test stubProvider
	status_test.go	computePhase truth table (10 rows)	pure
	spechash_test.go	hash stability, nil/empty normalization, sensitivity to every replacement-triggering field	pure
	health_test.go	Healthy-condition representation: Ready/Unhealthy phases, no-verdict, stale-verdict cleanup on replacement, nil snapshotter	fake client + fakeSnapshotter
	proxy_controller_test.go (envtest, ginkgo)	full lifecycles against a real API server: provision→Running, spec-change replacement, finalizer deletion, External tracking, Ready-through-health, quota-vs-permanent, adoption; all CEL rules (six invalid creates, mode/provider immutability incl. removal) and the default={} materialization	envtest apiserver + stub provider; the only place CEL and structural defaulting actually execute
internal/provider	name_test.go	deterministic naming: idempotency, ^proxy-[a-z2-7]{16}$, 10k-UID distinctness	pure
	errors_test.go	taxonomy: Class() mapping, errors.Is and errors.As through the multi-unwrap	pure
	config_test.go	providers-config parsing + fail-fast validation	pure
	metrics_test.go	WithMetrics decorator: classified result labels, error passthrough	fake recorder + static provider
internal/provider/kubernetes	kubernetes_test.go, pod_test.go	Create/Get/Delete/ListByTag over real Pod objects; pure buildPod/squidConf incl. the max_filedescriptors OOM-regression assertion	client/fake (real corev1.Pods, no kubelet)
	check_test.go	health check: a single cluster-wide list with Limit=1 and the managed-label selector; the reason for every API error class	client/fake + interceptor
internal/provider/gcp	insert_test.go, errors_test.go, gcp_test.go	field-by-field buildInsertRequest; HTTP-code classification table; zone-qualified IDs, 409-is-success, 9-row state mapping, ListByTag filter + ReturnPartialSuccess	pure builder needs no fake; the rest uses the flattened instancesAPI seam
	check_test.go	health check: request shape and the reason per HTTP code or network error; the real SDK client makes exactly one request with maxResults=1, even when a next page exists	fake seam + httptest
internal/provider/kvm	domainxml_test.go, volume_test.go, seed_test.go, id_test.go, metadata_test.go	pure builders: domain/volume XML unmarshalled back and checked field by field, seed ISO reopened with the same library, providerID round-trip, metadata in both namespace spellings	pure
	errors_test.go, conn_test.go	libvirt-code classification table; connection pool (reuse, redial, drop on transport error, cancel releases a blocked RPC, one dial per host under concurrency); SSH dial, host-key policy, keepalive	fake dialer; a real in-process x/crypto/ssh server forwarding a unix socket
	image_test.go, kvm_test.go	image cache (fresh fetch, ready, marker missing, checksum change/mismatch, 4xx/5xx, concurrent callers); Create resume after every partial step, leftover truncated seed, state/IP mapping, every Delete already-gone case, ListByTag across hosts / one down / unreadable domain; logging tiers (V0 silent, V1 one line per RPC, V2 XML) with cloud-init never logged	httptest image server + in-memory fakeAPI behind the libvirtAPI seam
	trace_test.go, conn_test.go, errors_test.go	every libvirtAPI method traced exactly once (guarded by reflection on the interface), results and errors pass through, libvirt error fields, detail keys only at V2, upload content never logged; pool connect/drop lines, no wrapping below V1	fakeAPI + funcr capture
	check_test.go, conn_test.go	health check: details, per-host verdicts in config order, hosts checked concurrently, a timeout that keeps the connection and refuses a second check while the first still waits, no waiting behind a reconcile's dial, the reason table; probe abandons without closing, but still drops on a transport error; waiting for a host's slot ends with the caller's deadline; an inactive or missing pool and daemon errors keep the connection; every dial stage tagged through the real SSH server, with classification unchanged	fakeAPI, fake dialers, in-process SSH server
	kvm_integration_test.go (-tags=integration)	one real VM: image fetch, Create, idempotent re-Create, Running with IP, ListByTag, Delete, pool state afterwards; TestKVMCheck_realHost runs the health check against the host and creates nothing	a real hypervisor — see below
internal/health	probe_test.go	probes through a real CONNECT-capable proxy stub to a real TLS server: tunnel success, refused CONNECT, wrong status, dead proxy, plain-http forward	httptest + hijacked bidirectional tunnel
	engine_test.go	thresholds, first-verdict, latency suppression matrix, dropped-event retry, stale-probe UID guard, tick scheduling/seeding/pruning, Start end-to-end	fake client.Reader + injected probeFn
internal/lease	store_test.go	capacity, MaxLeases=0, all three selection tie-breaks, target-scoped vs global cooldowns, expiry via fake clock, report-on-retained-lease, idempotent release, 40 concurrent acquires never exceeding capacity	injectable clock; the concurrency case is why -race matters
internal/discovery	server_test.go	auth matrix (incl. /healthz exemption), list filtering, grant shape, full 409 arithmetic, invalid TTL/body/result, idempotent release, report→cooldown→409 round trip, Start/shutdown	httptest over the real handler chain, fake cache reader, real lease.Store
internal/gc	gc_test.go	the true-orphan matrix (live/deleting/young/unlabelled all kept), broken-provider isolation, list-failure skips sweep, namespace guard, sweep loop	fake reader + canned-list provider
internal/providerhealth	status_test.go, monitor_test.go	buildStatus (the Ready condition, per-target transition times, truncation); objects created with the label and patched, the round timeout, no-checker Unknown, invalid names, transition-only logging, cleanup of labelled unconfigured objects, the checks-off path, ValidateSchedule	fake client with a cluster-scoped RESTMapper + funcr capture
	envtest_test.go	the generated ProviderHealth schema against a real API server: cluster scope, status subresource, required fields, list-map keys, the 1024-character message limit	envtest (skipped under -short)
internal/metrics	metrics_test.go	scrape-time collectors (GatherAndCompare) incl. the provider-target gauges, per-proxy series cleanup via ForgetProxy, label counts, fresh-registry-per-test property	prometheus/client_golang/testutil

Conventions (from CLAUDE.md): stdlib testing, table-driven by default, t.Parallel() where safe, tests next to source, ginkgo only in the envtest suite the scaffold generated.

What the automated suites deliberately do NOT cover

• New() constructors that dial real infrastructure: the kubernetes provider's New (connects to whatever your kubeconfig points at), the GCP provider's New and its realInstances SDK adapter (ADC + real Google endpoints). Both are thin; both are exercised by the kind run / real deployments. Their 0% coverage is by design — do not "fix" it.
• CEL and structural defaulting under the fake client: the fake client runs neither. Every unit test that relies on defaults calls the *OrDefault helpers; every CEL rule is asserted in the envtest suite instead.
• A container actually starting and serving: envtest has no kubelet, so a Pod created there sits Pending forever. Whether Squid really comes up and tunnels CONNECT is provable only on a real cluster — that is the kind verification's job, and it is exactly what caught the Squid OOM bug (below).
• Live GCP: no test talks to Google. The seam boundary (buildInsertRequest + classification) is tested exhaustively instead.
• Live libvirt, by default: the KVM unit tests run over an in-memory fake of the 20 libvirt calls the provider uses. Whether libvirt accepts the domain XML, whether a raw qcow2 upload plus pool refresh yields a usable backing image, and whether the guest actually boots and reports an address are provable only on a real host — that is the opt-in integration test below.

The KVM integration test (real hypervisor)

internal/provider/kvm/kvm_integration_test.go compiles only under -tags=integration and skips itself unless the environment names a host. It deviates from the "testcontainers for real infra" convention on purpose: libvirt needs /dev/kvm and a bridge to a DHCP network, and a container has neither. So it targets whichever hypervisor you point it at and leaves that host's pool with one more cached base image than it had.

export KVM_TEST_HOST=nas1.home.lan:22          # SSH endpoint
export KVM_TEST_USER=virt                      # must be in the libvirt group
export KVM_TEST_SSH_KEY=$HOME/.ssh/id_ed25519  # private key for that user
export KVM_TEST_KNOWN_HOSTS=$HOME/.ssh/kvm_known_hosts  # the hypervisors' host keys
export KVM_TEST_BRIDGE=br0                     # bridge to a network with DHCP
export KVM_TEST_IMAGE_URL=https://cloud.debian.org/images/cloud/trixie/latest/debian-13-generic-amd64.qcow2
make test-integration-kvm

KVM_TEST_KNOWN_HOSTS is technically optional, but leaving it out turns host-key verification off for the run (the log says so); keep a kvm_known_hosts holding just the hypervisors rather than pointing the test at your everyday known_hosts, so a re-keyed hypervisor is a one-line fix in a file nothing else uses.

Also optional: KVM_TEST_IMAGE_SHA256, KVM_TEST_POOL (default default), KVM_TEST_SOCKET, KVM_TEST_TIMEOUT (default 20m, must cover the first-time image download), KVM_TEST_KEEP to leave the VM behind for virsh console when something goes wrong, and KVM_TEST_LOG_LEVEL (default 1) for how much of the provider's logging the -v output shows: at 1 every libvirt RPC is one compact line (HH:MM:SS.mmm libvirt virDomainLookupByName domain=... duration=... error=... libvirtError=ErrNoDomain), interleaved with the test's own t.Log lines; 2 adds the domain and volume XML, metadata and addresses; 0 leaves only t.Log. The per-run constants (provider, host, domain name) are left off each line. See the Logging section of kvm-provider.md.

What one run asserts, in order: New dials the host; Create returns a hosts/<host>/domains/<name> ID (downloading the image on a cold host — the log prints how long it took); a second Create returns the same ID; Get reaches Running with an IPv4 within the budget, and the UID and CreatedAt survive the metadata round-trip through libvirt; ListByTag lists it; Delete removes it, Get says ErrNotFound, a second Delete is a no-op, and the pool holds the base image and its marker but no per-VM overlay or seed. It also tries a TCP connect to port 22 on the guest, but only logs the outcome — the machine running the test may not route to the bridge network.

TestKVMCheck_realHost runs the provider health check (kvm-provider.md) against the same host. It must find the host reachable and the pool running, and it creates nothing. It needs the same variables; to run only this test:

go test -tags=integration -count=1 -v -run TestKVMCheck ./internal/provider/kvm/

The test is not part of make test and not run in CI. Keep it compiling with go vet -tags=integration ./... when the provider's internals move.

The scaffolded make test-e2e suite

test/e2e/ is the kubebuilder-generated smoke suite (build image → kind cluster → deploy → assert the manager pod runs, publishes a Ready ProviderHealth for the kubernetes provider, and serves metrics that include that provider's proxy_operator_provider_target_up series). It compiles only under -tags=e2e, manages its own kind cluster (make test-e2e / make cleanup-test-e2e), and has been kept compiling (go vet -tags=e2e ./... is part of the routine) but is not part of make test and was not used for the release verification — the manual kind run below covers strictly more.

The OTel tracing spec (Tempo-gated)

test/e2e/tracing_test.go proves the full tracing pipeline against a real Tempo: deploy in kind with tracing enabled, create two kubernetes-provider proxies, delete them, and assert in Tempo that the traces exist with the documented span topology (Reconcile Proxy → reconcile.managed / provider.create / status.patch, and reconcile.delete / provider.delete on the way out).

It skips unless both env vars are set (so the rest of the suite runs anywhere). Homelab invocation:

TEMPO_URL=http://192.168.0.30:3200 \
OTLP_ENDPOINT=http://192.168.0.30:4318 \
make test-e2e

Worth knowing:

• Every span of a run carries the resource attribute test.run.id=e2e-<nanos> (injected via OTEL_RESOURCE_ATTRIBUTES, no code involved); the run ID is printed in the test log, and {resource.test.run.id="<id>"} in Grafana shows exactly that run.
• The spec preflights the OTLP endpoint from inside the cluster with a curl pod and fails fast with a clear message if it's unreachable — export failures are otherwise only visible at -zap-log-level=1.
• Proxy CRs are created in default, not the operator namespace: the squid pods carry no securityContext and the operator namespace enforces restricted PSS.

The kind verification run (the real end-to-end)

The README quickstart is the e2e test, run by hand before the MR: deploy the operator in-cluster on a throwaway kind cluster, watch a real Squid pod reach Ready via a real CONNECT probe, then exercise the whole lease lifecycle through the discovery API (list → lease → report rate_limited → same-target lease answered 409 inCooldown:1 → release 204 twice) and delete the CR to watch the finalizer remove the pod.

Worth knowing about that run (full detail in plans-executions/2026-08-07-1747-proxy-operator.md, "Verification" section): it was not a clean pass-through. It caught two real bugs the automated suites structurally could not —

1. make run-dev on a laptop can never produce a Ready kubernetes-provider proxy: health probes originate on the host, which cannot route to kind's pod network. The quickstart was rewritten to deploy in-cluster.
2. Squid was OOM-killed at startup: it sizes file-descriptor tables from RLIMIT_NOFILE, which containerd under kind sets effectively unlimited. Fixed with max_filedescriptors 1024 in the generated config, plus a regression assertion in pod_test.go.

That is the pattern to keep: when the kind run finds something, the fix lands with a unit-level regression test, so the manual run stays a discovery tool rather than a recurring gate.