egress-proxies-operator/docs/testing.md

# Testing

What tests exist, how to run them, and where the boundaries of the
automated suites are. The philosophy throughout: **test against the most
real double available** — a real envtest API server over a fake client, a
real CONNECT-capable proxy stub over a mocked HTTP client, the real lease
store under the discovery handlers — and leave the gaps that only real
infrastructure can close to the kind verification run, documented at the
bottom.

## Running

```sh
make test              # THE canonical run: codegen + fmt + vet + envtest setup,
                       # then `go test -race` on every package, with coverage
go test -short ./...   # skip the envtest suite (runs in <1s per package)
go tool cover -html=cover.out   # browse coverage from the last make test
```

Targeted runs:

```sh
go test -race ./internal/lease/                  # one package
go test -race -run TestAcquire ./internal/lease/ # one test (or a prefix)
go test -race -count=2 ./internal/health/        # flake-shaking: run twice
```

**Gotcha — plain `go test ./...` fails the controller package** with an
error like `/usr/local/kubebuilder/bin/etcd: no such file`. That is not a
code problem: the envtest suite needs `KUBEBUILDER_ASSETS` pointing at the
API-server/etcd binaries, which only `make test` sets up (via
`setup-envtest`). Either use `make test`, or export it once:

```sh
KUBEBUILDER_ASSETS="$PWD/bin/k8s/<version>-<os>-<arch>" go test -race ./...
```

Everything runs with `-race` — `make test` enforces it; keep the flag on
manual runs too.

## The suites, package by package

| Package | Files | What is covered | Test double / technique |
|---|---|---|---|
| `api/v1alpha1` | `helpers_test.go` | `EffectivePort/EffectiveHost/HealthCheckOrDefault/MaxLeasesOrDefault` | pure table-driven; CEL rules are **not** unit-testable — see envtest below |
| `internal/controller` | `reconcile_test.go` | every row of the reconciler's action table: create, poll, publish IP, replace, adopt, NotFound recovery, delete/finalizer, quota/permanent/transient errors, cloud-init resolution | `controller-runtime/pkg/client/fake` + an in-test `stubProvider` |
| | `status_test.go` | `computePhase` truth table (10 rows) | pure |
| | `spechash_test.go` | hash stability, nil/empty normalization, sensitivity to every replacement-triggering field | pure |
| | `health_test.go` | Healthy-condition representation: Ready/Unhealthy phases, no-verdict, stale-verdict cleanup on replacement, nil snapshotter | fake client + `fakeSnapshotter` |
| | `proxy_controller_test.go` (envtest, ginkgo) | full lifecycles against a **real API server**: provision→Running, spec-change replacement, finalizer deletion, External tracking, Ready-through-health, quota-vs-permanent, adoption; **all CEL rules** (six invalid creates, mode/provider immutability incl. removal) and the `default={}` materialization | envtest apiserver + stub provider; the only place CEL and structural defaulting actually execute |
| `internal/provider` | `name_test.go` | deterministic naming: idempotency, `^proxy-[a-z2-7]{16}$`, 10k-UID distinctness | pure |
| | `errors_test.go` | taxonomy: `Class()` mapping, `errors.Is` **and** `errors.As` through the multi-unwrap | pure |
| | `config_test.go` | providers-config parsing + fail-fast validation | pure |
| | `metrics_test.go` | `WithMetrics` decorator: classified result labels, error passthrough | fake recorder + static provider |
| `internal/provider/kubernetes` | `kubernetes_test.go`, `pod_test.go` | Create/Get/Delete/ListByTag over real Pod objects; pure `buildPod`/`squidConf` incl. the `max_filedescriptors` OOM-regression assertion | `client/fake` (real `corev1.Pod`s, no kubelet) |
| | `check_test.go` | health check: a single cluster-wide list with `Limit=1` and the managed-label selector; the reason for every API error class | `client/fake` + interceptor |
| `internal/provider/gcp` | `insert_test.go`, `errors_test.go`, `gcp_test.go` | field-by-field `buildInsertRequest`; HTTP-code classification table; zone-qualified IDs, 409-is-success, 9-row state mapping, ListByTag filter + `ReturnPartialSuccess` | pure builder needs **no fake**; the rest uses the flattened `instancesAPI` seam |
| | `check_test.go` | health check: request shape and the reason per HTTP code or network error; the **real SDK client** makes exactly one request with `maxResults=1`, even when a next page exists | fake seam + `httptest` |
| `internal/provider/kvm` | `domainxml_test.go`, `volume_test.go`, `seed_test.go`, `id_test.go`, `metadata_test.go` | pure builders: domain/volume XML unmarshalled back and checked field by field, seed ISO reopened with the same library, providerID round-trip, metadata in both namespace spellings | pure |
| | `errors_test.go`, `conn_test.go` | libvirt-code classification table; connection pool (reuse, redial, drop on transport error, cancel releases a blocked RPC, one dial per host under concurrency); SSH dial, host-key policy, keepalive | fake dialer; a **real in-process `x/crypto/ssh` server** forwarding a unix socket |
| | `image_test.go`, `kvm_test.go` | image cache (fresh fetch, ready, marker missing, checksum change/mismatch, 4xx/5xx, concurrent callers); Create resume after every partial step, leftover truncated seed, state/IP mapping, every Delete already-gone case, ListByTag across hosts / one down / unreadable domain; logging tiers (V0 silent, V1 one line per RPC, V2 XML) with cloud-init never logged | `httptest` image server + in-memory `fakeAPI` behind the `libvirtAPI` seam |
| | `trace_test.go`, `conn_test.go`, `errors_test.go` | every `libvirtAPI` method traced exactly once (guarded by reflection on the interface), results and errors pass through, libvirt error fields, detail keys only at V2, upload content never logged; pool connect/drop lines, no wrapping below V1 | `fakeAPI` + `funcr` capture |
| | `check_test.go`, `conn_test.go` | health check: details, per-host verdicts in config order, hosts checked concurrently, a timeout that keeps the connection and refuses a second check while the first still waits, no waiting behind a reconcile's dial, the reason table; `probe` abandons without closing, but still drops on a transport error; waiting for a host's slot ends with the caller's deadline; an inactive or missing pool and daemon errors keep the connection; every dial stage tagged through the real SSH server, with classification unchanged | `fakeAPI`, fake dialers, in-process SSH server |
| | `kvm_integration_test.go` (`-tags=integration`) | one real VM: image fetch, Create, idempotent re-Create, Running with IP, ListByTag, Delete, pool state afterwards; `TestKVMCheck_realHost` runs the health check against the host and creates nothing | a **real hypervisor** — see below |
| `internal/health` | `probe_test.go` | probes through a **real CONNECT-capable proxy stub** to a real TLS server: tunnel success, refused CONNECT, wrong status, dead proxy, plain-http forward | `httptest` + hijacked bidirectional tunnel |
| | `engine_test.go` | thresholds, first-verdict, latency suppression matrix, dropped-event retry, stale-probe UID guard, tick scheduling/seeding/pruning, `Start` end-to-end | fake `client.Reader` + injected `probeFn` |
| `internal/lease` | `store_test.go` | capacity, `MaxLeases=0`, all three selection tie-breaks, target-scoped vs global cooldowns, expiry via fake clock, report-on-retained-lease, idempotent release, **40 concurrent acquires never exceeding capacity** | injectable clock; the concurrency case is why `-race` matters |
| `internal/discovery` | `server_test.go` | auth matrix (incl. `/healthz` exemption), list filtering, grant shape, full 409 arithmetic, invalid TTL/body/result, idempotent release, report→cooldown→409 round trip, `Start`/shutdown | `httptest` over the real handler chain, fake cache reader, **real** `lease.Store` |
| `internal/gc` | `gc_test.go` | the true-orphan matrix (live/deleting/young/unlabelled all kept), broken-provider isolation, list-failure skips sweep, namespace guard, sweep loop | fake reader + canned-list provider |
| `internal/providerhealth` | `status_test.go`, `monitor_test.go` | `buildStatus` (the Ready condition, per-target transition times, truncation); objects created with the label and patched, the round timeout, no-checker Unknown, invalid names, transition-only logging, cleanup of labelled unconfigured objects, the checks-off path, `ValidateSchedule` | fake client with a cluster-scoped RESTMapper + `funcr` capture |
| | `envtest_test.go` | the generated `ProviderHealth` schema against a real API server: cluster scope, status subresource, required fields, list-map keys, the 1024-character message limit | envtest (skipped under `-short`) |
| `internal/metrics` | `metrics_test.go` | scrape-time collectors (`GatherAndCompare`) incl. the provider-target gauges, per-proxy series cleanup via `ForgetProxy`, label counts, fresh-registry-per-test property | `prometheus/client_golang/testutil` |

Conventions (from `CLAUDE.md`): stdlib `testing`, table-driven by default,
`t.Parallel()` where safe, tests next to source, ginkgo only in the
envtest suite the scaffold generated.

## What the automated suites deliberately do NOT cover

- **`New()` constructors that dial real infrastructure**: the kubernetes
  provider's `New` (connects to whatever your kubeconfig points at), the
  GCP provider's `New` and its `realInstances` SDK adapter (ADC + real
  Google endpoints). Both are thin; both are exercised by the kind run /
  real deployments. Their 0% coverage is by design — do not "fix" it.
- **CEL and structural defaulting under the fake client**: the fake
  client runs neither. Every unit test that relies on defaults calls the
  `*OrDefault` helpers; every CEL rule is asserted in the envtest suite
  instead.
- **A container actually starting and serving**: envtest has no kubelet,
  so a Pod created there sits Pending forever. Whether Squid really comes
  up and tunnels CONNECT is provable only on a real cluster — that is the
  kind verification's job, and it is exactly what caught the Squid OOM
  bug (below).
- **Live GCP**: no test talks to Google. The seam boundary
  (`buildInsertRequest` + classification) is tested exhaustively instead.
- **Live libvirt, by default**: the KVM unit tests run over an in-memory
  fake of the 20 libvirt calls the provider uses. Whether libvirt accepts
  the domain XML, whether a raw qcow2 upload plus pool refresh yields a
  usable backing image, and whether the guest actually boots and reports
  an address are provable only on a real host — that is the opt-in
  integration test below.

## The KVM integration test (real hypervisor)

`internal/provider/kvm/kvm_integration_test.go` compiles only under
`-tags=integration` and skips itself unless the environment names a
host. It deviates from the "testcontainers for real infra" convention on
purpose: libvirt needs `/dev/kvm` and a bridge to a DHCP network, and a
container has neither. So it targets whichever hypervisor you point it
at and leaves that host's pool with one more cached base image than it
had.

```sh
export KVM_TEST_HOST=nas1.home.lan:22          # SSH endpoint
export KVM_TEST_USER=virt                      # must be in the libvirt group
export KVM_TEST_SSH_KEY=$HOME/.ssh/id_ed25519  # private key for that user
export KVM_TEST_BRIDGE=br0                     # bridge to a network with DHCP
export KVM_TEST_IMAGE_URL=https://cloud.debian.org/images/cloud/trixie/latest/debian-13-generic-amd64.qcow2
make test-integration-kvm
```

Optional: `KVM_TEST_KNOWN_HOSTS` (without it host keys are **not**
verified — the log says so), `KVM_TEST_IMAGE_SHA256`, `KVM_TEST_POOL`
(default `default`), `KVM_TEST_SOCKET`, `KVM_TEST_TIMEOUT` (default
`20m`, must cover the first-time image download), `KVM_TEST_KEEP`
to leave the VM behind for `virsh console` when something goes wrong,
and `KVM_TEST_LOG_LEVEL` (default `1`) for how much of the provider's
logging the `-v` output shows: at `1` every libvirt RPC is one compact
line (`HH:MM:SS.mmm libvirt virDomainLookupByName domain=... duration=...
error=... libvirtError=ErrNoDomain`), interleaved with the test's own
`t.Log` lines; `2` adds the domain and volume XML, metadata and
addresses; `0` leaves only `t.Log`. The per-run constants (provider,
host, domain name) are left off each line. See the Logging section of
[kvm-provider.md](kvm-provider.md#logging).

What one run asserts, in order: `New` dials the host; `Create` returns a
`hosts/<host>/domains/<name>` ID (downloading the image on a cold host —
the log prints how long it took); a second `Create` returns the same ID;
`Get` reaches `Running` with an IPv4 within the budget, and the UID and
`CreatedAt` survive the metadata round-trip through libvirt; `ListByTag`
lists it; `Delete` removes it, `Get` says `ErrNotFound`, a second
`Delete` is a no-op, and the pool holds the base image and its marker
but no per-VM overlay or seed. It also tries a TCP connect to port 22 on
the guest, but only logs the outcome — the machine running the test may
not route to the bridge network.

`TestKVMCheck_realHost` runs the provider health check
([kvm-provider.md](kvm-provider.md#health-check)) against the same host.
It must find the host reachable and the pool running, and it creates
nothing. It needs the same variables; to run only this test:

```sh
go test -tags=integration -count=1 -v -run TestKVMCheck ./internal/provider/kvm/
```

The test is not part of `make test` and not run in CI. Keep it compiling
with `go vet -tags=integration ./...` when the provider's internals move.

## The scaffolded `make test-e2e` suite

`test/e2e/` is the kubebuilder-generated smoke suite (build image → kind
cluster → deploy → assert the manager pod runs, publishes a Ready
`ProviderHealth` for the `kubernetes` provider, and serves metrics that
include that provider's `proxy_operator_provider_target_up` series). It
compiles only under `-tags=e2e`, manages its own kind cluster
(`make test-e2e` / `make cleanup-test-e2e`), and has been kept compiling
(`go vet -tags=e2e ./...` is part of the routine) but is **not part of
`make test` and was not used for the release verification** — the manual
kind run below covers strictly more.

### The OTel tracing spec (Tempo-gated)

`test/e2e/tracing_test.go` proves the full tracing pipeline against a
**real Tempo**: deploy in kind with tracing enabled, create two
kubernetes-provider proxies, delete them, and assert in Tempo that the
traces exist with the documented span topology (`Reconcile Proxy` →
`reconcile.managed` / `provider.create` / `status.patch`, and
`reconcile.delete` / `provider.delete` on the way out).

It **skips unless both env vars are set** (so the rest of the suite runs
anywhere). Homelab invocation:

```bash
TEMPO_URL=http://192.168.0.30:3200 \
OTLP_ENDPOINT=http://192.168.0.30:4318 \
make test-e2e
```

Worth knowing:

- Every span of a run carries the resource attribute
  `test.run.id=e2e-<nanos>` (injected via `OTEL_RESOURCE_ATTRIBUTES`, no
  code involved); the run ID is printed in the test log, and
  `{resource.test.run.id="<id>"}` in Grafana shows exactly that run.
- The spec preflights the OTLP endpoint **from inside the cluster** with a
  curl pod and fails fast with a clear message if it's unreachable —
  export failures are otherwise only visible at `-zap-log-level=1`.
- Proxy CRs are created in `default`, not the operator namespace: the
  squid pods carry no securityContext and the operator namespace enforces
  restricted PSS.

## The kind verification run (the real end-to-end)

The README quickstart **is** the e2e test, run by hand before the MR:
deploy the operator in-cluster on a throwaway kind cluster, watch a real
Squid pod reach `Ready` via a real CONNECT probe, then exercise the whole
lease lifecycle through the discovery API (list → lease → report
`rate_limited` → same-target lease answered 409 `inCooldown:1` → release
204 twice) and delete the CR to watch the finalizer remove the pod.

Worth knowing about that run (full detail in
[plans-executions/2026-08-07-1747-proxy-operator.md](plans-executions/2026-08-07-1747-proxy-operator.md),
"Verification" section): it was not a clean pass-through. It caught two
real bugs the automated suites structurally could not —

1. `make run-dev` on a laptop can never produce a `Ready`
   kubernetes-provider proxy: health probes originate on the host, which
   cannot route to kind's pod network. The quickstart was rewritten to
   deploy in-cluster.
2. Squid was OOM-killed at startup: it sizes file-descriptor tables from
   `RLIMIT_NOFILE`, which containerd under kind sets effectively
   unlimited. Fixed with `max_filedescriptors 1024` in the generated
   config, plus a regression assertion in `pod_test.go`.

That is the pattern to keep: when the kind run finds something, the fix
lands **with a unit-level regression test**, so the manual run stays a
discovery tool rather than a recurring gate.