egress-proxies-operator/docs/testing.md

# Testing

What tests exist, how to run them, and where the boundaries of the
automated suites are. The philosophy throughout: **test against the most
real double available** — a real envtest API server over a fake client, a
real CONNECT-capable proxy stub over a mocked HTTP client, the real lease
store under the discovery handlers — and leave the gaps that only real
infrastructure can close to the kind verification run, documented at the
bottom.

## Running

```sh
make test              # THE canonical run: codegen + fmt + vet + envtest setup,
                       # then `go test -race` on every package, with coverage
go test -short ./...   # skip the envtest suite (runs in <1s per package)
go tool cover -html=cover.out   # browse coverage from the last make test
```

Targeted runs:

```sh
go test -race ./internal/lease/                  # one package
go test -race -run TestAcquire ./internal/lease/ # one test (or a prefix)
go test -race -count=2 ./internal/health/        # flake-shaking: run twice
```

**Gotcha — plain `go test ./...` fails the controller package** with an
error like `/usr/local/kubebuilder/bin/etcd: no such file`. That is not a
code problem: the envtest suite needs `KUBEBUILDER_ASSETS` pointing at the
API-server/etcd binaries, which only `make test` sets up (via
`setup-envtest`). Either use `make test`, or export it once:

```sh
KUBEBUILDER_ASSETS="$PWD/bin/k8s/<version>-<os>-<arch>" go test -race ./...
```

Everything runs with `-race` — `make test` enforces it; keep the flag on
manual runs too.

## The suites, package by package

| Package | Files | What is covered | Test double / technique |
|---|---|---|---|
| `api/v1alpha1` | `helpers_test.go` | `EffectivePort/EffectiveHost/HealthCheckOrDefault/MaxLeasesOrDefault` | pure table-driven; CEL rules are **not** unit-testable — see envtest below |
| `internal/controller` | `reconcile_test.go` | every row of the reconciler's action table: create, poll, publish IP, replace, adopt, NotFound recovery, delete/finalizer, quota/permanent/transient errors, cloud-init resolution | `controller-runtime/pkg/client/fake` + an in-test `stubProvider` |
| | `status_test.go` | `computePhase` truth table (10 rows) | pure |
| | `spechash_test.go` | hash stability, nil/empty normalization, sensitivity to every replacement-triggering field | pure |
| | `health_test.go` | Healthy-condition representation: Ready/Unhealthy phases, no-verdict, stale-verdict cleanup on replacement, nil snapshotter | fake client + `fakeSnapshotter` |
| | `proxy_controller_test.go` (envtest, ginkgo) | full lifecycles against a **real API server**: provision→Running, spec-change replacement, finalizer deletion, External tracking, Ready-through-health, quota-vs-permanent, adoption; **all CEL rules** (six invalid creates, mode/provider immutability incl. removal) and the `default={}` materialization | envtest apiserver + stub provider; the only place CEL and structural defaulting actually execute |
| `internal/provider` | `name_test.go` | deterministic naming: idempotency, `^proxy-[a-z2-7]{16}$`, 10k-UID distinctness | pure |
| | `errors_test.go` | taxonomy: `Class()` mapping, `errors.Is` **and** `errors.As` through the multi-unwrap | pure |
| | `config_test.go` | providers-config parsing + fail-fast validation | pure |
| | `metrics_test.go` | `WithMetrics` decorator: classified result labels, error passthrough | fake recorder + static provider |
| `internal/provider/kubernetes` | `kubernetes_test.go`, `pod_test.go` | Create/Get/Delete/ListByTag over real Pod objects; pure `buildPod`/`squidConf` incl. the `max_filedescriptors` OOM-regression assertion | `client/fake` (real `corev1.Pod`s, no kubelet) |
| | `check_test.go` | health check: a single cluster-wide list with `Limit=1` and the managed-label selector; the reason for every API error class | `client/fake` + interceptor |
| `internal/provider/gcp` | `insert_test.go`, `errors_test.go`, `gcp_test.go` | field-by-field `buildInsertRequest`; HTTP-code classification table; zone-qualified IDs, 409-is-success, 9-row state mapping, ListByTag filter + `ReturnPartialSuccess` | pure builder needs **no fake**; the rest uses the flattened `instancesAPI` seam |
| | `check_test.go` | health check: request shape and the reason per HTTP code or network error; the **real SDK client** makes exactly one request with `maxResults=1`, even when a next page exists | fake seam + `httptest` |
| `internal/provider/kvm` | `domainxml_test.go`, `volume_test.go`, `seed_test.go`, `id_test.go`, `metadata_test.go` | pure builders: domain/volume XML unmarshalled back and checked field by field, seed ISO reopened with the same library, providerID round-trip, metadata in both namespace spellings | pure |
| | `errors_test.go`, `conn_test.go` | libvirt-code classification table; connection pool (reuse, redial, drop on transport error, cancel releases a blocked RPC, one dial per host under concurrency); SSH dial, host-key policy, keepalive | fake dialer; a **real in-process `x/crypto/ssh` server** forwarding a unix socket |
| | `image_test.go`, `kvm_test.go` | image cache (fresh fetch, ready, marker missing, checksum change/mismatch, 4xx/5xx, concurrent callers); Create resume after every partial step, leftover truncated seed, state/IP mapping, every Delete already-gone case, ListByTag across hosts / one down / unreadable domain; logging tiers (V0 silent, V1 one line per RPC, V2 XML) with cloud-init never logged | `httptest` image server + in-memory `fakeAPI` behind the `libvirtAPI` seam |
| | `trace_test.go`, `conn_test.go`, `errors_test.go` | every `libvirtAPI` method traced exactly once (guarded by reflection on the interface), results and errors pass through, libvirt error fields, detail keys only at V2, upload content never logged; pool connect/drop lines, no wrapping below V1 | `fakeAPI` + `funcr` capture |
| | `check_test.go`, `conn_test.go` | health check: details, per-host verdicts in config order, hosts checked concurrently, a timeout that keeps the connection and refuses a second check while the first still waits, no waiting behind a reconcile's dial, the reason table; `probe` abandons without closing, but still drops on a transport error; waiting for a host's slot ends with the caller's deadline; an inactive or missing pool and daemon errors keep the connection; every dial stage tagged through the real SSH server, with classification unchanged | `fakeAPI`, fake dialers, in-process SSH server |
| | `kvm_integration_test.go` (`-tags=integration`) | one real VM: image fetch, Create, idempotent re-Create, Running with IP, ListByTag, Delete, pool state afterwards; `TestKVMCheck_realHost` runs the health check against the host and creates nothing | a **real hypervisor** — see below |
| `internal/provider/kvm2` | `xml_test.go`, `volume_test.go`, `seed_test.go`, `id_test.go`, `metadata_test.go`, `state_test.go` | pure builders: domain XML (file disks with resolved paths, UEFI with secure boot off, BIOS escape, the metadata element, the guest-agent channel), volume XML with and without a backing store, seed ISO reopened (files, label, user-data byte for byte, temp dir removed) and the vendor-data script vs. multipart rules, providerID round-trip and malformed forms, metadata in both namespace spellings, state mapping and the IPv4 filter | pure |
| | `errors_test.go`, `backend_test.go`, `hosts_test.go`, `ssh_test.go` | classification table (libvirt codes, SSH auth text, `knownhosts.KeyError`, missing files); the per-call budget adapter (deadline kills the connection and is Transient, cancel unblocks, killed by another caller, close idempotent) against a mocked libvirt socket; the host table (one dial per host under concurrency, redial after drop, `dialFresh` never cached, waiting ends with ctx); the real dialer end to end: socket forwarding, unknown and mismatched host keys, insecure mode, missing vs. unparsable key and known_hosts, wrong key, handshake timeout, cancel, every stage tagged, key material never in errors or logs | fake dialer; a **real in-process `x/crypto/ssh` server** forwarding a unix socket |
| | `image_test.go`, `create_test.go`, `get_test.go`, `delete_test.go`, `list_test.go`, `kvm2_test.go`, `provider_test.go` | image cache (fetch with the marker written last, cached, marker missing or without a digest, sha256 change or mismatch, 4xx/5xx, no Content-Length, short read, concurrent callers fetch once, cancel mid-stream, streamed without buffering, upload on a dedicated connection) and the pool-space quota precheck; Create resume after every partial step, returns only when running, adoption guard on UID, seed always rebuilt, cloud-init sentinel never logged; Get state and address mapping incl. agent-before-ARP; every Delete already-gone case, NVRAM flag, per-name lock; ListByTag across hosts, one down fails the whole call, unreadable domain skipped; `New` never dials and warns on missing files; every error wrapped with the configured name; logging tiers V0/V1/V2 | `httptest` image server + in-memory `fakeBackend` behind the `backend` seam |
| | `observe_test.go`, `check_test.go` | every seam method traced and logged exactly once (guarded by reflection), child spans under the parent, detail only at V2, upload bodies never logged, error fields; health check: one entry per host in config order, hosts checked concurrently, expired ctx returns promptly, never touches the pooled connection, the reason table, detail keys | `fakeBackend` + `funcr` capture + span recorder |
| | `kvm2_integration_test.go` (`-tags=integration`) | one real VM: `New` without a dial, `Check` Ready with details, Create, idempotent re-Create with an unchanged domain count, Running with IP and the metadata round-trip, ListByTag, Delete, NotFound, pool state afterwards, `kvm2.*` spans under `provider.create`; `TestKVM2Check_realHost` runs the health check against the host and creates nothing | a **real hypervisor** — see below |
| `internal/provider/nocloud` | `seed_test.go` | the NoCloud seed kvm2 and proxmox share: the ISO reopened (files, label, user-data byte for byte, temp dir removed, an empty vendor-data still a file, an empty name refused) and the vendor-data rules (a shell script, not cloud-config; multipart with configured extra vendor-data; the agent install switched off) | pure |
| `internal/provider/keyedlock` | `keyedlock_test.go` | the per-name lock kvm2 and proxmox share: the entry is removed once uncontended, waiters share one lock, a waiter gives up with its ctx | pure |
| `internal/provider/proxmox` | `id_test.go`, `mark_test.go`, `state_test.go`, `seed_test.go`, `errors_test.go`, `api_test.go` | pure pieces: providerID round-trip and malformed forms, the VMID derived from the UID (deterministic, in range, probing forward and wrapping), the description mark (keys, parsed from the captured clone, admin lines kept), state mapping and the IPv4 filter from captures, seed names and building (empty name Permanent without its payload, staging failure Transient); the `classify` table and the 500-message matchers against every captured error; the API types decoded from the captures | pure, over the responses Phase 0 captured from PVE 9.2.20 (`testdata/`) |
| | `client_test.go` | the thin HTTP client: the token read from its file on every request (missing Transient, unparsable Permanent), `apiError` from status, reason phrase and body, the data envelope, method, path and parameters of every seam call, the clone and destroy query shapes, the volid's slash escaped, HTTP/1.1 only, request timeout and cancel Transient and the client unaffected, a child span under a parent and none without, the token never in errors or logs, the CA file trusted and loaded lazily, an x509 failure Permanent, insecure mode, the upload streamed as multipart | `httptest.NewTLSServer` replaying the captures byte for byte |
| | `create_test.go`, `get_test.go`, `delete_test.go`, `list_test.go`, `task_test.go`, `proxmox_test.go`, `provider_test.go`, `fake_test.go` | Create: resume after every partial step and after a crash mid-clone, the VMID probe and ownership check, the template's shape refused before any clone, resize as a task, the seed overwritten without a delete, the net0 bridge and VLAN rewrite keeping the MAC, returns only when running, cloud-init sentinel never logged; Get: lock checked before name, the agent's messages, paused from `status/current`, hibernated, a reused VMID, four calls when running; Delete: every already-gone and not-ours case, the seed deleted before the destroy on the VM's node or on every node, a hibernated VM destroyed, the per-name lock; ListByTag: candidates by tag or name prefix, the mark decides, an unreadable VM skipped unless ctx ended; task and lock waits; `New` makes no request; the contract sweep: every method returns within a second of a cancel (blocked on its first seam call, and mid-operation), every error wrapped with the configured name, op and id, logging tiers V0/V1/V2 with the no-op lines | in-memory `fakeAPI` behind the `api` seam, its error texts checked against the captures |
| | `observe_test.go`, `check_test.go`, `provider_test.go` | the decorator: every seam method logs exactly one line (guarded by reflection), starts no span, identifiers at V1, detail at V2, upload body never logged, error fields; a Get through `provider.WithTracing` over the real client makes four `HTTP GET` spans, all children of `provider.get` and none carrying the token; health check: one entry per node in config order, nodes checked concurrently (a barrier, not a timer), expired ctx returns promptly, the reasons table at V2 with a sentinel secret, the transport reasons from real `net/http` and `crypto/tls` errors, detail keys | `fakeAPI` + `funcr` capture + span recorder; the real client over `httptest` for the tracing and transport-reason rows |
| | `proxmox_integration_test.go` (`-tags=integration`) | one real VM: `New` without a request, `Check` Ready with details, Create, idempotent re-Create with an unchanged VM count, Running with an IPv4 and the mark read back, ListByTag, Delete, NotFound, no VM and no seed left, HTTP spans under `provider.create`; `TestProxmoxCheck_realHost` runs the health check against the host and creates nothing. **Passed on pv1-1 (PVE 9.2.20), 2026-09-24**: Create 6.2 s, address 50 s later, whole run 61 s | a **real Proxmox VE host** — see below |
| `internal/health` | `probe_test.go` | probes through a **real CONNECT-capable proxy stub** to a real TLS server: tunnel success, refused CONNECT, wrong status, dead proxy, plain-http forward | `httptest` + hijacked bidirectional tunnel |
| | `engine_test.go` | thresholds, first-verdict, latency suppression matrix, dropped-event retry, stale-probe UID guard, tick scheduling/seeding/pruning, `Start` end-to-end | fake `client.Reader` + injected `probeFn` |
| `internal/lease` | `store_test.go` | capacity, `MaxLeases=0`, all three selection tie-breaks, target-scoped vs global cooldowns, expiry via fake clock, report-on-retained-lease, idempotent release, **40 concurrent acquires never exceeding capacity** | injectable clock; the concurrency case is why `-race` matters |
| `internal/discovery` | `server_test.go` | auth matrix (incl. `/healthz` exemption), list filtering, grant shape, full 409 arithmetic, invalid TTL/body/result, idempotent release, report→cooldown→409 round trip, `Start`/shutdown | `httptest` over the real handler chain, fake cache reader, **real** `lease.Store` |
| `internal/gc` | `gc_test.go` | the true-orphan matrix (live/deleting/young/unlabelled all kept), broken-provider isolation, list-failure skips sweep, namespace guard, sweep loop | fake reader + canned-list provider |
| `internal/providerhealth` | `status_test.go`, `monitor_test.go` | `buildStatus` (the Ready condition, per-target transition times, truncation); objects created with the label and patched, the round timeout, no-checker Unknown, invalid names, transition-only logging, cleanup of labelled unconfigured objects, the checks-off path, `ValidateSchedule` | fake client with a cluster-scoped RESTMapper + `funcr` capture |
| | `envtest_test.go` | the generated `ProviderHealth` schema against a real API server: cluster scope, status subresource, required fields, list-map keys, the 1024-character message limit | envtest (skipped under `-short`) |
| `internal/metrics` | `metrics_test.go` | scrape-time collectors (`GatherAndCompare`) incl. the provider-target gauges, per-proxy series cleanup via `ForgetProxy`, label counts, fresh-registry-per-test property | `prometheus/client_golang/testutil` |

Conventions (from `CLAUDE.md`): stdlib `testing`, table-driven by default,
`t.Parallel()` where safe, tests next to source, ginkgo only in the
envtest suite the scaffold generated.

## What the automated suites deliberately do NOT cover

- **`New()` constructors that dial real infrastructure**: the kubernetes
  provider's `New` (connects to whatever your kubeconfig points at), the
  GCP provider's `New` and its `realInstances` SDK adapter (ADC + real
  Google endpoints). Both are thin; both are exercised by the kind run /
  real deployments. Their 0% coverage is by design — do not "fix" it.
- **CEL and structural defaulting under the fake client**: the fake
  client runs neither. Every unit test that relies on defaults calls the
  `*OrDefault` helpers; every CEL rule is asserted in the envtest suite
  instead.
- **A container actually starting and serving**: envtest has no kubelet,
  so a Pod created there sits Pending forever. Whether Squid really comes
  up and tunnels CONNECT is provable only on a real cluster — that is the
  kind verification's job, and it is exactly what caught the Squid OOM
  bug (below).
- **Live GCP**: no test talks to Google. The seam boundary
  (`buildInsertRequest` + classification) is tested exhaustively instead.
- **Live libvirt, by default**: the KVM and kvm2 unit tests run over an
  in-memory fake of the 20 libvirt calls each provider uses. Whether
  libvirt accepts the domain XML, whether a raw qcow2 upload plus pool
  refresh yields a usable backing image, and whether the guest actually
  boots and reports an address are provable only on a real host — that is
  the opt-in integration test below (one per provider).

- **Live Proxmox, by default**: the proxmox unit tests run over an
  in-memory fake of the 16 seam calls, and the HTTP client over
  `httptest` replaying responses captured from PVE 9.2.20. Whether a
  clone, config write, seed upload and start yield a guest that boots and
  reports an address is provable only on a real host (the proxmox
  integration test below, which has passed on pv1-1). Two kinds of path
  have met only the fake:
  - **the cross-node paths**: a clone that targets another node, the seed
    deleted on every node when the VM's node is unknown, a Check round over
    several nodes. The test host is a single node, so the integration test
    cannot reach them either
    ([proxmox-provider-spec.md](proxmox-provider-spec.md) entry 24);
  - **an answer never captured**: the storage status for a storage the
    cluster knows but cannot use on a node (disabled there, or offline).
    Every 500 there is taken as `StorageNotFound`, and what the fake
    answers is its own guess. The three answers Phase 0 missed were
    captured on pv1-1 on 2026-09-24: a missing storage is a 400 with
    `storage: No such storage.`, a hibernated VM is destroyed without
    complaint (its suspend state volume with it), and a stop on a paused
    VM ends `OK` with the VM stopped.

## The KVM integration test (real hypervisor)

`internal/provider/kvm/kvm_integration_test.go` compiles only under
`-tags=integration` and skips itself unless the environment names a
host. It deviates from the "testcontainers for real infra" convention on
purpose: libvirt needs `/dev/kvm` and a bridge to a DHCP network, and a
container has neither. So it targets whichever hypervisor you point it
at and leaves that host's pool with one more cached base image than it
had.

```sh
export KVM_TEST_HOST=nas1.home.lan:22          # SSH endpoint
export KVM_TEST_USER=virt                      # must be in the libvirt group
export KVM_TEST_SSH_KEY=$HOME/.ssh/id_ed25519  # private key for that user
export KVM_TEST_KNOWN_HOSTS=$HOME/.ssh/kvm_known_hosts  # the hypervisors' host keys
export KVM_TEST_BRIDGE=br0                     # bridge to a network with DHCP
export KVM_TEST_IMAGE_URL=https://cloud.debian.org/images/cloud/trixie/latest/debian-13-generic-amd64.qcow2
make test-integration-kvm
```

`KVM_TEST_KNOWN_HOSTS` is technically optional, but leaving it out turns
host-key verification **off** for the run (the log says so); keep a
`kvm_known_hosts` holding just the hypervisors rather than pointing the
test at your everyday `known_hosts`, so a re-keyed hypervisor is a
one-line fix in a file nothing else uses.

Also optional: `KVM_TEST_IMAGE_SHA256`, `KVM_TEST_POOL`
(default `default`), `KVM_TEST_SOCKET`, `KVM_TEST_TIMEOUT` (default
`20m`, must cover the first-time image download), `KVM_TEST_KEEP`
to leave the VM behind for `virsh console` when something goes wrong,
and `KVM_TEST_LOG_LEVEL` (default `1`) for how much of the provider's
logging the `-v` output shows: at `1` every libvirt RPC is one compact
line (`HH:MM:SS.mmm libvirt virDomainLookupByName domain=... duration=...
error=... libvirtError=ErrNoDomain`), interleaved with the test's own
`t.Log` lines; `2` adds the domain and volume XML, metadata and
addresses; `0` leaves only `t.Log`. The per-run constants (provider,
host, domain name) are left off each line. See the Logging section of
[kvm-provider.md](kvm-provider.md#logging).

What one run asserts, in order: `New` dials the host; `Create` returns a
`hosts/<host>/domains/<name>` ID (downloading the image on a cold host —
the log prints how long it took); a second `Create` returns the same ID;
`Get` reaches `Running` with an IPv4 within the budget, and the UID and
`CreatedAt` survive the metadata round-trip through libvirt; `ListByTag`
lists it; `Delete` removes it, `Get` says `ErrNotFound`, a second
`Delete` is a no-op, and the pool holds the base image and its marker
but no per-VM overlay or seed. It also tries a TCP connect to port 22 on
the guest, but only logs the outcome — the machine running the test may
not route to the bridge network.

`TestKVMCheck_realHost` runs the provider health check
([kvm-provider.md](kvm-provider.md#health-check)) against the same host.
It must find the host reachable and the pool running, and it creates
nothing. It needs the same variables; to run only this test:

```sh
go test -tags=integration -count=1 -v -run TestKVMCheck ./internal/provider/kvm/
```

The test is not part of `make test` and not run in CI. Keep it compiling
with `go vet -tags=integration ./...` when the provider's internals move.

## The kvm2 integration test (real hypervisor)

`internal/provider/kvm2/kvm2_integration_test.go` compiles only under
`-tags=integration` and skips itself unless the environment names a
host. Like the kvm one it deviates from the "testcontainers for real
infra" convention on purpose: libvirt needs `/dev/kvm` and a bridge to a
DHCP network, and a container has neither. It targets whichever
hypervisor you point it at and leaves that host's pool with one more
cached base image (`base-test-image.qcow2` and its `.sha256` marker) than
it had. The kvm and kvm2 tests can share a host: kvm2 marks its domains
in its own metadata namespace and never lists or adopts the other's.

```sh
export KVM2_TEST_HOST=beelink:22               # SSH endpoint
export KVM2_TEST_USER=novakj                   # must be in the libvirt group
export KVM2_TEST_SSH_KEY=$HOME/.ssh/id_ed25519 # private key for that user
export KVM2_TEST_KNOWN_HOSTS=$HOME/.ssh/kvm_known_hosts  # the hypervisors' host keys
export KVM2_TEST_BRIDGE=br0                    # bridge to a network with DHCP
export KVM2_TEST_IMAGE_URL=https://cloud.debian.org/images/cloud/trixie/latest/debian-13-generic-amd64.qcow2
make test-integration-kvm2
```

`KVM2_TEST_KNOWN_HOSTS` is technically optional, but leaving it out turns
host-key verification **off** for the run (the log says so); keep a
`kvm_known_hosts` holding just the hypervisors rather than pointing the
test at your everyday `known_hosts`, so a re-keyed hypervisor is a
one-line fix in a file nothing else uses.

Also optional: `KVM2_TEST_IMAGE_SHA256` (verified after download and
compared with the cached marker), `KVM2_TEST_POOL` (default `default`),
`KVM2_TEST_SOCKET` (default `/var/run/libvirt/libvirt-sock`),
`KVM2_TEST_TIMEOUT` (default `20m`, the budget for the whole lifecycle;
it must cover the first-time image download), `KVM2_TEST_KEEP` to leave
the VM behind for `virsh console` when something goes wrong (the log then
prints the `virsh undefine --nvram` and `virsh vol-delete` lines that
clean it up), and `KVM2_TEST_LOG_LEVEL` (default `1`) for how much of the
provider's logging the `-v` output shows: at `1` every libvirt RPC is one
compact `libvirt call` line (`HH:MM:SS.mmm kvm2 "level"=1 "msg"="libvirt
call" "rpc"="DomainLookupByName" "domain"=... "duration"=...`, plus the
per-operation outcome lines and connection events), interleaved with the
test's own `t.Log` lines; `2` adds the domain and volume XML, metadata
and addresses; `0` leaves only `t.Log`. The per-run constants (provider,
host, domain name) are stated once in the first line and left off every
provider line. See the Logging section of
[kvm2-provider.md](kvm2-provider.md).

What one run asserts, in order, each step logged with its duration:

1. `New` returns within 100 ms and without dialling (no `libvirt host
   connected` line before the first call).
2. `Check` reports the one host `Ready` with reason `Reachable` and the
   `hostname`, `libvirtVersion`, `poolState` (`running`) and
   `poolAvailableGiB` details, which are logged.
3. Every later call goes through `provider.WithTracing` with an in-memory
   span recorder that the package's own spans also land in.
4. `Create` returns `hosts/kvm2-test/domains/<name>` (downloading the
   image on a cold pool — the log prints how long it took).
5. A second `Create` returns the same ID and the host's domain count,
   read over a dedicated connection, is unchanged.
6. `Get`, polled every 5 s, reaches `Running` with an IPv4 within the
   budget; its `UID` is the one Create wrote and `CreatedAt` is non-zero
   and within a few minutes of now. The time from Create's return to the
   first address is logged.
7. `ListByTag` contains the ID with that UID.
8. A TCP connect to port 22 on the guest, logged only — the machine
   running the test may not route to the bridge network.
9. `Delete` returns nil, `Get` is `ErrNotFound`, a second `Delete` is a
   no-op.
10. The pool, inspected over a dedicated connection, holds the base image
    and its marker but no `<name>.qcow2` or `<name>-seed.iso`.
11. Among the recorded spans, a `provider.create` span has at least one
    `kvm2.DomainDefineXML` and one `kvm2.dial` child, and no span
    attribute carries the cloud-init text.

With `KVM2_TEST_KEEP` steps 9 and 10 are skipped; otherwise a `t.Cleanup`
attempts the Delete even when an earlier assertion failed.

`TestKVM2Check_realHost` runs the provider health check
([kvm2-provider.md](kvm2-provider.md)) against the same host. It must
find the host reachable and the pool running, and it creates nothing. It
needs the same variables; to run only this test:

```sh
go test -tags=integration -count=1 -v -run TestKVM2Check ./internal/provider/kvm2/
```

The test is not part of `make test` and not run in CI. Keep it compiling
with `go vet -tags=integration ./...` when the provider's internals move.

## The proxmox integration test (real Proxmox VE host)

`internal/provider/proxmox/proxmox_integration_test.go` compiles only
under `-tags=integration` and skips itself unless the environment names a
host. It deviates from the testcontainers convention for the same reason
as the kvm tests: Proxmox VE is a hypervisor, not something a container
can stand in for. It needs a node with a cloud-init template without a
Proxmox cloud-init drive and with the guest agent enabled, and a token
with the privileges of [proxmox-provider-spec.md](proxmox-provider-spec.md)
entry 14: the role on `/` from the plan's "Before Phase 0" section
([2026-09-18-1502-proxmox-provider.md](plans/2026-09-18-1502-proxmox-provider.md)),
plus `Datastore.Allocate` on the ISO storage. A passing run leaves
nothing behind: its VM and its seed ISO are gone.

```sh
export PROXMOX_TEST_URL=https://192.168.0.8:8006           # host must be in the node certificate
export PROXMOX_TEST_TOKEN_FILE=$HOME/.config/proxmox-token  # one line: user@realm!tokenid=secret
export PROXMOX_TEST_CA_FILE=$HOME/.config/proxmox-ca.pem    # the node's pve-root-ca.pem
export PROXMOX_TEST_NODE=pv1-1
export PROXMOX_TEST_TEMPLATE_VMID=9001
export PROXMOX_TEST_ISO_STORAGE=proxy-seeds                 # holds iso content
export PROXMOX_TEST_VLAN_TAG=4                              # template 9001's net0 is untagged; proxies live on VLAN 4
make test-integration-proxmox
```

Required: `PROXMOX_TEST_URL`, `PROXMOX_TEST_TOKEN_FILE`,
`PROXMOX_TEST_NODE`, `PROXMOX_TEST_TEMPLATE_VMID` and
`PROXMOX_TEST_ISO_STORAGE`; the test skips when any is missing. Optional:
`PROXMOX_TEST_CA_FILE` (empty trusts the system roots),
`PROXMOX_TEST_INSECURE` (`true` skips certificate verification; the log
says so), `PROXMOX_TEST_STORAGE` (the clone's target storage; empty keeps
the template's), `PROXMOX_TEST_BRIDGE` and `PROXMOX_TEST_VLAN_TAG` (the
cloned `net0`'s bridge and tag; empty keeps the template's),
`PROXMOX_TEST_VMID_MIN` and `PROXMOX_TEST_VMID_MAX` (default
`100000`..`199999`), `PROXMOX_TEST_TIMEOUT` (default `20m`, the budget for
the whole lifecycle), `PROXMOX_TEST_KEEP` to leave the VM behind, and
`PROXMOX_TEST_LOG_LEVEL` (default `1`): at `1` every request is one
`proxmox api call` line with the call, node, VMID, UPID and duration,
plus the per-operation outcome lines; `2` adds request parameters, config
summaries and agent addresses; `0` leaves only `t.Log`. Provider, node,
name and VMID are stated once and left off every provider line. Without
`Datastore.Allocate` on the ISO storage the seed delete at the end of
Delete is a 403, and Delete fails as Permanent.

What one run asserts, in order, each step logged with its duration:

1. `New` returns within 100 ms and makes no request (no `proxmox api
   call` line before the first operation).
2. `Check` reports the node `Ready` with reason `Reachable` and the
   `pveVersion`, `kernel`, `tokenID`, `isoStorageAvailGiB` and
   `templates` (`1/1`) details, which are logged.
3. Every later call goes through `provider.WithTracing`, with an
   in-memory span recorder on the transport as well.
4. `Create` returns `vms/<vmid>/<name>` with the VMID in range. The
   machine type is 1 core, 1024 MiB and a 4 GiB disk, so a 3 GiB template
   is grown by the resize task.
5. A second `Create` returns the same ID and the cluster's QEMU VM count
   is unchanged.
6. `Get`, polled every 5 s, reaches `Running` with an IPv4 within the
   budget (a Transient answer while the guest boots is logged and polled
   through). Its `UID` and `CreatedAt` equal the mark in the VM's
   description, read back from its config, and `CreatedAt` is within a
   few minutes of now. The time from Create's return to the first address
   is logged.
7. `ListByTag` contains the ID with that UID and CreatedAt.
8. A TCP connect to port 22 on the guest, logged only: the machine
   running the test may not route to the VM's network.
9. `Delete` returns nil, `Get` is `ErrNotFound`, a second `Delete` is a
   no-op.
10. No VM with the name is left in the cluster's resources, and no
    `<isoStorage>:iso/<name>-seed.iso` in the storage's iso content. The
    content is read with one extra read-only GET of
    `/nodes/{node}/storage/{storage}/content?content=iso` made by the test
    itself (`storage_content_iso.json` is its captured shape): the seam
    has no method to list storage content, and does not get one for a
    test.
11. Among the recorded spans, a `provider.create` span has at least one
    `HTTP` child, and no span attribute carries the cloud-init text.

With `PROXMOX_TEST_KEEP` steps 9 and 10 are skipped, and at the end the
log prints what to run as root on the node to clean up:
`qm stop <vmid>`, `qm destroy <vmid> --purge` and
`pvesm free <isoStorage>:iso/<name>-seed.iso`. Otherwise a `t.Cleanup`
runs the Delete even when an earlier assertion failed. It finds the VM by
name, so a Create that failed halfway is cleaned up too.

Cost: one full clone of the template (about 2 s for the 3 GiB template on
pv1-1), one boot to an address (about 50 s), and a few seconds of
destroy; a run takes a couple of minutes. The Makefile target's 30 m
timeout covers the 20 m budget plus the cleanup.

`TestProxmoxCheck_realHost` runs the provider health check against the
same host. It must find the node reachable, the ISO storage usable and
the template present, and it creates nothing. It needs the same
variables; to run only this test:

```sh
go test -tags=integration -count=1 -v -run TestProxmoxCheck ./internal/provider/proxmox/
```

**Last run: 2026-09-24 on pv1-1** (Proxmox VE 9.2.20, kernel
7.0.14-17-pve, template 9001, `proxy-seeds`, VLAN tag 4). Both tests
passed; `go test` reported 60.9 s for the package.

| Step | Time |
|---|---|
| `Check` | 141 ms (125 ms in `TestProxmoxCheck_realHost`) |
| clone task, submit to done | about 3 s (seen done at the third 1 s poll) |
| `Create`, clone to started VM | 6.2 s |
| second `Create` (adoption, nothing created) | 32 ms |
| Create's return to the first address | 50.2 s (the agent install from the internet) |
| `Delete` (stop, seed delete, destroy), `Get`, second `Delete` | 3.1 s |
| `TestProxmox_lifecycle` | 59.7 s |

The first `Get`, 32 ms after the second Create, was Transient: the
listing still showed the new VM with `status: unknown`. The next poll 5 s
later saw it running. The Delete deleted the seed while the stopped VM's
`ide2` still referenced it, then destroyed the VM, and step 10 found
neither left. Like the kvm tests they are not part of `make test` and not
run in CI.

## The scaffolded `make test-e2e` suite

`test/e2e/` is the kubebuilder-generated smoke suite (build image → kind
cluster → deploy → assert the manager pod runs, publishes a Ready
`ProviderHealth` for the `kubernetes` provider, and serves metrics that
include that provider's `proxy_operator_provider_target_up` series). It
compiles only under `-tags=e2e`, manages its own kind cluster
(`make test-e2e` / `make cleanup-test-e2e`), and has been kept compiling
(`go vet -tags=e2e ./...` is part of the routine) but is **not part of
`make test` and was not used for the release verification** — the manual
kind run below covers strictly more.

### The OTel tracing spec (Tempo-gated)

`test/e2e/tracing_test.go` proves the full tracing pipeline against a
**real Tempo**: deploy in kind with tracing enabled, create two
kubernetes-provider proxies, delete them, and assert in Tempo that the
traces exist with the documented span topology (`Reconcile Proxy` →
`reconcile.managed` / `provider.create` / `status.patch`, and
`reconcile.delete` / `provider.delete` on the way out).

It **skips unless both env vars are set** (so the rest of the suite runs
anywhere). Homelab invocation:

```bash
TEMPO_URL=http://192.168.0.30:3200 \
OTLP_ENDPOINT=http://192.168.0.30:4318 \
make test-e2e
```

Worth knowing:

- Every span of a run carries the resource attribute
  `test.run.id=e2e-<nanos>` (injected via `OTEL_RESOURCE_ATTRIBUTES`, no
  code involved); the run ID is printed in the test log, and
  `{resource.test.run.id="<id>"}` in Grafana shows exactly that run.
- The spec preflights the OTLP endpoint **from inside the cluster** with a
  curl pod and fails fast with a clear message if it's unreachable —
  export failures are otherwise only visible at `-zap-log-level=1`.
- Proxy CRs are created in `default`, not the operator namespace: the
  squid pods carry no securityContext and the operator namespace enforces
  restricted PSS.

## The kind verification run (the real end-to-end)

The README quickstart **is** the e2e test, run by hand before the MR:
deploy the operator in-cluster on a throwaway kind cluster, watch a real
Squid pod reach `Ready` via a real CONNECT probe, then exercise the whole
lease lifecycle through the discovery API (list → lease → report
`rate_limited` → same-target lease answered 409 `inCooldown:1` → release
204 twice) and delete the CR to watch the finalizer remove the pod.

Worth knowing about that run (full detail in
[plans-executions/2026-08-07-1747-proxy-operator.md](plans-executions/2026-08-07-1747-proxy-operator.md),
"Verification" section): it was not a clean pass-through. It caught two
real bugs the automated suites structurally could not —

1. `make run-dev` on a laptop can never produce a `Ready`
   kubernetes-provider proxy: health probes originate on the host, which
   cannot route to kind's pod network. The quickstart was rewritten to
   deploy in-cluster.
2. Squid was OOM-killed at startup: it sizes file-descriptor tables from
   `RLIMIT_NOFILE`, which containerd under kind sets effectively
   unlimited. Fixed with `max_filedescriptors 1024` in the generated
   config, plus a regression assertion in `pod_test.go`.

That is the pattern to keep: when the kind run finds something, the fix
lands **with a unit-level regression test**, so the manual run stays a
discovery tool rather than a recurring gate.