egress-proxies-operator/docs/testing.md

Testing

What tests exist, how to run them, and where the boundaries of the automated suites are. The philosophy throughout: test against the most real double available — a real envtest API server over a fake client, a real CONNECT-capable proxy stub over a mocked HTTP client, the real lease store under the discovery handlers — and leave the gaps that only real infrastructure can close to the kind verification run, documented at the bottom.

Running

make test              # THE canonical run: codegen + fmt + vet + envtest setup,
                       # then `go test -race` on every package, with coverage
go test -short ./...   # skip the envtest suite (runs in <1s per package)
go tool cover -html=cover.out   # browse coverage from the last make test

Targeted runs:

go test -race ./internal/lease/                  # one package
go test -race -run TestAcquire ./internal/lease/ # one test (or a prefix)
go test -race -count=2 ./internal/health/        # flake-shaking: run twice

Gotcha — plain go test ./... fails the controller package with an error like /usr/local/kubebuilder/bin/etcd: no such file. That is not a code problem: the envtest suite needs KUBEBUILDER_ASSETS pointing at the API-server/etcd binaries, which only make test sets up (via setup-envtest). Either use make test, or export it once:

KUBEBUILDER_ASSETS="$PWD/bin/k8s/<version>-<os>-<arch>" go test -race ./...

Everything runs with -race — make test enforces it; keep the flag on manual runs too.

The suites, package by package

Package	Files	What is covered	Test double / technique
api/v1alpha1	helpers_test.go	EffectivePort/EffectiveHost/HealthCheckOrDefault/MaxLeasesOrDefault	pure table-driven; CEL rules are not unit-testable — see envtest below
internal/controller	reconcile_test.go	every row of the reconciler's action table: create, poll, publish IP, replace, adopt, NotFound recovery, delete/finalizer, quota/permanent/transient errors, cloud-init resolution	controller-runtime/pkg/client/fake + an in-test stubProvider
	status_test.go	computePhase truth table (10 rows)	pure
	spechash_test.go	hash stability, nil/empty normalization, sensitivity to every replacement-triggering field	pure
	health_test.go	Healthy-condition representation: Ready/Unhealthy phases, no-verdict, stale-verdict cleanup on replacement, nil snapshotter	fake client + fakeSnapshotter
	proxy_controller_test.go (envtest, ginkgo)	full lifecycles against a real API server: provision→Running, spec-change replacement, finalizer deletion, External tracking, Ready-through-health, quota-vs-permanent, adoption; all CEL rules (six invalid creates, mode/provider immutability incl. removal) and the default={} materialization	envtest apiserver + stub provider; the only place CEL and structural defaulting actually execute
internal/provider	name_test.go	deterministic naming: idempotency, ^proxy-[a-z2-7]{16}$, 10k-UID distinctness	pure
	errors_test.go	taxonomy: Class() mapping, errors.Is and errors.As through the multi-unwrap	pure
	config_test.go	providers-config parsing + fail-fast validation	pure
	metrics_test.go	WithMetrics decorator: classified result labels, error passthrough	fake recorder + static provider
internal/provider/kubernetes	kubernetes_test.go, pod_test.go	Create/Get/Delete/ListByTag over real Pod objects; pure buildPod/squidConf incl. the max_filedescriptors OOM-regression assertion	client/fake (real corev1.Pods, no kubelet)
	check_test.go	health check: a single cluster-wide list with Limit=1 and the managed-label selector; the reason for every API error class	client/fake + interceptor
internal/provider/gcp	insert_test.go, errors_test.go, gcp_test.go	field-by-field buildInsertRequest; HTTP-code classification table; zone-qualified IDs, 409-is-success, 9-row state mapping, ListByTag filter + ReturnPartialSuccess	pure builder needs no fake; the rest uses the flattened instancesAPI seam
	check_test.go	health check: request shape and the reason per HTTP code or network error; the real SDK client makes exactly one request with maxResults=1, even when a next page exists	fake seam + httptest
internal/provider/kvm	domainxml_test.go, volume_test.go, seed_test.go, id_test.go, metadata_test.go	pure builders: domain/volume XML unmarshalled back and checked field by field, seed ISO reopened with the same library, providerID round-trip, metadata in both namespace spellings	pure
	errors_test.go, conn_test.go	libvirt-code classification table; connection pool (reuse, redial, drop on transport error, cancel releases a blocked RPC, one dial per host under concurrency); SSH dial, host-key policy, keepalive	fake dialer; a real in-process x/crypto/ssh server forwarding a unix socket
	image_test.go, kvm_test.go	image cache (fresh fetch, ready, marker missing, checksum change/mismatch, 4xx/5xx, concurrent callers); Create resume after every partial step, leftover truncated seed, state/IP mapping, every Delete already-gone case, ListByTag across hosts / one down / unreadable domain; logging tiers (V0 silent, V1 one line per RPC, V2 XML) with cloud-init never logged	httptest image server + in-memory fakeAPI behind the libvirtAPI seam
	trace_test.go, conn_test.go, errors_test.go	every libvirtAPI method traced exactly once (guarded by reflection on the interface), results and errors pass through, libvirt error fields, detail keys only at V2, upload content never logged; pool connect/drop lines, no wrapping below V1	fakeAPI + funcr capture
	check_test.go, conn_test.go	health check: details, per-host verdicts in config order, hosts checked concurrently, a timeout that keeps the connection and refuses a second check while the first still waits, no waiting behind a reconcile's dial, the reason table; probe abandons without closing, but still drops on a transport error; waiting for a host's slot ends with the caller's deadline; an inactive or missing pool and daemon errors keep the connection; every dial stage tagged through the real SSH server, with classification unchanged	fakeAPI, fake dialers, in-process SSH server
	kvm_integration_test.go (-tags=integration)	one real VM: image fetch, Create, idempotent re-Create, Running with IP, ListByTag, Delete, pool state afterwards; TestKVMCheck_realHost runs the health check against the host and creates nothing	a real hypervisor — see below
internal/provider/kvm2	xml_test.go, volume_test.go, seed_test.go, id_test.go, metadata_test.go, state_test.go	pure builders: domain XML (file disks with resolved paths, UEFI with secure boot off, BIOS escape, the metadata element, the guest-agent channel), volume XML with and without a backing store, seed ISO reopened (files, label, user-data byte for byte, temp dir removed) and the vendor-data script vs. multipart rules, providerID round-trip and malformed forms, metadata in both namespace spellings, state mapping and the IPv4 filter	pure
	errors_test.go, backend_test.go, hosts_test.go, ssh_test.go	classification table (libvirt codes, SSH auth text, knownhosts.KeyError, missing files); the per-call budget adapter (deadline kills the connection and is Transient, cancel unblocks, killed by another caller, close idempotent) against a mocked libvirt socket; the host table (one dial per host under concurrency, redial after drop, dialFresh never cached, waiting ends with ctx); the real dialer end to end: socket forwarding, unknown and mismatched host keys, insecure mode, missing vs. unparsable key and known_hosts, wrong key, handshake timeout, cancel, every stage tagged, key material never in errors or logs	fake dialer; a real in-process x/crypto/ssh server forwarding a unix socket
	image_test.go, create_test.go, get_test.go, delete_test.go, list_test.go, kvm2_test.go, provider_test.go	image cache (fetch with the marker written last, cached, marker missing or without a digest, sha256 change or mismatch, 4xx/5xx, no Content-Length, short read, concurrent callers fetch once, cancel mid-stream, streamed without buffering, upload on a dedicated connection) and the pool-space quota precheck; Create resume after every partial step, returns only when running, adoption guard on UID, seed always rebuilt, cloud-init sentinel never logged; Get state and address mapping incl. agent-before-ARP; every Delete already-gone case, NVRAM flag, per-name lock; ListByTag across hosts, one down fails the whole call, unreadable domain skipped; New never dials and warns on missing files; every error wrapped with the configured name; logging tiers V0/V1/V2	httptest image server + in-memory fakeBackend behind the backend seam
	observe_test.go, check_test.go	every seam method traced and logged exactly once (guarded by reflection), child spans under the parent, detail only at V2, upload bodies never logged, error fields; health check: one entry per host in config order, hosts checked concurrently, expired ctx returns promptly, never touches the pooled connection, the reason table, detail keys	fakeBackend + funcr capture + span recorder
	kvm2_integration_test.go (-tags=integration)	one real VM: New without a dial, Check Ready with details, Create, idempotent re-Create with an unchanged domain count, Running with IP and the metadata round-trip, ListByTag, Delete, NotFound, pool state afterwards, kvm2.* spans under provider.create; TestKVM2Check_realHost runs the health check against the host and creates nothing	a real hypervisor — see below
internal/provider/nocloud	seed_test.go	the NoCloud seed kvm2 and proxmox share: the ISO reopened (files, label, user-data byte for byte, temp dir removed, an empty vendor-data still a file, an empty name refused) and the vendor-data rules (a shell script, not cloud-config; multipart with configured extra vendor-data; the agent install switched off)	pure
internal/provider/keyedlock	keyedlock_test.go	the per-name lock kvm2 and proxmox share: the entry is removed once uncontended, waiters share one lock, a waiter gives up with its ctx	pure
internal/provider/proxmox	id_test.go, mark_test.go, state_test.go, seed_test.go, errors_test.go, api_test.go	pure pieces: providerID round-trip and malformed forms, the VMID derived from the UID (deterministic, in range, probing forward and wrapping), the description mark (keys, parsed from the captured clone, admin lines kept), state mapping and the IPv4 filter from captures, seed names and building (empty name Permanent without its payload, staging failure Transient); the classify table and the 500-message matchers against every captured error; the API types decoded from the captures	pure, over the responses Phase 0 captured from PVE 9.2.20 (testdata/)
	client_test.go	the thin HTTP client: the token read from its file on every request (missing Transient, unparsable Permanent), apiError from status, reason phrase and body, the data envelope, method, path and parameters of every seam call, the clone and destroy query shapes, the volid's slash escaped, HTTP/1.1 only, request timeout and cancel Transient and the client unaffected, a child span under a parent and none without, the token never in errors or logs, the CA file trusted and loaded lazily, an x509 failure Permanent, insecure mode, the upload streamed as multipart	httptest.NewTLSServer replaying the captures byte for byte
	create_test.go, get_test.go, delete_test.go, list_test.go, task_test.go, proxmox_test.go, provider_test.go, fake_test.go	Create: resume after every partial step and after a crash mid-clone, the VMID probe and ownership check, the template's shape refused before any clone, resize as a task, the seed overwritten without a delete, the net0 bridge and VLAN rewrite keeping the MAC, returns only when running, cloud-init sentinel never logged; Get: lock checked before name, the agent's messages, paused from status/current, hibernated, a reused VMID, four calls when running; Delete: every already-gone and not-ours case, the seed deleted before the destroy on the VM's node or on every node, a hibernated VM destroyed, the per-name lock; ListByTag: candidates by tag or name prefix, the mark decides, an unreadable VM skipped unless ctx ended; task and lock waits; New makes no request; the contract sweep: every method returns within a second of a cancel (blocked on its first seam call, and mid-operation), every error wrapped with the configured name, op and id, logging tiers V0/V1/V2 with the no-op lines	in-memory fakeAPI behind the api seam, its error texts checked against the captures
	observe_test.go, check_test.go, provider_test.go	the decorator: every seam method logs exactly one line (guarded by reflection), starts no span, identifiers at V1, detail at V2, upload body never logged, error fields; a Get through provider.WithTracing over the real client makes four HTTP GET spans, all children of provider.get and none carrying the token; health check: one entry per node in config order, nodes checked concurrently (a barrier, not a timer), expired ctx returns promptly, the reasons table at V2 with a sentinel secret, the transport reasons from real net/http and crypto/tls errors, detail keys	fakeAPI + funcr capture + span recorder; the real client over httptest for the tracing and transport-reason rows
	proxmox_integration_test.go (-tags=integration)	one real VM: New without a request, Check Ready with details, Create, idempotent re-Create with an unchanged VM count, Running with an IPv4 and the mark read back, ListByTag, Delete, NotFound, no VM and no seed left, HTTP spans under provider.create; TestProxmoxCheck_realHost runs the health check against the host and creates nothing. Passed on pv1-1 (PVE 9.2.20), 2026-09-24: Create 6.2 s, address 50 s later, whole run 61 s	a real Proxmox VE host — see below
internal/health	probe_test.go	probes through a real CONNECT-capable proxy stub to a real TLS server: tunnel success, refused CONNECT, wrong status, dead proxy, plain-http forward	httptest + hijacked bidirectional tunnel
	engine_test.go	thresholds, first-verdict, latency suppression matrix, dropped-event retry, stale-probe UID guard, tick scheduling/seeding/pruning, Start end-to-end	fake client.Reader + injected probeFn
internal/lease	store_test.go	capacity, MaxLeases=0, all three selection tie-breaks, target-scoped vs global cooldowns, expiry via fake clock, report-on-retained-lease, idempotent release, 40 concurrent acquires never exceeding capacity	injectable clock; the concurrency case is why -race matters
internal/discovery	server_test.go	auth matrix (incl. /healthz exemption), list filtering, grant shape, full 409 arithmetic, invalid TTL/body/result, idempotent release, report→cooldown→409 round trip, Start/shutdown	httptest over the real handler chain, fake cache reader, real lease.Store
internal/gc	gc_test.go	the true-orphan matrix (live/deleting/young/unlabelled all kept), broken-provider isolation, list-failure skips sweep, namespace guard, sweep loop	fake reader + canned-list provider
internal/providerhealth	status_test.go, monitor_test.go	buildStatus (the Ready condition, per-target transition times, truncation); objects created with the label and patched, the round timeout, no-checker Unknown, invalid names, transition-only logging, cleanup of labelled unconfigured objects, the checks-off path, ValidateSchedule	fake client with a cluster-scoped RESTMapper + funcr capture
	envtest_test.go	the generated ProviderHealth schema against a real API server: cluster scope, status subresource, required fields, list-map keys, the 1024-character message limit	envtest (skipped under -short)
internal/metrics	metrics_test.go	scrape-time collectors (GatherAndCompare) incl. the provider-target gauges, per-proxy series cleanup via ForgetProxy, label counts, fresh-registry-per-test property	prometheus/client_golang/testutil

Conventions (from CLAUDE.md): stdlib testing, table-driven by default, t.Parallel() where safe, tests next to source, ginkgo only in the envtest suite the scaffold generated.

What the automated suites deliberately do NOT cover

• New() constructors that dial real infrastructure: the kubernetes provider's New (connects to whatever your kubeconfig points at), the GCP provider's New and its realInstances SDK adapter (ADC + real Google endpoints). Both are thin; both are exercised by the kind run / real deployments. Their 0% coverage is by design — do not "fix" it.

• CEL and structural defaulting under the fake client: the fake client runs neither. Every unit test that relies on defaults calls the *OrDefault helpers; every CEL rule is asserted in the envtest suite instead.

• A container actually starting and serving: envtest has no kubelet, so a Pod created there sits Pending forever. Whether Squid really comes up and tunnels CONNECT is provable only on a real cluster — that is the kind verification's job, and it is exactly what caught the Squid OOM bug (below).

• Live GCP: no test talks to Google. The seam boundary (buildInsertRequest + classification) is tested exhaustively instead.

• Live libvirt, by default: the KVM and kvm2 unit tests run over an in-memory fake of the 20 libvirt calls each provider uses. Whether libvirt accepts the domain XML, whether a raw qcow2 upload plus pool refresh yields a usable backing image, and whether the guest actually boots and reports an address are provable only on a real host — that is the opt-in integration test below (one per provider).

• Live Proxmox, by default: the proxmox unit tests run over an in-memory fake of the 16 seam calls, and the HTTP client over httptest replaying responses captured from PVE 9.2.20. Whether a clone, config write, seed upload and start yield a guest that boots and reports an address is provable only on a real host (the proxmox integration test below, which has passed on pv1-1). Two kinds of path have met only the fake:

  • the cross-node paths: a clone that targets another node, the seed deleted on every node when the VM's node is unknown, a Check round over several nodes. The test host is a single node, so the integration test cannot reach them either (proxmox-provider-spec.md entry 24);
  • an answer never captured: the storage status for a storage the cluster knows but cannot use on a node (disabled there, or offline). Every 500 there is taken as StorageNotFound, and what the fake answers is its own guess. The three answers Phase 0 missed were captured on pv1-1 on 2026-09-24: a missing storage is a 400 with storage: No such storage., a hibernated VM is destroyed without complaint (its suspend state volume with it), and a stop on a paused VM ends OK with the VM stopped.

The KVM integration test (real hypervisor)

internal/provider/kvm/kvm_integration_test.go compiles only under -tags=integration and skips itself unless the environment names a host. It deviates from the "testcontainers for real infra" convention on purpose: libvirt needs /dev/kvm and a bridge to a DHCP network, and a container has neither. So it targets whichever hypervisor you point it at and leaves that host's pool with one more cached base image than it had.

export KVM_TEST_HOST=nas1.home.lan:22          # SSH endpoint
export KVM_TEST_USER=virt                      # must be in the libvirt group
export KVM_TEST_SSH_KEY=$HOME/.ssh/id_ed25519  # private key for that user
export KVM_TEST_KNOWN_HOSTS=$HOME/.ssh/kvm_known_hosts  # the hypervisors' host keys
export KVM_TEST_BRIDGE=br0                     # bridge to a network with DHCP
export KVM_TEST_IMAGE_URL=https://cloud.debian.org/images/cloud/trixie/latest/debian-13-generic-amd64.qcow2
make test-integration-kvm

KVM_TEST_KNOWN_HOSTS is technically optional, but leaving it out turns host-key verification off for the run (the log says so); keep a kvm_known_hosts holding just the hypervisors rather than pointing the test at your everyday known_hosts, so a re-keyed hypervisor is a one-line fix in a file nothing else uses.

Also optional: KVM_TEST_IMAGE_SHA256, KVM_TEST_POOL (default default), KVM_TEST_SOCKET, KVM_TEST_TIMEOUT (default 20m, must cover the first-time image download), KVM_TEST_KEEP to leave the VM behind for virsh console when something goes wrong, and KVM_TEST_LOG_LEVEL (default 1) for how much of the provider's logging the -v output shows: at 1 every libvirt RPC is one compact line (HH:MM:SS.mmm libvirt virDomainLookupByName domain=... duration=... error=... libvirtError=ErrNoDomain), interleaved with the test's own t.Log lines; 2 adds the domain and volume XML, metadata and addresses; 0 leaves only t.Log. The per-run constants (provider, host, domain name) are left off each line. See the Logging section of kvm-provider.md.

What one run asserts, in order: New dials the host; Create returns a hosts/<host>/domains/<name> ID (downloading the image on a cold host — the log prints how long it took); a second Create returns the same ID; Get reaches Running with an IPv4 within the budget, and the UID and CreatedAt survive the metadata round-trip through libvirt; ListByTag lists it; Delete removes it, Get says ErrNotFound, a second Delete is a no-op, and the pool holds the base image and its marker but no per-VM overlay or seed. It also tries a TCP connect to port 22 on the guest, but only logs the outcome — the machine running the test may not route to the bridge network.

TestKVMCheck_realHost runs the provider health check (kvm-provider.md) against the same host. It must find the host reachable and the pool running, and it creates nothing. It needs the same variables; to run only this test:

go test -tags=integration -count=1 -v -run TestKVMCheck ./internal/provider/kvm/

The test is not part of make test and not run in CI. Keep it compiling with go vet -tags=integration ./... when the provider's internals move.

The kvm2 integration test (real hypervisor)

internal/provider/kvm2/kvm2_integration_test.go compiles only under -tags=integration and skips itself unless the environment names a host. Like the kvm one it deviates from the "testcontainers for real infra" convention on purpose: libvirt needs /dev/kvm and a bridge to a DHCP network, and a container has neither. It targets whichever hypervisor you point it at and leaves that host's pool with one more cached base image (base-test-image.qcow2 and its .sha256 marker) than it had. The kvm and kvm2 tests can share a host: kvm2 marks its domains in its own metadata namespace and never lists or adopts the other's.

export KVM2_TEST_HOST=beelink:22               # SSH endpoint
export KVM2_TEST_USER=novakj                   # must be in the libvirt group
export KVM2_TEST_SSH_KEY=$HOME/.ssh/id_ed25519 # private key for that user
export KVM2_TEST_KNOWN_HOSTS=$HOME/.ssh/kvm_known_hosts  # the hypervisors' host keys
export KVM2_TEST_BRIDGE=br0                    # bridge to a network with DHCP
export KVM2_TEST_IMAGE_URL=https://cloud.debian.org/images/cloud/trixie/latest/debian-13-generic-amd64.qcow2
make test-integration-kvm2

KVM2_TEST_KNOWN_HOSTS is technically optional, but leaving it out turns host-key verification off for the run (the log says so); keep a kvm_known_hosts holding just the hypervisors rather than pointing the test at your everyday known_hosts, so a re-keyed hypervisor is a one-line fix in a file nothing else uses.

Also optional: KVM2_TEST_IMAGE_SHA256 (verified after download and compared with the cached marker), KVM2_TEST_POOL (default default), KVM2_TEST_SOCKET (default /var/run/libvirt/libvirt-sock), KVM2_TEST_TIMEOUT (default 20m, the budget for the whole lifecycle; it must cover the first-time image download), KVM2_TEST_KEEP to leave the VM behind for virsh console when something goes wrong (the log then prints the virsh undefine --nvram and virsh vol-delete lines that clean it up), and KVM2_TEST_LOG_LEVEL (default 1) for how much of the provider's logging the -v output shows: at 1 every libvirt RPC is one compact libvirt call line (HH:MM:SS.mmm kvm2 "level"=1 "msg"="libvirt call" "rpc"="DomainLookupByName" "domain"=... "duration"=..., plus the per-operation outcome lines and connection events), interleaved with the test's own t.Log lines; 2 adds the domain and volume XML, metadata and addresses; 0 leaves only t.Log. The per-run constants (provider, host, domain name) are stated once in the first line and left off every provider line. See the Logging section of kvm2-provider.md.

What one run asserts, in order, each step logged with its duration:

1. New returns within 100 ms and without dialling (no libvirt host connected line before the first call).
2. Check reports the one host Ready with reason Reachable and the hostname, libvirtVersion, poolState (running) and poolAvailableGiB details, which are logged.
3. Every later call goes through provider.WithTracing with an in-memory span recorder that the package's own spans also land in.
4. Create returns hosts/kvm2-test/domains/<name> (downloading the image on a cold pool — the log prints how long it took).
5. A second Create returns the same ID and the host's domain count, read over a dedicated connection, is unchanged.
6. Get, polled every 5 s, reaches Running with an IPv4 within the budget; its UID is the one Create wrote and CreatedAt is non-zero and within a few minutes of now. The time from Create's return to the first address is logged.
7. ListByTag contains the ID with that UID.
8. A TCP connect to port 22 on the guest, logged only — the machine running the test may not route to the bridge network.
9. Delete returns nil, Get is ErrNotFound, a second Delete is a no-op.
10. The pool, inspected over a dedicated connection, holds the base image and its marker but no <name>.qcow2 or <name>-seed.iso.
11. Among the recorded spans, a provider.create span has at least one kvm2.DomainDefineXML and one kvm2.dial child, and no span attribute carries the cloud-init text.

With KVM2_TEST_KEEP steps 9 and 10 are skipped; otherwise a t.Cleanup attempts the Delete even when an earlier assertion failed.

TestKVM2Check_realHost runs the provider health check (kvm2-provider.md) against the same host. It must find the host reachable and the pool running, and it creates nothing. It needs the same variables; to run only this test:

go test -tags=integration -count=1 -v -run TestKVM2Check ./internal/provider/kvm2/

The test is not part of make test and not run in CI. Keep it compiling with go vet -tags=integration ./... when the provider's internals move.

The proxmox integration test (real Proxmox VE host)

internal/provider/proxmox/proxmox_integration_test.go compiles only under -tags=integration and skips itself unless the environment names a host. It deviates from the testcontainers convention for the same reason as the kvm tests: Proxmox VE is a hypervisor, not something a container can stand in for. It needs a node with a cloud-init template without a Proxmox cloud-init drive and with the guest agent enabled, and a token with the privileges of proxmox-provider-spec.md entry 14: the role on / from the plan's "Before Phase 0" section (2026-09-18-1502-proxmox-provider.md), plus Datastore.Allocate on the ISO storage. A passing run leaves nothing behind: its VM and its seed ISO are gone.

export PROXMOX_TEST_URL=https://192.168.0.8:8006           # host must be in the node certificate
export PROXMOX_TEST_TOKEN_FILE=$HOME/.config/proxmox-token  # one line: user@realm!tokenid=secret
export PROXMOX_TEST_CA_FILE=$HOME/.config/proxmox-ca.pem    # the node's pve-root-ca.pem
export PROXMOX_TEST_NODE=pv1-1
export PROXMOX_TEST_TEMPLATE_VMID=9001
export PROXMOX_TEST_ISO_STORAGE=proxy-seeds                 # holds iso content
export PROXMOX_TEST_VLAN_TAG=4                              # template 9001's net0 is untagged; proxies live on VLAN 4
make test-integration-proxmox

Required: PROXMOX_TEST_URL, PROXMOX_TEST_TOKEN_FILE, PROXMOX_TEST_NODE, PROXMOX_TEST_TEMPLATE_VMID and PROXMOX_TEST_ISO_STORAGE; the test skips when any is missing. Optional: PROXMOX_TEST_CA_FILE (empty trusts the system roots), PROXMOX_TEST_INSECURE (true skips certificate verification; the log says so), PROXMOX_TEST_STORAGE (the clone's target storage; empty keeps the template's), PROXMOX_TEST_BRIDGE and PROXMOX_TEST_VLAN_TAG (the cloned net0's bridge and tag; empty keeps the template's), PROXMOX_TEST_VMID_MIN and PROXMOX_TEST_VMID_MAX (default 100000..199999), PROXMOX_TEST_TIMEOUT (default 20m, the budget for the whole lifecycle), PROXMOX_TEST_KEEP to leave the VM behind, and PROXMOX_TEST_LOG_LEVEL (default 1): at 1 every request is one proxmox api call line with the call, node, VMID, UPID and duration, plus the per-operation outcome lines; 2 adds request parameters, config summaries and agent addresses; 0 leaves only t.Log. Provider, node, name and VMID are stated once and left off every provider line. Without Datastore.Allocate on the ISO storage the seed delete at the end of Delete is a 403, and Delete fails as Permanent.

What one run asserts, in order, each step logged with its duration:

1. New returns within 100 ms and makes no request (no proxmox api call line before the first operation).
2. Check reports the node Ready with reason Reachable and the pveVersion, kernel, tokenID, isoStorageAvailGiB and templates (1/1) details, which are logged.
3. Every later call goes through provider.WithTracing, with an in-memory span recorder on the transport as well.
4. Create returns vms/<vmid>/<name> with the VMID in range. The machine type is 1 core, 1024 MiB and a 4 GiB disk, so a 3 GiB template is grown by the resize task.
5. A second Create returns the same ID and the cluster's QEMU VM count is unchanged.
6. Get, polled every 5 s, reaches Running with an IPv4 within the budget (a Transient answer while the guest boots is logged and polled through). Its UID and CreatedAt equal the mark in the VM's description, read back from its config, and CreatedAt is within a few minutes of now. The time from Create's return to the first address is logged.
7. ListByTag contains the ID with that UID and CreatedAt.
8. A TCP connect to port 22 on the guest, logged only: the machine running the test may not route to the VM's network.
9. Delete returns nil, Get is ErrNotFound, a second Delete is a no-op.
10. No VM with the name is left in the cluster's resources, and no <isoStorage>:iso/<name>-seed.iso in the storage's iso content. The content is read with one extra read-only GET of /nodes/{node}/storage/{storage}/content?content=iso made by the test itself (storage_content_iso.json is its captured shape): the seam has no method to list storage content, and does not get one for a test.
11. Among the recorded spans, a provider.create span has at least one HTTP child, and no span attribute carries the cloud-init text.

With PROXMOX_TEST_KEEP steps 9 and 10 are skipped, and at the end the log prints what to run as root on the node to clean up: qm stop <vmid>, qm destroy <vmid> --purge and pvesm free <isoStorage>:iso/<name>-seed.iso. Otherwise a t.Cleanup runs the Delete even when an earlier assertion failed. It finds the VM by name, so a Create that failed halfway is cleaned up too.

Cost: one full clone of the template (about 2 s for the 3 GiB template on pv1-1), one boot to an address (about 50 s), and a few seconds of destroy; a run takes a couple of minutes. The Makefile target's 30 m timeout covers the 20 m budget plus the cleanup.

TestProxmoxCheck_realHost runs the provider health check against the same host. It must find the node reachable, the ISO storage usable and the template present, and it creates nothing. It needs the same variables; to run only this test:

go test -tags=integration -count=1 -v -run TestProxmoxCheck ./internal/provider/proxmox/

Last run: 2026-09-24 on pv1-1 (Proxmox VE 9.2.20, kernel 7.0.14-17-pve, template 9001, proxy-seeds, VLAN tag 4). Both tests passed; go test reported 60.9 s for the package.

Step	Time
Check	141 ms (125 ms in TestProxmoxCheck_realHost)
clone task, submit to done	about 3 s (seen done at the third 1 s poll)
Create, clone to started VM	6.2 s
second Create (adoption, nothing created)	32 ms
Create's return to the first address	50.2 s (the agent install from the internet)
Delete (stop, seed delete, destroy), Get, second Delete	3.1 s
TestProxmox_lifecycle	59.7 s

The first Get, 32 ms after the second Create, was Transient: the listing still showed the new VM with status: unknown. The next poll 5 s later saw it running. The Delete deleted the seed while the stopped VM's ide2 still referenced it, then destroyed the VM, and step 10 found neither left. Like the kvm tests they are not part of make test and not run in CI.

The scaffolded make test-e2e suite

test/e2e/ is the kubebuilder-generated smoke suite (build image → kind cluster → deploy → assert the manager pod runs, publishes a Ready ProviderHealth for the kubernetes provider, and serves metrics that include that provider's proxy_operator_provider_target_up series). It compiles only under -tags=e2e, manages its own kind cluster (make test-e2e / make cleanup-test-e2e), and has been kept compiling (go vet -tags=e2e ./... is part of the routine) but is not part of make test and was not used for the release verification — the manual kind run below covers strictly more.

The OTel tracing spec (Tempo-gated)

test/e2e/tracing_test.go proves the full tracing pipeline against a real Tempo: deploy in kind with tracing enabled, create two kubernetes-provider proxies, delete them, and assert in Tempo that the traces exist with the documented span topology (Reconcile Proxy → reconcile.managed / provider.create / status.patch, and reconcile.delete / provider.delete on the way out).

It skips unless both env vars are set (so the rest of the suite runs anywhere). Homelab invocation:

TEMPO_URL=http://192.168.0.30:3200 \
OTLP_ENDPOINT=http://192.168.0.30:4318 \
make test-e2e

Worth knowing:

• Every span of a run carries the resource attribute test.run.id=e2e-<nanos> (injected via OTEL_RESOURCE_ATTRIBUTES, no code involved); the run ID is printed in the test log, and {resource.test.run.id="<id>"} in Grafana shows exactly that run.
• The spec preflights the OTLP endpoint from inside the cluster with a curl pod and fails fast with a clear message if it's unreachable — export failures are otherwise only visible at -zap-log-level=1.
• Proxy CRs are created in default, not the operator namespace: the squid pods carry no securityContext and the operator namespace enforces restricted PSS.

The kind verification run (the real end-to-end)

The README quickstart is the e2e test, run by hand before the MR: deploy the operator in-cluster on a throwaway kind cluster, watch a real Squid pod reach Ready via a real CONNECT probe, then exercise the whole lease lifecycle through the discovery API (list → lease → report rate_limited → same-target lease answered 409 inCooldown:1 → release 204 twice) and delete the CR to watch the finalizer remove the pod.

Worth knowing about that run (full detail in plans-executions/2026-08-07-1747-proxy-operator.md, "Verification" section): it was not a clean pass-through. It caught two real bugs the automated suites structurally could not —

1. make run-dev on a laptop can never produce a Ready kubernetes-provider proxy: health probes originate on the host, which cannot route to kind's pod network. The quickstart was rewritten to deploy in-cluster.
2. Squid was OOM-killed at startup: it sizes file-descriptor tables from RLIMIT_NOFILE, which containerd under kind sets effectively unlimited. Fixed with max_filedescriptors 1024 in the generated config, plus a regression assertion in pod_test.go.

That is the pattern to keep: when the kind run finds something, the fix lands with a unit-level regression test, so the manual run stays a discovery tool rather than a recurring gate.