328 lines
25 KiB
Markdown
328 lines
25 KiB
Markdown
# Testing
|
|
|
|
What tests exist, how to run them, and where the boundaries of the
|
|
automated suites are. The philosophy throughout: **test against the most
|
|
real double available** — a real envtest API server over a fake client, a
|
|
real CONNECT-capable proxy stub over a mocked HTTP client, the real lease
|
|
store under the discovery handlers — and leave the gaps that only real
|
|
infrastructure can close to the kind verification run, documented at the
|
|
bottom.
|
|
|
|
## Running
|
|
|
|
```sh
|
|
make test # THE canonical run: codegen + fmt + vet + envtest setup,
|
|
# then `go test -race` on every package, with coverage
|
|
go test -short ./... # skip the envtest suite (runs in <1s per package)
|
|
go tool cover -html=cover.out # browse coverage from the last make test
|
|
```
|
|
|
|
Targeted runs:
|
|
|
|
```sh
|
|
go test -race ./internal/lease/ # one package
|
|
go test -race -run TestAcquire ./internal/lease/ # one test (or a prefix)
|
|
go test -race -count=2 ./internal/health/ # flake-shaking: run twice
|
|
```
|
|
|
|
**Gotcha — plain `go test ./...` fails the controller package** with an
|
|
error like `/usr/local/kubebuilder/bin/etcd: no such file`. That is not a
|
|
code problem: the envtest suite needs `KUBEBUILDER_ASSETS` pointing at the
|
|
API-server/etcd binaries, which only `make test` sets up (via
|
|
`setup-envtest`). Either use `make test`, or export it once:
|
|
|
|
```sh
|
|
KUBEBUILDER_ASSETS="$PWD/bin/k8s/<version>-<os>-<arch>" go test -race ./...
|
|
```
|
|
|
|
Everything runs with `-race` — `make test` enforces it; keep the flag on
|
|
manual runs too.
|
|
|
|
## The suites, package by package
|
|
|
|
| Package | Files | What is covered | Test double / technique |
|
|
|---|---|---|---|
|
|
| `api/v1alpha1` | `helpers_test.go` | `EffectivePort/EffectiveHost/HealthCheckOrDefault/MaxLeasesOrDefault` | pure table-driven; CEL rules are **not** unit-testable — see envtest below |
|
|
| `internal/controller` | `reconcile_test.go` | every row of the reconciler's action table: create, poll, publish IP, replace, adopt, NotFound recovery, delete/finalizer, quota/permanent/transient errors, cloud-init resolution | `controller-runtime/pkg/client/fake` + an in-test `stubProvider` |
|
|
| | `status_test.go` | `computePhase` truth table (10 rows) | pure |
|
|
| | `spechash_test.go` | hash stability, nil/empty normalization, sensitivity to every replacement-triggering field | pure |
|
|
| | `health_test.go` | Healthy-condition representation: Ready/Unhealthy phases, no-verdict, stale-verdict cleanup on replacement, nil snapshotter | fake client + `fakeSnapshotter` |
|
|
| | `proxy_controller_test.go` (envtest, ginkgo) | full lifecycles against a **real API server**: provision→Running, spec-change replacement, finalizer deletion, External tracking, Ready-through-health, quota-vs-permanent, adoption; **all CEL rules** (six invalid creates, mode/provider immutability incl. removal) and the `default={}` materialization | envtest apiserver + stub provider; the only place CEL and structural defaulting actually execute |
|
|
| `internal/provider` | `name_test.go` | deterministic naming: idempotency, `^proxy-[a-z2-7]{16}$`, 10k-UID distinctness | pure |
|
|
| | `errors_test.go` | taxonomy: `Class()` mapping, `errors.Is` **and** `errors.As` through the multi-unwrap | pure |
|
|
| | `config_test.go` | providers-config parsing + fail-fast validation | pure |
|
|
| | `metrics_test.go` | `WithMetrics` decorator: classified result labels, error passthrough | fake recorder + static provider |
|
|
| `internal/provider/kubernetes` | `kubernetes_test.go`, `pod_test.go` | Create/Get/Delete/ListByTag over real Pod objects; pure `buildPod`/`squidConf` incl. the `max_filedescriptors` OOM-regression assertion | `client/fake` (real `corev1.Pod`s, no kubelet) |
|
|
| | `check_test.go` | health check: a single cluster-wide list with `Limit=1` and the managed-label selector; the reason for every API error class | `client/fake` + interceptor |
|
|
| `internal/provider/gcp` | `insert_test.go`, `errors_test.go`, `gcp_test.go` | field-by-field `buildInsertRequest`; HTTP-code classification table; zone-qualified IDs, 409-is-success, 9-row state mapping, ListByTag filter + `ReturnPartialSuccess` | pure builder needs **no fake**; the rest uses the flattened `instancesAPI` seam |
|
|
| | `check_test.go` | health check: request shape and the reason per HTTP code or network error; the **real SDK client** makes exactly one request with `maxResults=1`, even when a next page exists | fake seam + `httptest` |
|
|
| `internal/provider/kvm` | `domainxml_test.go`, `volume_test.go`, `seed_test.go`, `id_test.go`, `metadata_test.go` | pure builders: domain/volume XML unmarshalled back and checked field by field, seed ISO reopened with the same library, providerID round-trip, metadata in both namespace spellings | pure |
|
|
| | `errors_test.go`, `conn_test.go` | libvirt-code classification table; connection pool (reuse, redial, drop on transport error, cancel releases a blocked RPC, one dial per host under concurrency); SSH dial, host-key policy, keepalive | fake dialer; a **real in-process `x/crypto/ssh` server** forwarding a unix socket |
|
|
| | `image_test.go`, `kvm_test.go` | image cache (fresh fetch, ready, marker missing, checksum change/mismatch, 4xx/5xx, concurrent callers); Create resume after every partial step, leftover truncated seed, state/IP mapping, every Delete already-gone case, ListByTag across hosts / one down / unreadable domain; logging tiers (V0 silent, V1 one line per RPC, V2 XML) with cloud-init never logged | `httptest` image server + in-memory `fakeAPI` behind the `libvirtAPI` seam |
|
|
| | `trace_test.go`, `conn_test.go`, `errors_test.go` | every `libvirtAPI` method traced exactly once (guarded by reflection on the interface), results and errors pass through, libvirt error fields, detail keys only at V2, upload content never logged; pool connect/drop lines, no wrapping below V1 | `fakeAPI` + `funcr` capture |
|
|
| | `check_test.go`, `conn_test.go` | health check: details, per-host verdicts in config order, hosts checked concurrently, a timeout that keeps the connection and refuses a second check while the first still waits, no waiting behind a reconcile's dial, the reason table; `probe` abandons without closing, but still drops on a transport error; waiting for a host's slot ends with the caller's deadline; an inactive or missing pool and daemon errors keep the connection; every dial stage tagged through the real SSH server, with classification unchanged | `fakeAPI`, fake dialers, in-process SSH server |
|
|
| | `kvm_integration_test.go` (`-tags=integration`) | one real VM: image fetch, Create, idempotent re-Create, Running with IP, ListByTag, Delete, pool state afterwards; `TestKVMCheck_realHost` runs the health check against the host and creates nothing | a **real hypervisor** — see below |
|
|
| `internal/provider/kvm2` | `xml_test.go`, `volume_test.go`, `seed_test.go`, `id_test.go`, `metadata_test.go`, `state_test.go` | pure builders: domain XML (file disks with resolved paths, UEFI with secure boot off, BIOS escape, the metadata element, the guest-agent channel), volume XML with and without a backing store, seed ISO reopened (files, label, user-data byte for byte, temp dir removed) and the vendor-data script vs. multipart rules, providerID round-trip and malformed forms, metadata in both namespace spellings, state mapping and the IPv4 filter | pure |
|
|
| | `errors_test.go`, `backend_test.go`, `hosts_test.go`, `ssh_test.go` | classification table (libvirt codes, SSH auth text, `knownhosts.KeyError`, missing files); the per-call budget adapter (deadline kills the connection and is Transient, cancel unblocks, killed by another caller, close idempotent) against a mocked libvirt socket; the host table (one dial per host under concurrency, redial after drop, `dialFresh` never cached, waiting ends with ctx); the real dialer end to end: socket forwarding, unknown and mismatched host keys, insecure mode, missing vs. unparsable key and known_hosts, wrong key, handshake timeout, cancel, every stage tagged, key material never in errors or logs | fake dialer; a **real in-process `x/crypto/ssh` server** forwarding a unix socket |
|
|
| | `image_test.go`, `create_test.go`, `get_test.go`, `delete_test.go`, `list_test.go`, `kvm2_test.go`, `provider_test.go` | image cache (fetch with the marker written last, cached, marker missing or without a digest, sha256 change or mismatch, 4xx/5xx, no Content-Length, short read, concurrent callers fetch once, cancel mid-stream, streamed without buffering, upload on a dedicated connection) and the pool-space quota precheck; Create resume after every partial step, returns only when running, adoption guard on UID, seed always rebuilt, cloud-init sentinel never logged; Get state and address mapping incl. agent-before-ARP; every Delete already-gone case, NVRAM flag, per-name lock; ListByTag across hosts, one down fails the whole call, unreadable domain skipped; `New` never dials and warns on missing files; every error wrapped with the configured name; logging tiers V0/V1/V2 | `httptest` image server + in-memory `fakeBackend` behind the `backend` seam |
|
|
| | `observe_test.go`, `check_test.go` | every seam method traced and logged exactly once (guarded by reflection), child spans under the parent, detail only at V2, upload bodies never logged, error fields; health check: one entry per host in config order, hosts checked concurrently, expired ctx returns promptly, never touches the pooled connection, the reason table, detail keys | `fakeBackend` + `funcr` capture + span recorder |
|
|
| | `kvm2_integration_test.go` (`-tags=integration`) | one real VM: `New` without a dial, `Check` Ready with details, Create, idempotent re-Create with an unchanged domain count, Running with IP and the metadata round-trip, ListByTag, Delete, NotFound, pool state afterwards, `kvm2.*` spans under `provider.create`; `TestKVM2Check_realHost` runs the health check against the host and creates nothing | a **real hypervisor** — see below |
|
|
| `internal/health` | `probe_test.go` | probes through a **real CONNECT-capable proxy stub** to a real TLS server: tunnel success, refused CONNECT, wrong status, dead proxy, plain-http forward | `httptest` + hijacked bidirectional tunnel |
|
|
| | `engine_test.go` | thresholds, first-verdict, latency suppression matrix, dropped-event retry, stale-probe UID guard, tick scheduling/seeding/pruning, `Start` end-to-end | fake `client.Reader` + injected `probeFn` |
|
|
| `internal/lease` | `store_test.go` | capacity, `MaxLeases=0`, all three selection tie-breaks, target-scoped vs global cooldowns, expiry via fake clock, report-on-retained-lease, idempotent release, **40 concurrent acquires never exceeding capacity** | injectable clock; the concurrency case is why `-race` matters |
|
|
| `internal/discovery` | `server_test.go` | auth matrix (incl. `/healthz` exemption), list filtering, grant shape, full 409 arithmetic, invalid TTL/body/result, idempotent release, report→cooldown→409 round trip, `Start`/shutdown | `httptest` over the real handler chain, fake cache reader, **real** `lease.Store` |
|
|
| `internal/gc` | `gc_test.go` | the true-orphan matrix (live/deleting/young/unlabelled all kept), broken-provider isolation, list-failure skips sweep, namespace guard, sweep loop | fake reader + canned-list provider |
|
|
| `internal/providerhealth` | `status_test.go`, `monitor_test.go` | `buildStatus` (the Ready condition, per-target transition times, truncation); objects created with the label and patched, the round timeout, no-checker Unknown, invalid names, transition-only logging, cleanup of labelled unconfigured objects, the checks-off path, `ValidateSchedule` | fake client with a cluster-scoped RESTMapper + `funcr` capture |
|
|
| | `envtest_test.go` | the generated `ProviderHealth` schema against a real API server: cluster scope, status subresource, required fields, list-map keys, the 1024-character message limit | envtest (skipped under `-short`) |
|
|
| `internal/metrics` | `metrics_test.go` | scrape-time collectors (`GatherAndCompare`) incl. the provider-target gauges, per-proxy series cleanup via `ForgetProxy`, label counts, fresh-registry-per-test property | `prometheus/client_golang/testutil` |
|
|
|
|
Conventions (from `CLAUDE.md`): stdlib `testing`, table-driven by default,
|
|
`t.Parallel()` where safe, tests next to source, ginkgo only in the
|
|
envtest suite the scaffold generated.
|
|
|
|
## What the automated suites deliberately do NOT cover
|
|
|
|
- **`New()` constructors that dial real infrastructure**: the kubernetes
|
|
provider's `New` (connects to whatever your kubeconfig points at), the
|
|
GCP provider's `New` and its `realInstances` SDK adapter (ADC + real
|
|
Google endpoints). Both are thin; both are exercised by the kind run /
|
|
real deployments. Their 0% coverage is by design — do not "fix" it.
|
|
- **CEL and structural defaulting under the fake client**: the fake
|
|
client runs neither. Every unit test that relies on defaults calls the
|
|
`*OrDefault` helpers; every CEL rule is asserted in the envtest suite
|
|
instead.
|
|
- **A container actually starting and serving**: envtest has no kubelet,
|
|
so a Pod created there sits Pending forever. Whether Squid really comes
|
|
up and tunnels CONNECT is provable only on a real cluster — that is the
|
|
kind verification's job, and it is exactly what caught the Squid OOM
|
|
bug (below).
|
|
- **Live GCP**: no test talks to Google. The seam boundary
|
|
(`buildInsertRequest` + classification) is tested exhaustively instead.
|
|
- **Live libvirt, by default**: the KVM and kvm2 unit tests run over an
|
|
in-memory fake of the 20 libvirt calls each provider uses. Whether
|
|
libvirt accepts the domain XML, whether a raw qcow2 upload plus pool
|
|
refresh yields a usable backing image, and whether the guest actually
|
|
boots and reports an address are provable only on a real host — that is
|
|
the opt-in integration test below (one per provider).
|
|
|
|
## The KVM integration test (real hypervisor)
|
|
|
|
`internal/provider/kvm/kvm_integration_test.go` compiles only under
|
|
`-tags=integration` and skips itself unless the environment names a
|
|
host. It deviates from the "testcontainers for real infra" convention on
|
|
purpose: libvirt needs `/dev/kvm` and a bridge to a DHCP network, and a
|
|
container has neither. So it targets whichever hypervisor you point it
|
|
at and leaves that host's pool with one more cached base image than it
|
|
had.
|
|
|
|
```sh
|
|
export KVM_TEST_HOST=nas1.home.lan:22 # SSH endpoint
|
|
export KVM_TEST_USER=virt # must be in the libvirt group
|
|
export KVM_TEST_SSH_KEY=$HOME/.ssh/id_ed25519 # private key for that user
|
|
export KVM_TEST_KNOWN_HOSTS=$HOME/.ssh/kvm_known_hosts # the hypervisors' host keys
|
|
export KVM_TEST_BRIDGE=br0 # bridge to a network with DHCP
|
|
export KVM_TEST_IMAGE_URL=https://cloud.debian.org/images/cloud/trixie/latest/debian-13-generic-amd64.qcow2
|
|
make test-integration-kvm
|
|
```
|
|
|
|
`KVM_TEST_KNOWN_HOSTS` is technically optional, but leaving it out turns
|
|
host-key verification **off** for the run (the log says so); keep a
|
|
`kvm_known_hosts` holding just the hypervisors rather than pointing the
|
|
test at your everyday `known_hosts`, so a re-keyed hypervisor is a
|
|
one-line fix in a file nothing else uses.
|
|
|
|
Also optional: `KVM_TEST_IMAGE_SHA256`, `KVM_TEST_POOL`
|
|
(default `default`), `KVM_TEST_SOCKET`, `KVM_TEST_TIMEOUT` (default
|
|
`20m`, must cover the first-time image download), `KVM_TEST_KEEP`
|
|
to leave the VM behind for `virsh console` when something goes wrong,
|
|
and `KVM_TEST_LOG_LEVEL` (default `1`) for how much of the provider's
|
|
logging the `-v` output shows: at `1` every libvirt RPC is one compact
|
|
line (`HH:MM:SS.mmm libvirt virDomainLookupByName domain=... duration=...
|
|
error=... libvirtError=ErrNoDomain`), interleaved with the test's own
|
|
`t.Log` lines; `2` adds the domain and volume XML, metadata and
|
|
addresses; `0` leaves only `t.Log`. The per-run constants (provider,
|
|
host, domain name) are left off each line. See the Logging section of
|
|
[kvm-provider.md](kvm-provider.md#logging).
|
|
|
|
What one run asserts, in order: `New` dials the host; `Create` returns a
|
|
`hosts/<host>/domains/<name>` ID (downloading the image on a cold host —
|
|
the log prints how long it took); a second `Create` returns the same ID;
|
|
`Get` reaches `Running` with an IPv4 within the budget, and the UID and
|
|
`CreatedAt` survive the metadata round-trip through libvirt; `ListByTag`
|
|
lists it; `Delete` removes it, `Get` says `ErrNotFound`, a second
|
|
`Delete` is a no-op, and the pool holds the base image and its marker
|
|
but no per-VM overlay or seed. It also tries a TCP connect to port 22 on
|
|
the guest, but only logs the outcome — the machine running the test may
|
|
not route to the bridge network.
|
|
|
|
`TestKVMCheck_realHost` runs the provider health check
|
|
([kvm-provider.md](kvm-provider.md#health-check)) against the same host.
|
|
It must find the host reachable and the pool running, and it creates
|
|
nothing. It needs the same variables; to run only this test:
|
|
|
|
```sh
|
|
go test -tags=integration -count=1 -v -run TestKVMCheck ./internal/provider/kvm/
|
|
```
|
|
|
|
The test is not part of `make test` and not run in CI. Keep it compiling
|
|
with `go vet -tags=integration ./...` when the provider's internals move.
|
|
|
|
## The kvm2 integration test (real hypervisor)
|
|
|
|
`internal/provider/kvm2/kvm2_integration_test.go` compiles only under
|
|
`-tags=integration` and skips itself unless the environment names a
|
|
host. Like the kvm one it deviates from the "testcontainers for real
|
|
infra" convention on purpose: libvirt needs `/dev/kvm` and a bridge to a
|
|
DHCP network, and a container has neither. It targets whichever
|
|
hypervisor you point it at and leaves that host's pool with one more
|
|
cached base image (`base-test-image.qcow2` and its `.sha256` marker) than
|
|
it had. The kvm and kvm2 tests can share a host: kvm2 marks its domains
|
|
in its own metadata namespace and never lists or adopts the other's.
|
|
|
|
```sh
|
|
export KVM2_TEST_HOST=beelink:22 # SSH endpoint
|
|
export KVM2_TEST_USER=novakj # must be in the libvirt group
|
|
export KVM2_TEST_SSH_KEY=$HOME/.ssh/id_ed25519 # private key for that user
|
|
export KVM2_TEST_KNOWN_HOSTS=$HOME/.ssh/kvm_known_hosts # the hypervisors' host keys
|
|
export KVM2_TEST_BRIDGE=br0 # bridge to a network with DHCP
|
|
export KVM2_TEST_IMAGE_URL=https://cloud.debian.org/images/cloud/trixie/latest/debian-13-generic-amd64.qcow2
|
|
make test-integration-kvm2
|
|
```
|
|
|
|
`KVM2_TEST_KNOWN_HOSTS` is technically optional, but leaving it out turns
|
|
host-key verification **off** for the run (the log says so); keep a
|
|
`kvm_known_hosts` holding just the hypervisors rather than pointing the
|
|
test at your everyday `known_hosts`, so a re-keyed hypervisor is a
|
|
one-line fix in a file nothing else uses.
|
|
|
|
Also optional: `KVM2_TEST_IMAGE_SHA256` (verified after download and
|
|
compared with the cached marker), `KVM2_TEST_POOL` (default `default`),
|
|
`KVM2_TEST_SOCKET` (default `/var/run/libvirt/libvirt-sock`),
|
|
`KVM2_TEST_TIMEOUT` (default `20m`, the budget for the whole lifecycle;
|
|
it must cover the first-time image download), `KVM2_TEST_KEEP` to leave
|
|
the VM behind for `virsh console` when something goes wrong (the log then
|
|
prints the `virsh undefine --nvram` and `virsh vol-delete` lines that
|
|
clean it up), and `KVM2_TEST_LOG_LEVEL` (default `1`) for how much of the
|
|
provider's logging the `-v` output shows: at `1` every libvirt RPC is one
|
|
compact `libvirt call` line (`HH:MM:SS.mmm kvm2 "level"=1 "msg"="libvirt
|
|
call" "rpc"="DomainLookupByName" "domain"=... "duration"=...`, plus the
|
|
per-operation outcome lines and connection events), interleaved with the
|
|
test's own `t.Log` lines; `2` adds the domain and volume XML, metadata
|
|
and addresses; `0` leaves only `t.Log`. The per-run constants (provider,
|
|
host, domain name) are stated once in the first line and left off every
|
|
provider line. See the Logging section of
|
|
[kvm2-provider.md](kvm2-provider.md).
|
|
|
|
What one run asserts, in order, each step logged with its duration:
|
|
|
|
1. `New` returns within 100 ms and without dialling (no `libvirt host
|
|
connected` line before the first call).
|
|
2. `Check` reports the one host `Ready` with reason `Reachable` and the
|
|
`hostname`, `libvirtVersion`, `poolState` (`running`) and
|
|
`poolAvailableGiB` details, which are logged.
|
|
3. Every later call goes through `provider.WithTracing` with an in-memory
|
|
span recorder that the package's own spans also land in.
|
|
4. `Create` returns `hosts/kvm2-test/domains/<name>` (downloading the
|
|
image on a cold pool — the log prints how long it took).
|
|
5. A second `Create` returns the same ID and the host's domain count,
|
|
read over a dedicated connection, is unchanged.
|
|
6. `Get`, polled every 5 s, reaches `Running` with an IPv4 within the
|
|
budget; its `UID` is the one Create wrote and `CreatedAt` is non-zero
|
|
and within a few minutes of now. The time from Create's return to the
|
|
first address is logged.
|
|
7. `ListByTag` contains the ID with that UID.
|
|
8. A TCP connect to port 22 on the guest, logged only — the machine
|
|
running the test may not route to the bridge network.
|
|
9. `Delete` returns nil, `Get` is `ErrNotFound`, a second `Delete` is a
|
|
no-op.
|
|
10. The pool, inspected over a dedicated connection, holds the base image
|
|
and its marker but no `<name>.qcow2` or `<name>-seed.iso`.
|
|
11. Among the recorded spans, a `provider.create` span has at least one
|
|
`kvm2.DomainDefineXML` and one `kvm2.dial` child, and no span
|
|
attribute carries the cloud-init text.
|
|
|
|
With `KVM2_TEST_KEEP` steps 9 and 10 are skipped; otherwise a `t.Cleanup`
|
|
attempts the Delete even when an earlier assertion failed.
|
|
|
|
`TestKVM2Check_realHost` runs the provider health check
|
|
([kvm2-provider.md](kvm2-provider.md)) against the same host. It must
|
|
find the host reachable and the pool running, and it creates nothing. It
|
|
needs the same variables; to run only this test:
|
|
|
|
```sh
|
|
go test -tags=integration -count=1 -v -run TestKVM2Check ./internal/provider/kvm2/
|
|
```
|
|
|
|
The test is not part of `make test` and not run in CI. Keep it compiling
|
|
with `go vet -tags=integration ./...` when the provider's internals move.
|
|
|
|
## The scaffolded `make test-e2e` suite
|
|
|
|
`test/e2e/` is the kubebuilder-generated smoke suite (build image → kind
|
|
cluster → deploy → assert the manager pod runs, publishes a Ready
|
|
`ProviderHealth` for the `kubernetes` provider, and serves metrics that
|
|
include that provider's `proxy_operator_provider_target_up` series). It
|
|
compiles only under `-tags=e2e`, manages its own kind cluster
|
|
(`make test-e2e` / `make cleanup-test-e2e`), and has been kept compiling
|
|
(`go vet -tags=e2e ./...` is part of the routine) but is **not part of
|
|
`make test` and was not used for the release verification** — the manual
|
|
kind run below covers strictly more.
|
|
|
|
### The OTel tracing spec (Tempo-gated)
|
|
|
|
`test/e2e/tracing_test.go` proves the full tracing pipeline against a
|
|
**real Tempo**: deploy in kind with tracing enabled, create two
|
|
kubernetes-provider proxies, delete them, and assert in Tempo that the
|
|
traces exist with the documented span topology (`Reconcile Proxy` →
|
|
`reconcile.managed` / `provider.create` / `status.patch`, and
|
|
`reconcile.delete` / `provider.delete` on the way out).
|
|
|
|
It **skips unless both env vars are set** (so the rest of the suite runs
|
|
anywhere). Homelab invocation:
|
|
|
|
```bash
|
|
TEMPO_URL=http://192.168.0.30:3200 \
|
|
OTLP_ENDPOINT=http://192.168.0.30:4318 \
|
|
make test-e2e
|
|
```
|
|
|
|
Worth knowing:
|
|
|
|
- Every span of a run carries the resource attribute
|
|
`test.run.id=e2e-<nanos>` (injected via `OTEL_RESOURCE_ATTRIBUTES`, no
|
|
code involved); the run ID is printed in the test log, and
|
|
`{resource.test.run.id="<id>"}` in Grafana shows exactly that run.
|
|
- The spec preflights the OTLP endpoint **from inside the cluster** with a
|
|
curl pod and fails fast with a clear message if it's unreachable —
|
|
export failures are otherwise only visible at `-zap-log-level=1`.
|
|
- Proxy CRs are created in `default`, not the operator namespace: the
|
|
squid pods carry no securityContext and the operator namespace enforces
|
|
restricted PSS.
|
|
|
|
## The kind verification run (the real end-to-end)
|
|
|
|
The README quickstart **is** the e2e test, run by hand before the MR:
|
|
deploy the operator in-cluster on a throwaway kind cluster, watch a real
|
|
Squid pod reach `Ready` via a real CONNECT probe, then exercise the whole
|
|
lease lifecycle through the discovery API (list → lease → report
|
|
`rate_limited` → same-target lease answered 409 `inCooldown:1` → release
|
|
204 twice) and delete the CR to watch the finalizer remove the pod.
|
|
|
|
Worth knowing about that run (full detail in
|
|
[plans-executions/2026-08-07-1747-proxy-operator.md](plans-executions/2026-08-07-1747-proxy-operator.md),
|
|
"Verification" section): it was not a clean pass-through. It caught two
|
|
real bugs the automated suites structurally could not —
|
|
|
|
1. `make run-dev` on a laptop can never produce a `Ready`
|
|
kubernetes-provider proxy: health probes originate on the host, which
|
|
cannot route to kind's pod network. The quickstart was rewritten to
|
|
deploy in-cluster.
|
|
2. Squid was OOM-killed at startup: it sizes file-descriptor tables from
|
|
`RLIMIT_NOFILE`, which containerd under kind sets effectively
|
|
unlimited. Fixed with `max_filedescriptors 1024` in the generated
|
|
config, plus a regression assertion in `pod_test.go`.
|
|
|
|
That is the pattern to keep: when the kind run finds something, the fix
|
|
lands **with a unit-level regression test**, so the manual run stays a
|
|
discovery tool rather than a recurring gate.
|