egress-proxies-operator/docs/plans/2026-08-24-1224-tracing-e2e.md

Plan: e2e test — OTel tracing against real Tempo

Created: 2026-08-24 12:24

Context

The tracing feature (MR #4, branch feat/otel-tracing) is covered by unit tests with an in-memory span recorder, but nothing proves the full pipeline: operator in a real cluster → OTLP export → Tempo ingest → queryable traces with the documented span topology. The homelab Tempo lives at http://192.168.0.30:3200 (query API — verified live from this host: /api/echo, TraceQL GET /api/search?q=…, GET /api/traces/<id>), with OTLP ingest on :4318 (HTTP, verified 200) and :4317 (gRPC, open). User decisions: Go Ginkgo e2e test (not a shell script), structural assertions (span tree, not just trace-exists), flow per the user's sketch: deploy operator in kind with tracing on → create kubernetes-provider proxies → delete them → shut down — with traces tagged for easy discovery per test run. This work continues on feat/otel-tracing; MR #4 stays the vehicle.

Design decisions (from review, with evidence)

• Proxy CRs go in the default namespace, not the operator namespace. The squid pods the kubernetes provider creates carry no securityContext (internal/provider/kubernetes/pod.go:33-49) and are created in the CR's namespace; the operator ns is labeled pod-security enforce=restricted (test/e2e/e2e_test.go:61-62) and would reject them. Pod RBAC is cluster-scoped (config/rbac/role.yaml:10), so default works. All kubectl calls use explicit -n.
• Anchor assertions on provider.create, not the root-span search. A name="Reconcile Proxy" hit may be the finalizer-add or a drift reconcile (no provider call inside). TraceQL matches span names anywhere in a trace, so search {resource.test.run.id="<runID>" && name="provider.create"}, fetch that traceID, then assert companion span names.
• Run identification with zero code changes: OTel resource attribute test.run.id=<runID> via OTEL_RESOURCE_ATTRIBUTES (env wins — resource.WithFromEnv() merges last, internal/tracing/tracing.go). Tempo vParquet searches arbitrary resource attrs without config; fallback query documented: {resource.service.name="egress-proxies-operator" && name="provider.create"}
  • start/end window (Unix seconds).
• OTLP preflight from inside the cluster: host-reachability of 192.168.0.30 doesn't prove pod-reachability from kind on macOS, and export failures log only at V(1) (invisible) — without a preflight, a broken path is a 2-minute opaque timeout. Reuse the suite's curl-pod pattern (e2e_test.go:218-248, incl. the restricted-PSS overrides JSON) to POST to <OTLP_ENDPOINT>/v1/traces and fail fast.
• No homelab defaults baked into the Makefile. The spec Skips unless TEMPO_URL/OTLP_ENDPOINT are set (otherwise everyone without that LAN host inherits a 2-minute failure). The copy-pasteable invocation lives in docs/testing.md.
• Ginkgo ordering: top-level container order is randomized, so the tracing Describe is fully self-contained (own ns create → make install → make deploy → teardown), same shape as the Manager Describe. kubectl delete ns blocks until termination, so no create/delete race between the two Describes.

Steps

Step 1 — test/e2e/tracing_test.go (new; //go:build e2e, package e2e)

Describe("OTel tracing", Ordered):

• BeforeAll:
  1. Read TEMPO_URL + OTLP_ENDPOINT; Skip("TEMPO_URL/OTLP_ENDPOINT not set") when empty.
  2. runID := "e2e-" + strconv.FormatInt(time.Now().UnixNano(), 10); print it to GinkgoWriter so the run is findable in Grafana by hand.
  3. Create operator ns + restricted-PSS label; make install; make deploy IMG=<managerImage> (same commands as e2e_test.go:54-75, via utils.Run).
  4. Pre-pull the proxy image to kill the biggest flake source: docker pull ubuntu/squid:6.6-24.04_edge + kind load docker-image (via utils.Run, honoring KIND/KIND_CLUSTER env like utils.LoadImageToKindClusterWithName).
  5. OTLP preflight: curl pod POSTing {} to <OTLP_ENDPOINT>/v1/traces, assert HTTP 200 in its logs (fail message names the endpoint).
  6. kubectl set env deployment/egress-proxies-operator-controller-manager -n <ns> OTEL_EXPORTER_OTLP_ENDPOINT=<OTLP_ENDPOINT> "OTEL_RESOURCE_ATTRIBUTES=k8s.pod.name=$(POD_NAME),k8s.namespace.name=$(POD_NAMESPACE),test.run.id=<runID>" — in-place update keeps the var after the downward-API vars ($(VAR) expansion needs that ordering); exec.Command means no shell mangling of $(). Then kubectl rollout status.
  7. Eventually: newest controller pod (sorted by creationTimestamp — a terminating old pod may otherwise be picked) logs contain "tracing enabled".
• It "creates proxies and reports reconcile traces":
  • Write 2 Proxy manifests (names proxy-tracing-e2e-1/2, provider: kubernetes, modeled on config/samples/proxy_kubernetes.yaml) to GinkgoT().TempDir() (absolute paths — utils.Run chdirs the process); kubectl apply -n default.
  • Eventually (5m — first squid start) .status.phase == "Ready" (confirmed field, api/v1alpha1/proxy_types.go:41).
  • Eventually: Tempo search {resource.test.run.id="<runID>" && name="provider.create"} ≥1 hit (BSP flushes every ~5s; 2m default timeout is ample).
  • Fetch that traceID via /api/traces/<id>; assert span names include Reconcile Proxy, reconcile.managed, provider.create, status.patch, and resource attr service.name=egress-proxies-operator.
• It "traces deletion":
  • kubectl delete proxy -n default …; Eventually CRs gone (finalizer → provider.delete, DeletionPoll 10s).
  • Eventually: Tempo search finds provider.delete for the runID; fetch and assert reconcile.delete in the same trace.
• AfterAll (= the sketch's "shut down operator"): delete leftover proxies in default; make undeploy (SIGTERM → tracingShutdown flushes); make uninstall; delete operator ns — errors discarded (_, _ = utils.Run(...)), mirroring e2e_test.go:79-95.
• Tempo client helpers (same file, stdlib only — the test process runs on the host, which reaches Tempo directly):
  • tempoSearch(traceql string) → GET {TEMPO_URL}/api/search?q=…&start=…&end=… (Unix seconds, window = suite start − 5m → now); minimal struct {Traces []struct{ TraceID string }}.
  • tempoTrace(id string) → /api/traces/<id>, decoded as OTLP-JSON (batches[].scopeSpans[].spans[].name, resource attrs as {key, value:{stringValue}}) — not Jaeger's shape. Helper flattens to a span-name set + resource attr map.

Step 2 — Makefile

• test-e2e: pass TEMPO_URL/OTLP_ENDPOINT through to go test env (no defaults) and add -timeout 30m — the suite already runs docker-build + kind-load in BeforeSuite plus 3–5m Eventuallys, and this adds a second full deploy cycle + image pulls; the 10m default will be exceeded on cold caches.

Step 3 — Docs

• docs/testing.md (§ e2e, currently "scaffold… only asserts manager runs and serves metrics"): document the tracing spec, its Skip gate, and the invocation: TEMPO_URL=http://192.168.0.30:3200 OTLP_ENDPOINT=http://192.168.0.30:4318 make test-e2e.
• Execution-log entries per step in docs/plans-executions/2026-08-24-1224-tracing-e2e.md.
• CHANGELOG entry only after the user confirms a green run (house rule).

Not doing (and why)

• No dedicated flush assertion on shutdown — teardown exercises the flush path, but attributing a specific late span to it is guesswork.
• No gRPC (4317) variant — http/protobuf is the operator default and the verified path; a variant run is a one-env-var change if ever wanted.
• No changes to internal/ — the whole test works through public surface (env vars, kubectl, Tempo API), which is the point of an e2e test.

Verification

1. TEMPO_URL=http://192.168.0.30:3200 OTLP_ENDPOINT=http://192.168.0.30:4318 make test-e2e → tracing spec passes (both Its); without the env vars → spec reports Skipped, remainder of suite unaffected.
2. In Grafana: TraceQL {resource.test.run.id="<runID printed in test log>"} shows the run's traces with the expected tree.
3. go vet -tags=e2e ./... clean (documented routine).
4. A deliberately wrong OTLP_ENDPOINT (e.g. port 9) fails fast in the preflight step with a clear message, not a 2-minute search timeout.