egress-proxies-operator/docs/plans-executions/2026-08-24-1224-tracing-e2e.md

Execution log: e2e test — OTel tracing against real Tempo

Plan: docs/plans/2026-08-24-1224-tracing-e2e.md

• Step 1 — test/e2e/tracing_test.go
• Step 2 — Makefile
• Step 3 — Docs

Steps 1–3 — spec, Makefile, docs (one commit)

The three steps landed together — the spec is one new file and the other two are its wiring. Describe("OTel tracing", Ordered) is fully self-contained (own ns create → make install/make deploy → teardown) because Ginkgo randomizes top-level container order, so it cannot share the Manager Describe's deployment. It reuses the package-level namespace / managerImage and the suite's idioms (utils.Run, curl-pod with the restricted-PSS overrides JSON, log-substring Eventuallys).

Judgment calls beyond the plan's letter:

• The squid pre-pull (docker pull + kind load) is best effort — a missing docker binary logs a note and continues rather than failing the spec; the 5m Ready timeout still covers an in-cluster pull.
• The OTLP preflight pod prints per-attempt HTTP codes and a final OTLP_OK/OTLP_UNREACHABLE marker; the assertion quotes the pod's output, so an unreachable endpoint names itself in the failure.
• kubectl set env is passed the literal OTEL_RESOURCE_ATTRIBUTES=...$(POD_NAME)... string via exec.Command — no shell involved, kubectl stores $() verbatim, and the in-place update keeps the var after the downward-API vars it references.
• Tempo helpers are stdlib-only; /api/traces/<id> is decoded as OTLP-JSON (batches[].scopeSpans[].spans[].name), which is Tempo's actual shape (not Jaeger's).

Verified so far: go vet -tags=e2e ./... clean.

Live run against homelab Tempo

First attempt failed before the suite started — Docker Desktop wasn't running (kind could not create the cluster), and the failure was masked to exit 0 by a | tail pipe on the make invocation (no pipefail in that shell). Rerun with Docker started first and no pipe:

TEMPO_URL=http://192.168.0.30:3200 OTLP_ENDPOINT=http://192.168.0.30:4318 make test-e2e

Result: SUCCESS — 4/4 specs (Manager smoke + both tracing Its), 219 s, kind cluster auto-deleted. Run id e2e-1787567624494810000; the traces are findable in Grafana with TraceQL {resource.test.run.id="e2e-1787567624494810000"}. Both proxies reached Ready on real squid pods; the create trace carried Reconcile Proxy / reconcile.managed / provider.create / status.patch and the expected resource attrs; the deletion trace carried reconcile.delete / provider.delete.

Worth noting: the OTLP preflight and both Tempo polls succeeded on the in-cluster → LAN path (kind on macOS reaches 192.168.0.30 through Docker Desktop's NAT), so no extra networking setup is needed on this machine.

The plan's negative verification also ran: with a deliberately wrong endpoint (OTLP_ENDPOINT=http://192.168.0.30:9999) the tracing spec failed in the BeforeAll preflight after ~52 s with OTLP endpoint http://192.168.0.30:9999 is not reachable from inside the kind cluster; curl output: OTLP_UNREACHABLE — a named, fast failure instead of a 2-minute opaque search timeout — and teardown still deleted the kind cluster (make cleanup-test-e2e run explicitly, since a failing go test skips the Makefile's cleanup step).