diff --git a/docs/plans/2026-08-24-1224-tracing-e2e.md b/docs/plans/2026-08-24-1224-tracing-e2e.md new file mode 100644 index 0000000..de3c31f --- /dev/null +++ b/docs/plans/2026-08-24-1224-tracing-e2e.md @@ -0,0 +1,150 @@ +# Plan: e2e test — OTel tracing against real Tempo +**Created:** 2026-08-24 12:24 + +## Context + +The tracing feature (MR #4, branch `feat/otel-tracing`) is covered by unit +tests with an in-memory span recorder, but nothing proves the full pipeline: +operator in a real cluster → OTLP export → Tempo ingest → queryable traces +with the documented span topology. The homelab Tempo lives at +`http://192.168.0.30:3200` (query API — verified live from this host: +`/api/echo`, TraceQL `GET /api/search?q=…`, `GET /api/traces/`), with +OTLP ingest on `:4318` (HTTP, verified 200) and `:4317` (gRPC, open). User +decisions: **Go Ginkgo e2e test** (not a shell script), **structural +assertions** (span tree, not just trace-exists), flow per the user's sketch: +deploy operator in kind with tracing on → create kubernetes-provider proxies +→ delete them → shut down — with traces tagged for easy discovery per test +run. This work continues on `feat/otel-tracing`; MR #4 stays the vehicle. + +## Design decisions (from review, with evidence) + +- **Proxy CRs go in the `default` namespace, not the operator namespace.** + The squid pods the kubernetes provider creates carry no securityContext + (internal/provider/kubernetes/pod.go:33-49) and are created in the CR's + namespace; the operator ns is labeled `pod-security enforce=restricted` + (test/e2e/e2e_test.go:61-62) and would reject them. Pod RBAC is + cluster-scoped (config/rbac/role.yaml:10), so `default` works. All + kubectl calls use explicit `-n`. +- **Anchor assertions on `provider.create`, not the root-span search.** A + `name="Reconcile Proxy"` hit may be the finalizer-add or a drift + reconcile (no provider call inside). TraceQL matches span names anywhere + in a trace, so search `{resource.test.run.id="" && name="provider.create"}`, + fetch *that* traceID, then assert companion span names. +- **Run identification with zero code changes:** OTel resource attribute + `test.run.id=` via `OTEL_RESOURCE_ATTRIBUTES` (env wins — + `resource.WithFromEnv()` merges last, internal/tracing/tracing.go). Tempo + vParquet searches arbitrary resource attrs without config; fallback query + documented: `{resource.service.name="egress-proxies-operator" && name="provider.create"}` + + start/end window (Unix **seconds**). +- **OTLP preflight from inside the cluster:** host-reachability of + 192.168.0.30 doesn't prove pod-reachability from kind on macOS, and + export failures log only at V(1) (invisible) — without a preflight, a + broken path is a 2-minute opaque timeout. Reuse the suite's curl-pod + pattern (e2e_test.go:218-248, incl. the restricted-PSS overrides JSON) + to POST to `/v1/traces` and fail fast. +- **No homelab defaults baked into the Makefile.** The spec Skips unless + `TEMPO_URL`/`OTLP_ENDPOINT` are set (otherwise everyone without that LAN + host inherits a 2-minute failure). The copy-pasteable invocation lives in + docs/testing.md. +- **Ginkgo ordering:** top-level container order is randomized, so the + tracing Describe is fully self-contained (own ns create → make install → + make deploy → teardown), same shape as the Manager Describe. `kubectl + delete ns` blocks until termination, so no create/delete race between + the two Describes. + +## Steps + +### Step 1 — `test/e2e/tracing_test.go` (new; `//go:build e2e`, package e2e) + +`Describe("OTel tracing", Ordered)`: + +- **BeforeAll:** + 1. Read `TEMPO_URL` + `OTLP_ENDPOINT`; `Skip("TEMPO_URL/OTLP_ENDPOINT not set")` when empty. + 2. `runID := "e2e-" + strconv.FormatInt(time.Now().UnixNano(), 10)`; + print it to `GinkgoWriter` so the run is findable in Grafana by hand. + 3. Create operator ns + restricted-PSS label; `make install`; + `make deploy IMG=` (same commands as e2e_test.go:54-75, + via `utils.Run`). + 4. Pre-pull the proxy image to kill the biggest flake source: + `docker pull ubuntu/squid:6.6-24.04_edge` + `kind load docker-image` + (via `utils.Run`, honoring `KIND`/`KIND_CLUSTER` env like + utils.LoadImageToKindClusterWithName). + 5. OTLP preflight: curl pod POSTing `{}` to `/v1/traces`, + assert HTTP 200 in its logs (fail message names the endpoint). + 6. `kubectl set env deployment/egress-proxies-operator-controller-manager + -n OTEL_EXPORTER_OTLP_ENDPOINT= + "OTEL_RESOURCE_ATTRIBUTES=k8s.pod.name=$(POD_NAME),k8s.namespace.name=$(POD_NAMESPACE),test.run.id="` + — in-place update keeps the var after the downward-API vars ($(VAR) + expansion needs that ordering); `exec.Command` means no shell mangling + of `$()`. Then `kubectl rollout status`. + 7. `Eventually`: newest controller pod (sorted by creationTimestamp — a + terminating old pod may otherwise be picked) logs contain + `"tracing enabled"`. +- **It "creates proxies and reports reconcile traces":** + - Write 2 Proxy manifests (names `proxy-tracing-e2e-1/2`, + `provider: kubernetes`, modeled on config/samples/proxy_kubernetes.yaml) + to `GinkgoT().TempDir()` (absolute paths — `utils.Run` chdirs the + process); `kubectl apply -n default`. + - `Eventually` (5m — first squid start) `.status.phase == "Ready"` + (confirmed field, api/v1alpha1/proxy_types.go:41). + - `Eventually`: Tempo search + `{resource.test.run.id="" && name="provider.create"}` ≥1 hit + (BSP flushes every ~5s; 2m default timeout is ample). + - Fetch that traceID via `/api/traces/`; assert span names include + `Reconcile Proxy`, `reconcile.managed`, `provider.create`, + `status.patch`, and resource attr `service.name=egress-proxies-operator`. +- **It "traces deletion":** + - `kubectl delete proxy -n default …`; `Eventually` CRs gone (finalizer + → `provider.delete`, DeletionPoll 10s). + - `Eventually`: Tempo search finds `provider.delete` for the runID; fetch + and assert `reconcile.delete` in the same trace. +- **AfterAll** (= the sketch's "shut down operator"): delete leftover + proxies in `default`; `make undeploy` (SIGTERM → `tracingShutdown` + flushes); `make uninstall`; delete operator ns — errors discarded + (`_, _ = utils.Run(...)`), mirroring e2e_test.go:79-95. +- **Tempo client helpers** (same file, stdlib only — the test process runs + on the host, which reaches Tempo directly): + - `tempoSearch(traceql string)` → `GET {TEMPO_URL}/api/search?q=…&start=…&end=…` + (Unix seconds, window = suite start − 5m → now); minimal struct + `{Traces []struct{ TraceID string }}`. + - `tempoTrace(id string)` → `/api/traces/`, decoded as **OTLP-JSON** + (`batches[].scopeSpans[].spans[].name`, resource attrs as + `{key, value:{stringValue}}`) — not Jaeger's shape. Helper flattens to + a span-name set + resource attr map. + +### Step 2 — Makefile + +- `test-e2e`: pass `TEMPO_URL`/`OTLP_ENDPOINT` through to `go test` env + (no defaults) and add `-timeout 30m` — the suite already runs + docker-build + kind-load in BeforeSuite plus 3–5m Eventuallys, and this + adds a second full deploy cycle + image pulls; the 10m default will be + exceeded on cold caches. + +### Step 3 — Docs + +- `docs/testing.md` (§ e2e, currently "scaffold… only asserts manager runs + and serves metrics"): document the tracing spec, its Skip gate, and the + invocation: + `TEMPO_URL=http://192.168.0.30:3200 OTLP_ENDPOINT=http://192.168.0.30:4318 make test-e2e`. +- Execution-log entries per step in `docs/plans-executions/2026-08-24-1224-tracing-e2e.md`. +- CHANGELOG entry only after the user confirms a green run (house rule). + +## Not doing (and why) + +- No dedicated flush assertion on shutdown — teardown exercises the flush + path, but attributing a specific late span to it is guesswork. +- No gRPC (4317) variant — http/protobuf is the operator default and the + verified path; a variant run is a one-env-var change if ever wanted. +- No changes to `internal/` — the whole test works through public surface + (env vars, kubectl, Tempo API), which is the point of an e2e test. + +## Verification + +1. `TEMPO_URL=http://192.168.0.30:3200 OTLP_ENDPOINT=http://192.168.0.30:4318 make test-e2e` + → tracing spec passes (both Its); without the env vars → spec reports + Skipped, remainder of suite unaffected. +2. In Grafana: TraceQL `{resource.test.run.id=""}` + shows the run's traces with the expected tree. +3. `go vet -tags=e2e ./...` clean (documented routine). +4. A deliberately wrong `OTLP_ENDPOINT` (e.g. port 9) fails fast in the + preflight step with a clear message, not a 2-minute search timeout.