egress-proxies-operator/docs/plans/2026-08-24-1224-tracing-e2e.md

# Plan: e2e test — OTel tracing against real Tempo
**Created:** 2026-08-24 12:24

## Context

The tracing feature (MR #4, branch `feat/otel-tracing`) is covered by unit
tests with an in-memory span recorder, but nothing proves the full pipeline:
operator in a real cluster → OTLP export → Tempo ingest → queryable traces
with the documented span topology. The homelab Tempo lives at
`http://192.168.0.30:3200` (query API — verified live from this host:
`/api/echo`, TraceQL `GET /api/search?q=…`, `GET /api/traces/<id>`), with
OTLP ingest on `:4318` (HTTP, verified 200) and `:4317` (gRPC, open). User
decisions: **Go Ginkgo e2e test** (not a shell script), **structural
assertions** (span tree, not just trace-exists), flow per the user's sketch:
deploy operator in kind with tracing on → create kubernetes-provider proxies
→ delete them → shut down — with traces tagged for easy discovery per test
run. This work continues on `feat/otel-tracing`; MR #4 stays the vehicle.

## Design decisions (from review, with evidence)

- **Proxy CRs go in the `default` namespace, not the operator namespace.**
  The squid pods the kubernetes provider creates carry no securityContext
  (internal/provider/kubernetes/pod.go:33-49) and are created in the CR's
  namespace; the operator ns is labeled `pod-security enforce=restricted`
  (test/e2e/e2e_test.go:61-62) and would reject them. Pod RBAC is
  cluster-scoped (config/rbac/role.yaml:10), so `default` works. All
  kubectl calls use explicit `-n`.
- **Anchor assertions on `provider.create`, not the root-span search.** A
  `name="Reconcile Proxy"` hit may be the finalizer-add or a drift
  reconcile (no provider call inside). TraceQL matches span names anywhere
  in a trace, so search `{resource.test.run.id="<runID>" && name="provider.create"}`,
  fetch *that* traceID, then assert companion span names.
- **Run identification with zero code changes:** OTel resource attribute
  `test.run.id=<runID>` via `OTEL_RESOURCE_ATTRIBUTES` (env wins —
  `resource.WithFromEnv()` merges last, internal/tracing/tracing.go). Tempo
  vParquet searches arbitrary resource attrs without config; fallback query
  documented: `{resource.service.name="egress-proxies-operator" && name="provider.create"}`
  + start/end window (Unix **seconds**).
- **OTLP preflight from inside the cluster:** host-reachability of
  192.168.0.30 doesn't prove pod-reachability from kind on macOS, and
  export failures log only at V(1) (invisible) — without a preflight, a
  broken path is a 2-minute opaque timeout. Reuse the suite's curl-pod
  pattern (e2e_test.go:218-248, incl. the restricted-PSS overrides JSON)
  to POST to `<OTLP_ENDPOINT>/v1/traces` and fail fast.
- **No homelab defaults baked into the Makefile.** The spec Skips unless
  `TEMPO_URL`/`OTLP_ENDPOINT` are set (otherwise everyone without that LAN
  host inherits a 2-minute failure). The copy-pasteable invocation lives in
  docs/testing.md.
- **Ginkgo ordering:** top-level container order is randomized, so the
  tracing Describe is fully self-contained (own ns create → make install →
  make deploy → teardown), same shape as the Manager Describe. `kubectl
  delete ns` blocks until termination, so no create/delete race between
  the two Describes.

## Steps

### Step 1 — `test/e2e/tracing_test.go` (new; `//go:build e2e`, package e2e)

`Describe("OTel tracing", Ordered)`:

- **BeforeAll:**
  1. Read `TEMPO_URL` + `OTLP_ENDPOINT`; `Skip("TEMPO_URL/OTLP_ENDPOINT not set")` when empty.
  2. `runID := "e2e-" + strconv.FormatInt(time.Now().UnixNano(), 10)`;
     print it to `GinkgoWriter` so the run is findable in Grafana by hand.
  3. Create operator ns + restricted-PSS label; `make install`;
     `make deploy IMG=<managerImage>` (same commands as e2e_test.go:54-75,
     via `utils.Run`).
  4. Pre-pull the proxy image to kill the biggest flake source:
     `docker pull ubuntu/squid:6.6-24.04_edge` + `kind load docker-image`
     (via `utils.Run`, honoring `KIND`/`KIND_CLUSTER` env like
     utils.LoadImageToKindClusterWithName).
  5. OTLP preflight: curl pod POSTing `{}` to `<OTLP_ENDPOINT>/v1/traces`,
     assert HTTP 200 in its logs (fail message names the endpoint).
  6. `kubectl set env deployment/egress-proxies-operator-controller-manager
     -n <ns> OTEL_EXPORTER_OTLP_ENDPOINT=<OTLP_ENDPOINT>
     "OTEL_RESOURCE_ATTRIBUTES=k8s.pod.name=$(POD_NAME),k8s.namespace.name=$(POD_NAMESPACE),test.run.id=<runID>"`
     — in-place update keeps the var after the downward-API vars ($(VAR)
     expansion needs that ordering); `exec.Command` means no shell mangling
     of `$()`. Then `kubectl rollout status`.
  7. `Eventually`: newest controller pod (sorted by creationTimestamp — a
     terminating old pod may otherwise be picked) logs contain
     `"tracing enabled"`.
- **It "creates proxies and reports reconcile traces":**
  - Write 2 Proxy manifests (names `proxy-tracing-e2e-1/2`,
    `provider: kubernetes`, modeled on config/samples/proxy_kubernetes.yaml)
    to `GinkgoT().TempDir()` (absolute paths — `utils.Run` chdirs the
    process); `kubectl apply -n default`.
  - `Eventually` (5m — first squid start) `.status.phase == "Ready"`
    (confirmed field, api/v1alpha1/proxy_types.go:41).
  - `Eventually`: Tempo search
    `{resource.test.run.id="<runID>" && name="provider.create"}` ≥1 hit
    (BSP flushes every ~5s; 2m default timeout is ample).
  - Fetch that traceID via `/api/traces/<id>`; assert span names include
    `Reconcile Proxy`, `reconcile.managed`, `provider.create`,
    `status.patch`, and resource attr `service.name=egress-proxies-operator`.
- **It "traces deletion":**
  - `kubectl delete proxy -n default …`; `Eventually` CRs gone (finalizer
    → `provider.delete`, DeletionPoll 10s).
  - `Eventually`: Tempo search finds `provider.delete` for the runID; fetch
    and assert `reconcile.delete` in the same trace.
- **AfterAll** (= the sketch's "shut down operator"): delete leftover
  proxies in `default`; `make undeploy` (SIGTERM → `tracingShutdown`
  flushes); `make uninstall`; delete operator ns — errors discarded
  (`_, _ = utils.Run(...)`), mirroring e2e_test.go:79-95.
- **Tempo client helpers** (same file, stdlib only — the test process runs
  on the host, which reaches Tempo directly):
  - `tempoSearch(traceql string)` → `GET {TEMPO_URL}/api/search?q=…&start=…&end=…`
    (Unix seconds, window = suite start − 5m → now); minimal struct
    `{Traces []struct{ TraceID string }}`.
  - `tempoTrace(id string)` → `/api/traces/<id>`, decoded as **OTLP-JSON**
    (`batches[].scopeSpans[].spans[].name`, resource attrs as
    `{key, value:{stringValue}}`) — not Jaeger's shape. Helper flattens to
    a span-name set + resource attr map.

### Step 2 — Makefile

- `test-e2e`: pass `TEMPO_URL`/`OTLP_ENDPOINT` through to `go test` env
  (no defaults) and add `-timeout 30m` — the suite already runs
  docker-build + kind-load in BeforeSuite plus 3–5m Eventuallys, and this
  adds a second full deploy cycle + image pulls; the 10m default will be
  exceeded on cold caches.

### Step 3 — Docs

- `docs/testing.md` (§ e2e, currently "scaffold… only asserts manager runs
  and serves metrics"): document the tracing spec, its Skip gate, and the
  invocation:
  `TEMPO_URL=http://192.168.0.30:3200 OTLP_ENDPOINT=http://192.168.0.30:4318 make test-e2e`.
- Execution-log entries per step in `docs/plans-executions/2026-08-24-1224-tracing-e2e.md`.
- CHANGELOG entry only after the user confirms a green run (house rule).

## Not doing (and why)

- No dedicated flush assertion on shutdown — teardown exercises the flush
  path, but attributing a specific late span to it is guesswork.
- No gRPC (4317) variant — http/protobuf is the operator default and the
  verified path; a variant run is a one-env-var change if ever wanted.
- No changes to `internal/` — the whole test works through public surface
  (env vars, kubectl, Tempo API), which is the point of an e2e test.

## Verification

1. `TEMPO_URL=http://192.168.0.30:3200 OTLP_ENDPOINT=http://192.168.0.30:4318 make test-e2e`
   → tracing spec passes (both Its); without the env vars → spec reports
   Skipped, remainder of suite unaffected.
2. In Grafana: TraceQL `{resource.test.run.id="<runID printed in test log>"}`
   shows the run's traces with the expected tree.
3. `go vet -tags=e2e ./...` clean (documented routine).
4. A deliberately wrong `OTLP_ENDPOINT` (e.g. port 9) fails fast in the
   preflight step with a clear message, not a 2-minute search timeout.