# Execution log: e2e test — OTel tracing against real Tempo

Plan: `docs/plans/2026-08-24-1224-tracing-e2e.md`

- [x] Step 1 — `test/e2e/tracing_test.go`
- [x] Step 2 — Makefile
- [x] Step 3 — Docs

## Steps 1–3 — spec, Makefile, docs (one commit)

The three steps landed together — the spec is one new file and the other
two are its wiring. `Describe("OTel tracing", Ordered)` is fully
self-contained (own ns create → `make install`/`make deploy` → teardown)
because Ginkgo randomizes top-level container order, so it cannot share the
Manager Describe's deployment. It reuses the package-level `namespace` /
`managerImage` and the suite's idioms (`utils.Run`, curl-pod with the
restricted-PSS overrides JSON, log-substring `Eventually`s).

Judgment calls beyond the plan's letter:

- The squid pre-pull (`docker pull` + `kind load`) is **best effort** — a
  missing docker binary logs a note and continues rather than failing the
  spec; the 5m Ready timeout still covers an in-cluster pull.
- The OTLP preflight pod prints per-attempt HTTP codes and a final
  `OTLP_OK`/`OTLP_UNREACHABLE` marker; the assertion quotes the pod's
  output, so an unreachable endpoint names itself in the failure.
- `kubectl set env` is passed the literal
  `OTEL_RESOURCE_ATTRIBUTES=...$(POD_NAME)...` string via `exec.Command` —
  no shell involved, kubectl stores `$()` verbatim, and the in-place update
  keeps the var after the downward-API vars it references.
- Tempo helpers are stdlib-only; `/api/traces/<id>` is decoded as
  OTLP-JSON (`batches[].scopeSpans[].spans[].name`), which is Tempo's
  actual shape (not Jaeger's).

Verified so far: `go vet -tags=e2e ./...` clean.

## Live run against homelab Tempo

First attempt failed before the suite started — Docker Desktop wasn't
running (`kind` could not create the cluster), and the failure was masked
to exit 0 by a `| tail` pipe on the make invocation (no pipefail in that
shell). Rerun with Docker started first and no pipe:

```bash
TEMPO_URL=http://192.168.0.30:3200 OTLP_ENDPOINT=http://192.168.0.30:4318 make test-e2e
```

Result: **SUCCESS — 4/4 specs (Manager smoke + both tracing Its), 219 s**,
kind cluster auto-deleted. Run id `e2e-1787567624494810000`; the traces are
findable in Grafana with TraceQL
`{resource.test.run.id="e2e-1787567624494810000"}`. Both proxies reached
`Ready` on real squid pods; the create trace carried
`Reconcile Proxy` / `reconcile.managed` / `provider.create` /
`status.patch` and the expected resource attrs; the deletion trace carried
`reconcile.delete` / `provider.delete`.

Worth noting: the OTLP preflight and both Tempo polls succeeded on the
in-cluster → LAN path (kind on macOS reaches 192.168.0.30 through Docker
Desktop's NAT), so no extra networking setup is needed on this machine.

The plan's negative verification also ran: with a deliberately wrong
endpoint (`OTLP_ENDPOINT=http://192.168.0.30:9999`) the tracing spec
failed in the BeforeAll preflight after ~52 s with
`OTLP endpoint http://192.168.0.30:9999 is not reachable from inside the
kind cluster; curl output: OTLP_UNREACHABLE` — a named, fast failure
instead of a 2-minute opaque search timeout — and teardown still deleted
the kind cluster (`make cleanup-test-e2e` run explicitly, since a failing
`go test` skips the Makefile's cleanup step).