69 lines
3.3 KiB
Markdown
69 lines
3.3 KiB
Markdown
# Execution log: e2e test — OTel tracing against real Tempo
|
||
|
||
Plan: `docs/plans/2026-08-24-1224-tracing-e2e.md`
|
||
|
||
- [x] Step 1 — `test/e2e/tracing_test.go`
|
||
- [x] Step 2 — Makefile
|
||
- [x] Step 3 — Docs
|
||
|
||
## Steps 1–3 — spec, Makefile, docs (one commit)
|
||
|
||
The three steps landed together — the spec is one new file and the other
|
||
two are its wiring. `Describe("OTel tracing", Ordered)` is fully
|
||
self-contained (own ns create → `make install`/`make deploy` → teardown)
|
||
because Ginkgo randomizes top-level container order, so it cannot share the
|
||
Manager Describe's deployment. It reuses the package-level `namespace` /
|
||
`managerImage` and the suite's idioms (`utils.Run`, curl-pod with the
|
||
restricted-PSS overrides JSON, log-substring `Eventually`s).
|
||
|
||
Judgment calls beyond the plan's letter:
|
||
|
||
- The squid pre-pull (`docker pull` + `kind load`) is **best effort** — a
|
||
missing docker binary logs a note and continues rather than failing the
|
||
spec; the 5m Ready timeout still covers an in-cluster pull.
|
||
- The OTLP preflight pod prints per-attempt HTTP codes and a final
|
||
`OTLP_OK`/`OTLP_UNREACHABLE` marker; the assertion quotes the pod's
|
||
output, so an unreachable endpoint names itself in the failure.
|
||
- `kubectl set env` is passed the literal
|
||
`OTEL_RESOURCE_ATTRIBUTES=...$(POD_NAME)...` string via `exec.Command` —
|
||
no shell involved, kubectl stores `$()` verbatim, and the in-place update
|
||
keeps the var after the downward-API vars it references.
|
||
- Tempo helpers are stdlib-only; `/api/traces/<id>` is decoded as
|
||
OTLP-JSON (`batches[].scopeSpans[].spans[].name`), which is Tempo's
|
||
actual shape (not Jaeger's).
|
||
|
||
Verified so far: `go vet -tags=e2e ./...` clean.
|
||
|
||
## Live run against homelab Tempo
|
||
|
||
First attempt failed before the suite started — Docker Desktop wasn't
|
||
running (`kind` could not create the cluster), and the failure was masked
|
||
to exit 0 by a `| tail` pipe on the make invocation (no pipefail in that
|
||
shell). Rerun with Docker started first and no pipe:
|
||
|
||
```bash
|
||
TEMPO_URL=http://192.168.0.30:3200 OTLP_ENDPOINT=http://192.168.0.30:4318 make test-e2e
|
||
```
|
||
|
||
Result: **SUCCESS — 4/4 specs (Manager smoke + both tracing Its), 219 s**,
|
||
kind cluster auto-deleted. Run id `e2e-1787567624494810000`; the traces are
|
||
findable in Grafana with TraceQL
|
||
`{resource.test.run.id="e2e-1787567624494810000"}`. Both proxies reached
|
||
`Ready` on real squid pods; the create trace carried
|
||
`Reconcile Proxy` / `reconcile.managed` / `provider.create` /
|
||
`status.patch` and the expected resource attrs; the deletion trace carried
|
||
`reconcile.delete` / `provider.delete`.
|
||
|
||
Worth noting: the OTLP preflight and both Tempo polls succeeded on the
|
||
in-cluster → LAN path (kind on macOS reaches 192.168.0.30 through Docker
|
||
Desktop's NAT), so no extra networking setup is needed on this machine.
|
||
|
||
The plan's negative verification also ran: with a deliberately wrong
|
||
endpoint (`OTLP_ENDPOINT=http://192.168.0.30:9999`) the tracing spec
|
||
failed in the BeforeAll preflight after ~52 s with
|
||
`OTLP endpoint http://192.168.0.30:9999 is not reachable from inside the
|
||
kind cluster; curl output: OTLP_UNREACHABLE` — a named, fast failure
|
||
instead of a 2-minute opaque search timeout — and teardown still deleted
|
||
the kind cluster (`make cleanup-test-e2e` run explicitly, since a failing
|
||
`go test` skips the Makefile's cleanup step).
|