OpenTelemetry tracing integrated with logr/zap logging #4
150
docs/plans/2026-08-24-1224-tracing-e2e.md
Normal file
150
docs/plans/2026-08-24-1224-tracing-e2e.md
Normal file
@@ -0,0 +1,150 @@
|
|||||||
|
# Plan: e2e test — OTel tracing against real Tempo
|
||||||
|
**Created:** 2026-08-24 12:24
|
||||||
|
|
||||||
|
## Context
|
||||||
|
|
||||||
|
The tracing feature (MR #4, branch `feat/otel-tracing`) is covered by unit
|
||||||
|
tests with an in-memory span recorder, but nothing proves the full pipeline:
|
||||||
|
operator in a real cluster → OTLP export → Tempo ingest → queryable traces
|
||||||
|
with the documented span topology. The homelab Tempo lives at
|
||||||
|
`http://192.168.0.30:3200` (query API — verified live from this host:
|
||||||
|
`/api/echo`, TraceQL `GET /api/search?q=…`, `GET /api/traces/<id>`), with
|
||||||
|
OTLP ingest on `:4318` (HTTP, verified 200) and `:4317` (gRPC, open). User
|
||||||
|
decisions: **Go Ginkgo e2e test** (not a shell script), **structural
|
||||||
|
assertions** (span tree, not just trace-exists), flow per the user's sketch:
|
||||||
|
deploy operator in kind with tracing on → create kubernetes-provider proxies
|
||||||
|
→ delete them → shut down — with traces tagged for easy discovery per test
|
||||||
|
run. This work continues on `feat/otel-tracing`; MR #4 stays the vehicle.
|
||||||
|
|
||||||
|
## Design decisions (from review, with evidence)
|
||||||
|
|
||||||
|
- **Proxy CRs go in the `default` namespace, not the operator namespace.**
|
||||||
|
The squid pods the kubernetes provider creates carry no securityContext
|
||||||
|
(internal/provider/kubernetes/pod.go:33-49) and are created in the CR's
|
||||||
|
namespace; the operator ns is labeled `pod-security enforce=restricted`
|
||||||
|
(test/e2e/e2e_test.go:61-62) and would reject them. Pod RBAC is
|
||||||
|
cluster-scoped (config/rbac/role.yaml:10), so `default` works. All
|
||||||
|
kubectl calls use explicit `-n`.
|
||||||
|
- **Anchor assertions on `provider.create`, not the root-span search.** A
|
||||||
|
`name="Reconcile Proxy"` hit may be the finalizer-add or a drift
|
||||||
|
reconcile (no provider call inside). TraceQL matches span names anywhere
|
||||||
|
in a trace, so search `{resource.test.run.id="<runID>" && name="provider.create"}`,
|
||||||
|
fetch *that* traceID, then assert companion span names.
|
||||||
|
- **Run identification with zero code changes:** OTel resource attribute
|
||||||
|
`test.run.id=<runID>` via `OTEL_RESOURCE_ATTRIBUTES` (env wins —
|
||||||
|
`resource.WithFromEnv()` merges last, internal/tracing/tracing.go). Tempo
|
||||||
|
vParquet searches arbitrary resource attrs without config; fallback query
|
||||||
|
documented: `{resource.service.name="egress-proxies-operator" && name="provider.create"}`
|
||||||
|
+ start/end window (Unix **seconds**).
|
||||||
|
- **OTLP preflight from inside the cluster:** host-reachability of
|
||||||
|
192.168.0.30 doesn't prove pod-reachability from kind on macOS, and
|
||||||
|
export failures log only at V(1) (invisible) — without a preflight, a
|
||||||
|
broken path is a 2-minute opaque timeout. Reuse the suite's curl-pod
|
||||||
|
pattern (e2e_test.go:218-248, incl. the restricted-PSS overrides JSON)
|
||||||
|
to POST to `<OTLP_ENDPOINT>/v1/traces` and fail fast.
|
||||||
|
- **No homelab defaults baked into the Makefile.** The spec Skips unless
|
||||||
|
`TEMPO_URL`/`OTLP_ENDPOINT` are set (otherwise everyone without that LAN
|
||||||
|
host inherits a 2-minute failure). The copy-pasteable invocation lives in
|
||||||
|
docs/testing.md.
|
||||||
|
- **Ginkgo ordering:** top-level container order is randomized, so the
|
||||||
|
tracing Describe is fully self-contained (own ns create → make install →
|
||||||
|
make deploy → teardown), same shape as the Manager Describe. `kubectl
|
||||||
|
delete ns` blocks until termination, so no create/delete race between
|
||||||
|
the two Describes.
|
||||||
|
|
||||||
|
## Steps
|
||||||
|
|
||||||
|
### Step 1 — `test/e2e/tracing_test.go` (new; `//go:build e2e`, package e2e)
|
||||||
|
|
||||||
|
`Describe("OTel tracing", Ordered)`:
|
||||||
|
|
||||||
|
- **BeforeAll:**
|
||||||
|
1. Read `TEMPO_URL` + `OTLP_ENDPOINT`; `Skip("TEMPO_URL/OTLP_ENDPOINT not set")` when empty.
|
||||||
|
2. `runID := "e2e-" + strconv.FormatInt(time.Now().UnixNano(), 10)`;
|
||||||
|
print it to `GinkgoWriter` so the run is findable in Grafana by hand.
|
||||||
|
3. Create operator ns + restricted-PSS label; `make install`;
|
||||||
|
`make deploy IMG=<managerImage>` (same commands as e2e_test.go:54-75,
|
||||||
|
via `utils.Run`).
|
||||||
|
4. Pre-pull the proxy image to kill the biggest flake source:
|
||||||
|
`docker pull ubuntu/squid:6.6-24.04_edge` + `kind load docker-image`
|
||||||
|
(via `utils.Run`, honoring `KIND`/`KIND_CLUSTER` env like
|
||||||
|
utils.LoadImageToKindClusterWithName).
|
||||||
|
5. OTLP preflight: curl pod POSTing `{}` to `<OTLP_ENDPOINT>/v1/traces`,
|
||||||
|
assert HTTP 200 in its logs (fail message names the endpoint).
|
||||||
|
6. `kubectl set env deployment/egress-proxies-operator-controller-manager
|
||||||
|
-n <ns> OTEL_EXPORTER_OTLP_ENDPOINT=<OTLP_ENDPOINT>
|
||||||
|
"OTEL_RESOURCE_ATTRIBUTES=k8s.pod.name=$(POD_NAME),k8s.namespace.name=$(POD_NAMESPACE),test.run.id=<runID>"`
|
||||||
|
— in-place update keeps the var after the downward-API vars ($(VAR)
|
||||||
|
expansion needs that ordering); `exec.Command` means no shell mangling
|
||||||
|
of `$()`. Then `kubectl rollout status`.
|
||||||
|
7. `Eventually`: newest controller pod (sorted by creationTimestamp — a
|
||||||
|
terminating old pod may otherwise be picked) logs contain
|
||||||
|
`"tracing enabled"`.
|
||||||
|
- **It "creates proxies and reports reconcile traces":**
|
||||||
|
- Write 2 Proxy manifests (names `proxy-tracing-e2e-1/2`,
|
||||||
|
`provider: kubernetes`, modeled on config/samples/proxy_kubernetes.yaml)
|
||||||
|
to `GinkgoT().TempDir()` (absolute paths — `utils.Run` chdirs the
|
||||||
|
process); `kubectl apply -n default`.
|
||||||
|
- `Eventually` (5m — first squid start) `.status.phase == "Ready"`
|
||||||
|
(confirmed field, api/v1alpha1/proxy_types.go:41).
|
||||||
|
- `Eventually`: Tempo search
|
||||||
|
`{resource.test.run.id="<runID>" && name="provider.create"}` ≥1 hit
|
||||||
|
(BSP flushes every ~5s; 2m default timeout is ample).
|
||||||
|
- Fetch that traceID via `/api/traces/<id>`; assert span names include
|
||||||
|
`Reconcile Proxy`, `reconcile.managed`, `provider.create`,
|
||||||
|
`status.patch`, and resource attr `service.name=egress-proxies-operator`.
|
||||||
|
- **It "traces deletion":**
|
||||||
|
- `kubectl delete proxy -n default …`; `Eventually` CRs gone (finalizer
|
||||||
|
→ `provider.delete`, DeletionPoll 10s).
|
||||||
|
- `Eventually`: Tempo search finds `provider.delete` for the runID; fetch
|
||||||
|
and assert `reconcile.delete` in the same trace.
|
||||||
|
- **AfterAll** (= the sketch's "shut down operator"): delete leftover
|
||||||
|
proxies in `default`; `make undeploy` (SIGTERM → `tracingShutdown`
|
||||||
|
flushes); `make uninstall`; delete operator ns — errors discarded
|
||||||
|
(`_, _ = utils.Run(...)`), mirroring e2e_test.go:79-95.
|
||||||
|
- **Tempo client helpers** (same file, stdlib only — the test process runs
|
||||||
|
on the host, which reaches Tempo directly):
|
||||||
|
- `tempoSearch(traceql string)` → `GET {TEMPO_URL}/api/search?q=…&start=…&end=…`
|
||||||
|
(Unix seconds, window = suite start − 5m → now); minimal struct
|
||||||
|
`{Traces []struct{ TraceID string }}`.
|
||||||
|
- `tempoTrace(id string)` → `/api/traces/<id>`, decoded as **OTLP-JSON**
|
||||||
|
(`batches[].scopeSpans[].spans[].name`, resource attrs as
|
||||||
|
`{key, value:{stringValue}}`) — not Jaeger's shape. Helper flattens to
|
||||||
|
a span-name set + resource attr map.
|
||||||
|
|
||||||
|
### Step 2 — Makefile
|
||||||
|
|
||||||
|
- `test-e2e`: pass `TEMPO_URL`/`OTLP_ENDPOINT` through to `go test` env
|
||||||
|
(no defaults) and add `-timeout 30m` — the suite already runs
|
||||||
|
docker-build + kind-load in BeforeSuite plus 3–5m Eventuallys, and this
|
||||||
|
adds a second full deploy cycle + image pulls; the 10m default will be
|
||||||
|
exceeded on cold caches.
|
||||||
|
|
||||||
|
### Step 3 — Docs
|
||||||
|
|
||||||
|
- `docs/testing.md` (§ e2e, currently "scaffold… only asserts manager runs
|
||||||
|
and serves metrics"): document the tracing spec, its Skip gate, and the
|
||||||
|
invocation:
|
||||||
|
`TEMPO_URL=http://192.168.0.30:3200 OTLP_ENDPOINT=http://192.168.0.30:4318 make test-e2e`.
|
||||||
|
- Execution-log entries per step in `docs/plans-executions/2026-08-24-1224-tracing-e2e.md`.
|
||||||
|
- CHANGELOG entry only after the user confirms a green run (house rule).
|
||||||
|
|
||||||
|
## Not doing (and why)
|
||||||
|
|
||||||
|
- No dedicated flush assertion on shutdown — teardown exercises the flush
|
||||||
|
path, but attributing a specific late span to it is guesswork.
|
||||||
|
- No gRPC (4317) variant — http/protobuf is the operator default and the
|
||||||
|
verified path; a variant run is a one-env-var change if ever wanted.
|
||||||
|
- No changes to `internal/` — the whole test works through public surface
|
||||||
|
(env vars, kubectl, Tempo API), which is the point of an e2e test.
|
||||||
|
|
||||||
|
## Verification
|
||||||
|
|
||||||
|
1. `TEMPO_URL=http://192.168.0.30:3200 OTLP_ENDPOINT=http://192.168.0.30:4318 make test-e2e`
|
||||||
|
→ tracing spec passes (both Its); without the env vars → spec reports
|
||||||
|
Skipped, remainder of suite unaffected.
|
||||||
|
2. In Grafana: TraceQL `{resource.test.run.id="<runID printed in test log>"}`
|
||||||
|
shows the run's traces with the expected tree.
|
||||||
|
3. `go vet -tags=e2e ./...` clean (documented routine).
|
||||||
|
4. A deliberately wrong `OTLP_ENDPOINT` (e.g. port 9) fails fast in the
|
||||||
|
preflight step with a clear message, not a 2-minute search timeout.
|
||||||
Reference in New Issue
Block a user