egress-proxies-operator/docs/plans-executions/2026-08-24-1025-otel-tracing.md

Execution log: OpenTelemetry tracing integrated with logr/zap logging

Plan: docs/plans/2026-08-24-1025-otel-tracing.md

• Step 1 — Dependencies
• Step 2 — New package internal/tracing
• Step 3 — provider.WithTracing decorator
• Step 4 — cmd/main.go wiring
• Step 5 — Reconciler spans
• Step 6 — Discovery server
• Step 7 — GC + health
• Step 8 — GCP wire-log enrichment
• Step 9 — Manifests + docs

Step 1 — Dependencies

Aligned the pre-existing indirect skew (otel core v1.44.0 vs otlptrace exporters v1.40.0) and added the new direct deps in one shot:

go get go.opentelemetry.io/otel@v1.45.0 \
  go.opentelemetry.io/otel/sdk@v1.45.0 \
  go.opentelemetry.io/otel/trace@v1.45.0 \
  go.opentelemetry.io/otel/exporters/otlp/otlptrace/otlptracegrpc@v1.45.0 \
  go.opentelemetry.io/otel/exporters/otlp/otlptrace/otlptracehttp@v1.45.0 \
  go.opentelemetry.io/otel/exporters/stdout/stdouttrace@v1.45.0 \
  go.opentelemetry.io/contrib/instrumentation/net/http/otelhttp@v0.70.0
go mod tidy

MVS side-effects worth recording: logr v1.4.3→v1.4.4, httpsnoop v1.0.4→v1.1.0, grpc-gateway/v2 v2.27.7→v2.29.0, proto/otlp v1.9.0→v1.11.0, plus a genproto/googleapis/api pseudo-version bump. Full make test passed against the bumped graph (google.golang.org/api v0.292.0 and k8s v0.36 tolerate otelhttp v0.70.0).

Worth noting: the first go mod tidy ran before any first-party code imported otlptracehttp/stdouttrace, so it silently dropped those two modules again; the Step 2 tidy re-added them. The plan's semconv question resolved to semconv/v1.43.0 — that's what sdk@v1.45.0/resource/builtin.go imports, so first-party code uses the same version to avoid ErrSchemaURLConflict in the common path.

Step 2 — internal/tracing package

Landed as planned: tracing.go (env-gated Setup, hand-rolled exporter selection), logger.go (Start/StartSpan/ContextWithLogger with the base-logger ctx key that prevents duplicate traceID zap fields on nested spans), reconciler.go, transport.go, http.go, options.go; tests for all of it (first use of sdk/trace/tracetest in the repo).

Two deviations from the plan's letter:

• k8s transport gating is a custom RoundTripper, not otelhttp.WithFilter (parentGatedTransport): requests without a parent span bypass the otel transport entirely, so the no-root-spans-from-informers guarantee doesn't depend on otelhttp filter semantics for transports.
• HTTPMiddleware must copy the route pattern back. The logger-injecting inner handler wraps the request via WithContext (a shallow copy), so the mux records the matched pattern on the copy while otelhttp's post-routing span rename reads the original. Found by the middleware test (span named "GET" instead of "GET /v1/things/{id}"); fixed with r.Pattern = r2.Pattern after next.ServeHTTP.

Also: both the transport and the middleware pass explicit W3C propagators instead of relying on the global, so behavior is deterministic under tests and when tracing is disabled. Setup tests reset the global provider to a fresh noop per case — restoring otel's own default delegate triggers a "Setting tracer provider to its current value" warning from the SDK.