egress-proxies-operator/docs/plans-executions/2026-08-24-1025-otel-tracing.md

Execution log: OpenTelemetry tracing integrated with logr/zap logging

Plan: docs/plans/2026-08-24-1025-otel-tracing.md

• Step 1 — Dependencies
• Step 2 — New package internal/tracing
• Step 3 — provider.WithTracing decorator
• Step 4 — cmd/main.go wiring
• Step 5 — Reconciler spans
• Step 6 — Discovery server
• Step 7 — GC + health
• Step 8 — GCP wire-log enrichment
• Step 9 — Manifests + docs

Step 1 — Dependencies

Aligned the pre-existing indirect skew (otel core v1.44.0 vs otlptrace exporters v1.40.0) and added the new direct deps in one shot:

go get go.opentelemetry.io/otel@v1.45.0 \
  go.opentelemetry.io/otel/sdk@v1.45.0 \
  go.opentelemetry.io/otel/trace@v1.45.0 \
  go.opentelemetry.io/otel/exporters/otlp/otlptrace/otlptracegrpc@v1.45.0 \
  go.opentelemetry.io/otel/exporters/otlp/otlptrace/otlptracehttp@v1.45.0 \
  go.opentelemetry.io/otel/exporters/stdout/stdouttrace@v1.45.0 \
  go.opentelemetry.io/contrib/instrumentation/net/http/otelhttp@v0.70.0
go mod tidy

MVS side-effects worth recording: logr v1.4.3→v1.4.4, httpsnoop v1.0.4→v1.1.0, grpc-gateway/v2 v2.27.7→v2.29.0, proto/otlp v1.9.0→v1.11.0, plus a genproto/googleapis/api pseudo-version bump. Full make test passed against the bumped graph (google.golang.org/api v0.292.0 and k8s v0.36 tolerate otelhttp v0.70.0).

Worth noting: the first go mod tidy ran before any first-party code imported otlptracehttp/stdouttrace, so it silently dropped those two modules again; the Step 2 tidy re-added them. The plan's semconv question resolved to semconv/v1.43.0 — that's what sdk@v1.45.0/resource/builtin.go imports, so first-party code uses the same version to avoid ErrSchemaURLConflict in the common path.

Step 2 — internal/tracing package

Landed as planned: tracing.go (env-gated Setup, hand-rolled exporter selection), logger.go (Start/StartSpan/ContextWithLogger with the base-logger ctx key that prevents duplicate traceID zap fields on nested spans), reconciler.go, transport.go, http.go, options.go; tests for all of it (first use of sdk/trace/tracetest in the repo).

Two deviations from the plan's letter:

• k8s transport gating is a custom RoundTripper, not otelhttp.WithFilter (parentGatedTransport): requests without a parent span bypass the otel transport entirely, so the no-root-spans-from-informers guarantee doesn't depend on otelhttp filter semantics for transports.
• HTTPMiddleware must copy the route pattern back. The logger-injecting inner handler wraps the request via WithContext (a shallow copy), so the mux records the matched pattern on the copy while otelhttp's post-routing span rename reads the original. Found by the middleware test (span named "GET" instead of "GET /v1/things/{id}"); fixed with r.Pattern = r2.Pattern after next.ServeHTTP.

Also: both the transport and the middleware pass explicit W3C propagators instead of relying on the global, so behavior is deterministic under tests and when tracing is disabled. Setup tests reset the global provider to a fresh noop per case — restoring otel's own default delegate triggers a "Setting tracer provider to its current value" warning from the SDK.

Step 3 — provider.WithTracing

internal/provider/tracing.go mirrors metrics.go exactly (same wrap shape, same resultLabel classification reused as the provider.result span attribute); tests reuse staticProvider from metrics_test.go. ErrNotFound maps to span status Ok as planned — the reconciler polls Get to NotFound during replacement/deletion, so it's an answer, not a failure.

One small API addition to internal/tracing for this: exported tracing.Tracer(opts...), since the decorator lives in package provider and couldn't reach the unexported option resolver. The cmd/main.go wiring (WithTracing outermost around WithMetrics) lands with Step 4's commit — same file, one commit.

Step 4 — cmd/main.go wiring

As planned: tracing.Setup right after ctrl.SetLogger; manager rest.Config wrapped via restCfg := ctrl.GetConfigOrDie(); restCfg.Wrap(...) (the previous inline call left nowhere to wrap); providers decorated tracing- outermost; explicit trace flush after mgr.Start returns on both the error and clean paths, with a fresh 10s context since the signal ctx is already cancelled by then.

The kubernetes provider grew NewWithTransportWrapper(ctx, cfg, wrap) (its client is built from its own ctrl.GetConfig(), invisible to the manager's wrapped config); New now delegates with a nil wrapper, so newWithClient tests stayed untouched, and main registers a closure — the same pattern the gcp constructor already used for wire-log options.

Deviation: the plan put --trace-health-probes here, but the flag needs the health.Engine.TraceProbes field that Step 7 introduces — moved there to keep every commit compiling.