OpenTelemetry tracing integrated with logr/zap logging #4

kacerr · 2026-08-24T11:30:28+02:00

kacerr commented

2026-08-24 11:30:28 +02:00

Adds OTel tracing across the operator without disturbing the existing logging system.

Off by default: activates only when OTEL_EXPORTER_OTLP_ENDPOINT / OTEL_TRACES_EXPORTER is set; OTEL_SDK_DISABLED=true and OTEL_TRACES_EXPORTER=none force off. No collector -> no SDK, no-op spans, byte-identical logs.
Traces: one per reconcile (root span + managed/replace/delete/status.patch sub-spans), per discovery API request (route-named server spans, incoming traceparent honored), per GC sweep; per health probe behind --trace-health-probes (default off, ~1/s/proxy volume).
Child spans: provider calls via a WithTracing decorator (mirrors WithMetrics, reuses the error taxonomy), k8s API calls via a parent-gated otelhttp rest.Config transport (informer watches/leader election never become root spans), GCP HTTP via the SDK's built-in otelhttp transport.
Log correlation: every log line inside a traced operation carries traceID/spanID (lowerCamel, matching reconcileID); nested spans re-derive from a captured base logger so zap never emits duplicate keys. GCP V(5) wire logs are enriched too.
Config: standard OTEL_* env vars (sampler, endpoint, protocol, resource attrs); manifests ship a commented-out example block.
Tests throughout (first use of tracetest in the repo); lint clean vs main (--new-from-rev); full suite green.

Plan: docs/plans/2026-08-24-1025-otel-tracing.md, execution log: docs/plans-executions/2026-08-24-1025-otel-tracing.md.

Left for a follow-up: CHANGELOG entry once verified live against a collector (verification steps are in the plan).

Adds OTel tracing across the operator without disturbing the existing logging system. - **Off by default**: activates only when OTEL_EXPORTER_OTLP_ENDPOINT / OTEL_TRACES_EXPORTER is set; OTEL_SDK_DISABLED=true and OTEL_TRACES_EXPORTER=none force off. No collector -> no SDK, no-op spans, byte-identical logs. - **Traces**: one per reconcile (root span + managed/replace/delete/status.patch sub-spans), per discovery API request (route-named server spans, incoming traceparent honored), per GC sweep; per health probe behind --trace-health-probes (default off, ~1/s/proxy volume). - **Child spans**: provider calls via a WithTracing decorator (mirrors WithMetrics, reuses the error taxonomy), k8s API calls via a parent-gated otelhttp rest.Config transport (informer watches/leader election never become root spans), GCP HTTP via the SDK's built-in otelhttp transport. - **Log correlation**: every log line inside a traced operation carries traceID/spanID (lowerCamel, matching reconcileID); nested spans re-derive from a captured base logger so zap never emits duplicate keys. GCP V(5) wire logs are enriched too. - **Config**: standard OTEL_* env vars (sampler, endpoint, protocol, resource attrs); manifests ship a commented-out example block. - Tests throughout (first use of tracetest in the repo); lint clean vs main (--new-from-rev); full suite green. Plan: docs/plans/2026-08-24-1025-otel-tracing.md, execution log: docs/plans-executions/2026-08-24-1025-otel-tracing.md. Left for a follow-up: CHANGELOG entry once verified live against a collector (verification steps are in the plan).

kacerr added 10 commits 2026-08-24 11:30:29 +02:00

Add plan: OTel tracing integrated with logr/zap logging 328877e000

Co-Authored-By: Claude <noreply@anthropic.com>

Align OTel deps at v1.45.0 and add trace exporter modules b1a16774bf

Co-Authored-By: Claude <noreply@anthropic.com>

Add internal/tracing: env-gated OTel setup, span/log helpers, decorators 6b34f68469

Co-Authored-By: Claude <noreply@anthropic.com>

Add provider.WithTracing decorator mirroring WithMetrics d15b06ec45

Co-Authored-By: Claude <noreply@anthropic.com>

Wire tracing into the composition root 8e4f3b9b13

Setup after SetLogger, wrapped rest configs (manager + kubernetes
provider), tracing-outermost provider decorators, and an explicit
trace flush after mgr.Start returns (os.Exit skips defers).

Co-Authored-By: Claude <noreply@anthropic.com>

Trace reconciles: root span wrapper plus state-machine sub-spans 896bf89a19

Co-Authored-By: Claude <noreply@anthropic.com>

Trace discovery API requests and enrich request logs ec7267b8de

Co-Authored-By: Claude <noreply@anthropic.com>

Trace GC sweeps and (opt-in) health probes bca32f10d3

Co-Authored-By: Claude <noreply@anthropic.com>

Enrich GCP wire logs with trace context from the request ctx 53c0d77ef5

Co-Authored-By: Claude <noreply@anthropic.com>

Add tracing manifests and docs; clean up branch lint findings aeb4115c72

Manager env block (downward-API resource attrs, commented OTLP
examples), architecture §10 + Decisions entries, README section.
Lint: goconst constants, gofmt, logcheck (Setup now takes its logger
from ctx via logf.FromContext).

Co-Authored-By: Claude <noreply@anthropic.com>

kacerr added 3 commits 2026-08-24 12:39:07 +02:00

Add plan: e2e test of OTel tracing against real Tempo 026acea279

Co-Authored-By: Claude <noreply@anthropic.com>

Add Tempo-gated e2e test for OTel tracing e691105f89

Co-Authored-By: Claude <noreply@anthropic.com>

Record live and negative Tempo e2e runs in the execution log f8b911f1db

Co-Authored-By: Claude <noreply@anthropic.com>

This pull request can be merged automatically.

You are not authorized to merge this pull request.

View command line instructions

Checkout

From your project repository, check out a new branch and test the changes.

git fetch -u origin feat/otel-tracing:feat/otel-tracing

git checkout feat/otel-tracing

Sign in to join this conversation.

1 Participants

Notifications

Due Date

No due date set.

Dependencies

No dependencies set.

Reference: kacerr/egress-proxies-operator#4