70 lines
3.2 KiB
Markdown
70 lines
3.2 KiB
Markdown
# Execution log: OpenTelemetry tracing integrated with logr/zap logging
|
|
|
|
Plan: `docs/plans/2026-08-24-1025-otel-tracing.md`
|
|
|
|
- [x] Step 1 — Dependencies
|
|
- [x] Step 2 — New package `internal/tracing`
|
|
- [ ] Step 3 — `provider.WithTracing` decorator
|
|
- [ ] Step 4 — `cmd/main.go` wiring
|
|
- [ ] Step 5 — Reconciler spans
|
|
- [ ] Step 6 — Discovery server
|
|
- [ ] Step 7 — GC + health
|
|
- [ ] Step 8 — GCP wire-log enrichment
|
|
- [ ] Step 9 — Manifests + docs
|
|
|
|
## Step 1 — Dependencies
|
|
|
|
Aligned the pre-existing indirect skew (otel core v1.44.0 vs otlptrace exporters
|
|
v1.40.0) and added the new direct deps in one shot:
|
|
|
|
```bash
|
|
go get go.opentelemetry.io/otel@v1.45.0 \
|
|
go.opentelemetry.io/otel/sdk@v1.45.0 \
|
|
go.opentelemetry.io/otel/trace@v1.45.0 \
|
|
go.opentelemetry.io/otel/exporters/otlp/otlptrace/otlptracegrpc@v1.45.0 \
|
|
go.opentelemetry.io/otel/exporters/otlp/otlptrace/otlptracehttp@v1.45.0 \
|
|
go.opentelemetry.io/otel/exporters/stdout/stdouttrace@v1.45.0 \
|
|
go.opentelemetry.io/contrib/instrumentation/net/http/otelhttp@v0.70.0
|
|
go mod tidy
|
|
```
|
|
|
|
MVS side-effects worth recording: `logr v1.4.3→v1.4.4`, `httpsnoop
|
|
v1.0.4→v1.1.0`, `grpc-gateway/v2 v2.27.7→v2.29.0`, `proto/otlp v1.9.0→v1.11.0`,
|
|
plus a `genproto/googleapis/api` pseudo-version bump. Full `make test` passed
|
|
against the bumped graph (`google.golang.org/api v0.292.0` and k8s v0.36
|
|
tolerate otelhttp v0.70.0).
|
|
|
|
Worth noting: the first `go mod tidy` ran *before* any first-party code
|
|
imported `otlptracehttp`/`stdouttrace`, so it silently dropped those two
|
|
modules again; the Step 2 tidy re-added them. The plan's semconv question
|
|
resolved to `semconv/v1.43.0` — that's what `sdk@v1.45.0/resource/builtin.go`
|
|
imports, so first-party code uses the same version to avoid
|
|
`ErrSchemaURLConflict` in the common path.
|
|
|
|
## Step 2 — `internal/tracing` package
|
|
|
|
Landed as planned: `tracing.go` (env-gated `Setup`, hand-rolled exporter
|
|
selection), `logger.go` (`Start`/`StartSpan`/`ContextWithLogger` with the
|
|
base-logger ctx key that prevents duplicate `traceID` zap fields on nested
|
|
spans), `reconciler.go`, `transport.go`, `http.go`, `options.go`; tests for
|
|
all of it (first use of `sdk/trace/tracetest` in the repo).
|
|
|
|
Two deviations from the plan's letter:
|
|
|
|
- **k8s transport gating is a custom RoundTripper, not `otelhttp.WithFilter`**
|
|
(`parentGatedTransport`): requests without a parent span bypass the otel
|
|
transport entirely, so the no-root-spans-from-informers guarantee doesn't
|
|
depend on otelhttp filter semantics for transports.
|
|
- **`HTTPMiddleware` must copy the route pattern back.** The logger-injecting
|
|
inner handler wraps the request via `WithContext` (a shallow copy), so the
|
|
mux records the matched pattern on the copy while otelhttp's post-routing
|
|
span rename reads the original. Found by the middleware test (span named
|
|
`"GET"` instead of `"GET /v1/things/{id}"`); fixed with `r.Pattern =
|
|
r2.Pattern` after `next.ServeHTTP`.
|
|
|
|
Also: both the transport and the middleware pass explicit W3C propagators
|
|
instead of relying on the global, so behavior is deterministic under tests
|
|
and when tracing is disabled. `Setup` tests reset the global provider to a
|
|
fresh noop per case — restoring otel's own default delegate triggers a
|
|
"Setting tracer provider to its current value" warning from the SDK.
|